The word “want” is ambiguous here. We tend to formalize it by asking what it’s getting at, and then defining goal states that can be programmed into an AI (like capturing the opponent’s king).

But biological entities experience want first and foremost as a feeling accompanied by a relatively open-ended catalog of goal states that might address it.

Mother Nature can’t build goal-directed machines without those feelings, but we can. And we do. And then we conclude that after all the subjectivity those feelings create is an illusion.

But this is a mistake. As is the belief that because it needn’t be reduced to formal computational rules in order to generate the desired behavior, it’s not part of what it means to be intelligent.