LLMs are trained to predict what linguistic token should come next statistically, based on the prompt they're given.

That's not the same as having a picture of what your words refer to in your mind.

You can tease out the difference by asking LLMs to respond to a question for which such mental modeling is necessary, and for which simply connecting linguistic tokens won't work. For example, by asking it which side - white or black - would win more often in a chess match if the board were shaped as a torus.

And of course, LLMs are inert until you give them a prompt. They aren't the far from equilibrium process that minds created by natural selection are, so they don't have any feelings about what they're producing.

Put those two things together - an inability to appreciate what their words refer to, or to care even if they could - and you should obviously not expect them to produce good poetry.