I don't necessarily disagree that LLMs don't "think" in the sense people think, but "count letters" is a pretty obviously terrible example given what the inputs and outputs of LLMs actually look like

imagine I locked you in a room, and in this room is a terminal that asks you questions in english that you answer in english. unbeknownst to you, the people asking the questions are all speaking chinese, and both the inputs and outputs are run through a translator on the way in and out

you receive a question "how many characters in the word for "America"? you may well be able to answer the question, if you've seen enough inputs to deduce that 美国 likely has 2 characters, but I don't think being unable to do so says much about your intelligence one way or the other

what we're asking LLMs to do when we tell them to count letters in words is really to reverse-engineer the token embedding process, which is not an easy task in the slightest! I'm frankly surprised they're as good at it as they are