We are accustomed to thinking that anything generating language must have a mind behind it. But that's not true, not anymore. Not saying true AI is impossible, just saying the standard for making that call has to be much much higher than you're making it. en.wikipedia.org/wiki/ELIZA_e...
See also: bsky.app/profile/kjhe... Nobody is home. It's not thinking, it's extruding homogenized thought-like product.
I don't necessarily disagree that LLMs don't "think" in the sense people think, but "count letters" is a pretty obviously terrible example given what the inputs and outputs of LLMs actually look like
imagine I locked you in a room, and in this room is a terminal that asks you questions in english that you answer in english. unbeknownst to you, the people asking the questions are all speaking chinese, and both the inputs and outputs are run through a translator on the way in and out
you receive a question "how many characters in the word for "America"? you may well be able to answer the question, if you've seen enough inputs to deduce that 美国 likely has 2 characters, but I don't think being unable to do so says much about your intelligence one way or the other
what we're asking LLMs to do when we tell them to count letters in words is really to reverse-engineer the token embedding process, which is not an easy task in the slightest! I'm frankly surprised they're as good at it as they are