These things don’t think. Their answers aren’t the product of reason but the output of a probabilistic algorithm that finds the next most likely word based on a library of weighted associations. It doesn’t understand what it says. It’s hyper-advanced autocomplete.
If you ask it “can you feel?” it’s not engaging in critical self-analysis—it’s providing the most likely series of words that would be the response to that question, based on its available weighted data set built by consuming human-written material without the consent of the authors.
A million monkeys hammering on typewriters could eventually produce the works of Shakespeare but that doesn’t mean they understand it or that they Wrote It. This is a similar mechanism, a more guided and sculpted version of monkeys hammering typewriters.