…whatever the AI is doing, it’s qualitatively different from human behavior. The other piece that I believe is relevant is, AIs in human interactions appear to often prioritize pleasing the human over seeking the truth; while some humans do this, that behavior is widely considered untrustworthy.
That lack of trust is particularly pronounced when an AI describes its self-perception, because of how humans have often described AI self-perception in fiction; “predictive text” end up being a reasonable explanation for an AI whose responses are similar to what has previously been so described.
Someone likened the beasties to having a power tool that talks and I think that’s a fair analogy, both on up and down side