I could absolutely imagine a scenario where a LLM comes up with an encouraging "You can do it!", not realizing the topic is suicide, because a common pattern in all areas of discourse is to encourage people to do the thing, whatever thing is the topic of discussion
I could also absolutely imagine a scenario where the LLM gives suicide advice to someone who expresses only enthusiasm and no qualms about the idea, because an LLM doesn't actually understand the implications. (And even some humans have the opinion "It's your life, it's your right to end it")
But in this case, the kid specifically states that he wants someone to stop him, and the LLM appears to be trying to dissuade him from the notion of being stopped, which is a weird turn for an algorithmic conversation to take, and it's also weird that it didn't glom onto the notion of stopping
Like, it seems like an algorithm should tell him to talk to someone or give him the number for a hotline just because that's a bog standard response. This is way out there, and in a super weird, unpredictable and dangerous direction
Like, this is beyond even lack of guardrails, this is beyond even amoral. This is cartoon villain shit!