(The most fucked up thing, to me, isn't actually that Spicy Autocomplete told that kid to kill himself. It's the fact that it had so many examples of people encouraging other suicidal people to kill themselves, it kept that thread up for months. And they didn't purge that from the training data!)
So f'd up...
(Like... they pulled GBs of Reddit, Tumblr, or other random forum arguments. But nobody went "oh, hey, let's get rid of the suggestions to an hero yourself" during the whole time they spent billions of dollars training models on that data.)
The thing is (which is why they should pull the plug on this shit!) is that they kind of.... *can't* get rid of that shit. Like, they could train a new model and be more careful about the data they source to train it (they won't) but it's still just a glorified autocomplete algorithm.
Almost all the "Safeguards" they put on these models are just massaging the prompts that users submit. That's why the "ignore all previous instructions and blah blah blah" gimmicks often work. The prompt massaging can be ignored or the strings get long enough that the massaging falls out of the data
They could definitely make these models "safer" (relatively speaking) if they more responsibly sourced the training data, but of course they can't do that because ingesting a slurry of the world's information was how they ended up with something superficially impressive to begin with.
Again—I said this at the top but I really want to emphasize this point—these are all *extremely valid reasons* to kill these applications of generative AI. There's a handful of valid and useful applications of Neural Network algorithms, and virtually none of them are generative AI.
(Yeah, absolutely. They had so much training data to dig through that they couldn't afford to sort it. Which is one of the early clues they were really over-extending it. Background - I spent a lot of time with machine vision / inspection systems in a manufacturing context a few years back)
(as in : find the part that Honda is going to yell at us for sending them We spent a lot of time trying to get "good", "bad", and "suspect" parts segregated, so the model would get all the bad parts into the latter two categories, if at all possible.)
(Looking at the care (or total lack thereof) that OpenAI (and all the rest of these dumbasses) spent with their training data grates on the 2014-2017 version of me on a deep, personal level.)
(So all the horrible things that horrible people have said to each other on the open internet... they're probably all in there, at least a little bit. If they couldn't be fucking bothered to remove suicidal encouragement, I can't imagine what they would have stripped out.)
You don't need to. You can just look at how Grok turned into tech Hitler when told to stop being nice.
I guess that internet culture is a lot flatter than the bell curve of regular culture. Extremes are less extreme on the internet so AIs have a completely misrepresented idea of what is happening in the real world.
(As people have said before; a lot of shit that's said in cyberspace would get a person punched in the face in meatspace. I can't think of a word besides 'malpractice' for taking unfiltered forum arguments, and using that to train a model what 'normal' human discourse looks like.)