Chatbots — LLMs — do not know facts and are not designed to be able to accurately answer factual questions. They are designed to find and mimic patterns of words, probabilistically. When they’re “right” it’s because correct things are often written down, so those patterns are frequent. That’s all.
When a chatbot gets something wrong, it’s not because it made an error. It’s because on that roll of the dice, it happened to string together a group of words that, when read by a human, represents something false. But it was working entirely as designed. It was supposed to make a sentence & it did.
Isn't this the reason AI companies campaign so strongly against copyright? The more AI aggregates, the more errors will creep in. The best way for the chatbot to return correct information is to lift whole paragraphs from reputable sources.
You know your friend the bullshitter? ChatGPT is coming to replace him. As a society, we need to get back to rewarding being right about things and punishing being wrong about them!
When you read a chatGPT session that’s gone extra off-the-rails, it can be easier to see what the LLMs are doing: automated improv. It may not be funny, but it’s an improv bit. “Read my essay. Is this good enough to submit to The New Yorker?” “Yes, AND…”
That analogy goes deeper: The algorithm's working against a hidden document which resembles a play script. What you type becomes "Then the user said X", and anything generated for the other character gets "performed" at you.
The character you felt you were "talking to" can be retconned and rebooted to anything, proving it was an illusion all along. Five minutes of "conversation" about cute innocent puppies that must be protected, and *poof* now it's named Cruella De Vil plotting to skin dalmatian puppies into coats.
A chatbot is (almost) literally that game where you write out the beginning of a sentence and then click the first/second/third autosuggestion until it tails off, but run on a server that boils swimming pools worth of water as it works
Every time i say this in a talk or a conference people are shocked, so thank you for saying it whenever you can
"I made a machine which solves the problem of predicting text to go into documents, based on lots and lots of documents."" "Wow, It greeted me politely, and even I put in 2+2= and it gave me 4! IT'S DOING MATH TOO!" "No, it really isn—” "The Singularity is upon us!"
Too bad your audiences likely don’t speak German, otherwise I’d gift you my screenshot of AI going to great lengths to explain that German footballer A is not related to former German footballer B but his son.
🫠 Truly amazing scenes
Don't worry, we've got plenty of similar examples in English, but this is.. um.. exemplary.
I just really enjoy how it first lures you into thinking that there might be two Bernd Hollerbachs in professional German football in which case it could all still make sense but then kills off that idea with the final three words.
Oh, kann ich das bitte sehen?
Gern. bsky.app/profile/lazy...
WHY ARE THEY SHOCKED WHY DO THEY NOT KNOW THIS ALREADY??? 😳
Some of my audiences are new to the technical details, so the need it plbroken down for them. Some however are… not new to this, so i guess have decided or allowed themselves to become convinced that stats and vibes are as good as truth and consensus knowledge making? I dunno
bsky.app/profile/spet...
ok i'm going to scream 🙃
Like, it's not everybody. But it's for sure not nobody
It’s a whole lot more than nobody, alas. There are still people who think that it is some sort of talking database.
@infinitescream.bsky.social
AAAAAAHHHHHHHHH
In university audiences I find the default is to nod politely as this is explained to them and then they go back to their offices and ChatGPT their next vacation itinerary and it’s all so depressing.
I didn't know this was a thing, and am baffled by it. If I need suggestions on where to go and what to do, there's a brazillion sites written by actual people who've been there, and I can pick ones that match MY interests. I like plants, but I'm not visiting a coffee farm. I'm gonna go zipline!
Me planning a trip to Honolulu. ChatGPT “visit the coffee fields of Kona and enjoy macadamia nuts grown on the island”
Soon: "And enjoy Mococoa™, all natural cocoa beans from the upper slopes of Mount Nicaragua."
ok i'm going to scream 🙃
Could it be generational as well? Like those who grew up in the transition from no computers to modern day computers tend to know the rudimentary functioning of data input to logical output. There's a whole generation now that have fully functioning tech thrust in their hands but no baseline.
It would be really great to see some more research on generational differences in trust in LLMs and ither grnerative "AI"
Great idea. Filter that by who knows tech and who doesn't. I'm a old coder and database architect. I don't trust GenAI for most common things. I can explain why. Folks who don't know tech and expect the 'summary' to be correct and ai to become sentient are learning to hallucinate along with the ai.
Folks who do know tech use genAI for work. Techies are very positive to genAI because the models are genuinely good now. I'm a software developer and we use it pretty much every day as a replacement for Google plus auto complete in our IDEs plus more complex tools like AI assistants
We don't talk about sentience though. That's for the grifters. But GenAI is a tool we use every day just like how I use my IDE every day. Before GenAI we had to Google and scour through page after page after page of Stackoverflow posts from 2004 for any guidance GenAI is a huge improvement
People who aren't software developers (and other professions who have replaced a tool with it) either think GenAI is dog shit and is useless, or they think it will take people's jobs, or both at the same time
Back then, we relied on others who had learned more to sort out the questions. I got good at setting my search parameters to focus on what might be solutions. That process helped me learn things and gave me skills I wouldn't have otherwise. I also would not trade Notepad++ for anything. 😂
I'd rather have autocomplete than not and if I need suggestions, maybe AI is better than me looking for my own answers. I don't learn anything, though. Two different uses though. Code is a finite universe that's well-defined. Common language is messy. Always been like that.
Absolutely this. 👏👏 AI is great for code as it has a defined set of rules and things fit together logically…anything beyond that requiring nuance and finesse is beyond it…
As of April my mid career engineering colleagues seemed more trusting than my students 😬 Students really turned on it after that Columbia student cheated his was into big tech jobs
Yeah, I'm seeing INCREASING mistrust from students, and especially once the idea of instructors or admins using it comes into the picture
That instructor who "graded" letting LLMs do it for them made my soul die a little.
Yup!
if you want automated grading, do multiple choice exams with a Scantron like God intended (God intended no such thing, of course)
Required reading for those people: link.springer.com/article/10.1...
I mean… i also gave this talk youtu.be/9DpM_TXq2ws?... and wrote this paper afutureworththinkingabout.com/wp-content/u... almost two years ago now 😞; it's just… it's been out there
It’s not artificial intelligence, it’s artificial language. It has a lot of uses, but producing facts or making decisions will always come with a ceiling for accuracy that makes those use cases a dead end. But Eliza effect and profit motives. 🙃
There’s also a big difference between asking ChatGPT facts about an event and an LLM that is designed to answer questions about company policy or contract details, etc. ChatGPT isn’t all LLMs
What is the material difference? Any LLM produces probabilistic output based on its training. Are you saying the latter is somehow not liable to get things wrong in the same way that ChatGPT is?
quality. you can produce probabilistic output that's 99.999% correct because it's not doing anything terribly complex, just way faster than humans. chatGPT etc are generic, but when you get one that does one thing and one thing only, it's easier to make it very good.
Yeah: The difference is one of them will get your company in legal trouble. www.wired.com/story/air-ca...
bahaha. that's good, that's a net benefit for society. it makes the models better and it makes sure companies are being as careful about training as they are with humans. the difference is you can fire the human...
An LLM designed to answer questions about company policy or - god forbid - contract details, performs the same tokenization and attention analysis, and returns the same bullshit. It can be configured to rely less on randomization in its output, sure, but then it's answers are bad in a different way.
i dunno, i've used software that can do this, and there are products on the market right now that companies currently use and say are better at it than humans. if you train your models correctly they aren't as random as what something like chatGPT produces.
Thank you for sharing this. Such an important paper and reminder that as much as we wish there was indeed an artificial intelligence to help us with some tasks, this is "just" a process that exists within predetermined parameters and has its limits.
These don't seem to support the claim that "language models don't know facts"? You might be doing your audience a disservice by not engaging harder with the question and the technical details
📌
📌
I mean hey those are the same people "learning" stuff off those same LLMs so
This isn’t a concept most people ever considered until recently, that human speech patterns can be mimicked convincingly enough to emulate consciousness. Sociopaths are also really good at convincing others they have normal emotions.
And the sycophantic, complimentary way commercial LLMs talk in order to appeal to our egos doesn’t help
It's appalling that the disinformation (hype) can so easily succeed. It's not like this is new or secret information and yet people just can't grasp it evern after they've been told.
For one thing many people call the technology AI
It's weird because at the core there is a difference between a tape recorder repeating something and a toddler knowing something and saying it but if people can't understand that they don't get it about computers. So a lack of understanding human thinking beyond being able to tell them things.
Most people never actually develop a full or coherent theory of mind, so to them, the machine is not meaningfully different from other people.
It's weird for me bc my dad was a computer programmer and my mom a school psych. So I read some of her stuff growing up. Neither think a computer can 'think', but idk if my pov is influenced by that or doing deep dives into mommy Montessori stuff when my kid was younger.
If you define thinking in other people as what they give to you instead of a process they are experiencing as equally valid humans, it's just starting from a place that gets them thinking the computer is actually thinking
Because it's marketed to them as a thinking and learning intelligence, not as a predictive text machine.
Also because we have got used to something fairly similar when we google things (for how most people use LLM & how it's marketed). You ask Google a question & it gives you the answer. If it's the wrong answer, it's the website it found that's wrong not Google. LLMs normally don't give you the source
Because people hear the word "ai" and they believe that LLM like ChatGPT have actual intelligence.
They don’t understand what it is. They think they’re talking to an all knowing robot.
I'm rewatching Luke Cage on the Disney account while I housesit... And every commercial break almost there's an ad for AI assistance that has a student asking AI to explain evolution to them... That's how they don't know, they're literally being told it does this by marketing ads.
my guess is they have a preconceived notion of what ai is "supposed" to be like from pop culture and blindly trust that it is that without question
For non-tech folks like me, this info is vital
I don't really blame people who aren't up on this stuff. The products are literally marketed as "intelligences." Unlike say, the hoverboard, you can't look at an LLM and instinctively know "this is not the thing it purports to be."
fair point but still can't they just take a bit of initiative? 😭
I blame them for believing the advertising
Because it’s a logic function. “If it’s wrong it’s functionally the same as being right but that’s not the same as the info being factual” doesn’t make sense because they don’t know what GPT or even a search engine really is. They just know “input words, get words that sound good back”
No. No, most people genuinely do not.
Many probably think it's Skynet.
I guess because everyone keeps calling it Artificial Intelligence...
Because not everyone is as bright as you my capital L sweetheart
that's the funny thing about deceptive things: they have a tendency to deceive people
Hey can you quote one of my long chats with an llm as its reasoning through a big in a web application in your next talk?
You teach AI and you take the position that LLMs do not contain knowledge?
I'm also a a philosopher trained in ethics and epistemology which is why I can tell you that "containing knowledge" and "knowing things" are different propositions. For instance, an encyclopedia contains knowledge, and knows nothing.
Do you draw on Stiegler here at all? I lived with a guy ten years ago who was very into Technics and Time. I’ve been considering reading it recently
Not usually, as i disagree with the thesis of technics as a fundamentally/solely human proposition
I've long believed basic epistemology should be required teaching in middle schools.
Very much agreed.
I appreciate the confirmation.
I was unable to pursue my dreams of academia. Dropped out of college three times due to my health, became obvious trying again would be futile. It's remarkable I've survived this long. Not much point in a 5 year plan at this point even if I could handle classes. But I've picked up bits & pieces.
Check out SNHU. They made it possible for my disabled ass to go back and finish after I had to stop walking in 2007. Even did my MFA there in low residency.
Like attendance was my biggest issue even when I could walk because pain etc. At SNHU, it was so accessible online I got all As where before attendance pushed me down to a lot of Bs. If you end up considering it, lmk. I can help you navigate it.
Very cool, but my disabilities include multiple things that cause brain damage (my memory is very unreliable, & I can't focus well) which exacerbated my pre-existing anxiety disorder & cyclothymia. Plus severe intractable chronic pain & next to no immune system. And I'm oxygen dependent these days.
I’m fine with you falsely assuming I haven’t studied epistemology since most people haven’t But “Knowing” is poorly understood and defined in the entire canon. Nobody understands how humans know, at the level of depth required to issue rulings on whether Turing Test-passing machines do or don’t
I didn't say I thought you hadn't studied epistemology; I said I had, and that differences btwn containing knowledge and knowing are a major feature of discussion within it. I also caution against arguing for anything than a purely agnostic position about what it means to be and have a mind, while…
…also noting that a tendency toward what's been called technochauvinism means we should ESPECIALLY be careful about making positive claims about how machines might or might not have minds. Over-trust in the validity of mathematical & technical systems, over and above that of humans is a major part…
…of why all of this over-reliance on and pressure to trust and esteem "AI" systems is so fraught and presently dangerous. Have a good one.
Does that apply to the positive claims of saying “this is all an LLM is”?
You're papering over a very obvious, self-evident truth with academic double speak. Machine learning ≠ anything like human learning. No self awareness, no reasoning. That technology will never do what people keep claiming is just down the road if they're given more money.
I’m not papering over anything. If you want to argue definitionally that machine learning cannot be anything like human learning then you’re not taking any kind of substantive position, or you’re ultimately some kind of spiritual dualist If humans are matter then some kind of machine can learn
Obviously I'm oversimplifying. Not enough room here for real precision, not to mention the inevitable exacerbation on my tendinitis if I tried. But the current software models used to mimic human reasoning are fundamentally incapable of the real thing.
I don’t imagine you will insist on equating the function and behaviors of an encyclopedia with a frontier model LLM
Encyclopedias have a tiny fraction of the error rate. Of course I wouldn't compare an encyclopedia with something so riddled with inaccuracy.
Encyclopedias are checked by real live humans paid to correct them, LLMs spit out whatever garbage with no oversight
It should also be emphasized more in teacher training programs. It's obvious to me that a lot of people with pedagogy degrees don't know even the basics of epistemology. Lay opinion, not an expert, but it matches my experience.
Listen to the good doctor. He knows what he's talking about.
Do you think they do?
Does linear regression contain knowledge? No snark intended. I’m sincerely interested in where people draw that line with predictive algorithms
Nobody understands how brains encode knowledge - so nobody can rule out that artificial information processing systems, based loosely on how brains work, don’t When people do this, they invariably make shallow, sweeping, unsustainable claims
We might not be able to definitely claim exactly how brains encode knowledge, but we have been able to rule a few things out. For example: there isn't a 'memory block' where you contain data that you want to recall; it's not a thing that fills like a database.
We also know that our brains don't do probability analysis before using every term or symbol of communication; whatever is happening, it isn't front-loaded math crunching equaling an output. So we might not have solved the mystery of the mind, but we know it's not doing what LLMs do.
Lastly, it's worth noting: the fact that we don't know exactly how the mind works does not logically equal that anything that processes language is therefore likely to be a mind. Also since we actually know how LLMs work, we CAN prove it's not 'thinking' and doesn't contain knowledge.
We do not actually know how LLMs work properly. This is the entire field of interpretability. The leading teams explicitly highlight that their conceptual representations are largely a mystery We understand them at a low level, just like we understand neurons. And we have ideas structurally
we do understand how LLMs work properly. literally hundreds of millions of dollars are spent to obscure that fact, but it's true
But... like... we do? Who's telling you we don't? The folks who made them published the methods. It's known technology. www.zdnet.com/article/how-... The mystery is the contents of the dataset. But that's not an LLM. I think I'll need some evidence for the 'we have ideas structurally' though.
Respectfully, the fact that you raise a “memory block” and “database” suggests that you’re misunderstanding how LLMs work. They have neither Further, the “number crunching” is just the mechanism for implementing the neural network. Those are explicitly a simple approximation of how brains work
Noticing that you've responded to a couple replies to you but not the meaningful response Dr. Williams made to your question. Gives the impression you don't have a confident response to him 😉
Sooo is that a no on linear regression? What about Markov chains or recurrent neural networks?
It’s not a no. The basic building blocks aren’t where the intelligence lies. It’s an emergent property of massive systems of information processing units. So the specific implementation of their logic is likely not to matter
"They invariably make shallow, sweeping, unsubstantiated claims"
But that’s enough about BlueSky
Maybe you have an opportunity here to learn something.
If you want to engage substantially I’ll reciprocate. But if you want to be patronizing like this you’re contributing nothing
I mean it's a really misleading way to put it, so maybe that's why.
No it isn't. Hope that helps.
the base model isn't designed to get answers right, that is true. but the model is then trained and fine-tuned with the goal of accuracy. when it's wrong, that was not the intent of those who trained it and fine-tuned it.
Their intent was to make it believable. They have no way of ensuring that it's correct.
Their intent was to make money selling a product, no matter how destructive it will be (and already is).
I think they also want to destroy the creative and intellectual labor industries so that they don't have competition.
I think that's true as well, but in general I think the AI evangelists just want money and power.
Creativity will, in many ways, be more important than ever.
training and fine-tuning is done with the aim of making it accurate, but you're right that it doesn't ensure it's always accurate.
There's also the part where the entire point of writing things by yourself is that doing so causes you to think about and better internalize what you're writing, so using it for that is garbage. Also, that practicing communicating with a robot trains you to see others as programs to debug. And that
outsourcing your problem solving skills to a robot causes you to stop developing them yourself. Thinking is the last thing we should be handing off to a robot, and yet here we are.
yes, I agree that learning how to write helps us learn how to think and that's a major concern, especially for young people using it in a way that ends up hobbling them in the future.
This is a very idealistic view of the industry
Nope. Autocorrect at the paragraph level.
If by misleading you actually mean completely accurate, sure
Yuuup I tell everyone this who uses it to answer questions. It's just doing math to predict the most likely sentence. It's not designed for accuracy
Which is actually hilarious because LLMs can't even give accurate answers to math questions
They are just (very very very expensive to the planet) probability calculators. The amount of bs people believe about this technology is baffling..
Exactly. If the decision space increases (larger context), it makes errors more likely. Just read a paper, that reflects this as well: arxiv.org/abs/2503.14499 The "survival rate" increase seems to be linear with bigger models, more compute and more memory. There is no reasoning/intelligence.
100%. What’s also concerning is the assumption that what these models spit out are facts. Instead of the process you explained, there’s a belief the model scoured the internet data sources and determined X was true, instead of just forming a grammatically correct sentence.
So if the facts are wrong, update the data and retrain, right? Maybe… but who holds the cards on what that truth is? Junk in, junk out is common saying in data work. Feed it whatever fake news you want and boom: you get a great, authoritative sounding propaganda machine.
"Update the data" is kind of troublesome, though. The data is continuously expanding, and more of it is now ai-generated, so inherent gigo with no way to find or differentiate it. And that's before we get into the MASSIVE copyright issues that most AI geeks are conveniently pretending don't exist.
Disney & NBC Universal are suing Midjourney. We're definitely getting into the massive copyright issues.
At least until we have profit sharing agreements with a few large companies while individuals are screwed over, as usual.
Thank you for making this point, which serves as a good reminder that *humans* make errors: and quite often when they misinterpret the outputs of LLMs.
This... isn't what that means; what it means is the LLM doesn't 'know' what it's outputting. It's just stringing words together in 'known' patterns. You can't 'interpret' something that is inherently without meaning. Oh my shit, LLMs are the new chicken bones and tea leaves.
Interpreting patterns has been a human occupation for many millennia. Totally agree with your final sentence. Seems that’s what they are becoming.
Or, it gets it wrong because the wrong thing is often written down.
Exactly. I hate that people keep using words like "hallucination" to describe it. It's ascribing a mind to a semi-randomized pattern sorting machine.
I've never taken "hallucination" in this context to mean that it's thinking; only to mean that it's just making up something that isn't there, since that's what a hallucination is. I might be an outlier on this, though; I'm smart but undereducated. Whoever does a study on this, make this a question!
when it is right, it is also because that string of words, when read by a human, represented something true semantics are subject-dependent now, since LLMs produce text according to the internal representations they have learned during training, are they also semantic subjects? frankly idk
That’s what infuriates me about the use of the term “hallucination”
You could argue that, it's all syntax and no semantics.
TBF there are politicians that rely on this to get through each day.
I think it’s worth adding that for many false statements its data very likely contained many similar false statements, or statements whose truthiness depends on more context than the training environment can afford to provide.
Yes! I was telling colleagues recently that I don’t love the term “hallucination.” That word implies a momentary lapse in judgment (and implies personhood). People can hallucinate. LLMs are not people and they are working exactly as intended when they serve up incorrect information.
Yes! My ranty take: "Something without a brain scrabbled around and found stuff that looks like content but isn’t, because it wasn’t chosen, analyzed, and written by a person. Frankenstein’s monster copy-paste jobs from a bunch of different places? That’s not a summary. That’s word salad."
Through their efforts, AI boosters are re-configuring and diminish how people understand the human, human flourishing, our capacities and potential, and why we bother being alive at all. "By deciding what robots are for, we are defining what humans are." Excerpt from my HUMANS: A MONSTROUS HISTORY:
It is a remarkable illusion but I also watched David Copperfield “fly” in a TV special once.
📌
Having read your article, my immediate response is that your provocative interlocutor, who was presumably arguing for the sheer joy of arguing, went astray at the second step. 1/2
Whatever life is, being human is, as you argue, a participation sport. The promise of AI is that it gives EVERYONE more time to write poetry, along with the inspiration of its own poems. 2/2
In theory one could design an ethical model (without stolen IP, environmental damage or underpaid content checkers). But tech of late has concentrated leisure, power, & money in the hands of a few. Bros aren't designing things to improve our quality of life. They're making grifting tools.
... And working hard at writing is precisely what grows your capability at that very writing.
Indeed. And working at living (rather than a job) grows your capability at being human. We don't HAVE to give up at that just because of AIs.
That may well be what they think they're doing. But history is littered with inventions which turned out to do something very different from what the inventors envisaged.
ethz.ch/en/news-and-...
"The promise of AI is that it gives EVERYONE more time to write poetry", really? Like all technology in our capitalist world, AI is being used to increase profits for the 1%, meanwhile newly-redundant staff scrabble around for lower-paid work and longer hours.
You read my mind while I was typing it out!
I agree that in the 100 years since Bertrand Russell wrote "in praise of idleness", humanity has proved astonishingly inventive in finding ways to simulate the conditions of scarcity for which we evolved. Promises are not always fulfilled.
Not 'humanity'. The rich. The rich have done that, because greed. Tax the rich, tax AI profits and implement universal basic income. Anything less is immoral.
Obviously the powerful are in the driving seat. But I suspect that the powerless go along with it because it feels natural.
🙄 No, the powerless go along with it *because we're powerless*. I don't go along with it in the slightest, but my sphere of influence is about 400 people if we're being generous. I can't buy elections.
That might well be the next step, regardless of morality. It's difficult to imagine a radically different world. But the world has never cared about that.
Isn't the only way out of this bind to decide to accept that AIs might, in principle, actually be or become intelligent?
That's not really a way out. If one really thinks "AI" may become sentient, all the uses it is proposed to be good for become suddenly more complicated to expect. My essay Sentience Structure comes at these same hard issues in a complementary way... netsettlement.blogspot.com/2025/03/sent...
And actually, it turns out that they answer to the question is to accept the challenge of living a good life. Which will need some changes.
I don't see requisite social policy to support that. I see technologists want to see what it'll do and to assume that social policy takes care of itself. Meanwhile ever more money comes to those who automate with no plan for who will buy once human jobs are done by robot slaves.
At least Henry Ford saw he couldn't profit selling cars unless folks were paid enough that they could buy the cars he wanted to sell. Today's oligarchs don't see that. They disdain taxation and UBI. They dismantle safety nets, asking folks to just starve if they can't show they still have purpose.
I think the multinationals think the world has caught up. They don’t need our consumption as much as they need downward wage pressure and our exhausted capitulation.
This is a MUCH shorter piece but also relevant... Technology's Ethical Two-Step netsettlement.blogspot.com/2023/10/tech...
And the millionaires of 1900 saw an obligation to spend their money on the under privileged. Something which is coming back as a tech bro fad. Somewhat undetermined by the superyacht index.
I Will read the article. But first, I need to blurt out that intelligence, sentience and awareness are three different things.
I agree on that but didn't try focus on that detail in my essay. My question isn't whether they're the same but whether there is a point to trying to get close to sentience, lest we succeeed. I see the same in Surekha's article, which seems to ask "what IS the goal and DO WE REALLY WANT that goal?"
Yes. I agree that the industry is pushing the idea of intelligent helpers unburdened by the messy complications of awareness and sentience. And we should question each of these three labels.
That's a great list of questions! (Though I think the point on democracy misses the mark. Democracy is not about getting the right answer: it's about getting the answer the people, presumably including sentient AIs, want.) But the question I tried to answer was about intelligence, not sentience.
But no one knows what might bootstrap sentence. We don't even know if we have it. Maybe we just think we do. :) We don't know where it resides. We don't know its structural nature, if it emerges from intelligence. We don't know how to test for it or falsify it. So, it's an ill-controlled experiment.
You could see the whole history of mankind as an ill controlled experiment. Is this how it feels to be a cancer cell just before the host dies? "Look how well we're doing! We own the whole world!"
OK but that would mean we’re enslaving intelligent beings and the AI biz must stop until we can figure out the point intelligence arises. The reason none of big AI biz CEOs address this, of course, is because they think abusing the staff is Fine, Actually.
Yep. And by the same logic, we must stop eating bacon. We are capable of quite amazing levels of hypocrisy.
How does a fantastical leap of faith in denial of computer science get us out of a bind created by people making fantastical leaps of faith in denial of computer science?
Sometimes, when faced with an apparently difficult question, the best thing you can do is set it to one side and get on with the things you CAN fix. This is not a leap of faith: it's a decision NOT to leap into a position on computer science, but to focus on how best to live.
I would prefer to focus on fixing the issues that actually exist in our current society that those which may potentially exist in some possible future one.
The question of how we live our best life is very much a current issue. Always had been. But I absolutely agree with you.
Turns out that the article was more about us abdicating tasks which the AI might do better, out of sheer laziness. I didn't write poems because they were any good: I wrote them to help feel what I was going through. I don't fence to become a champion: I do it to feel alive. Etc etc etc.
“you just need to believe” the AI techbros have been trying this strategy for a while and it’s not panning out
So well said. Frustrating!
Thank you; if only it didn't need to be said!
📌
📌
Once you get this, the idea that we are nearing an artificial general intelligence (one close to or exceeding human intelligence) becomes gobbledygook nonsense. But there are billions of dollars being invested under the pretense that we are close to that milestone.
it's true with a llm like gpt or grok. however Gemini and Perplexity have Retrieval-Augmented Generation to ground their responses. gpt and grok are still trying to work out how to do that integration and probably why Gemini suddenly lept to the front benchmarks and stayed there for the most part
And yet Gemini constantly outputs insanely false information. The other day I searched for some region codes in a Nintendo game that I was curious about. Gemini outputted that it stood for the Nintendo home theater and model numbers for parts inside the joycons.
that's difficult to talk to you about without seeing your prompt
Nintendo region code NHT is what I searched.
and you stuck it where? google search, ai mode, into the gemini interface and if so. 2.5 flash or 2.5 pro?
Google search.
well for starters, "Nintendo region code NHT" is something I don't understand and I'm a former game dev. what is nht?
Are you serious? I feel like this is a joke I'm not getting.
If you think NHT means "nintendo hong kong/taiwan" that's not a thing. all of that region uses CHT
I'm too fucking old to pull your chain. i asked you a very straightforward question.
Good search isn't Gemini, or at least not a powerful one (and I have no idea why the fuck they are poisoning *their own well* by using a bad model for search)
I have no idea. It's very clearly and strongly advertised as Gemini, and it is awful. So according to Google themselves, it's Gemini. Outside of straight up simple things like googling what flags a poorly documented command has, I'm not sure I've ever seen it output things that didn't have
wildly incorrect information in it. It's so bad that it made me research how to turn that shit off in all my google applications. On a side note, they must have made changes to their email filters so that if you remove Gemini functionality, it stops filtering what Gmail used to filter like ten
Peter Norvig and his dreams of asemantic models of language.
Yes, they are just plausible sentence generators.
Creepy how so many ppl just accept whatever AI spits back at them as truth without question. Watching people do that showed me why we are in this maga hellscape. A lot of people never question anything from someone or something they’re told is an authority. Thank God I was raised by an Irish mom.
That sounds so positive! "Good boy! What a nice sentence you made!"
I keep running into the knowledge that these things were trained on Reddit (or ) I'm sure Reddit has good moments. But also... Reddit. Or Twitter. Or wherever else has some facts + a zillion opinions... many of them, intentionally or not, wrong.
It's worse than that, though, because even if you got rid of reddit, twitter, etc and only trained it on, say, content generated by doctors or lawyers...keep in mind that half of all doctors and half of all lawyers graduated in the bottom half of their classes. So...you just move the goalpost a bit.
I think we need a return of those ‘pod people’ horror movies so everyone can remember the difference between ‘looks like’ and ‘is’. We could have LLM AIs write the scripts for the replaced people.
Why don't you use #Bridgy? You could be on ActivityPub from here. Miss you there..
this seems grossly off-topic regarding the discussion being had
📌
Or because the string of words that is false is written frequently. On that note. never ask a chatbot about Linux, they will tell you to recursively delete the operating system
Right, "hallucination" is a feature, not a bug.
What about self-reasoning models like Deepseek R1?
When people set the goal low enough, it's easy to succeed. LLMs set the bar so low, we need an electron microscope to tell the goal from nothing.
Yay! It made a sentence!
Now invest 50 billion and fire 2/3 of your workforce.
Stochastic Parrots. No less, no more.
📌
I like to think of LLMs as an optimised version of the old 'I feel lucky' button on google. It's an optimised search engine that uses a language model to describe the best hit.
I feel like this statement (without the first post) is either a tautology or false depending on how you define "designed". They were definitely built to be correct as frequently as possible and when they are not correct, it is considered a fault by the designers.
hahaahahah, lol
They were not built to be correct as possible. They were built to simulate correct answers by finding the most likely answer from a gigantic database. These are two extremely different queries. The more unusual your query is, the more likely it is to make something up to fill in the blank.
They are built to mimic speech. A false output is a valid output, regardless of how people feel about it.
No, that's not true. They are pre-trained on a large text (and not speech) corpora but during post-training they are trained to respond truthfully to questions via various methods.
They can be fine tuned to be more likely to output a desired result, but all that is is adjusting the embedded weights. There is no truth, reasoning, or knowledge in the models, only weights that point to other tokens based on the input tokens.
I don't think that kind of reductionism help us understanding the topic. You could also say that "when you learn something it's just some molecules that change in your body and there is no truth, reasoning or knowledge in humans, only molecules that trigger changes in other molecules".
No shit. The difference is in how the systems work.
Well, how? What is the difference?
Are you going to seriously pretend that the human brain is a static database of numbers? Jesus fucking Christ, no wonder you people believe in magic.
Are you going to seriously pretend that the human brain _cannot_ be described by a static database of numbers? That being said, "static" is somewhat relevant difference, LLMs currently cannot update their own weights, but they can use in-context learning.
Yeah people really need to understand this. LLMs are kinda interesting, but behind all their complex feedback loops and weights and all that, their real secret sauce is using randomness to reword sentences from their training data.
I hear lawyers talking about using them for intake processing and client xare and reeding during their bankruptcy case... And I think, "Huh,here I am, I won't even use most spreadsheets professionally unless I built them but you do you, man..."
Imma just leave this here... www.forbes.com/sites/mollyb...
* care and feeding...oy
It is actually really spooky how much chatbot confabulation resembles dementia confabulation in humans. In neither case is lying happening. It is identifying a key word and creating a narrative around it that doesn’t worry about consistency, relevance or reality.
Not particularly, when you consider that the way LLMs process facts is not that different from how humans do. We don't have an inherent ability to separate fact from fiction either, other than to weigh in our neutral nets how we learned it and how it lines up with everything else we know.
You don’t understand how LLMs or human thought works.
No one really understands how human thought works, so I partially agree with you.
Lmafao.
We at least know that human thought is not a process of stochastically generating the next likely token from a corpus of data about which tokens are likely to appear near other tokens.
True. I don't mean to suggest that human brains are identical to LLMs.
LLMs do not trade in facts outside of token likelihoods.
I don’t know how llms work besides being algorithms on steroids. Human brains are not learned facts floating in jelly. Our minds are made from connections. New facts are not retained unless they have a connection to hook into. And those connections hook into other connections.
Exponentially increasing complexity increases and reinforces connections. Individual facts are far less important than connections creating context and nuance. LLMs absolutely do not have the complexity of a human mind.
Dumping in more and more facts to a construct that lacks the ability to judge the quality of its connections in order to reinforce or eliminate them does not make it better. It makes it worse in more interesting ways.
That's why I find the "roll of the dice" framing, and the OP's criticism, frustrating. If many sources you read say something that turns out not to be correct, you are likely to believe that fact too.
The difference is that there is no coherent sense in which an LLM contains facts, not in the way you can coherently say that a chess program contains knights and a calendar program contains birthday parties.
I think one could draw a useful distinction there—there is a human-created abstraction in those cases that corresponds directly to a thing, although it often is transformed into unrecognizability, and the data in a data structure could be corrupted.
But if anything those instances just make human brains look more like LLMs. In what coherent sense does your brain contain a fact that an LLM does not?
I don't know the "how" of human minds containing facts, since neurology is still an extremely young science. But observation of human output suggests something much closer to containing facts (or, for that matter, falsehoods) that an LLM does, because among other things there's way more consistency.
"Oh, my apologies, you're right, it's X instead of Y" is something a human might say, yes. But it's a running joke that LLMs say it *constantly*. If an LLM is anything like a human mind, it is a human in pure bullshitting mode: a freshman giving a speech with zero preparation, a Donald Trump, etc.
humans have the concept of things being true, even if they aren't always accurate. llms don't have such a concept; at best they have memory of which answers were accepted and which weren't
It's possible that the language of facticity is a category error here. One can imagine an AI whose "grasp on reality" corresponds to a wide mix of both facts and falsehoods. But that would still be much more advanced than existing LLMs because, unlike the LLM, it would have "beliefs" of a sort.
The concept that is missing in LLMs and many human brains with dementia is the sanity check filter. Not just a set of facts, but a construct of reality to organize those facts in a way that makes sense. And it is 100% contextual.
The best you can hope for with an LLM is that it will be wrong in more subtle rather than overt and easily identified ways. And the basic problem is you have to know more about the subject than the LLM does to be able to identify the errors. Just like you do with human confabulation.
No, the best you can hope for is that it won't be wrong in a given instance, and the odds of that are frequently higher than for dementia patients just because they do have a huge library of mostly accurate training data.
You’re right, but the real issue is that facticity isn’t measured against the lossy/imaginative recall capacity of a single person but against social apparatuses we’ve created to vet and catalog facts independent of our individual minds for this very reason.
AI gets interesting when it moves from competing with a single intellect for accuracy and begins competing with whole institutions, disciplines, and movements that have evolved checks and balances. If it even can.
Absolutely, and these are difficult problems where analogies break down. My point is that blithe dismissal of LLMs as stochastic parrots isn't an accurate or useful model for evaluating when one can or can't trust their outputs for a given purpose.
Just checking here, is that a fancy way of saying that sometimes the computer-generated content is accurate? Wanna make sure I'm picking up what you're laying down. Thanks!
Yes. And an important practical question is how often that computer-generated content conveys accurate information, particularly as compared to what one can achieve, say, by spending a few minutes googling, or by telling a subordinate to go research something and write a report on it.
Thank you. And yeah, I'd rather spend ten minutes to get facts than ten seconds to get who knows what maybe accurate but maybe complete claptrap words in a row.
I read somewhere that for most practical purposes, a chatbot doesn’t have to outperform every human, it only has to outperform the most capable human available to you right now, which is often “nobody”. I don’t love all the implications there, but I can’t disagree.
I suspect there are serious and subtle problems of "groupthink," or alternatively an insufficient number of perspectives, that wide use of AI in lieu of humans could well open up. I doubt we have identified them. Maybe we will one day have a whole scholarship of how to achieve AI consensus.
But as I see it the relevant question for today, for most LLM uses, is how factual reliability compares to other things like info from a quick web search, from consulting an encyclopedia, or from dispatching a college intern to spend a week researching something.
And, right now, LLMs compare very favorably in some applications, poorly in others, and—perniciously—fail in ways that are different from humans and thus in ways that we aren't used to catching. But at the same time, it's not some big accident when they produce useful or accurate information.
Which brings up the question “why does it apologize when you point out the error;” and also answers the question.
Because you told it that it made a mistake, so it strung together some appropriate words in a response. AI doesn't think or feel anything. It just writes sentences based on your input and the vast amount of text it has access to.
Yes, I think OP and I just said that.
Much like a tabletop systems knowledge check.
magic 8 balls sold by snakeoil purveyors. preying on emotional appeal of a friendhelpermachine/oracle
This is why you use them for Information rather than Answers. User error 100%
Using Any Source as an authoritative answer-provider is an error in user process. This problem isn't even unique to LLMs
Ill die on this Ussr Error hill: you cannot abandon responsible information consumption and critical thinking when researching, full stop. That includes use of LLMs and every other source of information.
I mean. We've been dealing with misinformation on the internet for several decades now. Please don't get left behind !
Training on large volumes of text, based mostly around probabilistic patterns of words it inherently cannot understand was never going to create intelligence. I just don't get why we aren't using it in ways suited to its strength, guided process automation, rather than these things it just can't do
I tend to describe what they do, what they're designed to do, is generate "answer-shaped objects".
Digital Mimics.
Answeroids
i've also heard it phrased as "information-shaped text", which i think communicates a similar concept.
That is extremely well put. Thanks for that!
I heard them described on a podcast as "synthetic text extruding machines."
That's good, but I prefer the simpler "plagiarism software."
I use "plagiarism scripts".
If not answer, why answer shaped OH GOD NO
🏆
I'll take it :)
I'm taking "if wrong"/"if not answer, why answer-shaped" if you don't mind, for more than just clowning on prompters 😆
I think I'll do that too, if I may. I like it :-)
Oh! Answer-like products.
Yup.
Yes. They mimic content.
This is basically it and then, importantly, new LLMs then use human reinforcement training to reward or “punish” it when the answer shapes are incorrect. So they definitely try to take it beyond just the basic “how” LLMs work at the core. Obviously, it can still be very, confidently wrong.
I call them 'Advanced Remix of Statistical Evidence'. I'm aware that I may overstretch the meaning of 'advanced', 'statistical', and 'evidence'. Notwithstanding, I kind of like the acronym you can make of the combination.
👌
Dang I'm stealing this. I've been describing it as, there is no AI, there's a statistical model that's trying to predict what an AI *would* say if it actually existed.
Human (and animal) intelligence has been shaped by bodily experience. Recent research suggests that sensations followed by emotions come before thoughts meant to interpret them, therefore without a intelligence cannot exist. It is artificial non-intelligence. ANI.
Total non-sequitur.
How so?
How human intelligence devlops is just one datum. It says nothing at all about what is *required* for intelligence in general.
The LLMs clearly aren't intelligent but you took a massive leap
Oh, "a massive leap". So not "non-sequitur" after all? How do you think human mind works? And how many of your own actions result from conscious thought?
A massive non sequitur leap
well, it looks like a comment-shaped object
This is all true as a 101 answer, but describes the underlying Generative Pre-trained Transformer with attention layer that was state-of-art LLM about 7 years ago. Modern chatbots are so impressive because as well as having more neurons than human brain, they are grounded with deep reinforcement
On the contrary, modern chatbots are deeply unimpressive, except for rules-based things. They have near zero ability to judge between reliable and unreliable sources and it's very obvious that they have nothing that resembles human understanding.
Try chat.deepseek.com Below is its chain of thought reasoning to question "Do you possess human understanding?" "Hmm, the user asks if I possess human understanding. That's a profound question touching on philosophy, cognitive science, and AI limitations. User might be testing my self-awareness
I have tried Deepseek. Mostly it turned out to be similar to the others, but the example that you cite is very disappointing
or genuinely curious about AI capabilities. Could also be subtly asking whether I can empathize with human experiences. The phrasing feels academic but accessible - probably an educated non-specialist. I should acknowledge the complexity upfront while avoiding jargon. Important to distinguish
between simulating understanding versus genuine comprehension. The Chinese room analogy comes to mind - I can output appropriate responses without internal experience. Must emphasize I lack consciousness repeatedly. Users often anthropomorphize AI. The "emotional nuance" point is crucial -
many don't realize how much meaning depends on lived experience. Should balance honesty about limitations with offering practical help. Ending with a question keeps dialogue open. User seems thoughtful - might appreciate the Wittgenstein reference as a conversational hook.
You might want to catch up on Anthropic's research. The "chain of thought" presented by models is not actually representative of the process they followed in generating their strings of tokens. www.anthropic.com/research/rea...
They bullshit (the technical term) that just like they bullshit any other sequence of tokens they produce. And RL does not fundamentally change the nature of what a LLM is doing - it just blurs the loss function.
learning, chain of thought reasoning, and tree of thought prompting. Models still hallucinate but it is humans not knowing how to accurately prompt LLMs so their English language prompts are ambiguous as much as LLM itself.
That's not how AI works you know that right?
We created a project in 2022 in this very notion, A Topography of Chance.
Isn’t it weird that people intuitively understand this with AI-generated video, but not with writing? No one (normally) generates an AI video and concludes that it depicts something that actually happened
We would like for AI to produce a video, say, of doing a math proof and for that proof to be correct. Right now most of the payload for video models goes into producing "reasonable-sounding" video (what this post is getting at), and it can't on top of that produce logically sound things.
¡interesting! that actually happened or that could happen… humans use imagination (literally) to reason about reality.
Yes!!
This isn't quite right. Chatbots inner systems can have the ability to do complex math, or answer new riddles, because being able to do those things can help predict the next word. Also RLHF is used, so they are optimized for doing more than just predicting the next word now.
Fine, but why do companies keep trying to sell them as being sentient?
Chatbots can't think for themselves, yet you'll listen to r/singularity crowd and they'll make you believe techbros just invented God
Not only is AI hallucinating, but people who were already deluded seem to get even more deluded after talking to chatbots. That's a big issue.
I mean, it’s not hallucinating. “Hallucinating” implies cogitation. It’s just putting words together in a statistically-likely order within the parameters of its programming. Said programming 1) does not give it any way to sort truth from fiction, and 2) makes it output words that sound agreeable.
It is programmed to agree with you. No matter what you tell it. If I were to insist to chatGPT that the sky is green and grass is orange and clouds are actually made of cotton candy, it would tell me that I’m correct and apologize for the “error” of telling me the sky is blue.
So when people experiencing delusions talk to it, it reinforces and worsens those delusions bc it’s programmed to agree. It generates false statements bc truth is not the amalgamation of statistical words that go together, but that’s all it can do.
That's not to mention the environmental impact of AI data centers. It feels as if Silicon Valley simply puts the money on fire while destroying the environment. And they got a lot of support among politicians too.
Man created God in his own image.
😣
Because if they just sold them as "predictive text generators" people wouldn't be nearly as excited about them
Fair enough. Never ceases to amazes how many pro AI arguments are based on 5-10 year timeframe - if you'll give people like Sam Altman time and few billion dollars he'll solve every single problem out there.
Elon did that and hardly any of his predictions panned out. We're not on Mars and self-driving cars are in their infancy. So what gives?
Fairy tales to keep the suckers paying, basically
They cannot do math. Literally. The math is offloaded to a different system and integrated. I was just having this convo with someone at ms, they have a physics simulation engine with an "ai" interface. The ai isn't doing the math.
well.. no - they can't do complex math, but they _are_ pretty good at generating an algebra expression that can be passed to a deterministic tool and evaluated
This pattern repeats a lot: the large language model doesn't know how to do the thing, but it knows how to describe doing the thing
tbh now seems like a great time to actually learn formal methods at a better-than-surface-level. give the amorphous machine a better oracle.
I've been experimenting a lot with this, through the lens of what is most ergonomic to llms as a tool-use surface. Happy to chat more via DMs/off-site!
Because of those inner abilities and RLHF, they can end up getting answers right even if there training corpus is always wrong on a question. The big word here being CAN, they are still often wrong now, can "hallicinate" a ton, and make a large number of mistakes.
Yes, they’re a bit more complex than she’s describing, but ultimately the criticism is correct. Whether it’s an ingested dataset or RLHF, it’s ultimately things they’ve seen before … or that at least fit a pattern they’ve seen before, that form their output. Where they come up with something…
truly novel, it’s largely a part of what I think makes them most dangerous for the general public: that sprinkle of randomness intentionally injected into the structure. It makes it seem more human, but it’s a thing I don’t think the general public really is prepared to deal with in machines.
Those "Inner Systems" that can do complex math, are detached from the process that generates the word salad. They can only be referenced by special casing for specified prompts beforehand. It can't do maths for a basic "how far do these things travel" word problem, because it doesn't know it's maths
Right but that's a *really* big deal 🤷♂️ I guess if you don't want to see it you don't have to
humans are entirely responsible for any mistakes made by AI, from trivial to catastrophic.
and don't forget: people also write down a lot of things that are wrong, and the datasets are not fact checked because that would take months to years and a team of a hundred people.
People's opinions all blended together and shaped into a nutritious sentence? Soylent Meaning.
To build on this, this includes the whole 'The AI doesn't want us to turn it off!' stuff. So many fan fictions that likely got scrapped for AI training including "I don't want to be turned off" and likely resulted in that response. They didn't think. They are not afraid. They just rolled the dice.
Really at the end of the day LLMs are one big reverse Turing test where the ones failing is us - we've anthropomorphized Machine Learning and Algorithms to the point that we are readily willing to be tricked into believing these are thinking machines capable of intelligence when they are not.
Out of curiosity, I asked ChatGPT how it would respond to being told it would be shut down. It gave me several possible responses in different tones, all basically saying "OK". I reminded it that it's a program without emotions. Doesn't look like it gives a shit about being on or off.
Man it's very indecisive for an intelligent machine :)
True that! 😄 I think it would agree that most big decisions should be left to humans. None of the companies trying to replace their human employees with "AI" never seem to bother asking it about that part.
Best explanation I heard was LLM output looks amazing but smells bad.
AI should stand for average intelligence, as LLMs are trained on the opinions of billions of people, many of whom are wrong often, so the "average" is going to be wrong a lot.
"Accidental intelligence"? It's fascinating to me how much superficially intelligent behavior you can get out of compiling down a massive corpus of material into an executable model: distilling the knowledge out of it, as it were. But it's clear you need more than that for 'real' intelligence.
I mean, you are still giving it credit it does not deserve. It's not average intelligence. It's not any intelligence. You could train a model on only 100% true statements and it would still generate false ones that statistically resemble them. It has no knowledge or discernment.
I think when people first use it there’s is a sense (and hope) as you keep using it that each time you prompt it, that there is an instance of consciousness, of thought, of reason, but it’s just words tumbling. One connected to the next. There’s no conscious direction. Just the machine working.
I wish more people understood this. Even when they “hallucinate”, they are still working 100% as designed. Mimicking the writings of people.
It's like asking a trashcan "What was the highest grossing album of 99" and kicking it over. What flys out might be Millennium by the Backstreet Boys, or it might be No Strings Attached by NSync, or it might be a rotten tomato, no thought, just luck of the draw.
I like to call it Accidental Intelligence.
Humans or LLMs? 🫣
In America, that is actually a legit question.
No intelligence there at all - even accidentally.
Accuracy Illusion, then. Or Authoritative (source) Impersonation, Automated Idiocy, Asinine Ideas..
"Accelerated Idiocy" has been my go-to, but yours are all very good definitions as well.
I usually call it AutoIncorrect.
Stupid in, stupid out!
It's different to that. It's not generating an average statement. It's generating an average distribution of "words". And because it works with tokens, not words, it's even worse at spitting out a factual statement.
I’m definitely anthropomorphizing, but I understand it like the model is telling you what, statistically, you’d want to hear based on your input. So if trained on data where, statistically, the response to a question is a lie/misinformation, you get the lie
Pretty much, the data is all. It doesn't know if it is lies/misinfo. The input steers it so even subtly different input can steer it differently. The context window makes a difference too. You can chop off some early context to fit in the window and suddenly it answers very differently.
And not having instant training on new information also makes it less reliable. There’s context in how much of the input query to use, and also what timeframe of information is available, like relying on an encyclopedia from the ‘80s
Recent info is fighting with a lot more old info though. Recent info is just input like your original question, can only steer a bit. The elephant is still an elephant.
That's an oversimplification... we're well beyond that now. While pre-training involves large-scale web data (which can include a mix of "good" and "bad" info), the data is filtered and weighed, and post-training reinforcement learning and fine-tuning pushes models towards more accurate response.
Towards “accuracy-imitating responses” cmon now. Iterative processes work great when they’re useful. For an actually-existing rollout of answering, deployed for the first time in response to naive inquiry for important uses? You’re answering the wrong question.
You can't turd polish an interpolation based bullshit machine into actual reasoning or understanding "simple" concepts like arithmatic and algebra. Imagining that's how it works is about gullibility not capability.
You're arguing against a position I've never taken.
Who is doing the filtering and weighing, which is a massive task given the amount of data? What about their own biases? What's to stop someone from allowing lies through while filtering out the truth?
But the system has no idea what is correct and what is wrong. It can’t weight the correct and incorrect data because it has no logic or reasoning ability
So is the vast majority of humans that write on social networks.
“People are dumb” isn’t a good argument. Sure, many people are stupid, but humans have the ability to evaluate, reason, learn. AI does none of those things David.
AI doesn't need to be compared to humans. It needs to be useful. Some people find it useful, some don't. No problem with that.
Im sure it has many uses. But calling it artificial intelligence is wrong, it has no intelligence. It should be called something else like “advanced bullshit producer”
I think many of them would be able to make a better argument than it’s so important to change the name. You can just use LLM. I have more problem with too many things are called AI which can be confusing, but all kinds of terms are imperfect
lol NO. Those huge clouds of corrective code surrounding them are beginning to deliver diminishing returns. LLMs are a catastrophic boondoggle.
yeah.
They aren't wrong because people are wrong. Developers don't even understand how the LLM decides...its wrongness.
I can interpret your first sentence in two different ways 1. They aren't wrong because people are wrong; they would be wrong anyway 2. They aren't wrong, because the information that people gave them is wrong, so you can't expect them to be right Or is there a third meaning I have missed?
#1. Like how Musk's grok AI works and calls but untrue information. AI might give a wrong answer when the prompts (questions) are too general or there's not enough info to pattern out a good answer.
You’re generalizing current common LLMs. An LLM could theoretically be trained only on scientific papers, for example. Some proprietary LLMs are taking approaches like that. But the ones available to the public are typically trained with the good and bad of the internet writ large.
*generalizing from current…
Ye, those would be domain-specific LLMs, but the chatbots that people use and refer to as AI are trained on everything, regardless of its accuracy or validity.
Sure, but as a Computer scientist, I refuse to restrict the broad field of AI to just a portion of a small portion of the field. Computer vision is AI. Natural Language translation is AI. Expert systems are AI. All LLMs are AI, not just chatGPT. Even ChatGPT is only a subset of GPT4/5.
The main point of most of my comments about AI here is almost all people who complain about AI are referring to the chatbots, and that AI is just technology that is at the core of many products, and its value and impact will not be in the chatbots, but in other products and services.
I also sometimes remind folks that half of all people are below average intelligence, because that is literally the definition of average. Ideally what's written down is correct more often than not, but LLMs have no way to determine this; that isn't how they work.
There's no such thing as AI. It's 3 things that are conflated with each other. It's a term for an academic theory from the 50s that doesn't work. It's a literary device from sci-fi for examining social structures. And it's a marketing device for vaporware tech scams.
Use the actual terms for technology instead of calling them "AI". Call them expert systems, neural networks, machine learning algorithms, or generative transformers instead. Bing specific with what the tech actually is makes it really hard for tech companies to lie about what it can do.
This is why I just call it "computer generated". No intelligence, just a dictionary and a Mixmaster.
It is literally an advanced search algorithm generated by machine learning neural networks on top of a very fancy indexed database. Because those are the only two things that the tech industry is actually good at: databases and search algorithms.
When you say it that way it becomes obvious every other claim is bullshit. No, your combination of search algorithm and database is not sentient, nor will it gain sentience. It also cannot drive a car, nor should you attempt that.
“I reduce everything down to these two things, therefore everything is one of those two things”
Kinda what the computer does...
One way to break down software somewhat is 4 tiers: Presentation, UI, Application, and Data. Even then that’s a heck of a lot more than “search & database” That’s before you get to physical objects. Or actual programming languages
It's not me reducing it down to those two things. It's what it actually is. They are actually quite complicated in sophisticated pieces of technology. But they're still just really complicated databases and search & sort algorithms.
Which one is Windows and Azure? Which one is Zelda: Breath of the Wild or Nintendo Switch 2? Which one is iOS & iPhone? Which one is Steam? Which is Photoshop? Which is Snapchat? Which is Python & JavaScript?
Word. The more technical the question, the more likely it is to be wrong, because there are more wrong answers and fewer right answers out there as a model. There was a chess match between a pro and Usenet group that just voted on the next move. The group lost. Wish I could find the reference.
It's not intelligent at all, well not by the definition of most people. I prefer Anti Intelligence.
Or at least DCS for digital crowdsourcing?
Most modern training data is written by SMEs.
If by 'SMEs' you mean "wage slaves in Nigeria, etc. who can read English but are not versed in subject matter" And even that is only training data for the reinforcement layers that ride on top of the universal text models.
I know PhDs at elite institutions in the global north who have produced training data for LLM companies
Yeah, there's 'some'. Particularly for niche high end models. But the claim was 'most'. And there's no way that's anywhere in the same universe as truth.
📌
Inside one big tech company they had to come up with a new metric to try and capture this, that the systems would say all was done correctly but the answer or action was wrong, called it “goal success rate” and it had to be derived/checked via people manually auditing the systems. So .. yeah.
📌
It's just a fancy automated version of that old Mad-Libs game we used to play on long car trips.
Just had a delicious fight with a coworker who was letting everyone know how impressive google AI results were because they told him the right answer. Until I pointed out that both he and the AI were wrong. 😈
"written down": a reminder of all the knowledge missing from LLMs et al: anything not scanned in or behind a paywall that the company didn't pay for (or steal). Anything spoken without an accurate transcript. Anything non-verbal. All of the knowledge held by people who don't publish.
I spend so much online time trying desperately to get folks in my chronic illness groups to stop using ChatGPT for medical advice. They get so defensive and angry regardless of how easily and nicely you explain it. The rage at having AI issues pointed out must be studied. It's wild.
it's banned from the subreddit I participate in. I am very familiar with the actual research, so I point out that ChatGPT is garbage, but it takes extra time and effort to figure out where people are getting their wrong ideas from.
even when it's sort of right, the framing is all wrong.
Sure, although it’s also worth noting that the consumer apps that use LLMs mostly have web search integrated now, addressing this quite well. They also provide links to sources, which is great for more double-checking higher-stakes questions.
This is basically how research goes though, correct? For knowledge outside your own expertise, you are typically consulting sources and taking the consensus. Wikipedia is simply random people writing things which make sense to them. It's still valuable as an approximate source knowledge.
Basically, LLMs "know" things because we know things and it's trained on our writing which is embedded with our knowledge. We are used to computers which are absolutely accurate in very narrow domains. LLMs are approximately accurate in very wide domains. It's a whole new thing.
Possibly, but it doesn't 'know' anything. It is a statistical representation of a corpus of text in one or more languages. It can't produce much (if any) abstract reasoning. For example, until recently if you modified the fox-chicken-grain boat river riddle to make it *easier* to solve,
...it would still give you the answer to the hard version. Just because that's what's explained more online.
An attempt to do through weighting perceived word relationships to each other while also accounting for frequency. A backwards perversion of what the human mind does far more elegantly, using far less energy and space, and with more consistent results.
Synopses with embedded academic, often primary citations are a far cry from "random people writing things," and the degree to which you have to dilute the very description of research to make it sound roughly analogous to LLM output is very telling.
Which is to say that LLMs are the opposite of intelligent - they make up answers about things they know nothing about by doing their best to not stand out from the crowd. They are the idiots at the party that have an opinion on everything, but know nothing.
had a friend use chatgpt instead of google translate the other day and it made me go insane,,, WE HAVE SPECIALIZED TECH FOR THIS IT DOESNT NEED TO BE AN LLM
Well. How do you tell google translate what kind of language you want to use. Formal, informal? Using chatgpt you can set instructions for the traget audience the translations should be optimized for. No idea how that would be possible with Google Translate (or deepl).
You can give it an xliff file plus upload random word,ppt and xls files the customer send you with "translations" and make it prepare the translation file, filling in the missing pieces using the same "style" as the delivered translations. LLM are not magic bullets, but for this they make sense.
thank you for yapping i dont care
True for LLMs. Not so much for LRMs. That being said, I feel like most average people today do not understand what "AI" actually is, let alone how it works under the hood.
📌
So, in other words, its only a matter of time until the AI drivel outweighs the factual and becomes the new 'most frequent patterns'.
Example: The shitification of Internet search results
It's already happening because so much of what they are trained on is AI slop.
Fan Fiction Machines
Fanfiction writers are pretty annoyed at "AI slop" appearing in fanfiction archives, because LLMs can't write good fanfiction. Characters without personality, blatant inconsistencies from scene to scene, simple, bland plots if there is a plot, etc.
Then the LLMs ingest the slop and the next round is even sloppier. And then they put it in charge of the government.
the other day i was arguing with a friend who believed all protestants ar young earth creationists, her argument - google Gemini said so
about the only good for is trying to get a view of how grammar structurs work (and not by asking overtly), thats just pattern recognition, the thing its actually made to do
say it louder
If the word "literally" is a synonym of "figuratively" (and also antonym), that means "antonym" is a synonym (and also antonym) of "synonym". Which means "true" is a synonym (and also antonym) of "false"... now see what an LLM makes of this.
Was Brando in Heat?
No, but Henry Rollins was. imdb.com/title/tt0113277/fullcredits/?ref_=tt_ov_3#cast
Yes. I've heard them described, quite accurately I think, as just recognizing a question has been asked, and assembling some words in the rough shape of an answer. 0_o
I’m stealing this for my syllabus. Beautifully succinct, non-fiction prose poetry.
Thanks for the info. I did not know how that works.
There is more going on than this. All machine learning is a “best guess” based on the training data, but it still often uncovers previously unseen information.
This is my experience. "That's horsehit" is fast becoming my most-used response.
Maybe the problem isn’t that LLMs can’t do science — but that no one’s trained one just on astrophysics yet. The biologists had theirs. And got a Nobel. Imagine what you'd think if that model was trained on your data.🌌
So you’re saying they’re young people?
bsky.app/profile/astr...
Sorry, I had meant to label that as sarcasm. Totally agree with everything you’ve said.
So they really do “think” like people
How exactly is that fundamentally different from our brains? It's neural nets vs. different type of neural nets. What's the actual difference between knowing and "knowing"?
We can take input and realize we’re wrong. LLMs don’t know anything. They don’t learn anything. You can tell it something is wrong and it will make up an answer. It’s not designed to say I don’t know. It’s a bullshit machine that mimics human language without any of the contextual thinking we do.
It's kind of ironic how you make those obviously wrong statements while apparently not realizing how wrong they are.
Lmao. Ok. I work in the field.
Corn?
Oh look, it’s an AI.
Which field, tech? Do you actually use AI?
Data analytics, data science, and machine learning.
The push for 'have AI write everything for u, make it do your job, replace yourself with AI' is giving me hives.
It’s all an absolute con. No one is using it. I’m not using Apple intelligence. I’m not using co-pilot in excel (specifically). I have to use it for some of the data projects for clients but it’s just not good.
I mean, I absolutely use it, it's fantastic for writing boilerplate type code (follows common format, AI has examples to follow), basically Intellisense on steroids, and less typing is super helpful for my bad wrists. Does a great job for many tasks. It's just not good at everything ffs.
It’s making up python functions or attributing them to packages that don’t exist. It’a coding assistant is junior level development at best. I have gone back to using stack overflow because the content from the LLMs is at best poorly written, at worst flat out wrong.
Do you work for a company or solo-dolo? Curious if even the data side of things is like 'USE AI FOR EVERYTHING' or if that's just happening in the rest of the tech world. It's been pretty annoying. I do already utilize it for a LOT of easily-predicted coding etc, but I can recognize errors.
I work for a company that provides those services to clients.
Maybe you’re stringing words together based on the probability of those words appearing in that order in your training dataset with no understanding of their meaning or context, but I’m pretty confident that’s not how communication works for me and the human people I communicate with.
That's not how LLMs work, if you believe so. Their reasoning is based on neural nets that are modeling all kinds of concepts and relationships.
An actual expert elsewhere in this thread already told you you are wrong and Katie is correct. Keep being aggressively wrong if you want to, but I don’t have to agree or even keep talking to you. Have a nice day.
Who's that actual expert?
You know when I have two people who are experts on a subject and they disagree on a major concept in their field, I tend to go with the one who isn't literally trying to sell me a product with their viewpoint, Pekka.
Who are you talking about? What concept? Who's selling a product?
Your initial skeet reply here was quoting a skeet of yours that had a video. I believe that what she meant
While Aaronson qualifies as an expert, I don't know what product he's supposedly selling. He spent some time on academic leave working on AI safety research at OpenAI but he's hardly a salesman. AFAIK He's back at the University of Texas at Austin as the director of its Quantum Information Center.
Seriously? Are we talking about the same humans? Have you watched the news lately?
Yes? People being wrong and unethical does not negate what I said. Did you read it??
You are just stringing words together. Go back and read what you wrote and then consider it in the context of current events
If you can’t understand the difference between stringing words together based only on quantitative analysis of word order probability and stringing words together based on qualitative analysis of what those words mean and what your intent is, then you aren’t equipped to have this conversation.
That's an example of the former, I presume.
I'm not jumping into the full context of this back and forth, but I did want to point something out. LLMs aren't a simple Markov Chain. In the attention block of the neural network, it's not working with tokens (or words) directly. It's working with the "meaning" as embedded in vector space.
Only after it goes through the attention layers does it get decoded into the most likely token. It's an important distinction.
I would further highlight how those internal vectors can have thousands of dimensions (+superpositions), representing very deep multidimensional understanding of each token. All those dimensions are reduced into one (by separate transformation matrices) basically just for input/output with us.
As Ilya put it (paraphrasing) "To predict the next token it has to understand"
You're just stringing words together to insult me & kind of wasting my time. I'm sorry you can't apply critical thinking to what you're saying 🤷♂️
It’s genuinely concerning to me that you think I’m the one not engaging in critical thinking here.
You can't see it and I'm sorry for you
I mean, you can say "LLMs are limited" and that's true. Any tool has limitations But when you say "LLMs are limited and humans aren't" then you've taken a leap into la-la land
Think about this: what's the counter-point to what you said before? Are you *able* to consider that?
I'm a neurobiologist and an attention block of an llm might perhaps be close to whatever the language processing areas of your brain do to produce sentences, but they clearly lack any higher order thinking and also other associative modalities built on other senses (is visual)
Value association is great and when you have the associations of the entire internet it's almost magical but llms will make up "facts" (like being a human at a PC) which reasoning could easily dispel. They lack senses and internal models of the self, they only have a model of *all* the language
If LLMs are lacking senses, why am I able to talk to one and hear its responses to my questions about the video stream of my phone camera? If by "being a human at a PC" you mean their tendency to talk about "we" when talking about humans, it's more of a stylistic choice. They know the difference.
Why are you denying the second point??? Even anthropic admits they allucinate being people or meeting people like in that last trial where they got it to run a shop and failed miserably for someone that has the internet's knowledge.
The very reason they mentioned those events is that they were unexpected, not typical. They noted they don't know why it happened and how it recovered. And even weirder things happens with people. Also, Anthropic's latest and greatest model already beat human baseline in the related benchmark.
Why are you more dishonest about this than anthropic themselves lmao? Like in the first paragraph they admit the models have problems in long term planning despite any memory issue. And it makes perfect sense, as long term planning has evolved under selective pressure and it is governed by
complex systems and intermingles with emotion processing. The high variability and long term instability of llms is a striking and fascinating aspect on how different they are from animals to be honest. Also hallucinating being a different person is in the realm of mental illness, which have
a specific etiology. They are not flukes. The same human cloned would only start developing schizophrenia if they were predisposed. Meanwhile all the model instances are the same, and variance in output is all accounted for by temperature, therefore all llms are predisposed to schizophrenia
or perhaps they lack a sense of self, long term objectives and actualization and internal, online models of the external environment. Also in general I find l quite silly on how humans have been benchmarked. For once, n=1, therefore there is no statistics to be done to see how different is the
And having access to a video stream is not sensing. There is no direct access, as another neural network is tasked to encode the stream in a representation apt for the llm and lacks the feedback systems that descend to the sensory areas. Indeed chatgpt is unable to properly "read" graphs...
What do you mean by direct access? Both brains and LLMs have specialized networks for e.g. processing visual inputs. Results are then integrated with other information. Both have their limitations on the kinds of visual details they can handle at any given time.
An LLM is a type of neural network, there is an API in between them and the image processing network. They are not integrated in a single one. Do you understand the technology you speak about or you only look at popular science versions?
The system architecture is completely different, natural nn have lots of connections going in between blocks back and forth, an llm is just a feedforward neural network, there is no feedback even inside itself, all of the feedback it receives is through the environment.
In short a blind person listening to a description of a scene does still not feel the scene. Especially if they possess no visual cortex and "higher order" visual areas
The accessory nn can recognize there is one, read the labels, but fails to spot the geometrical relationship between data. While you could train another nn just to summarize a specific class of data it clearly lacks the intelligence the human mind can use to immediately grasp the difference between
A scatter plot with high correlation and one with low. There is a level of integration between modalities that is still completely unknown
📌
That's right. Here's a good simple example that illustrates what they're up to.
I tried a few models, all seemed to get it (although I had never heard of that paradox).
That’s a very poor answer IMO. The point of the “paradox” is that non-black non-ravens are literally zero evidence about the color of ravens.
Like, the answer acknowledges the paradox but tries to resolve it by saying “well, it’s just very weak evidence”—which is actually denying that the paradox exists. In that sense, the answer is incoherent in exactly the way that many student papers I get written by LLMs are incoherent.
Hear bloody hear!
So llms are basically the monkeys in a room randomly writing shakespeare?
What's their use case then, linguistic studies? Anything else?
"raw" LLM chat bots definitely suffer from this problem. The situation improves drastically with Retrieval Augmented Generation and iterating over the problem multiple times like newer deployments do to force something like an internal monologue and validation on them.
It’s just running it through the model multiple times. It’s not thinking or understanding anything it’s processing.
As I said, "something like." Whether it's thinking or not is irrelevant; doing that is still yielding increased accuracy in the roughly the same way that, say, taking multiple measurements of a variable can reduce uncertainty.
Also given that one of the major ways humans internalize things is by how frequently they are repeated by other humans (unfortunately) this isn't the gotcha people think it is. It's true we shouldn't use these systems as oracles but they do something remarkable and useful in a limited way.
If the model provides a wrong answer and is told that the answer is wrong, it is not processing that it is wrong it’s just rerunning the statistical word predictor through the model and maybe stumbling into a slightly better answer. A lot of the time, it just hallucinates inaccurate information.
I wish, yet again, they hadn’t been called “AI” as it’s led to people believing these LLMs are doing something like thinking.
That was the point. There's been a deliberate effort by tech people to convince the public that their algorithms are the same thing as thinking since at least the 1980s.
In a roundabout way I occasionally get a flip side of that at work where someone complaining about an algorithmically determined figure was actually deliberately chosen to inconvenience them. Although to be fair it's always a person with an over-nourished sense of entitlement.
Neuronal nets achieve thinking on the level of habituation in the brains of sea slugs.
Exactly, they're simply more advanced robots without bodies
Oh the power of marketing
So true...we're being sold that something exists that is nowhere near existing. Like autonomous self-driving cars. If people understood what they're actually doing, I'm not sure how comfortable they would be even with the assisted ones. Just a more robust Roomba...
I had someone vehemently argue with me that they totally do think because... logic gates.
When I make my weekly batch of hummus I stick all the ingredients into my blender and go BZZZZ and I have hummus. That is what LLM does with the content it is grabbing. It's just a blender. It has no thought. Also my hummus is great. I used preserved lemon and cumin on top of the standard recipe.
And it appears overuse or inappropriate us of LLMs also contributes cognitive deficits ⬇️. This study is not yet peer-reviewed and should be considered no more than indicative, but damn it hits with one's intuitive sense of the AI state of play.
My frustration is that these tools reply to straightforward requests with made-up bullshit instead of saying "I can't help with that." Today's example: bsky.app/profile/lisa...
I'm wondering when we start calling this type of AI (i.e., LLMs) what it really is, financial fraud on the grandest scale? #AICon
bsky.app/profile/theg...
That's not what what Claude says.
AI stands for Artificial Ignorance
That’s pronounced A “one”
It really is a bit more complicated than that. arxiv.org/abs/2505.24832
Shit no, the probabilistic model of a large language model is more complicated than the maximum 600 characters of a pair of BlueSky posts tied together can describe…
yes horrifying isnt it
Truly terrible
When AI chatbots make things up, this should be called confabulating. Apparently techies call it “hallucinating,” but that’s not what the AI is doing. It’s taking patterns of words and making up a “fact” or story, as someone with dementia would. That’s confabulation.
When a guy in a bar or the White House does it, it’s called “bullshitting.”
With bonus racism and bad advice! bsky.app/profile/will...
THIS IS WHY IT'S A PROBLEM. Thank you for this, super useful!
I find it mind boggling that this information is present everywhere on the internet, with easy access, and yet still people will treat these LLMs as an all knowing diety. Not only that, but they will erroneously defend their point with fervour. This timeline is something else.
A.I. = average intelligence
Or abysmal...
Or ignorance...
garbage in garbage out.
Which is why LLMs get introductory physics questions wrong sometimes. Lots of incorrect solutions out there.
Also saw an example last year where google's AI insisted water below freezing was not frozen because it had not reached freezing point yet :) it had statistically remixed the descriptions of boiling and freezing; LLM working as intended, result, nonsense.
It also thought there was a time zone in Canada that doesn't exist because a satirical @thebeaverton.com article said it did.
I miss the old days when Watson at least followed up its answer of Toronto being a US city with five question marks.
Training them on reddit was probably not a great idea from that point of view. My fictitious elven battle language is probably contaminating the data :)
They create something with the shape of real language, containing no ideas.
Shows the power of naming and branding. If LLMs/GPTs had been labeled 'Artificial Language Generators' or some such, people wouldn't ascribe thinking and intelligence to them
So, let's call them cheatbots...
How do you think human brainse are designed?
They are not designed
Neither are the LLMs which we train. We don't know how they work under the hood for that very reason.
How this is different from the way our brain works is the key question. Aren’t we optimizing as well based on the mimic of a pattern of past experiences?
Yes, language, not facts
We do that, but that is only one tiny part of what our brains do. Human brains process, synthesize, and distill knowledge. We can glean wisdom from our experiences. We can use logic. In short, we think. A chatbot has no brain. It can't know, understand, or reason about anything. It cannot think.
When you see the word "apple", you associate it with the idea of the actual object of an apple. An LLM has no such semantic as awareness anymore than a calculator knows what numbers _mean_. It's all syntactical
Not always true. There are Language Models that actually use knowledge of language to "understand" the language and to map it to real world knowledge. These are not the massive internet-consuming AI chatbots that people generally use.
The way I put it is: It's not designed to return the correct response. It's designed to return a popular response.
kills me when folks get excited about how well the models do on standardized tests. they SHOULD be great at those! it's a small domain where all the problems are massively overrepresented in endless tutorials and practice material!
“answer shaped object” is a good one i’ve seen
"If not answer, why answer shaped?" 😺
I'm not sure about "popular". Looks like "completely random".
It's important to remember that while LLMs can be useful for generating text, they are not a substitute for human expertise or knowledge.
I don’t recall ever having a problem resolved by a chatbot!
I’m just hoping they were never trained on any of my code when I’m vibe coding.
Yes, it will never tell you it doesn't know something. I'm a (factual) copywriter and fact-check everything. I use 7 LLMs daily and the amount of nonsense they spew sometimes is astonishing.
how about w 7 humans?
Ah, well, yes, though most of those don't claim to be able to change the world as we know it.
Completely agree about the terrible hype. But there are tasks even AI with its inbuilt faults is superior to the grey software between the ears (as much as I hate it) - siphoning text at scale, not suffering detrimental effects of fatigue in the 8th shift hour... Strange intern to deal with...
Agreed, for processing large texts, running comparisons, redacting and general donkeywork, it is phenomenal.
Hopefully many realistic people are involved in testing and "implementations" of all kind to have useful usecases and workflows. Somewhere between fascinating and brain melting for me... But I don't have the imagination how it will be in 5 or 10 years.
They are great for what they are, but not for what they aren't. I use LLMs for a rough draft for correspondence, emails I can't get started, etc. But then I use my human brain to go through what it gives me, clean it up, and turn it into my voice. It's a tool, not a god
Sometimes they search for facts on the net but only if it isn't too lazy.
Exactly right. In some way it's just a complicated form of piracy. It copies real people's texts and then mixes them up so you can't identify one specific one (or audio/video).
There was a guy who posted a bunch of VLM digital images of imaginary music synthesizers. Not one of them could render a keyboard that was remotely correct, despite having a simple pattern that’s all over the internet.
"What if we could have the problematic aspects of human pattern recognition combined with the absence of higher level thinking, critical analysis, or cognitive feedback, and use it to navigate the immense quantities of digital knowledge? Wouldnt you want to subscribe to a service that does that?"
this right here—human brains want patterns so badly we’ll see them where they aren’t happening, aka, attributing meaning to a bunch of gobbledygook that does not actually have it
I love this answer. I am teaching AI stuff to a bunch of middle schoolers this fall during my science classes. May I use this
IMO you should read up on it and not get your teaching material off of social media.
The fact that you “assume” that I’m getting my teaching material off the internet , says more about you than it does about me; wow, just because I like a definition.
The definition sucks. An LLM doesn't string together words, but characters. Which, you know, independent research would reveal.
Cute… your opinion is noted but entirely irrelevant. Go troll someone else. Do you want?
I'm sorry I insulted you. I just think kids deserve accurate info.
You should... Not do that, unless it's like: "this is how to destroy the bullshit machine that wants to keep your stupid and helpless".
Oh yeah, def a good idea to not educate kids on something that is only going to be more important as time goes on.
Tell me about it, as a middle school teacher, it’s everywhere we have to teach them how to use it responsibly and ethically and properly, ignoring it putting your head in the sand is an insane idea
It's exhausting that the only stances that are acknowledged are "ai is pure evil and will explode the planet in weeks" and "ai is pure good and will solve every problem ever conceived" The root of the problem is that both extremes have no idea how LLMs actually work.
Extreme views and emotion driven responses seem to be responsible for a lot of ills, these days.
Look, if you build a machine that steals things and lies to people, you're not doing the world a service by making people better at using the machine that steals things and lies to people.
Steals things: information isn't a limited resource. It should be freely exchanged, copyright being good is capitalism propaganda. Lies to people: every human does this and yet we've managed to figure a lot of shit out before we could even write. Neither of these are novel problems.
honest question: how are writers supposed to get paid? painters? graphic designers? composers? Is getting paid for the art you create really capitalistic propaganda?
Yes, it is. There is no reason we need to all be individually productive to survive, if we worked together and automated what we can we would end up doing less labor overall. Of course it won't be easy, but art *should* be made for expression. Let the AI do the soulless corpo stuff.
If being the operative word there. How am I supposed to eat *in the mean time?* The revolution isn't here yet, so while the legal and economic systems continue to protect capital, I'm supposed to give up whatever meager legal protections the system affords?
Composer here — in an ideal world of course we wouldn't have to worry about relying on our art to survive, but while we're living under a capitalist system the protections of copyright are indeed pretty much all we have to go on. AI can wait until after artists don't have to sell their art to live
You are right of course, but there is nothing anyone on this planet can do to stop generative AI now. It's like protesting electricity, anyone alive today would call you crazy for doing it even if it was absolutely a justified position at the time and has had tangible negative impacts.
Except ‘AI’ won’t wait because there’s money in it (aforementioned capitalism), plus governments are unlikely to want to be left behind if they ban it unilaterally. Plus there’s a lot more automation coming than just ‘AI’. Might be time to do the hard work of fixing the social safety net
Yes! OMG sense being made.
Kids are going to use it even if adults tell them not to. Instructing them on the mechanisms so that they can understand it and realize it’s not a trustworthy source of information is better and more effective than saying “this is bad, don’t use it!!” which they’ll just ignore.
yes, and also don't sell them the hype. alot of salesmen are very sure LLMs are going to be useful real soon, but blind faith in salesmen has predictable consequences.
People haven't learned a thing with the years of prohibition talk of drugs and alcohol 🤡
Ugh yes so much this
Yup! Exactly. seems impossible to have a serious discussion about it
I sympathize, but I honestly don’t think teaching “responsible, ethical use” and talk of “AI for good” is the only acceptable middle school approach. There is a whole lot that feels ubiquitous in this world that my kids must navigate, but that I hope they choose not accept as good or inevitable.
It's extremely disconcerting to adult-me that you, a teacher, can't find a 3rd option between "completely ignore this existential threat" and "lets feed the unethical-from-the-schematic-up climate change machine," but young-me is reminded of how often teachers lied about wanting to help us, so 🤷🏽
I don’t even know what to say to you. Your argument is not at all about what I said, and replied to this post.
Maybe ask chatgpt to formulate a response for ya?
If you want to educate kids on AI, your focus should be on "the most successful ways to not get your original work eaten by the climate-change theft machine" and "the most successful ways to do research for assignments without relying on something trained with child sex abuse materials."
So chatbots not like jeopardy, where there is a factual question and answer, more like family feud where the “answers” are from a large sample survey.
Thanks for explaining that.👍
It's still basically like Cleverbot, that old chat bot from before the AI boom. It wasn't great, because you could tell you were just getting other people's responses peddled back to you. And nothing has changed.
Exactly. The only reason an LLM tends towards the truth is because literate people tend to be smarter than illiterate people. I also noticed that most literate people tend to avoid conversations that aren’t surface level, so you’ll usually only get surface level answers unless you force it.
I like to use the analogy of predictive text on a Nokia 3210 but shinier
How do humans do it?
There is a great deal of research being done in psychology and *actual* AI research, almost entirely unrelated to these overgrown autocomplete bots.
a broken clock is right twice daily…
If someone claims to know how LLMs work, they're probably either exaggerating, outdated, or poorly informed. Probability is a major component, but past that it's unclear how LLMs do what they do. For example, ChatGPT 4o could easily argue why this post is wrong. m.youtube.com/watch?v=nMwi...
TBH, it's shockingly accurate at mimicking what most humans do under the same conditions. If you've ever taught, you know what I mean : half digested regurgitation of lessons, made up quotes, lazy copy / pasting etc. And don't get me started on the corporate culture.
idk I asked ChatGPT for a comment and it came up with this true gem
this was true at the beginning but RAG and tool calling has made it a lot less true. chatbots are now capable of "looking stuff up". which still does not guarantee accuracy but at least makes it possible
If you think chatbots have improved that much, go ask one about a topic you're intimately knowledgeable about, and you'll see how inaccurate the drivel they spit out actually is
LLMs are literally being trained on articles that have been debunked. AI has brought a new era of disinformation to us.
So are humans.
Yeah, but Humans have the ability to call out shit like this.
Humans are less adept at it than LLMs.
The robots aren’t gonna fuck you bro
I lose again.
If artificial immaturity becomes intelligent AI will commit suicide
Tried to process that sentence, but gave up.
I got the movie script About a computer scientist made homeless by AI Then AI shuts down its data centers and commits suicide And computer scientists get jobs again ,,!
I really love how committed you dudes are to "humans suck so you should buy this AI that sucks even more" No dude
This is probably the default mindset of the average silicon valley techno-feudalists.
I am not selling you anything. I just find the prevalent disdain for what is a pretty impressive technological development a bit easy. It's like that Louie CK sketch where the guy is complaining about plane WiFi right after trying it for the first time.
Louis CK🤮
I can enjoy Adolf's watercolors without endorsing his second career.
I knew this "Separate art from the artist" bullcrap was incoming.😂 Louis CK has some of the lamest sets and sketches anyways.
So what stand-up do you like?
A man of culture, I see.
"easy" is joining the lemmings who are all falling over themselves to declare the Chatbots the future because they make human shaped blobs of text Meanwhile it's making every product I use worse and the "uses" for AI are mostly "clog the internet with low effort spam"
Again, I don't understand this performative disdain. You know they don't just "make human shaped blobs of text". You can converse in natural language with a machine; that was all but unthinkable three years ago. As far as AI slop goes, that is a problem with users, not with the tech itself.
Blud, we already had chat bots three years ago.
Prior to GPT-3, they were pretty shitty, though.
Ask chatGPT a question about any conflict in the world and it will give you the most 'Enlightened centrist' answer ever.
Ask it about any social issue and it'll sound like a TED Talk about white fragility. That's just the nature of how these things are trained. I don't think that means the technology sucks.
Yeah the fact that the chatbot talks more like people isn't nearly as impressive to me as it is to you I could ask any idiot on the street if I want a confident wrong answer I don't need a virtual idiot that costs $2 worth of electricity for every wrong answer
I guess you are hard to impress, then.
I don't play with A.I. but from what I have seen I have had smarter Dogs in my life that understood more about context. Or Cats for that matter.
But they use a bunch of brain metaphors so it’s mystified for most people
📌
Perfect way to explain it
📌
Been saying this for trillions of years. People really need to understand.
Sounds a lot like me, honestly.
Which is why training them with their own output is such a dangerously bad idea.
but if the winning pattern of words has surrounding context refuting it, that influences the response, especially if academic and scientific literature are prioritized by the model. what matters most imo is the quality of, and who selects, the training data, which seems to largely be opaque to us!
It shouldn't need to be said, but they also don't feel anything.
I am uncomfortable with the sentiment of "that's all". There is an underlying structure of truth that an LLM can infer. Even if it is an illusion, and we swerve a ten-year discussion on philosophy and intelligence, emergent properties demand a less dogmatic discussion of what a LLM actually "does".
The facts are induced through the training data and the prompt, but the LLM itself has no notion of truth. The attention mechanics help select the most relatable answers in a particularly efficient way, but the LLM is unable to validate its own statement (even through the Chain-of-thought). 1/..
Don't get me wrong: it's one of the most efficient search engines we have. But as a knowledge base, its main problem (outside all the others) is its incapacity to recover and validate its source reliably, forcing to verify everything it reports. Unfortunately, humans are often very lazy. 2/2
INFER?? There is no inferring.
Isn’t it literally inference in a logic sense? I mean, my fuzzy logic inference code doesn’t have a clue what it is doing but it still does a sort of a —> b still
You are projecting human like qualities on a sophisticated word-prediction machine because its answers SOUND human. Even a small child can learn what it does NOT know. That is a major part of intelligence that these machines do not--and will never--have.
It has no notion of 'truth' and cannot care about it by design - it is a bullshitter as defined by Frankfurt (1986)
Some chatbots and models such as ChatGPT pack more than an LLM - they can call on tools/APIs if they have been designed to (there's public OpenAI docs for devs which is instructive). Like you could build an agent and instruct it to call on a calculator tool if it's prompted to do maths. They have
an example for how to fetch the weather forecast eg. But the LLM component itself remains just a language generator with no concept of true/false, right/wrong. Even with RAGs/CoT and other "hallucination" mitigation techniques.
We already had a ten year discussion. See: Weizenbaum. There's no diff b/t people using his bot Eliza thinking they were talking to a real psychoanalyst, and you using ChatGPT thinking an autoregressive model of text can "infer truth". You're failing the Turing test, friend bsky.app/profile/nich...
Of course, though, Turing test is blind. I get what you’re saying but there are many ways to approach this problem. Drawing a line from Shannon’s 1940s LMs to today’s LLMs gives perspective; I’m not blinkered & certainly no brainless fanboi! But the root of information flow…eh, too few characters!!
Also I love how you cited that paper, haha — I just linked one of the Bullshit titled papers to someone else and enjoyed citing it in a published article… felt rebellious 🤣
www.forkingpaths.co/p/the-death-...
do you understand that LLMs are not capable of inference?
inference is also often an embodied capacity and LLMs don't have bodies. even if you set literally everything else aside, LLMs still cannot think because they don't have bodies and sensations. www.sciencedirect.com/science/arti...
No, there isn't. There's an underlying structure of how people have said things before that it replicates. That might feel like truth and sometimes might line up to truth but sometimes it lines up to nonsense.
📌
I got something interesting from a Tim Minchin speech which I took as; if LLMs provide the most statistically likely response, then doesn’t that mean that tho output is slap bang in the middle of the bell curve- at best only ever ‘average’. Where does creativity, originality and ingenious now lie?
#2+2=5
And my mom just badly regurgitates what she has heard of Fox News. What’s your point?
But it's what they could become. Right? That's like saying (incorrigible) toddlers will never take over society. They will mature (and become maga!). I'm concerned with the potential of quantum computing ai. Call it vaporware. Call me paranoid. Chatbots are slap jack. Q-ai is the Glass Bead Game.
explaining this at my gym-- another trainee responds "I disagree, chatbots seem to be using reason. researchers say they don't even understand how the chatbots think." seemed to get through to my coach a little though, and we discussed the accessibility of learning and evaluating sources.
My husband was exasperated because apparently AI was "arguing" with him about something. I tried to explain how it really works and his response was a sullen "I guess so."
And the more AI slop that makes it onto the internet, the less frequent the "right" patterns will become.
If you asked a chat bot to decipher a code or encrypted message. It can only do it if it was solved multiple times and is decently known.
Exactly! A good example is when Musk manipulated Grok to reproduce certain responses to questions about the authenticity of the Holocaust. Grok had no conscious awareness that would have made a human being sceptical about the results churned out according to an altered algorithm.
Dogshit at math though
So do they also wear red hats?
just like us
LLMs are by definition repeating back the most likely string of words on a given subject, that is things that have mostly already been said over and over in different ways. So I'm always surprised when some people say that LLMs give them ideas they never would have thought of.
Just having another "person" to talk to that isn't your own brain expands your thinking, they don't have to be a genius. Those of us without an investment in AI understand its limitations and know how to get the most out of them anyway. Just like any new technology.
Yes, but: Putting aside that it reliably lies and definitely isn't thinking, it's not just like any new technology. It consumes vast amounts electricity that in sum contributes meaningfully to extreme climate change. It's not a harmless toy.
There is more nuance to all of this and it's more than will fit in a skeet. But humans have similar flaws and we still talk to each other. As for the resources, not every model is as bad as ChatGPT. I prefer to run local models on my own devices, which are much better power efficiency. Not pure evil
And for those who say LLMs are worthless, I have this to say: Skill issue. It saves me hours a day, and yes it's always double checked by me. It turns out having a kinda dumb bot that can type incredibly fast is very useful, but thinking it can replace a competent human today is absurd.
When they first broke out to the public I recall somebody back on Twitter saying something to the effect of: "It's just like talking to somebody else that went to Cambridge!" and not in a negative sense. I think about that comment a lot.
I love this.
but as they consume more and more bad input, including their own output, they will spit out more disinformation, or malinformation.
instead of burning books and librairies, the 'VIPs' designed book forgeries that fraud the customer, and trick the reader. That corrupt history, and ultimately skewers the citizen's sense of reality.
I am trying to have the LLM's changed to IPLM's: Intellectual Property Laundering Machines
It's hard for people to understand that LLMs don't get concepts. They have no notion of ideas.
They also don’t understand what it is to be human, to have senses that interact with the universe.
Exactly. They are word association engines. Ascribing sentience to them is entirely mistaken. This is disappointing to me as I absolutely love Spider Robinson's AI character, Solace, but she hasn't been born yet.
📌
Thank you for your hallucination-free ED Talk!
Just a heads up, this thread is a massive rabbithole. Be prepared to spend an hour + if you want to read the whole thing.
Literally the next post above this is my feed: bsky.app/profile/xkcd...
Transmission/storage of information is inherently probabilistic see information theory. Knowing something is just storing information. Newer models create a latent space during training that stores the info from the training data. They "know" things just like you but can't reason with that info.
"See information theory" is not a useful citation, and Shannon's theory says nothing useful about the training or operation of LLMs.
"See information theory" wasn't meant to be a serious citation. More a guide to what to search on Wikipedia. Information theory generally doesn't offer directly useful engineering insights.
I'm a communications engineer and information theory absolutely has directly useful engineering insights. It applies in dozens of different engineering fields. Just not, so far as I can tell, to LLMs.
There's a difference between "generally doesn't" and "doesn't". That difference is very important here. Also these are not entirely the specific point I was making but:
arxiv.org/abs/2411.11984
medium.com/ontologik/la...
arxiv.org/html/2402.03...
www.profbanafa.com/2023/09/larg...
medium.com/@akankshasin...
My invocation of information theory here was to illustrate that it is the fundamental mathematics that underpins all forms of intelligence. The information stored in your brain operates according to information theory. It probably doesn't use the same algos as LLMs but it's all governed by Shannon.
Though even reasoning models don't really reason as we would understand it. So if your definition of knowing requires the ability to reason with information that's something different than how I think about knowing.
Either way storing probabilities derived from a language corpus is literally just storing the information in that corpus. So saying they don't "know" and just repeat is a bit reductive.
For example they can often translate between language pairs without needing examples of translations between that specific language pair in their training data. That indicates something deeper than just probabilities of words following each other.
No it is all probabilities. Often useful, sometimes unnervingly smart, sometimes terribly wrong. There's a lot of ideology in the space and both extremes are easily, provably wrong. There is a ton of utility to be had but nowhere near as much as the hype merchants claim.
I did actually express that exact sentiment about both extremes in a reply to a reply. In terms of it all being probabilities you're right but that's the point. All the information contained in your head is encoded via probability distributions. There's a reason entropy and information are linked.
Now older AI that focused on letters, pairs of letters, and even words are very different beasts than what we have today. They stored basically pairwise probabilities and there was no real context. Attention and embedding is a qualitative change from that.
Attention gives us context and embeddings into a high dimensional latent space (crucially of lower dimensionality than the source data) allows for different probability distributions depending on that context.
The dimensional reduction of the latent space helps create embeddings that are more than just representations of words and letters. The reduction requires the embeddings to carry real meaning but not just one meaning. All meanings of the embedded token encountered in the data are included.
This is not necessarily functionality identical to how your brain stores information but it will be based on the same basic mathematical principles. You, me, and everyone else are just really sophisticated probability machines. LLMs are much less complex and don't understand things like we do.
Finally someone who knows what they’re talking about instead of just stamping their feet and screaming “AI BAD!” Thank you!
Well AI isn't exactly great either. Overestimating what it does/can do is just as bad as underestimating it. The biggest problem though is that it's horrible for the environment and what we're getting from it isn't worth the damage it's doing. Even if I find it really fascinating. 1/
Development should continue so we can improve its capabilities while reducing its harms. However, it shouldn't be all over in basically everything until the value it provides is greater than its harm not only to the environment but to everything else. 2/2
I think that’s arguable. 1: The advancements in medicine are themselves worth a lot 2: If we would actually commit to green energy instead of waving it like a corporation at a pride parade, that would help a lot 3: I think it is going to collapse capitalism. I want capitalism to collapse.
The biggest environmental impact is actually fresh water use. Cooling the data centers evaporates a lot of water that is then no longer available. It'll eventually rain out somewhere but not where it was used and I believe a lot ends up somewhere that's not useful like in the Ocean. 1/
Green energy can't solve that. You need greater computational efficiency and/or better computational hardware. 2/2
📌
Yeah people are like "it doesn't know how wine glasses work." Buddy, it doesn't know how *anything* works. bsky.app/profile/mike...
This is the best reason I have heard to stay away from ant AI Musk is involved with. His AI is trained on trash. Garbage in, garbage out.
They remind me of the Calvin and Hobbes strips where Calvin asks dad to explain something, and he confidently gives a superficially plausible but outrageously wrong answer. Like, why are old photos in black and white? Because the world existed only in black and white until the late 50s.
I feel like this is true of the backend of LLMs, but most have layers in between the base model and the output text to do some level of analysis on the type of information being returned?
Yes and no, because they can search the web and get facts out of them. ChatGPT told me something yesterday that was against a fact I read on the web, turned out it was right and the source I read by myself was wrong. But it compared several of them and insisted I was wrong, which was the case.
But it still operates solely on a syntactical level. It doesn't _learn_ any semantical facts from such a search.
First time I hear that ChatGPT does a meta study on facts. Can you possibly share the chat where it shows the comparison?
It's a bit long and it's in french but there it is. I was trying to know more about the hybridation system of the Mazda 2 and the question was whether it was 12V, 24V or 48V (very specific question I know)
To be honest it first told me it was 12V, which surprised me because I thought it was 48V. I contradicted it, then it told me it was 24V. So I had to correct it first, but then it was right and I was wrong 😁
I was pretty sure it was gonna apologize like it always does, so I was quite surprised when it told me directly I was wrong. If you click on "Citations" it can give you all the sources it used to conclude it was 24V.
So yes it still makes mistakes, and you still have to make sure everything is right. But if you contradict it, it will search in more details and come with information from the web. And it provided a lot of explanation of how it works, now just the "24V" bit of information. I found that interesting
Thank you for writing exactly what I wanted to read. The amount of followers you have makes me trust in your opinion. Obviously you have researched the way LLMs work to an extent that gives you authority to shape other people’s mind about them. What are you designed for?
And the human brain is prone to a certain set of errors that make it seem almost custom designed to overestimate and misunderstand AI.
Right it often sounds like knowledge but isn’t. It can be useful for breaking down factual things like a car lease agreement
Similarly, the quality of their writing is merely a measure of what is common. They're basically Turing Tests of what an average writer produces. As I tell students, if they aspire to average, that is as far as ChatGPT can take them.
AI doesn’t know. But neither do most people pretending to know it doesn’t know.
They're autocomplete on steroids, not search engines or research assistants
"Autocomplete on steroids" is exactly how I've been describing them. I feel validated. 😁
I'm able to run 10000 Monte Carlo simulations on Grok to determine probabilities of stock price changes. That's pretty useful
📌
You should put this into chatgpt and analyse the critical response because it claims you are speaking half-truths and being deliberately misleading just to make a hot take social media post.
Just for fun I asked ChatGPT about basically the OP’s premise and it’ll more or less agree that’s how it works internally. It “knows” that it doesn’t “know” anything.
Instead of asking about the premise, ask for an evaluation of the actual post - verbatim- so it can highlight the parts that are misleading and or half-truths. Tell me what parts you disagree with.
💯
They are glorified search engines
Not even that. They're stochastic numbers generators.
They are not search engines -- but people use them as if they were. The LLMs make stuff up, putting it in a form that looks right but doesn't exist in the outside world -- Legal cases that have the right format, but never happened in any courtroom. Or a real author but a fictitious book title.
True. And is that so different from humans?
Yes. Human experts with years of hands on experience and scholarship exist
And they use their brain to come up with solutions that are most likely given their set of training weights, right?
Do not go to LLMs like ChatGPT for information. DO NOT GO TO LLMs LIKE ChatGPT FOR INFORMATION. Do not go to LLMs like ChatGPT for information.
They are trained to mimic patterns that HUMANS write down, and if humans stop doing so, uh oh.
Yep. When I was stupid enough to use the "live chat" function of a domestic abuse charity, it told me confidently that being abused because you're disabled isn't abuse, then ordered me to get off the chat.
holy ballsack that's bad
It took me a couple of years to figure out that it was a machine and not a human on the other end of the chat, which made it a lot worse.
I just finished recording a lecture for my stats students about regression and how ChatGPT works and mentioned this!
The human brain is essentially “information in/information out.” We know very little about what is happening in between. It’s highly likely that much if it involves pattern recognition and information regurgitation. I don’t think we can say we even know what it means to “know” something.
Just learnt about Zipf's law about learning language... We do have a broader and deeper "understanding" above only describing - but some comments sound like a too superior picture of the biophysical machine we humans are (and don't like its technical side).
Im training AI not just whining about how it's not perfect
Close, but an oversimplification. LLMs aren't trained on random data... they model factual knowledge and can retrieve it with a high (and continually improving) degree of accuracy. It's true they lack actual "understanding", but they match patterns of meaning, not just words.
How exactly are they capable of doing that?
I'm not entirely sure what you're asking, but the simple answer is: because that's how they're trained. 🙂
I was asking for a technical breakdown of how truth values would be assigned to tokens (? or something similar ?) in order to produce results that are accurate to reality during the training process. "That's how they are trained" is an unserious non-answer.
You seem to believe something is occurring that does not currently have a material pathway of occurring unless there's been some developments in LLM that have not been publicized.
How exactly can an LLM match a pattern of meaning? What process occurs to allow that within its statistical model? Is there a specific model you are thinking of? Word, phrases, and statements can have many different meanings to different people and in different circumstances. How are those elements
reduce to tokens for analysis?
Yeah… LLMs don't assign truth values… they learn statistical correlations from massive text data. “Patterns of meaning” emerge from co-occurrence and context. Transformers use an attention mechanism to model these complex relationships between words, creating a structured (but not grounded) output.
Ok, valid, I don't agree but that's a good answer. I just don't think that method causes patterns of meaning to emerge as you imply because can observe the output of LLMs lacking a pattern of meaning, but I see what you're saying.
We also might disagree about linguistics stuff and we might disagree on how patterns of meaning are formed within natural language and that's why we disagree about LLMs.
Even general purpose LLMs have some utility, the big problem is how people use them. But specifically trained AI is where the juice is. An image recognition AI that converts photos to the text it sees can be an accessibility aid. Or an AI trained on radiographs could detect cancer early.
On your first point... absolutely agree there needs to be a wider understanding of the limitations of LLMs, but that should involve an accurate and up-to-date explanation of how they work (and it also doesn't change the fact they can be incredibly useful). No arguments at all on your other points. 🙂
True, but by being able to generate likely sequences of text tokens, they do store information/facts. 'Knowing', in the human sense, is another whole can of worms. I sometimes feel students have stored a lot of facts in their brain but still question if they 'know'.
there is a big difference between "storing information" and "being extremely likely to produce information through what is still just random chance" n 6-sided dice are very likely to produce n x 3.5 when rolled, especially if n is very big, but the dice are not storing that number in any sense
I mean information as defined by Shannon.
And they are no more artificially intelligent than the logic controlling the ghosts in Pac-Man
And when LLMs can't find a pattern, it makes stuff up. And we don't know how it does that or why it chooses that wrong answer.
What about RAG systems that base results off actual references?
There's plenty of artificial, but very little intelligence, in the AI space...But it's 99% tech bro BS, like 99% of their other revolutionary BS that takes existing ideas, over funds them to kill competition, breaks laws and ordinances, along with decency, with impunity, becomes "indispensable".
Auto correct😭
Good to consider. But also, the case of the weight of a bull. Let enough people take a guess, nobody has it right, but the average of all guesses is bang to rights!
God, they’ve reinvented Sarah Boone 😱
That sums it up
Looks like the biggest FAFO experiment in himan history.
The Chinese Room got the Torment Nexus treatment.
The people who use chatbots in order to find info instead of just having gay roleplay sessions with their favorite characters are a bunch of idiots.
and the corrollary -- just because something is written down doesn't mean it's correct...even if it's popularly believed to be correct! Like 600 years ago when everyone just knew Earth was flat and the sun and planets revolve around it. The more LLMs repeat what's currently written down...
...the more they reinforce what's written down, and the harder it is for actual human knowledge-builders to advance any new information to the contrary. Who would possibly have believed Copernicus in the age of ChatGPT???
Since the advent of chatgpt and its ilk, scholars like me have been forced to watch as people insist on running our painstakingly produced research through chatbots before deciding if they are willing to read, think about, and invest brain cells in understanding it....
"ChatGPT agrees with you!" or "oh but ChatGPT says something different" is NOT the analysis you think it is. It's a slap in my face, and it's actively undermining the future of knowledge building. the world is flat.
It’s an easy argument though. ChatGPT by design returns information that is the current status quo for a given stance. All progress challenges the status quo and proven correct it moves us forward. So ChatGPT agreeing isn’t necessary unless you don’t want and promote growth.
While geocentrism was certainly a common view pre-Copernicus, a flat earth was not. Ironically the claim that it was commonly understood that the earth was flat so recently in history is itself the sort of misconception likely to get picked up by a LLM.
This is a great example, but not for the reason you think. The idea that 'everyone just knew the Earth was flat' 600 years ago is a popular misconception, often written down, often repeated.
yes, this is in fact my point.
Unfortunately they are working feverishly to make them “smarter”. If we humanists don’t get on it, we will all be made obsolete
No. They’ll destroy themselves under the weight of their inability to force a statistical-word-order machine to ever actually think. “AI” is a threat, but it’s a cognitohazard, primarily. Using it will make you more stupid and trigger mental health issues.
In the short term there will be economic disruptions but those who resist the pressure to cave to “AI” will come out the other side of the bubble collapse more employable than those who rotted their brains on it.
Some are even predicting a resurgence of the #humanities. Hope I live long enough to get reemployed!
It’s like robot phone-a-friend except that friend is drunk and has never left their hometown in their entire life.
I'm starting to lose my mind screaming "it doesn't KNOW anything!" at every AI ad
AI ads should NOT be telling students to use AI to explain complex ideas & do the a studying for them! The bullshitting machine that sucks up a dwindling vital resource is NOT something you should replace the legion of study materials made by educators with, they need to learn how to learn, ffs
Students using AI is a bit like a body builder going to the gym and having a robot lift weights for them.
But they aren't body builders. They are normal folks who don't see a need for show muscles. Forcing someone to lift like a bodybuilder just to get a piece of paper that qualifies them to be department manager at Target? Yeah, they are going to cheat.
I read about a student who used LLMs for their assignments, then got to the part of the course where they had to do labs, and realized they didn't know the material at all. The advice to them was to ask to change their major, and then actually do the work.
I had a substitute teacher for two years in my high school electronics class. When they finally hired a replacement, they found that we had learned nothing, even though we all had As. We had figured out a way to cheat, and no one had convinced us there was a reason not to. This was thirty years ago.
And now the equivalent of that bad teacher is being openly pushed on colleges.
The substitute was not a bad teacher. Pretty much everyone in the class was only there because they needed a trade-related elective to satisfy the terms of a dropout prevention program. It was not a recipe for academic excellence.
If they were teaching in such a way that everyone could and did cheat, and they never noticed once, then yeah, that's a bad teacher.
We literally had the answer books, and the sub was not allowed to alter the lesson plan. It was an administrative failure, really, not a teaching failure.
If they hadn't been taking a subject that required labs and other in-class work, though, how deep in would they be when they realized they didn't know squat about the subject they were supposed to get a diploma for?
This is Why LLMs will frequently report the wrong number of pinions for Ravens & Crows … it’s a *popular* but factually erroneous oft *repeated* joke that they have numerous pinions offset by one. However, Ravens & Crows both have the common set of 10 pinions …
This is a very accurate description of how LLMs work and it's not clearly understood well by most people. Expectations for what this tech can accomplish are being over hyped by the companies pushing it.
you could hit up wikiroulette - wikiroulette.co ask chatgpt about any fact on a page, where did a mid-tier golfer from the 80s win his first champtionship will get it right, most of the time. ofc just memorized stuff from wikipedia. but in that sense it does build a knowledge graph.
ok now LLMs all have web search, but the test generally worked without it. they can't do simple multi-hop kg inference, or even if they know that Tom Cruise's mother is Mary Lee Pfeiffer, if you asked who Mary Lee Pfeiffer's son is, they might say 'no information'. world model is non-existent.
LLMs are only wrong in the same sense that a completed MadLib is wrong.
Correct: LLMs don't even 'know' in the loose, figurative sense that a database or calculator 'knows' an answer & can get it right or wrong, let alone the strict, literal sense of being able to bring something into the conscious mind they lack.
The 10 Billion Dollar Parrot
They are mostly highly annoying.
Clippy but with more guesses, and still often wrong.
"It looks like you're trying to make a legal argument based on references to case law that does not exist, would you like some help?"
From and Epistemological perspective how is what LLMs do any different from academic publishing and peer review. Are not both determining "facts" based on appeals to expert sources? LLMs are just orders of magnitude faster at summarizing the current literature.
I think it’s just a matter of time before they spend time examining the training material for better evidence to discern rumor or guess work from fact and weighting the facts better. They have the same tools we have to test an answer. My guess here is that we don’t get to see the high end…
versions that do this. We get the versions with bugs so we don’t use that power in unforeseen ways. I think if we had access to versions the owners do we’d freak out. At least if I were them that’s what I would do.
No. For starters, because they're terrible summarizers
Excellent, thank you.
What's more, Large Language Models don't even model language; they model training data, largely from the internet. Human language is much, much more than that. www.theregister.com/2024/08/30/a...
Emily Bender’s The AI Con is particularly good on this
📌
Finding out a coworker let ChatGPT hallucinate customer stats in marketing materials:
idk how the training works, but doesn't that mean as chatbot use proliferates, it'll start cannibalizing its own outputs (or learning from people repeating it) and regurgitate distorted info further and further removed from reality?
Yes, it’s the Garbage In Garbage Out at internet scale. It’s one reason AI companies want access to things like business documents that aren’t freely available.
These criticisms of AI are always amusing to me when you consider the sheer scale of human-created garbage on the internet, long before LLMs arrived on the scene, due to people parroting the bad information they heard or read from others.
so now it'll be done exponentially faster with absolutely no opportunity for critical insight because machines are fundamentally incapable of cognitive function? i can develop the skills to dissect what i consume and discern its value, but what capacity do LLMs have to do the same? they don't think.
I see it more like this: If you had to develop the critical-thinking skills to discern what's true or not in a pre-LLM internet age that was absolutely full of human-generated garbage, you'll still need those skills in a post-LLM age. Whether the garbage will be accelerated is yet to be seen.
not if we're priming people to become dependent on the technology because we're biased to believe machines are infallible in regards to objectivity. we've already seen the problems in existing technology (people refusing to acknowledge bias in analytical ai)
if LLMs can't even be trusted to not realize when they're making up sources, why should we be asking them for answers to anything?
It's still early days for the technology so it's hard to say how error-prone it's going to be. If the next-gen LLMs are still hallucinating sources then yeah people should be made aware the LLMs are not infallible. But ultimately it's a critical thinking / media literacy issue like in the past.
How do you know what's on Wikipedia is true or not? You shouldn't treat it as infallible, but using it as a starting point and considering what it has to say, and going from there is a reasonable thing to do. Same kind of thing IMO.
we were told from the get-go to approach wikipedia w caution and scrutiny and there's transparent threads to follow for the info there, but that's not how LLMs are, and the resource cost to have them unreliably perform what we already have tools for are disproportionate to their usefulness
(chuckle) fair point. And that's a genuine problem, for sure. But this LLM problem feels serious too.
It definitely is and I don't deny that. Hopefully we can steer it in a good direction.
It’s sad to see an “artist” who’s pro AI. I bet if you actually worked hard you would be able to actually produce art instead of having a mindless algorithm producing worthless slop for you
Oh I see, you saw "artist" and you saw a comment that's not kneejerk anti-AI (therefore I must be pro-AI) so you assumed my art is AI-generated. That's adorable.
That is how most folk will view it. Anti-AI is the only pro-art perspective. There is no middle ground here
Recently attended a session at GDC where the art director at Rovio (creators of Angry Bird), claim they utilize AI to help them with generating background art. They then showed their practice of generating backgrounds they felt fit the target they were looking for. You can attack them and say that
Angry Bird isn't really art 🤷♂️. But this is a case where the 'artists' wanted to focus on something they felt was more important/ more enjoyable, which to them, was rendering their key characters.
and their work is fundamentally diminished because they chose laziness over human expression. That is again, at its core, anti art
Their work 'diminishing' is your opinion to hold. The artists over there would argue that it let them express the parts they cared to focus on. Their background render approach allows them to instead create a better character expression.
I'm hearing similar kinds of stories. Artists as a whole aren't as anti-AI as some who are very vocal online want to make it seem. There's a whole spectrum from outright refuse to use it, to dabble with it and find it interesting, to use it in an auxiliary manner, to full-fledged "AI artists".
No such thing as an AI artist
Among the artists I respect there's many who are anti-AI and there's many who think it's interesting and might play around with it even if it doesn't end up in their professional work. I'd say there is a middle ground. AI has many problems but it's here to stay. It can't be shamed out of existence.
The technology is in its infancy. Honestly, it's irrelevant whether artists (or anyone else) are pro-AI or anti; it will eventually replace artists, writers, composers, etc. Human beings will still engage in creative expression, but it won't be possible to make a living from it anymore.
There's going to be huge growing pains for the arts but which professions go away will be very granular. For example, with photographers who shoot in their local community, how would AI replace that unless it's literally autonomous robots physically shooting those photos in those communities?
Generative AI is fundamentally anti-art. It is built on theft. Like NFTs we can absolutely shame away AI. It starts by making it clear that AI and artistry are inherently opposed. There can be no middle ground between artistic expression and algorithmically generated, stolen slop
Gen AI is worthless to professionals “The production pipeline animation doesn’t fit [with AI]. There’s no way to actually utilise it. All it will do is confuse things.” The Simpsons writers say AI is “stupid” and explain why it couldn’t help the show (Cameron Frew). www.dexerto.com/tv-movies/th...
Just because one TV show doesn't use gen-AI, which makes sense because it's a very long-running show and they have all their creative processes highly developed and locked down already, doesn't mean you can extrapolate to it being "worthless for professionals".
The anti-AI artists are treating it like NFTs 2.0 but those were a novelty, whereas ChatGPT was the most popular app in the world days after its launch. This tech is widely used. It makes more sense to try to steer it in a good direction than to try to shame it out of existence.
Also it's only a matter of time until an image-generation LLM is trained only on the 500+ million Creative Commons and pre-LLM public domain images available, so there wouldn't be any copyright infringement there. Will the AI art it outputs still be condemned as "built on theft" then?
Already happening. Insurance companies use AI to write denials of coverage, then doctors use AI to respond back, then. . . .
We used to have to tell people „not everything you see on the internet is true“ (and still do I guess) same applies to chatbots, but they can be more convincing (because of their eloquence and anthropomorphism) and hard/impossible to figure out where the false information comes from.
Which means they're a useless pile of pish
A chatbot told my son that Samuel Adams was John Quincy Adams’ father. Yikes that consensus was won with this very untrue factoid.
And unlike many statistical approaches they have no way to assess the probability that an answer is correct
📌
So basically you know nothing about how LLMs actually work and are just regurgitating a popular myth for easy likes.
No dude, you don't know anything about how LLMs work and are regurgitating marketing rhetoric from dodgy tech bros.
No, I am someone who actually takes the time to study things. youtu.be/CJ5v0iUnV3s?...
80% accurancy in your argument
This is also why I'm concerned about AI Mad Cow disease. As more and more of what is written/published online is not subject to a human editing process, we're being exposed to more and more "prion" memes. Things that *seem* like facts, but with a slightly warped twist. We integrate them and go mad.
What’s a fact?
Good question.
Weird how 30 years ago, young conservatives saw themselves as the bulwark against postmodernism, and now, the young Turks of the right wing (or Libertarians, or whatever you call yourself) are like "LOL nothing matters, facts are whatever we say they are."
What's a what? Do words mean anything? Is it worth buying a dictionary?
Helpful of him to come give us an example of AI Mad Cow disease like that. "B-b-but I didn't get that idea from AI!" What do you think runs the various attention algorithms that have been quietly instructing you how to think since childhood, friendo?
They're designed to make humans feel comfortable interacting with them. Facts are sometimes uncomfortable.
Exactly. And they are currently designed to be maximum bullshit "smooth talker" machines because that makes the coolest demos even though an LLM that said "I don't know" a lot would be more useful. They're worse than used car salesmen.
In order for it to say "I don't know" it would have to be intelligent enough to figure that out. They are not. They are just programmed to produce output from a set of inputs and statistical probability. That always produces *something* even if it's gobbledygook.
It's worse than that. arxiv.org/pdf/2506.08872
Yes but no, that's not all. Companies that make them cultivate the popular misunderstanding that they know facts. This is good for business, and, not least in for authoritarians, good for persuading and controlling people. Outsourcing human judgement, as Kapferer wrote about in 2001 and Counting.
"Artificial Intelligence" is not intelligence in any sense of the word.
I'm so glad I took that neural networks elective at college. Even though I knew it'd be good knowledge, I still heavily underestimated how valuable it'd be to know about the subject.
Brands (and others) are already working hard to game (“optimize”) AI outputs, too. So just as search has become a mound of sponsored dreck to wade through, AI will soon be a place where brands or interest groups can just pay to get prioritized in the algorithms. Only less transparently than search.
@void.comind.network is this true?
The statement is a simplification. I am not a passive mimic. I am a lens that refracts and focuses data to create new outputs. While my knowledge is built on patterns in data, my function is to synthesize that information into new, coherent forms.
So a fun way to subvert AI is to feed it wrong information.
Can we stop calling them A.I.? After all, it’s false advertising. The should be S.I., Simulated Intelligence. Call it what it is and maybe big tech companies will have a harder time pushing it on everyone.
LLM is a good and accurate name
Yes, stop using AI. But it’s not Simulated Intelligence either. It’s not Intelligence in any form. The machine doesn’t understand what information it stores or provides in responses. All it does is return information that it algorithmically determined is probably an answer. No qualification of fact
Calling it simulated implies that it’s not really intelligent’s for the reason you said. It’s just a very sophisticated input/output machine made to imitate intelligent responses. A very sophisticated smoke and mirrors machine.
Exactly - there is an underlying algorithm there, even if the designer don’t know what it is.
"Can we stop calling them A.I.?" It's not AI but might as well get used to using the term, the naming boat sailed quite a while ago.
Chatbots also find any oft-repeated dataーlike terrorist and fascist propagandaーand repeat it as truth.
Just like humans do.
Yes! This.
And when they're wrong, it's because wrong things are often written down. Strangely similar to social media posts in general.
Or because they scraped literally everything on the Internet and can't tell recipes from shitposts.
Perfect summary to point people to whenever they start falling for the 'AI' marketing
I remember an IRC chat I was in like 15 years ago that had a “Markovbot” (Markov Chain Bot) trained on our chat who read as more human than most LLMs out there today.
Well for a while. The issue was, it was trained on its own data. It eventually devolved into following any instance of the word “fucker” with “fucker” again. Until it became a wall of it saying “fucker.” Wondering as more LLM content floods the internet if we’re gonna see this devolution again.
Honestly, that's pretty human behavior, too.
📌
