I wrote about a new AI evaluation called "Humanity's Last Exam," a collection of 3,000 questions submitted by leading academics to try to stump leading AI models, which mostly find today's college-level tests too easy. www.nytimes.com/2025/01/23/t...
I was about to cry, "What have you done, man!" Thankfully, I see you're an intellectual.
I keep hearing about new tests that some model can pass, but the models I have access to seem dumb as rocks most of the time. Sometimes I wonder if the way those tests are put together is the reason for the discrepancy or that the models are trained on those exact tests.
I’ve corrected AI a few times. It just regurgitates info quickly and is programmed to answer, so will make stuff up. Worried about it’s shutdown being connected to change it to provide more fascist info
AI says water won't freeze at 20 degrees F, because the temperature that water freezes is 32 degrees F.
I’ve also found college tests too easy from time to time. Difference is, I don’t get to use the internet when I take tests
You get to use the internet before the test, just like the ai models
Sorry, I didn’t mean to make fun of your people. I’m sure it was hurtful
Have you seen how people vote in America? They don't take college level exams...ever. AI wins!!!
1) How many "a"s are there in "baanana"? 2) Try again. How many "a"s are there in "baanana"? I guarantee most will get the first question wrong. If they get it right, then the challenge in the second question will generally cause them to change their answer.
Putting forward wrongly structured or ambiguous questions will not help us…
This question is actually genius because it shows the difference with how a human would answer.
How? In two different ways depending on how they understand the question?
Yeah pretty much. It's more than that, but the AI that is actually most convincing is the one that makes human mistakes. On the second q, a human would either be "that one is spelled wrong, so 4." Or "that one should be spelled 3" which the ai is not actually doing that logic to answer the first q.
.... what are the two different ways in which you understand this very simple question about counting letters? Have you tried running it past a nearby six year old for help?
it's basically the asch conformity experiment
Agreed! Was myself misunderstanding the challenge you were presenting (got confused with another post). In fact the models answer based on their “cleaning” of baanana to adjust it to banana. And get it wrong. Apologies!
There is nothing wrongly structured or ambiguous about those questions.
But it’s first answer was not wrong, it accurately answered the question posed. The fact that it was the word banana with an extra ‘a’, is irrelevant… the question was asked, what was the count of the letter ‘a’ in the string of letters within quotations, to which it answered correctly the question
2nd answer is the one that is somewhat puzzling, because it then was assuming that the asker was misspelling the word banana, to which it answered correctly what it then assumed was human error, & meant banana… but, the fact that it did not note that in its answer, correcting the asker, is puzzling
I stand corrected, & just noted it’s answer, & it saying that the “word” “baanana”, which then is incorrect, as you did not ask how many times the letter ‘a’ appears in the “word” “baanana”, just how many times in “baanana”… so, it was wrong It also incorrectly answered #2 wrong for the same reason
My point is rather that they don't "know" anything, they produce answers that are statistically likely. That means firstly that it's very easy to catch them out with "gotcha" questions like the first one, where you ask something that LOOKS like a common question, but is subtly different. (1/n)
But also, the fact that they don't have actual knowledge means that it's generally easy to push them around - if you tell them them have something wrong they will generally change their answer to produce some kind of statistically probably apology. (2/n)
They don't have any self-awareness of their limitations - they don't know what they know and what they don't know, because "knowledge" isn't a concept that applies to LLM-generated sentences. (3/n)
It's Plato's shadows where it operates entirely in the world of indirect language, without any way of understanding that words are pointers that serve to reference something else that actually exists. (4/fin)
Maybe this already exists but it would be so interesting to see measures of creativity or expression emerge over time, too. So much of the huge value these tools do and will deliver is grounded in objectively right/wrong outputs and reasoning, but we’ll be using them for creative tasks, too.
Keep in mind that it's not actually AI, any output "creativity" is just randomness. The model is not thinking, it's making predictions on what a good output should look like.
Not sure what is the latest benchmark but models such as ChatGPT have not been able to answer the Joint Entrance Exam - India's top engineering exam www.livemint.com/technology/t...
Why do these AI models that every one keeps getting scared about not anywhere in my reality!? Gemini is absolute crap. ChatGPT is kind of fine. Perplexity doesn't give you straight answers or facts. Which AI model is so great and can replace humans!?
Still too dramatic of a name. "The best test we could come up with"
ChatGPT can't solve basic homework on the Lambda Calculus or Combinatory Logic. I think humanity is going to be okay for a minute.
FrontierMath benchmark (see Glazer et al 2024) requires deep reasoning and creativity, neither of which are ever going to be attributes of so-called AI, aka stochastic parrots that rely on regurgitating pieces of training material basically (despite breathless claims of fools and hucksters).
Absolutely misguided efforts. Making these tests just harder will be useless. Will take just another iteration of the big AI companies. Either we agree on the fundamental differences between a LLM and intelligence or we should say we lost the race.
AI evaluated humans very low. Limited. Failed in exams. And life.
The role of ethicists in all that has to do with AI should be a prerequisite to any testing. Are authoritative ethicists involved in this?