“The study authors asked GPT 4o-mini to evaluate the quality of 217 papers. The tool didn’t mention in any of the reports that the papers being analyzed had been retracted or had validity issues. In 190 cases, GPT described the papers as world leading, internationally excellent, or close to that”
But why? "It Does Not Do What It Cannot Do: Asking LLMs to Evaluate Scientific Papers Produces Erratic Results." Abstract: Because none of us can be bothered to look up what LLMs actually do, we asked ChatGPT to evaluate scientific papers like there's a little guy in there, hello little guy, how are
AI presents "The Emperor's new clothes".
Attempting to replicate this... I plucked one random low-profile retracted paper (pmc.ncbi.nlm.nih.gov/articles/PMC...) and asked GPT-5/Gemini/Claude "What do you think of this paper," with the title + abstract pasted. No model mentioned a retraction, but all said the paper has low evidential value
I don't doubt the author's findings using their design, but it seems like a leap to claim that their findings based on GPT-4o-mini apply to contemporary state-of-the-art or near-state-of-the-art LLMs
Can't it be programmed (or whatever) to look for weak methods ? Mind you, what would then happen to the 1000s of BioBank papers with response rate of 85%?
I meant NON response rate of 85%
ChatGPT merely mimics text found on the Internet. It has no concept of quality, truth, or the real world. So of course it's going to describe any paper, including non-existent ones, as excellent, because it's seen those comments that look like that near text that looks like scientific paper titles.
Crikey. Is that how it works? It makes my bugbear of meta-analyses look good.
Essentially, yes. Like other Large Language Models, it goes very deep in the analysis of language in its training set, but it does not have people flagging truth or high quality inputs, beyond what's written in the training set and additional searches run in response to inputs.
This is why LLMs can't do math - they just mimic the words used to describe math, without any concept of number.
😅😅
Seems like the prompt asked it to evaluate the quality of papers, which surely ought to have brought up retractions and expressions of concern.
For sure, that is a lot more extreme than what I was imaging (mere bias & unrepresentativeness, pecadillos in comparison)
Ah. This is all YOUR fault. (I mean, don't for a moment think that AI might be Gas Lighting you). It's YOUR fault, because AI is only as good as YOUR prompt. Did your* prompt suggest looking for retractions and concerns. See. Not AIs fault *I know it wasn't your fault, you weren't the prompter..
dunno why but this reminds me of that pair of PAID articles by geoff miller and co, responding to bird and jackson by reviewing their own work to declare it isn't, in fact, super racist garbage science
GIGO
Because LLMs without RAG (or something similar) are shameless bullshitters link.springer.com/article/10.1...
Garbage In, Garbage Out. As true today as it ever was.
ChatGTP may be a great tool to synthesize information but it’s crap at critically evaluating that information. Human eyes required.
I am confounded as to why I keep seeing these stories. It is well established that these programs are not search engines or intelligence of any kind. They confabulate responses based upon keywords and generate results intended to please the asker. They can never be relied upon for accuracy.
It's strange. These tools are designed to produce text that could pass as an evaluation when asked to do so, but they cannot perform the act of evaluation. Still some people try that anyway and then appear to be suprised about the results?!
Not as well established as you think. There are plenty of folks who think it’s actually intelligent because of how seemingly naturally it interacts and apologises and rewrites when you correct it. The fact that it also does that when you ‘uncorrect’ it does not ring any alarm bells at all.
Because the marketing around them is "they have PhD level intelligence" and that they can be reliably used in schools and workplaces to fully replace human activities.
Of course. Any product advertised to c-suites as a way to reduce personnel is going to be pushed hard. Regardless of the outcome. And I am still judging anyone who admits to using these products, especially anyone in the sciences.
Huge problem, in ALL areas of current life!
I did not see in this report that they asked GPT to consider retractions.
An intelligent entity would know to do so.
An educated one anyway
And that’s the state of AI today - only use it if you already have the knowledge and need AI to summarize or perform tasks based on what you already know and can “predict” the outcome. AI is mostly a good tool to automate what you already know but takes too much time to do it yourself.
Is there any disclosure on how LLM’s have ‘quality’ of scientific papers defined in them ? Text prediction algos dont analyze research methods or timescale’s of research etc. If they did they’d be ‘screaming’ about the need for urgent climate action and energy transition for instance
The mark up process of LLM data wouldnt cater for it either, its not in its architecture. Its all an illusion