Yet!

How do you put citations to papers that DONT EXIST?? That just sounds… so… idiotic?? Like you’re actively expecting people not to check your sources, which, the fact it’s about openAI makes me not too surprised

Gemini will even talk about fake citations as if they are real for awhile.

I like to work with AI but the Idea to let it write something for you instead of just proofreading something for you is just wild.

I actually use it in the reverse, to draft tables or summary text and then review it. But I usually have a structured text that I expect it to produce from multiple documents.

The problem being that some people don't see this as a problem 🥺

Huh, l use o3 + Deep Research almost daily and have yet to get a fake reference. It’s been great at finding important papers in a research niche for focused lit reviews. My main problem with is how credulous it is wrt claimed findings but as long as I read everything it’s citing, it’s great

Out of curiosity, where do those two links go? (I wonder if I miss times where it mis-describes a paper but still links to something useful)

💯 I find that the made up citations issue is real, but seems to vary with model and mode. "Deep research" doesn't usually make stuff up, but it does repeatedly cite the one not very good source - a Frontiers article, a Wikipedia page, etc - it happens to have access to on a given topic.

The made up links often go to some other paper, maybe related authors or topic. The imaginary references are often v credible, with authors who work on the topic and real journal names, etc.

I find exactly this using Gemini deep research when I fed it lit review titles I give to students. The references it provides are real, and the links go to the right place. They are on topic, but almost exclusively to low quality papers, and rarely provides any critical analysis of what it cites.

But then u find that is also true of the majority of student lit reviews.

When it generates wrong citations, they often look very credible: a reference author, a title in line with what you are looking for, and a reputable journal. Almost too good to be true, which should be the hint that it's fake.

They’re all links and I have yet to open one that’s not very topical to my query

In this case, the link goes to a completely different paper.

Exactly, or they do exist but are mis-cited

They were just saying any old shit five months ago

So, I asked ChatGPT about this. They cited a paper by A. Pseudo saying that it can pull accurate citations. I couldn't find the paper referenced but that may just be a skill issue on my part.

Is it a statistical model of words, or a search engine? If it searches, it's not an AI, or am I missing something?

letmegooglethat.com?q=openAI+%22...

"make"

Not a problem in their opinion though! If AI reads what AI writes, does citation even matter?

I'm sure there must be other problems with it...?

To be honest, with the ~20$/month version, I have not found a single invented citation. Often they are not the most relevant ones, but they do exist. Maybe I have just been lucky?

You've just been lucky. This is Deep Research, the $20/month OpenAI tool specifically designed for doing citation-based research reports.

I hope it lasts That said, they are seldom useful because as I wrote they rarely are the most relevant ones

We are truly living in an age of blatant disinformation and rampant misinformation.

There are details in here because RAGs are exactly for this but most people just yolo prompt into a vanilla ChatGPT and get surprised when it makes up things.

This is OpenAI's paid RAG system Deep Research.

And it is still making up references?

Yes it still hallucinates. From the article.

If they have a db with all the processed articles it could pretty trivially to double check if it hallucinated (run a query and feed to the ai again, iterate until conclusion) so I am not sure why they don't do this.

1/ Most likely, because it could be used as evidence of their piracy and make them further liable in lawsuits. Currently the NYT is suing them for training of their whole body of work and in one of their latest acts they are asking OpenAI to keep user prompts. Terrible for privacy of the users, but-

2/ Considering the recent two lawsuits that wrapped up against Meta and Anthropic where one of the judges said “the use of purchased material without a license was fair, the piracy wasn’t”. Having a database of evidence of your possible crime is not in their interest.

I would consider this a legitimate and excellent usecase for GenAI. (I bet Elsevier has really devious plans on this) Even if you just collect all content from Arxiv (with permission) that would be a huge win. If you can cut time on background research, it would speed up R&D

I am trying to find some reference if they actually use a RAG (a vector database with all scientific articles processed) but to me it looks like they are doing the research with online queries which is unfeasible to be complete (you won't have enough recall for professional background research)

I think academic RAG eg Scopus AI or academic deep search like Undermind.ai search via api a bounded database with uniqueids so they can check if a generated citation exists. Openai/Gemini Deep research do a "live search" of the Web making its harder to verify fake citations.

Yes. That's the point of my post.

Don't quibble with a gift horse

Even if that gift horse is making pretty much every facet of my job worse?

I was most assuredly not serious about that