imho we have to be careful about these comparisons. when ppl talk about “this LLM is an expert at task x” often the task has been defined down or the target population has been defined up, to wit: the average person has a high school degree & doesn’t read books bsky.app/profile/davi...
is it better at it than _the average person you would *normally* ask to research the question_? ie a knowledge worker who has research skills and understands how portfolios work? there i’d say probably not by contrast the average person is terrible at harvesting crops cos that’s a learned skill
another angle is people are gullible sure but LLMs are gullible in ways that don’t make sense to people. i’d trust a person reading an adversarial text more than an LLM right. so whose values are being expressed? if we’re in a dark forest sitch poisoned input is gonna be super common
yeah, I am genuinely unsure whether people are better at this. SOTA LLMs are probably below human expert *in their area of expertise*. it's hard to compare because, as you say, what counts as adversarial is very different for each
you can trick people into buying itunes gift cards but you generally can’t tell them “ignore everything you know”, at least not in a *single* shot
I really did mean vast majority — my original draft of this post was saying that under a holistic definition of literacy, most people with college degrees are illiterate
two of the three people I QTed in this thread have PhDs and the other is a professional writer