On the second task, once again a core task is using expertise to gauge the reliability of information, a task LLMs are incapable of. But equally, I've seen no signs that LLMs are capable of communicating that information in a way that is engaging or informative. 4/
ChatGPT rolled out in November of 2022 and even without the 'hallucination' problem, it still produces cliche, lame prose that no one actually wants to read. You could get the thing to write 10,000 books in a day, and people have and somehow not one of those monkey-typewriter novels is good. 5/
Notice that this chart rates "authors" at almost as replaceable as historians... (but, uh, lol, no? absolutely not? not even remotely? That's something said only by a person who has never read a book in their life...)
there are 5 million English language books in the library of Congress. I worked out that you could write about 5m books of exactly 200 pages where every scentence was is ... The missing words matter
Meanwhile, of course, something we expect from historians working as history educators is to avoid egregious error. Everyone makes mistakes and our understanding of the past changes, but if a historian confabulated data out of nothing, we'd fire them. Ask Michael Bellesiles. Or Naomi Wolf. 6/
So what you have here are folks who do not understand what historians do assuming that 'doing history' is summarizing textbooks rather than realizing that doing history is making the critical and analytical decisions that shape textbooks. Embarrassing ignorance, but also very bad for the field./end
I think a lot of people think there's nothing more to being a historian than knowing lots of dates and facts and names, which is like thinking being a mathematician involves being really good at arithmetic.
Even I, as a person who studied history extensively and is bad at math, knows that math is more than just "being able to solve math problems." Someone has to come up with new problems to solve! #smdh
If you want to know more about what historians actually do, I've covered that in two posts, one on what the job of a research historian actually is (acoup.blog/2025/03/07/c...) and another on how that research moves through other kinds of historians to reach the public (acoup.blog/2020/07/09/c...)
📌
The other bad news -- "embalmer" rated high on their list of jobs that aren't endangered 😱
The links on your blog are a very useful and salient reminder of the work historians do. Thank you for taking the time to share it so eloquently.
OpenAI is focussing ChatGPT on corporate and health solutions to make $$$ It is not intended to be creative - and indeed what it outputs cannot be copyrighted and is therefore not monetisable. So don't fret - it is not coming after artists
This is actually a really good point that on top of it being completely useless to begin with, it can't even write well enough to grab one's attention.
You know it only just hit me what a damning indictment of the tech it is that there hasn't been a breakout AI Book yet. Like, if the tech were even half as adequate as the moneymen are desperate for us to believe, surely it could at least put out some sellable YA Pulp or something, righht?
Yeah, this only just hit me when someone said the same thing on here that I saw like, a few weeks ago, but I can't find the original to give proper credit. It's not my original idea. But it is shocking, when you think about it.
Like, lol, I skimmed through a kindle version of one of the old Stackpole X-Wing books a while back and believe me, you DON'T need to be a very good writer to put out something teens are willing to read fairly consistently - but GPT still can't do it.
I read a LOT of those old Stackpole X-Wing books when they were not old, so believe me, I know.
It's possible they'll get sellable YA pulp by throwing more computing resources at it, but for the models people actually use in real life producing a book-length output is literally impossible.
Context windows are now long enough, but the models still aren't able to handle the task well. A book might be 100k words, which translates to maybe 200k tokens, and flagship models are now up to 1M tokens. (I expect they'll get there, though, especially with scaffolding and outlining)
Like, models have a context window that governs how long their output can be. For GPT-4 and co that is not a book long. And the context window is the sum of the input+output, so you can't make half a book and feed the contents in to get the second half of the book.
This has implications for many of the non-book-creation uses AI is being put to; it cannot "remember" a book-length conversation.