It makes me think sometimes that folks enamored with this technology just fundamentally don't understand what intelligence is? Like, to me intelligence is the screwing up, screwing around, getting bored, getting excited, sitting with /something/ for years.
And there's no way to shortcut that because the long path /is/ the intelligence. Having spent the time gives one the flexibility and history to do the cool shit, like make new connections and build meaningful ideas. AND actually integrate that shit into your life.
I think a lot about how the core lesson of my artistic journey is probably "make what your hands want you to make, not what your head thinks you want to make". I was only able to actually learn that by fucking it up for decades, and now seeing the payoff for a few years.
Even /if/ an LLM could spit out that exact lesson to me, it /still/ wouldn't be nearly equivalent to the sum total of my life experiences that led me to that idea myself. All this to say, the LLM cannot have "PHD level intelligence" because it didn't fucking wrestle with something for years.
The wrestling /is/ the intelligence. Not the facts or figures - which the LLM can't even reliably get right anyway.