But they are extremely intelligent, in that they can quickly sort and understand patterns in information. So you begin feeding them everything anyone has ever written or said about anything ever. And then you ask them a question. "What's the best way to cook chicken?"
Our genius-in-a-box knows everything anyone has ever said or written about chicken. Recipes, reviews, etc. But they've never eaten chicken themselves. Or seen one, even. What can they *possibly do* from inside the box, other than tell you what *everyone else* has already said about chicken?
It doesn't matter how smart the genius-in-a-box is: it can only be as smart as everything you feed into it. That's literally all it has to work with. Nothing comes out of the black box that wasn't put into it.
Can it make interesting connections with information that we hadn't noticed before? *For sure.* Patterns are its bread and butter. But it is still bound to what we give it. It can't do anything else. If you want actual progress, a human being has to go out there and make more data.
This is all to say 1. I think AI can be an AMAZING learning tool. Infinitely patient, infinitely capable of repackaging the same idea in different examples. 2. I think AI is great for surfacing solutions people have already found. (A fancy web search basically)
But I think if you want to solve new problems with original, useful decisions, the genius in the box cannot help you. Because the genius is stuck in the box. They can't know anything we didn't already know.