too expensive for anyone to do, I do agree with you that there are reasons to be concerned about oligopoly. Especially because the real value of these tools will emerge as they are integrated with things like operating systems and email systems, where oligopoly is already a problem.
This is the key point.. Oligopoly will take ultimate control [in this current world of Oligarchy] of GenAI as features/functions plateau. Particularly because there isn't anything proprietary about these foundation models. Just like email, OS etc
But I'm afraid the real appeal of the bubble theory is the tacit suggestion that "things individuals are currently doing are really unsustainable, and they will be forced to stop."
We're ALREADY seeing instances of where individuals are being forced to stop doing things due to the sustainability issues, so I wouldn't be so quick to categorically reject that could happen, as it already is. Whether this is an isolated incident or a canary in the coal mine is left to be seen.
You can vibe-code just fine with models that can be run locally for free on ~$1500 hardware. You can synthesize images and video similarly. So, uh, I don't see how to square this belief that its so unknown what would stick around or how costly it really is.
You can vibe-code just fine with the existing models that have been trained on large collections of code from places like GitHub. When the webdev community moves to its next language or framework and you don't have a model trained for you on the new code, you're going to hit a wall pretty quickly.
That's going to create some interesting pressures and barriers against movement, as well as amplify existing rich-get-richer tendencies in language development. Even before LLMs, lots of people still code in stuff like C or heck, even Fortran in the scientific community.
Practically though, if you've got example code (and especially if the language isn't some totally alien thing - just different reserved words and boilerplate), updating by training a Lora or even putting documentation into the prompt isn't out of the question even for individuals.
I'd LOVE to see some numbers on what the minimal viable training set was for a programming language. A few weeks back, I used Amazon's Q to navigate their CDK language - which is pretty niche, but Amazon has the benefit of seeing ALL CDK code that's actually deployed.
our languages are all so similar that a lot of this transfers, i think. libraries tend to have ~~pretty similar calling patterns. i have had pretty good luck for "just give it docstrings and it can Do The Thing"
Maybe you could get a scaling law for lines-of-code-between-bugs versus lines-of-example-code, but I think the language details will matter a lot. Using, say, tables of particular memory addresses for functions of a new system-on-chip is different than say 'Python but with {} instead of whitespace'
Extending from one language to another is actually going to be very easy. It's not a back to the drawing board situation at all. arxiv.org/abs/2310.16937
The issue isn't so much going back to the first step, it's avoiding going back far enough that one spends more time verifying LLM code than learning to write it the classical way. Thanks for sharing that paper - I'm going to spend some more time with it. However, in my initial review...
... it only looked at code syntax in pretty small encapsulated chunks (seeded from coding competitions). That's the low-hanging fruit when it comes to writing software. Oftentimes, it's just as important to understand the underlying philosophy of a software platform, and that's where things tend...
... to change when a new language or framework is released. To be sure, you want to maintain enough continuity with the old world so some human knowledge transfer can happen (no need to reinvent conditional statements), but translating between higher-level constructs like multiprocessing is where...
I'm really waiting for a general theory of conceptual attachment from these things. I suspect there are pretty simple patterns for when a model keeps new things separate, versus when it abstracts them in a way that allows linkage to what it already can do.
Also, we're getting dozens of foundation models a month right now. There are paths that lead from that world to one where we're darning our used clothing, saving bacon drippings, and only training one model every other year. But those paths involve thermonuclear war or smallpox. +
I mean you should save your bacon drippings anyway, that's some of the best cooking lipid available
You could reduce investment in this industry by two orders of magnitude — assassinate 200 leading researchers to prevent any further technical advances — and we would really *still* be quite capable of updating models to reflect new languages and recent events.
I firmly believe "brilliant researcher" is more a social niche than some sort of phenotype
If all AI companies shut down their APIs and closed their doors, maybe I couldn't have an LLM analyze an entire book series for me, but I could still certainly have one analyze 10 chapters. Maybe I couldn't let one work on a codebase but I could still use it to vectorize for loops into PyTorch, etc.