The underlying reason was that groups of people who weren't mostly High-IQ 10x Code Ninjas could make useful stuff with a few months' practice, reading and learning. That was not the scalable disruptive innovation vibe needed to dominate the world with the web. CS guys knew what the web should be:
Functional; it should be nothing but functions, not documents, not styles, not semantics, not ecosystems of open standards. No one liked it, especially if they foresaw the complexity nightmare it would (and did) create. But we 'had' to learn it, so we'd have to hire for it, so we had to build in it.
Let’s not forget the “programming vs coding” discourse.
Omg I wish I could
I miss the semantic web