ooooh, I had down/up mixed. That fits my (post hoc) intuition for hierarchical models in the sense that each layer is passing a code, if the code is changed a lot at the first telephone pass the ultimate output will be wrong-er
Replies
we do a lot of things to try and fix this -- layer norms, batch norms (maybe; they work but we don't know why), skip connections -- but once the model starts developing polysemanticity, everything becomes entangled in a way which is very difficult to unstick.
jul 22, 2025, 8:28 pm • 1 0 • view
Is this still true in character rather than token models? Trying to figure out if it's more of a structural thing (always lowest layer least plastic) or if there's an early-semantic "mess" where layers must start accepting the previous layers' framing of the world
jul 22, 2025, 8:31 pm • 1 0 • view
if this helps: character and token models should look much the same, other than the embedding layer, iff they are well trained and you are lookin at the activations after at least one attention pass
jul 22, 2025, 8:38 pm • 3 0 • view
like. the character embeddings will be dogshit simple, but to perform well the internal activations will take on the same shape after you've run attention to mix in the information you specifically want
jul 22, 2025, 8:40 pm • 1 0 • view
this is true even of things which don't deal with words at all, although i have a strong intuition that models which deal with hierarchical signal decomposition (i.e., diffusion models) probably have less of a hard time with it than models where the signal is extremely white. (images are pink)
jul 22, 2025, 8:34 pm • 2 0 • view
this is neat, thanks!!
jul 22, 2025, 8:39 pm • 1 0 • view
We fundamentally don't know anything about what the (necessary) "fixed-points" of concepts are. There must be some latent structure the concept is being "passed" around in a way that is de-constructable, but it just happens that location is not it and no competitors are clear frontrunners
jul 22, 2025, 8:28 pm • 2 0 • view
yeah that is one of my biggest problems with neuroscience: addressability in a system which seems substantially nonlocal
jul 22, 2025, 8:32 pm • 3 0 • view
Some competitors/supplements to spatial just in case you want to read more are (in any combination): - Topological (spatial or functional, I think spatial topological models are silly tho) - Frequency/power spectrum-ish stuff - Functional connectivity kinds of things - Whatever Friston is on about
jul 22, 2025, 8:36 pm • 2 0 • view
yeah, i was actually just thinking through how i'd build addressability via cumulative noise, and, like, "well, if the noise spectrum were 1/f, then so long as the dominant frequencies in the characteristic noise were fixed points (and we ran a high-pass filter) then plasticity wouldn't be that bad.
jul 22, 2025, 8:39 pm • 2 0 • view
I'm pretty sure fixed-ish oscillation frequencies (alpha, beta, whatnot) do (in theory) serve the purpose of addressability in the brain in the frequency band/functional connectivity literature, but I don't have a great source
jul 22, 2025, 8:42 pm • 1 0 • view
But that's on the order of like, "OK, we're doing forward messages now" "OK backward messages time" back and forth a few times a second, not like "this came from x=20, y=-3 in the visual field"
jul 22, 2025, 8:45 pm • 1 0 • view
For more understood neural (not well understood) solutions to addressability, hippocampal conceptual-spatial maps are neat too (www.jneurosci.org/content/40/3...)
jul 22, 2025, 8:37 pm • 2 0 • view