Note: LLM internals are emphatically not something I know much about. The following should be treated as a set of ad hoc reflections based on some cursory reading on Mech Interp research and a lot of enthusiasm for a specific set of philosophical problems.
Mechanistic interpretability is a branch of AI research that attempts to identify logical structures within a neural network (e.g. an LLM) by observing how data is transformed as it passes through the system, and how specific sorts of inputs interact with various portions of the system as they’re processed. The goal of this research is to identify a set of logical structures that perform the same transformations as the network itself, characterizing operative features and circuits within the network and (maybe, potentially, eventually) expressing these as an independent formal system. This idea is really interesting on a bunch of levels, both philosophical and practical, so I’d like to talk about it a bit.
First off, on a practical level, I think it’s a great idea to try and reduce what we might call the “organically” developed structures present in these models or neural networks into human-intelligible subroutines expressible in language. If we figure out how to do it, and how to do it at the scale of the big frontier models, the benefits could obviously be stupendous. It would make conceivable the possibility of radically compressing the big models, being able to tune their behavior by tinkering with the internal routines, splitting them off into different independently functional cognitive sub-models, and so on. It could also democratize access and control of AI (I’m fantasizing here), reducing the computational costs of running these models by several orders of magnitude. But who knows if that’s achievable.
On a philosophical level, the idea is interesting too. The question of whether it’s possible to actually succeed at a mechanistic interpretation of a model is related to a question that was popular in mid-century anglo-analytic philosophy about whether there are mutually untranslatable languages (see, e.g. Donald Davidson, who attacks the idea of mutual untranslatability in “On the very idea of a conceptual scheme”, or W. V. O. Quine’s work in Word and Object).
To make the connection explicit, let’s frame the problem this way: Suppose you have some kind of structured space that has a bunch of information in it (for our purposes this “space” is Claude, Gemini, etc.). Can you accurately describe what’s going on in that space by translating the structures present in it into a different linguistic domain?
In formal languages this is provably possible. We can compile our java into C or whatever. Any program that’s expressible in one Turing complete language is expressible in all the others, etc.
In natural language, various factors make the “accuracy” of translation a hard problem. For one thing, word substitutions are never going to be 1:1, if not on the basis of inexact matches between their usage, then because of their phonemic differences, their contrastive and connotative relationships with all the other words in the language, etc. Additionally, natural languages are not just “one thing”. They include a massive number of local and contextual variants that flavor expression, and evolve constantly, irregularly, from place to place and over time.
Natural languages are shaped not just by collective use, but by social shifts, patterns of misuse, cross-cultural interactions, and so on. To imagine a language is, as Wittgenstein says, “to imagine a form of life”. And, perhaps maddeningly (if hopes to reduce natural language to the “ideal” of formal language) all of this complexity exists simultaneously, with a high degree of ambiguity, and any particular corpus of text or speech in a given language is at best a sample identifying points within an arbitrarily complex manifold.1
All of this is to say that the business of achieving a perfectly accurate translation is hard, first of all, because the domains between which one is attempting a translation are so riddled with ambiguity and complexity that a truly perfect translation would require both the origin and target language to be identical.
Now, you might find this perspective somewhat odd because, after all, translations between natural languages exist and we make them all the time. Moreover, languages are not free-floating networks of signs and linguistic constructions. Language relates to the world, and our experience of the world is reflected in language. If I meet a German speaker and want to know the word for “ball”, I can point to a ball and ask the word for it. Now I have a translation of that word. Was that so hard?
But here the problem of translation goes one level deeper. We can think of languages themselves as means by which we “translate” the world of our experience into words. But here the “interpretability” problem starts to show up. When I translate the world of my experience into words, I am always doing so by means of the concepts and linguistic behaviors available to me. The word concept (conceptus) itself betrays this, since a concept is the means by which we choose which things to hold together in thought as one. The conceptual interpretation of experience in language happens by means of a set of habitual (usually almost invisible) abstractions. When we describe, we do so by choosing which aspects of the undifferentiated manifold of experience to regard as “objects”, and under what characteristics to name or describe them. This problem of (usually unconscious, inescapably prejudicial, heavily situational) interpreting things is, as Hans-Georg Gadamer said “the mode of the whole human experience of the world” (see “The Universality of the Hermeneutical Problem” in his Philosophical Hermeneutics).
As an aside, one thing I find really fascinating about this is the extent to which, starting from different branches of the western philosophical tradition, with very different academic priorities, research interests, and methodologies, both continental European and Anglo-American philosophers have largely coalesced around the same set of insights on the topics of language, meaning, and interpretation.
But let’s get back to our original problem, which I’m going to state as follows: “Given a space in which a huge amount of information is encoded in ways we do not understand, how can we decompose the internals of that space into intelligible logical forms?” Clearly this is possible, because if it were basically impossible we would have no capacity for understanding the world at large. So the question is: once we have achieved such a decomposition, how do we know if it’s the right one? How do we know if our concept of the model as expressed in intelligible formalisms is actually equivalent to the information encoded in the model itself?
How would you know if you’d achieved a sufficiently “accurate” translation from one natural language to another? How would you know whether you’ve achieved a scientific model that is equivalent to “the world at large”? Well, from a perspective that aspires to achieve the level of rigor you get with formal languages, it’s impossible. You can’t reduce a stochastically generated model that’s rife with chaotic variations to a set of tidy formalisms. But if you swap out that aspiration for something more pragmatic, then the question becomes: can I substitute a set of expressions or linguistic behaviors in the target language that will yield the same sorts of results in the same sorts of situations as in the source material, at least in the respects that I care about? Obviously this is achievable. But it then raises a different question: What is it that you care about?
If we consider, instead of a source or target language, a big neural network (Claude, etc.) that’s been stochastically trained off a bunch of problems, scenarios, a big corpus of text, etc., it’s clear that we’ve already decided “what we care about” when we made the model. “What we care about” is its behavioral fidelity to the training we gave it. Virtually the same thing applies to the people doing work to translate the model’s undifferentiated mess of neural networks into specific intelligible subroutines. In other words, the mechanistic interpretation of a model as a composition of formal subroutines is simply a way of reducing the information space by deciding what you care about, much like how, when I say “the hammer is too heavy”, I’m honing in on a bunch of characteristics of the stuff present before me to decide what matters in context, ignoring all the other possible characteristics or aspects of the situation I could have chosen to describe instead.
I think the big mistake from all this would be to assume that if you succeed at decomposing a model into a mechanistic interpretation, you’ve somehow nailed down the “true meaning” or “true structure” of the model’s internals. Because in reality your selection criteria for the decomposition determined what sort of decomposition you got out. Which is fine, because ultimately we want a model that behaves a certain way—our selection criteria for the interpretation are the same as the selection criteria we used when training the model itself. We’ve just added another step on the transpilation pipeline. “First encode all of this data in the form of a massive transformer with a bunch of weights. Then smush that transformer down into something with fewer dimensions. Then render that simplified thing in a formal language.” Sounds great.
But I think it’s important to notice the big difference between two claims you could make, either about the model you’re working with, or about the formalization of the model into a set of mechanical abstractions. You can claim, for example, that “this thing is artificially intelligent and will perform tasks within this domain to a high degree of fidelity based on our acceptance criteria”. Or on the other hand you could claim that “this thing encodes within itself the sum of all available human knowledge and speech, and its behavior is on the whole a compressed representation of what people at large tend to express in language about various topics”. The latter is not strictly a “useful” object, but it has the opportunity to be interesting and even surprising in ways that the former is much less likely to be.
Returning to the concept of translating between languages, if you were to develop an automated translation service and then decompose it into a set of rigid formalisms that functionally capture the behavior of the translator, what would you lose along the way? The flavor, the subtle connections, the ambiguities. There is a fuzziness that’s possible in “organic”, extremely complex highly dimensional information spaces that would presumably get lost when you reduce them down to a formal mechanism. So the question is: how effective is your decomposition algorithm at preserving what’s useful in the behavior of your model, and how much of the unintended fuzzy behavior you captured when training the model in the first place do you really want to preserve—how much can you preserve, after the edges have all been sharpened and you’re dealing with something much more like a clock than a brain?
I think the same thing is likely to apply to the decomposition of LLMs into formal features and modules, which is not a knock on this work at all, but another interesting point to consider, and it reduces back to the really befuddling question about all LLMs and related entities: What exactly is this thing that we’re building, and what is it for?
One of the joys of the present moment is that philosophical reflection and the frontiers of computer science have suddenly drawn close to each other, so that one can make puns on “manifold” with reference to both its Kantian sense (as the undifferentiated substrate of pre-cognitive sense experience) and its mathematical sense as a locally euclidean topological space of arbitrary dimensions.