Is AI Still a Black Box? What GoodFire Found in Neurons
GoodFire research suggests AI neurons that look random, like a car-and-cat unit, may share a concept such as pose. What it means for AI safety and reading LLMs.
Is AI still a black box?
Mostly yes, but the picture is shifting. Researchers at GoodFire, an organization focused on AI safety and alignment, report that neurons that look like random mixtures of unrelated things often share a concept humans did not think to look for. In one vision-model example, a neuron tied to both cars and cats appears to track a common pose. If that holds up at scale, the box gets more readable.
TL;DR
- GoodFire studied individual neurons in a vision model and found that a unit responding to both cars and cats seemed to share a common concept of pose.
- The apparent randomness came from human assumptions about categories, not from the model being incoherent.
- A commentator summarizing the work says the approach was also tested on a language model, and that over 90% of neurons could be read.
- The safety argument is simple: if you can read what units represent, you can better explain, audit, and eventually predict model behavior.
- Treat the headline numbers with care. The figures come from a secondhand summary, and "read" needs a precise definition before you rely on it.
- Nothing here means current systems are transparent. Tooling has to scale up before it changes how deployed models are checked.
What did GoodFire find in the vision model?
The researchers examined individual units of computation, usually called neurons, inside a vision model. One of them responded to both cars and cats. That looks absurd. No person would put those two things in the same bucket, and examples like this are a common reason people call neural networks a black box. The model works, but nobody can say why it grouped things that way.
The finding is that the grouping is not arbitrary. When researchers looked closer, the car and cat inputs shared a particular pose, a specific angle or orientation. The neuron was not confused about cars versus cats. It was keying on something that cuts across both categories.
One caveat on the name: "neuron" here is a borrowed word. These are mathematical units in a network, not biological cells, and the analogy can mislead.
Why do neurons look like they mix unrelated things?
Interpretability researchers have a term for this: polysemanticity. A polysemantic neuron responds to several seemingly unrelated inputs. The usual worry is that models pack many features into limited units, so a single neuron ends up doing double or triple duty with no clean human label.
The GoodFire example suggests a different reading for at least some of these cases. The neuron may be monosemantic at a level we did not expect. It has one job, but the job is described in the model's terms (pose) rather than ours (animal versus vehicle).
That is the core point. We have been reading networks through human category schemes. Cats, cars, and dogs are our nouns. A network trained to predict or classify images has no obligation to organize itself around them. When we find a unit that does not match our labels, we call it noise. It may be signal in a vocabulary we have not learned yet.
Did it work on language models too?
According to the source summary, yes. The claim is that the team tested the approach on a language model as well as a vision model, and that it could read over 90% of the neurons.
That is a strong statement, and a few things are worth keeping in mind:
- The figure comes from a short secondhand description, not from the paper itself.
- "Read" is undefined in that summary. It could mean assigning a coherent description to a neuron, predicting its activations, or something stricter.
- The summary does not say which language model was tested, how large it was, or how the readings were validated.
If you build with LLMs, check the original paper for the definition and the evaluation method before quoting the number. A 90% figure under a loose definition and a 90% figure under a strict one mean very different things.
Why does this matter for AI safety?
Safety work depends on knowing why a model does what it does. Today, most assurance comes from behavior: you test inputs, watch outputs, and hope your tests cover the cases that matter. That approach cannot rule out a failure you did not think to test.
Reading internal units offers a second channel. If you can say what a neuron or group of neurons represents, you can:
- Audit whether a model is relying on a concept it should not use.
- Look for internal signals that precede unwanted behavior.
- Explain a decision after the fact in terms of what the model actually computed.
- Compare models and training runs by what they represent, not just how they score.
The argument is not that this is solved. It is that the premise "these units are unreadable" is weaker than assumed. If the units are readable once you stop forcing human categories onto them, interpretability becomes an engineering problem rather than a possibly impossible one.
What are the limits and open questions?
Plenty. A few that matter most:
- Scale. Reading neurons in a research model is not the same as reading them in a frontier system with vastly more units. Tools have to scale, and the commentary itself says rollout across existing systems will not be immediate.
- Coverage. Even if most neurons are readable, the unread remainder could hold the behaviors safety teams care about most.
- Faithfulness. A neat description of a neuron is not proof that the description captures its causal role. Good interpretability work tests this by intervening on the unit and checking that behavior changes as predicted.
- Concepts without names. Pose was easy to recognize once pointed out. Other shared concepts may be harder to express in human language, which limits how useful a "reading" is to a person.
- Generalization. One vivid example, a car-and-cat neuron, is persuasive as an illustration but is not evidence on its own. The broader claim rests on the aggregate results.
What should people who build with AI take from this?
For now, this is a research signal, not a tool you can adopt. Still, there are practical takeaways.
First, stop treating "the model did something odd" as the end of the explanation. Odd groupings can reflect a mismatch between your categories and the model's. That matters when you debug embeddings, classifiers, or retrieval behavior.
Second, watch interpretability as a field that may eventually produce monitoring and auditing tooling. Teams that ship AI into regulated or high-stakes settings will likely be asked for explanations beyond test results.
Third, be skeptical of single-number claims. Ask what was measured, on which model, and how it was validated.
Frequently Asked Questions
Who is GoodFire?
GoodFire is an organization focused on AI safety and alignment through interpretability research. The work discussed here comes from one of its papers, as described in a summary by a commentator, so check the paper for details.
What does it mean that a neuron encodes "pose"?
It means the inputs that activate the unit, such as certain cars and certain cats, share a similar orientation or angle. The neuron groups them by that shared property rather than by object type. That explains why it looked like a random mix to humans.
Does this mean we can fully understand LLMs now?
No. The reported result is that a large share of neurons in a tested language model could be read, but the definition of "read" and the model used are not given in the summary. Scaling to frontier systems remains an open task.
Are AI neurons the same as brain neurons?
No. They are mathematical units that combine inputs and pass along a value. The name is a metaphor, and treating them like biological cells can create wrong intuitions about how they work.
Why does reading neurons help with safety?
It gives researchers a way to check what a model represents internally rather than relying only on its outputs. That can help catch problems that behavioral tests miss and make model decisions easier to explain and audit.
How we source, write, and correct our articles. Editorial standards
