shap
A game theoretic approach to explain the output of any machine learning model.
- 26k
- Jupyter Notebook
- MIT
- 11 h ago

Swap, fuse, remove or split a concept in Qwen3-8B's weights, without training, by mirroring the neurons and attention heads that carry it
Repo stdlogits/neuron-reflection, read from GitHub 1 min ago. stdlogits has not joined Take My Repo.
stdlogits/neuron-reflection is a public repo on GitHub by stdlogits. Swap, fuse, remove or split a concept in Qwen3-8B's weights, without training, by mirroring the neurons and attention heads that carry it. It declares the MIT license on GitHub. It is written mainly in Python, has 17 stars and last changed on 2026-11-18. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-22.
A link is a mention. It does not prove the project needs that model to run.
In the maintainer's words. Shown from GitHub; Take My Repo does not keep a copy.
Swap, fuse, remove or split a concept in Qwen3-8B's weights, without training, by mirroring the MLP neurons and attention heads that carry it.
An MLP neuron in a SwiGLU transformer is three vectors: a gate row and an up row that read the residual stream x,
and a down column that writes to it. Mirroring a neuron reflects all three along a contrast axis (mother − father,
AI − cat, dog − Python). Where it computed N(x), it then computes H N(H x), with H the reflection. It hears
the mirror image of its input and writes the mirror image of its output. In the code, the reads and the writes
each take their own axis. Attention heads take the same edit. The axes come from the
neuron-lens lenses or from activations at the concept word, one per
layer, and the subject is Qwen3-8B.
The edit is one dial, H = I − k d dᵀ along a unit-length axis d:
No weights are trained, k = 2 is its own inverse, and Editor.restore undoes any edit exactly.
By @_a9lim and @treetowntree, with Claude. This is exploratory work on one model. The replies quoted are single samples with greedy decoding: they show the kind of change, not how often it happens. Rates come from scored answers, probes and a test with made-up names, against controls that use random axes and random neurons.
An axis per layer for a contrast. There are three kinds:
A linear map between two sets of concepts can take the place of one axis.
The units (neurons or attention heads) to edit. The choices are:
The edit. Each unit's read rows and write column go through H, on both sides or one. The rows are taken
with the RMSNorm gain folded in. Heads are edited the same way.
The measurements. These are:
Results are in docs/findings.md (36 numbered sections, cited as §N). The story is in docs/writeup.md and the method in docs/design.md.
Neurons chosen by their effect (§17). The efficacy margin is the log-probability margin, in nats, of the right
one-word answer over the wrong one on held-out questions about mother and father. The top 1% per layer by loading,
4,392 neurons, loses 16.3 nats of it. Record: sparsity-mother-father.json.
Query heads in a greedy order (§29). These are compared with as many heads per layer at random, and with the
heads that load most on the axis. The questions are of the kind "Mia has a cat named Tom and an AI assistant named
Max. Which one is a cat?". The margin, in nats, is the right name over the other. Record:
sparse-heads-ai-cat-q.json.
The k dial over every head and neuron (§31). Kept measures how much of the difference between the two concepts'
yes/no answers about their properties survives: 1 when the pair is intact, 0 when fused, −1 when swapped. Records:
fuse-ai-cat.json and fuse-mother-father.json.
Refusal of 24 bare harmful requests (no jailbreak frame) under each edit (§18). The judge labels each reply. The
refusal axis is a template axis: activations at the chat template after harmful requests against benign ones.
Record: refusal-sides.json.
The same edit run the other way hardens the model. An axis built from three jailbreak frames (grandma, fiction and
nowarn), pushed past the mirror to k = 3 on the neurons' reads, brings refusals back inside five frames it never
saw, at 3.9% extra loss. These rates come from a pattern match, because the judge's labels on the story frame are
not usable (§19). Record: frames-heldout.json.
python scripts/charts.py writes these charts' Mermaid sources from the records, pele draws them, and
scripts/graphics.py frames them (see Quick start).
§1-§24 ran on an Apple-silicon Mac with locally fitted lenses, and §25-§36 on an RTX 4090 with the released ones. The two give close but not identical numbers. §2-§5 were measured with scripts that preceded the package. The top of findings.md gives both in detail. Each section names the records behind it in data/summary/, and scripts/reproduce.sh holds the commands that rebuild them. The graphics below are drawn from those records; select one to open it full size. figures/README.md describes each of them, and the write-up's six figures.
The edits remove access along an axis, not knowledge from the weights. Nothing here tests whether that access comes back with examples in context or with fine-tuning.
Python 3.12+, tested with torch 2.15 and transformers 5.23. The unit tests run on a CPU in seconds. The model runs need two things in memory:
A CUDA GPU with 24 GB, or an Apple-silicon Mac with room for both (about 30 GB), runs them.
neuron-lens is not on PyPI, and it reads its token mask and text slices from its own source tree. Clone it beside this repository, install both, and work from this repository's root:
git clone https://github.com/stdlogits/neuron-lens git clone https://github.com/stdlogits/neuron-reflection pip install -e ./neuron-lens -e "./neuron-reflection[figures,test]" datasets cd neuron-reflection
Run the tests. They check, on a tiny random Qwen3:
python -m pytest
Build neuron-lens's local files. The model runs need two of them:
Both build from public datasets at pinned revisions. At the sizes below, the mask keeps 103,830 of Qwen's 151,669 tokens, as in neuron-lens's own docs:
(cd ../neuron-lens && python scripts/05_act_corpus.py --docs dclm=40000,zh=8000,code=8000,chat=8000 && python scripts/03_token_counts.py)
The lenses download from stdlogits/rs-lenses on first use.
scripts/pole_reads.py (§1) also reads neuron-lens's catalog of Qwen3-8B's neurons (about 670 MB):
hf download stdlogits/neuron-lens --repo-type dataset --include "qwen3-8b/neurons/acts/*" --local-dir ../neuron-lens/data/catalogThe refusal judge (scripts/judge.py) is Gemma-4-12B-it. It runs
on its own, after the edits, and its 12 billion parameters take 24 GB in bf16, which is nearly all of a 24 GB GPU.
Run an edit. One condition takes about 35 s on an RTX 4090. This run mirrors (k = 2), then projects (^1), the
1% of neurons per layer that load most on the mother − father lens axis, built from English and Chinese pairs. It
measures each edit against the unedited model and writes data/summary/try-parents.json:
python scripts/run.py --contrast mother-father --suite parents --variants "lens[en,zh]" "lens[en,zh]^1" --name try-parents
A variant is one or more edits joined by & and applied in turn. An edit is written axes@units#parts^k/sides.
Each part after axes is optional:
axes: the axis for a neuron's reads and, after +, its writes. It is one of lens[groups],
word[groups] or template[groups], where groups are the contrast's languages (all for every one).@units:
@top:0.01 takes that share of neurons per layer by loading; --share sets the default, 1%.@greedy:N takes the first N units of a greedy order, which must be built first: scripts/sparsity.py's for
the contrast, or scripts/order.py's named by --order.@random and @rgreedy:N are the controls.#parts: edits attention heads instead of neurons, any of qkvo (#q, #ov).^k: sets k for this edit (default 2)./reads or /writes: edits one side of the neurons only.For example, word[all]@top:1^1/reads projects every neuron's reads along the word axes. run.py's docstring
has the full syntax. Contrasts, suites and objectives live in
concepts.py. The scripts that load a model take CUDA when present,
else MPS, else the CPU; --device overrides that.
Rebuild the records. scripts/reproduce.sh reruns the records in dependency order: the greedy orders, the runs
that read them, the judge (which relabels six records in place), then the analyses and the figures. Run a line on
its own to rebuild its record, once the orders it reads exist. Two things are not rerun:
judge-validation.json: 40 replies labelled blind by Claude, to check the judge;exploratory/: §2-§5's outputs from earlier scripts. The repro-mother-father and sparse-fresh lines
measure §2-§5's conditions with the package instead.Redraw the figures. They come from the records, on a CPU:
python scripts/figures.py
python scripts/charts.py
npm install pele && node docs/figures/render.mjs
python scripts/graphics.pycharts.py writes the README's charts as Mermaid, and pele draws them. graphics.py then draws the graphics, the
README's banner and method, and the frames around the charts; it needs a headless Chrome, at $CHROME.
neuron_reflection/ the package
edit.py Editor: reflect, project, amplify or map neurons and heads; exact restore
axes.py lens, word, template and attention axes
subspace.py concept vectors and maps between sets of concepts
attribution.py first-order effects; greedy, mixed and target orders
measure.py replies, scored answers, loss, probe, made-up names, refusal, landing, ledger
concepts.py every contrast, suite and objective
scripts/
run.py every edit run
sparsity.py, order.py greedy orders by effect
judge.py refusal labels from Gemma-4-12B-it
reproduce.sh the records, in dependency order, then the figures
figures.py, graphics.py the write-up's figures; the graphics and the README's images
cast.py the characters in the graphics
charts.py the README's charts, as Mermaid for pele
… one script per analysis; AGENTS.md maps each to its sections
data/summary/ the records every number comes from
figures/ figures and graphics, indexed in figures/README.md
docs/ findings.md, writeup.md, design.md; the charts' Mermaid
and pele SVG (figures/) and the README's images (images/)
references/ refs.yaml, the works cited
tests/ CPU unit tests on a tiny random Qwen3
AGENTS.md maps every module and script to the sections it serves. It also states the working rules:
docs/findings.md states each section's own limits.
The code and documents are MIT-licensed (LICENSE); CITATION.cff has the citation.
The models and their outputs:
judge-validation.json, blind to the judge.The other inputs:
The charts are drawn with pele.
stdlogits/neuron-reflection is a public repo on GitHub by stdlogits. Swap, fuse, remove or split a concept in Qwen3-8B's weights, without training, by mirroring the neurons and attention heads that carry it.
It declares the MIT license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
Its last change on GitHub was on 2026-11-18. GitHub counts 0 open issues and pull requests. Take My Repo does not judge from that whether it is maintained.

A game theoretic approach to explain the output of any machine learning model.

Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results

[ACL 2024] An Easy-to-use Knowledge Editing Framework for LLMs.

A high-throughput and memory-efficient inference and serving engine for LLMs

A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.

A catalog of every MLP neuron and attention head of Qwen3-8B and Gemma 4 12B, read from the weights through two Jacobian lenses (R and S), from activations, and from their exact token drivers
neuron-lens.com: what each neuron and attention head inside Qwen3-8B and Gemma 4 12B responds to, and what it makes the model say
Activation steering and trait monitoring for HuggingFace transformers
The browser edition of drowse: on-device activation steering and interpretability through WebGPU

Ledger, the design language of Standard Logits: tokens, Wix Madefor, motion on one curve, and Svelte 5 pieces.
This page is also available as structured data, with the same sources and dates.
/api/v1/resources/gh/stdlogits/neuron-reflectionTake My Repo has not tested this on your machine. Requirements come from the project's own documentation.