Sparse Weight Decomposition
for Efficient Circuit Extraction

Abstract

Dense pretrained transformers do not naturally expose interpretable units for circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations or training sparse models, incurring substantial additional computation while potentially introducing a fidelity gap between the representation being analyzed and the original pretrained model. We propose Sparse Weight Decomposition (SWD), which reparameterizes pretrained linear projections by factorizing each weight matrix into two sparse factors whose shared intermediate coordinates serve as individually addressable circuit units. Without training a separate replacement network, this parametric representation supports the same scoring, selection, and ablation circuit extraction workflow used for methods that learn sparse features. Across single-matrix replacements, SWD matches the held-out fidelity achieved by Transcoder and other strong baselines while using less than 1% of the data that those baselines use to train their replacements. For matched replacement fidelity, SWD reaches the same circuit sufficiency and necessity targets with fewer active read/write edges and selected units across tasks on GPT-2, Qwen2.5, and Qwen3.5-27B. We further show that SWD remains effective for full-model replacement of all attention and MLP weight matrices after fine-tuning the nonzero factor values. Finally, SWD also features a zero-data variant, allowing broader use of mechanistic interpretability analysis (e.g., per-step analysis).

1 Introduction

Circuit analysis seeks a small set of internal units that is sufficient to reproduce a behavior and necessary in the sense that ablating it predictably degrades that behavior (Bhaskar et al., 2024; Conmy et al., 2023; Syed et al., 2023). Pretrained transformers, however, do not expose such units: they implement task behavior through dense linear projections, where individual neurons can be polysemantic and task-relevant effects are distributed across many activations and parameters (Cunningham et al., 2023; Elhage et al., 2022). This leaves a gap between the sparse causal explanations we seek and the dense computational substrate we aim to analyze.

Existing approaches obtain interpretable units for circuit analysis in several ways. Sparse autoencoders and Transcoders learn feature dictionaries or replacement modules in activation space; parameter-decomposition methods learn components of the model’s weights; and sparse pretraining builds sparse connectivity into the model during pretraining (Braun et al., 2025; Bushnaq et al., 2025, 2026; Cunningham et al., 2023; Dunefsky et al., 2024; Gao et al., 2025). Each of these three families provides useful units for analysis, but requires training or optimizing an additional representation or model beyond the dense pretrained checkpoint. The reported overheads can be substantial. For example, weight-sparse models require $100$–$1000\times$ more training and inference compute than dense models of comparable capability (Gao et al., 2025).

Alternatively, a natural question is whether circuit units can be obtained directly from a pretrained checkpoint, avoiding the cost of learning auxiliary representations altogether. NaNA (Xue & Andrzejak, 2026) applies exact SVD directly to MLP weights, ranks the resulting components by their contribution to a specified target token, and shows that a small set of top-ranked components can recover the target prediction without training an auxiliary representation. However, each retained component has dense input and output vectors, so retaining only a few components does not necessarily yield a compact circuit in terms of active connections. Weight-sparse transformers are trained from scratch with sparse weights and activations and show that sparse connectivity can yield substantially smaller and more human-understandable task circuits at comparable pretraining loss (Gao et al., 2025). This result suggests that sparse connectivity is a useful structural constraint for circuit analysis. We therefore ask whether weight-side sparse connectivity can be introduced after pretraining by reparameterizing an existing dense checkpoint. To pursue this question, we draw on Double Sparse Factorization from model compression, which approximates a dense matrix as the product of two sparse matrices (Boza & Macko, 2025). We evaluate whether this sparse decomposition can preserve the original model’s behavior while yielding task circuits with fewer active connections.

Our key idea is to obtain circuit units directly from pretrained weights via sparse factorization, avoiding the cost of training an auxiliary representation. Specifically, we introduce Sparse Weight Decomposition (SWD), which reparameterizes a pretrained dense linear projection with weight matrix $\mW$ as $\mW\approx \mA\mB$, where $\mA$ and $\mB$ are sparse factor matrices. Each intermediate coordinate $i$ is a bottleneck unit: it reads through $\mA_{:,i}$ and writes through $\mB_{i,:}$, defining a fixed rank-one path $\mA_{:,i}\mB_{i,:}$ with sparse read and write connections. Its scalar activation can be independently scored, selected, and ablated using the same circuit-extraction workflow applied to learned sparse features (Bhaskar et al., 2024; Conmy et al., 2023; Miller et al., 2024; Syed et al., 2023; Wang et al., 2022).

In our experiments, we evaluate both replacement fidelity (how closely each replacement preserves dense-model behavior before circuit extraction) and the circuit cost-quality tradeoff (how many selected units or active read/write edges are required to reach each sufficiency or necessity target). We first replace one MLP matrix in each of GPT-2 Small (Radford et al., 2019) and three Qwen2.5 (Qwen et al., 2025) models of different sizes, comparing SWD with activation- and parameter-space baselines under matched pre-pruning cross-entropy (CE) loss, as detailed in Section 3.3. To test scalability to larger models, we further apply the same single-matrix replacement and circuit-extraction protocol to Qwen3.5-27B (Qwen Team, 2026). We simultaneously replace all 48 attention and MLP weight matrices across the 12 transformer blocks of GPT-2 Small to test whether SWD works when local approximation errors accumulate (Section 3.4). As controls, exact full-rank dense factorizations reproduce the original weight matrix without error, testing whether low circuit cost comes from factorization alone or from sparse read/write connectivity (Section 3.5). Finally, a zero-data variant constructs factors from weights alone, omitting calibration activations (input activations collected on calibration data), to test whether the model alone can be turned into useful bottleneck units (Section 3.6).

Our contributions are as follows:

• We repurpose and evaluate sparse weight factorization as a circuit-extraction method: its bottleneck units can be scored and pruned without training a separate replacement network.

• Compared with baselines such as Transcoder and VPD, SWD reaches matched replacement fidelity using less than $1\%$ as much data and, at that fidelity, SWD requires fewer active read/write edges to attain the same sufficiency and necessity thresholds.

• SWD scales across model sizes and, after fixed-support fine-tuning, remains effective when all attention and MLP weight matrices are replaced simultaneously. It also admits a variant that requires no calibration data.

2 Methodology

2.1 Sparse Weight Decomposition

Sparsity on the edges. A dense matrix has many factorizations, but an exact dense reparameterization such as Singular Value Decomposition (SVD) still leaves every bottleneck unit connected to almost every input and output dimension. Motivated by evidence that weight-sparse models preserve more tractable circuits (Gao et al., 2025), SWD places sparsity on the factorization’s read and write edges (weights). Each bottleneck unit therefore reads from and writes to only a small set of input/output dimensions. Section 3.5 shows that the resulting cost-quality tradeoff advantage does not come from exact factorization alone.

Objective. For a dense matrix $\mW \in \mathbb{R}^{d_{\mathrm{in}} \times d_{\mathrm{out}}}$, SWD seeks sparse factors $\mA \in \mathbb{R}^{d_{\mathrm{in}} \times m}$ and $\mB \in \mathbb{R}^{m \times d_{\mathrm{out}}}$ such that $\mW \approx \widehat{\mW}=\mA\mB$, where $m$ is the number of bottleneck units. Table 3 in Appendix A summarizes the recurring notation used throughout the paper. Under a fixed nonzero budget $K$, the sparse product cannot in general preserve the action of $\mW$ equally well in every input direction, so the optimization must decide where to allocate approximation error (Boza & Macko, 2025). The Frobenius objective $\|\mW-\mA\mB\|_F^2$ nevertheless weights all directions equally so that at the same sparsity, it can spend limited fitting capacity on directions rarely encountered by the model, and leave avoidably large error on the activations the model actually processes (Section 3.6). Our default objective therefore minimizes reconstruction error on calibration activations:

where $\vh$ is a calibration-activation vector and $K$ is the total number of nonzero factor entries. In experiments, we report $K$ as a sparsity level $s=1 - \frac{K}{\|\mW\|_0}$ (larger $s$, smaller $K$). Since

the Gram matrix $\mG=\mathbb{E}_{\vh\sim\mathcal{D}_{\mathrm{cal}}}[\vh^\top\vh]$ directs the limited fitting capacity toward input directions with greater mass under the model’s activation distribution. When calibration activations are unavailable, we also have a zero-data variant: set $\mG=\mI$, reducing the objective to $\|\mW-\mA\mB\|_F^2$, which we evaluate in Section 3.6.

Solver. Because the $\ell_0$-constrained objective is nonconvex and NP-Hard, we approximately optimize it using the Double Sparse Factorization (DSF) heuristic (Boza & Macko, 2025). We split the total budget into fixed factor-specific budgets, $K_A+K_B=K$, and alternate between updating $\mA$ with $\mB$ fixed and updating $\mB$ with $\mA$ fixed. Each block update is an $\ell_0$-constrained regression problem, which DSF solves through Alternating Direction Method of Multipliers (ADMM) (Boyd et al., 2011) iterations consisting of: 1. a regularized least-squares update, 2. A hard-thresholding projection onto the prescribed nonzero budget, and 3. a dual-variable update. After the prescribed outer iterations, DSF freezes the per-matrix supports and the matrix $\mA$ and locally refits the surviving entries of the matrix $\mB$ against the activation-weighted reconstruction objective.

For a single matrix, this is the entire SWD process. When many weight matrices are replaced simultaneously, however, their local approximation errors can accumulate. In the full-model replacement experiment, we therefore minimize next-token cross-entropy over the nonzero factor values while keeping all other pretrained parameters fixed. Here, fixed support means that the zero/nonzero patterns of $\mA$ and $\mB$ are frozen, so optimization updates only values at existing nonzero entries. This preserves the sparse read/write connectivity and its corresponding edge count. We refer to the replacement after fixed-support fine-tuning as SWD-FT and evaluate it in Section 3.4; implementation and hyperparameter details are given below.

Takeaway. SWD turns a dense matrix into two sparse factors so that every bottleneck unit becomes a rank-one path with few read and write edges.

2.2 Circuit Extraction from Sparse Factors

The factorized replacement can be written as a sum of additive read–write paths. For an input-activation vector $\vh$, define

As shown in Figure 1, bottleneck unit $i$ reads $\vh$ through the sparse vector $\mA_{:,i}$ and writes the resulting scalar along the sparse vector $\mB_{i,:}$; their supports therefore specify which input and output coordinates the path touches. Moreover, replacing $z_i(\vh)$ changes only the $i$th term in the sum, leaving the other paths unchanged. This makes each bottleneck unit individually addressable for circuit interventions.

We rank bottleneck units by a task-margin attribution score and evaluate nested top-$k$ prefixes, as detailed below. Given this ranking, a candidate circuit is a subset $S\subseteq\{0,\dots,m-1\}$. Let $\mu_i$ be the reference activation for bottleneck unit $i$. Following the mean-ablation convention used in prior circuit analysis (Wang et al., 2022), we set $\mu_i$ to be the mean activation on the circuit extraction training split. We form two complementary interventions:

The keep intervention tests sufficiency: how much task behavior is retained by only keeping those units in $S$. The ablation intervention tests necessity: how much task behavior is lost by removing those units in $S$. In both cases, the intervened output is obtained by substituting the corresponding activations into $\sum_i z_i \mB_{i,:}$.

These interventions are comparable across methods only when the unpruned replacements share the same pre-pruning behavior; otherwise scores are measured against different reference computations. The detailed replacement-fidelity matching rule and the common attribution-based $S$ selection protocol are specified in Section 3.2.

Takeaway. Each bottleneck is an independently ablatable path. We rank paths on circuit training examples, and then test sufficiency and necessity on the test set.

3 Experiments

Our experiments track two criteria throughout: replacement fidelity and circuit cost-quality tradeoff, which is defined in Section 3.2. We first replace one MLP matrix in each of GPT-2 Small and the Qwen2.5 models from 0.5B to 3B with SWD, Transcoder, and VPD variants, then extend this single-matrix comparison to Qwen3.5-27B to test scalability, and evaluate sufficiency and necessity under matched pre-pruning cross-entropy. We then report full-model replacement results, isolate the contribution of sparse read/write edges from exact dense controls, evaluate a zero-data variant, and qualitatively inspect selected units and circuit. Models and replacement targets are summarized below.

3.1 Baselines

We compare SWD with three approaches that provide units for circuit analysis: Transcoders learn sparse activation-space replacements for MLP computations (Dunefsky et al., 2024); sparse-pretrained models impose weight sparsity during pretraining (Gao et al., 2025); and VPD learns parameter components with an input-dependent selector (Bushnaq et al., 2026). We evaluate two VPD configurations. VPD-KL denotes the original VPD configuration, whose fidelity term is model-level Kullback–Leibler (KL) divergence. VPD-Recon-CI retains the same component parameterization and causal-importance (CI) selector but replaces this term with local activation reconstruction. VPD-KL does not reach the fidelity required for CE-matched circuit comparison in our single-matrix replacement experiments, so we report it only for replacement quality; VPD-Recon-CI is used for circuit comparisons. Its configuration is provided below, while companion replacement-quality metrics are reported with the single-matrix results. Since sparse pretraining produces an entire sparse model rather than a single-matrix replacement, we compare against it only in the full-model replacement experiment (Section 3.4). Table 1 summarizes how these methods differ in representation, sparsity, optimization, data requirements, and applicable scope. Full baseline settings are provided below.

3.2 Evaluation Protocol

Our evaluation protocol largely follows the task-based circuit-discovery setup used by ACDC/auto-circuit and Edge-Pruning (Bhaskar et al., 2024; Conmy et al., 2023; Miller et al., 2024), together with first-order attribution scoring from attribution patching (Syed et al., 2023). Evaluation proceeds in two stages. First, we evaluate replacement fidelity using CE delta (replacement CE minus dense-model CE), supplemented where applicable by KL and activation relative mean-squared error (relative MSE; the relative error between the dense and replacement outputs of the replaced projection). Detailed definitions are provided with the replacement-quality metrics and settings. All methods use FineWeb-Edu for training or calibration, with a disjoint split reserved for evaluation (Lozhkov et al., 2024). We report token–CE loss curves to evaluate data efficiency.

Since circuit scores are computed relative to each method’s own replacement, unless explicitly stated otherwise, we compare circuit performance only between replacements whose pre-pruning CE differs by at most $0.001$. Second, for circuit extraction, we use greater-than, indirect-object identification (IOI), docstring, and gendered-pronoun task families from public Edge-Pruning and auto-circuit datasets (Bhaskar et al., 2024; Conmy et al., 2023; Miller et al., 2024). For each task $\tau$, we compute unit scores and mean-ablation statistics on the circuit extraction training split (circuit_train). The resulting ranking defines a nested family of top-$k$ prefixes $S_k$. We then evaluate these fixed prefixes on the held-out test split (circuit_test) and report held-out sufficiency and necessity as functions of circuit cost. Data sources, split sizes, and the Qwen task subsets are detailed below.

For each method, we apply this common selection protocol to the candidate units exposed by its representation. Units are ranked by positive first-order task-margin attribution, as detailed below. Ablated units are replaced by their mean activation on circuit_train; thus $M_{\mathrm{keep}}(S)$ retains the selected units and mean-ablates the remainder, whereas $M_{\mathrm{abl}}(S)$ mean-ablates the selected units and retains the remainder. Following prior causal circuit analyses (Wang et al., 2022), we use mean ablation for the main comparisons because it removes input-dependent variation while preserving each unit’s average activation level. Because circuit evaluations can be sensitive to the ablation methodology, including the value assigned to ablated activations (Miller et al., 2024), we also test whether our conclusions depend on this choice. The zero-ablation analysis below keeps the checkpoints, unit rankings, and top-$k$ sets fixed and repeats the GPT-2 single-matrix evaluation using zero rather than mean ablation. Let $g_{\tau}(M,\vx)$ be the task logit margin for example $\vx$: the model’s average logit for correct task answers minus its average logit for distractor answers. We define $Q_{\tau}(M;\mathcal{D})=\mathbb{E}_{\vx\in\mathcal{D}}[g_{\tau}(M,\vx)]$ as the average task score of model $M$ on split $\mathcal{D}$, with $M_{\mathrm{unpruned}}$ denoting the replacement before circuit pruning. We report method-relative sufficiency,

and

We measure circuit size by selected units and active edges. An active edge is a nonzero weight in the read or write vector of a selected unit. For a selected set $S$, let $\mathbf{r}_i$ and $\mathbf{w}_i$ denote the read and write weight vectors of unit $i$, and let $S_{\mathrm{eff}}\subseteq S$ contain the selected units with at least one nonzero read edge and one nonzero write edge. We define

The active-edge count describes the structural connectivity of $S$ and is the same whether $S$ is retained for sufficiency or ablated for necessity. For each sufficiency or necessity value, we report the minimum selected units or minimum active edges among the evaluated top-$k$ sets that reach that value. Method-specific definitions are given in the earlier SWD settings and baseline settings.

3.3 Single-Matrix Replacement

We begin by replacing the GPT-2 Small layer-8 MLP output projection, mlp.c_proj, with each method’s corresponding sparse representation. Figure 2 plots CE delta from the dense model against data used. SWD reaches low replacement error with substantially less data: SWD ($s{=}0.5$) reaches low CE delta after a few thousand tokens versus roughly $10^6$ for the baselines, whereas VPD-KL improves only after much longer optimizer replay and then plateaus. Because its CE cannot be matched to the other methods, we retain VPD-KL as a replacement-quality reference but exclude it from circuit extraction evaluation. Activation relative MSE, KL, and full settings are documented later in this section.

Figure 3 compares performance at matched pre-pruning CE, with one sparser SWD setting shown separately as a non-matched control. Across all four tasks, SWD attains the same sufficiency and necessity targets with substantially fewer active edges than Transcoder and VPD-Recon-CI.

The two SWD settings expose a trade-off between replacement fidelity and circuit size. SWD with $s{=}0.5$ achieves lower replacement CE and is therefore used in the CE-matched comparison, whereas the more aggressive $s{=}0.75$ setting incurs a modest fidelity penalty but requires fewer active edges to retain the same task behavior or induce the same performance drop when ablated. Thus, increasing factor sparsity can reduce circuit size even when it slightly reduces replacement fidelity.

This advantage is not specific to mean ablation: when the checkpoints, unit rankings, and top-$k$ sets are held fixed and zero ablation is used instead, SWD continues to require fewer active edges across all four tasks, as shown below.

Companion results using selected units are reported below; the corresponding settings are given in the earlier baseline settings.

We repeat the single-matrix comparison on Qwen2.5 models from 0.5B to 3B and extend it to Qwen3.5-27B. Figure 4 reports replacement quality on Qwen2.5-3B and Qwen3.5-27B. Further details are provided below. In both models, SWD reaches low CE delta with substantially fewer tokens than the trained baselines.

Figure 5 shows that SWD reaches comparable sufficiency and necessity on Qwen2.5-3B with fewer active edges, consistent with the GPT-2 results. Results for the other Qwen2.5 model sizes are reported below.

At the larger scale, Figure 6 shows the same pattern on Qwen3.5-27B: SWD reaches comparable sufficiency and necessity with fewer active edges. Further details for Qwen3.5-27B are provided below.

In the experiments above, each method replaces only one MLP matrix. The whole-MLP comparison below uses a standard Transcoder to replace the entire MLP, while SWD replaces both MLP weight matrices with sparse factorizations and retains the original GELU.

Takeaway. SWD requires substantially less data than trained replacement baselines to reach matched replacement fidelity, while achieving better circuit sufficiency and necessity with fewer selected units and active edges.

3.4 Full-Model Replacement

In this experiment, we replace all 48 attention and MLP weight matrices across the 12 transformer blocks of GPT-2 Small. The embeddings, layer-normalization modules, nonlinearities, and LM head remain unchanged. We compare with weight-sparse pretraining (Gao et al., 2025) at an approximately matched budget of $26.77$M active weights (about $68\%$ sparsity relative to the dense transformer-body weight matrices); the exact training recipe and accounting are given in the earlier sparse-pretraining configuration.

Approximation error now accumulates across layers: the sparse factorization reaches held-out CE $3.90$. We therefore apply fixed-support fine-tuning, updating only the nonzero factor values while keeping the sparse supports (and hence the edge count) fixed. The resulting SWD-FT replacement reaches CE $3.44$, slightly below the matched sparse-pretraining checkpoint’s $3.45$, while using under $1\%$ of its token budget ($20.6$M tokens in total, i.e. $4.19$M for factorization and $16.38$M for fixed-support fine-tuning, versus $2.884$B tokens for sparse pretraining). Table 2 summarizes this comparison; the full curve is shown below.

For task circuits, we prune SWD bottleneck units and the sparse-pretrained model’s latent channels under their corresponding protocols (Section 3.2), then compare the minimum active edges needed to reach each held-out sufficiency or necessity target, as shown in Figure 7. At the approximately matched budget, the methods are broadly comparable on greater-than and gendered-pronoun. On docstring and IOI, the sparse-pretraining circuit results have near-zero or negative necessity drop, whereas SWD-FT remains effective on all four tasks.

Takeaway. Fixed-support fine-tuning restores replacement quality and matches sparse pretraining at an approximately equal active-weight budget and circuit cost-quality tradeoff with far fewer training tokens.

3.5 Ablation: Exact Dense Reparameterizations

The fact that SWD requires fewer circuit edges could arise either from its sparse read/write matrices or simply from factorizing the original matrix into intermediate units. To distinguish these explanations, we compare SWD with two exact factorizations of the same GPT-2 Small output-projection matrix. The first is a full-rank SVD, using the same exact rank-one decomposition as NaNA (Xue & Andrzejak, 2026); the second is Random-B, which uses a seeded random orthogonal matrix for $\mB$ and computes $\mA$ from $\mW$ and $\mB$. Both factorizations reproduce the target matrix exactly, but their factor matrices are dense. We apply the same task-level attribution and mean-ablation procedure to all three methods.

Figure 8 shows that the exact SVD and Random-B factorizations require more active edges than SWD to retain the same task behavior under sufficiency evaluation or induce the same performance drop under necessity evaluation. Since this pattern holds for both the SVD basis and random orthogonal bases, the comparison isolates the contribution of sparse read/write structure from factorization alone. The construction, fidelity checks, and companion selected-unit results are provided below.

Takeaway. The exact SVD and Random-B factorizations require more active edges than SWD to preserve the same task behavior or produce the same ablation effect, isolating the contribution of sparse read/write structure.

3.6 Zero-Data Factorization

The previous section shows that sparse structure helps SWD achieve low-cost circuit results. We next ask whether constructing such bottleneck units requires calibration activations. Calibration activations may be unavailable and can tie the factorization to a particular data distribution, so we evaluate the zero-data SWD variant defined in Section 2.1, which depends only on checkpoint weights. Because zero-data SWD requires only the current weights, it can in principle be applied at every training step without collecting calibration activations, enabling step-by-step mechanistic analysis of how circuit structure emerges and evolves during training. We leave such training-trajectory analysis to future work. On the GPT-2 output-projection surface, we vary $s$ from $0.125$ to $0.875$ and measure two complementary notions of fidelity: relative Frobenius error $\|\mW-\mA\mB\|_F/\|\mW\|_F$ in parameter space and held-out CE delta in model behavior. The factorization and evaluation settings are given below.

As shown in Figure 9, zero-data SWD remains closer to $\mW$ throughout all sparsity levels, reaching relative error $0.366$ versus $0.415$ for activation-calibrated SWD at $s=0.75$. Conversely, activation-calibrated SWD achieves lower CE under aggressive sparsification because it allocates the limited edge budget toward the distribution of activation directions. Thus zero-data SWD better preserves the raw weights, whereas calibration better preserves behavior on typical model activations.

To further analyze the circuit properties of the zero-data factors, we apply the same protocol in the zero-data circuit results below and show that zero-data bottlenecks remain useful for task circuits.

Takeaway. Calibration data improves replacement fidelity, but useful circuit units can still be recovered directly from checkpoint weights without activation data.

3.7 Qualitative Results

We conclude the experimental analysis with three qualitative views of SWD bottleneck units. We first examine whether units selected by task attribution have recognizable semantic hypotheses, then test whether an individual unit supports a targeted directional edit, and finally use the factors to diagnose attention computation and interactions between units. These analyses complement the aggregate circuit results above by illustrating what the extracted units represent and how they can be manipulated.

3.7.1 Semantic Audit of a Task Circuit

We audit GreaterThan on the GPT-2 Small all-layer mlp.c_proj surface by ranking all $9{,}208$ bottleneck units with the train-split positive first-order attribution used for pruning. The dashboard below displays every unit in the global top-$512$ prefix, with six audited units from layers 6, 8, and 10 pinned at the top. For each audited unit, we evaluate $2{,}048$ task-independent WikiText-2 tokens, collect its 20 highest-magnitude activation contexts, and use GPT-5.5 to summarize recurring patterns into a tentative semantic hypothesis. These contexts and labels do not affect the attribution score, rank, or circuit membership.

The six audited units have global ranks 32–129. Four respond to numbers, quantities, or measurements, while the other two respond to punctuation or species names. This alignment between task attribution and task-independent semantic labels complements the held-out sufficiency and necessity results, showing that the selected units are both behaviorally important and semantically recognizable without implying that they are globally monosemantic.

3.7.2 Targeted Bottleneck-Unit Editing

Having examined what attribution-selected bottleneck units respond to, we next test whether an individual SWD bottleneck unit can support a directional intervention. To ensure a common base model across editing methods, every edit starts from the original dense GPT-2 Small layer-8 mlp.c_proj weight. We select c205 by screening eight candidate prompt–answer–foil triples for positive target activation and answer-direction write alignment. On the prompt The opposite of up is, we use its read vector to construct a rank-one update along the answer–foil unembedding direction defined by down and left. We compare this edit with a randomly selected unit using the same calibration, rank-4 LoRA trained for each target shift, and a target-conditioned dense rank-one oracle.

Figure 11 shows the efficacy–locality tradeoff over the full edit sweep. At the strongest positive SWD setting, the answer–foil margin increases by $0.216$, while mean final-token KL is $4.02\times10^{-5}$ on seven unrelated factual prompts and $1.96\times10^{-5}$ on a separate 504-token global evaluation sample. The c205 edit remains farther left than the random-unit control and rank-4 LoRA at comparable positive target-margin changes, indicating lower measured side effects; LoRA reaches larger absolute margin changes, but at substantially higher non-target KL. The target-conditioned dense rank-one oracle achieves the strongest tradeoff, as expected from using the target activation directly. Thus, this example demonstrates a more precise edit through an extracted SWD bottleneck direction than through the practical controls. The exact update, sweep grid, unit-selection rule, and LoRA training details are provided in the edit construction and control settings below.

3.7.3 Mechanistic Diagnostics

The preceding sections used MLP bottleneck units for semantic labeling and targeted editing. We now ask whether SWD also exposes a causal mechanism inside attention. Figure 12 previews a selected layer-9, head-3 case: the SWD reconstruction retains the dense attention structure with mean KL $0.0405$ (top), whereas removing q266 redirects attention sharply toward the first token, raising the mean KL to $2.496$ and producing a maximum probability change of $0.775$ (bottom). We next trace this result from reconstruction fidelity through static QK screening and prompt-level replay to bottleneck-unit ablation.

The attention case above isolates a causal effect for one selected query unit. For MLPs, however, interpreting bottleneck units one at a time raises a different question: how independent are their contributions? Post-GELU c_proj contributions are nearly uncorrelated apart from a small structured tail, whereas pre-GELU c_fc bottleneck units interact more broadly through the nonlinearity. We next distinguish correlations between additive output paths from non-additivity introduced by GELU.

Takeaway. Across semantic auditing, targeted editing, and mechanistic diagnostics, SWD bottleneck units provide interpretable and manipulable units for analyzing model computation, while the observed interactions caution against treating every unit as independent or monosemantic.

4 Related Work

Task-level circuit discovery and evaluation. Mechanistic interpretability (MI) seeks small computational subgraphs whose interventions explain a model behavior (Olah et al., 2020). Manual transformer-circuit analyses have isolated mechanisms such as induction heads, indirect-object identification, and greater-than comparison (Elhage et al., 2021; Hanna et al., 2023; Olsson et al., 2022; Wang et al., 2022). Causal tracing and activation patching localize behavior by exchanging or ablating internal states, while ACDC, attribution patching, and edge pruning make the search over components or edges more scalable (Bhaskar et al., 2024; Conmy et al., 2023; Meng et al., 2022; Syed et al., 2023). Recent evaluations emphasize that circuit conclusions depend on the intervention, metric, and granularity, and therefore favor held-out causal tests and concise recovered pathways over overlap with a single presumed circuit (Hanna et al., 2024; Miller et al., 2024; Mueller et al., 2025; Zhang & Nanda, 2024). Whereas these methods primarily decide which nodes or edges to retain in an existing computational basis, SWD introduces an alternative basis of addressable bottleneck units inside otherwise dense linear maps. It is thus complementary to circuit-search algorithms: attribution and mean ablation can be applied to the induced units, with held-out sufficiency and necessity measuring circuit quality and selected units and active edges measuring circuit size.

Sparse activation features and learned replacement modules. Superposition makes native neurons polysemantic and motivates learning overcomplete sparse feature dictionaries (Elhage et al., 2022). Sparse autoencoders (SAEs) decompose residual-stream or sublayer activations into sparsely active features, and large releases such as Gemma Scope make such dictionaries available across layers and model scales (Bricken et al., 2023; Cunningham et al., 2023; Lieberum et al., 2024). Sparse feature circuits connect SAE features into causal graphs that support intervention and editing (Marks et al., 2025). Transcoders instead learn a sparse-feature module that approximates an MLP’s input–output computation, avoiding some difficulties in tracing an SAE feature through the original nonlinear MLP; attribution graphs extend this replacement-based view to larger computation graphs (Ameisen et al., 2025; Dunefsky et al., 2024; Kamath et al., 2025). Like these approaches, SWD exposes scalar units that can be scored, selected, and intervened on. SAEs and Transcoders, however, fit a new activation dictionary or replacement network on an activation corpus, whereas SWD factorizes a checkpoint matrix and sparsifies each unit’s parameter-side read and write vectors. This construction applies uniformly to MLP and attention projections, and defines each unit as a sparse rank-one path without assuming that it is a monosemantic activation feature. We test the practical effect of this distinction through the matched-fidelity Transcoder comparison.

Parameter-space interpretability. Parameter-decomposition methods seek simpler additive components directly in model weights. Attribution-based Parameter Decomposition (APD) optimizes for parameter faithfulness, sparse per-input component use, and component simplicity; Stochastic Parameter Decomposition (SPD) develops a more scalable stochastic formulation (Braun et al., 2025; Bushnaq et al., 2025). VPD extends this family to language-model parameters using an input-dependent selector, and sparse attention decomposition uses SVD-derived parameter components to trace query–key computations (Bushnaq et al., 2026; Franco & Crovella, 2024). NaNA treats each rank-one component in the exact SVD of an MLP weight matrix as a detector–effector unit (DEU). It scores each DEU by how strongly it is activated by the input and how much it supports a specified target token, then retains or removes the highest-scoring units to test their effect on the prediction (Xue & Andrzejak, 2026). SWD also constructs fixed rank-one components from model weights, but each component consists of one sparse column of $\mA$ and one sparse row of $\mB$. SWD constructs these components independently of the downstream circuit task and subsequently ranks them using a common task-margin attribution protocol; VPD instead learns input-dependent component use as part of the decomposition.

Sparse models and post-training pruning. Weight pruning shows that substantial parameter sparsity can be recovered from dense language models with little or no retraining. SparseGPT uses approximate second-order information for one-shot pruning, while Wanda scores weights using both magnitude and observed input activations (Frantar & Alistarh, 2023; Sun et al., 2024). A complementary approach builds weight and activation sparsity into pretraining; such sparse-pretrained transformers can preserve more tractable circuits than comparable dense models, albeit at the cost of training a new model (Gao et al., 2025). These results motivate the hypothesis that useful sparse structure can be recovered from a dense checkpoint, but do not directly provide the intervention surface used by SWD. Pruning removes scalar edges without creating intermediate units that can be independently scored and ablated, while sparse pretraining changes the model from the start. SWD instead introduces bottleneck units with sparse incoming and outgoing edges post hoc. The sparse-pretraining comparison tests the shared sparse-connectivity hypothesis, while the exact dense-factor controls distinguish the effect of sparse edges from that of factorization alone.

Matrix factorization and compression. Classical matrix methods impose low-rank, non-negativity, or sparsity to obtain compact representations (d’Aspremont et al., 2004; Lee & Seung, 1999; Mairal et al., 2010). For neural-network compression, quantized sparse weight decomposition combines structural constraints, while activation-aware and truncation-aware SVD methods use calibration statistics to preserve the action of a weight matrix on likely inputs (Kuzmin et al., 2022; X. Wang et al., 2025; Yuan et al., 2023). NaNA uses rank-one components from an exact SVD as intervention units (Xue & Andrzejak, 2026). In an exact SVD, however, the left and right singular vectors are generally dense, so each component connects to most input and output dimensions. Double Sparse Factorization (DSF) instead approximates a dense weight matrix as the product of two sparse matrices (Boza & Macko, 2025). SWD adopts DSF and uses calibration inputs when fitting its two sparse matrices so that the replacement preserves the outputs of the original dense matrix. The exact-SVD controls in Section 3.5 apply the same attribution and ablation protocol to dense SVD components and sparse SWD components, testing whether sparse read/write connections reduce the number of selected units and active edges required to reach the same sufficiency and necessity levels.

5 Limitations

Efficient extraction is not complete understanding. SWD reaches matched replacement fidelity while using less than $1\%$ of the data that trained baselines use to train their replacements, and it exposes circuits with fewer active edges. This local-to-global gap is shared by other MI approaches: learned feature dictionaries and task-circuit methods provide tractable local objects without, by themselves, yielding a complete account of the model (Ameisen et al., 2025; Conmy et al., 2023; Cunningham et al., 2023; Syed et al., 2023). Thus, our efficiency gains do not resolve the broader concern that a large nonlinear model may not admit a compact, human-understandable mechanistic account (Hendrycks & Hiscott, 2025). Our sufficiency and necessity results establish that selected bottleneck units causally affect held-out task margins, but they do not explain the model globally or guarantee coverage of rare, safety-relevant cases.

SWD circuits are local and non-unique. Identifiability of a two-factor sparse matrix decomposition requires additional structural conditions and is, even then, only defined up to unavoidable scaling and permutation symmetries (Zheng et al., 2021). SWD does not establish such conditions for its approximate learned decompositions, so the resulting bottleneck units can depend on the objective, calibration distribution, initialization, and sparsity budget. Moreover, SWD generally intervenes on an approximate replacement $\mA\mB$: matched cross-entropy and reconstruction control average drift but cannot guarantee agreement with the original dense model on every input. First-order ranking can miss behaviorally important edges, and mean-ablation conclusions can vary with the intervention and evaluation protocol (Hanna et al., 2024; Miller et al., 2024; Zhang & Nanda, 2024); redundant or nonlinear mechanisms are also suggested by the MLP interaction results above. Recent controlled evidence further shows that exact circuit claims can change with the reported graph, pruning threshold, query/key representation, and comparison granularity, even when coarser summaries remain stable (Sheng & Fu, 2026). Our comparisons hold the task data, ranking rule, ablation policy, top-$k$ evaluation, and cost accounting fixed across methods; this supports controlled comparisons within our chosen protocol but does not establish invariance to alternative extraction or reporting choices. The extracted circuits should therefore be interpreted as task- and protocol-specific causal explanations, not unique or complete mechanistic ground truth.

Scaling verification remains open. Our Qwen3.5-27B experiment covers one matrix, full-model replacement is evaluated only on GPT-2 Small, and the semantic audit examines six model-labeled bottleneck units. Thus, reducing factorization data and circuit size does not remove the need to verify hypotheses across decompositions, prompt distributions, and adversarial edge cases. Following the top-down alternative emphasized by Hendrycks & Hiscott (2025), SWD is best used alongside behavioral and representation-level analyses rather than as a certificate of understanding or safety.

Takeaway. SWD makes local circuit hypotheses substantially cheaper to extract and test, but it does not eliminate MI’s non-uniqueness, distributional, verification, or scaling limitations.

6 Conclusion

Sparse Weight Decomposition turns dense pretrained projections into bottleneck units with sparse read/write connectivity that can be used directly for circuit extraction. At matched replacement fidelity, SWD achieves comparable circuit sufficiency and necessity with fewer selected units and active edges than learned sparse replacements across GPT-2, Qwen2.5, and Qwen3.5-27B. Fixed-support fine-tuning extends the method to full-model replacement, and a zero-data variant recovers useful bottleneck units from checkpoint weights alone. Exact SVD and Random-B factorizations isolate the contribution of sparse read/write structure from factorization alone. The resulting parameter-side surface supports compact circuit selection and causal testing in dense models.

References

1. Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. (2024). Finding Transformer Circuits with Edge Pruning. Advances in Neural Information Processing Systems (NeurIPS).

2. Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, and others. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/methods.html.

3. Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. (2011). Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Foundations and Trends in Machine Learning. 3(1), 1–122. doi:10.1561/2200000016.

4. Vladimir Boza and Vladimir Macko. (2025). Two Sparse Matrices Are Better Than One: Sparsifying Neural Networks with Double Sparse Factorization. International Conference on Learning Representations. https://openreview.net/forum?id=DwiwOcK1B7.

5. Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. (2025). Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition. arXiv preprint arXiv:2501.14926.

6. Trenton Bricken, Adly Templeton, Joshua Batson, and others. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features.

7. Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin, Bart Bussmann, Nathan Hu, Michael Ivanitskiy, Linda Linsefors, and Lee Sharkey. (2026). Interpreting Language Model Parameters. Technical Report. https://www.goodfire.ai/research/interpreting-lm-parameters.

8. Lucius Bushnaq, Dan Braun, and Lee Sharkey. (2025). Stochastic Parameter Decomposition. arXiv preprint arXiv:2506.20790.

9. Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. (2018). Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges. IEEE Signal Processing Magazine. 35(1), 126–136. doi:10.1109/MSP.2017.2765695.

10. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adria Garriga-Alonso. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. arXiv preprint arXiv:2304.14997.

11. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. (2023). Sparse Autoencoders Find Highly Interpretable Features in Language Models. arXiv preprint arXiv:2309.08600.

12. Alexandre d’Aspremont, Laurent Ghaoui, Michael Jordan, and Gert Lanckriet. (2004). A direct formulation for sparse PCA using semidefinite programming. Advances in neural information processing systems. 17.

13. Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. (2020). Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey. Proceedings of the IEEE. 108(4), 485–532. doi:10.1109/JPROC.2020.2976475.

14. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. (2024). Transcoders Find Interpretable LLM Feature Circuits. arXiv preprint arXiv:2406.11944.

15. Nelson Elhage, Tristan Hume, Catherine Olsson, and others. (2022). Toy Models of Superposition. Transformer Circuits Thread. https://transformer-circuits.pub/2022/toy_model/index.html.

16. Nelson Elhage, Neel Nanda, Catherine Olsson, and others. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html.

17. Elias Frantar and Dan Alistarh. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. Proceedings of the 40th International Conference on Machine Learning. 202. https://proceedings.mlr.press/v202/frantar23a.html.

18. Gabriel Franco and Mark Crovella. (2024). Sparse Attention Decomposition Applied to Circuit Tracing. https://arxiv.org/abs/2410.00340.

19. Michael Hanna, Ollie Liu, and Alexandre Variengien. (2023). How does GPT-2 Compute Greater-Than? Interpreting Mathematical Abilities in a Pre-Trained Language Model. Advances in Neural Information Processing Systems.

20. Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. (2024). Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. First Conference on Language Modeling. https://openreview.net/forum?id=TZ0CCGDcuT.

21. Dan Hendrycks and Laura Hiscott. (2025). The Misguided Quest for Mechanistic AI Interpretability. AI Frontiers. Guest Commentary. https://ai-frontiers.org/articles/the-misguided-quest-for-mechanistic-ai-interpretability.

22. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations. https://openreview.net/forum?id=nZeVKeeFYf9.

23. Joseph Miller, Bilal Chughtai, and William Saunders. (2024). Transformer Circuit Faithfulness Metrics Are Not Robust. First Conference on Language Modeling. https://openreview.net/forum?id=zSf8PJyQb2.

24. Harish Kamath, Emmanuel Ameisen, Isaac Kauvar, Rodrigo Luger, Wes Gurnee, Adam Pearce, Sam Zimmerman, Joshua Batson, Thomas Conerly, Chris Olah, and Jack Lindsey. (2025). Tracing Attention Computation Through Feature Interactions. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attention-qk/index.html.

25. Andrey Kuzmin, Mart van Baalen, Markus Nagel, and Arash Behboodi. (2022). Quantized Sparse Weight Decomposition for Neural Network Compression. arXiv preprint arXiv:2207.11048.

26. Daniel D. Lee and H. Sebastian Seung. (1999). Learning the Parts of Objects by Non-negative Matrix Factorization. Nature. 401(6755), 788–791. doi:10.1038/44565.

27. Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, and Dan Mossing. (2025). Weight-sparse transformers have interpretable circuits. https://arxiv.org/abs/2511.13653.

28. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. (2024). Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. 278–300. https://aclanthology.org/2024.blackboxnlp-1.19/.

29. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. (2024). FineWeb-Edu: the Finest Collection of Educational Content. Hugging Face. doi:10.57967/hf/2497.

30. Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. (2010). Online learning for matrix factorization and sparse coding.. Journal of Machine Learning Research. 11(1).

31. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. (2025). Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. International Conference on Learning Representations. https://openreview.net/forum?id=I4e82CIDxv.

32. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. (2022). Locating and Editing Factual Associations in GPT. arXiv preprint arXiv:2202.05262.

33. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. (2017). Pointer Sentinel Mixture Models. International Conference on Learning Representations. https://openreview.net/forum?id=Byj72udxe.

34. Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek, Amir Zur, David Bau, and Yonatan Belinkov. (2025). MIB: A Mechanistic Interpretability Benchmark. Proceedings of the 42nd International Conference on Machine Learning. 267, 45069–45108. https://proceedings.mlr.press/v267/mueller25a.html.

35. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. (2020). Zoom In: An Introduction to Circuits. Distill. doi:10.23915/distill.00024.001.

36. Catherine Olsson, Nelson Elhage, Neel Nanda, and others. (2022). In-context Learning and Induction Heads. Transformer Circuits Thread. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.

37. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. (2025). Qwen2.5 Technical Report. https://arxiv.org/abs/2412.15115.

38. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. (2019). Language Models are Unsupervised Multitask Learners.

39. Yang Sheng and Jie Fu. (2026). Circuit Claims Depend on What Is Extracted and How It Is Compared. arXiv preprint arXiv:2607.18921. https://arxiv.org/abs/2607.18921.

40. Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. (2024). A simple and effective pruning approach for large language models. International Conference on Learning Representations. 2024, 4942–4964.

41. Aaquib Syed, Can Rager, and Arthur Conmy. (2023). Attribution Patching Outperforms Automated Circuit Discovery. arXiv preprint arXiv:2310.10348.

42. Michael Tsang, Dehua Cheng, and Yan Liu. (2018). Detecting Statistical Interactions from Neural Network Weights. International Conference on Learning Representations. https://openreview.net/forum?id=ByOfBggRZ.

43. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. (2022). Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. arXiv preprint arXiv:2211.00593.

44. Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. (2025). SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression. International Conference on Learning Representations. https://openreview.net/forum?id=LNYIUouhdt.

45. Min Xue and Artur Andrzejak. (2026). SVD as a Fast Interpretability Method for Transformers. Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=7tt8TwMjdJ.

46. Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. (2023). ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models. arXiv preprint arXiv:2312.05821. https://arxiv.org/abs/2312.05821.

47. Fred Zhang and Neel Nanda. (2024). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. International Conference on Learning Representations. https://openreview.net/forum?id=9eJv5PS27Q.

48. L’eon Zheng, Elisa Riccietti, and R’emi Gribonval. (2021). Identifiability in Exact Two-Layer Sparse Matrix Factorization. arXiv preprint arXiv:2110.01235. https://arxiv.org/abs/2110.01235.

49. Qwen Team. (2026). Qwen3.5: Accelerating Productivity with Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5.

A Notation

Table 3 consolidates the recurring notation used across the main text and appendices. We typeset scalars and indices in ordinary italic, vectors in bold lowercase, and matrices in bold uppercase; sets, datasets, models, and scalar-valued functions remain nonbold. Layer, head, and unit identifiers are zero-based, whereas displayed attribution ranks are one-based. For method-specific terms, we call an SWD intermediate coordinate a bottleneck unit, a Transcoder hidden element a feature, a VPD element a parameter component, and a sparse-pretrained hidden element a latent channel. We use unit as the method-agnostic umbrella term in cross-method comparisons and selected-unit figures.

The support of a factor is the set of its nonzero entries, and its factor mask is the corresponding binary indicator. A unit is valid when both its read and write vectors contain at least one nonzero entry; $S_{\mathrm{eff}}$ is the valid subset of a selected set.

B Experimental Details

This section consolidates the settings shared across experiments. We first specify the replacement surfaces, then give the SWD optimization, circuit-selection protocol, task data, and baseline configurations. Throughout our experiments, replacement fidelity is measured on data excluded from replacement fitting. For circuit comparisons, unit scores and mean-ablation values are estimated on circuit_train, and the resulting nested top-$k$ prefixes are evaluated on circuit_test.

B.1 Models and Replacement Targets

Table 4 summarizes the computational surfaces used in the paper. Within each single-matrix comparison, all methods replace the same weight matrix from the pretrained model. The MLP-replacement experiment is reported separately because a standard Transcoder replaces the entire MLP, whereas the full-model experiment replaces all attention and MLP weight matrices across the transformer blocks and therefore tests accumulated approximation error.

Setting	Model	Replacement surface
GPT-2 single matrix	GPT-2 Small	Layer 8 mlp.c_proj
Qwen2.5 scaling	Qwen2.5 0.5B, 1.5B, 3B	Middle-layer mlp.down_proj: layers 12, 14, and 18, respectively
Qwen3.5 single matrix	Qwen3.5-27B	Layer 31 mlp.down_proj
GPT-2 MLP replacement	GPT-2 Small	Layer 8 standard MLP, comprising mlp.c_fc and mlp.c_proj
GPT-2 full-model replacement	GPT-2 Small	attn.c_attn, attn.c_proj, mlp.c_fc, and mlp.c_proj in all 12 blocks
Zero-data	GPT-2 Small	Layer 8 mlp.c_proj, matching the GPT-2 single-matrix experiment
Qualitative audit	GPT-2 Small	mlp.c_proj in all 12 blocks; selected bottleneck units are visualized from layers 6, 8, and 10

Table 4. Models and replacement surfaces. Each comparison uses a common surface across the methods shown in that comparison.

B.2 SWD Factorization and Fixed-Support Fine-Tuning

We implement SWD with the activation-aware Double Sparse Factorization solver of Boza & Macko (2025). Standard decompositions use inner width $m=\min(d_{\mathrm{in}},d_{\mathrm{out}})$. The following configuration is fixed across a sparsity sweep; only the total nonzero budget $K$ changes.

• Budget and initialization. The square factor receives density $0.16$ for square targets and $0.25$ for rectangular targets, with the remaining entries in $K$ assigned to the other factor. We rescale the target using the diagonal of the calibration Gram matrix, initialize the square factor to the identity and the other factor to a magnitude-pruned rescaled target, and initialize the dual variables to zero.

• Alternating optimization. Each outer iteration updates both factors in turn. We use 40 outer iterations for the GPT-2 single-matrix and full-model experiments and 8 for the Qwen single-matrix experiments.

• ADMM block updates. Each block update uses 5 inner iterations, penalty $\rho=1$, and ridge coefficient $10^{-2}$. Hard thresholding recomputes the prescribed support during the first 2 inner iterations and then holds it fixed. At outer iteration $t$ of $T$, the first solve uses $\rho_{\mathrm{init}}=\min\{1,t/(T-3)\}^{3}$; subsequent inner solves use $\rho=1$.

• Finalization and precision. After alternating optimization, 20 fixed-support ADMM iterations refit the selected entries of one factor while holding the other factor and both factor supports fixed. Factorization uses float32 with TF32 matrix multiplication disabled. This local finalization is distinct from the model-level fixed-support fine-tuning used in the full-model experiment below.

• Data used to fit SWD replacements. Each point on an SWD replacement-quality curve corresponds to an independently fitted SWD replacement using the stated amount of calibration data; the points are not successive checkpoints from one training run. Table 5 separates the data ranges evaluated in these curves from the fixed SWD replacements subsequently used for circuit extraction.

Experiment	CE-curve data (tokens)	Fixed replacement used for circuit extraction
GPT-2 single matrix	1,024–4.19M	$s{=}0.5$: 16,384 tokens; CE delta $0.000889$
		$s{=}0.75$: 16,384 tokens; CE delta $0.008292$
Qwen2.5-0.5B	1,024–33.55M	$s{=}0.5$: 1,024 tokens; CE delta $0.001222$
		$s{=}0.75$: 1.05M tokens; CE delta $0.005010$
Qwen2.5-1.5B	1,024–33.55M	$s{=}0.5$: 1,024 tokens; CE delta $0.000733$
		$s{=}0.75$: 1.05M tokens; CE delta $0.001222$
Qwen2.5-3B	1,024–33.55M	$s{=}0.5$: 2,048 tokens; CE delta $0.000733$
		$s{=}0.75$: 1.05M tokens; CE delta $0.000611$
Qwen3.5-27B	1,024–16.78M	$s{=}0.5$: 2,048 tokens; CE delta $-0.000427$
		$s{=}0.75$: 1.05M tokens; CE delta $-0.000448$
GPT-2 MLP replacement	1,024–33.55M	$s{=}0.5$: 16,384 tokens; CE delta $0.004003$
		$s{=}0.75$: 16,384 tokens; CE delta $0.020491$
GPT-2 full model	4.19M (factorization) + 16.38M (fine-tuning)	Final SWD-FT replacement (20.57M total tokens)
Zero-data SWD	0	Fixed zero-data factors at each evaluated sparsity

Table 5. Data used for SWD replacement-quality results and the fixed SWD replacements used for circuit extraction. CE delta is replacement CE minus dense-model CE. The third column identifies the fixed replacement whose units are subsequently scored and ablated; it does not count the task data used for circuit scoring and evaluation.

In the primary GPT-2 and Qwen2.5 single-matrix comparisons, $s=0.5$ is the CE-matched SWD setting and $s=0.75$ is a higher-sparsity reference. Both fixed SWD settings are included in the Qwen3.5-27B circuit evaluation. In the MLP-replacement experiment, $s=0.5$ is the CE-matched SWD setting and $s=0.75$ is a higher-sparsity reference; SWD composes the two projection factorizations as specified in the whole-MLP comparison below. The full-model experiment instead compares methods at a matched active-weight budget, and zero-data SWD uses no calibration data (see the zero-data setting below).

Circuit cost. Each selected unit is one SWD bottleneck unit. Its read and write vectors are $\mA_{:,i}$ and $\mB_{i,:}$, so its active-edge count is $\|\mA_{:,i}\|_0+\|\mB_{i,:}\|_0$.

For the full-model experiment, approximation errors from the 48 replaced weight matrices accumulate. We therefore apply model-level fixed-support fine-tuning after sparse factorization. All pretrained parameters and binary factor masks remain frozen, and gradients update only factor values at existing nonzero locations. Fine-tuning thus preserves both the $26{,}770{,}800$-nonzero budget and the read/write connectivity used for circuit-cost accounting.

• Objective and data. Fixed-support fine-tuning minimizes autoregressive next-token cross-entropy on 4.19M unique training tokens. We train with batch size 8 for 2,000 steps, totaling 16.38M consumed tokens.

• Optimizer. We use AdamW with peak learning rate $5\times10^{-7}$, $(\beta_1,\beta_2)=(0.9,0.95)$, $\epsilon=0.1$, and weight decay $10^{-3}$.

• Schedule and gradient control. The learning rate warms up linearly over the first 1% of steps and then decays linearly to zero. Before each optimizer step, the global factor-gradient RMS is normalized to $1.0$.

• Precision and seed. Fine-tuning uses bfloat16 training, float32 evaluation, and seed 0.

B.3 Unit Scoring and Circuit Selection

Candidate units are scored under the unpruned replacement. SWD exposes bottleneck units, Transcoders expose hidden features, and the VPD variants expose parameter components. For the sparse-pretrained model, which has no explicit replacement bottleneck, we prune latent channels following its native setup (Gao et al., 2025).

Let $g_{\tau}(M,\vx)$ be the task logit margin for example $\vx$, computed from the next-token logits as the average correct-answer logit minus the average distractor-answer logit, and let $z_i(\vx)$ be the activation of candidate unit $i$ (for SWD, $z_i(\vx)=\vh_{\ell,t}(\vx)\mA_{:,i}$ at the task-scoring position). The train-split mean activation used for ablation is

$$\mu_i = \mathbb{E}_{\vx \in \mathcal{D}_{\tau,\mathrm{train}}}\left[z_i(\vx)\right].$$

The default score is positive first-order task-margin attribution (Bushnaq et al., 2026; Syed et al., 2023),

$$a_i = \mathbb{E}_{\vx \in \mathcal{D}_{\tau,\mathrm{train}}} \left[\max\!\left(0,\, z_i(\vx)\frac{\partial g_{\tau}}{\partial z_i(\vx)}\right)\right].$$

Here $\partial g_{\tau}/\partial z_i$ is the gradient of the task margin with respect to unit $i$‘s activation, so $a_i$ accumulates only positive first-order contributions to the margin. We sort each method’s candidate units once by $a_i$ on circuit_train; this fixed ranking defines the nested top-$k$ prefixes evaluated on held-out circuit_test.

B.4 Data Sources and Splits

B.4.1 Language-Model Data

All data-dependent replacement methods use FineWeb-Edu for fitting or training (Lozhkov et al., 2024). SWD uses subsets of this corpus to fit its sparse factors, whereas the trained baselines use it for optimization. Replacement quality is evaluated on a fixed, disjoint FineWeb-Edu split shared by all methods within each comparison. Accordingly, data used counts calibration tokens for SWD and optimizer-consumed tokens for the trained baselines. Method-specific data amounts and the checkpoints used for circuit extraction are reported in the SWD settings and baseline settings above. The zero-data experiment uses no text data.

B.4.2 Circuit-Task Data

Circuit extraction uses fixed public examples from Edge-Pruning (Bhaskar et al., 2024) and auto-circuit (Conmy et al., 2023; Miller et al., 2024). These task examples are separate from the FineWeb-Edu data above and are not included in the replacement fitting or training data. The training split is used to estimate unit scores and mean-ablation values; the test split is used only to evaluate the resulting fixed top-$k$ sets.

Task	Public source	GPT-2	Qwen2.5	Qwen3.5-27B
Greater-than	Edge-Pruning, a918efd8	80,000 / 12,240	Not used	Not used
IOI	auto-circuit, 8fc5afd8	600 / 200	600 / 200	600 / 200
Docstring	auto-circuit, 8fc5afd8	600 / 200	600 / 200	600 / 200
Gendered pronoun	Edge-Pruning, a918efd8	3,024 / 378	3,024 / 378	2,887 / 370

Table 6. Public circuit-task sources and train/test example counts. The linked repository revisions fix the source data used in our experiments.

GPT-2 uses all four tasks. Qwen2.5 and Qwen3.5-27B use IOI, docstring, and gendered pronoun. Greater-than is omitted for Qwen because its numerical answers do not satisfy the single-token next-token evaluation used in these experiments. The Qwen3.5-27B counts report the examples retained after model-specific tokenization compatibility checks.

B.5 Baseline Configurations

B.5.1 Transcoder

The Transcoder baselines use sparse hidden-feature replacements for the same target surface as the corresponding SWD comparison. Below we report the settings needed to identify the plotted Transcoder lines and the checkpoints used for circuit extraction.

Circuit cost. Each selected unit is one Transcoder hidden feature. Its active edges are the nonzero encoder and decoder weights connected to that feature.

GPT-2 single-matrix replacement. For the GPT-2 single-matrix experiment, the Transcoder replaces the layer 8 MLP output projection.

• Target: GPT-2 Small layer 8 mlp.c_proj.

• Widths: TC3k: Transcoder with hidden width 3072, TC12k: Transcoder with hidden width 12,288.

• Feature sparsity penalty: L1 coefficient $5\times10^{-5}$, seed 42.

• Replacement-quality curve: measured from 1,024 to 4.19M training tokens.

• Checkpoint for circuit extraction: 2.048M training tokens; CE deltas are 0.000529 for TC3k and 0.000979 for TC12k.

Qwen2.5 down-projection scaling. For the Qwen2.5 scaling experiment, each Transcoder replaces the middle-layer MLP down-projection of the corresponding model size: layer 12 for Qwen2.5-0.5B, layer 14 for Qwen2.5-1.5B, and layer 18 for Qwen2.5-3B.

• Widths: TC3k and TC12k, with hidden widths 3,072 and 12,288.

• Feature sparsity penalty: L1 coefficient $5\times10^{-5}$, seed 42.

• Replacement-quality curve: measured from 1,024 to 33.55M training tokens.

• Checkpoints for circuit extraction: Qwen2.5-0.5B uses 25.17M training tokens for both widths, with CE deltas 0.001654 for TC3k and 0.002032 for TC12k. Qwen2.5-1.5B uses 33.55M tokens for TC3k and 25.17M for TC12k, with CE deltas $-0.000057$ and $-0.000098$, respectively. Qwen2.5-3B uses 25.17M tokens for both widths, with CE deltas 0.001516 for TC3k and 0.001229 for TC12k.

Qwen3.5-27B down projection. For Qwen3.5-27B, each Transcoder replaces layer 31 mlp.down_proj.

• Widths: TC24k and TC48k, with hidden widths 24,576 and 49,152.

• Feature sparsity penalty: L1 coefficient $5\times10^{-5}$, seed 42.

• Replacement-quality curves: measured from 1,024 tokens; TC24k is shown through 134.22M training tokens and TC48k through 67.11M training tokens.

• Checkpoints for circuit extraction: 134.22M training tokens for TC24k and 67.11M training tokens for TC48K, with CE deltas 0.000398 and 0.000695, respectively.

GPT-2 MLP replacement. For the MLP-replacement comparison, we use the standard TC24k Transcoder setting with hidden width 24,576, L1 coefficient $1.4\times10^{-4}$, and seed 42. Circuit extraction uses only the 6.29M-token checkpoint, with CE delta 0.003456. The complete replacement-quality trajectory is reported in the whole-MLP comparison below.

B.5.2 VPD-KL and VPD-Recon-CI

VPD-KL uses the original VPD component parameterization, learned causal-importance (CI) mask, and model-level KL fidelity term. VPD-Recon-CI retains the same parameterization and mask but replaces this term with local activation reconstruction. The differing fidelity terms are

$$\begin{aligned} \mathcal{L}_{\mathrm{fid}}^{\mathrm{VPD\text{-}KL}} &= \mathbb{E}_{\vx,t} \!\left[ D_{\mathrm{KL}} \!\left( p_{\mathrm{dense},t}(\cdot\mid \vx) \,\|\, p_{\mathrm{VPD\text{-}KL},t}(\cdot\mid \vx) \right) \right], \\ \mathcal{L}_{\mathrm{fid}}^{\mathrm{VPD\text{-}Recon\text{-}CI}} &= \mathbb{E}_{\vx,t,j} \!\left[ \left( \widehat{y}^{\mathrm{CI}}_{\ell,t,j}(\vx)-y_{\ell,t,j}(\vx) \right)^2 \right]. \end{aligned}$$

Here $\vx$ denotes a token sequence, $t$ a next-token position, $\ell$ the target module, and $j$ one coordinate of that module’s output. The distributions $p_{\mathrm{dense},t}$ and $p_{\mathrm{VPD\text{-}KL},t}$ are the dense and VPD-KL next-token distributions. The values $y_{\ell,t,j}$ and $\widehat{y}^{\mathrm{CI}}_{\ell,t,j}$ are the corresponding dense and CI-masked replacement outputs; the expectation in the reconstruction objective averages over all batch elements, token positions, and output coordinates, matching the elementwise MSE used in training. The first objective constrains the final model output; the second directly constrains the local replacement. The equations isolate the fidelity terms; mask-minimality regularization is specified below. The CI network predicts a per-component mask, after which the masked rank-one components are recombined into the replacement output. VPD-KL is therefore reported only as a replacement-quality reference when its CE cannot be matched.

Circuit cost. Each selected unit is one VPD parameter component. Its active edges are the nonzero entries in that component’s read and write vectors. The CI predictor is not part of this read/write edge count.

GPT-2 single-matrix replacement. For the GPT-2 single-matrix experiment, VPD-Recon-CI replaces the layer 8 MLP output projection.

• Target: GPT-2 Small layer 8 mlp.c_proj.

• Components and CI predictor: $C=3{,}072$ components, seed 0, and a vector-MLP CI predictor of hidden width 128.

• Objective: MSE between the CI-masked replacement and dense projection outputs, plus a CI minimality penalty of $10^{-4}$.

• Checkpoint for replacement quality and circuit extraction: 4.19M training tokens, with CE delta 0.000873. We reuse this fixed checkpoint for both replacement-quality and circuit results.

Qwen2.5 down-projection scaling. For the Qwen2.5 scaling experiment, VPD-Recon-CI replaces the same middle-layer MLP down-projection used by SWD and the Transcoders.

• Targets: layer 12 mlp.down_proj for Qwen2.5-0.5B, layer 14 for Qwen2.5-1.5B, and layer 18 for Qwen2.5-3B.

• Components and objective: $C=12{,}288$ components and the same local CI-masked reconstruction objective as in the GPT-2 single-matrix experiment.

• Checkpoints for circuit extraction: 16.78M training tokens for Qwen2.5-0.5B, 4.19M for Qwen2.5-1.5B, and 16.78M for Qwen2.5-3B. Their CE deltas are 0.000729, 0.001186, and 0.000906, respectively. The corresponding circuit results are reported in the Qwen2.5 settings and results below.

Qwen3.5-27B down projection. For Qwen3.5-27B, VPD-Recon-CI replaces layer 31 mlp.down_proj.

• Components and CI predictor: $C=12{,}288$ components, seed 0, and a vector-MLP CI predictor of hidden width 12.

• Objective: MSE between the CI-masked replacement and dense projection outputs, plus a CI minimality coefficient of $3\times10^{-4}$.

• Checkpoint for circuit extraction: 200.00M training tokens, with CE delta 0.001847. It is retained as the parameter-decomposition reference; the corresponding circuit results are reported in the Qwen3.5-27B settings and results below.

GPT-2 MLP replacement. For the MLP-replacement experiment, VPD-Recon-CI replaces the complete layer 8 MLP.

• Components and objective: $C=3{,}072$ components with the local MLP-reconstruction objective.

• Replacement-quality curve: measured from 1,024 to 16.78M training tokens.

• Checkpoint for circuit extraction: 4.19M training tokens, with CE delta 0.003522.

• VPD-KL reference: $C=3{,}072$, seed 0, and the native model-level KL objective. Its replacement-quality curve is measured from 4,096 to 1B training tokens; it is not used for circuit extraction. Full results are reported in the whole-MLP comparison below.

B.5.3 Sparse-Pretrained Reference

The sparse-pretraining reference is used in the GPT-2 Small full-model comparison. Unlike the other baselines, it is a separately trained sparse model with GPT-2 Small dimensions, rather than a sparse replacement of the Hugging Face GPT-2 checkpoint.

Circuit cost. For circuit extraction, the 3,072 post-GELU MLP channels in each layer form 36,864 candidate units; attention and residual channels are not candidate units. Each selected unit is one MLP hidden channel. Its active edges are the nonzero incoming and outgoing weights connected to that channel. For channel $j$, this count is

$$\left\|\mW_{\mathrm{c\_fc}}[:,j]\right\|_0+ \left\|\mW_{\mathrm{c\_proj}}[j,:]\right\|_0+ \mathbf{1}\!\left[b_{\mathrm{c\_fc},j}\ne0\right].$$

GPT-2 full-model comparison.

• Architecture: 12 layers, residual width 768, 12 attention heads of width 64, MLP width 3,072, context length 1,024, and vocabulary size 50,257. The model uses learned positional embeddings, LayerNorm, tied token-embedding and language-model-head weights, biases, zero dropout, and standard nn.GELU. Activation sparsity is disabled. The use of nn.GELU differs from the gelu_new implementation in the Hugging Face GPT-2 checkpoint used by SWD.

• Training data: FineWeb-Edu documents are separated by end-of-sequence tokens and packed into 1,024-token examples; a disjoint FineWeb-Edu split is held out for evaluation. The global batch contains 128 examples, or 131,072 token presentations per optimizer update.

• Evaluation checkpoints: The replacement-quality and circuit results use separately trained models with identical configurations, each evaluated after 2,883,584,000 token presentations. Their held-out CE values are 3.450415 and 3.450323, respectively. The complete training schedule contains 4,999,872,512 token presentations, so the evaluated models are not terminal checkpoints.

• Matched active-weight budget: At the evaluated data point, the scheduled density is 0.315194486. The 48 transformer-block matrices contain 26,770,920 nonzeros (realized density 0.315194307), compared with 26,770,800 nonzeros in SWD-FT; the difference is 120 weights. This accounting excludes biases, normalization parameters, embeddings, and the tied language-model head. Across all trainable two-dimensional matrices, including token and positional embeddings, 39,184,479 of 124,318,464 entries are nonzero.

Optimization and sparsification. Training uses seed 0, bfloat16, and AdamW with base learning rate $3\times10^{-4}$, $(\beta_1,\beta_2)=(0.9,0.95)$, $\epsilon=0.1$, and weight decay $10^{-3}$. Let $n$ denote cumulative token presentations, with $n_0=49{,}938{,}432$, $n_1=3{,}999{,}793{,}152$, and $N=4{,}999{,}872{,}512$. The scheduled nonzero density is

$$d(n)= \begin{cases} 1,&n<n_0,\\ 0.2^{(n-n_0)/(n_1-n_0)},&n_0\leq n<n_1,\\ 0.2,&n\geq n_1. \end{cases}$$

Thus density remains one during the first 1% of training, decays exponentially to the target density 0.2 by 80% of training, and then remains fixed. The learning rate is

$$\operatorname{lr}(n)=3\times10^{-4}\frac{h(n)}{\sqrt{d(n)}}, \qquad h(n)= \begin{cases} n/n_0,&n<n_0,\\ 1-(n-n_0)/(N-n_0),&n\geq n_0. \end{cases}$$

After every optimizer update, each trainable two-dimensional matrix is magnitude-projected to its scheduled nonzero budget while retaining at least one weight per output row. The gradient routine computes a global RMS over all available parameter-gradient entries and, when this RMS exceeds $10^{-8}$, divides all gradients by $\mathrm{RMS}+10^{-5}$.

C Single-Matrix Replacement Results

C.1 Replacement Quality beyond Cross-Entropy

In the single-matrix comparison above, we measure replacement quality primarily by cross-entropy delta. We additionally report two complementary metrics on GPT-2 Small layer 8 mlp.c_proj and the three Qwen2.5 mlp.down_proj surfaces to verify that the comparison is not specific to CE.

Metrics. Let $\mY_{\mathrm{dense}}$ and $\mY_{\mathrm{rep}}$ stack the dense and replacement outputs of the target projection over all evaluated examples and token positions. The activation relative MSE is a local, single-layer measure of how well the replacement reproduces that projection’s output,

$$\mathrm{relMSE} = \frac{\lVert \mY_{\mathrm{rep}} - \mY_{\mathrm{dense}}\rVert_F^2} {\lVert \mY_{\mathrm{dense}}\rVert_F^2}.$$

The numerator and denominator are aggregated over the complete evaluation set before taking the ratio; we do not average per-example relative errors. The KL is a global measure that inserts the replacement back into the full model and compares next-token distributions on a held-out language-model split,

$$\mathrm{KL} = \mathbb{E}_{t}\,\mathrm{KL}\!\big(p^{\mathrm{dense}}_t \,\|\, p^{\mathrm{rep}}_t\big),$$

with the dense model as the reference distribution and the expectation taken over held-out tokens $t$. Lower is better for both metrics. The x-axis reports data used in tokens, rather than compute, and uses a broken log scale so the short-horizon comparison and the VPD-KL long tail are both visible. We report activation relative MSE first because it measures the replaced layer directly, and KL second because it measures how that local error propagates to the model’s output.

Both metrics reproduce the CE ordering across all single-matrix replacements. SWD reaches low error with little data, whereas Transcoder and VPD-Recon-CI approach it after roughly $10^{6}$ optimizer-replay tokens. VPD-KL is the weakest local replacement because its native objective targets model-level logits instead of the projection output (see the VPD settings above). No checkpoint on its extended GPT-2 trajectory, from 4,096 to 1.6384B tokens, meets the CE-matching criterion, showing that the fidelity gap is not an early-training artifact.

C.2 GPT-2 Selected-Unit Circuit Results

Figure 16 presents the GPT-2 Small layer 8 mlp.c_proj circuit results from Figure 3, using selected units rather than active edges as the cost axis. The main CE-matched comparison uses SWD ($s{=}0.5$), TC3k, TC12k, and VPD-Recon-CI; a sparser SWD ($s{=}0.75$) is retained as a non-CE-matched sparse control.

C.3 Robustness under Zero Ablation

The main circuit results replace ablated units with their mean activation on circuit_train. To test whether SWD’s advantage depends on using mean ablation rather than zero ablation, we repeat the GPT-2 single-matrix circuit evaluation from Section 3.3 using zero ablation. We keep the method checkpoints, unit rankings, top-$k$ sets, task examples, and definitions of selected units and active edges fixed. The only change is the intervention value: for sufficiency, units outside the selected set are set to zero; for necessity, the selected units are set to zero. No unit scores or rankings are recomputed.

Figure 17 shows that SWD’s active-edge advantage persists under zero ablation. At sufficiency retention of at least $1.0$, SWD with $s=0.75$ has the lowest observed active-edge cost on all four tasks; on GreaterThan, it requires $12{,}303$ active edges, compared with $184{,}320$ for TC3k. The necessity results show the same overall separation, with the SWD curves reaching comparable necessity drops at lower active-edge costs than the trained baselines.

The selected-unit comparison in Figure 18 shows the same pattern. Together, these results show that SWD’s advantage in the GPT-2 single-matrix experiment is not specific to using mean activation as the ablation reference.

C.4 Qwen2.5 Circuit Results

The Qwen2.5 comparison targets the middle-layer mlp.down_proj in each model size. VPD-Recon-CI uses $C=12{,}288$ components and the same local reconstruction objective as in the GPT-2 experiment. Table 7 specifies the target modules and fixed checkpoints used for circuit extraction. VPD-KL appears only in the replacement-quality figures because its fidelity is insufficient for a controlled circuit comparison. Figure 19 reports replacement quality for all three model sizes. Figures 20 and 21 complement the 3B result in Figure 5 above with the 0.5B and 1.5B circuit results.

Model and target module	Checkpoint for circuit extraction	CE delta
Qwen2.5 0.5B, layer 12 mlp.down_proj	16.78M tokens used	0.000729
Qwen2.5 1.5B, layer 14 mlp.down_proj	4.19M tokens used	0.001186
Qwen2.5 3B, layer 18 mlp.down_proj	16.78M tokens used	0.000906

Table 7. Qwen2.5 VPD-Recon-CI settings. All rows use $C=12{,}288$ components and replace one mlp.down_proj.

Figures 22–24 recast the same three Qwen2.5 circuit sweeps with selected units as the cost axis, separating selected-unit count from read/write connectivity, as in the GPT-2 companion (Figure 16).

C.5 Qwen3.5-27B Circuit Results

The Qwen3.5-27B comparison targets layer 31 mlp.down_proj. All methods use the same IOI, docstring, and gendered-pronoun task data and the circuit-selection protocol described above. Figure 6 reports active edges, and Figure 25 reports the same circuit results using selected units. Table 8 lists the fixed checkpoints used for circuit extraction.

Method	Setting	Checkpoint for circuit extraction	CE delta
SWD	$s=0.5$	2,048 calibration tokens	$-0.000427$
SWD	$s=0.75$	1.05M calibration tokens	$-0.000448$
Transcoder	width 24,576	134.22M training tokens	$0.000398$
Transcoder	width 49,152	67.11M training tokens	$0.000695$
VPD-Recon-CI	$C=12{,}288$	200.00M training tokens	$0.001847$

Table 8. Qwen3.5-27B checkpoints used for circuit extraction. CE delta is measured against the same dense model and held-out language-model data.

Relative to SWD with $s=0.5$, the CE-delta differences are $0.000022$ for SWD with $s=0.75$, $0.000824$ for TC24k, $0.001121$ for TC48k, and $0.002274$ for VPD-Recon-CI. TC48k is slightly outside the $0.001$ matching tolerance. VPD-Recon-CI has the largest difference and is retained as a reference for the parameter-decomposition baseline.

D Full-Model Replacement

Figure 26 shows the replacement-quality trajectory behind Table 2. The SWD point before fine-tuning appears at 4.19M calibration tokens with CE 3.90. Fixed-support fine-tuning then lowers the CE along the SWD-FT curve without changing the active-weight budget, reaching CE 3.44 at 20.6M total tokens. This matches the sparse-pretraining baseline’s CE 3.45, which is reached after 2.884B tokens.

E MLP Replacement

The main single-matrix experiment constrains every method to replace mlp.c_proj. Standard Transcoders normally replace the entire MLP, so we also compare on the GPT-2 Small layer 8 MLP input–output map. For SWD, this means factorizing the weight matrices of the two linear projections, mlp.c_fc and mlp.c_proj, while keeping the original GELU nonlinearity $\phi$ between them:

$$\widehat{\mathrm{MLP}}(\vh) = \phi\!\left(\vh\mA_{\mathrm{fc}}\mB_{\mathrm{fc}}+\vb_{\mathrm{fc}}\right) \mA_{\mathrm{proj}}\mB_{\mathrm{proj}}+\vb_{\mathrm{proj}}.$$

Thus, SWD changes only the two linear projections, while the nonlinearity is the original GELU used by the dense model.

Method	Replacement configuration	Checkpoint for circuit extraction
SWD	Sequential factorization of c_fc and c_proj; $s\in\{0.5,0.75\}$	16,384 tokens used
Transcoder	Standard MLP Transcoder (settings above)	6.29M tokens used; CE delta 0.003456
VPD-Recon-CI	Local MLP-reconstruction objective; $C=3{,}072$	4.19M tokens used
VPD-KL	Native KL-logit objective; $C=3{,}072$, seed 0	Replacement only

Table 9. Fixed configurations for the GPT-2 Small layer 8 MLP-replacement comparison. SWD tokens are calibration tokens; Transcoder and VPD-Recon-CI tokens are optimizer replay. VPD-KL is excluded from circuit extraction because its replacement fidelity is not matched.

Replacement-quality trajectories share a 16.78M-token horizon; the VPD-KL curve is extended to 1B tokens to test whether longer optimization closes its fidelity gap. Across CE delta, KL, and activation relative MSE, it remains the weakest replacement. As in the single-matrix experiment, we retain it for replacement-quality context but exclude it from task-circuit pruning.

We then apply the same circuit extraction protocol to the fixed MLP-replacement checkpoints in Table 9. The primary matched comparison uses SWD with $s=0.5$, a single TC24k checkpoint at 6.29M tokens, and the 4.19M-token VPD-Recon-CI checkpoint; SWD with $s=0.75$ provides a higher-sparsity reference. Figures 28 and 29 report the resulting cost-quality tradeoff using active edges and selected units as the cost axes. On this broader surface, the results are more mixed than in the single-matrix setting: SWD remains competitive on the active-edge axis and often requires fewer edges to reach the same necessity drop, while the standard MLP Transcoder is competitive in several selected-unit and sufficiency comparisons. This comparison complements the single-matrix results by testing the same protocol on the MLP-replacement surface.

F Exact-Dense Reparameterization Controls

The exact dense controls in Figure 8 reparameterize GPT-2 Small layer-8 mlp.c_proj without approximation or sparsification. They test whether bottleneck re-expression alone yields SWD’s low-cost circuits. The SVD control uses the same exact rank-one decomposition used by NaNA (Xue & Andrzejak, 2026). To measure how sparsity in the read and write vectors affects the number of selected units and active edges required, we keep the task data, positive first-order task-margin attribution, mean ablation, and figure plotting fixed across SVD, Random-B, and SWD. This shared protocol compares dense and sparse read/write connections without changing how units are ranked or ablated. Control configurations.

• SWD ($s=0.5$): fixed CE-matched decomposition on the same mlp.c_proj surface; the primary matched-fidelity SWD line.

• SWD ($s=0.75$): higher-sparsity SWD reference on the same mlp.c_proj surface.

• SVD Exact: full-rank dense SVD of mlp.c_proj (rank 768, $d_{\mathrm{in}}=3072$, $d_{\mathrm{out}}=768$); an exact dense control that is neither low-rank nor sparse.

• Random-B (seed mean): random orthogonal exact dense control with $\mB=\mQ$, $\mA=\mW\mQ^\top$. Results are averaged over 10 runs (seeds 0–9), and the band shows one standard deviation above and below the mean.

Both dense controls use the same cost accounting: $k$ bottleneck units, or $k(d_{\mathrm{in}}+d_{\mathrm{out}})$ active edges. They exactly reparameterize the GPT-2 layer 8 mlp.c_proj matrix, whereas the SWD lines expose sparse read/write structure.

For the full-rank SVD control, the audited effective orientation is

$$\mW_{\mathrm{eff}} = \mA_{\mathrm{svd}}\mB_{\mathrm{svd}}, \qquad \mA_{\mathrm{svd}} = \mU\mSigma,\quad \mB_{\mathrm{svd}} = \mV^\top,$$

using all rank-768 singular directions. For the Random-B control,

$$\mB_{\mathrm{rand}} = \mQ, \qquad \mA_{\mathrm{rand}} = \mW_{\mathrm{eff}}\mQ^\top, \qquad \mW_{\mathrm{eff}} = \mA_{\mathrm{rand}}\mB_{\mathrm{rand}},$$

where $\mQ$ is a seeded random orthogonal matrix from Gaussian QR. In both cases $\mW_{\mathrm{eff}}$ reproduces the original mlp.c_proj map exactly.

Both controls are numerically exact: SVD has mean CE delta $1.58\times10^{-6}$ and KL $1.81\times10^{-7}$; across 10 Random-B seeds, CE delta ranges from $-2.48\times10^{-6}$ to $3.40\times10^{-6}$ and mean KL is $1.89\times10^{-7}$. Their need for more active edges therefore cannot be attributed to poorer replacement fidelity. On the selected-unit axis (Figure 30), SVD and Random-B are competitive on IOI and at low docstring necessity levels, while SWD requires fewer units in most other regions. The larger and more consistent difference appears on the active-edge axis, where SWD’s sparse read/write factors reduce the number of active connections per selected unit.

G Zero-Data Analysis

G.1 Factorization and Evaluation Setting

To isolate the role of calibration activations, we keep the standard SWD solver, factor dimensions, and nonzero-budget schedule fixed and replace only the activation Gram matrix with the identity:

$$\mH^\top \mH \leftarrow \mI, \qquad \min_{\mA,\mB}\|\mW-\mA\mB\|_F^2 \quad \mathrm{s.t.}\quad \|\mA\|_0+\|\mB\|_0\leq K .$$

No calibration activations are used to choose the supports or fit their surviving values. We factorize the GPT-2 Small layer 8 mlp.c_proj from the main single-matrix experiment over $s\in\{0.125,0.25,0.375,0.5,0.625,0.75,0.875\}$. Larger $s$ leaves fewer active scalars. The fidelity plots below compare each resulting product $\mA\mB$ with the corresponding activation-calibrated factorization at the same sparsity.

G.2 Circuit Results

Replacement fidelity alone does not establish that the bottleneck units remain useful after circuit pruning. We therefore apply the common unit-scoring and top-$k$ protocol from Section 3.2 to every zero-data sparsity checkpoint. For each of greater-than, IOI, docstring, and gendered-pronoun, the bottleneck-unit ranking and ablation means are computed on circuit_train, then frozen before sufficiency and necessity are evaluated on circuit_test. Figure 31 reports the minimum active edges and selected units needed to reach each held-out quality threshold. The Transcoder curves use the fixed checkpoints from the single-matrix experiments, listed in the Transcoder settings above, as contextual references.

The task results in Figure 31 suggest that zero-data SWD factors are not merely good matrix approximations; their bottleneck units remain useful for circuit extraction after pruning. It also shows that many zero-data task-sparsity settings retain high held-out sufficiency at small cost: at $s=0.375$, the docstring circuit reaches $0.962$ test sufficiency with only 3 selected units, and the greater-than circuit reaches $0.986$ with 256 units. Sufficiency is easiest to retain on greater-than, docstring, and IOI, while gendered-pronoun is the weakest task, with fewer high-sufficiency low-cost points.

Useful circuit structure can therefore be recovered directly from pretrained weights without calibration data. Activation-aware SWD remains stronger under aggressive sparsification, but the zero-data result shows that its bottleneck units are not solely an artifact of a chosen calibration distribution.

H GreaterThan Case Study: Semantic Audit

This section provides additional details for the semantic audit above. We rank all $9{,}208$ valid mlp.c_proj bottleneck units in GPT-2 Small using the GreaterThan attribution score defined in the unit-scoring protocol above. The dashboard highlights the two highest-ranked units in layers 6, 8, and 10, and Table 10 extends the comparison to the top three units in each layer. The global ranks are computed over the full candidate pool.

For each bottleneck unit, we then evaluate 2,048 task-independent WikiText-2 tokens and collect the 20 surrounding text passages associated with its largest activation magnitudes. GPT-5.5 summarizes the recurring patterns in these passages into a short semantic hypothesis. The passages and semantic labels are not used for unit ranking or circuit selection.

Layer	Bottleneck unit	Layer order	Global rank	Score	Semantic hypothesis
6	c380	1	108	10.2	measure/unit
6	c723	2	129	9.7	punctuation/function
6	c558	3	152	9.3	mixed lexical
8	c264	1	76	11.3	quantity
8	c511	2	84	11.0	measure/place
8	c728	3	123	9.8	event/time
10	c757	1	32	14.4	numeric/year
10	c627	2	63	11.8	species/name
10	c481	3	85	10.9	numeric/year

Table 10. Top-three GreaterThan bottleneck units in each displayed layer. Layer order is used only for visualization; global rank is over all $9{,}208$ candidates. Semantic hypotheses are task-independent and do not affect ranking.

The six audited bottleneck units have global ranks 32–129 and all fall within the global top-$512$ circuit prefix. Table 11 gives representative activation contexts for these units. Four respond to numbers, quantities, or measurements, while the other two respond to punctuation or species names. The dominant pattern is therefore quantitative, with two units reflecting the surrounding syntactic or lexical context.

Bottleneck unit	Semantic hypothesis	Top-activating excerpts
L6 c380	measure/unit	“carapace length of 80–85 millimetres”; “usually 23–38 cm”; “a carapace length of 15 mm”.
L6 c723	punctuation/function	Repeated comma-token activations in clauses such as “higher than that of H. americanus, and the European species” and around numeric punctuation.
L8 c264	quantity	“estimated that only 1 larva”; “not normally deeper than 50 m (160 ft)”; “published in 1758”.
L8 c511	measure/place	“48 kilometres or 30 miles northwest of Gothenburg”; “for up to 12 months”; “body length up to 60 centimetres”.
L10 c757	numeric/year	“published in 1758”; “Homarus Weber, 1795”; “Fabricius, 1775”.
L10 c627	species/name	Repeated activations on species-name fragments in “H. americanus”, “H. gammarus”, and “Homarus americanus”.

Table 11. Representative task-independent top-activation contexts for the six bottleneck units highlighted in the dashboard. Bold marks the token or phrase most relevant to each tentative semantic hypothesis.

Overall, the semantic audit gives the GreaterThan circuit a more concrete interpretation. Most audited bottleneck units respond to numbers, years, quantities, or measurements in task-independent text, consistent with their high GreaterThan attribution. Together with the held-out sufficiency and necessity results, this shows that the selected SWD bottleneck units are both behaviorally important and semantically recognizable.

I Targeted Editing of a Single SWD Bottleneck Unit

Our design adapts the model-editing template of making a localized low-rank weight update and evaluating both target efficacy and locality on unrelated inputs (Meng et al., 2022). To remove replacement-fidelity differences from the comparison, every method starts from the same original dense GPT-2 Small layer-8 mlp.c_proj weight $\mW$. The SWD intervention uses the read direction of one bottleneck unit to materialize the dense rank-one update induced by changing that unit’s write direction. We include a random-unit control, a target-conditioned dense rank-one oracle, and rank-4 LoRA (Hu et al., 2022). The source SWD factorization uses $s=0.75$, 768 bottleneck units, seed 0, and 131,072 WikiText-2 calibration tokens (Merity et al., 2017).

The main circuit experiments establish that SWD bottleneck units support targeted ablation. Here we ask a complementary question: can the read direction of one unit support a localized directional edit? We study the prompt

The opposite of up is answer: down, foil: left.

I.1 Editing Construction and Reproducibility Details

The dense model assigns the answer an answer–foil logit margin of $3.261101$. Let $\va_c=\mA_{:,c}\in\mathbb{R}^{3072}$ be the read vector of bottleneck unit $c$, let $\vh^\star\in\mathbb{R}^{1\times3072}$ be the final-position input to mlp.c_proj, and let

$$\vd=\vu_{\mathrm{ans}}-\vu_{\mathrm{foil}}, \qquad \overline{\vd}=\frac{\vd}{\lVert\vd\rVert_2},$$

where $\vu_{\mathrm{ans}},\vu_{\mathrm{foil}}\in\mathbb{R}^{768}$ are the column-vector forms of the corresponding unembedding rows. For requested local answer-direction shift $\delta$, the SWD-guided edit is

$$\mW_c(\delta) = \mW+\alpha_c(\delta)\va_c\overline{\vd}^{\top}, \qquad \alpha_c(\delta) = \frac{\delta} {(\vh^\star\va_c)\lVert\vd\rVert_2}.$$

It therefore satisfies

$$\vh^\star\bigl(\mW_c(\delta)-\mW\bigr)\vd=\delta.$$

This is the materialized dense increment induced by the corresponding write-row update, since

$$\mB'_{c,:} = \mB_{c,:}+\alpha_c(\delta)\overline{\vd}^{\top} \quad\Longrightarrow\quad \mA\mB'-\mA\mB = \alpha_c(\delta)\mA_{:,c}\overline{\vd}^{\top}.$$

We apply this increment to the original dense $\mW$, rather than using $\mA\mB$ as the base, so that all editing methods are evaluated from the same model. The requested-shift grid is

$$\delta\in\{-2,-1,-0.5,0,0.5,1,2,3\}.$$

Unit c205 was selected by screening eight candidate prompt–answer–foil triples for positive target activation and answer-direction write alignment, retaining the top 24 positive units per prompt and incorporating stored task importance and rank in the selection score. The random-unit control c540 was sampled with seed 17 from units whose absolute target activation lies between the 35th and 85th percentiles. It uses the same answer direction and requested-shift calibration as c205. The target-conditioned dense rank-one oracle replaces the SWD read vector by the normalized target activation,

$$\vr_{\mathrm{dense}} = \frac{\vh^{\star\top}}{\lVert\vh^\star\rVert_2}, \qquad \mW_{\mathrm{dense}}(\delta) = \mW+\alpha_{\mathrm{dense}}(\delta) \vr_{\mathrm{dense}}\overline{\vd}^{\top},$$

with $\alpha_{\mathrm{dense}}(\delta)$ calibrated by the same local answer-direction constraint. Rank-4 LoRA uses $\Delta\mW=\mL_A\mL_B$, with $\mL_A\in\mathbb{R}^{3072\times4}$ initialized as $0.01\mathcal{N}(0,1)$ and $\mL_B\in\mathbb{R}^{4\times768}$ initialized to zero. A separate LoRA update is trained for each requested final-margin shift using AdamW for 180 steps with learning rate $0.08$, zero weight decay, and an $L_2$ coefficient of $10^{-4}$. The Pareto comparison uses the realized final-margin change of every method.

I.2 Additional Evaluation Results

At requested local shift $\delta=3$, the c205 coefficient is $\alpha_{205}=0.423220$, and the realized final answer–foil margin change is $0.215797$. Mean dense-to-edited final-token KL over the seven non-target prompts is $4.02375\times10^{-5}$. On the separate 504-token global evaluation sample, the corresponding mean KL is $1.96415\times10^{-5}$.

Figure 11 shows the expected ordering. The target-conditioned dense rank-one oracle gives the strongest tradeoff. The c205 edit is next: it opens farther vertically and remains farther left than the random-unit control. Rank-4 LoRA attains larger absolute margin changes, but at substantially higher non-target KL. Thus, on this audited target and probe set, an SWD-guided bottleneck-unit edit provides a more precise intervention than the random-unit and LoRA controls, though it does not match the target-conditioned dense rank-one oracle.

Takeaway. For the audited target, materializing the write-direction edit associated with one selected SWD bottleneck unit changes the answer margin with lower measured side effects than the random-unit and LoRA controls, though not the target-conditioned dense rank-one oracle.

J Attention Bottleneck Units: From Static QK Geometry to Prompt-Local Effects

To analyze attention at the unit-pair level, we combine the QK-circuit decomposition of attention (Elhage et al., 2021), sparse weight-space attention decomposition for circuit tracing (Franco & Crovella, 2024), and feature-level QK attribution (Kamath et al., 2025). Because a pre-softmax attention score is bilinear in its query- and key-side inputs, an additive feature decomposition rewrites that score as a sum of query–key feature-pair terms. SWD supplies parameter-side query and key bottleneck units for this analysis. We first verify that the factorization preserves the dense attention pattern, then rank unit pairs by their static geometry and validate the strongest candidates on concrete prompts with causal intervention.

The factors come from the full-model experiment in Section 3.4, which replaces every attention and MLP weight matrix across the transformer blocks. Here we use its sparse Q and K bottleneck units to test whether parameter-side candidates correspond to attention computations that are active on real tokens and sensitive to intervention.

J.1 Attention Factorization Fidelity

We first measure activation-weighted relative reconstruction SSE for the packed Q/K/V weight matrix attn.c_attn and the attention output weight matrix attn.c_proj in all 12 GPT-2 Small layers. As shown in Figure 32 (top), the Q/K/V reconstruction error remains below $4\%$ in every layer. The attention output matrix is less accurate in the middle layers, where its error reaches approximately $6\%$. The prompt-level analysis below therefore uses the better-preserved attn.c_attn factors and separately checks the reconstructed attention pattern against the dense head.

J.2 Static QK Candidate Screen

For one head $h$, write the reconstructed query and key at token positions $t$ and $u$ as

$$\begin{aligned} \vq_t^h &= \sum_a z_a(\vh_t)\vb^{\mathrm{Q}}_{a,h},\\ \vk_u^h &= \sum_b z_b(\vh_u)\vb^{\mathrm{K}}_{b,h}. \end{aligned}$$

Expanding their scaled dot product separates each attention score into bottleneck-unit-pair terms:

$$\begin{aligned} \operatorname{score}_{tu}^h &= \sum_{a,b}z_a(\vh_t)z_b(\vh_u)\gamma_{ab}^h,\\ \gamma_{ab}^h &= \frac{\langle \vb^{\mathrm{Q}}_{a,h},\vb^{\mathrm{K}}_{b,h}\rangle}{\sqrt{64}}. \end{aligned}$$

The static coefficient $\gamma_{ab}^h$ measures the interaction between a Q write direction and a K write direction; the activation product $z_a(\vh_t)z_b(\vh_u)$ determines whether that interaction contributes on a particular prompt and token pair.

For each of 12 layers and 12 heads, we retain the 16 Q and 16 K bottleneck units with the largest head-specific write norms, evaluate all 256 cross-pairs, and summarize the head by its largest $|\gamma_{ab}^h|$. Figure 32 (bottom) shows that the strongest candidates occur mainly in later layers. We advance the five strongest pairs in every head to prompt replay.

J.3 Prompt Replay and Bottleneck-Unit Intervention

We evaluate the five strongest static pairs from each layer and head on three prompts, for

$$12\ \text{layers} \times 12\ \text{heads} \times 5\ \text{pairs} \times 3\ \text{prompts} =2{,}160$$

prompt–head–pair evaluations. The clearest response occurs on When Mary gave John the book, he thanked in layer 9, head 3, for pair q266$\times$k64.

Before interpreting the pair, we compare the dense attention pattern with the pattern obtained after replacing the packed Q/K/V weight matrix by its SWD reconstruction. Figure 12 (top) shows that the main attention structure is retained; the mean dense-to-reconstructed attention KL is $0.0405$. We then evaluate the full pair contribution

$$z_{266}(\vh_t)z_{64}(\vh_u)\gamma_{266,64}^h$$

at every causally valid query–key position. As shown in Figure 33, the pair contributes negatively toward the first key, When, and positively toward most later keys, with signed mean $+4.26$ over valid positions.

Finally, we remove q266 from the reconstructed Q slice and recompute the head. Attention shifts sharply toward the first token: relative to the intact reconstructed pattern, mean attention KL rises to $2.496$ and the maximum probability change is $0.775$ (Figure 12, bottom). This reversal agrees with the signed contribution map: removing q266 eliminates the unit’s suppression of When and its support for later keys.

Together, these checks close the chain from static geometry to causal effect. SWD closely reconstructs the selected head, static QK geometry identifies q266$\times$k64, prompt replay shows a coherent token-level contribution, and ablating q266 reverses that pattern. This is a single selected attention case, but it shows that SWD’s attention bottleneck units can expose a concrete and manipulable part of a head’s computation.

Takeaway. Static QK geometry, prompt-level contribution, and bottleneck-unit ablation agree on the same attention mechanism, showing that SWD exposes a concrete and manipulable part of the head’s computation.

K Interaction Patterns among MLP Bottleneck Units

To formalize these two regimes, we draw on feature-based circuit analyses that express an MLP output as a sum of component contributions (Dunefsky et al., 2024), work on superposition showing that the corresponding features can overlap (Elhage et al., 2022), and measures of feature interaction based on departures from additivity across nonlinear activations (Tsang et al., 2018). We examine correlation between additive output contributions after GELU and contextual non-additivity when bottleneck-unit contributions are combined before GELU.

We analyze the two MLP weight matrices separately in GPT-2 Small layer 8. For the output matrix mlp.c_proj, bottleneck-unit contributions are combined linearly after GELU, so we measure correlation between their residual-stream contributions. For the input matrix mlp.c_fc, the vector contributions of the bottleneck units are summed before GELU, so we measure their pairwise contextual finite difference across the nonlinearity.

K.1 c_proj: Linear Contributions Are Usually Weakly Correlated

The linear analysis uses all 768 bottleneck units of the layer-8 mlp.c_proj factorization with $s=0.75$ and 2,048 held-out token activations. Bottleneck unit $i$ reads a scalar $z_i=\vh\mA_{:,i}$ and contributes the vector $z_i\mB_{i,:}$ to the residual stream. We define its output-variance energy as

$$\operatorname{Var}(z_i)\lVert \mB_{i,:}\rVert_2^2.$$

The 96 highest-energy bottleneck units are retained for the readable matrix views in Figure 34; distributional statistics use all 768 units.

For each bottleneck-unit pair, we measure absolute token-level read correlation, absolute cosine similarity between write directions, and the correlation of the complete residual contributions. Read and write overlap are each common in isolation, but they rarely occur for the same pair. Across all units, the 95th-percentile absolute read correlation is $0.207$ and the 95th-percentile absolute write cosine is $0.071$, whereas the 95th-percentile absolute contribution correlation is only $0.0049$; even its 99th percentile is $0.0089$.

The exceptional tail is compact. Bottleneck units 373, 447, 481, and 266 recur among the strongest positive pairs; the leading c373–c481 pair has contribution correlation $0.345$. In that pair, opposed read patterns combine with opposed write directions, so the two sign reversals yield positively correlated residual contributions. Thus, linear c_proj bottleneck units are usually weakly correlated as complete pathways, with a small set of structured exceptions.

K.2 c_fc and GELU: Contextual Pair Interactions

The nonlinear diagnostic uses the 1,536-bottleneck-unit layer-8 mlp.c_fc factorization and 8,192 held-out token activations. We replay the tokens with the lower-layer c_fc replacements installed, so the layer-8 inputs follow the sequentially reconstructed model. To obtain a readable pair matrix, we retain 128 units with the largest factor-rescaling-invariant pre-GELU downstream energy,

$$R_i = \mathbb{E}_t[z_{t,i}^2] \left\lVert \mB_{i,:}\mW_{\mathrm{proj}}\right\rVert_2^2,$$

giving $128\times127/2=8{,}128$ unordered pairs.

Let $\vc_{t,i}=z_{t,i}\mB_{i,:}$ be unit $i$‘s vector contribution to the c_fc output, and let

$$\vh_t = \vb_{\mathrm{fc}}+\sum_a\vc_{t,a}$$

be the complete reconstructed preactivation. For each pair, we compute the contextual finite difference

$$\begin{aligned} \boldsymbol{\Delta}_{t,ij} ={}& \operatorname{GELU}(\vh_t) -\operatorname{GELU}(\vh_t-\vc_{t,i}) -\operatorname{GELU}(\vh_t-\vc_{t,j})\\ &+\operatorname{GELU}(\vh_t-\vc_{t,i}-\vc_{t,j}). \end{aligned}$$

We then project this interaction through the original MLP output matrix, $\vq_{t,ij}=\boldsymbol{\Delta}_{t,ij}\mW_{\mathrm{proj}}$. Define the contextual marginal residual effect of unit $i$ as

$$\vm_{t,i} = \left[ \operatorname{GELU}(\vh_t) -\operatorname{GELU}(\vh_t-\vc_{t,i}) \right]\mW_{\mathrm{proj}}, \qquad E_i=\sum_t\lVert\vm_{t,i}\rVert_2^2.$$

The normalized interaction magnitude is

$$R_{ij} = \frac{ \left(\sum_t\lVert\vq_{t,ij}\rVert_2^2\right)^{1/2} }{ (E_iE_j)^{1/4} }.$$

Pair interactions are widespread but typically moderate. Across the 8,128 pairs, mean $R$ is $0.093$, the median is $0.089$, the 95th percentile is $0.132$, the 99th percentile is $0.154$, and the maximum is $0.221$. In total, 2,130 pairs have $R\geq0.10$, 113 have $R\geq0.15$, and four have $R\geq0.20$. Figure 36 shows a dense central band together with a thinner upper tail. The brighter first rows and columns indicate that several high-energy units interact moderately with many partners, while isolated bright cells reveal additional pair-specific effects.

The two MLP locations therefore show complementary structure. After GELU, most c_proj bottleneck paths make weakly correlated residual-stream contributions, with a compact set of exceptions. Across GELU, the c_fc finite differences reveal a dense field of moderate interactions and a thinner tail organized around recurring high-energy units. Thus, output-projection paths are usually well described by their additive contributions, whereas the effects of some input-projection paths are more informative when examined together with their strongest interaction partners.

Takeaway. Most post-GELU c_proj paths are weakly correlated, while GELU couples preactivation contributions into widespread moderate interactions and a smaller upper tail of recurring unit pairs.