I've been into RL for about six months, and more recently into what RL does to language models. Around the same time I got interested in AI safety and started looking for fellowships. Last spring I applied to the Astra Fellowship, the OpenAI Safety Fellowship, and the Anthropic Fellows Program, ¹ and didn't get into any of them. This summer I decided to try again with MATS Winter 2027 Cohort, specifically Neel Nanda's MATS12.0 stream. I've followed Neel on X for a while and most of what I know about pragmatic interpretability, comes from his posts. For applications to his group, he put out a document for applicants on how to do good interp research, with a list of project ideas. ² I chose to work on a question related to understanding reasoning models.

#1. The Research Question

A lot of what we know about the internals of reasoning models comes from studying distilled models, not RL-trained ones. That is mainly because the RL-trained models are huge and expensive and thus difficult to run interp experiments on. So it is important to find out if their reasoning mechanisms transfer. The answer is not that obvious. Nothing guarantees they ended up with the same machinery.

Two papers help me in my project. First, Bogdan et al. ³ ask which sentences in a chain of thought matter. They answer it by resampling: they cut the trace at a sentence, let the model regenerate the rest many times, and see how often the answer changes. They found that only a few sentences, mostly planning and backtracking ones, carry most of the causal weight. This is a claim about how reasoning is organized which is what an imitation objective doesn't see. For a distill, nothing tells it which ones were load-bearing for the teacher. So their result raises a question: is the anchor structure a property of reasoning, or a property of the traces that got copied? It turns out that their method also provides the right tool to answer this question. Second, Venhoff et al. ⁴ go one level deeper, from sentences to activations. They label spans of a reasoning trace by behavior (backtracking, uncertainty, adding knowledge, and so on), average the residual-stream activations on those spans, and subtract the average over all reasoning tokens. What's left is a direction. Add that direction back into the residual stream at generation time and the model does more of the behavior; subtract it and the model does less. The part that matters for me is that they show the same recipe works across three R1 distills with different architectures, which suggests these directions are a stable property of how these models reason.

Results from both papers were measured on R1 ⁵ distills only, however, the distill learned to reason by imitating R1's text under next-token loss; but, R1 learned it through RL on outcomes. Whether those two signals build the same internal machinery is assumed in this literature. In the steering paper, ⁴ the three models differ in architecture but share a teacher. To separate those explanations you have to do the opposite where you hold the base model fixed and vary how it learned to reason, and it is what this project tries to work on.

Research question: For two 7B reasoning models with the same base (Qwen2.5-7B), one trained to reason only by RL on verifiable outcomes with no SFT (Open-Reasoner-Zero-7B), the other trained only by supervised fine-tuning on R1 reasoning traces with no RL (OpenThinker-7B). On MATH-500 ⁶:

Do the same kinds of steps carry the causal weight in both models?
Does a steering direction extracted from one model induce the same behavior in the other?

I call the two models mechanistically similar if both answers are yes. If the traces look alike but either answer is no, distillation might have transferred the form of the reasoning without its organization. ⁷

#2. Setup

#2.1 The Model Pair

I needed two models with the same base and the same size, where one learned to reason only through RL and the other only through distillation. R1 and its distills would be a bad choice, since R1 is too big. DeepScaleR-style models isn't suitable either, because they apply RL on top of a distill. So, I used:

	Model	Trained from
RL only	`Open-Reasoner-Zero-7B`	Qwen2.5-7B base, vanilla PPO, rule-based rewards, no SFT
Distill only	`OpenThinker-7B`	Qwen2.5-7B-Instruct, SFT on 114k R1 traces

Note that, ORZ ⁸ starts from the base model and OpenThinker ⁹ from Instruct, so there is an instruction-tuning stage on one side only. Their CoT formats are also different: ORZ uses <think>/<answer>, OpenThinker uses <|begin_of_thought|>/<|begin_of_solution|>.

#2.2 My Budget

I ran everything on one L40S (48 GB) on Lightning AI, which bills per GPU-hour ($3.24/hour). I developed with the GPU switched off and only turned it on for runs. The total was about 10.80 GPU-hours for $35.00.

This limited budget constrained in the following ways:

Bogdan et al. [3], used 100 rollouts per sentence. I used 10.
Bogdan et al. [3], used ~10x more problems. I used 7.
The sentence labeler had to be Qwen2.5-32B-Instruct in AWQ 4-bit, because 32B in bf16 doesn't fit on 48 GB next to anything else.

#2.3 Hypotheses

Two models share a reasoning mechanism if:

(a) Causal load: the same sentence categories determine whether the answer is correct (Experiment 1).
(b) Representation: a steering vector for a category in one model induces that category in the other (Experiment 2).

P0 (premise): Both models produce the same sentence categories. If not, a mechanistic difference should be expected.
H0: Both (a) and (b) both hold.
H1: After removing importance values below the noise floor (|Δacc| ≥ 0.2), the categories differ in one of two ways. Either a different category carries the most importance, or a category that appears in both models carries importance in only one.
H2: An OpenThinker vector that steers OpenThinker coherently has no coherent effect on ORZ beyond the unsteered baseline and a random vector of the same norm.

*With 7 problems and 10 rollouts per boundary, H0 can be rejected but not confirmed. Because a null result cannot separate shared mechanisms from estimator noise.

#3. Do Both Models Depend on the Same Kinds of Steps? (Experiment 1)

#3.1 Method

I adopted the resampling procedure from Bogdan et al. [3], with two simplifications for compute. Take a trace the model got right and split it into sentences $s_1, \dots, s_T$ . For each position $i$ , keep the prefix $s_1 \dots s_i$ , throw away everything after, and let the model regenerate the rest $n$ times. Let $\mathrm{acc}_i$ be the fraction of those $n$ continuations that land on the correct answer. Then

\mathrm{imp}_i = \mathrm{acc}_{i+1} - \mathrm{acc}_i

is how much fixing sentence $i+1$ moved the odds of being right. A big jump means that sentence was load-bearing. This is causal in the interventional sense, which is the point of the method.

Bogdan et al. [3] measures importance as a KL divergence between answer distributions, and they filter to resampled sentences that are semantically dissimilar to the original (their counterfactual importance). I use the accuracy difference, which their code also supports and which is the natural quantity for binary-graded math, and I skip the dissimilarity filter, which needs an embedding model I didn't have room for on the GPU. The likely effect is that my estimates are slightly conservative, since paraphrase rollouts dilute the signal.

The aggregation is novel where I labelled each sentence into one of six categories taken from [4] and computed the share of importance per category:

P(c) = \frac{\sum_{i : \mathrm{cat}(s_i) = c} |\mathrm{imp}_i|}{\sum_i |\mathrm{imp}_i|}

#3.2 The Noise Problem

With $n = 10$ , $\mathrm{acc}_i$ is quantized to tenths and a single flipped rollout moves it by 0.1. My first run made this painfully obvious. For a 40-sentence ORZ trace, sentence 0 was a real anchor ( $\mathrm{acc}_0 = 0.5 \to 1.0$ ) and everything after it was $\pm 0.1$ jitter. But there were 24 deduction sentences and 15 adding-knowledge sentences, and their jitter summed into category shares of 0.23 and 0.38 that meant nothing.

The fix is to threshold before aggregating, $|\mathrm{imp}_i| > 0.2$ , i.e. two rollouts' worth, and then average per-problem profiles with equal weight so a 119-sentence trace doesn't outvote a 33-sentence one. The cost is that several problems end up with no suprathreshold sentence at all and contribute a zero profile, which is why the bars in Figure 1 don't sum to 1.

Share of thresholded causal importance by sentence category for ORZ and OpenThinker. — Figure 1: Share of thresholded causal importance by sentence category, 7 problems per model. Bars are means; the error bars are $\pm 1$ std and should be read as " $n$ is small," not as confidence intervals.

#4. Does a Steering Direction Transfer? (Experiment 2)

#4.1 Method

For a behavior $c$ and layer $\ell$ , the steering vector is a difference of means over residual-stream activations $h^{(\ell)}_t$ :

v_c^{(\ell)} = \mathbb{E}_{t \in c}\left[h^{(\ell)}_t\right] - \mathbb{E}_{t \in \text{all reasoning tokens}}\left[h^{(\ell)}_t\right]

To steer, add it at every position during generation,

h^{(\ell)}_t \leftarrow h^{(\ell)}_t + \alpha \, v_c^{(\ell)}, \quad \ell \in \{12, \dots, 18\}

and measure the fraction of output sentences the labeler tags as category $c$ , as a function of $\alpha$ . I extracted from OpenThinker only, whereas ORZ has no backtracking or uncertainty sentences to extract from.

#4.2 Within-Model Control

A negative transfer result is only worth reporting if the vector demonstrably works on its own model, since steering is a causal probe and a dead probe fails everywhere. So the first thing I checked was within-model. i.e. does this vector do anything to the model it was extracted from? I add the vector across 7 layers, so the coefficient compounds much faster than a single-layer injection would suggest. At

\alpha = 1

the model already looped, and by

\alpha = 8

it was producing repeated Chinese gibberish.

Coefficient $\alpha$	Backtracking fraction	Uncertainty fraction
0	0.040	0.043
0.2	0.097	0.087
0.4	0.039	0.211

The two behaviors respond differently, and only one of them is monotonic. Observe that uncertainty rises with $\alpha$ . Backtracking doubles at $\alpha = 0.2$ and then falls back at $0.4$ , which looks like a failure of the vector until you read the generations. At the higher coefficient the model collapses into doubt loops, and the labeler scores those as uncertainty rather than backtracking. That gives me what I needed from this experiment.

#4.3 Transfer to ORZ

Steered-category sentence fraction versus steering coefficient for OpenThinker, ORZ, and a random-vector control. — Figure 2: Steered-category sentence fraction vs. coefficient $\alpha$ . Blue: OpenThinker's own vector on OpenThinker. Orange: OpenThinker's vector applied to ORZ. Green: a random vector of matched norm applied to ORZ.

\alpha = 0.2

, where OpenThinker responds cleanly, ORZ does nothing. Zero. At

\alpha = 0.4

ORZ starts backtracking (0.23), and at

\alpha = 0.8

it is at 0.67, but by then the text is fraying and the labeler is counting vocabulary, not reasoning. The random control is flat at zero everywhere, for both behaviors. So whatever ORZ is doing at 0.4 and 0.8 is specific to the backtracking direction, not generic degradation.

#5. Results

P0 fails. If the two models reasoned the same way, the traces themselves would look alike, and they don't. Across the 210 labeled units per model:

Model	Backtracking units	Uncertainty units	Sentences per trace
ORZ	0	2	37–98
OpenThinker	22	11	116–1005

ORZ reasons linearly and almost never doubts itself. OpenThinker checks and revises constantly, at up to ten times the length.

H1 is supported. The causal weight sits in a different place in each model. ORZ's importance concentrates on setup and recall, at 0.26 for initializing and 0.18 for adding-knowledge, while OpenThinker's concentrates on deduction at 0.43. Deduction makes up half of ORZ's units, 105 of 210, but carries only 6% of its importance, which is indicative of the causal structure.

H2 is supported. The steering vectors work within the model they came from, and they do not cross over. At $\alpha = 0.2$ they roughly double OpenThinker's backtracking fraction (0.04 → 0.10) and its uncertainty fraction (0.04 → 0.09), and the output stays coherent. The same vectors do nothing to ORZ at that strength. ORZ only responds at $\alpha = 0.4$ – $0.8$ , by which point its output is already degrading. A random vector of the same norm has no effect at any strength, so what I am seeing is specific to the backtracking direction.

Therefore, both hypotheses hold and P0 fails, which means that the two models differ at every level we measured, starting with the surface.

#5.1 Inconclusive Findings

Why the vector fails to transfer. My bet is that it is either one of these papers:
- Non-aligned directions: the models encode backtracking along related but different directions, so pushing along one does little in the other.
- Same direction, different scale: ORZ's residual norms are larger, which makes a given $\alpha$ a weaker push there, while ORZ needs more of it.
The random control rules out nonspecific damage, but it says nothing about which of these is true. Re-running the sweep with $\alpha$ scaled by each layer's residual norm would decide it, but I didn not run that experiment.
How large the H1 differences. The pattern is clear but the exact shares are not reliable, because the standard deviations exceed the means. On 3 of 7 ORZ problems and 2 of 7 OpenThinker problems, no step passed the noise threshold at all. So, each importance unit takes the label of its first sentence, and OpenThinker's units run up to 33 sentences.
Where the backtracking direction comes. I extracted it from ORZ rather than from Qwen2.5-7B, so I cannot say whether it was already present in the base model. Nor can I say whether distillation is what taught OpenThinker to use it, and the fact that OpenThinker starts from the instruct model muddies that comparison further.

I expected the risk to be that distillation produces a hollow imitation of reasoning, but the data point is the other way around. At 7B, the RL model is the one missing the reflective behavior. Strangely, a direction capable of inducing that behavior is already present in ORZ, which fits a broader line of evidence that reasoning training amplifies what the base model already has rather than building new capabilities. So, steering vectors trained with an RL objective on a frozen base model can recover most of the gain from full RL fine-tuning ¹⁰, and base models already contain the reasoning behaviors that thinking models learn to deploy ¹¹.

#5.2 Next Steps

Run the norm-normalized steering sweep. This is the obvious one, and it decides between the two possible interpretation in 5.1 above.
Add Qwen2.5-7B-Instruct as a third condition. If OpenThinker's vectors transfer to Instruct about as well as they transfer to ORZ, the shared structure is inherited from the base rather than from R1.
Extract from ORZ anyway, using whatever handful of backtracking-adjacent sentences turn up across many traces, and test ORZ → OpenThinker. Transfer should be tested in both directions.
Raise the rollouts to $n = 20$ on the same 7 problems. Halving the noise floor probably buys more than adding problems does.
Add a second RL model, General-Reasoner-Qwen2.5-7B, which is also zero-RL from the Qwen2.5-7B base, to check that the "no backtracking" result isn't an artifact of ORZ's prompt or reward.

#5.3 Limitations

The numbers here rest on a small sample. Only seven problems with ten rollouts per sentence, and three eval problems for steering. Coherence was judged by reading the outputs rather than measured. The labeler is a 4-bit 32B model whose labels I hand-checked on a sample but not systematically. The base-versus-instruct and CoT-format confounds are unaddressed, and the 0.4 and 0.8 transfer numbers come from degraded text. I suggest that anyone reading this take every magnitude in this post as provisional, and the categorical claims such as the zeros, and the specificity relative to a random vector as the actual result.

#6. On Using AI

I used Claude throughout. For scoping the project down from something I'd have needed weeks for, writing most of the pipeline code via Claude Code, and arguing with me about interpretation. This write-up, however, is largely my interpretation of the experimental results.

#7. References

[1]
I didn't go past the third round at Anthropic Fellows Program.
[2]
See Neel Nanda's MATS 12.0
[3]
Bogdan, Paul C., Uzay Macar, Neel Nanda, and Arthur Conmy. "Thought Anchors: Which LLM Reasoning Steps Matter?." arXiv preprint arXiv:2506.19143 (2025).
[4]
Venhoff, Constantin, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. "Understanding reasoning in thinking language models via steering vectors." arXiv preprint arXiv:2506.18167 (2025).
[5]
Guo, Daya, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).
[6]
See MATH-500 dataset
[7]
The two outcomes aren't equally informative at this scale. With seven problems and ten rollouts per sentence, a yes on both could mean the mechanisms are shared or could mean the measurement is too coarse to see a difference; I can't distinguish those. A no is harder to get by accident. So going in, divergence is the result I can defend and agreement is the result I'd have to hedge.
[8]
Hu, Jingcheng, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. "Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model." Advances in Neural Information Processing Systems 38 (2026): 162239-162262.
[9]
OpenThoughts team. OpenThoughts: Data Recipes for Reasoning Models, 2025.
[10]
Sinii, Viacheslav, Nikita Balagansky, Gleb Gerasimov, Daniil Laptev, Yaroslav Aksenov, Vadim Kurochkin, Alexey Gorbatovski, Boris Shaposhnikov, and Daniil Gavrilov. "Small vectors, big effects: A mechanistic study of rl-induced reasoning via steering vectors." arXiv preprint arXiv:2509.06608 (2025).
[11]
Venhoff, Constantin, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. "Base models know how to reason, thinking models learn when." arXiv preprint arXiv:2510.07364 (2025).