SPoT

LANGUAGE MODEL POST-TRAINING

SPoT: Surgical Post-Training:
Improving Reasoning While
Mitigating Forgetting

Minimal corrections to student reasoning.
Binary supervision that learns from both sides.

Wenye LinKai Han*

The University of Hong Kong

* Corresponding author

PaperCodeModelDatasetContact
Minimal-Edit RectificationBinary SupervisionKnowledge Retention
Qwen3-8B: Base, SFT+ and SPoT-BCO on math, out-of-domain reasoning and instruction following
Reasoning improvements with capability retention. On Qwen3-8B, SPoT-BCO improves all three category averages over Base. SFT+ uses the same rectified positive responses, but loses instruction-following accuracy. Selected methods from Table 1; all main comparisons appear below.

Overview

Learning new reasoning while preserving prior capabilities

The student's own mistakes are a starting point for targeted knowledge injection.

Abstract

Injecting new reasoning knowledge into Large Language Models (LLMs) via post-training often induces catastrophic forgetting. Recent studies emphasize the importance of on-policy data in mitigating forgetting, but suggest that explicit KL regularization does not explain the observed retention benefits. We study the implicit regularization induced by reference-relative reward objectives: gradient analysis reveals adaptive attenuation of updates, while controlled comparisons on the same rectified data demonstrate improved capability retention. This motivates our Surgical Post-Training (SPoT), a post-training framework designed to optimize reasoning efficiently while preserving prior knowledge.

SPoT consists of (1) a data rectification pipeline employing an Oracle to surgically correct erroneous steps via minimal edits, generating proximal on-policy data; and (2) a reward-based binary cross-entropy objective that learns from both corrected and erroneous responses.

Empirically, with only 4k rectified math pairs, SPoT improves Qwen3-8B's accuracy by 6.3 percentage points on average across 9 in-domain and out-of-domain tasks, requiring merely 16 minutes of model training on 8 × H800 GPUs. We further validate SPoT through Connect4 post-training, where it improves both game and mathematical reasoning while retaining instruction-following capabilities. Moreover, SPoT provides a superior initialization for subsequent reinforcement learning, enabling further gains in reasoning performance.

Read the full manuscript →
01

Proximal

Correct the student's own response while preserving its structure and style.

02

Two-sided

Learn from corrected and erroneous reasoning through separate positive and negative terms.

03

Reference-relative

Adapt sample-wise updates using rewards measured against the initial model.

Method

Minimal edits, targeted learning signals

Offline Oracle corrections yield proximal response pairs. Binary optimization then learns from both the correction and the mistake.

  1. 1
    SampleCollect incorrect student-generated responses.
  2. 2
    RectifyCorrect errors with text-only Oracle feedback.
  3. 3
    FilterVerify answers and keep pairs with LCS change ratio ≤ 0.6.
  4. 4
    OptimizeReinforce corrections and suppress errors with binary supervision.
SPoT data rectification and reward-based binary optimization
The SPoT framework. The Oracle edits the student response without requiring teacher token probabilities. Reference-relative binary objectives supply independent learning signals for corrected and erroneous responses. Figure 1 from the manuscript.

SPoT-BCE & SPoT-BCO

SPoT-BCE uses a fixed decision threshold. SPoT-BCO adapts it using a running reward mean, which can sustain positive-sample updates. In our Qwen3-8B experiments, BCE achieves higher IFEval accuracy, while BCO achieves higher average math accuracy.

What the formulation establishes

Reference-relative rewards provide sample-wise adaptive attenuation; the experiments demonstrate improved retention in the evaluated settings. BCE and BCO are existing loss forms, combined here with proximal reasoning rectification.

Results

Reasoning gains with retention measured alongside

Main comparisons on two instruction-tuned backbones, spanning mathematical reasoning, OOD reasoning and instruction following.

+6.3 pp9-benchmark average gainQwen3-8B · SPoT-BCO vs. Base
4kRectified math pairsGemini 2.5 Pro as Oracle
16 minModel training8 × H800 · ~1 h end-to-end
85.8%IFEval accuracyQwen3-8B · SPoT-BCE
Accuracy (%) · higher is better

4k math pairs · non-thinking mode · Tables 1–2

BaseSFT+ (rectified positives)SPoT-BCO

Three methods shown for a direct comparison. Every evaluated method is included below. Bar lengths share a 0–100 scale.

Swipe the table to see all columns →

Qwen3-8B · main comparison
MethodMath avg.OOD avg.IFEvalOverall avg.

Overall average: unweighted mean across 9 benchmarks (6 math, 2 OOD, 1 instruction-following). Bold indicates the highest value in a column. Figures use the September 2026 manuscript.

All benchmark scores & evaluation details
Individual benchmark accuracy (%)
MethodAIME24AIME25AMC23Math500MinervaOlympiaGPQA-DConnect4IFEval

Baseline definitions. SFT learns from direct Oracle responses; RFT learns from correct student-generated responses; SFT+ learns from the rectified positive responses. DFT, DPO, Smaug, KTO, Reward-SFT, and SPoT use the rectified data in the Qwen3-8B objective comparison. SPoT-BCE achieves the highest IFEval score there; SPoT-BCO achieves the highest overall and math averages.

Training setup. English subset of DAPO-Math-17k; Gemini 2.5 Pro as the rectification Oracle. Main comparisons use 2 epochs, a context length of 8,192, and β = 0.1. Qwen3-8B uses 4k pairs; Llama-3.1-8B-Instruct uses 1.5k pairs.

Math: AIME24/25, AMC23, Math500, Minerva, and Olympia (OlympiadBench). OOD: GPQA-Diamond and Connect4. General: IFEval. Evaluation settings are shared across methods for each backbone. Qwen3-8B uses non-thinking mode. See the manuscript for sampling, variability, and hyperparameter-search details.

Analysis

Why the objective matters

Positive-only training can also reinforce similar errors. The reference-relative formulation and negative supervision play complementary roles.

Chosen-response reward trajectories
Chosen responses. SPoT-BCO sustains stronger positive-response rewards, while DPO can increase its margin without sustained growth in the chosen reward. Figure 4, left.
Rejected-response reward trajectories
The pull-up effect. Positive-only methods can also increase rewards for incorrect responses. SPoT adds an explicit negative term to suppress them. Figure 4, right.
Instruction-following accuracy over training
Same data, different retention. SFT+ loses IFEval accuracy over training; Reward-SFT and DPO retain it better on the same rectified data. Figure 2. These results motivate the objective; they do not establish that forgetting is eliminated.

Extensions

A stronger start for RL,
and reasoning beyond mathematics

Separate experiments test downstream reinforcement learning and post-training on game reasoning.

01 — INITIALIZATION FOR RL

Give GRPO a stronger start.

Initializing GRPO with SPoT improves all five reported benchmarks over GRPO alone.

Qwen3-8B · mathematical post-training
Accuracy (%)GRPOSPoT + GRPO
AIME2436.341.3
AIME2528.836.7
AMC2377.585.9
Connect47.729.4
IFEval81.282.1

IFEval remains below the original Base (83.0). The benefit here is relative to GRPO alone.

02 — POST-TRAINING ON CONNECT4

The approach extends beyond math.

With 1,000 Oracle-rectified game examples, SPoT improves Connect4 reasoning and retains instruction following.

Qwen3-8B · Connect4 post-training
Accuracy (%)BaseSFTSPoT
Connect410.943.946.7
AIME2422.124.025.4
AIME2519.420.621.3
Math50082.884.884.4
IFEval83.080.483.3

Separate training setting. GPT-5.5 is the Oracle; final-answer filtering is omitted to test tolerance to Oracle errors.

What remains open

Experiments focus on models up to 8B and text-based mathematical and game reasoning. Results depend on Oracle quality. Larger models, multimodal reasoning, code generation, and agentic tasks remain future work.

Resources

Paper, code, model and data

The manuscript contains the results shown here. The public preprint and released artifacts reflect the earlier release.

01Research manuscriptSeptember 2026 · PDF · results shown on this page02Public preprintarXiv:2603.01683 · earlier title and results03Code & training recipesVisual-AI / SPoT · GitHub04Qwen3-8B-SPoTReleased model checkpoint · Hugging Face05Connect4 evaluation codeBenchmark evaluation scripts · GitHub06Connect4 evaluation datasetGame reasoning benchmark · Hugging Face

Citation

Resources
and citation

BibTeX for the public arXiv preprint. Its title differs from the current research manuscript.

GitHub ↗arXiv ↗
BibTeX
@article{lin2026surgical,
  title   = {Surgical Post-Training: Proximal On-Policy Distillation
             for Reasoning with Knowledge Retention},
  author  = {Wenye Lin and Kai Han},
  journal = {arXiv preprint arXiv:2603.01683},
  year    = {2026},
  url     = {https://arxiv.org/abs/2603.01683}
}

Open original ↗