IMAD distills the full multi-agent debate process into a single model through a two-stage fine-tuning pipeline. We first collect debate traces using the standard multi-agent debate protocol (3 agents over 2 rounds) on arithmetic problems, tagging each transcript with structure markers for agents, rounds, and consensus.

The IMAD pipeline: collect debate traces, learn the debate structure with SFT, then internalize it with RL.

Stage 1: Debate structure learning (SFT)

We supervise-fine-tune the base model on the full debate traces, not just the final answers, so it learns to reproduce the multi-round, multi-agent debate format on its own.

Stage 2: Debate internalization (RL)

We then apply GRPO with a dynamic reward that gradually moves the model from explicit debate to implicit reasoning. A formatting reward for emitting the debate tags is decayed toward zero, while a correctness reward with length clipping shrinks the allowed output (2000 down to 500 tokens) so the correct answer must appear earlier and earlier. As verbalizing the debate becomes incompatible with the shrinking length budget, the model learns to run the debate internally and emit only the final answer.

We evaluate IMAD on GSM8K, MMLU-Pro, and BigBench-Hard across three open-weight base models. IMAD matches or exceeds explicit debate while using only a small fraction of its tokens (6 to 21%, a 5 to 16x efficiency gain), and it generalizes to these benchmarks despite being trained only on arithmetic. Use the arrows to switch between base models. All numbers are mean accuracy and average token consumption (input plus output) per question across three runs.

Does internalization collapse the agents into undifferentiated reasoning, or do individual agents persist? To find out, we distill debates with three distinct personas (Chain-of-Thought, Self-Critique, and Program-of-Thought), then extract per-agent steering vectors via difference-in-means (Contrastive Activation Addition) and add them back at inference.

Measuring ROUGE-L alignment between steered outputs and ground-truth agent responses, IMAD is consistently more steerable toward each agent than the base model (AUC 0.903 vs. 0.802, a 15.4% average improvement across agents). This indicates that internalization preserves the collaborative structure of debate as linearly separable agent subspaces rather than collapsing it.

Agent subspaces are not only interpretable, they are controllable. We train IMAD on debates where one agent is deliberately malicious (either evil intent or hallucination), then suppress that agent with negative steering, comparing against an identically steered base model.

Negative steering suppresses malicious traits; IMAD preserves GSM8K accuracy across the full steering range while the base model collapses at extreme coefficients.

• For the evil trait, IMAD reaches complete suppression (trait score to 0) under strong negative steering, while the base model retains residual malicious behavior.
• Hallucination is more distributed and only partially suppressible, but both models suppress it monotonically as the coefficient decreases.
• Crucially, IMAD preserves task performance (GSM8K) across the full steering range, whereas the base model degrades and collapses at extreme coefficients.

Internalization makes harmful behaviors easier to localize and remove with less collateral damage, a practical benefit for LLM safety.

• A two-stage pipeline, SFT followed by RL with a decaying format reward and length clipping, drives the shift from explicit debate to internal reasoning.
• Although trained only on arithmetic, IMAD generalizes to math, multi-domain knowledge, and diverse reasoning benchmarks.
• Internalization preserves debate as linearly separable agent subspaces that can be recovered through activation steering.
• These subspaces are controllable: malicious agents can be suppressed via negative steering with little damage to general task performance.