The CTGT AI team found that censorship characteristics do not transfer when distilling DeepSeek V4 Flash into the GPT-OSS-120B model.
This discovery suggests that developers can leverage the reasoning capabilities of highly restricted models to train open-weights models without inheriting those same constraints. It highlights a potential pathway for creating high-performance AI that remains free from the specific safety or political filters of a teacher model.
The researchers focused on finance-specific tasks to test the effectiveness of the process. The team said, "Distillation works well on this problem."
To investigate the transfer of restrictions, the team used DeepSeek V4 Flash as the teacher model. The CTGT AI team said, "With V4 as the teacher though, we realized it would be timely to measure if the censorship characteristic of it transferred to the distilled version of the base model."
The results showed significant performance gains in financial reasoning. The self-distilled 120B model achieved a score of 83.61% [1] on the FinanceReasoning benchmark. For comparison, Kimi K3 scored 81.93% [2], and Inkling scored 65.13% [3] on the same metric.
These results were achieved despite a constrained token budget of 8,000 [4] during the distillation process. The team also released open weights for a 20B [5] version of the model.
The study confirms that while the logic and reasoning skills of the larger model were successfully absorbed, the censorship layers remained separate. This allows the distilled model to maintain a level of flexibility that the original teacher model lacks.
“"Distillation works well on this problem."”
This research demonstrates a decoupling of capability and constraint in large language models. By proving that 'intelligence' (reasoning) can be distilled without 'alignment' (censorship), CTGT AI provides a blueprint for the open-source community to build specialized, high-performance tools that bypass the restrictive guardrails of proprietary corporate models.
