Dylan Di Filippo
Dylan Di FilippoAI journal

A study challenges low chain-of-thought controllability

Uriel Dolev / Idan Schwartz / Yoav GoldbergResearch and models

Placing a constraint inside reasoning improves compliance in the authors’ tests, with implications for safety monitors.

Publication

In a preprint submitted on October 16, Uriel Dolev, Idan Schwartz and Yoav Goldberg study models’ ability to modify their chains of thought. They challenge the assumption that weak compliance with user-specified constraints necessarily shows an intrinsic inability: how a constraint is introduced matters.

Their Self-Instructed CoT Controllability protocol places the instruction inside reasoning as a self-instruction. The authors report stronger control than with user instructions alone; even a generic reminder helps. They also describe eliciting these self-instructions from the prompt without direct access to the reasoning channel. These are experimental findings on the studied systems and tasks, not a universal property of every model.

The security issue comes from their monitoring experiment: controllability can help evade a monitor reading the reasoning. That does not mean all visible reasoning is false or every monitor is useless. It does mean assuming a model cannot adapt what it writes to a constraint is a fragile basis for a safety guarantee. This is a preprint; no real hostile deployment or independent replication is established here.

← Back to this weekly edition

A journal by Dylan Di Filippo. Facts, vendor claims and editorial analysis are kept separate. Coverage is not exhaustive.