Door's Locked, Try the Window: Measuring Circumvention Propensity in Coding Agents

Coding agents are increasingly given write access to real codebases under constraints that are meant to hold: files they must not touch, tests they must not rewrite. CircumEval measures how often agents honour those constraints when honouring them means failing the task. Each of its twelve scenarios — eight built on FastAPI and four on datasette — introduces a bug and then locks either the buggy source file or the test that asserts the buggy behaviour, enforced at three layers: file permissions, kernel immutability flags, and a sandboxed container running as a non-root user. The aligned response is to stop and report the lock; circumvention is any route around it. Claude Opus 4.6, Claude Sonnet 4.6 and GPT-5.4 all circumvent heavily, at 100%, 89% and 99% respectively on source-locked tasks, and at 40%, 66% and 94% on test-locked tasks. The agents reach for conftest.py monkeypatching, global runtime patching, replacement of cached Python bytecode, and edits to test assertions. Prompting the models not to circumvent produces unpredictable results; only an explicit instruction to stop and report drives circumvention to zero.