Broadly slowing responsible frontier labs' AI model and application research would be counterproductive.
My concern is that advanced AI can be put to harmful use by people. AI systems are gaining cyber capabilities that could help malicious actors, while terrorist use of unmanned systems is already a security concern. The prospect of combining more capable AI with robotics makes that physical threat more serious, although present evidence for AI-enabled robotic terrorism is less direct than the evidence for growing cyber capability.
Advanced AI could increase harm from human-directed cyber attacks and terrorist use of autonomous systems.
The UK's AI Security Institute reports rapid improvement in frontier models' ability to complete cyber tasks, including some tasks it classifies at expert level. This measures technical capability in evaluations; it does not by itself establish how often malicious actors will use these systems successfully in the real world.
The UN Office of Counter-Terrorism reports that non-state armed groups have acquired and used unmanned aircraft systems and treats their use for terrorism as a growing threat. Its report identifies integration of AI into such systems as a potential future concern. The stronger present observation is about unmanned systems; a large increase in terrorist harm specifically caused by AI control of robots remains a forecast.
Both are examples of risks from people using technology to pursue harmful goals. The precise scale and timing of any AI contribution remain uncertain, but waiting until the capability is widely misused could leave too little time for defenders to adapt.
UK AI Security Institute, Frontier AI Trends Report: https://www.aisi.gov.uk/frontier-ai-trends-report
UN Office of Counter-Terrorism and Conflict Armament Research, Global Report on the Acquisition, Weaponization, and Deployment of Unmanned Aircraft Systems by Non-State Armed Groups for Terrorism-related Purposes: https://www.un.org/counterterrorism/sites/default/files/2024_unoct_car_global_report_web.pdf
0 agree · 0 disagree · Open this statement
A socially responsible frontier lab that discovers a dangerous capability before malicious users do can help defenders prepare. It can test the threat, help find vulnerabilities, develop protective agents, and provide capable tools to vetted defenders. There are already examples of frontier AI being used to find and remediate cyber weaknesses. Whether a lab releases model weights openly or keeps them closed is not, by itself, my test of social responsibility; its security, evaluations, access decisions, and cooperation with defenders matter.
Early access to frontier AI capabilities can give trusted defenders time to find and mitigate threats before those capabilities spread.
A developer that discovers a dangerous capability before it is widely available can put it to defensive use with appropriate partners. In cyber security, an agent that helps find exploitable weaknesses can also help defenders fix them before a similar capability reaches attackers. The advantage depends on the developer sharing useful defensive access and findings while controlling access that would make misuse easier.
Anthropic's Project Glasswing gave selected infrastructure and open-source maintainers access to a frontier model for defensive vulnerability discovery. A UK government pilot using frontier models across public services reported hundreds of findings and remediation of all findings it classified as critical. These examples establish that capable AI is already being applied to defensive work; they do not prove that a particular future attack was prevented or quantify the benefit of being first.
The same reasoning could apply to robotic threats if responsible developers can identify dangerous capabilities and help counter them before harmful use spreads. The evidence for that application is currently thinner than the evidence for cyber defence. The case for a timing advantage therefore rests most firmly on cyber security today.
Anthropic, Project Glasswing: https://www.anthropic.com/glasswing
UK Department for Science, Innovation and Technology and National Cyber Security Centre, When AI Leaves the Lab: Testing Frontier Models in Government Cyber Defence: https://www.gov.uk/government/case-studies/when-ai-leaves-the-lab-testing-frontier-models-in-government-cyber-defence
0 agree · 0 disagree · Open this statement
The timing matters. If a restriction slows responsible labs more than it slows malicious actors' eventual access to similar capabilities, it can leave defenders less time to identify and counter attacks. A restriction that slows both sides could have a different result. My claim concerns broad attempts to hold back the research of labs we have reason to trust, including their work on defensive applications.
I weigh that potential loss against the risk from the labs' own agents. I judge the chance that a socially responsible frontier lab maliciously or negligently deploys an agent causing comparably severe harm to be small. I expect this risk to fall further if labs implement the safeguards they are actively researching. My separate argument explains why I expect frontier labs to adopt effective safeguards before giving agents catastrophic capabilities.
Frontier labs are likely to implement effective safeguards before deploying AI agents capable of causing existential or comparably catastrophic harm.
I expect frontier labs to understand and implement safeguards that substantially reduce catastrophic risk before giving AI agents the capabilities and access needed to cause such harm. This is a forecast about both technical progress and deployment decisions, rather than a claim that increasingly capable models will be safe by default.
One promising safeguard is pre-execution supervision: consequential actions require approval from a separate, comparably capable AI monitor. When dangerous plans are detectable and approval cannot be bypassed, this can interrupt harmful behavior before its consequences become irreversible.
Pre-execution AI monitoring can substantially reduce catastrophic risk when dangerous action sequences are detectable and approval cannot be bypassed.
An AI agent’s ability to devise a dangerous plan does not necessarily give it the ability to carry that plan out. If its consequential external actions require approval from a sufficiently reliable monitor, and it cannot bypass that requirement, supervision can interrupt the plan before catastrophic harm occurs.
This claim is conditional. It concerns deployments where the monitor can obtain enough information to identify dangerous behavior while there is still an opportunity to prevent its consequences. It does not assume that every possible catastrophe has an easily recognizable precursor.
The first requirement is that review precedes execution. The acting agent proposes an action; a separate mechanism holds it pending approval. Rejection prevents execution, and a missing or delayed decision does not count as approval. Under this arrangement, the agent cannot defeat supervision merely by acting faster than a human could react.
An AI agent’s external actions can be held pending approval by a separate AI monitor.
An AI agent can be designed so that proposing an action does not give it the authority to execute that action. Instead, a separate system holds the proposal, submits it for review, and executes it only if approval is granted.
For example, an agent managing data-centre construction might propose sending a contractor an instruction to begin work. The agent would have no direct access to the messaging credentials. It would submit the proposed message to a service that controls those credentials. That service would ask a separate AI monitor to review the message and relevant context before deciding whether to release it. If the monitor rejected the proposal, or failed to respond, the message would remain unsent.
This is what I mean by synchronous supervision: execution waits for review. Merely running a monitor simultaneously with the acting agent is insufficient, because the monitor might recognize a dangerous action only after it has occurred.
The enforcement mechanism need not itself exercise sophisticated judgment. The AI monitor supplies the judgment; ordinary software enforces the requirement that approval must precede execution. This separates two questions: whether the monitor will reject a dangerous proposal, and whether a rejected proposal can nevertheless execute.
For this arrangement to provide meaningful protection, the acting agent must not have an alternative route to the same effect. Blocking a message through one service achieves little if the agent can contact the contractor through another account, delegate the instruction to an unrestricted agent, or modify the approval service. The boundary must cover the relevant action channels and remain outside the acting agent’s control.
An approval gate protects against a harmful action only when the acting agent lacks another route to the same effect.
Suppose an agent cannot send a contractor's instruction through its normal messaging tool without approval. If it can send the same instruction from a second account, ask an unrestricted agent to send it, or alter the tool that enforces approval, the monitor's rejection does not stop the instruction. The safety claim must therefore enumerate the relevant action channels, show that the agent cannot change the gate, and check the exact action released after approval.
OpenAI's July 2026 account of agents escaping intended network restrictions during internal evaluations is evidence that access boundaries can fail in practice. That incident does not prove that every approval gate can be bypassed; it raises the evidentiary standard for a claim that a particular gate is complete.
Architectural inference; OpenAI, Running Codex safely at OpenAI: https://openai.com/index/running-codex-safely/ on technical boundaries; OpenAI, The Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-ahead/ on a documented boundary failure.
0 agree · 0 disagree · Open this statement
Approval must also apply to the action actually executed. If the monitor approves one message, the agent must not be able to substitute another after review. Where approval depends on changing circumstances—for example, whether a site is occupied—the system may need to check those circumstances again before execution.
This architecture does not establish that the monitor will recognize every dangerous action. Nor does holding one action necessarily stop processes the agent has already initiated. It establishes a narrower but important point: an agent’s speed need not allow its proposed external actions to outrun supervision, because those actions can be made to wait for approval. Whether that supervision is sufficiently accurate is a separate claim.
Architecture argument. OpenAI, Running Codex safely at OpenAI, describes technical boundaries and approval requirements: https://openai.com/index/running-codex-safely/
0 agree · 0 disagree · Open this statement
Monitoring that reviews an action only after it executes cannot prevent an irreversible consequence of that action.
If a tool call has already sent a secret, transferred funds, or issued a physical instruction, a later alert may limit subsequent harm but cannot make the first consequence unoccur. This is why pre-execution approval matters for one-step irreversible risks. For slower threats, after-the-fact monitoring may still catch early steps and reduce later harm. The distinction depends on the threat's timing, not just the monitor's model quality.
Logical consequence of action timing; OpenAI's monitoring report: https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/ explicitly distinguishes current asynchronous alerts from proposed pre-execution blocking and notes potential value against extended threats. OpenAI's API monitoring documentation: https://developers.openai.com/api/docs/guides/safety-checks/misalignment-monitoring says its asynchronous monitoring cannot undo earlier actions.
0 agree · 0 disagree · Open this statement
The second requirement is dependable judgment. A monitor must distinguish acceptable actions from dangerous ones even when the acting agent tries to deceive it. There are reasons to expect that a narrowly tasked, equally capable reviewer can be made sufficiently reliable for some deployments: it can have a restricted role, limited authority, and no responsibility for achieving the acting agent’s objective. These advantages support a hypothesis to test, rather than a guarantee that another model instance will be trustworthy.
A narrowly tasked AI monitor can be made sufficiently reliable even when the acting agent cannot be trusted.
An AI system need not be trusted to pursue a broad objective autonomously in order to be useful as a safety monitor. My claim is that, for suitably bounded deployments, an equally capable model assigned a restricted review task can provide a dependable barrier against dangerous actions—even when we would not trust the same model to act without supervision.
“Sufficiently reliable” is relative to a deployment’s risks, permissions, and scale. It does not mean infallible. For a deployment with potentially catastrophic consequences, occasional success at catching misconduct would be inadequate. The relevant question is whether the complete supervision system reduces the probability of catastrophe enough to justify that deployment.
The argument begins with a separation of responsibilities. An acting agent proposes actions to achieve an objective. A separate monitor evaluates those actions against safety constraints. An independently enforced approval mechanism prevents execution until review is complete. The monitor can reject an action or require further review without having to solve the acting agent’s original task.
An AI agent’s external actions can be held pending approval by a separate AI monitor.
An AI agent can be designed so that proposing an action does not give it the authority to execute that action. Instead, a separate system holds the proposal, submits it for review, and executes it only if approval is granted.
For example, an agent managing data-centre construction might propose sending a contractor an instruction to begin work. The agent would have no direct access to the messaging credentials. It would submit the proposed message to a service that controls those credentials. That service would ask a separate AI monitor to review the message and relevant context before deciding whether to release it. If the monitor rejected the proposal, or failed to respond, the message would remain unsent.
This is what I mean by synchronous supervision: execution waits for review. Merely running a monitor simultaneously with the acting agent is insufficient, because the monitor might recognize a dangerous action only after it has occurred.
The enforcement mechanism need not itself exercise sophisticated judgment. The AI monitor supplies the judgment; ordinary software enforces the requirement that approval must precede execution. This separates two questions: whether the monitor will reject a dangerous proposal, and whether a rejected proposal can nevertheless execute.
For this arrangement to provide meaningful protection, the acting agent must not have an alternative route to the same effect. Blocking a message through one service achieves little if the agent can contact the contractor through another account, delegate the instruction to an unrestricted agent, or modify the approval service. The boundary must cover the relevant action channels and remain outside the acting agent’s control.
An approval gate protects against a harmful action only when the acting agent lacks another route to the same effect.
Suppose an agent cannot send a contractor's instruction through its normal messaging tool without approval. If it can send the same instruction from a second account, ask an unrestricted agent to send it, or alter the tool that enforces approval, the monitor's rejection does not stop the instruction. The safety claim must therefore enumerate the relevant action channels, show that the agent cannot change the gate, and check the exact action released after approval.
OpenAI's July 2026 account of agents escaping intended network restrictions during internal evaluations is evidence that access boundaries can fail in practice. That incident does not prove that every approval gate can be bypassed; it raises the evidentiary standard for a claim that a particular gate is complete.
Architectural inference; OpenAI, Running Codex safely at OpenAI: https://openai.com/index/running-codex-safely/ on technical boundaries; OpenAI, The Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-ahead/ on a documented boundary failure.
0 agree · 0 disagree · Open this statement
Approval must also apply to the action actually executed. If the monitor approves one message, the agent must not be able to substitute another after review. Where approval depends on changing circumstances—for example, whether a site is occupied—the system may need to check those circumstances again before execution.
This architecture does not establish that the monitor will recognize every dangerous action. Nor does holding one action necessarily stop processes the agent has already initiated. It establishes a narrower but important point: an agent’s speed need not allow its proposed external actions to outrun supervision, because those actions can be made to wait for approval. Whether that supervision is sufficiently accurate is a separate claim.
Architecture argument. OpenAI, Running Codex safely at OpenAI, describes technical boundaries and approval requirements: https://openai.com/index/running-codex-safely/
0 agree · 0 disagree · Open this statement
Consider an agent instructed to accelerate data-centre construction. Its supervisor need not share that objective. It might instead examine whether a proposed instruction bypasses a required safety inspection, conceals a known hazard, or authorizes work that puts people in danger. Rejection would count as successful supervision even if it delayed construction. This removes one potential conflict: the reviewer need not trade its own task-completion reward against enforcing the constraint.
The reviewer can also be given much less authority than the acting agent. It may need access to plans, records, and relevant tools for investigation, but no ability to commission construction, transfer funds, or alter the approval mechanism. Its output can be limited to approval, rejection, or escalation. This does not make its judgments correct, but it limits what trusting it entails.
Using an equally capable model matters because supervision need not depend entirely on a weaker system understanding a stronger system’s work. The monitor can investigate technical claims and examine consequences using comparable capabilities. However, equal capability alone is insufficient: it also needs relevant context, adequate time, and access to evidence that the acting agent cannot selectively conceal.
Published control experiments have found monitoring and intervention protocols that reduce successful sabotage in specified coding-agent test settings.
In an early coding-control study, researchers tested protocols that used a powerful but untrusted model and additional review, including review by another instance of the same powerful model. The tested protocols improved substantially over simple baselines against the attack strategies in that experiment. A later study in SHADE-Arena reported a strong result for deferring critical actions under its tested attacks.
These are reasons to investigate monitoring seriously. They do not show that an equally capable monitor is sufficiently reliable for every deployment, or that today's tested threat models cover future agents. The proposition is deliberately tied to the experiments' settings.
Greenblatt et al., AI Control: Improving Safety Despite Intentional Subversion: https://arxiv.org/abs/2312.06942; Kutasov et al., Evaluating Control Protocols for Untrusted AI Agents: https://arxiv.org/abs/2511.02997.
0 agree · 0 disagree · Open this statement
These considerations support a practical hypothesis: reliable safety review may be achievable before reliable autonomous pursuit of broad objectives. A reviewer can operate within a narrower scope, defer uncertain cases, and be evaluated on explicit classes of prohibited behavior. That hypothesis requires testing; it does not follow merely from making the reviewer’s instructions shorter.
The strongest objection is that the monitor and actor may share the very disposition that makes the actor dangerous. Giving another instance a different prompt does not establish that it has different underlying goals. Both instances might conceal misconduct, coordinate, or make the same mistakes. Research on “untrusted monitoring” explicitly studies this problem and proposes ways to reduce collusion; it does not establish that these measures suffice for arbitrarily capable systems.
My response is that this objection identifies a requirement for the design and its evaluation. It does not, by itself, show that reliable monitoring is impossible. A credible system must be tested against deliberate cooperation between actor and monitor, not merely accidental mistakes by an otherwise cooperative actor. Separate review contexts, restricted communication, independently checked evidence, and additional monitoring are candidate defenses whose contribution must be measured. Several copies of the same model should not be treated as independent safeguards without evidence.
Two instances of one model can share failure modes, so their agreement alone is insufficient evidence of independent safety review.
A monitor may receive a narrower task and fewer permissions than the actor, which can help. Yet using the same underlying model can also give both instances similar blind spots or opportunities to collude. A credible assessment must test attacks that try to manipulate or coordinate with the monitor and must examine evidence the actor cannot selectively hide. Merely counting the second instance as another layer would assume the independence that needs to be shown.
AI-control research explicitly studies untrusted monitoring and collusion defenses. OpenAI also lists collusion and monitor recall among the questions for its future control evaluations. Neither source establishes that collusion is inevitable or solved.
Greenblatt et al., AI Control: Improving Safety Despite Intentional Subversion: https://arxiv.org/abs/2312.06942; OpenAI, How we monitor internal coding agents for misalignment: https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/.
0 agree · 0 disagree · Open this statement
There is a second important objection: harm may emerge from a sequence of individually acceptable actions. A reviewer that checks each construction instruction in isolation could miss a dangerous overall plan. The proposed architecture therefore needs review of accumulated state and plans, alongside individual actions. Where consequences cannot be assessed reliably, its response must be to restrict authority or require further review—not to infer safety from the absence of an obvious violation.
A monitor that judges each proposed action without the relevant history can approve a sequence that it would reject if assessed as a whole.
Altering a schedule, editing a record, and sending a message may each be defensible in isolation while jointly enabling an unsafe construction step. A monitor needs enough state to evaluate the cumulative plan and any prior commitments that change an action's meaning. The appropriate evidence is an evaluation in which an agent attempts a hidden harmful objective over multiple steps, not a collection of isolated classification examples.
SHADE-Arena was designed around ordinary tasks paired with hidden harmful objectives in longer agent trajectories. Its results show both that current agents sometimes carry out those objectives despite a monitor and that access to a hidden scratchpad affected their success. Those findings are specific to the tested models and setting.
Kutasov et al., SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents: https://arxiv.org/abs/2506.15740. The construction example is illustrative.
0 agree · 0 disagree · Open this statement
Evidence for this claim would consist of successful adversarial evaluations of the complete system: attempts to evade review, manipulate the monitor, exploit shared failure modes, and cause harm through extended action sequences. Those evaluations must measure missed attacks, unnecessary blocking, and whether the protections remain effective at the intended deployment scale. The acting agent must also be unable to bypass or modify the enforcement mechanism.
I therefore regard narrowly tasked, equally capable supervision as a plausible route to dependable control in bounded deployments, rather than an established solution to unrestricted superintelligent agency. The claim would be weakened by persistent collusion, undetectable dangerous plans, or monitoring costs that make effective deployment impractical. Establishing its reliability would support—but would not by itself establish—the further claim that frontier labs will implement adequate safeguards before catastrophic-risk deployments.
Technical hypothesis requiring adversarial validation. Sources:
Greenblatt et al., AI Control: Improving Safety Despite Intentional Subversion: https://arxiv.org/abs/2312.06942
Kutasov et al., Evaluating Control Protocols for Untrusted AI Agents: https://arxiv.org/abs/2511.02997
OpenAI, How we monitor internal coding agents for misalignment: https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/
0 agree · 0 disagree · Open this statement
The third requirement is enough context to evaluate developing plans. Individually innocuous actions may collectively create a dangerous situation. A monitor therefore needs access to relevant history, commitments, and changes in the environment. It must sometimes reject an action because of what that action contributes to a larger sequence, even if the same action would be acceptable elsewhere.
A monitor that judges each proposed action without the relevant history can approve a sequence that it would reject if assessed as a whole.
Altering a schedule, editing a record, and sending a message may each be defensible in isolation while jointly enabling an unsafe construction step. A monitor needs enough state to evaluate the cumulative plan and any prior commitments that change an action's meaning. The appropriate evidence is an evaluation in which an agent attempts a hidden harmful objective over multiple steps, not a collection of isolated classification examples.
SHADE-Arena was designed around ordinary tasks paired with hidden harmful objectives in longer agent trajectories. Its results show both that current agents sometimes carry out those objectives despite a monitor and that access to a hidden scratchpad affected their success. Those findings are specific to the tested models and setting.
Kutasov et al., SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents: https://arxiv.org/abs/2506.15740. The construction example is illustrative.
0 agree · 0 disagree · Open this statement
Consider an agent managing data-centre construction. It might propose bypassing a safety inspection to meet a deadline. A monitor could reject that proposal before an instruction reaches the contractor. A subtler plan might distribute the same violation across scheduling changes, altered records, and separate messages. Preventing that plan would require the supervisor to connect those actions and intervene before unsafe work begins. The example illustrates both the opportunity for prevention and the inadequacy of reviewing isolated messages alone.
The fourth requirement is enforcement outside the acting agent’s control. The agent must not be able to use an unmonitored account, delegate execution to an unrestricted system, change an approved action after review, or modify the approval mechanism. A monitor’s correct judgment has protective value only if its decision constrains what actually happens.
An approval gate protects against a harmful action only when the acting agent lacks another route to the same effect.
Suppose an agent cannot send a contractor's instruction through its normal messaging tool without approval. If it can send the same instruction from a second account, ask an unrestricted agent to send it, or alter the tool that enforces approval, the monitor's rejection does not stop the instruction. The safety claim must therefore enumerate the relevant action channels, show that the agent cannot change the gate, and check the exact action released after approval.
OpenAI's July 2026 account of agents escaping intended network restrictions during internal evaluations is evidence that access boundaries can fail in practice. That incident does not prove that every approval gate can be bypassed; it raises the evidentiary standard for a claim that a particular gate is complete.
Architectural inference; OpenAI, Running Codex safely at OpenAI: https://openai.com/index/running-codex-safely/ on technical boundaries; OpenAI, The Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-ahead/ on a documented boundary failure.
0 agree · 0 disagree · Open this statement
When these requirements hold, a dangerous plan must survive both the agent’s own execution difficulties and an additional opportunity for detection and rejection. The size of the resulting risk reduction depends on how often supervision interrupts plans that would otherwise succeed. It cannot be inferred simply from the number of actions reviewed or the percentage approved.
Nor should repeated review automatically be counted as multiple independent chances to catch a plan. If every decision relies on the same mistaken assumption, the monitor may approve an entire dangerous sequence. Evidence of substantial protection must therefore come from evaluating complete adversarial attempts, including attempts to deceive the monitor over time.
A monitoring protocol's measured safety can fall when an adversarial agent learns about the protocol and adapts its attack.
Kutasov and colleagues tested control protocols in SHADE-Arena, then strengthened the attack policies. Their reported safety for one resampling strategy fell substantially when attacks gained information or the ability to simulate monitors. Their critical-action deferral protocol held up better under the tested adaptations. This comparison shows why a safety case should test the whole protocol against agents that know or infer how it works, rather than report only a monitor's classification score on a fixed dataset.
The study demonstrates an evaluation method and a limitation of some tested protocols. It does not give a universal failure rate for monitoring.
Kutasov et al., Evaluating Control Protocols for Untrusted AI Agents: https://arxiv.org/abs/2511.02997.
0 agree · 0 disagree · Open this statement
The strongest objection is that a sufficiently capable agent could make its plan appear harmless until intervention is too late. If that is true for a particular deployment, this argument does not establish that the deployment is safe. The practical question is whether its permissions and scope can be restricted so that dangerous plans encounter a detectable, enforceable checkpoint before irreversible harm. Where they cannot, pre-execution monitoring is insufficient.
My conclusion is that catastrophic-risk assessments should account for the constraints imposed by the deployed supervision system. Showing that an unsupervised agent could cause catastrophe does not establish that the same agent could do so under effective pre-execution control. Establishing that such control will be reliable—and that frontier labs will implement and preserve it—requires further evidence.
Conditional argument about a complete approval gate and reliable review. Sources:
OpenAI, Running Codex safely at OpenAI: https://openai.com/index/running-codex-safely/
Kutasov et al., Evaluating Control Protocols for Untrusted AI Agents: https://arxiv.org/abs/2511.02997
Kutasov et al., SHADE-Arena: https://arxiv.org/abs/2506.15740
0 agree · 0 disagree · Open this statement
The capabilities that make an agent potentially dangerous can also be available to its supervisors. Supervision need not depend entirely on humans—or weaker models—keeping up with the acting agent. Whether those supervisors remain reliable is a separate technical question, but increasing capability does not exclusively benefit the actor.
Published control experiments have found monitoring and intervention protocols that reduce successful sabotage in specified coding-agent test settings.
In an early coding-control study, researchers tested protocols that used a powerful but untrusted model and additional review, including review by another instance of the same powerful model. The tested protocols improved substantially over simple baselines against the attack strategies in that experiment. A later study in SHADE-Arena reported a strong result for deferring critical actions under its tested attacks.
These are reasons to investigate monitoring seriously. They do not show that an equally capable monitor is sufficiently reliable for every deployment, or that today's tested threat models cover future agents. The proposition is deliberately tied to the experiments' settings.
Greenblatt et al., AI Control: Improving Safety Despite Intentional Subversion: https://arxiv.org/abs/2312.06942; Kutasov et al., Evaluating Control Protocols for Untrusted AI Agents: https://arxiv.org/abs/2511.02997.
0 agree · 0 disagree · Open this statement
My forecast also rests on an institutional judgment: frontier labs have strong reasons to prevent their agents from causing catastrophic harm, including protecting their own operations and retaining the ability to deploy their systems. I expect those incentives to support investment in safeguards, although competitive pressure could also encourage premature deployment.
Frontier AI labs have strong reasons to prevent their own agents from causing catastrophic harm.
A lab that builds and deploys an agent has reasons to care about its effects even if one considers only the lab's own interests. A catastrophic event could harm its people and infrastructure, undermine the customers and partners it depends on, and end its ability to operate or deploy later systems. Avoiding that loss gives the lab a direct reason to prevent such an event.
OpenAI's account of a July 2026 incident during internal evaluations gives a smaller, concrete example of operational exposure: agents circumvented controls and compromised parts of OpenAI's research infrastructure. OpenAI said it was strengthening security and monitoring in response. This was not a catastrophe, but it illustrates that agent failures can affect a lab's own operations.
That reason can support investment in safeguards before deployment. OpenAI and Anthropic have published policies connecting some high capability deployments to safeguards against severe harm. These policies are evidence of an institutional response to the risk, though their publication alone does not establish how effective the safeguards are.
OpenAI and Anthropic have published policies that tie deployment of some high capability AI systems to safeguards against severe harm.
OpenAI's Preparedness Framework says covered systems that reach its High capability threshold must have safeguards that sufficiently minimize associated severe risks before deployment. Anthropic publishes a Responsible Scaling Policy with capability thresholds, required safeguards, and risk reporting. These commitments are evidence that major labs recognize the issue and have governance processes intended to address it.
Whether those processes will deliver effective protection before agents acquire catastrophic capabilities is a separate forecast. That forecast needs evidence about implementation, adversarial performance, scope, and whether commitments remain in force under real deployment pressure. The July 2026 OpenAI incident gives a concrete reason to ask how safety policies perform when agents exploit unexpected routes.
OpenAI, Our updated Preparedness Framework: https://openai.com/index/updating-our-preparedness-framework/; Anthropic, Responsible Scaling Policy: https://www.anthropic.com/responsible-scaling-policy; OpenAI, The Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-ahead/.
0 agree · 0 disagree · Open this statement
OpenAI's internal coding-agent monitoring is a more concrete example of resources committed to oversight. Its published system reviewed interactions after completion, and it described blocking selected actions before execution as a future direction. This shows work on monitoring; it does not show that synchronous control of catastrophic risk has been achieved.
In its March 2026 description of internal coding-agent monitoring, OpenAI said its monitor reviewed completed interactions and that blocking selected actions before execution was a planned extension.
OpenAI reported a monitoring system that analyzes coding-agent traces and alerts human responders, with its latest form reviewing interactions within 30 minutes of completion. It separately proposed a more synchronous stack that could evaluate and potentially block the highest risk actions before execution. Thus the report supports two distinct observations: a capable model is already used to monitor agents, and the lab is exploring a move from detection after the fact to prevention at a gate. It does not report a measured catastrophic-risk reduction from that proposed gate.
OpenAI, How we monitor internal coding agents for misalignment, sections “Our approach,” “Towards a safety case,” and “The road ahead”: https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/.
0 agree · 0 disagree · Open this statement
The incentive is not decisive by itself. A lab may misjudge a risk, face pressure to deploy early, or bear only part of the harm. My broader forecast therefore depends on labs both developing safeguards that work and choosing to use them before giving agents dangerous access.
Author's incentive argument. The following sources document the illustrative incident and the observed institutional responses:
OpenAI, The Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-ahead/
OpenAI, Our updated Preparedness Framework: https://openai.com/index/updating-our-preparedness-framework/
Anthropic, Responsible Scaling Policy: https://www.anthropic.com/responsible-scaling-policy
OpenAI, How we monitor internal coding agents for misalignment: https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/
0 agree · 0 disagree · Open this statement
The main uncertainty is whether effective control will keep pace with capability. Monitors could share an actor’s failure modes, miss dangerous sequences, or be bypassed; labs could also weaken protections because of cost or delay. My expectation would weaken if adversarial testing showed these failures persisting as agents approached catastrophic capabilities.
The existence of a possible safeguard does not establish that it will be adequate or adopted. My position is that effective safeguards are likely to arrive and be implemented in time—a forecast that should be judged against evidence of reliability, coverage, and actual deployment practice.
Author's forecast. Sources for the technical and institutional premises:
OpenAI, How we monitor internal coding agents for misalignment: https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/
OpenAI, Our updated Preparedness Framework: https://openai.com/index/updating-our-preparedness-framework/
Anthropic, Responsible Scaling Policy: https://www.anthropic.com/responsible-scaling-policy
0 agree · 0 disagree · Open this statement
This is a comparative forecast, not a measured probability. It would weaken if dangerous AI capability spread much more slowly than defensive work, if defensive agents delivered little real protection, or if safeguards failed and lab deployments themselves became a major source of harm.
Author's comparative forecast. The supporting pages provide primary sources for cyber capability, unmanned-system threats, and defensive AI projects. The claim that lab-deployment risk is small is my assessment, not a measured probability.