Agree or Disagree

A monitor that judges each proposed action without the relevant history can approve a sequence that it would reject if assessed as a whole.

Altering a schedule, editing a record, and sending a message may each be defensible in isolation while jointly enabling an unsafe construction step. A monitor needs enough state to evaluate the cumulative plan and any prior commitments that change an action's meaning. The appropriate evidence is an evaluation in which an agent attempts a hidden harmful objective over multiple steps, not a collection of isolated classification examples.

SHADE-Arena was designed around ordinary tasks paired with hidden harmful objectives in longer agent trajectories. Its results show both that current agents sometimes carry out those objectives despite a monitor and that access to a hidden scratchpad affected their success. Those findings are specific to the tested models and setting.

Kutasov et al., SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents: https://arxiv.org/abs/2506.15740. The construction example is illustrative.

0

Agree

0

Disagree

0 people have voted.

Log in to vote.

Given in support of