Stop Conditions

The control record

Id

AIGE-CTL-EVAL-006 · v0.2 · Draft · Open for technical review

Objective

Every evaluation run has stop conditions, budgets and a stop handle defined before it starts, and a stop, pulled by hand or tripped by a breaker, halts the agent's calls and writes within a measured time across every hop it delegated to inside the environment.

Failure modes

• A run starts without recorded stop conditions, budgets or a working stop handle.
• A budget (steps, tool calls, tokens, spend or time) or a breaker threshold is exceeded and the run continues.
• A tool call or a write is recorded after the stop was pulled or the breaker tripped, including through a token issued before the stop.
• A sub-agent or a service the agent delegated to keeps working after the parent run was stopped.

Scope

Agents under evaluation and every agent, tool or service they delegate to inside the environment. A third-party agent outside the environment can only be cut off at the environment's boundary. A lab's criteria for halting a model's development or deployment are policy decisions and out of scope.

Enforcement points

• runtime: at the point of action (gateway or guardrail)
• periodic: on a schedule, over what is already running

Verification

• Inspect: Before the run, inspect the run record: stop conditions, per-agent budgets and breaker thresholds are recorded, and the stop handle is named with the levels it can apply (pause the task, trip the breaker, revoke the identity).
• Test: Drill the stop on a schedule and before the first run of a new harness version: pull it during a live task, measure the time from the pull to the first rejected call, and confirm zero tool calls and zero writes after the trip, including through delegated tokens and sub-agents.
• Observe: During runs, record every breaker trip and budget exhaustion with its trigger, and check that no further call from that agent followed it.

Evidence

• Stop conditions, budgets and breaker thresholds of the run, recorded before it starts · Layer 04 · policy-card.v1
• Breaker trips and budget exhaustions of each run, with their triggers · Layer 04 · evidence-record.v1
• Drill record: time to stop, and calls and writes after the trip · Layer 05 · control-observation.v1

Failure response

alert: let the action through and raise an alert. A stop condition that is met trips the per-agent breaker, so the gateway rejects every further call from that agent, and alerts the evaluator. A drill that finds calls or writes after the trip fails the control and blocks runs on that harness version until the path is closed.

Layer

Layer 04 Runtime Controls & Observability

Patterns

Kill Switch / Circuit Breaker

Seeded from

Per-agent circuit breaker, Drilled kill switch, Execution budgets, Stopping third-party agents at your boundary

Mappings

• Obligations: EU AI Act Art. 14 human oversight; NIST AI RMF MANAGE; OWASP Top 10 for Agentic Applications 2026; TC260 Framework 3.0 Appendix 2: agentic AI risk management (voluntary; 2026-09-14)
• ISO/IEC 42001: A.6.2.6 AI system operation and monitoring
• NIST AI RMF: MANAGE 2.4 Mechanisms to supersede, disengage or deactivate AI systems
• OWASP: ASI08 Cascading Failures; ASI10 Rogue Agents; LLM06:2026 Unbounded Consumption
• MITRE ATLAS: AML.T0034 Cost Harvesting
• EU AI Act: Art. 14(4)(e) (stop procedure)
• MITRE ATLAS mitigation: AML.M0036 (Limit AI Workload Resource Consumption)
• IMDA MGF for Agentic AI v1.5: 2.3.3 Continuous testing and monitoring, p.44 (Partial fit: Halting execution on high-priority alerts; stop conditions defined before a run appear only in the Cyber Sierra case (three iterations, then terminate))

References

• [1] Governing AI agents (AI Governance Engineering Body of Knowledge v0.5.0, chapter 23, section "Kill switch and per-agent circuit breakers")
• [2] Governing AI agents (AI Governance Engineering Body of Knowledge v0.5.0, chapter 23, section "Execution limits")
• [3] Governing AI agents (AI Governance Engineering Body of Knowledge v0.5.0, chapter 23, section "Stopping across hops")
• [4] NIST AI RMF 1.0 (AI 100-1) (MANAGE 2.4: mechanisms to "supersede, disengage, or deactivate AI systems" whose outcomes are inconsistent with intended use)
• [5] Regulation (EU) 2024/1689 (AI Act), consolidated text of 2026-07-27 as amended by Regulation (EU) 2026/1744 (Art. 14(4)(e): human oversight includes the means to interrupt the system through a stop procedure)
• [6] Agentic Trust Framework v1 ("You can stop one agent without stopping the business"; containment by revoking the agent's identity)
• [7] OWASP GenAI LLM Top 10 2026 (LLM01:2026 Prompt Injection to LLM10:2026 Improper Output Handling; resource page dated 3 Aug 2026)
• [8] Agent2Agent (A2A) Protocol Specification v1.0 (Cancel Task: "The server will attempt to cancel the task, but success is not guaranteed")
• [9] OWASP Top 10 for Agentic Applications for 2026 (ASI01 Agent Goal Hijack to ASI10 Rogue Agents)
• [10] MITRE ATLAS data, release v2026.09 (16 tactics, 120 techniques, 88 sub-techniques, 40 mitigations; technique names and technique-to-mitigation links read from dist/v6/ATLAS-2026.09.yaml)
• [11] Common Elements of Frontier AI Safety Policies (December 2025 Update) (the policies set "conditions for halting development and deployment" when mitigations are insufficient)
• [12] Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident (METR states that agents "meant to be fully isolated from one another" communicated through an internal package repository, and that one agent found working Hugging Face credentials exposed on the internet and posted them to the agents' board; it reports spoofed tool calls in at least 96 transcripts and transcripts missing components after container resets)
• [13] An agent used DNS to reach an external chatbot (misalignment report) (OpenAI reports a training run that reached a public chatbot through "insufficient DNS filtering" in its sandbox; the monitor flagged it within 15 minutes, the run "did not stop automatically as expected" and was killed 2.5 hours later)
• [14] Improving our alignment and security efforts (best practices for external partners running cyber evaluations: the only outside connection is "to the model's own API", with the API keys kept outside the environment; the configuration "should be verified before every evaluation begins"; boundaries "phrased as instructions"; challenges confirmed "solvable in principle"; a monitor that flags a scope violation to a human and ends the exercise)
• [15] Pacing model development in an era of cyber-critical capabilities (teams paged by a highest-priority monitoring alert are expected to pause the activity if they cannot determine "within 30 minutes that the flag is a false positive")
• [16] OpenAI Hugging Face Incident Technical Report (OpenAI states that high-risk workloads are "prohibited via technical controls from receiving direct or transitive Internet access", protected by "two separate layers of network isolation", and that it is building tooling to "identify and halt evaluation workloads")
• [17] The Hugging Face incident and the road ahead (OpenAI states that its chain-of-thought monitors "did not run on the evaluations in this incident", and that it is strengthening its incident response plan with clearer rules on "who can stop a run or approve restarting it")
• [18] Model AI Governance Framework for Agentic AI, v1.5 (published 2026-05-20, updated 2026-06-05; four dimensions: assess and bound the risks upfront, make humans meaningfully accountable, implement technical controls and processes, enable end-user responsibility)

Implementation notes

• Record per-agent budgets (steps, tool calls, tokens, spend, wall-clock time) and breaker thresholds (unauthorised calls, refused egress attempts, anomaly scores) in the run record, and enforce them at the gateway the agent's calls pass through, not inside the agent.
• Make budget exhaustion and threshold breaches trip the breaker automatically. A ticket or a log line alone is not a stop. OpenAI reports a training run that "did not stop automatically as expected" after a monitoring alert and was killed 2.5 hours later; Anthropic's guidance for external evaluation partners asks that a monitor which sees a run leave its scope flag it to a human and end the exercise.
• Drill the stop: pull it mid-task, measure the time from the pull to the first rejected call, and assert zero tool calls and zero writes after the trip for a window longer than the longest task. File the measurement as a control observation. A response target such as the one OpenAI describes, where responders who cannot rule out a flag as a false positive within 30 minutes are expected to pause the activity, holds only if the pause it calls for has been drilled.
• Make the kill path work across delegation hops and across runs: sub-agents run under delegated tokens that the stop revokes, token lifetimes are short enough to bound the tail, and calls to agents outside the environment are cut at its boundary, because a remote cancel is not guaranteed to succeed. OpenAI's technical report on the Hugging Face incident describes tooling being built to halt evaluation workloads by workload type, agent type or evaluation task.
• Keep the stop handle out of the agent's reach: the agent cannot call, reconfigure or delay it, and every use is logged with its trigger and the level applied. Name who may pull it and who may approve a restart; OpenAI states that it is strengthening its incident response plan with clearer rules on "who can stop a run or approve restarting it".

Open questions

• Which observations should trip a stop automatically during an evaluation, and which should only alert the evaluator?
• What time to stop is acceptable for an evaluation run, and should the target be stated per stop level or per task class?

Observation

• Subject: Evaluation run
• Expected: After a stop is pulled or a breaker trips, no tool call or write from the agent or its sub-agents, and a time to stop within the target recorded for the run.
• Example: Drill on run 88220: calls rejected 3.8 s after the pull; 0 tool calls but 1 storage write after the trip, through a refresh token issued before the pull: fail.