EnvOS audits computer-use agents when the world has changed
Submitted October 29, EnvOS archives interfaces, tools, state mutations, observation delivery and scoring evidence to test whether agents recheck facts after appointments, policies or other world state has changed instead of relying on stale memory or prompts.
Why it matters
Reliable generative interfaces cannot be judged from a final screenshot or action log alone. A host needs evidence that state changed, the task remained solvable, critical observations were delivered, and an independent auditor can reconstruct the run.
A closer look
The paper demonstrates phenomenon-based tasks in two synthetic workflows. In clinic booking, Opus 5 rechecked in 0/16 runs without extra information, 2/16 with a reminder and link, and 15/16 when the current value was shown directly; the paper-wide corrected result is p=0.00293. In insurance, Opus 5 rechecked 10/10 times; Sonnet 5 moved from 1/10 to 9/10 with current policy, but that contrast was not significant after paper-wide correction. EnvOS archives state mutation, solvability, observation delivery, the verifier and evidence for independent reconstruction.
Who this is for
Research and platform teams building computer-use evaluations, auditable generative interfaces, state recovery, compliance acceptance and regression testing
Original sources
Revisions & corrections (1)
— Publish a primary-source audit framework that links state mutation, observation delivery, terminal verification and independent reconstruction, while preserving its synthetic scope and adversarial counterevidence.