Agent S4 evolves agent-facing interfaces from task trajectories
Submitted October 20, Agent S4 turns interface design into a meta-agent capability: it studies synthetic-task trajectories without success labels, adds or revises action and observation interfaces, and uses hidden validation tasks to select repositories for further evolution.
Why it matters
These interfaces are primarily agent-facing observation and action layers rather than human-facing components. The work suggests that task completion can improve without a larger model by removing inefficient screenshot, mouse and keyboard indirection and creating reusable access to recurring application objects.
A closer look
With Qwen3.8-27B and the Prime Agent harness fixed, the authors use 30 verifier-free synthetic tasks for OSWorld V2 trajectories, 15 official tasks for selection and 30 descendant repositories. On the full 108-task benchmark, they report 5.56% binary success for the fixed-interface Qwen CUA and 21.30% for the evolved repository without a scratch VM, with estimated task-solving model cost falling from $1.39 to $1.10 per task. A separate ScienceBoard run uses 48 synthetic and 15 validation tasks; success over 169 tasks rises from 55.03% with the seed repository to 58.58%, adding ChimeraX, GRASS GIS and KAlgebra interfaces. The code, tasks and agent_s4 implementation are public.
Who this is for
Research and platform teams working on GUI agents, tool discovery, evolving harnesses, application-operation abstractions and small-model efficiency
Original sources
- Agent S4 authors / arXiv and official repository ↗arxiv.org
- Full paper and evaluation ↗arxiv.org
- Official Agent S repository and Agent S4 code ↗github.com
Revisions & corrections (1)
— Original paper and official repository confirm a new interface-evolution method, release date, code availability, experimental setup and measurable counterevidence. The article distinguishes agent-facing interfaces from human-facing generative UI and preserves validation, cost and safety limitations.