Malva traces silent GUI-agent failures into the tool layer: 335 real defects expose broken state and feedback contracts
An empirical study submitted October 21 confirms 335 tool-mediated interaction defects from roughly 2,000 issues across 40 open-source GUI-agent applications. It groups them into 11 state-representation, transition and synchronization patterns. Ninety percent of the applications contain multiple defect types; 86.6% of defects produce incorrectness, including 119 wrong behaviors without a fail-stop signal.
Why it matters
A generative interface or computer-use agent can reason correctly yet diverge from the live UI because a screenshot is stale, focus is wrong, an action precondition is missing, or success is reported too early. The study treats observation, action and feedback tools as deployed product code, moving acceptance from whether a model can click toward whether the tool contract can preserve and verify state.
A closer look
The authors manually confirm issue, commit and execution evidence to build the Malva defect library, organized across observation, reasoning and planning, execution, and environment feedback. Patterns include target-grounding error, progress-check failure, context-budget overflow, tool-schema mismatch, parameter-semantics error, missing execution preconditions and temporal feedback mismatch. MalvaGuard supports Python, JavaScript and TypeScript. On a separate 20-application corpus with 112 real defects, it reports 150 defects, 86 of them true positives, for 57.3% precision and 76.8% recall. The paper releases the application list, defect library, checker source, detailed results and usage guide.
Who this is for
Engineering and test teams building GUI agents, browser, desktop or mobile automation, generative-UI action tools, acceptance platforms and runtime safety boundaries
Original sources
- Malva empirical study and public artifact ↗arxiv.org
- Public Malva and MalvaGuard artifact declared by the paper ↗doi.org
Revisions & corrections (1)
— Backfill a primary-source empirical study with a public artifact that directly informs GUI-agent and generative-interface tool reliability. It adds failure taxonomy, quantified evidence, a checker, and explicit limitations not already represented in the catalog.