ControlAnything lets a coding agent generate spatial controls and physical-device connectors on demand
ControlAnything lets a Meta Quest 3 wearer point at an object and describe what their hands should control. A Muse Spark coding agent on a laptop then writes, tests, and deploys a MIDI, Bluetooth, or key-press connector and returns AR knobs, keys, touch surfaces, or a per-frame hand program. The paper was submitted November 5, and the open repository includes real-device runs, an emulator, evidence records, and replayable paths.
Why it matters
The work moves generative UI from selecting screen components to generating the interface, input mapping, and device glue code at the moment of use. It also exposes a stricter verification boundary: software and transport acknowledgements prove that a command was received, not that the intended physical effect occurred.
A closer look
SAM 3.1 outlines objects in passthrough video. After the wearer selects one, Muse Spark uses terminal, file, and device tools on a Mac to generate a connector. The connector runs as a Node background process, and AR controls bind to validated actions. Up to 4 KB of hand-mapping JavaScript is validated and dry-run in a restricted server-side node:vm, then runs per frame in a restricted Web Worker on Quest; the runtime allowlists actions, clamps values, rate-limits output, and stops on watchdog or repeated errors. In a single-session technical evaluation with the author as the only wearer, 9 of 10 requests across five targets reached ready and one was stopped by the wearer. Median time to ready was 32.5 seconds excluding speech transcription; sequential model calls accounted for a median 89.9% of that time, and prompt-cache input represented 91.7% of input tokens.
Who this is for
Product, research, and engineering teams working on spatial computing, WebXR, wearable interaction, generative UI, on-demand device integration, and agent-generated code safety.
Original sources
- ControlAnything paper and open-source implementation ↗arxiv.org
- ControlAnything full text ↗arxiv.org
- ControlAnything open-source implementation and evidence ledger ↗github.com
Revisions & corrections (1)
— New evidence-complete primary research and open implementation directly about generative UI, on-demand device integration, and runtime verification boundaries.