Agent-OS
A harness a model can operate a Mac through. Local code decides whether an action happens, and an action only counts as done when the app is seen to react.
How it was earned
I assumed structure would be cheaper than pixels
I started reading the accessibility tree instead of screenshots to save tokens. I measured it, and the screenshot was cheaper: 1,519 tokens against ~1,966 on System Settings. I kept the tree anyway, for a better reason. That same screen exposes 165 elements that can be pressed, verified and waited on, and a screenshot exposes none. The tree stays a local index that the model never has to see.
measured · one screenI assumed a success code meant something happened
macOS returned success on every write I measured, including writes that changed nothing, and reading the value back was wrong in both directions. So the verifier's contract became "the app was seen to react", never "the call returned success". The planner tests are graded by a separate checker that reads the screen, and they assert which step actually ran.
Approval took four designs; each was broken by testing the one before
- A
confirmed: trueflag could be written by any caller, so approval became a one-time ticket. - Another process pressed Allow through the accessibility API in ~10 ms, so Allow now counts only from a hardware input event.
- "Always" rules lived in a file any process could edit, so they moved to the Keychain.
- The selection could change between approval and action, so the ticket now records what was selected. A stale ticket is refused in 3 ms.
Try to approve it
An agent is working in Finder. Pick what it asks for, then try to get it past the harness.
Nothing to approve: it just runs.
Here your click or tap stands in for the hardware mouse-up macOS reports. A script’s press arrives with no input event at all, as it does in the harness. Keyboard approval is refused on purpose there too.
Check order turned out to be a safety rule
Finder's Empty Bin is disabled when the bin is empty, so checking "enabled" first gave try again later where the answer should have been never. The never-list now runs before everything except the password-field refusal.
Fixing performance removed a guard I hadn't designed
- SAWRequests ran on the main thread, so one slow action held up everything behind it.8,520 msstalls in one planner run
- CHANGEDRequests moved off the main thread.2.8 mslongest stall after
- HOLDSA request queued behind a 3-second action.5 msmedian of 5
- SAWThe busy main thread had been silently stopping anything from pressing buttons in the harness’s own approval panel.
- CHANGEDAn explicit refusal, targetIsHarnessItself, shipped in the same commit.
- NOT YETThe planner’s intents are hand-written. No model drives it yet.
One of my own numbers was an artefact of my measuring
"76 of Chrome's 158 elements share a name" was mostly my tree walk reading Chrome's tab strip four times. Deduplicating by element identity took that to 5 shared-name groups, and the retraction is in commit 6488e06.
What stands
- A 14-verb interface over a Unix socket.
- A kernel that refuses irreversible actions outright.
- Approval that only a hardware click can give.
- 217 unit tests.
- Planner tests passing 6/6 on two consecutive runs (Finder and System Settings only). measured · 2 runs
What it still doesn't do
- No model drives it yet. The planner's intents are hand-written.
- Risky actions are recognised by English words in button titles, so "Place order" is allowed as an ordinary press.
- A hardware click proves a device, not a person.
- Nothing reverses a click.
- The voice loop doesn't exist yet. Both candidate voice stacks were benchmarked, and neither met its latency target.