Agent-OS
Software built so an AI model can use a Mac safely: code on the Mac, not the model, decides whether an action happens, and an action only counts once the app visibly reacts.
How it was earned
I assumed structure would be cheaper than pixels
I started reading the accessibility tree instead of screenshots to save tokens. I measured it, and the screenshot was cheaper: 1,519 tokens against ~1,966 on System Settings. I kept the tree anyway, for a better reason. That same screen exposes 165 elements that can be pressed, verified and waited on, and a screenshot exposes none. The tree stays a local index that the model never has to see.
measured · one screenI assumed a success code meant something happened
macOS returned success on every write I measured, including writes that changed nothing, and reading the value back was wrong in both directions. So the verifier's contract became "the app was seen to react", never "the call returned success". The planner tests are graded by a separate checker that reads the screen, and they assert which step actually ran.
Approval took four designs; each was broken by testing the one before
- A
confirmed: trueflag could be written by any caller, so approval became a one-time ticket. - Another process pressed Allow through the accessibility API in ~10 ms, so Allow now counts only from a hardware input event.
- "Always" rules lived in a file any process could edit, so they moved to the Keychain.
- The selection could change between approval and action, so the ticket now records what was selected. A stale ticket is refused in 3 ms.
Try to approve it
An agent is working in Finder. Pick what it asks for, then try to get it past the harness.
Nothing to approve: it just runs.
Here your click or tap stands in for the hardware mouse-up macOS reports. A script’s press arrives with no input event at all, as it does in the harness. Keyboard approval is refused on purpose there too.
Check order turned out to be a safety rule
Finder's Empty Bin is disabled when the bin is empty, so checking "enabled" first gave try again later where the answer should have been never. The never-list now runs before everything except the password-field refusal.
Fixing performance removed a guard I hadn't designed
- SAWRequests ran on the main thread, so one slow action held up everything behind it.8,520 msstalls in one planner run
- CHANGEDRequests moved off the main thread.2.8 mslongest stall after
- HOLDSA request queued behind a 3-second action.5 msmedian of 5
- SAWThe slowness was hiding a hole: the harness’s own approval panel was only safe because it was too busy to respond.
- CHANGEDSo I added a rule that refuses it outright, in the same change.
- NOT YETNo memory across sessions, and apps without a readable structure are out of reach.
One of my own numbers was an artefact of my measuring
"76 of Chrome's 158 elements share a name" was mostly my tree walk reading Chrome's tab strip four times. Deduplicating by element identity took that to 5 shared-name groups, and the retraction is in commit 6488e06.
What stands
- A 14-verb interface over a Unix socket.
- A kernel that refuses irreversible actions outright.
- Approval that only a hardware click can give.
- Planner tests passing 6/6 on two consecutive runs (Finder and System Settings only). measured · 2 runs
What it still doesn't do
- No memory across sessions, and apps without a readable structure are out of reach.
- Risky actions are recognised by English words in button titles, so "Place order" is allowed as an ordinary press.
- "The app reacted" isn't "the right thing happened". Verification checks that something changed, not that it was what you meant.
- A hardware click proves a device, not a person.
- Nothing reverses a click.