We have accumulated over 4 million web agent trajectories from real knowledge work in real browsers. Our agent does not drive from screenshots — every observation is a compact accessibility tree in which each node carries a stable id, and every action in the trajectory names one of those ids. That makes each (state, action) pair exactly reconstructable.
Below are three complete runs, published in full. Walk them step by step, read the planner's reasoning, and inspect the exact page state behind every decision.
18
observations published
39,786
tree nodes in them
5,777
link targets
13
per-decision cycles
A screenshot shows you the viewport. On one of the pages in the samples below — an Amazon search results page — the viewport is a few percent of the action space. The accessibility tree for that single page holds 8,699 nodes and 1,137 link targets, all of them addressable, all of them in the state.
| Approach | Observation | Action space | Off-screen content | Reconstructable? |
|---|---|---|---|---|
| Screenshot only | Rendered viewport pixels | Pixel coordinates, inferred from the image | Lost — everything below the fold is absent from the state | Only if you keep the image; coordinates do not survive a layout change |
| Raw DOM / HTML | Full serialised markup | CSS or XPath selectors | Present, but buried in framework noise and hidden subtrees | In principle, but selectors break on class-name churn and the payload is enormous |
| Semantic accessibility treewhat we ship | One line per node: role, accessible name, stable integer id, plus a separate href record | The integer ids themselves — every action in the trajectory names one | Fully present. The whole page is in the state regardless of scroll position | Exactly. Every (observation, action) pair joins by accTreeId and element id |
Each node is one line: [role] accessible name [id=N]. Every element_id in the trajectory indexes one of those labels, so joining an action back to the node it touched is a lookup, not an inference.
Each tree records the previous observation in the same run under provenance.parentTreeId. Nodes are observations, edges are the actions between them, and across many runs on one site those graphs merge.
Links live in a separate elementLinkRecord. The agent is shown an enriched view; we store the lean text so the observation stays byte-stable and the enrichment stays derivable.
The agent does not emit one model call per click. A planner reads the page once and writes a JavaScript plan script that runs in a sandbox, calling browser helpers directly. When a step is too dynamic to script blind, the script delegates to a sub-agent that does run a classical per-step loop. Both levels are recorded, and they want modelling separately.
One observation in, a whole script of actions out. In the Ashby run a single planner pass produces a nine-thousand-character program that then drives twelve browser calls with no further model round-trips.
This is not a per-step MDP, and we do not pretend it is. Because the script runs without model round-trips, intra-script page transitions are genuinely unobserved. That is a property of the policy, stated plainly in every bundle's README.
Inside an rtrvr.act delegation, agentSteps[] is a classical observe→decide→act loop. Each entry names exactly the observation that decision saw.
Failures are kept verbatim — for example Element with ID '128' not found in DOM — usually followed by the model's recovery on the next observation. Those error→recovery pairs are typically the highest-signal part of a run.
Reward signal · Per action
functions[].response
Every tool call carries the runtime's own response string — "Success", or the verbatim failure such as "Element with ID '128' not found in DOM".
Reward signal · Per step
status / error
Each stored step is independently labelled, so a step that errored is separable from the run that still succeeded around it.
Reward signal · Per task
status + code_plan_review.verdict + output
A terminal verification pass re-observes the page and certifies completion, giving a task-level label grounded in a fresh observation rather than a self-report.
This is the measured mix of knowledge work running on the platform, not a taxonomy we invented on a whiteboard. Shares are taken over a recent one-week production window, deduplicated to distinct tasks and normalised across these eight categories.
Pull structured records off listings, dashboards, search results and directories, usually into a sheet. The broadest category by number of distinct users.
Typical horizon
10–100+ actions, often fanned across many tabs
What makes it hard
Pagination and lazy loading mean the observation is never complete. The agent has to decide when it has seen enough, not just what is on screen.
Shares describe the mix within this slice of knowledge work. They are not a claim about total platform volume, and the categories are not mutually exclusive at the margin — an extraction task that ends in a spreadsheet touches two of them.
Nothing here is a mock-up. These are production runs, served as the same static JSON you get in the download. Pick a step to read the planner's reasoning and the program it wrote; pick a decision to load the exact page state that decision saw, with the element it acted on highlighted.
The entire instruction
“apply to this job”
Three words of instruction become a filled, uploaded and submitted ATS application.
Caveat: Zero failed actions. Clean runs are useful for behaviour cloning but carry no recovery signal — for that, see the Amazon run.
This bundle
Every bundle ships this same README. Four files, one join key, no proprietary format.
workflow.json the trajectory trees/<id>.json one accessibility-tree observation per file, keyed by accTreeId trees/INDEX.json accTreeId -> url, tree size, link count SUMMARY.json machine-readable counts, including tree coverage and missingIds
trees = {t["accTreeId"]: t for t in load_all("trees/*.json")}
for step in workflow["multiSteps"]:
for decision in step.get("agentSteps", []):
obs = trees[decision["accTreeId"]] # state
acts = decision.get("functions", []) # action(s)
why = decision.get("thought") # reasoning trace
# the next decision's accTreeId is the resulting stateStated plainly, because you will find it yourself: these are live-web runs, so the sites have changed since capture. The trees are the durable record; the URLs may no longer render the same page. Tree coverage is reported per bundle in SUMMARY.json, and where a planner-pass observation is missing it is listed by id rather than quietly dropped.
We can export by task shape, site, horizon length, failure density, or surface — extension runs in a real logged-in browser, or cloud-browser runs we drive ourselves. Tell us what an environment or an eval needs to look like and we will cut it.