Browser agent data and evals

Train agents on work that really happened.

rtrvr has served 35,000+ users across more than 7 million live-web runs. We turn those runs into training data, evals, and browser environments for agents that need to act, recover, and verify.

Explore the runs Work with rtrvr

Plain-text version for agents and crawlers: /data.md

Production run

jobs.ashbyhq.com

Task

“apply to this job”

Page state105 interactive elements

textboxNameid=61

textboxEmailid=66

buttonUpload fileid=75

textboxEvaluation systemid=97

buttonSubmit applicationid=112

Actionupload_file · element 75

Success

Final checkfresh page state

Complete

Real lines from the job-application sample below.

7M+
live-web runs: 35K+
users served: 81.4%
Halluminate Web Bench, ranked #1: 3
complete runs to inspect

The production signal

A run contains more than clicks.

It records what the agent saw, what it tried, what failed, how it recovered, and whether the work was actually complete.

01Page state

02Plans and tools

03Failure and recovery

04Verified outcome

Why this data matters

Where browser agents still fail.

Real runs show whether an agent can find the right control, carry state, recover, and verify the final result.

Find the right control

Pages can contain thousands of interactive elements, including controls outside the viewport.

Keep the state

Long tasks cross tabs, redirects, logins, apps, and human approval turns.

Recover in place

A stale element or failed tool call should repair the plan, not end the run.

Know when it is done

The agent must check the result instead of treating its own last action as proof.

Three production samples

Open a run. Inspect every step.

Choose a task, select a tool call, and load the exact page state the agent used. The viewer reads the same JSON that ships in each bundle.

workflow.json

Job applicationsjobs.ashbyhq.comdeepseek-v4-flash

Instruction “apply to this job”

Loading the run

Production task mix

The work our users ran across the web.

A recent one-week window, deduplicated to distinct tasks and grouped by function. Use the full mix for a generalist agent, or start with the slice your model needs.

0128.8%

Data extraction & scraping

Pull structured records from listings, dashboards, and directories, usually into a sheet.

Typical horizon

10–100+ actions, often fanned across many tabs

0219.0%

Job applications

Applications through Ashby, Greenhouse, Lever, and Workday, including files, fields, and essay answers.

Typical horizon

10–40 actions on one long form

0312.0%

Docs, Sheets & Drive ops

Creating and editing docs and spreadsheets, and moving results between the browser and Google Workspace.

Typical horizon

5–50 actions, frequently as the sink of another task

0411.2%

LinkedIn networking & outreach

Finding people, verifying identity, sending connection requests and messages, and confirming delivery.

Typical horizon

5–20 actions per contact

058.2%

E-learning & assessments

Course modules, quizzes, and certification flows on SkillsBuild, SCORM players, and publisher courseware.

Typical horizon

Very long, often 50–500+ actions across many pages

067.5%

Business forms & ops

Government portals, invoicing systems, CRMs, and internal tools, including checkout flows that pause for approval.

Typical horizon

10–60 actions, often multi-step wizards

076.9%

Content generation in-page

Drafting and posting inside the target app's own editor, including CMS bodies and rich-text fields.

Typical horizon

5–25 actions

086.4%

Social media actions

Posting, replying, reacting, and scheduling across social platforms, usually from a list of targets.

Typical horizon

3–15 actions per target, repeated

Work with rtrvr

Start with the corpus. Build what is missing.

We work with teams building browser agents, language action models (LAMs), and world models on offline policy data, online browser environments, evals, and verifiers.

Offline policy

Learn from production runs

Use a focused cut of the corpus for post-training, imitation, or offline policy learning.

Generalist or specialist by function, vertical, site, horizon, or failure type
Page states, plans, tool calls, errors, recoveries, and outcomes
New runs, redaction, and delivery format scoped with your team

Online policy

Train against the live web

Co-design browser environments for models that learn by acting, recovering, and checking results.

Task families and starting-state setup drawn from observed work
Browser, API, network, and code actions in the same environment
Rewards and terminal checks anchored to the actual end state

Evals

Measure the misses that matter

Build evals around the tasks your model is close to solving but still fails in production.

Task taxonomies grounded in real browser work
Short tasks and workflows that run for 30–60 minutes
Verifiers for success, recovery, safety, and efficiency

For coding-agent teams

Browser work is code work.

Our planner writes code against the page, executes it, receives typed errors, repairs the program, and stops only after a final verification. The same loop matters for coding agents: long-horizon plans, tool execution, recovery, and checked completion.

PlanExecuteRepairVerify

Talk to the rtrvr team Download sample runs

Research

What we learned building the agent.

Our public work on page understanding, agent harnesses, reliability, and cost. The commercial data and evals are scoped with each partner.

Aug 2025

SOTA on Halluminate Web Bench

81.4% across 323 tasks without screenshots.

Dec 2025

DOM intelligence architecture

How we build a richer, smaller page representation.

May 2026

Replacing tool loops with eval()

Why the planner writes a program instead of one click at a time.

Jun 2026

Code as plan

A text-only agent that executes, repairs, and continues.

Aug 2026

The self-improving action layer

How production runs become better site knowledge.

Aug 2026

Nothing but the cache

Byte-stable observations and a 90% lower token bill.

Sample data

Download the full sample runs.

Each ZIP includes the trajectory, every available page state, an index, summary counts, and the schema README.

Bundle contents

workflow.json      full trajectory
trees/<id>.json    captured page states
trees/INDEX.json   IDs, URLs, and counts
SUMMARY.json       coverage and totals

Citation

@misc{rtrvr2026trajectories,
  title        = {Web Agent Trajectories from Production Traffic},
  author       = {{rtrvr.ai}},
  year         = {2026},
  howpublished = {\url{https://rtrvr.ai/data}},
  note         = {Production browser-agent runs captured as enriched accessibility-tree
                  observations with typed actions and verified outcomes}
}

Work with rtrvr

Improve agents on real browser work.

Bring us one task your model misses. We will build the data, browser environment, eval, or verifier needed to improve it.

Talk to the rtrvr team30 minutes · bring one task or failure case

@misc{rtrvr2026trajectories, title = {Web Agent Trajectories from Production Traffic}, author = {{rtrvr.ai}}, year = {2026}, howpublished = {\url{https://rtrvr.ai/data}}, note = {Production browser-agent runs captured as enriched accessibility-tree observations with typed actions and verified outcomes} }