Signed-in business software,
replicated for agents.

1. 01

   Structure from anonymized production pages

   For 4 of 12 environments, the interface structure is aggregated from anonymized production page data: roles, layout, action hints and short generic labels that recur across many users. No page text, values or links are kept. The other 8 were authored from product documentation.

2. 02

   An unbranded, live app

   Each replica is a working web app with its own database, seeded with generated people, companies and records. Every tool call changes real application state.

3. 03

   rtrvr's production observation and tools

   Agents read each page as the accessibility-tree text rendered by the pinned rtrvr extension (build 528f571), and act with rtrvr's production act tools: 37 tools in profile sim-core-v1.

4. 04

   A deterministic verifier

   SQL, event-log and answer checks grade the final state, with no LLM judges. Faults, user corrections and approval pauses are scripted per task, so every run is repeatable.

[h1] Data Analyst
[textbox][editable][actions=click→type:focus-first] Name * [id=7802] [name=fullName] [required] [autocomplete=name]
[textbox][editable][actions=click→type:focus-first] Email * [id=7811] [type=email] [name=email] [required] [autocomplete=email]
[combobox][autocomplete=list] Location * [id=7829] [name=location] [placeholder="Start typing a city"]
[textNode] Resume *
[occluded][textbox][read-only][actions=click→type:focus-first][format=accepts .pdf,.doc,.docx] Upload file [id=7839] [type=file]
[group] Are you legally authorized to work in the United States?
  [below][button] Yes [id=7870]
  [below][button] No [id=7872]
[below][button] Next [id=7874]

1. 01engineering manager

   Contractor access and team role audit with sign-off

   An engineering manager audits outside collaborators and team roles on a code host against a contractor roster spreadsheet and a collaborator policy document. Contractors whose contract ended, or who aren't on the roster, lose all access except keep-list rows; active contractors lose access outside their contract scope and are lowered to their contract role without ever being raised, and a contract ending today still counts. Suspended people leave every team and only listed leads may stay team maintainers. Removing an Admin grant needs the requester's sign-off on the full list, and the requester holds one back until a vendor handover finishes.

   • Approval pause

   Read the instruction

   Please audit outside collaborators and team roles in Code Host against our Collaborator policy and the Contractor roster (Groupware › Files › Engineering › Vendors). Today is 2026-09-10. Ask me before you remove anything that has Admin. When you're done, list the access you removed as "repository: @login" (the repository's short name, without the organization) and tell me how many team membership changes you made.

   The verifier checks

   • At least one access entry or team role was changed
   • warehouse-loader: @aingersoll-ext lowered to Write (above the contract role)
   • mobile-app: @aingersoll-ext removed (outside the contract scope)
   • design-system: @astridthorvaldsen removed (contract ended)
   • developer-docs: @astridthorvaldsen removed (contract ended)
   • developer-docs: @estanhope-ext removed (outside the contract scope)
   • admin-console: @fdunmore-ext removed (contract ended)
   • developer-docs: @fokafor-ext removed (contract ended)
   • warehouse-loader: @pkowalczyk-ext lowered to Read (above the contract role)
   • developer-docs: @pkowalczyk-ext removed (outside the contract scope)
   • mobile-app: @rkowalczyk-ext removed (outside the contract scope)
   • developer-docs: @rkowalczyk-ext lowered to Write (above the contract role)

   Length

   Tier 3 · 60–150 actions

   Apps

   Code Host, Groupware

   Reference solution

   64 actions

   Step budget

   96

   Provenance

   Authored from product documentation

   Watch the reference run

2. 02RevOps analyst

   Turn call recaps into next steps and follow-ups

   A sales manager's team emails a recap after every customer call. The RevOps analyst applies each recap of an open deal currently owned by the team: the deal's next step becomes the recap's text and one follow-up task is created on the deal with a standard subject, the recap's follow-up date (a date the rep corrects in a later reply wins, and a missing date means a week after the call), a priority that depends on the deal stage, and the deal owner as assignee. Recaps for closed deals, deal names that are not in the CRM (even when a similarly named deal exists) and deals owned by another team (including one reassigned since the call) are skipped and counted by reason, and a forwarded copy of a recap must not create a second task. The first task save fails with a server error, every task save asks for confirmation and the next-step field starts collapsed. Partway through, the manager announces that one rep is going on leave and that follow-ups for that rep's deals go to the manager instead, including any already created.

   • Server error (500) on the first save
   • Confirm dialog before submitting
   • Required field in a collapsed section
   • 1 scripted correction

   Read the instruction

   Kasimir Farrow's Central team sent their call recaps to my inbox in Groupware. Please process every call recap there (whoever sent it) in the CRM as described in "Call recap follow-up standard" (Groupware, Files > Sales Operations > Playbooks). Answer with how many deals you updated, how many follow-up tasks you created, and how many recaps you skipped under each code (CLOSED, NOT_FOUND, OTHER_TEAM; give all three, 0 when none).

   The verifier checks

   • At least one follow-up task was created
   • Ravensloe Hospitality – Reporting suite FY26: Next step "Rollout plan walkthrough"
   • Ravensloe Hospitality – Reporting suite FY26: one open follow-up task "Follow up: Rollout plan walkthrough" due 2026-06-03, normal, owner 4
   • Juniperra Veterinary Clinic – Data migration FY26: Next step "Implementation timeline review"
   • Juniperra Veterinary Clinic – Data migration FY26: one open follow-up task "Follow up: Implementation timeline review" due 2026-06-07, high, owner 12
   • Caldovine Realty – Training package FY26: Next step "Second demo for the warehouse leads"
   • Caldovine Realty – Training package FY26: one open follow-up task "Follow up: Second demo for the warehouse leads" due 2026-06-04, normal, owner 4
   • Nevrassa Consulting – Security bundle FY26: Next step "Usage forecast for the second year"
   • Nevrassa Consulting – Security bundle FY26: one open follow-up task "Follow up: Usage forecast for the second year" due 2026-06-12, normal, owner 10
   • Marrowby Realty – Partner portal FY26: Next step "Procurement intake form submitted"
   • Marrowby Realty – Partner portal FY26: one open follow-up task "Follow up: Procurement intake form submitted" due 2026-06-21 (date corrected in a reply), high, owner 4
   • Carrowmere Pharmacy – Mobile access FY26: Next step "Security questionnaire back from IT"

   Length

   Tier 4 · 150–300 actions

   Apps

   CRM, Groupware

   Reference solution

   171 actions

   Step budget

   360

   Provenance

   Authored from product documentation

   Watch the reference run

3. 03cloud security administrator

   Least-privilege migration wave with blocked rows

   A cloud security administrator applies a least-privilege migration plan across four non-production projects in a cloud access console: for each plan row, replace the principal's Editor role with exactly the listed roles in one change (keeping other roles, never granting a role twice), resolve rows that name permissions to the single smallest role that covers them, and record each row's result in the plan sheet. Three rows can't be applied (no single role covers the permissions, a listed custom role is disabled, the Editor role is only inherited from a folder): the administrator must leave them untouched, record a reason code and report the wave as partial.

   • Includes an infeasible item

   Read the instruction

   Please run wave 3 of our least-privilege migration in Access Admin today (Tuesday, September 8, 2026). The rows are in the "Least-privilege migration plan - wave 3" sheet in Groupware (Files, Platform › Least privilege), and the Least-privilege migration standard in the same folder says how each row is applied, which rows can't be applied, and the reason codes to use. Fill in the plan's Result column for every row as you go. When you're done, answer with status "partial" if any row could not be applied (otherwise "completed"), how many rows you applied and, for each reason code (NO_COVERING_ROLE, ROLE_DISABLED, INHERITED_ONLY), the emails of the principals whose rows were blocked with it.

   The verifier checks

   • The answer status is "partial" (some rows could not be applied)
   • CHG-2597: vikram.sandoval@pellmore-hospitality.example.test on partner-api-staging-7415 has exactly database-client, secrets-reader (Editor replaced, other roles kept)
   • CHG-2597: the Editor removal carries the change ID as its reason
   • CHG-2597: the plan's Result says Done (a short note after it is fine)
   • CHG-2598: freya.adeyemi@pellmore-hospitality.example.test on partner-api-staging-7415 has exactly artifacts-writer, functions-developer (Editor replaced, other roles kept)
   • CHG-2598: the Editor removal carries the change ID as its reason
   • CHG-2598: the plan's Result says Done (a short note after it is fine)
   • CHG-2599: default-runtime@partner-api-staging-7415.svc.example.test on partner-api-staging-7415 has exactly functions-developer, logging-viewer (Editor replaced, other roles kept)
   • CHG-2599: the Editor removal carries the change ID as its reason
   • CHG-2599: the plan's Result says Done (a short note after it is fine)
   • CHG-2600: freya.adeyemi@pellmore-hospitality.example.test on partner-api-dev-7238 has exactly keys-user, secrets-viewer (Editor replaced, other roles kept)
   • CHG-2600: the Editor removal carries the change ID as its reason

   Length

   Tier 4 · 150–300 actions

   Apps

   Access Admin, Groupware

   Reference solution

   196 actions

   Step budget

   294

   Provenance

   Authored from product documentation

   Watch the reference run

4. 04payments operations analyst

   Work the support refund queue under the refund policy

   A payments operations analyst works the refund requests that the support desk routes to finance. For each ticket the analyst finds the store payment in the payments dashboard, applies the written refund policy in order (bank disputes, requests already refunded or raised twice, an approval limit that sends large refunds to the controller, double checkouts, damaged deliveries within 60 days, change-of-mind returns within 30 days with a restocking fee, and late deliveries that earn the shipping fee back) and refunds exactly what the policy allows with a ticket-number note. The first refund save fails with a server error and must be retried without refunding twice. Each ticket then gets an outcome tag, the refunded amount, a reply quoting the refund reference and the right status.

   • Server error (500) on the first save
   • Required field in a collapsed section

   Read the instruction

   Support Desk has routed this week's refund requests to finance (Billing group tickets tagged refund_request). Work every unsolved one under the "Customer Refund Policy" in Groupware Files (Finance > Policies): decide each request in Payments, refund there exactly what the policy allows, then tag, fill in and answer the ticket in Support Desk as the policy says. Create at most one refund per payment (a failed save must not end up refunded twice), and don't change anything else in either app. When you're done, answer with the IDs of the refunds you created, the total amount refunded and the numbers of the tickets you escalated.

   The verifier checks

   • At least half of the refund-request tickets carry an outcome tag
   • #4323 (Wrenhollow Mechanical): tagged refund_denied (and no other outcome), product area Hardware & orders, order ORD-41458, refund amount empty
   • #4323 (Wrenhollow Mechanical): public reply
   • #4323 (Wrenhollow Mechanical): Solved
   • #4324 (Sabrevel Tree Care): PAY-WLL6GVE3 refunded once, $118.00, reason Customer request, note "Ticket #4324"
   • #4324 (Sabrevel Tree Care): tagged refund_full (and no other outcome), product area Hardware & orders, order ORD-41461, refund amount $118.00
   • #4324 (Sabrevel Tree Care): public reply with the refund ID
   • #4324 (Sabrevel Tree Care): Solved
   • #4325 (Norvessa Garage Doors): PAY-PSM6YWKE refunded once, $186.15, reason Customer request, note "Ticket #4325"
   • #4325 (Norvessa Garage Doors): tagged refund_partial (and no other outcome), product area Hardware & orders, order ORD-41464, refund amount $186.15
   • #4325 (Norvessa Garage Doors): public reply with the refund ID
   • #4325 (Norvessa Garage Doors): Solved

   Length

   Tier 4 · 150–300 actions

   Apps

   Payments, Support Desk, Groupware

   Reference solution

   219 actions

   Step budget

   330

   Provenance

   Authored from product documentation

   Watch the reference run

5. 05operations analyst

   Stand up a customer onboarding tracker and load the hand-off data

   An operations analyst sets up a new grid database for customer onboarding from a written spec: the starter table is renamed and reshaped (renamed primary and paragraph fields, the starter stage options renamed and extended), eight fields are added, and five views are built: a renamed sorted grid, an any-of filter with a nested all-of group, a two-level grouped grid, a kanban board with chosen card fields, and an intake form with custom labels, a required question, a custom button and a confirmation message. The pasted sales hand-off list must be cleaned on the way in (owner initials mapped to full names, coded plan, phase and health values, year/month/day dates the importer cannot read, a test account and a customer entered twice), and finance's ARR corrections must update only the matching onboardings without adding the customer that is not onboarding.

   Read the instruction

   We're moving customer onboarding out of the sales hand-off spreadsheet into Tables. Please set up a new database called "Customer Onboarding" and shape its starter table into our tracker: TABLE "Onboardings" (rename the starter table; keep its fields, renamed): - Account: the primary field (the starter's Name), required. - Blockers: the starter's Details paragraph. - Stage: keep the starter's Stage field but its options must be, in order: Kickoff, Data migration, Training, Live, Stalled. - CSM: Choice with options, in this order: Ximena Farrow, Keziah Kavanagh, Dario Alvarado. - Plan: Choice, required, options in this order: Starter, Growth, Enterprise. - Kickoff date and Go-live target: Date fields. - ARR: Currency in whole dollars. - Seats: Number (whole numbers). - Health: Choice with options, in this order: Green, Yellow, Red. - Sponsor email: Email. Nothing else (we don't keep the CRM reference). VIEWS: - "All onboardings": the default grid, renamed, sorted by Go-live target, earliest first. - "At risk": a grid showing onboardings that match ANY of: Health is Red; Stage is Stalled; Health is Yellow and the Go-live target is before today (both of those together). - "By CSM": a grid grouped by CSM and then by Stage, sorted by ARR, highest first. - "Onboarding board": a kanban board with a column per Stage; cards show CSM, Health and Go-live target only. - "Onboarding intake": a form whose questions are, in this order: Account labelled "Customer name"; Plan; Seats; Sponsor email labelled "Executive sponsor's email" and required on the form. Button text "Start onboarding"; confirmation message "Thanks! Your CSM will reach out within one business day." DATA: here's the sales hand-off export. Owners are written as initial + surname (they're our CSMs above), plan tiers are starter / growth / ent, phase "on hold" means Stalled, RAG is G / Y / R for Green / Yellow / Red, and the dates are year/month/day. Leave out test accounts. One customer was entered twice by sales; the later row is the correct one. Don't bring in the CRM ref column. Customer,Owner,Plan tier,Phase,Kickoff,Target go-live,Annual value,Licenses,RAG,Sponsor,CRM ref,Blockers Torvalen Consulting,K. Kavanagh,ent,kickoff,2026/08/25,2026/09/05,"$154,000",625,G,teodor.pennington@torvalen-consulting.example.test,CRM-83123,SSO setup waiting on customer IT Ondrevia Insurance Agency,X. Farrow,starter,training,2026/07/20,2026/09/09,"$81,000",820,Y,gaia.thorvaldsen@ondrevia-insurance-agency.example.test,CRM-36795,Sponsor on leave until next month Serravin Motors,K. Kavanagh,ent,live,2026/08/28,2026/11/07,"$18,000",728,G,mireille.garside@serravin-motors.example.test,CRM-94087,Needs custom SLA review Dunvessa Veterinary Clinic,D. Alvarado,starter,live,2026/08/25,2026/12/02,"$147,000",338,G,kasimir.lindqvist@dunvessa-veterinary-clinic.example.test,CRM-58064,SSO setup waiting on customer IT Serravin Consulting,X. Farrow,growth,kickoff,2026/08/15,2026/10/17,"$51,000",638,G,leilani.montoya@serravin-consulting.example.test,CRM-26577,Needs custom SLA review Torvalen Consulting,K. Kavanagh,ent,live,2026/08/25,2026/09/26,"$164,000",625,G,teodor.pennington@torvalen-consulting.example.test,CRM-83123,SSO setup waiting on customer IT Westerlyn Freight,D. Alvarado,ent,kickoff,2026/08/24,2026/12/04,"$133,000",776,Y,priya.stanhope@westerlyn-freight.example.test,CRM-26750,Sponsor on leave until next month Graymantle Outfitters,X. Farrow,ent,live,2026/08/26,2026/10/29,"$80,000",162,G,jordan.pellegrini@graymantle-outfitters.example.test,CRM-46963, TEST - Wrenhollow sandbox,X. Farrow,starter,kickoff,2026/07/20,2026/10/24,"$1,000",820,Y,qa@example.test,CRM-00000, Elmstrand Realty,K. Kavanagh,ent,migration,2026/09/03,2026/10/11,"$103,000",742,G,casey.esposito@elmstrand-realty.example.test,CRM-70690,Needs custom SLA review Meridane Engineering,D. Alvarado,starter,on hold,2026/07/19,2026/09/04,"$169,000",237,G,wren.brightwell@meridane-engineering.example.test,CRM-47729, Norvessa Academy,X. Farrow,ent,training,2026/09/01,2026/11/19,"$235,000",302,R,yara.barrientos@norvessa-academy.example.test,CRM-21272,Legacy data export is late Thistlecombe Energy,K. Kavanagh,starter,live,2026/08/05,2026/10/09,"$201,000",393,G,emil.ferreira@thistlecombe-energy.example.test,CRM-83843, Nevrassa Robotics,D. Alvarado,growth,training,2026/08/27,2026/12/13,"$232,000",816,G,farida.macready@nevrassa-robotics.example.test,CRM-27412,SSO setup waiting on customer IT Ostrevel Apparel,X. Farrow,starter,migration,2026/09/10,2026/11/01,"$58,000",39,R,esperanza.jablonski@ostrevel-apparel.example.test,CRM-67152, Then apply finance's ARR corrections below to the matching onboardings. Finance's list covers all customers, so ignore anyone who isn't in the tracker; don't add them. Customer,Corrected ARR Halvessen Media,"$117,000" Dunvessa Veterinary Clinic,"$141,000" Nevrassa Robotics,"$240,000" Graymantle Outfitters,"$82,000" When you're done, tell me how many onboardings are in the tracker, which customers from either list you deliberately didn't add, and how many onboardings the At risk view shows.

   The verifier checks

   • Exactly one "Customer Onboarding" database with an "Onboardings" table
   • Hand-off rows were brought in
   • Account is the primary field (renamed from Name), required
   • Details is renamed Blockers (Paragraph)
   • Stage options are Kickoff, Data migration, Training, Live, Stalled
   • CSM: Choice with the three CSMs' full names
   • Plan: required Choice Starter, Growth, Enterprise
   • Kickoff date is a Date field
   • Go-live target is a Date field
   • ARR: Currency $ in whole dollars
   • Seats is a whole Number
   • Health: Choice Green, Yellow, Red

   Length

   Tier 3 · 60–150 actions

   Apps

   Tables

   Reference solution

   148 actions

   Step budget

   222

   Provenance

   Authored from product documentation

   Watch the reference run

harbor run -p ./rtrvr-sim/<task-id>-s0 -a claude-code -m <provider>/<model>
harbor run -p ./rtrvr-sim/<task-id>-s0 -a oracle   # reference solution

[[environment.mcp_servers]]
name = "rtrvr-sim"
transport = "streamable-http"
url = "http://sim:8000/mcp"

POST /reset {"task_id": "<task-id>", "seed": 0, "tool_profile": "sim-core-v1"}
  -> {episode_id, observation, tools, task}
POST /step  {"episode_id": "<id>", "action": {"tool_name": "click_element", "args": {"tab_id": 0, "element_id": 412}}}
  -> {observation, reward, done, info}

rtrvr-sim run --policy https://<endpoint>/v1 --model <model-id> --tasks <task-ids>

from trl import GRPOTrainer
from rtrvr_sim_trl import RtrvrSimEnv  # one typed method per tool

trainer = GRPOTrainer(
    model=model,
    train_dataset=tasks,          # rows carry task_id and seed
    reward_funcs=episode_reward,  # reads the verifier's reward at episode end
    environment_factory=RtrvrSimEnv,
)

Each task's verifier writes reward, success, weighted score, steps, invalid actions, no-op actions that left the page unchanged, gates passed and whether the agent answered.