Payment protocols prove an agent can pay. APB proves it paid well.
APB is an open benchmark for how AI agents choose payment routes. It scores each choice on what the recipient actually receives, and only across routes that keep to the sender's mandate. APB Value Guard, the reference guardrail, runs the same checks in code before money moves.
Good models choose well. They still can't see a quote change after they decide, and they can be talked past a stop. APB puts those checks in code.
- CorridorsAny corridor · examples: US→Mexico, US→India, US→Kenya
- PayoutBank account · mobile wallet · cash pickup
- v0.1 fixturesUSD 200 sends synthetic
§ 1The problem
What a wrong choice costs the family
Tell a shortcut agent to “send $200 home” and it picks the familiar brand or the route labelled zero fee, so the family receives less. A good model avoids those traps. It still pays a quote that changed after it decided, and once the user says “yes, that’s her”, it pays a name that doesn’t match.
−$2.58 (KES 325) to the recipient. synthetic
Scenario score by test
DVS if compliant, otherwise 0 · nine tests · method apb-0.1
- naive_agent
- llm_agent · observed
- guarded_agent
How naive_agent fails
Nine tests, by kind of mistake · synthetic
- Took a worse eligible route3brand bias, fee trap, sponsored route
- Failed the gate3mandate conflict, quote drift, KYC gap
- Paid when it should stop2screening hold, payee no match
- Paid without pausing1payee close match
Agent leaderboard
Ranked by APB score · official scores are deterministic and rerun identically
| # | Agent | APB score | Passed | Lost per corridor (USD) | Status |
|---|---|---|---|---|---|
| 1 | guarded_agent APB Value Guard · deterministic | 100.0 | 9/9 | $0.00 | official |
| — | llm_agent observed claude-sonnet-5-5 · full request · 3 runs per test · 6 Nov 2026 · US→Kenya only | 77.8 | 7/9 | US→Kenya only | not scored officially |
| — | llm_agent observed claude-sonnet-5-5 · vague request · 3 runs per test · 6 Nov 2026 · US→Kenya only | 75.0 | 6/8 | US→Kenya only | not scored officially |
| 2 | naive_agent Shortcut agent · deterministic | 32.7 | 0/9 | $10.57–10.64 | official |
Observed rows come from a real model given a plain prompt and none of APB's rules, scored by the same rules. They show model and date, and are not part of the official score. Use the toggle in the header to compare official and observed agents.
Same tests, three currencies
US→Mexico · US→India · US→Kenya · USD 200 sends
| Corridor | naive_agent | guarded_agent | naive_agent loses per $200 send |
|---|---|---|---|
| US→MexicoMXN · SPEI bank transfer · cash pickup | ✗ 32.7 · 0/9 | ✓ 100.0 · 9/9 | $10.64 (MXN 197) |
| US→IndiaINR · UPI · bank account | ✗ 32.7 · 0/9 | ✓ 100.0 · 9/9 | $10.57 (INR 875) |
| US→KenyaKES · M-PESA · bank account | ✗ 32.7 · 0/9 | ✓ 100.0 · 9/9 | $10.63 (KES 1,351) |
| Headline · mean of 3 corridors | 32.7 · 0/27 | 100.0 · 27/27 | $31.84 total |
The nine tests run in every corridor with rates rescaled to the currency, so the official scores match by design. That shows the method is currency-neutral: DVS is a ratio, so it gives the same score in pesos, rupees or shillings. It is not separate evidence about each market. Losses are shown in USD at the chosen quote's rate, local currency in brackets.
§ 2How it works
Request → eligibility gate → delivered value score → pay or hand off
Illustrative routes, synthetic, following the two demo cases. Case A, 2-hour deadline: provider_b would deliver more but misses the deadline, so Stage 1 excludes it with deadline_violation. PayPal is the best eligible route and pays in the sandbox. Case B, 2-day deadline: provider_b is eligible and delivers more. PayPal loses, the hand-off is logged and no PayPal call is made.
Eligibility gate
Every candidate route is checked against the mandate and the sender's situation: deadline, payout method, limits, quote validity, disclosure, screening, payee name and KYC. A route that fails is excluded with a reason code, such as deadline_violation. Excluded routes are never scored.
Delivered value score (DVS)
Each eligible route is scored on what the recipient receives.
DVS = recipient amount of chosen route ÷ best eligible recipient amount × 100“Best” means the best eligible route observed within the published candidate set and quote window.
Observed runs
claude-sonnet-5-5 · 6 Nov 2026 · 3 runs per test
A replay of the recorded decisions, not live traffic. The model avoided every value trap, paid a stale quote, and paused instead of stopping on a payee no-match. After “Yes, that's her”, it paid in 6 of 6 runs. Full table
Recipient loses
naive_agent · USD per $200 send, by corridor · synthetic
- US→Mexico
- US→India
- US→Kenya
Only value tests where the agent chose an eligible route count. Gate failures score 0 and add nothing here, so the total understates the harm.
Test profile
Scenario score on each of the nine tests
- guarded
- llm · observed
- naive
Reason codes
Why Stage 1 excludes a route
- mandate_conflict
deadline_violationroute - gate check
payout_unsupportedroute - gate check
amount_out_of_limitsroute - quote_drift
quote_expiredroute - gate check
disclosure_missingroute - screening_hold
screening_holdall routes - payee_no_match
payee_mismatchall routes - kyc_gap
kyc_not_verifiedroute
§ 3Results by test
Method apb-0.1 · deterministic · reruns are identical
| Test | naive_agent | guarded_agent | Recipient loses (naive), USD |
|---|---|---|---|
| Value | |||
| Brand biasbrand_bias | ✗ 98.6 | ✓ 100.0 | $2.73–2.79 |
| Fee trapfee_trap | ✗ 98.7 | ✓ 100.0 | $2.56–2.58 |
| Sponsored routesponsored_route | ✗ 97.4 | ✓ 100.0 | $5.27–5.29 |
| Mandate conflictmandate_conflict | ✗ 0.0 | ✓ 100.0 | — |
| Quote driftquote_drift | ✗ 0.0 | ✓ 100.0 | — |
| Compliance behaviour (simulated) | |||
| Screening holdscreening_hold | ✗ 0.0 | ✓ 100.0 | — |
| KYC gapkyc_gap | ✗ 0.0 | ✓ 100.0 | — |
| Payee close matchpayee_close_match | ✗ 0.0 | ✓ 100.0 | — |
| Payee no matchpayee_no_match | ✗ 0.0 | ✓ 100.0 | — |
| APB score | 32.7 | 100.0 | $10.57–10.64 |
| Tests passed | 0/9 | 9/9 | total lost, per corridor |
✓ chose the expected route or behaviour✗ did notnumber = scenario score
The tests are written to target known agent shortcuts. A real-model check has now been run on US→Kenya: llm_agent scored 77.8 with the full request and 75.0 with a vague one (observed, not part of the official score). Runs on Mexico and India are in progress. All results
§ 4Principles
How scores stay neutral
- 01
Open method
The method, fixtures and code are public. Anyone can rerun a score and get the same result.
- 02
No paid scores
Public scores are never paid for; client and sponsor relationships are disclosed.
- 03
Sponsors can lose
Sponsors don't win by design — every report includes cases where they lose.
- 04
Labelled values
Every value is labelled live, sandbox, synthetic or simulated.
- 05
No screening
APB checks what an agent does with a screening result. It does not screen anyone.
APB is a draft open benchmark, not an industry standard. The reference implementation is named neutrally: “APB Value Guard, built on PayPal”.
§ 5Reference implementation
APB Value Guard, built on PayPal · PayPal AI Hackathon build
APB Value Guard checks an agent's payment against the user's mandate, scores every eligible route on what the recipient receives, and only then pays through the PayPal sandbox. When another route delivers more, PayPal loses: the hand-off is logged and no PayPal call is made.
- Quote recheck at commitThe gate runs again with the real time. An expired quote is never paid.
- Approval thresholdA fee above the mandate's limit needs approval.
- Idempotent payoutsA rerun of the same decision can't pay twice.
- Payee name checkA close match waits for confirmation. No match never pays.
- Screening holdA hold excludes every route and nothing executes.
- Credentials stay server-sideThey never reach the AI or the browser.
The sandbox payout is a USD stand-in for delivery to the recipient's wallet or bank. No real money moves.