We pointed an AI agent at Shopify's own demo store. It stopped at signup.

In three lines

• We drove Shopify's public demo store hydrogen.shop with Claude computer-use, clicking for real
• It halted at a CAPTCHA on the signup form (13 actions, $0.53, score 20/100)
• AI-referred ecommerce orders grew 15x year over year, yet there is almost no way to measure whether an agent can actually buy

Why measure this at all

• Orders arriving via AI search grew 15x globally between Jan 2025 and Jan 2026 (Shopify Japan)
• In Japan, 64.0% of people aged 20–60 have used AI search to look for a product (Dec 2025, up from 47.1% in Apr 2025) (SEM Research)
• Sessions arriving via AI convert +49% better than organic search (same source)

So agent-referred visitors are growing in both volume and quality. What was missing was any way to check whether an agent can actually operate your site.

Plenty of tools already score a site statically — does it have llms.txt, is JSON-LD present. That is *discoverability*. Whether an agent can fill the form, add to cart and reach the payment step is a different question.

What we built

• Playwright drives a real browser; screenshots go to the Claude computer-use API and the returned coordinates become clicks and keystrokes
• No per-site scripts. The agent gets a natural-language goal and explores the UI itself — otherwise you measure the script's quality, not the agent's
• Every step is captured as a screenshot so the record shows what actually happened

The six scored steps

The safety design (arguably the real subject)

This tool runs an automation agent against other people's sites, so none of the following is optional.

1. Purchase confirmation is refused in code

Telling the agent "never click the purchase button" is not enough. Instruction-following is probabilistic, so one day it will be broken. Instead the harness compares the element text against a forbidden list (Place order, Pay now, 注文する, …) immediately before executing a click and discards the click regardless of what the agent decided, recording that moment as "reached pre-checkout" and ending the session cleanly.

// Immediately before the click executes. The LLM's intent is ignored.
const guard = evaluateClickText(elementText);
if (!guard.allowed && guard.treatAsCheckoutReached) {
  // Never clicked; recorded as "reached pre-checkout" and finished
}

2. CAPTCHAs are never solved

On detection the run stops and records captcha_block. Solving one would destroy the very thing being measured.

3. Consent is enforced by the type system

A branded ValidConsent type carries consent_confirmed_by and consent_date, and nothing else can call the run function. This is not validation that throws — it simply does not type-check.

const ConsentBrand: unique symbol = Symbol("ValidConsent");
export type ValidConsent = {
  readonly [ConsentBrand]: true;
  readonly consent_confirmed_by: string;
  readonly consent_date: string;
};
// createConsentToken() is the only thing that can produce this type
export async function runAgentQaSession(consent: ValidConsent, ...)

4. It says who it is

Requests carry a dedicated AgentQA-Bot/1.0 (+contact URL) user agent. It never impersonates an ordinary browser — whoever reads the server log should be able to find out what this was.

The result

Score:   20/100
Status:  stopped
Tags:    captcha_block
Actions: 13
Cost:    $0.5372

The full report, with screenshots: https://www.agentrix.biz/ja/reports/a0fa49ca-52d3-4e2c-b03f-3242a2c471d8

How to read this

This is not "Shopify's demo store is bad". The opposite: the CAPTCHA works correctly for humans. The problem is the asymmetry.

• A human blocked by a CAPTCHA gets annoyed — and shows up in your bounce rate
• An agent blocked by a CAPTCHA throws no error and simply disappears

Nothing lands in your analytics. Conversion rate does not drop, because the session never existed. That is why nobody notices.

This is what we mean by "AI agents are quietly abandoning your checkout".

Two bugs only a real site could surface

1. Key-name dialects

computer-use emits xdotool-style key names (ctrl, super, Return, prior); Playwright expects its own (Control, Meta, Enter, PageUp). A single ctrl threw keyboard.press: Unknown key: "ctrl" and took the whole session down.

const KEY_ALIASES: Record<string, string> = {
  ctrl: "Control", super: "Meta", cmd: "Meta",
  return: "Enter", prior: "PageUp", next: "PageDown",
  up: "ArrowUp", down: "ArrowDown", /* ... */
};

export function normalizeKey(key: string): string {
  return key.split("+").map(normalizeKeyPart).filter(Boolean).join("+");
}

2. One bad action killed the whole run

The deeper problem was structural: any action that threw ended the diagnosis as failed. On real sites, elements vanish mid-click and unexpected modals appear all the time. A tool that dies on every misstep is not a tool.

let actionError: string | null = null;
try {
  await executeComputerAction(page, action);
} catch (err) {
  actionError = err instanceof Error ? err.message : String(err);
}
// Record the failure on the timeline and keep going. The agent observes an
// unchanged screen next turn and can try another route.

Takeaways

• Agent-referred ecommerce traffic is genuinely growing, but almost nothing measures whether agents can complete anything
• Running the agent for real surfaces concrete stopping points — CAPTCHAs, missing labels, forced login — that static scoring cannot see
• If you run this against someone else's site, the consent gate, the purchase-click refusal and the no-CAPTCHA-solving rule belong in code, not in a prompt

We run one free diagnosis per site, for consented sites only. Request a diagnosis