What happened
Visa tested 22 AI shopping agents built on models from Google, Meta, OpenAI, Anthropic, and eight other labs. In 23.1% of runs, the agent bought a product that did not meet the shopper's needs, leading to potential disputes. Visa's Agentic Commerce Benchmark highlights that even top models fail and that a more expensive model doesn't solve consistency issues. The company aims to establish itself in the agentic payments landscape, where liability for agent-initiated purchases remains unresolved. The test suggests a significant financial impact from disputes, positioning Visa as a potential arbiter in this emerging space.
Key facts & figures
- ·Visa tested 22 AI shopping agents built on models from Google, Meta, OpenAI, Anthropic, and eight other labs.
- ·In 23.1% of runs, the AI agent purchased a product that did not meet the shopper's needs.
- ·Visa's Agentic Commerce Benchmark reported a 57.2% pass rate for the agents.
- ·The article highlights that no party has yet agreed on liability when a shopper disputes an agent-initiated transaction.
- ·Visa aims to position itself as a central figure in the agentic payments disputes landscape.
Why it matters — our perspective
Visa's benchmark provides crucial data on AI agent reliability, highlighting liability and dispute risks that will shape the future of agentic commerce and payments.
Key questions for investors and operators
When evaluating transactions, technology stacks, or commercial partnerships in this space, we recommend looking closely at three questions:
Companies & platforms involved
Visa tested 22 AI shopping agents and announced the Agentic Commerce Benchmark.
The AI shopping agents were built on models from Google.
The AI shopping agents were built on models from Meta.
The AI shopping agents were built on models from OpenAI.
The AI shopping agents were built on models from Anthropic.