Visa put 22 AI models through everyday shopping tests, and nearly half of their attempts failed.
Its new Agentic Commerce Benchmark tested models from 12 AI labs across 15 shopping scenarios involving TVs, laptops and personal care appliances.
The benchmark assessed whether AI models could identify customer needs through conversation and make suitable purchasing decisions.
They were also expected to avoid making a purchase when no suitable product was available.
More Than Half of Failed Tests Still Ended in a Purchase
Overall, the models achieved a pass rate of 57.2 percent.
The best-performing models passed 82.2 percent of attempts, while the lowest-scoring models passed 28.9 percent.
More than half (54 percent) of failed attempts still resulted in a purchase.
The AI either selected a product that did not meet the customer’s identified requirements or failed to uncover all their needs.
Models that uncovered more customer requirements generally performed better.
The five models that identified the most hidden requirements achieved an average pass rate of 70.2 percent, compared with 44.4 percent for the five that identified the fewest.
Consistency remained a problem. Each model was given three attempts at every shopping scenario, but all three attempts were successful in just 34.2 percent of model-scenario combinations.
Higher-cost models also performed better on average.
The five most expensive models passed 67.1 percent of attempts, compared with 50.2 percent for the five least expensive.
Can Consumers Trust AI to Shop?
Visa’s Trust Index, released in September 2026, found that while 72 percent of U.S. consumers had used an AI assistant, only 23 percent trusted AI to handle payments on their behalf.
“AI agents have the potential to power the next era of commerce, but trust is key to making it work.
Visa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry.”
said Rubail Birwadker, Global Head of Growth at Visa.
Visa plans to expand the benchmark to cover more shopping scenarios and product categories.
Future testing will also include checkout, payments and order placement.
The company has shared the benchmark with AI model providers for feedback and made the research and interactive tests publicly available through Visa Payments Frontier.
Featured image: Edited by Fintech News Singapore, based on image by vecstock via Magnific
