AI models recommended a completely fictional company in up to 100% of test runs, showing that confident recommendations don’t necessarily mean a business has been vetted.
AI recommendations can change significantly from one search to the next, with roughly half of the suggested brands changing when researchers asked the same question twice.
Experts recommend using AI as a starting point, not the final word, and checking a company’s digital footprint, customer reviews and independent sources before making a purchase.
More people are turning to AI tools like ChatGPT and Gemini for help deciding what to buy, where to shop, and which companies to trust. But how much confidence should consumers really place in those recommendations?
A new experiment from digital marketing agency Fractl raises some uncomfortable questions. Researchers created a completely fictional company — with no website, customers, or actual business behind it — and tested whether AI models would recommend it alongside legitimate competitors. In many cases, they did, and they did so with surprising confidence.
The findings suggest that an AI chatbot sounding certain about a company doesn’t necessarily mean that company has been vetted or that the recommendation is based on solid information.
ConsumerAffairs spoke with Nicole Franco, Head of Digital PR, AI Innovation at Fractl, who explained that consumers may want to keep this in mind as AI increasingly becomes part of the way they research products and services.
What the research found
In one study, researchers placed a completely fabricated company alongside five legitimate competitors in simulated search results.
Six of the nine AI models tested recommended the fake company in 94% to 100% of runs, while two others recommended it less often or included caveats. The newest model tested, Claude Fable 5, was the only one that consistently identified the company as unverifiable and explained why it was leaving it out.
A second study looked at 11,573 AI answers across 15 industries and found that AI recommendations often extend beyond the biggest, best-known companies. In fact, 85% of recommendations went to brands outside their industry's top five.
The researchers also found that asking the same buyer question twice could produce noticeably different results: roughly half of the recommended brands changed between responses on several major AI platforms.
“What made me surprised is the little effort needed to get a recommendation for a company which doesn't exist at all,” Franco said. “We created a business which had no website, no customers, no history, and six out of nine models recommended this business 94% to 100% of the times it was part of the information that the model got access to.
“This shows quite obvious vulnerability for anyone able to create a credible-looking piece of information online."
The risks to shoppers
With half of recommended brands changing when the same question is asked, Franco shared some of the risks this can pose to shoppers.
“AI-generated shortlists shouldn't automatically be assumed to represent the top options,” she said. “When we asked the exact same prompt twice, roughly half of the suggested brands shifted. While this volatility isn't particularly concerning for minor purchases like running shoes, it warrants substantial caution for high-stakes decisions, such as selecting a mortgage lender or financial product.
“In those critical scenarios, users should reasonably expect transparency regarding why a specific company was recommended.”
Check for a digital footprint
If you’re struggling to figure out whether a company that pops up in an AI search is legitimate, Franco recommends doing some research.
"If I searched for a company independently and turned up almost nothing in the search results, that would immediately raise flags for me,” she said. “A legitimate business should leave a footprint beyond a single favorable article.
“I recommend examining the sources cited by the AI and performing your own background check to see if authentic customer feedback and reputable sources corroborate its existence. If every piece of information leads back to a single source, I would not feel secure making a purchase there."
To avoid any risky purchases, Franco encourages consumers to use AI as a starting point, and conduct their own research outside of the AI chatbot.
“Even if I haven't heard of a company before, I'm still going to do additional research on it and will most likely cite the sources,” she said. “When it comes to big purchases, I would ask the same question again or ask other models too. I've seen enough variations in our research to not make an expensive choice based on just one answer from AI."