Google’s AI Overviews is sozzled and blotto, as P.G. Wodehouse would have described it, when nonsensical phrases are thrown at it.
The feature involves the automated answers at the top of most search queries. If users key in made-up phrases and the word “meaning” is attached at the end of the search, chances are AI Overviews will answer as if the gibberish is a real saying, and the results will include what it means.
“He was a cat who flushed a toilet” is what The Telegraph keyed in and Google AI Overviews answered: “The statement… describes a cat that has learned to manipulate the flushing mechanism of a toilet. This can happen through observation and repetition, leading the cat to associate the flush handle or button with the engaging sound of water and its movement.”
The tech giant’s chatbot “hallucinates” from time to time when asked for a meaning to made-up sayings. Instead of drubbing the question, the algorithm accepts it as authentic and then attempts to offer a meaning.
It’s silly that AI Overviews thinks “Bob Dylan is a poor man’s riches” is a phrase addressing “themes of social justice, poverty, and the struggles of ordinary people”, but it’s also a reminder that generative AI continues to fall short. To complicate matters, at times, reference links are offered to add a sheen of authority.
At the end of every AI Overviews note, Google adds a disclaimer: “Generative AI is experimental.” Ultimately, the technology uses a large language model-based system that, at an elementary level, involves placing one most-likely word after another. It explains why AI Overviews attempts to explain phrases even if they mean nothing.
The problem doesn’t appear to affect the LLM used for ChatGPT. When this newspaper asked for an explanation for “he was a cat who flushed a toilet”, we were told this is “not a common or established idiom, but it can be interpreted metaphorically or humorously”.
AI Overviews in Search now has “1.5 billion users per month”, Google CEO Sundar Pichai said in a statement as part of Alphabet’s Q1 2025 earnings. The company started to widely roll out the feature last May and has continued to expand the tool with updates showing AI Overviews for more types of queries as Google tries to take on other AI-powered search tools like ChatGPT Search and Perplexity.
Addressing the issue, a Google spokesperson has told Wired: “When people do nonsensical or ‘false premise’ searches, our systems will try to find the most relevant results based on the limited web content available…. This is true of search overall, and in some cases, AI Overviews will also trigger in an effort to provide helpful context.”