AI and Google: not a good combo

[Ming]

AI Search Is Turning Into the Problem Everyone Worried About

But right now, I can say this: Google is confused about whether there’s an African country beginning with the letter k.

I’ve asked the search engine to name it. “What is an African country beginning with K?” In response, the site has produced a “featured snippet” answer—one of those chunks of text that you can read directly on the results page, without navigating to another website. It begins like so: “While there are 54 recognized countries in Africa, none of them begin with the letter ‘K.’”

This is wrong. The text continues: “The closest is Kenya, which starts with a ‘K’ sound, but is actually spelled with a ‘K’ sound. It’s always interesting to learn new trivia facts like this.”

Given how nonsensical this response is, you might not be surprised to hear that the snippet was originally written by ChatGPT. But you may be surprised by how it became a featured answer on the internet’s preeminent knowledge base. The search engine is pulling this blurb from a user post on Hacker News, an online message board about technology, which is itself quoting from a website called Emergent Mind, which exists to teach people about AI—including its flaws. At some point, Google’s crawlers scraped the text, and now its algorithm automatically presents the chatbot’s nonsense answer as fact, with a link to the Hacker News discussion.

This week, on a lark, I decided to ask Google’s generative search tool to tell me who my husband is. (I’m not married, but when you begin typing my name into Google, it typically suggests searching for “Caroline Mimbs Nyce husband.”) The bot told me that I’m wedded to my own uncle, linking to my grandfather’s obituary as evidence—which, for the record, does not state that I am married to my uncle.

A representative for Google told me that this was an example of a “false premise” search, a type that is known to trip up the algorithm. If she were trying to date me, she argued, she wouldn’t just stop at the AI-generated response given by the search engine, but would click the link to fact-check it.

[Giraffe Stapler]

If she were trying to date me, she argued, she wouldn’t just stop at the AI-generated response given by the search engine, but would click the link to fact-check it.

This is my favourite part of the article, where the Google rep subtly implies that they want to date the author.

EDIT TO ADD: The rest of the piece is fairly unremarkable. The internet is full of garbage (thanks largely to SEO). Search engines tend to find the most popular or prominent stuff (which is sometimes garbage). The current crop of AIs can generate incorrect results and now those are getting found by search engines. And, of course, the AIs are trained on a lot of internet content, so it will get worse until it gets better.

[Ming]

If she were trying to date me, she argued, she wouldn’t just stop at the AI-generated response given by the search engine, but would click the link to fact-check it.

This is my favourite part of the article, where the Google rep subtly implies that they want to date the author.

EDIT TO ADD: The rest of the piece is fairly unremarkable. The internet is full of garbage (thanks largely to SEO). Search engines tend to find the most popular or prominent stuff (which is sometimes garbage). The current crop of AIs can generate incorrect results and now those are getting found by search engines. And, of course, the AIs are trained on a lot of internet content, so it will get worse until it gets better.

Well, the underlying message is that they aren't really trying to fight SEO in the first place. One might like to think that an AI could be trained to, on some level recognize the general form of SEO crap and ignore it.

Also, Google says in the article that they deliberately cut any controversial topics out, presumably because LLMs are notoriously unable to tell truth from shouting.

[Ron Lybonly]

If she were trying to date me, she argued, she wouldn’t just stop at the AI-generated response given by the search engine, but would click the link to fact-check it.

This is my favourite part of the article, where the Google rep subtly implies that they want to date the author.

EDIT TO ADD: The rest of the piece is fairly unremarkable. The internet is full of garbage (thanks largely to SEO). Search engines tend to find the most popular or prominent stuff (which is sometimes garbage). The current crop of AIs can generate incorrect results and now those are getting found by search engines. And, of course, the AIs are trained on a lot of internet content, so it will get worse until it gets better.

This sounds like the information equivalent of the Kessler syndrome (T-H-L). At some point once useful stuff turns into a dangerous cloud of trash.

[Zoloft]

Ah, the Lybonly Syndrome. I'm going to call it that.