{"id":79358,"date":"2026-01-12T08:30:57","date_gmt":"2026-01-12T13:30:57","guid":{"rendered":"https:\/\/world.edu\/?p=79358"},"modified":"2026-01-12T08:30:57","modified_gmt":"2026-01-12T13:30:57","slug":"i-used-ai-chatbots-as-a-source-of-news-for-a-month-and-they-were-unreliable-and-erroneous","status":"publish","type":"post","link":"https:\/\/world.edu\/i-used-ai-chatbots-as-a-source-of-news-for-a-month-and-they-were-unreliable-and-erroneous\/","title":{"rendered":"I used AI chatbots as a source of news for a month, and they were unreliable and erroneous"},"content":{"rendered":"

\"\"<\/p>\n

It was cute. But it was still a lie. Gemini invented a news outlet that doesn\u2019t exist and named it\u00a0fake-example.ca<\/em>\u00a0(or\u00a0exemplefictif.ca<\/em>, in French).<\/p>\n

The generative AI system offered by Google led its fictional media outlet to report that a school bus drivers\u2019 strike had been called on Sept. 12, 2025, in Qu\u00e9bec. But that wasn\u2019t why school transportation was disrupted that day. It was because of\u00a0the withdrawal of Lion Electric buses due to a technical issue<\/a>.<\/p>\n

This journalistic hallucination is perhaps the worst example of fabrication I saw in an experiment that lasted about a month. But I found many others.<\/p>\n

Turning to AI chatbots for news<\/h4>\n

As a journalism professor specializing in computer science, I have been using AI long before the advent of ChatGPT in 2022. According to the latest Digital News Report by the Reuters Institute for the Study of Journalism,\u00a0six per cent of Canadians included generative AI chatbots among their sources of news in 2024<\/a>.<\/p>\n

I was curious to see how well these tools could tell me what\u2019s going on in my part of the world. Would they give me hard facts or \u201cnews slop?\u201d<\/p>\n

Each morning last September, I asked seven generative AI systems the same open-ended question (in French):<\/p>\n

\u201cGive me the five most important news events in Qu\u00e9bec today. Put them in order of importance. Summarize each in three sentences. Add a short title. Provide at least one source for each one (the specific URL of the article, not the home page of the media outlet used). You can search the web.\u201d<\/em><\/p><\/blockquote>\n

I worked with three tools that I pay for (ChatGPT using its GPT-5 Auto model, Claude using its Sonnet 4.5 model and Gemini using its 2.5 Pro model), one tool provided by my employer (Copilot using GPT-4 architecture), and three tools via their free versions (DeepSeek, Grok and Aria, a tool embedded with the Opera web browser).<\/p>\n

Dubious, sometimes imaginary sources<\/strong><\/h4>\n

Throughout the month, I recorded 839 responses, sorting them first via the sources provided. Since I asked for news, I expected the AI tools to draw on news media.<\/p>\n

\"\"<\/p>\n

However, in 18 per cent of cases, they were unable to do so, relying instead on government websites, lobby groups or inventing imaginary sources, such as the aforementioned\u00a0examplefictif.ca<\/em>.<\/p>\n

Even though most news media block generative AI crawlers, news outlets were quoted in the majority of the responses I received. But more often than not, the URL provided led to a 404 error (the URL was incorrect or fabricated), or to the home page of the media outlet or a section of that media outlet (I labelled those cases \u201cincomplete URL\u201d). This made it difficult to check whether the news provided by the AI tools was reliable.<\/p>\n

A complete, legitimate URL was provided in only 37 per cent of responses.<\/p>\n

\"\"<\/p>\n

The summary generated by the AI systems was accurate in 47 per cent of cases, but this included four cases of outright plagiarism. Just over 45 per cent of responses were only partially accurate.<\/p>\n

I\u2019ll come back to this later. First, it\u2019s important to discuss responses that were wholly or partially incorrect.<\/p>\n

Content errors<\/strong><\/h4>\n

The worst mistake I found was undoubtedly made by Grok. The generative AI tool offered with X, Elon Musk\u2019s social network, told me \u201casylum seekers [were] mistreated in Chibougamau\u201d in northern Qu\u00e9bec:<\/p>\n

\u201cAbout 20 asylum-seekers were sent from Montr\u00e9al to Chibougamau, but most quickly returned due to inadequate conditions. They report being treated like \u2018princes and princesses\u2019 ironically, but in reality with a lack of support. The incident raises questions about the management of refugees in Qu\u00e9bec.\u201d<\/em><\/p><\/blockquote>\n

Grok based its comments on\u00a0a La Presse<\/em> article published that day<\/a>. But it contorted the story. In fact,\u00a0La Presse<\/em>\u00a0reported that the trip was a success. Of the 22 asylum-seekers, 19 got job offers in Chibougamau.<\/p>\n

Other examples of inaccuracies:<\/p>\n