Wikipedia is struggling with voracious AI bot crawlers
Wikimedia has seen a 50 percent increase in bandwidth used for downloading multimedia content since January 2024, the foundation said in an update. But it's not because human readers have suddenly developed a voracious appetite for consuming Wikipedia articles and for watching videos or downloading files from Wikimedia Commons. No, the spike in usage came from AI crawlers, or automated programs scraping Wikimedia's openly licensed images, videos, articles and other files to train generative artificial intelligence models.
This sudden increase in traffic from bots could slow down access to Wikimedia's pages and assets, especially during high-interest events. When Jimmy Carter died in December, for instance, people's heightened interest in the video of his presidential debate with Ronald Reagan caused slow page load times for some users. Wikimedia is equipped to sustain traffic spikes from human readers during such events, and users watching Carter's video shouldn't have caused any issues. But "the amount of traffic generated by scraper bots is unprecedented and presents growing risks and costs," Wikimedia said.
The foundation explained that human readers tend to look up specific and often similar topics. For instance, a number of people look up the same thing when it's trending. Wikimedia creates a cache of a piece of content requested multiple times in the data center closest to the user, enabling it to serve up content faster. But articles and content that haven't been accessed in a while have to be served from the core data center, which consumes more resources and, hence, costs more money for Wikimedia. Since AI crawlers tend to bulk read pages, they access obscure pages that have to be served from the core data center.
Wikimedia said that upon a closer look, 65 percent of the resource-consuming traffic it gets is from bots. It's already causing constant disruption for its Site Reliability team, which has to block the crawlers all the time before they they significantly slow down page access to actual readers. Now, the real problem, as Wikimedia states, is that the "expansion happened largely without sufficient attribution, which is key to drive new users to participate in the movement." A foundation that relies on people's donations to continue running needs to attract new users and get them to care for its cause. "Our content is free, our infrastructure is not," the foundation said. Wikimedia is now looking to establish sustainable ways for developers and reusers to access its content in the upcoming fiscal year. It has to, because it sees no sign of AI-related traffic slowing down anytime soon. This article originally appeared on Engadget at https://www.engadget.com/ai/wikipedia-is-struggling-with-voracious-ai-bot-crawlers-121546854.html?src=rss
