boot13
In the never-ending search for content to feed into their Large Language Models, ‘AI’ companies are using distributed networks of bots (automated programs) to download the contents of web sites everywhere on the Internet. Many sites are being overwhelmed by this traffic. Some site owners are being forced to upgrade their hosting service, while others have simply shut down.
If your response to that is to point out that Google does the exact same thing, and has for years, you’d be right, except for a few important differences:
- Google’s searchbot doesn’t try to download every page on every web site. It’s much more selective about what it grabs.
- Googlebot clearly identifies itself, which makes it much easier to classify its requests and limit its impact. The vast majority of ‘AI’ scraper bots falsely identify themselves as ordinary web users.
- Google publishes a list of host IP ranges used by Googlebot, and they consist of specific, limited subnets that are clearly owned by Google. The vast majority of ‘AI’ scraper bots are spread around the world, on largely unrelated groups of hosts, often running on virtual private servers owned by VPS providers.
- When Googlebot crawls your site, one or two hosts read a small batch of related/linked pages, then go away and come back for more later. ‘AI’ crawlers spread their requests across hundreds or even thousands of hosts, each one reading as little as one page, which makes identifying and blocking them difficult.
All of these things make the job of managing traffic for a web site much more difficult than it was before the current ‘AI’ boom.
In my work on managing traffic for clients, I have been lucky in that two of them are not legally allowed to sell to anyone outside of Canada. For those clients, I eventually just blocked traffic from every country except Canada. Traffic from major search providers is always allowed, and special procedures are required for certificate updates, but this made a huge difference. Web sites that need traffic from other countries obviously can’t use this drastic approach.
Thankfully, all of this pointless and damaging traffic will stop when the current ‘AI’ bubble bursts, and that can’t come soon enough for me.
The Register talks about this, describing the new ‘AI’ crawlers as site destroyers, which is accurate.
Here’s Drew Curtis, the owner/operator of the long running humour site fark.com, talking about this problem.
The EFF posted about this problem and suggests some solutions.
There has been some progress on ‘local models’, which refers to running LLM ‘AI’ software locally (i.e. on your own computer). The idea is that if you can run an LLM on your own computer, you don’t have to pay for the use of commercial services. And local models do actually work, but with much less useful results. Your laptop just doesn’t have the processing power to produce truly useful results. According to Sean Goedecke, this is unlikely to change, even with significant increases in processing power. So the ability to run local models will not have any effect on the deluge of traffic hitting web sites.
In the small dialog that appears, enter a name for the new search (I call it ‘Google Classic’), the URL 
First up, there’s been an 
Meanwhile, Google is planning to 
) to