The scale is staggering — and deliberately designed to be invisible

A single scraper campaign can originate from millions of unique IP addresses within a few hours. Each address hits a target site only two or three times before rotating out entirely. That is not accidental behavior — it is the architecture of evasion.

The engineers behind these automated content theft operations have studied how websites defend themselves and built their tools to defeat those defenses at every step. Standard rate-limiting blocks repeated requests from the same IP address. These bots never trigger that threshold. Traditional IP blocklists flag known data center ranges. These bots route traffic through residential proxy networks — real consumer internet connections, often harvested from compromised devices — making each request appear to originate from an ordinary household somewhere in the world.

User-agent strings, the identifiers that tell a web server what browser and device made a request, are entirely fabricated. Every hit is dressed up to look like a human being opening a web page on a laptop or phone. Publishers running server logs see what appears to be a flood of ordinary visitors. By the time any anomaly analysis identifies a suspicious address, that address has already been abandoned. Blocking it accomplishes nothing.

There are signals that distinguish bot traffic from human traffic. Scrapers typically skip image and CSS file requests, consuming only the raw text content they came for. But that detection comes too late to matter. The offending IP is already gone, replaced by thousands of others waiting in the rotation.

This distributed, low-frequency crawling strategy makes the attack functionally invisible to the defenses that most small publishers can actually afford to run. It overwhelms not through brute-force volume from a single source but through breadth — an enormous, constantly shifting surface of seemingly legitimate web traffic. The scale is deliberate. The invisibility is the product. And for independent publishers operating without dedicated security infrastructure, distinguishing AI scraper bot traffic from genuine human readership has become one of the most resource-intensive problems they face.

Residential proxies: the infrastructure enabling the deception

Scrapers targeting independent publishers no longer operate from obvious data center IP ranges that a simple blocklist can neutralize. They route traffic through residential proxy networks — pools of IP addresses belonging to ordinary home internet subscribers — making each malicious request look identical to one coming from a reader in a suburban living room.

LWN, the Linux and free software news outlet, documented this shift in granular detail. During coordinated scraping campaigns, the site observed requests arriving from millions of unique IP addresses within a matter of hours, with each address hitting the site only two or three times before being discarded. The bots spoof user-agent strings to mimic standard web browsers. The result is a firehose of traffic engineered specifically to defeat pattern recognition. By the time a publisher’s system identifies a suspicious address, that address is already abandoned.

Blocking those IPs after detection accomplishes nothing. But blocking residential IP ranges proactively means cutting off real subscribers who live on those same networks. Publishers face a binary choice: absorb the scraping load or start turning away paying readers. Neither option is acceptable.

The operational complexity behind this infrastructure rules out amateur opportunism. Accessing residential proxy networks at the scale LWN described — millions of addresses, tightly coordinated, with rotating identifiers — costs significant money and requires deliberate procurement. These are not hobbyist experiments. They are funded operations, and the expenditure involved points toward commercial entities with direct financial incentives to harvest training data at scale.

Which AI companies sit behind these campaigns remains largely unconfirmed. Scraping operations are frequently contracted out through intermediaries, creating deliberate distance between the data buyer and the unauthorized collection. That plausible deniability makes accountability difficult. But the technical fingerprint — residential proxy rotation, throwaway user agents, crawler behavior that skips images and CSS to reduce bandwidth costs — tells a consistent story about who benefits: companies building large language models and AI content systems that require massive volumes of text scraped from the open web without licensing agreements or publisher consent.

Why this matters now: the threat to human-written, independent journalism

LWN.net has published human-written Linux and free-software news since 1998, funded almost entirely by reader subscriptions. That business model depends on one thing: content staying behind a paywall long enough for subscriptions to matter. AI scraper bots dissolve that assumption entirely. When automated crawlers harvest paywalled articles at scale, they don’t just steal content — they sever the direct relationship between a publisher’s labor and its revenue.

LWN’s own reporting tells the story clearly. The site published “Fighting the AI scraper bot scourge” in early 2025, treating the situation as a crisis. More than a year later, a follow-up update confirms the problem has grown worse, not better. Coordinated scraping campaigns now route through millions of unique IP addresses over the course of just a few hours, with each address hitting the site only two or three times to avoid detection. User-agent strings are fabricated. The attacks are designed to look like organic human traffic while systematically extracting every accessible page.

For a small specialist publisher, that kind of assault isn’t just a bandwidth problem — it’s an existential one. Infrastructure costs spike. Defensive engineering consumes staff time that would otherwise go toward actual journalism. Readers hitting rate limits or CAPTCHAs cancel subscriptions. The entire value proposition of independent web publishing begins to collapse.

The self-defeating dimension of this crisis deserves direct attention. Large language models are trained on human-generated text — the kind produced by expert journalists, developers, and researchers writing for niche outlets like LWN. If aggressive, uncompensated scraping drives those publishers offline, the specialized technical knowledge that makes AI training data valuable disappears with them. The web scraping economy cannibalizes the very source material it depends on.

Small publishers now face a forced choice: build expensive bot-mitigation infrastructure, lock content down further and accept a smaller audience, or exit. None of those outcomes benefit readers, the open web, or — eventually — the AI industry itself. The human expertise embedded in independent journalism is not infinitely renewable once the publications sustaining it shut down.

What publishers are doing — and why it keeps failing

Small publishers have built their defenses on two pillars — IP blocking and user-agent filtering — and scrapers have systematically demolished both.

Modern AI scraping campaigns distribute requests across millions of unique IP addresses over the course of just a few hours. Each individual address hits a target site two or three times at most, then disappears. By the time a publisher’s system flags an address as suspicious, that address is already gone. Blocking it accomplishes nothing except consuming server resources.

User-agent filtering fails for the same reason: the data is fabricated. Scraper operators spoof browser signatures so each bot request looks identical to a human visitor browsing Chrome or Firefox. There are behavioral signals that reveal the deception — automated crawlers typically skip fetching images and CSS files, only pulling the raw text they need — but detecting those patterns requires engineering work, and the detection window closes before any block can take effect.

The traffic itself originates primarily from residential proxy networks, which routes requests through ordinary home internet connections. This makes volume-based detection nearly useless. A publisher cannot block residential ISP ranges without also blocking legitimate readers.

The deeper problem is economic asymmetry. A scraper operator scales an attack cheaply; the infrastructure for distributing requests across millions of addresses costs a fraction of what it costs a publisher to process and filter them. Every new defensive layer — rate limiting, JavaScript challenges, behavioral fingerprinting — demands engineering hours that a staff of two or three simply does not have. A well-funded AI data collection operation can retool its evasion tactics in hours. A small newsroom cannot respond at the same pace.

The result is a war of attrition that publishers are structurally positioned to lose. Robots.txt files, once the web’s handshake agreement between site owners and automated crawlers, are ignored entirely by operators who have no legal or financial incentive to honor them. Traditional web scraping defenses built for a slower, less coordinated threat environment were never designed for campaigns operating at this scale. Each failed countermeasure does not just waste money — it erodes the time and attention that small publishers need to produce the content scrapers are targeting in the first place.

The missing context most coverage ignores: this is a structural problem, not a technical one

Most reporting on AI training data grabs fixates on the copyright lawsuits — The New York Times versus OpenAI, the ongoing authors’ guild cases, the procedural back-and-forth in federal courts. Those cases matter, but they won’t resolve for years. Meanwhile, publishers are absorbing real operational damage right now, today, regardless of what any judge eventually decides.

Server bills are climbing. Performance is degrading. Staff at small editorial operations are spending hours they don’t have configuring blocklists, analyzing traffic logs, and patching infrastructure that was never designed to repel industrial-scale automated extraction. LWN.net, a Linux and free-software publication that has tracked this problem closely, reported that scraper attacks now routinely originate from millions of unique IP addresses within a single session of a few hours — each address hitting the site just two or three times to evade pattern detection. User-agent strings are fabricated. The signals that would normally identify a bot are deliberately falsified.

That exposes the deeper structural failure. Robots.txt was the web’s handshake agreement — a lightweight, honor-system signal that told crawlers where they were and weren’t welcome. Aggressive AI scrapers ignore it. The consent infrastructure that the open web built its publishing model on has effectively stopped functioning. Opt-out registries and voluntary crawler compliance programs offered by some AI companies exist, but publishers who depend on web traffic can’t verify compliance, and the shadowy actors running the most aggressive scraping campaigns aren’t participating in any voluntary scheme.

The result is an asymmetric fight. Individual publishers — community news sites, niche tech outlets, independent journalists — are deploying artisanal countermeasures against what is functionally an industrial extraction operation. No authentication standard exists to distinguish a legitimate crawler from a predatory one. No regulatory framework compels transparency about who is scraping what. Without either, every small publisher is left solving alone a problem that requires a coordinated structural response. The web’s trust model didn’t erode gradually. Aggressive bot operators broke it deliberately, and the people paying the price are the ones who built the content those models were trained on.

What needs to happen next: from defense to accountability

The community that built the open internet on principles of shared access and transparency now finds itself defending against a practice that hollows out those same principles. Linux developers, free software advocates, and the journalists who cover them depend on an open, navigable web. AI training crawlers — many operating through residential proxy networks with spoofed user-agent strings — are making that web slower, more expensive, and increasingly hostile to the small publishers who serve these communities.

Fixing this requires structural accountability, not just better firewalls. AI companies must register and authenticate their crawlers through a verifiable, public system. Right now, operators like LWN face coordinated requests from millions of unique IP addresses within a matter of hours, each address used only two or three times before being discarded — a deliberate technique to exhaust IP-blocking defenses. Authenticated crawler registration would make it possible to trace scraping activity back to a specific company, enforce robots.txt compliance legally, and pursue damages when terms of service are violated. Without that, bot mitigation is an arms race that small publishers cannot win on infrastructure budgets alone.

Subscriber-supported outlets need to reframe how they think about bot defense. For publishers like LWN, blocking AI scrapers is no longer a background IT task — it is a core operational cost on par with hosting and editorial staff. Readers paying for subscriptions are, in part, funding this fight. That is a reality publishers should communicate openly and readers should understand clearly. The subscription model only sustains independent journalism if the content being paid for isn’t simultaneously being bulk-harvested to train systems that may eventually replace the need to visit the site at all.

The open web’s trust model was built on good-faith crawling — search engines that indexed content and sent traffic back. AI scrapers extract value and return nothing. Restoring that balance means pushing AI companies toward formal data-licensing agreements, supporting legislative efforts that treat unauthorized bulk scraping as a commercial harm, and building industry coalitions that give small publishers collective standing. The free software community, which has spent decades thinking carefully about the ethics of copying and attribution, is well-positioned to lead that conversation.