The internet is being rewritten by machines.
Government data is the raw material. Who gets paid?
AI agents, autonomous scrapers, and data-hungry models are visiting government websites at exponentially growing rates. This isn't the future — it's happening now. The question isn't whether to allow it. It's whether to get paid for it.
In 2020, bot traffic was mostly search engines indexing pages. In 2026, it's autonomous agents retrieving data, filling forms, and executing tasks — at machine speed, around the clock.
The traffic visiting your website has changed.
Your model for it hasn't.
In 2020, most bot traffic was search engines indexing pages. Harmless, even helpful. Google's crawler made your records findable. Bing's made them reachable. The deal was clear: you let them crawl, they sent you human visitors. Everyone benefited.
In 2024, AI companies began scraping at unprecedented scale to train models. The traffic looked similar to search engines — same protocols, same patterns — but the purpose was different. They weren't sending you visitors. They were extracting raw material.
In 2026 and beyond, autonomous AI agents are visiting sites to retrieve data, fill forms, answer questions, and execute tasks — all at machine speed, around the clock. They don't read pages. They pull structured records. They don't browse. They harvest.
Government websites are prime targets: they hold authoritative, structured, public data that AI agents need. Property records. Permits. Meeting minutes. Budgets. The raw material of civic life.
This traffic will not decrease. It will multiply. The infrastructure costs will grow. Without a plan, municipalities pay more and earn nothing.S-116
The instinct is to fear it.
The instinct is understandable. It's also insufficient.
When officials first learn that more than half their web traffic is automated, the reflex is defensive. Block it. Ignore it. Hope it stops. These reactions aren't stupid — they're human. But each one fails, and each one fails in a way that leaves the municipality worse off than before.
Block bots entirely
Breaks search engine indexing, accessibility tools, legitimate research, and journalism. Throwing out the good with the bad — and the bad bots ignore blocks anyway.
Ignore the problem
Infrastructure costs grow silently. Performance degrades for human residents. Staff time is consumed managing traffic spikes you never planned for.
Hope it goes away
It won't. AI adoption is accelerating, not slowing. Every quarter without a program is a quarter of free data extraction at your expense.
Robots.txt is a polite request, not an access control. The bad bots ignore it. The good bots honor it — which means blocking bots blocks the cooperative ones and leaves the extractive ones alone. RFC 9309 is explicit: "These rules are not a form of access authorization."S-101
Don't block the agents.
Build the tollbooth.
The right response isn't fear. It's infrastructure. Governments can build systems that identify bot vs. human traffic, require registration and API keys for automated access, charge fees for commercial automated access — like tolls on a highway — and keep access free for residents, journalists, and researchers. The technology exists. The legal precedent exists. What's been missing is the municipal will and the municipal build.
Turn a cost center into a revenue stream
Bandwidth, server load, and staff time currently consumed by bot traffic become a metered, billed service. The same access that cost you money starts paying for itself.
Control what automated systems can and cannot see
Decide which datasets are exposed to agents, at what rate, under what terms. Move from passive scraping to governed access — on your conditions, not theirs.
Protect resident data and privacy
Commercial access doesn't mean open access. Keep records free for residents, journalists, and researchers while gating bulk commercial extraction behind registration and fees.
Fund better digital services
Revenue from automated access programs can be reinvested in the civic technology most budgets can't afford — modern APIs, faster websites, better data infrastructure.
Set the terms of engagement
The AI economy's rules are being written now. Without a program, those rules get written for you — by the bots. A municipal program sets the precedent your community will live under.
Residents free. Commercial access paid.
A FOIL request and a bot crawling your website are not the same thing. FOIL gives every person the right to request specific records. It doesn't give commercial bots the right to bulk-download your entire site for free. Paid, registered, metered API access is an alternative to scraping — not a replacement for public records law.
Governments already charge for this.
Just not yours. Not yet.
Charging for automated access isn't a radical idea. It's standard practice at every level of government above the municipality. Federal agencies, state agencies, and even sister municipalities have built paid-access programs. The pattern is the same: free for the public, tiered fees for commercial use.
Charges $0.10/page for federal court records — automated access is paid, not free.
Three fee categories including a commercial use tier. Federal precedent for charging by purpose.
Sell bulk vehicle data to insurers, dealers, and data brokers under regulated terms.
Tiered API access — free public feeds plus paid high-volume commercial tiers.
Operates paid enterprise API tiers for commercial weather data consumers.
Free public access plus commercial redistribution licensing through the API program.
The precedent is everywhere above the municipal level. What's missing isn't the legal theory — it's the local infrastructure. Municipalities haven't built the gateways, the billing, or the legislation. PAID packages all three so a town or county can stand up a program in months, not years.
PAID doesn't just research the problem.
We build the solution.
Three pillars. Each one is a body of work a municipality can adopt, adapt, or hand to a vendor. Together they form a complete program — from the legal authority to charge, to the technical infrastructure that meters and bills, to the data quality that makes the access worth paying for.
Policy
Model legislation, legal frameworks, cost-recovery and revenue models. The legal scaffolding that makes paid automated access lawful, defensible, and FOIL-compatible.
Infrastructure
API gateways, bot management, metered access, billing systems. The technical scaffolding that identifies, routes, and charges for automated traffic.
Data Readiness
Data inventory, format conversion, API publication, quality control. The work that makes your public data worth paying for — clean, structured, machine-readable.
The window to lead is open.
It will not stay open.
Every month without a program, a municipality is subsidizing commercial data extraction with its own infrastructure budget. Bandwidth, server load, staff time — all paid by the taxpayer, all consumed for free by commercial extractors.
The first municipalities to adopt paid access programs will set the standard. They'll have the most experienced programs, the most refined fee structures, and the strongest legal precedent. Late adopters will be working from their playbook.
The window to lead is now — before the AI economy's norms are set by the bots, not by the governments.
The agent economy is the most significant change to web traffic in a generation. Treated as a threat, it drains budgets. Treated as an opportunity, it funds the civic technology most municipalities can't otherwise afford. The difference is infrastructure and policy — both of which can be built.
- Bot traffic grows every quarter — infrastructure spend grows with it
- AI training corpora are being built now; data extracted free today won't be paid for retroactively
- Fee structures and legal frameworks set early become the template others copy
- Resident-facing performance degrades as bot load rises without management
The agents are coming.
Will you profit, or will you pay?
The research lays out the landscape. The services lay out the build. The choice — whether your municipality treats the agent economy as a cost or a dividend — is yours.