Charging for the Bots

New York municipalities publish vast amounts of public information on their websites — meeting minutes, budgets, property records, permits, contracts, and notices. Under the NY Freedom of Information Law (FOIL), any person can request copies of these records at low statutory fees ($0.25 per page for paper copies, or the actual cost of reproduction for electronic records) S-011, S-202. But the same information is also being harvested at high volume by automated software programs — “bots,” “crawlers,” or “scrapers” — that visit municipal websites at machine speed, downloading thousands or hundreds of thousands of pages, consuming server capacity and bandwidth, and reselling or monetizing the data for commercial purposes. Industry research indicates that bots account for approximately 53% of all web activity, and “bad bots” alone make up roughly 40% of all traffic S-116.

This paper examines whether and how a New York municipality can lawfully establish a paid automated-access program — a registered, key-based, rate-limited system (typically an API, or Application Programming Interface) through which commercial bots pay a fee tied to the actual cost of providing that access, while residents, journalists, and researchers continue to access the same public information for free through the regular website and through FOIL.

The research finds:

1. A municipality likely has the legal authority to adopt a local law creating such a program, under NY’s home rule provisions (NY Constitution Article IX; Municipal Home Rule Law § 10) S-002, S-012. The program would be a voluntary, alternative, premium channel that supplements — and does not replace — free public website access and free FOIL access S-011.
2. The critical legal distinction is between a FOIL request (a formal, written demand for records that triggers statutory deadlines and fee caps) and passive website access (reading already-published information). A bot crawling a public website is not, by that act alone, making a FOIL request S-011.
3. There is a “public access floor” the municipality cannot go below. A municipality cannot eliminate free public access to records that FOIL requires be available, nor charge more than FOIL permits for records obtained through FOIL S-011, S-202.
4. Fees should be structured as cost recovery, not revenue generation. Every government fee statute examined — federal FOIA S-001, S-201, NY FOIL S-011, S-202, the federal PACER court-records system S-204, S-205, and the E-Government Act of 2002 S-016 — is built on a cost-recovery standard.
5. The technical tools exist and are affordable. Cloudflare offers free bot management S-110. AWS WAF Bot Control starts at $10/month S-115. Open-source API gateways like Kong provide authentication, rate limiting, and usage logging at no software cost S-106.
6. Realistic revenue is modest. A small/medium NY municipality (population 25,000–150,000) should expect, conservatively, $500–$75,000/year in gross revenue, with net revenue likely $0–$40,000/year after program costs.
7. The data must be cleaned and structured first. A paid automated-access tier only makes sense after the municipality has inventoried its data, converted it to machine-readable formats (CSV, JSON, GeoJSON — not PDFs), published it with metadata, and established reliable update cadences S-301.
8. Several significant legal questions remain unresolved and require review by qualified municipal counsel before any ordinance is enacted S-011, S-015.

This paper synthesizes legal, technical, economic, and data-organization research into a single coherent document. It is accompanied by standalone companion deliverables: a model ordinance, an implementation playbook, and a data organization guide. The recommended path is to start with cost recovery, a free resident tier, clean data first, and a phased rollout over approximately 12 months.