The internet is in its guano years

On November 13, Reddit is turning off RSS.

If you’ve never used it, RSS is the plumbing that lets a program ask a website “anything new?” and get a tidy list back. Netscape shipped the first version in 1999. It’s exactly the kind of boring, open technology the web runs on (your podcast app almost certainly depends on it).

Reddit announced the shutdown on September 30, along with the end of its public API in March 2027, and the reason it gave was that RSS had become a “common surface for large-scale scraping.”

Reddit didn’t say who was doing the scraping. It didn’t need to. It has two lawsuits going right now, one against Anthropic and one against Perplexity and three scraping companies, and both are about AI companies taking Reddit posts.

I’d already bumped into this in a small way. When I drafted Reddit posts for a couple of my articles, I wanted each subreddit’s rules in front of me, and my agent couldn’t read them. Reddit blocks automated reads, so it worked from third-party snapshots instead. A site made of people talking to each other doesn’t want machines reading along anymore, unless they pay.

Meanwhile, this site’s robots.txt is four lines long, and the second one is Allow: /. Every crawler is welcome here, the AI ones included. So I’m on both sides of this thing: I give my words away on purpose, and the platforms I’d share them on are busy putting up fences and price tags.

I wanted to understand what’s actually going on, so I went through the numbers: API prices, licensing deals, lawsuits, and some very strange new jobs. The more I read, the more it reminded me of bird poop.

Bird droppings worth a war

In the 1840s, the most valuable thing in Peru was bird droppings.

The Chincha Islands, off Peru’s coast, almost never get rain, so the droppings of seabirds had piled up there for centuries without washing away. Guano turned out to be a superb fertilizer, full of the nitrogen European farms were short of. Peru shipped 24,701 tons of it in 1845 and 316,116 tons in 1853, by which point guano was about three-quarters of everything the country exported. For decades, guano was the Peruvian government’s main source of money.

The rush reached Washington. In 1856 the US Congress passed the Guano Islands Act, which let any American citizen claim any unclaimed island with guano on it. Dozens were claimed, and a few, Midway among them (yes, that Midway), are still American today. In 1864 Spain seized the Chincha Islands outright, and a war followed. In 1879 Chile went to war with Peru and Bolivia over the desert next door, which was full of nitrate, the other big natural source of the same nitrogen. Bolivia lost its coastline.

Deposits that took centuries to build were mostly dug out in about forty years, and the guano boom was over by 1880. The world moved on to Chile’s nitrate mines.

Then, in 1909, a chemist named Fritz Haber showed the company BASF that he could make ammonia from the nitrogen in the air. BASF opened a plant in 1913, and by 1933 Chile’s nitrate cartel was gone. The scarce thing people had fought wars over became something a factory made.

For about twenty years, we did the bird part. We answered questions on Stack Overflow, argued on Reddit, wrote threads on Twitter and left reviews everywhere, and it all piled up on platforms that mostly used it to sell ads. Nicholas Carr saw the shape of this in 2006 and called it digital sharecropping. “It’s a sharecropping system, but the sharecroppers are generally happy,” he wrote. We were, mostly. It was fun.

Then ChatGPT came out in November 2022, and the droppings became fertilizer.

nitrogen, 1840–1933human data, 2004–?

1. 1A byproduct piles up

   Then: Seabird droppings build up for centuries on Peru’s rainless Chincha Islands.

   Now: Twenty years of posts, answers and reviews build up on Reddit, Stack Overflow and Twitter.

2. 2Someone finds out what it’s worth

   Then: Peru ships 24,701 tons of guano in 1845 and 316,116 in 1853. In 1856 the US lets its citizens claim any unclaimed guano island.

   Now: ChatGPT ships in November 2022. Within five months, Twitter and Reddit both say they’ll charge for their APIs.

3. 3Fences, and fights

   Then: Spain seizes the Chincha Islands in 1864. In 1879 Chile goes to war with Peru and Bolivia over nitrate fields.

   Now: Reddit sues over scraping, X bans AI training on its posts, and Reddit is switching off RSS.

4. 4Someone makes it from air

   Then: Fritz Haber makes ammonia from air in 1909, and BASF opens a plant in 1913. Chile’s nitrate cartel is gone by 1933.

   Now: Labs already train on data that models wrote. Whether it can replace people is the open question.

The fences go up

Twitter went first, and it had been practicing. Back in August 2012 it capped every third-party Twitter app at 100,000 users. In February 2023, nine weeks after ChatGPT came out, it said it would end free API access. By May, the Basic tier cost $100 a month and Pro cost $5,000.

It kept going. Basic doubled to $200 in October 2024. In March 2025, xAI, Elon Musk’s AI company, bought X in a deal that valued it at $33 billion, and two months later X changed its developer terms to forbid using its posts to train AI models.

In February 2026, X moved its API to pay-per-use, with free access only for “public utility” apps. That same month, SpaceX merged with xAI, so Twitter is now, technically, part of a rocket company. I promise I’m not making that up.

Reddit’s turn came in April 2023. The company said it would start charging for its API, and CEO Steve Huffman explained why to the New York Times: “The Reddit corpus of data is really valuable.” The prices came out at the end of May, and Christian Selig, who made the popular Apollo app, did the math in public. Apollo made about 7 billion requests a month. At $0.24 per 1,000 calls, that came to about $20 million a year, for one app.

More than 7,000 subreddits went dark in protest on June 12. Apollo shut down on June 30 anyway. Eight months later, Reddit’s IPO filing listed $203 million in data licensing contracts, and Reuters reported that the Google deal alone was worth about $60 million a year. OpenAI signed with Reddit in May 2024.

The data money is small, though. Reddit’s “other revenue,” the line where licensing lives, was $140 million in 2025, out of $2.2 billion in total. That’s about 6%. It’s growing (up 24% from a year earlier in the second quarter of 2026), but the ads still pay the bills.

My read is that the fences are about the next negotiation. In July, the Wall Street Journal reported that Reddit had weighed cutting off Google’s AI access while the two renegotiate. Ten weeks later, Reddit announced it was ending RSS.

Allclosedsoldsued

1. 2012
   1. AugclosedTwitter caps third-party apps at 100,000 users each.
2. 2023
   1. FebclosedTwitter ends free API access. By May, Basic costs $100 a month and Pro $5,000.
   2. AprclosedReddit says it will charge for its API. Apollo’s bill comes to about $20 million a year.
   3. JunclosedMore than 7,000 subreddits go dark in protest. Apollo shuts down on June 30.
3. 2024
   1. FebsoldReddit and Google sign a deal, reported at $60 million a year. Reddit’s IPO filing lists $203 million in data contracts.
   2. MaysoldOpenAI signs with Reddit, Stack Overflow and News Corp (reported at $250 million over five years).
4. 2025
   1. MarsoldxAI buys X in a deal that values X at $33 billion.
   2. JunclosedX’s developer terms ban training AI models on X content.
   3. JunsuedReddit sues Anthropic, alleging its bots kept scraping the site.
   4. JunsoldMeta pays $14.3 billion for 49% of Scale AI, a data-labeling company.
   5. OctsuedReddit sues Perplexity and three scraping companies, alleging they took Reddit posts through Google’s results.
   6. NovclosedReddit ends self-serve API keys. New developers need approval.
5. 2026
   1. FebclosedX moves its API to pay per use. Free access only for “public utility” apps.
   2. JulsoldAnthropic’s $1.5 billion settlement with authors gets final approval: about $3,100 per book.
   3. SeptclosedReddit says RSS feeds end on November 13 and the public API ends in March 2027.

What a sentence costs

Put the prices in one place and they read like a receipt from a very strange store.

THE PRICE OF WHAT PEOPLE WROTE
itemized, 2023–2026

1. 1,000 Reddit API calls (2023 price)$0.24
2. An hour of an expert’s time (Mercor’s average rate)$103
3. A month of X’s Basic API (2024 price)$200
4. One book (Anthropic’s settlement, per work)$3,100
5. An airline’s emails and Teams chats (Spirit, proposed)$10M
6. A year of The New York Times, for Amazon^r$20–25M
7. A year of Reddit, for Google^r$60M
8. Five years of News Corp, for OpenAI^r$250M+
9. 49% of Scale AI, for Meta$14.3B
10. All of X, folded into xAI$33B

Reddit’s data revenue, 2025$140M

Paid to the people who wrote the posts$0.00

^r reported, never confirmed by the companies

A few of these need a story.

The book price comes from Anthropic’s $1.5 billion settlement with authors. A judge had ruled in June 2025 that training on books Anthropic bought was fair use, but building a library out of pirated copies wasn’t. The settlement covers 482,460 works at about $3,100 each, and it got final approval on July 20. The first checks are expected by mid-November.

The airline is Spirit, which is in bankruptcy. It has proposed selling about 100 million emails and 500 million Microsoft Teams messages to Google, to train AI, for $10 million. Employee records and payroll data are on the list too. Rep. Steven Horsford and Sen. Elizabeth Warren asked both companies to hold off until the workers are protected.

And the $0.00 line is the deal we all accepted when we signed up. You post for free, and Reddit gets a license to what you posted.

The open web has its own version of this. For decades, the deal with search engines was simple: let them crawl your pages, and they’ll send people back. In July 2025, Cloudflare measured how well that deal still holds.

Google still sends about one visitor for every five or so pages it crawls. The AI companies were taking far more than they sent back.

And Google’s half of the deal is fading too. Pew found that people who saw an AI summary at the top of a Google search clicked a regular result on 8% of visits, against 15% without one, and they clicked a link inside the summary only 1% of the time. SparkToro estimates that 68% of US Google searches in early 2026 ended without a click at all.

So the web started putting up its own fences. Between 2023 and 2024, robots.txt rules shut AI crawlers out of at least 28% of the most important sources in a widely used training dataset, according to the Data Provenance Initiative.

On July 1, 2025, Cloudflare started blocking AI crawlers by default on new sites and launched a way to charge them per crawl. Since September 15, it also blocks training and agent bots by default on pages that carry ads, for new sites and for free accounts that haven’t changed the setting.

The deposit is running out

The guano took about forty years to dig out. The data version is going faster, and Stack Overflow is the clearest chart I found.

Stack Overflow peaked at 207,244 new questions in March 2014. In December 2025 it got 3,323, fewer than in August 2008, its first month. May 2026 had 1,342. That’s a drop of more than 99%.

Ouch.

People stopped asking Stack Overflow and started asking chatbots that learned, in part, from Stack Overflow. The company licensed its archive to Google and OpenAI in 2024, and last December it renamed that business Stack Data Licensing. The archive is still valuable. Hardly anyone is adding to it.

Zoom out and it’s the same story at web scale. In 2024, Epoch AI estimated that all the public text humans have written adds up to about 300 trillion tokens, and that at the current pace models would use all of it somewhere between 2026 and 2032.

The new deposits are thinner, too. In August, Pew estimated that about one in ten English web pages in a July 2026 sample showed signs of being written or substantially edited by AI, and more than a third of pages published since ChatGPT did. Graphite’s count puts AI at about half of new online articles.

Can’t the models just train on their own writing? Partly. Microsoft’s Phi-4 got 40% of its training data from synthetic text. But a 2024 paper in Nature showed that models trained over and over on their own output lose the rare, unusual cases first, and then the defects pile up. Other researchers found that mixing synthetic data in with real data, instead of replacing the real data, avoids most of that. So for now, fresh human data is the thing the synthetic data gets mixed into, and there’s less of it every month.

When the old deposit runs low, buyers go looking for new ones. The biggest one left is the work people do every day.

Your job is the next deposit

In the late 1890s, Frederick Taylor timed men loading pig iron at Bethlehem Steel. His idea, laid out in The Principles of Scientific Management in 1911, was that managers should gather up the knowledge that had always lived in workers’ heads and hands, and reduce it “to rules, laws, and formulae.” Once the knowledge was written down, it belonged to the company, and anyone could be taught it.

If you’ve ever written an onboarding doc so the next person wouldn’t have to ask you, you’ve done a small version of this, for free.

AI labs are doing a version of that now, and they pay well for it. Mercor, which hires professionals to write and grade the examples models learn from, was valued at $10 billion in October 2025 and was reportedly in talks at $20 billion this July. Its site says it pays out more than $4 million a day, at an average rate of $103 an hour.

Mercor isn’t alone. Handshake AI’s training work hit about $1 billion in gross annualized revenue in April. Micro1 hit $500 million in August and raised money at a $4 billion valuation in September. Doctors, lawyers, bankers and programmers are getting paid, often very well, to write down how they do their jobs.

In January, Wired reported that OpenAI and Handshake AI had asked contractors to upload real work from their past and current jobs: Word documents, PDFs, spreadsheets, slide decks, code repositories. Contractors were asked to scrub them first with a ChatGPT tool. OpenAI declined to comment.

When the people aren’t available, the leftovers get sold. Spirit is one example. This month, MIT Technology Review reported that AI labs are buying Slack, email and code archives from startups that have shut down, and that one wind-down service, SimpleClosure, has done more than 150 of these deals. Reuters reported in April that Meta was logging its own employees’ keystrokes, mouse movements and screenshots to train agents. The program was reportedly suspended around September after a data-exposure problem.

So a day of work now makes two things: whatever you were paid to make, and a record of how you made it. Until a few years ago, nobody would have paid for the second one.

The catch is in the shape. The money for writing a skill down is a one-time spike. Once a model has learned the routine part of a job, nobody needs to pay for that part again, from you or from anyone else.

For now, the people who work alongside the models are doing well. PwC’s 2026 AI Jobs Barometer, built on more than a billion job ads in 27 countries, found that jobs asking for AI skills advertise a 62% wage premium. That tells you who’s paid more this year, not what lasts.

My guess is that the durable part is whatever can’t be scraped once and reused. Firsthand experience is one: the model knows what was written down last year, not what happened in your hospital or your codebase this morning. Judgment someone has to sign for is another, because a model can draft the audit, but a person still gets sued. Trust is the third, the kind built between particular people over particular years.

If a skill can be written down as a checklist, assume someone is paying to write it down right now.

Four ways this could go

To guess where this goes, I put the possibilities on two axes: whether fresh human data stays scarce or synthetic data gets good enough, and whether data ends up fenced and priced or open and free.

2010202320262031, a guess

fenced and pricedopen and free

The licensed webPlatforms meter access, publishers sign deals, collectives license in bulk. Human data has a price list.

Fences around an empty fieldPlatforms guard data that synthetic data made cheap, like Chile’s nitrate fields after 1913.

The scraping warEveryone wants the data and nobody can fence it well. Lawsuits, bot blockers and logins everywhere.

The exhaust eraData is a byproduct nobody pays for, and the APIs are free. The web of 2010.

the web

fresh human data stays scarcesynthetic data is good enough

Licensing deals, pay per crawl, lawsuits. Most of the big piles of human writing now have a price.

The licensed web. We’ve built this before. In 1914, a group of composers founded ASCAP because restaurants and hotels were playing their music without paying them. In 1917, the Supreme Court sided with them in a case about a New York restaurant, and Justice Oliver Wendell Holmes wrote, “If music did not pay, it would be given up.” A century later, radio stations still pay through collective licenses like that.

Data has the early pieces of the same thing. More than 1,500 organizations back RSL, a standard for licensing content to AI, including AP, the Guardian, Stack Overflow and Cloudflare. The UK’s Copyright Licensing Agency has announced a collective license for AI training. Wikimedia’s paid service now counts Amazon, Meta, Microsoft, Mistral and Perplexity as partners, alongside Google. As of September, though, no AI company had been publicly named as an RSL licensee.

The scraping war. Everybody wants the data and nobody can fence it well, so it goes to court and behind logins. That’s already happening. The New York Times and OpenAI filed for summary judgment in September. In July, a judge let most of Reddit’s case against Perplexity go ahead. Wikimedia says it now blocks or throttles about 30% of automated requests, around 1.5 billion a day. The cost of that future falls on whoever runs the servers, and on anyone who liked the web open.

Fences around an empty field. This is the Haber moment, with the fences still standing. If synthetic data gets good enough, the platforms keep guarding something nobody needs, like Chile’s nitrate fields after 1913. I’d watch the share of synthetic data in training. It’s already 40% for at least one smaller model, Phi-4, and the big labs don’t publish theirs. I wouldn’t bet on this one before 2031, though, because the research so far says synthetic data works best mixed into fresh human data, not instead of it.

The exhaust era, again. Synthetic data wins and the fences come down too, so data goes back to being a byproduct nobody pays for. That’s where we were in 2010, and I don’t see a path back to it unless synthetic data wins outright and the lawsuits all go the AI companies’ way.

Paying people directly for their data has been proposed for years. Jaron Lanier and Glen Weyl suggested union-like groups that would bargain over people’s data in Harvard Business Review in 2018, and California Governor Gavin Newsom proposed a “data dividend” in his 2019 State of the State address. I couldn’t find a bill, a program or a single payment that came out of either.

That matches the guano age, when Peru’s government and its contractors kept about 71% of the final sale price. The birds got nothing. When creators did get paid, as composers did through ASCAP, it was because they organized.

My read of the trajectory is the “2031, a guess” setting in the chart above: the market splits in two. The archive gets cheap. That’s everything written before about 2023, which the models have already absorbed and synthetic data can increasingly stand in for. Fresh, firsthand, verified-human data keeps getting fenced and priced, because it’s the one thing a model can’t make out of what it already knows. If that’s right, the most valuable data in 2031 is whatever happened this week, written down by the person it happened to.

Back to RSS

Reddit’s feeds go dark on November 13. If your words only live on someone else’s platform, they’re part of someone else’s deposit, and the owner decides what they’re worth and who gets to read them.

I keep my writing here first, on a domain I pay for, in plain files I can move anywhere. If you’ve been writing for years on platforms you don’t control, it’s worth exporting a copy this week, while the export buttons still work.

This site’s robots.txt still says Allow: /, which, after everything above, is either generous or naive. I haven’t decided which. What does yours say, and did you pick it or did your host? Tell me in the comments.

Keep reading.

All articles

• Article · 15 min readThe compiler was supposed to end programming tooIn 1954, IBM said FORTRAN would virtually eliminate coding and debugging. What the numbers say about AI and software work in 2026, and my guess at a developer's Tuesday in 2031.Read article
• Article · 22 min readWho holds the off switch?The same coffee, bought in five kinds of society in 2035, and who gets to know about it. What the 2026 numbers say about privacy and control, and my guesses for where the privacy business goes next.Read article
• Article · 7 min readScoring 131,072 universesHow Rule Safari finds the interesting Life-like rules without being told which ones are famous, and the twelve-generation trick that made it work.Read article

A thought to add?

A good question, another perspective, or something this made you think about. There’s room for it here.

Loading the conversation…

Display name

Use a name or an alias. No email address needed.

Your comment

Leave this empty

Comments are reviewed before appearing. Your display name and message will be public once approved. Privacy

Submit comment