Exactly one year ago today, on September 10, 2025, a consortium led by Eckart Walther (co-creator of the RSS format) and Doug Leeds (former CEO of Ask.com) launched a standard with a simple goal: give robots.txt the nuance it has lacked since 1994. Until now, you had two options when facing an AI crawler — allow it or block it. RSL (Really Simple Licensing) adds a third path: allow it, provided it respects licensing terms, which can include payment. A year later, the protocol has become an official specification, and a handful of major publishers have actually put it into production. Here's what it does, what it doesn't do yet, and whether it makes sense for a site like yours in 2026.
The problem robots.txt doesn't solve
The classic robots.txt file (RFC 9309) is binary: Allow or Disallow, per user-agent. It says nothing about what a crawler is allowed to do with content once it has fetched it — use it to train a model, generate a one-off answer, or resell it to a third party. That's exactly the gap publishers started filling by hand back in 2023-2024, bluntly blocking GPTBot altogether — often at the cost of also losing eligibility for citations in ChatGPT or Perplexity, for lack of distinguishing training crawlers from citation crawlers (see our guide on selectively blocking AI crawlers).
RSL formalizes that distinction and goes further: it lets you attach economic conditions to access. The 1.0 specification reached official Recommendation status on December 10, 2025, backed by publishers including Reddit, Yahoo, Medium, O'Reilly, BuzzFeed, USA Today, and Vox Media, with Cloudflare and Akamai committing to implement it at the infrastructure level.
How it works technically
RSL adds a new directive to robots.txt, independent of the usual User-agent blocks:
License: https://yoursite.com/license.xml
User-agent: Googlebot
Allow: /
The License: directive points to an XML file (license.xml) describing usage terms: what a crawler is allowed to do (search, generative answering, model training), and under what financial conditions. The standard supports several models: free, mandatory attribution, subscription, pay-per-crawl, or pay-per-inference (billed each time an AI actually reuses the content in an answer). A site can declare multiple licenses — for example, a free one for standard indexing and a paid one for training.
On paper, it's elegant. In practice, just like robots.txt itself, nothing forces a crawler to read this file or honor the terms it contains — RSL remains a declaration of intent, not a technical lock. Its strength depends entirely on how many players (both crawlers AND infrastructure providers) choose to enforce it.
RSL, TDMRep, Google-Extended, llms.txt: don't mix up the mechanisms
These four tools now coexist in the landscape, and they don't do the same thing:
| Mechanism | Where | What it does |
|---|---|---|
Disallow (robots.txt) | Domain root | Blocks or allows access, unconditionally |
| Google-Extended | robots.txt | Specifically blocks training of Google's models (Gemini) |
| TDM Reservation Protocol | robots.txt + metadata | Reserves your text and data mining rights (EU framework) |
llms.txt | Domain root | Points AI systems to a condensed, readable version of your content (see our llms.txt guide) |
| RSL | robots.txt + license.xml | Adds usage AND payment conditions, negotiable by use type |
RSL is the only one of the five to introduce an explicit monetary dimension. To go further on the distinction between training crawlers and citation crawlers that underpins much of this mechanics, our complete guide to robots.txt for AI bots breaks down bot by bot who does what.
What's actually in production, not just announced
Plenty of articles list RSL adopters by simply quoting press releases. We checked directly in public robots.txt files instead. Result, as of this writing: medium.com/robots.txt does contain the line License: https://medium.com/license.xml — the directive is live, not just announced. On the other hand, a publisher cited among the protocol's earliest backers, Yahoo, has no visible License: directive in its public robots.txt as of this date. The gap between "signed the launch press release" and "actually published the directive" is real — check for yourself before citing a site as an RSL reference in a case study or a sales pitch.
This check takes thirty seconds: open https://domain.com/robots.txt and look for the License: line. It's the only test that matters, far more reliable than a row of logos on a press page.
Should you implement it on your site in 2026?
Be honest about what RSL can and can't get you today. It won't guarantee a check from the AI giants — the actual payment ecosystem (who pays whom, on what basis) is still embryonic, driven mostly by the RSL Collective and a handful of bilateral deals. What it does deliver right now: a public, machine-readable declaration of intent, useful if a regulatory or contractual framework shifts in your favor, and a clear signal sent to crawlers that already choose to honor it (Cloudflare bakes it into its bot management infrastructure).
It makes the most sense if you publish a meaningful volume of original, high-value content (research, proprietary data, analysis), you already have the bandwidth to track the protocol's evolution, or your host (Cloudflare in particular) offers a turnkey implementation. It makes less sense if your content is already widely duplicated elsewhere on the web, or if your priority remains visibility (AI citations) rather than monetizing crawl access — in that case, focus first on citability fundamentals rather than licensing.
How to add it to your site
- Decide your usage terms per crawler type (training, generative answering, standard indexing) and the associated economic model (free, attribution, subscription, pay-per-crawl, pay-per-inference).
- Generate your
license.xmlfile following the standard's XML vocabulary — the official rslstandard.org site provides ready-to-adapt license templates. - Host that file at a stable URL and add the line
License: https://yourdomain.com/license.xmlto yourrobots.txt, independent of your existingUser-agentblocks. - Verify the implementation by reloading your production
robots.txtand confirming the directive actually appears in the served version (not just in your local environment). - Document the decision: keep a record of the date and the terms you chose, useful if you ever need to prove your content was under an explicit license before a crawler used it.
Key takeaways
- RSL adds a
License:directive torobots.txt, pointing to alicense.xmlfile that defines usage terms and, optionally, payment. - It became an official specification on December 10, 2025, a year after launch, backed by major publishers and infrastructure commitments from Cloudflare and Akamai.
- It's neither Google-Extended (Google training only), nor the TDM Reservation Protocol (rights reservation, EU framework), nor
llms.txt(condensed content for AI) — these are four complementary mechanisms, not interchangeable ones. - Just like
robots.txt, nothing forces a crawler to honor a declared license: always verify whether a site has actually published the directive before relying on it. - Prioritize implementation if you publish original, high-value content and have the bandwidth to track the protocol's evolution; otherwise, focus first on your citability across AI engines.
Before diving into this kind of advanced optimization, make sure your site's fundamentals hold up: run your free audit to get your SEO and GEO score out of 100. To go deeper on how AI answer engines actually cite content, beyond licensing questions, our partner Tesseract Studio has published a complete guide to GEO.