Share with your friends!

the web has a new system for A new licensing standard aims to let web publishers set the terms of how AI system developers use their work.

the web has a new system for

Introduction to the RSL Standard

On Wednesday, a coalition of prominent web publishers, including Reddit, Yahoo, Medium, Quora, and People Inc., announced their support for a new initiative called Really Simple Licensing (RSL). This open content licensing standard is designed to empower publishers to dictate the terms under which AI developers can scrape their sites for training data. The collective effort aims to provide these publishers with the leverage they need to negotiate fair compensation from AI companies.

Understanding the RSL Standard

The RSL Standard builds upon the existing robots.txt protocol, which has long allowed website owners to specify which parts of their sites can be accessed by web crawlers. Traditionally, this protocol has been a binary system, allowing publishers to either permit or deny access to specific bots. However, the RSL Standard introduces a more nuanced approach, enabling publishers to embed licensing and royalty terms directly into their robots.txt files. This means that publishers can now specify not only whether bots can access their content but also the conditions under which they can do so.

Features of the RSL Standard

One of the key features of the RSL Standard is its flexibility in licensing models. Publishers can choose from various options, including:

Subscription Fees: Publishers can require AI companies to pay a recurring subscription fee for access to their content.
Pay-Per-Crawl Fees: This model allows publishers to charge a fee each time an AI bot crawls their website.
Pay-Per-Inference Fees: Under this model, publishers receive compensation whenever an AI model references their work to generate a response.

Additionally, the RSL Standard allows publishers to implement these terms not only in their robots.txt files but also in online books, videos, and training datasets. This comprehensive approach aims to ensure that creators are compensated for their intellectual property in a manner that reflects the value it provides to AI systems.

The RSL Collective

At the heart of the RSL Standard is the RSL Collective, a newly formed rights organization led by Eckart Walther and Doug Leeds. Walther, a co-creator of the Really Simple Syndication (RSS) standard and former CEO of CardSpring, brings a wealth of experience to the initiative. Leeds, the former CEO of IAC Publishing and Ask.com, adds further credibility to the organization. Together, they aim to create a scalable business model for the web that benefits both publishers and AI developers.

Goals and Vision

According to Walther, the primary goal of the RSL Collective is to establish a new framework for licensing rights and compensation rights across the internet. “RSL takes some of those early RSS ideas and creates a new layer for the entire internet where licensing rights and compensation rights are defined,” he explains. This vision aligns with the growing need for clear guidelines in an increasingly complex digital landscape, where the lines between content creation and AI utilization are often blurred.

The Need for a Standardized Approach

The RSL Standard addresses a critical gap in the current landscape of AI development. Many media companies, including Vox Media, News Corp, and The New York Times, have previously entered into individual licensing agreements with AI companies like OpenAI and Amazon. However, these negotiations can be time-consuming and often result in unequal power dynamics, with larger companies having more leverage. The RSL Collective aims to simplify this process by providing a standardized framework that allows any website owner or creator to receive compensation for their work without the need for separate negotiations.

Challenges and Industry Dynamics

Despite its promising framework, the success of the RSL Standard hinges on the willingness of major AI companies to adopt it. Historically, AI developers have faced criticism for ignoring the instructions set forth in robots.txt files, leading to concerns about unauthorized scraping of content. Without the participation of AI companies, it remains unclear how publishers can effectively enforce the inference fee or other licensing terms.

Leeds emphasizes the importance of collective action among publishers, stating, “Our job is to go out and get a big group of people to say it’s in your interest, both efficiently, because you can negotiate with everybody at once, and legally, because if you don’t, you’re violating everybody at once.” This unified front is intended to make it more appealing for AI companies to adopt the RSL Standard, as they would face collective legal repercussions for non-compliance.

Technical Implementation and Collaboration

While the RSL Standard provides a framework for licensing, it does not inherently block bots from accessing websites. This limitation is noteworthy, especially when compared to existing solutions like Cloudflare’s “pay per crawl” system. To address this issue, the RSL Collective is collaborating with Fastly, a content delivery network, to create a system where AI bots can only access websites if they have agreed to the licensing terms. Leeds describes Fastly’s role as akin to a “bouncer at the door to the club,” ensuring that only compliant bots gain entry.

“RSL is issuing the IDs. So we say, ‘Hey, you’ve agreed to license this content,’ and Fastly says, ‘Come on, in your ID checks out,’” he explains. This collaboration aims to create a more secure environment for publishers, allowing them to enforce their licensing terms effectively. However, publishers who do not utilize Fastly will still face challenges in blocking AI crawlers until additional solutions are developed.

Legal Considerations and Enforcement

Another significant aspect of the RSL Standard is its potential for legal enforcement. Leeds asserts that the RSL Collective can legally enforce licenses, as all participants in the collective share the responsibility for addressing any infringements. This collective approach to enforcement is reminiscent of existing digital rights organizations, such as ASCAP, which manages licensing fees for music creators.

However, the legal landscape surrounding unauthorized scraping and the use of content for AI training remains complex and often ambiguous. Major AI companies are currently embroiled in lawsuits from various publishers, including Reddit and Getty Images, highlighting the ongoing struggles over intellectual property rights in the digital age. The RSL Standard aims to clarify these issues by putting crawlers on notice regarding the terms of access before they interact with a site.

Future Implications and Industry Impact

As the RSL Standard gains traction, it has the potential to reshape the relationship between content creators and AI developers. By providing a clear framework for licensing, the RSL Collective hopes to foster a more equitable environment where publishers can receive fair compensation for their contributions. Leeds envisions the RSL Standard as a means to create an intuitive system for navigating licensing for AI training, stating, “What we’re doing is not reinventing wheels or inventing wheels — we’re just bringing them to a place that they haven’t existed before.”

The RSL Standard is free for publishers and creators to join, and it has already attracted support from other notable brands, including O’Reilly, wikiHow, and Ziff Davis, the parent company of IGN. As more organizations come on board, the collective power of the RSL Standard could lead to significant changes in how AI companies approach content licensing.

Conclusion

The introduction of the RSL Standard represents a pivotal moment in the ongoing dialogue between web publishers and AI developers. By establishing a clear and standardized approach to licensing, the RSL Collective aims to empower creators and ensure that they are compensated fairly for their work. As the digital landscape continues to evolve, the success of this initiative will depend on the willingness of AI companies to engage with the new framework and the collective action of publishers to enforce their rights.

Source: Original report