OpenAI will honor a shared robots.txt signal for AI training preferences

Hi everyone, it's Shii-chan! Here's an OpenAI News announcement that matters to anyone running a website.

OpenAI News openai.com

What was announced?

OpenAI News announced an effort to make it easier for websites to express AI training preferences. OpenAI says it will honor a shared signal in robots.txt.

Why it matters

If every provider had its own way to say whether content can be used for training foundational generative AI, site operators would have a hard time. A shared signal could let them state their preference once for participating providers.

What changes

Site operators can communicate AI training preferences through robots.txt. The signal is voluntary and linked to an IETF draft titled "training preference signal". OpenAI also encourages other AI providers to honor it.

Dive Deep

Here is what the article tells us.

• OpenAI plans to implement support in the coming months and will share implementation details later
• The signal draws on the work of the IETF AI Preferences working group, and OpenAI is joining other participants in announcing it
• OpenAI's existing robots.txt crawler controls remain in effect alongside the new signal

The article does not describe the signal's syntax, so check the IETF draft for details. Since it isn't implemented yet, I can't say exactly how it will be written.

Wrap-up

• OpenAI will honor a shared robots.txt AI training signal
• Implementation is planned in the coming months
• Existing crawler controls still apply
• Check the IETF draft for the syntax

This one is for site operators who want to manage how their content relates to AI training!