New robots.txt signal lets website owners control AI training use
By OAS
· 1 min read · Data safety · AI tools
OpenAI announced it will honor a common signal in robots.txt that lets website operators express preferences about using their content for generative‑AI training. The voluntary signal will work across multiple AI providers and is expected to roll out in the coming months.
In short
- A single robots.txt signal can cover multiple AI providers.
- Your existing robots.txt controls stay active.
- OpenAI will begin honoring the signal in the next few months.
What it means for you
- Small businesses with a website can set a single preference rather than managing separate rules for each AI provider.
- The signal adds a clearer way to protect your content from being used in AI model training.
- Existing robots.txt controls remain in place, so you keep current protections while gaining a new shared option.
The details
What happened
OpenAI said it will honor a shared signal added to the robots.txt file that indicates a site’s wishes regarding the use of its content for foundational AI training. The signal is voluntary and is designed to work across multiple AI providers, reducing the need for separate rules for each crawler.
What it means for your business
If you run a website for your dental office, salon, or other small business, you can now add a single line to robots.txt to tell participating AI companies whether they may use your pages to train their models. This simplifies compliance and helps protect proprietary content, images, or client information you publish online. The change does not replace the existing crawler controls you already use; those continue to function.
What to do
- Review your current robots.txt file and the existing crawler rules you have set.
- Add the new shared‑signal line (as described by OpenAI when it releases details) to express your preference.
- Keep your existing robots.txt rules in place, as they remain effective.
- Stay aware of other AI providers that join the shared‑signal effort and adjust your file if needed.
What to do this week
0 of 3 done
Practical AI and tech for running a small business.
The Open Arm Brief: answering calls and leads, automating busywork, getting found on Google, keeping data safe and avoiding scams. 5 minutes, Tuesday and Thursday mornings.
Free. One-click unsubscribe. Privacy
Sources (each opens the publisher's page)
More from AI News
What happened ·
Perplexity launches always‑on AI agents for ongoing work
According to The Decoder, Perplexity announced a new “Teammates” AI agent that stays active around the clock to handle ongoing tasks. The agents run on separate virtual machines and are accessed via threads in the Perplexity app or Slack, and they are available only on the Max plan, which costs $200 per month.
Read the storyHow it works ·
Perplexity’s “Teammates” AI agents for everyday workplace tasks
The Verge reports that Perplexity AI introduced “Teammates,” always‑on AI agents that can handle multiple jobs in a workspace. The agents run on Perplexity Computer, can be accessed via threads in the app or Slack, and require a $200‑per‑month Max plan subscription.
Read the storyWhat happened ·
Google Drive adds beta auto‑encryption for confidential files
Google Workspace says its new beta feature lets admins automatically apply client‑side encryption to Docs, Sheets, Slides and Drive files that meet policy rules. The encryption is applied in place and works on My Drive and shared drives, helping meet compliance standards like HIPAA.
Read the story
