Source
Say it once: introducing Bot Preference SynC
Cloudflare is launching Bot Preference Sync, a feature that automatically updates a website's robots.txt file to match the AI bot configurations set in the Cloudflare dashboard. It synchronizes stated preferences with edge-enforced blocks for Search, Agent, and Training traffic.
Why it matters
Website owners can prevent AI models from training on their content while still allowing search engine indexing. This helps publishers protect their intellectual property without losing discoverability for human visitors.
The details
- Available to all customers, including those on the Free tier.
- Prepends updates to existing robots.txt files to maintain other directives.
- Training bots must meet transparency criteria to avoid being blocked.
Show entities and relationshipsHide entities and relationships
In this article
Key connections
Cloudflare owns Bot Preference Sync
Cloudflare developed and launched Bot Preference Sync for AI traffic management.
Cloudflare owns BotBase
Cloudflare maintains BotBase to classify and track verified web bots.
Cloudflare uses Bot Management
Cloudflare utilizes bot management technology to mitigate unwanted bot traffic.
Bot Preference Sync uses robots.txt
Bot Preference Sync updates and prepends AI bot policy rules to robots.txt files.
Bot Preference Sync uses BotBase
Bot Preference Sync uses bot lists tracked in BotBase to update robots.txt directives.
Bot Preference Sync is related to Cloudflare Radar
AI bots complying with Bot Preference Sync transparency criteria are tracked in Cloudflare Radar.
Show 2 more connectionsShow fewer connections
Bot Preference Sync is related to Web Crawling
Bot Preference Sync manages permissions and rules for web crawling bots.
Bot Preference Sync is related to Artificial Intelligence
Bot Preference Sync controls how AI models access website content for training and agents.
Related events
Cloudflare Announces Bot Preference Sync for AI Traffic Management
Get the weekly recap
The stories like this one, picked and explained — once a week, straight to your inbox.