Get started

This guide instructs you on how to:

• View AI crawlers that are interacting with pages in your domain (a Cloudflare zone).
• Use AI Audit to block individual crawlers from accessing your content.

Prerequisites

1. Sign up for a Cloudflare account ↗.
2. Connect your domain to Cloudflare.
3. Make sure your domain is proxying traffic through Cloudflare.

1. Block specific AI crawlers

All plans

To only block specific AI crawlers:

1. Go to AI Audit.
2. From the AI Crawlers tab, go to the list of AI crawlers.
3. Review the AI crawlers that are accessing your domain.
4. From the Block column, select the option Block.

Quality of AI crawler detection

On the free plan, AI Audit identifies AI crawlers based on their user agent strings ↗. This enables AI Audit to detect easy-to-detect (well-known) AI crawlers.

Upgrade your plan to enable a more thorough detection using Cloudflare's Bot Management detection ID field.

Paid plans

To only block specific AI crawlers:

1. Go to AI Audit.
2. From the AI Crawlers tab, go to the list of AI crawlers.
3. Review key information such as:
   • Bot operator
   • Number of requests sent by the AI crawler
   • Whether the AI crawler is allowed according to your robots.txt, or the number of times the AI crawler has violated your robots.txt.
4. From the Block column, toggle the switch for AI crawlers you wish to block.

For more information, refer to Manage AI crawlers.

You can also create more complex rules when taking action on AI crawlers, using Cloudflare WAF. For more information on creating more specific rules, refer to Create a custom rule in the dashboard.

2. Review detected AI crawlers

All plans

Review the AI crawlers detected on your site.

1. Go to AI Audit.
2. From the Metrics tab, view key metrics on AI crawlers.

Note that on free plans, the Metrics tab only displays metrics for the past 24 hours.

Plan comparison

All plans	Enterprise plans with Bot Management
AI crawler detection via user agent strings ↗	Advanced AI crawler detection via Bot Management detection ID
Maximum 24-hour analytics window	Configurable analytics timeframes
Allow/block controls	Allow/block controls, and the ability to charge AI crawlers using pay per crawl

Next steps

• Manage AI crawlers with granular allow/block controls.
• Analyze AI traffic to understand crawler patterns and content popularity.
• Explore pay per crawl to test content monetization options (private beta).

Related resources

Refer to the following related resources:

• Cloudflare blog: Start auditing and controlling the AI models accessing your content ↗
• Block AI crawlers that do not adhere to recommended guidelines using Cloudflare AI Labyrinth.
• Direct AI crawlers with managed robots.txt.