AI Readiness & Discovery · Free generator

Content-Signal Generator

Define how automated systems may use your website content for search, AI-generated answers and model training. Generate Content Signals for robots.txt and separate HTTP header or HTML meta tag controls for indexing and snippets.

Content Signals express content-use preferences. X-Robots-Tag and meta robots govern search indexing and presentation for crawlers that support them. They are not interchangeable.

1. Choose your policy

Start with a policy preset, then customize each use. A blank signal means no preference expressed.

Search indexing search

Traditional search results with links and short excerpts. Does not include AI-generated search summaries.

AI answer input ai-input

Retrieval, grounding and real-time input to a generative AI response.

Model training ai-train

Training or fine-tuning AI models on your content.

2. Choose where the policy applies

Important: adding a new User-agent group can change which existing Allow or Disallow rules match a bot. Merge carefully into an existing group when possible.

Leave unchecked to avoid accidentally changing crawl permissions. Content-Signal and Allow are independent.

3. Generate your policy

Live preview

Choose an output format. Robots.txt controls content-use preferences. HTTP headers and HTML meta tags control indexing and snippets.

Indexing and snippet settings

These settings are independent of Content Signals. They do not tell AI companies whether training is permitted.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

What this output means
Search: allow; AI input: allow; Training: restrict.
This is a snippet, not a complete replacement for your current robots.txt. Preserve existing Sitemap, Allow, Disallow and crawler-specific rules when deploying.

4. Publish and verify

  1. Open your live https://yourdomain.com/robots.txt.
  2. Locate the relevant User-agent group and add the Content-Signal directive there.
  3. Publish through your hosting platform, CMS or CDN and clear cached copies.
  4. Reopen the public URL and confirm the expected policy.
Check AI crawler compatibility

What are Content Signals?

Content Signals are a publisher preference vocabulary introduced by Cloudflare for machine-readable statements about how automated systems may use website content. The directives are placed inside robots.txt groups. They let publishers distinguish traditional search use, real-time AI input and training, instead of treating all AI usage as one decision.

The policy uses values such as search=yes, ai-input=no and ai-train=no. Leaving out a field is intentionally neutral. It does not automatically mean permission granted or denied.

Why are Content Signals important?

AI products can use web content in different ways. A traditional search index may link to your pages, a retrieval system may use a passage to ground an answer, and a model developer may use data during training. A publisher can support some uses and restrict others. Content Signals make that intent easier to communicate in a recognizable, machine-readable form.

They are useful for expressing a policy, but they are not a guarantee of compliance. Crawl access, legal rights, AI product behavior and citation outcomes remain separate questions.

How to read the generated output

User-agent: *

This group targets generic crawlers unless a crawler-specific group takes precedence under its robots.txt implementation.

Content-Signal: search=yes, ai-input=yes, ai-train=no

The example expresses support for search and AI input, while restricting model training. It does not tell a bot to fetch or avoid any particular URL path.

Where should you make the changes?

The public location is the site's root /robots.txt. On a hosted site, that file can be generated by the CMS or a plugin, served by your web server, or managed at the CDN layer. If Cloudflare manages your robots.txt, review its managed robots settings rather than editing a local file that will not appear publicly.

For custom hosting, verify the final output served over HTTPS. Place the directive in the correct group, retain unrelated robots rules, and check that your CDN cache is not serving an outdated version. Avoid copying the entire generated sample over an existing robots.txt.

How to use HTTP headers and HTML meta tags

Use the HTTP header tab to create an X-Robots-Tag for a server, CMS, CDN or PHP application response. The Apache and Nginx samples show server-level directives and should be placed in the configuration appropriate for your intended page scope. In PHP, send the header before output. To inspect a published response, use your browser network panel or curl -I https://example.com/page.

Use the HTML meta tag tab for an individual HTML document. Insert it in the page <head> through a template, CMS setting or SEO plugin. Search engines need to fetch the page to see its header or meta directive. If robots.txt blocks that URL, these controls may not be discovered. Avoid accidentally applying noindex to important pages.

Do not paste ai-train=no into an X-Robots-Tag or robots meta tag. It is not a documented indexing directive. Some Google-specific snippet limits can affect how content is used in Google AI search experiences, but these are not universal controls over AI training.

Content Signals versus robots.txt, X-Robots-Tag and meta robots

MechanismPrimary purposeWhere it lives
Allow / DisallowPath-level crawl access preferences under RFC 9309robots.txt
Content-SignalUse preferences for search, AI input and training under an emerging extensionrobots.txt, by user-agent group
X-Robots-TagSupported indexing and snippet restrictionsHTTP response header
Meta robotsSupported indexing and snippet restrictions for HTML pagesHTML head
CDN / WAF / authenticationPractical access enforcementInfrastructure and application configuration

Search engines may ignore unknown robots.txt extensions. Blocking crawling may also prevent a crawler from seeing on-page indexing directives. Test each mechanism independently before changing your sitewide policy.

How Content Signals are evolving

The three core signals cover search, AI input and AI training. Cloudflare has also begun experimenting with an additional content-use or use dimension for content retention and reuse. Support differs by implementation, so this generator focuses on the three established Cloudflare policy fields and avoids presenting optional extensions as universally recognized standards.

Content Signals are not part of the core IETF robots.txt specification, and support across AI crawlers remains uneven. Revisit vendor documentation when adopting new directives or changing an organizational content policy.

A practical AI readiness workflow

First decide how you want content to be used. Generate and publish the policy. Next, review whether major AI crawlers can reach the pages they need. Finally, audit discoverability, content formats and agent protocol signals across your site.

AI Crawlers Compatibility CheckerSite Agent Readiness CheckerAll AEO tools

Frequently asked questions

It generates Content Signals for robots.txt and separately creates X-Robots-Tag and HTML meta robots snippets for indexing and snippet controls. These mechanisms have different purposes and do not provide interchangeable AI-training controls.

Choose content-use preferences and a user-agent for the robots.txt panel. Choose indexing, link following and snippet controls for the HTTP header and HTML meta panels. Switch between output tabs, then copy or save the relevant snippet and publish it in the appropriate location.

It expresses permission for traditional search indexing and result presentation under the Content Signals definition. It does not automatically permit AI-generated search summaries, and it does not override a Disallow rule.

It expresses permission for using retrieved content as input to AI systems, including retrieval augmented generation or grounding for AI-generated responses. It is separate from model training.

It expresses a restriction on using the content for AI model training or fine-tuning under the Content Signals policy. Its practical effect depends on the crawler or operator recognizing and honoring the policy.

That field is omitted. Under the Content Signals policy, an absent field conveys no permission or restriction through this vocabulary. Omission is not equivalent to no.

Add the generated directive within the intended User-agent group in the robots.txt file served from your site root, such as https://example.com/robots.txt. Check your CMS, server configuration, CDN, or Cloudflare managed robots.txt settings to find the authoritative file.

No. Traditional Allow and Disallow rules tell compliant crawlers which paths may be fetched. Content-Signal describes preferences for how fetched content is used. Authentication, firewalls and CDN controls enforce access more directly.

No. Content-Signal is a proposed extension promoted by Cloudflare, not a directive defined by the IETF robots.txt standard. Some crawlers may ignore it.

Not by itself. The directive communicates publisher preferences. Operators may not recognize or honor it, and its legal impact depends on jurisdiction and circumstances. For enforceable access restrictions, use appropriate technical controls and obtain legal advice where necessary.

Yes. A common choice is search=yes, ai-input=yes, ai-train=no. This signals that search and AI answer grounding are welcome while training is restricted, subject to provider support.

Yes. Use search=yes, ai-input=no, ai-train=no. Note that this is an expressed preference and not a crawler access block.

Visit your live /robots.txt URL, verify the directive appears in the correct User-agent group, then recheck after any CDN cache has cleared. Check crawler documentation and server or CDN logs to understand actual behavior.

No. ai-train=no is part of Cloudflare proposed robots.txt Content Signals vocabulary, not a standard X-Robots-Tag or HTML meta robots directive. Adding it there would not make it an understood AI-training restriction.

Configure the HTTP response header for the intended URLs in your server, application, CMS or CDN. The generator provides a raw header plus Apache, Nginx and PHP examples. Verify a live response using browser developer tools or curl -I.

Place the generated tag inside the HTML head of the intended page, typically through an SEO plugin, template or CMS head settings. Meta robots tags do not control PDF files or other non-HTML resources.

Yes. For cooperative crawlers that support the directive, use noindex in an HTTP X-Robots-Tag or HTML meta robots tag. The crawler must generally be allowed to fetch the URL to read that directive.

nosnippet requests that no text snippet be shown. max-snippet:N sets a maximum number of characters for snippets in supported search results. These are search presentation rules, not a universal ban on AI model training.

Usually one appropriate mechanism is enough. X-Robots-Tag works for HTML and non-HTML responses and meta robots applies to HTML. If both are present, restrictive rules can combine or take precedence for crawlers that recognize them.

Google documents robots, googlebot and googlebot-news handling for search directives. Arbitrary AI crawler tokens in meta tags are not a reliable way to control those bots. The generator uses general robots meta tags and optionally Google-specific tags.

These are page-level indexing and snippet controls recognized by supporting search engines. They differ from Content Signals, which express website content-use preferences. They do not universally control AI training or retrieval use.

Cloudflare has introduced an additional content-use or use vocabulary describing retention and reuse levels in some products. Adoption and semantics are still evolving, so this generator uses the three core Content Signals fields and links to current documentation for additional signals.

No. Content Signals communicate intended permissions. They do not improve rankings by themselves or guarantee that AI systems will crawl, summarize, cite or avoid a page.

References and further reading

These primary sources explain the policy proposal, the standard robots.txt model and the difference between crawl controls and indexing directives.