# Content-Signal Generator Generate Content Signals for robots.txt plus X-Robots-Tag HTTP headers and HTML meta robots directives. Compare content-use preferences with indexing controls and learn where to implement them. Interactive generator: https://www.hota.digital/t/content-signal-generator ## Supported signals - search: conventional search indexing and result links or excerpts; not AI-generated summaries. - ai-input: retrieval and grounding for AI responses. - ai-train: model training and fine-tuning. Values are yes or no; a missing field has no indicated preference. This Cloudflare proposal extends robots.txt and is not part of RFC 9309. ## Example ```text User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no ``` The example does not change Allow or Disallow crawl rules. Copy directives into the appropriate existing robots.txt group. HTTP X-Robots-Tag and HTML meta robots directives are separate search indexing controls. Example HTTP header: X-Robots-Tag: noindex, nofollow. Example HTML: . Neither expresses ai-train=no. ## Related checks - [AI Crawlers Compatibility Checker](https://www.hota.digital/t/ai-crawlers-compatibility-checker) - [Site Agent Readiness Checker](https://www.hota.digital/t/site-agent-readiness-checker) ## FAQ ### What does the Content-Signal Generator do? It generates Content Signals for robots.txt and separately creates X-Robots-Tag and HTML meta robots snippets for indexing and snippet controls. These mechanisms have different purposes and do not provide interchangeable AI-training controls. ### How do I use this generator? Choose content-use preferences and a user-agent for the robots.txt panel. Choose indexing, link following and snippet controls for the HTTP header and HTML meta panels. Switch between output tabs, then copy or save the relevant snippet and publish it in the appropriate location. ### What does search=yes mean? It expresses permission for traditional search indexing and result presentation under the Content Signals definition. It does not automatically permit AI-generated search summaries, and it does not override a Disallow rule. ### What does ai-input=yes mean? It expresses permission for using retrieved content as input to AI systems, including retrieval augmented generation or grounding for AI-generated responses. It is separate from model training. ### What does ai-train=no mean? It expresses a restriction on using the content for AI model training or fine-tuning under the Content Signals policy. Its practical effect depends on the crawler or operator recognizing and honoring the policy. ### What if I leave one signal unspecified? That field is omitted. Under the Content Signals policy, an absent field conveys no permission or restriction through this vocabulary. Omission is not equivalent to no. ### Where do I add Content-Signal? Add the generated directive within the intended User-agent group in the robots.txt file served from your site root, such as https://example.com/robots.txt. Check your CMS, server configuration, CDN, or Cloudflare managed robots.txt settings to find the authoritative file. ### Does Content-Signal control crawler access? No. Traditional Allow and Disallow rules tell compliant crawlers which paths may be fetched. Content-Signal describes preferences for how fetched content is used. Authentication, firewalls and CDN controls enforce access more directly. ### Is Content-Signal part of RFC 9309 robots.txt? No. Content-Signal is a proposed extension promoted by Cloudflare, not a directive defined by the IETF robots.txt standard. Some crawlers may ignore it. ### Does this prevent AI companies from using my content? Not by itself. The directive communicates publisher preferences. Operators may not recognize or honor it, and its legal impact depends on jurisdiction and circumstances. For enforceable access restrictions, use appropriate technical controls and obtain legal advice where necessary. ### Can I allow AI search while blocking AI training? Yes. A common choice is search=yes, ai-input=yes, ai-train=no. This signals that search and AI answer grounding are welcome while training is restricted, subject to provider support. ### Can I block AI usage but keep traditional search? Yes. Use search=yes, ai-input=no, ai-train=no. Note that this is an expressed preference and not a crawler access block. ### How do I check whether my rules are published? Visit your live /robots.txt URL, verify the directive appears in the correct User-agent group, then recheck after any CDN cache has cleared. Check crawler documentation and server or CDN logs to understand actual behavior. ### Can Content-Signal: ai-train=no be used inside an X-Robots-Tag or meta robots tag? No. ai-train=no is part of Cloudflare proposed robots.txt Content Signals vocabulary, not a standard X-Robots-Tag or HTML meta robots directive. Adding it there would not make it an understood AI-training restriction. ### Where do I add an X-Robots-Tag header? Configure the HTTP response header for the intended URLs in your server, application, CMS or CDN. The generator provides a raw header plus Apache, Nginx and PHP examples. Verify a live response using browser developer tools or curl -I. ### Where do I add a meta robots tag? Place the generated tag inside the HTML head of the intended page, typically through an SEO plugin, template or CMS head settings. Meta robots tags do not control PDF files or other non-HTML resources. ### Can I stop search indexing without blocking crawling? Yes. For cooperative crawlers that support the directive, use noindex in an HTTP X-Robots-Tag or HTML meta robots tag. The crawler must generally be allowed to fetch the URL to read that directive. ### What is the difference between nosnippet and max-snippet? nosnippet requests that no text snippet be shown. max-snippet:N sets a maximum number of characters for snippets in supported search results. These are search presentation rules, not a universal ban on AI model training. ### Do I need both an X-Robots-Tag and a meta robots tag? Usually one appropriate mechanism is enough. X-Robots-Tag works for HTML and non-HTML responses and meta robots applies to HTML. If both are present, restrictive rules can combine or take precedence for crawlers that recognize them. ### Does Google support custom bot names in meta robots tags? Google documents robots, googlebot and googlebot-news handling for search directives. Arbitrary AI crawler tokens in meta tags are not a reliable way to control those bots. The generator uses general robots meta tags and optionally Google-specific tags. ### What about X-Robots-Tag and meta robots? These are page-level indexing and snippet controls recognized by supporting search engines. They differ from Content Signals, which express website content-use preferences. They do not universally control AI training or retrieval use. ### How is the policy evolving? Cloudflare has introduced an additional content-use or use vocabulary describing retention and reuse levels in some products. Adoption and semantics are still evolving, so this generator uses the three core Content Signals fields and links to current documentation for additional signals. ### Will this improve SEO rankings or guarantee AI citations? No. Content Signals communicate intended permissions. They do not improve rankings by themselves or guarantee that AI systems will crawl, summarize, cite or avoid a page. ## Authoritative references - [Cloudflare: Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) - [Cloudflare: Managed robots.txt and Content Signals](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) - [Cloudflare: robots.txt monitoring](https://developers.cloudflare.com/ai-crawl-control/features/track-robots-txt/) - [IETF RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) - [Google Search Central: robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro) - [Google Search Central: Robots meta tag and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) - [Cloudflare: Content use in the Crawl endpoint](https://developers.cloudflare.com/changelog/post/2026-08-31-crawl-content-use/)