Decide what search and AI may read on your site
Build a robots.txt that allows or blocks each crawler, and an llms.txt that gives AI tools a short map of your site. Free, and the files are built in your browser.
Enter your URL
Optional. We fill in the form from your current robots.txt, sitemap and homepage.
Set the rules
Allow or block each crawler and pick the pages AI tools should know about.
Copy both files
Download robots.txt and llms.txt and upload them to the root of your domain.
robots.txt
Tells crawlers which parts of your site they may read. Most search and AI crawlers follow it.
Search engines
Classic web search. Blocking these takes your pages out of search results.
GooglebotGoogle
Google Search, including AI Overviews. Blocking it takes your site out of Google.
BingbotMicrosoft
Bing search. Its index also feeds Copilot and other search engines.
AI search
Index pages so AI assistants can show and link them in answers.
OAI-SearchBotOpenAI
Shows your pages in ChatGPT search answers.
Claude-SearchBotAnthropic
Indexes pages for search results in Claude.
PerplexityBotPerplexity
Indexes pages for Perplexity answers. Perplexity says it does not train on them.
AI training
Collect pages to train AI models. Blocking them does not take you out of search.
GPTBotOpenAI
Collects pages that may be used to train OpenAI's models.
ClaudeBotAnthropic
Collects pages that may be used to train Anthropic's models.
Google-ExtendedGoogle
Decides whether Google may use your pages to train Gemini and ground its answers. Google Search is not affected.
Applebot-ExtendedApple
Decides whether Apple may use your pages to train its AI models. Siri and Spotlight search are not affected.
Meta-ExternalAgentMeta
Collects pages to train Meta's AI models.
CCBotCommon Crawl
Builds the open Common Crawl archive, which many AI companies train on.
BytespiderByteDance
Collects pages for the AI models of ByteDance, the company behind TikTok.
On request
Open a page when a person asks an AI assistant about it. Some of these may not follow robots.txt.
ChatGPT-UserOpenAI
Opens a page when a ChatGPT user asks for it. OpenAI says robots.txt may not apply.
Claude-UserAnthropic
Opens a page when a Claude user asks for it.
Perplexity-UserPerplexity
Opens a page when a Perplexity user asks for it. Perplexity says it generally ignores robots.txt.
Apply to every crawler you allow. One path per line, e.g. /wp-admin/
Paths inside a blocked folder that stay open, e.g. /wp-admin/admin-ajax.php
# Made with https://www.webvise.io/tools/robots-llms-generator
User-agent: *
Allow: /
This file replaces your current robots.txt. Rules for crawlers not on this list are not carried over.
llms.txt
A short Markdown file that tells AI tools who you are and which pages matter most. It follows the format proposed at llmstxt.org.
Sections and links
Group your most useful pages. Name a section "Optional" for links AI tools may skip when space is short.
# Website
Where the files go
Both are plain text files. Once they are live, open them in your browser to check.
See our SEO & GEO servicerobots.txt
Upload it to the root of your domain, so it opens at yoursite.com/robots.txt. Crawlers read it before anything else.
llms.txt
Put it next to robots.txt, at yoursite.com/llms.txt. AI tools that support the format read it to find your key pages.
WordPress
WordPress writes its own robots.txt. A file you upload to the root replaces it. Many SEO plugins also have a robots.txt editor where you can paste the rules.