Free tool

robots.txt for AI,
knowing what each bot does.

Some bots cite you in ChatGPT, Claude or Perplexity answers; others only collect text to train models. Choose who gets in and copy a block ready to paste.

Start from a recipe

Your robots.txt block

9 allowed · 5 blocked

robots.txt
# AI crawlers · generated with Tarsier · https://te-ve-la-ia.onrender.com/robots-txt
# Paste it at the end of your robots.txt and delete any other lines naming these bots.

# Allowed to read the site
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
Allow: /

# Not allowed to read the site
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

Bot by bot

They cite you in their answers

They index the web to answer with links to their sources. If they can't get in, you don't show up.

They read your page when someone asks

They come in the moment a person asks a question or pastes your link.

They train and they answer

One key for both: shut them out and they won't cite you either.

They only train models

They collect text for future versions. Blocking them doesn't take you out of any answer.

If you want to fine-tune

One path per line, like /cart/ or /my-account/. A bot with its own group stops reading the "User-agent: *" rules: if you close something there, repeat it here.

It helps them find all your pages.

How to add it to your site

  1. Copy the block. As is, comments included: they tell whoever opens the file later what each part does.

  2. Paste it at the end of your robots.txt. That's the file at yourdomain.com/robots.txt. In WordPress you edit it from Yoast or Rank Math; in Shopify, in the robots.txt.liquid template; in Wix, in the SEO tools. If you don't have one, this block works as the whole file.

  3. Delete anything already there about these bots. If a bot appears in two places, both count, and an old "Allow: /" opens the door even if further down it says "Disallow: /".

  4. Check that it works. robots.txt is only half of it: the firewall or the CDN can turn a bot away even if it's allowed here. The free analysis tests each one.

What robots.txt doesn't do

And your site, does it let AI in?

The free analysis reads your robots.txt bot by bot, tests whether your server turns them away and compares what a person sees with what the AI receives.

Analyze my site for free

Frequently asked questions

  1. If I block GPTBot, do I disappear from ChatGPT?

    No. GPTBot collects text to train models. To search and cite you, ChatGPT uses OAI-SearchBot, and to open a page someone asks for, ChatGPT-User. You can block the first and let the other two in.

  2. Does blocking Google-Extended take me out of AI Overviews?

    No. AI Overviews is part of Search and uses what Googlebot reads. Google-Extended only decides whether Gemini trains on your site and uses it to answer in its own app.

  3. What about ClaudeBot?

    It only trains. Claude searches with Claude-SearchBot and opens the pages it's asked for with Claude-User, so you can block ClaudeBot and stay in its answers.

  4. How long until they comply?

    Each bot re-reads robots.txt on its own schedule; Google, for example, caches it for up to 24 hours. There's no one to notify.

  5. Where does the file have to be?

    At the root of the domain: yourdomain.com/robots.txt. Inside a folder nobody reads it, and each subdomain needs its own.