Training only

GPTBot: what it is and how to allow or block it

Blocking it keeps your text out of future model training, but doesn't remove you from answers: other bots handle that. It's a legitimate choice either way.

Check whether GPTBot can read your site

Tarsier tests your site with this and other AI crawlers and tells you who gets in, who doesn't and why.

What it is

OpenAI GPTBot. Collects text to train OpenAI's models. Blocking it doesn't take you out of ChatGPT.

Like most AI crawlers, it reads the HTML exactly as your server sends it: whatever your site paints with JavaScript may never reach it.

How it identifies itself

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)

This is what it sends in the User-Agent header. In robots.txt you only use the short name: GPTBot.

The lines for your robots.txt

Allow it

User-agent: GPTBot
Allow: /

Block it

User-agent: GPTBot
Disallow: /
Build the full robots.txt

Other crawlers