Quotes you in answers

OAI-SearchBot: what it is and how to allow or block it

If you block it, ChatGPT Search can't read your site when someone asks about what you do, and it won't quote you. To show up in answers, let it in.

Check whether OAI-SearchBot can read your site

Tarsier tests your site with this and other AI crawlers and tells you who gets in, who doesn't and why.

What it is

ChatGPT Search. Indexes the web for ChatGPT search. Without it, ChatGPT won't link to you as a source.

Like most AI crawlers, it reads the HTML exactly as your server sends it: whatever your site paints with JavaScript may never reach it.

How it identifies itself

Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)

This is what it sends in the User-Agent header. In robots.txt you only use the short name: OAI-SearchBot.

The lines for your robots.txt

Allow it

User-agent: OAI-SearchBot
Allow: /

Block it

User-agent: OAI-SearchBot
Disallow: /
Build the full robots.txt

Other crawlers