Trains and answers

meta-externalagent: what it is and how to allow or block it

The same name decides two things: whether they train on your text and whether they use you to answer. Blocking it closes both doors at once.

Check whether meta-externalagent can read your site

Tarsier tests your site with this and other AI crawlers and tells you who gets in, who doesn't and why.

What it is

Meta AI. Collects text to train Meta's models and for Meta AI's answers.

Like most AI crawlers, it reads the HTML exactly as your server sends it: whatever your site paints with JavaScript may never reach it.

How it identifies itself

meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

This is what it sends in the User-Agent header. In robots.txt you only use the short name: meta-externalagent.

The lines for your robots.txt

Allow it

User-agent: meta-externalagent
Allow: /

Block it

User-agent: meta-externalagent
Disallow: /
Build the full robots.txt

Other crawlers