Today people often ask ChatGPT, Perplexity or Claude directly, without going through Google. Showing up in those answers takes optimizing for ChatGPT and the other assistants, and it starts with a practical question: can their bots read your site?
These assistants don’t browse like a person: they use dedicated bots that download pages. If a bot is turned away or finds no text, the page doesn’t exist for the assistant, even if you rank first on Google.
Four checks to run
- robots.txt Open
yoursite.com/robots.txtand look for rules blocking bots such as GPTBot, ClaudeBot or PerplexityBot. Some blocks are old and nobody chose them on purpose. - Firewall and CDN A bot can be allowed by robots.txt and still turned away by the firewall. It happened on auditseo.tools: with the “training” category switched off on Cloudflare, four bots got a 403 error despite an open robots.txt.
- Text in the HTML Many AI bots don’t run JavaScript. If content only appears after scripts load, they may find an empty page.
- llms.txt A text file at the root of the site with a summary of your main content. It’s cheap to make, but not every assistant says it reads it: treat it as an extra, not a ranking lever.
Which bots to allow
Allowing AI bots is an editorial choice, not an obligation. Bots fall into three groups you can control separately:
- Search They read pages so they can cite them as sources in assistants’ answers.
- Agents They open a page in real time to answer a user’s question, like ChatGPT-User.
- Training They collect text to train models, like GPTBot or CCBot.
If you want to appear in answers, you usually need to allow search and agents. Training can be blocked on its own if you don’t want your text feeding the models.
FAQ
How do I write an llms.txt file?
The format is Markdown: a title with the site name, a short description and a list of links to your most useful pages. Save the file at the root of the site as llms.txt.
In practice: a line # Site name, a description starting with >, then a list of links, each with a one-line note. A few well-chosen pages are enough, such as the home page, services and main guides. After publishing, check that yoursite.com/llms.txt opens in the browser.
Does blocking AI bots hurt my SEO on Google?
No. AI bots and Googlebot are separate systems: blocking GPTBot or ClaudeBot doesn’t change your ranking in traditional search results.
The risk is touching the wrong rules. If you block Googlebot, the site disappears from Google. Before editing robots.txt, check which lines you’re changing and for which bot.
How do I know if an AI assistant reads my site?
Paste the address of one of your pages into ChatGPT or Perplexity and ask for a summary. If the answer is generic or wrong, there’s an access or structure problem.
For a more precise check, look at your server or CDN logs: search for requests from bots such as GPTBot or ClaudeBot and read the response code. A 200 means the page was served, a 403 means the firewall turned it away. Our free audit tests the main bots and shows each one’s response code.
Do AI Overviews need a different kind of optimization?
According to Google’s documentation, no: to appear in AI Overviews you need the same requirements as traditional search. The page must be indexed and eligible to show a snippet.
There’s no dedicated markup or file. What matters is useful content, with a clear answer in the first lines. Blocking Google-Extended doesn’t remove your site from AI Overviews: that control concerns the use of content for Gemini models. Nobody can guarantee a mention in an answer.
How do I check whether my text is visible without JavaScript?
Open the page source and search for a sentence from your article: if you find it, bots that don’t run JavaScript can see it.
In Chrome, view the source with Ctrl+U (Cmd+Opt+U on Mac). Don’t use developer tools: they show the page already processed by the browser, not the one a bot receives. If the sentence is missing, the content depends on scripts and the most robust fix is server-side rendering.
This article is for informational purposes. SEO results depend on many factors and timelines vary by site.