How to get recommended by ChatGPT and AI search: crawlers, llms.txt and being cited

How AI assistants decide what to recommend

When ChatGPT answers with web search, it doesn’t rely only on what it learned in training. It searches the web, reads a handful of pages and builds an answer from them, usually with links to its sources. Google’s AI Overviews and AI Mode work on top of Google Search’s own index. Perplexity, Claude and others run their own search crawlers.

So two things decide whether you show up. First, the assistant has to be able to find and read your pages. Second, your brand has to look like a good answer – on your own site and in what other sites say about you. Generative engine optimization (GEO), often called AI SEO, is mostly normal SEO with extra weight on clear answers and third-party mentions.

Let the right AI crawlers in (robots.txt)

AI companies use different crawlers for different jobs, and you can allow or block each one separately in your robots.txt at yoursite.com/robots.txt. The distinction that matters most: search crawlers fetch pages to answer questions and link to you, training crawlers collect content to train models.

A common setup is to allow the search crawlers and, if you want, opt out of training. Anything you don’t block is allowed by default, so the Allow lines below only make your intent explicit. Also check that a broad User-agent: * rule, your host or your CDN’s bot protection isn’t blocking these crawlers by accident – some providers offer a one-click setting to block AI bots.

# AI search crawlers: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# AI training: opted out (optional)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

• OAI-SearchBot: OpenAI’s crawler for ChatGPT search. According to OpenAI, it must be allowed for your pages to appear in ChatGPT’s search answers.
• GPTBot: collects content for training OpenAI’s models. Blocking it doesn’t remove you from ChatGPT search.
• ChatGPT-User: fetches a page when someone asks ChatGPT to open or read it.
• Google-Extended: a token that controls whether Google may use your content for Gemini training and grounding. It doesn’t affect Google Search or AI Overviews.
• PerplexityBot (Perplexity), Claude-SearchBot and ClaudeBot (Anthropic’s search and training crawlers) and Applebot-Extended (Apple’s opt-out for AI training).

Get indexed in Bing and Google

ChatGPT search also relies on third-party search providers, and Bing is widely reported to be one of them. OpenAI doesn’t publish the details, but being properly indexed in Bing can only help. Verify your site in Bing Webmaster Tools (you can import it from Google Search Console in a few clicks), submit your sitemap and check that your important pages are indexed. Sending new and changed URLs via IndexNow gets them to Bing faster.

Google’s AI Overviews and AI Mode don’t need a separate setup: they draw on the normal Google index. A page that isn’t indexed can’t be used – if Search Console shows problems, see why a page isn’t indexed.

Finally, make sure your key content is in the HTML. Many crawlers don’t run JavaScript like a browser does, so text that only appears after scripts load may never be read. Open the page source (not the element inspector) and check that your headings and main text are there.

Write content an AI can quote

AI answers are built from passages of text. Pages that state things clearly and specifically are much easier to use than pages full of slogans. A quick website SEO check is a good start, then go through your key pages with these points:

• Answer the question directly near the top, then explain. “We deliver in Hamburg on weekdays from 7 am” beats “We bring freshness to your door.”
• Use your customers’ questions as headings: prices, opening hours, who it’s for, how it compares to alternatives.
• Publish original facts: your own prices, test results, specs and experience. Things only you can say are reasons to cite you.
• Keep brand facts consistent everywhere – name, address, what you do – on your site, your Google Business Profile, social profiles and directories.
• Keep pages fresh. Update outdated prices and facts, and show an updated date when you’ve changed something substantial.
• Add structured data such as Organization, LocalBusiness or Product where it fits. It guarantees nothing, but it makes facts about you unambiguous.

Get mentioned on other sites

When someone asks “What’s the best …?”, assistants lean heavily on what others say: review platforms, comparison articles, “best of” lists, forums and communities like Reddit. A brand that’s only praised on its own website is a weak candidate.

Ask your main questions yourself and note which sources keep showing up in the AI answers. Those are the places to be: ask happy customers for reviews on the platforms that matter in your niche, suggest your product to authors of comparison articles, and answer questions in communities honestly and with disclosure. Links help, but plain mentions of your brand next to the topic count too – our guide on checking backlinks for free shows who already links to you.

llms.txt: what it is and what it isn’t

llms.txt is a proposal from 2024: a Markdown file at yoursite.com/llms.txt that gives language models a short summary of your site and links to your most important pages.

Be realistic about it. No major AI provider has confirmed that it uses llms.txt, and Google has said it doesn’t. There’s no proven effect on rankings or citations. It’s quick to create and does no harm – but spend your real effort on the points above. A minimal file looks like this:

# Example Bakery

> Family bakery in Hamburg since 1998. Sourdough bread, cakes and catering, delivery on weekdays.

## Key pages

- [Products and prices](https://example.com/products): bread, rolls and cakes with current prices
- [Delivery](https://example.com/delivery): areas, times and minimum order
- [Contact](https://example.com/contact): address, opening hours, phone

Test whether you’re being named

There’s no Search Console for ChatGPT, so test by hand. Write down five to ten questions a real customer would ask – not your brand name, but the problem: “sourdough bakery Hamburg Altona” or “invoicing app for freelancers”.

Ask them in ChatGPT with search turned on, in Perplexity and in Google (look at the AI Overview if one appears). Note whether you’re named, who is named instead and which sources are linked. Answers vary from run to run, so repeat this regularly – patterns over several weeks are what count.

Your server logs can also show whether OAI-SearchBot, PerplexityBot and others actually visit. In your analytics, referrals from chatgpt.com or perplexity.ai show people clicking through from AI answers.

Check your AI visibility with Rankpilot

Rankpilot is an iPhone and iPad app for website SEO, coming soon to the App Store. Its AI visibility tool reads your robots.txt, shows which AI crawlers are allowed or blocked and checks whether you have an llms.txt. The code generator creates structured data for your pages, and the Bing Webmaster Tools connection and IndexNow submission help you get indexed in Bing.

With Rankpilot Pro, the AI answers feature asks ChatGPT with web search your customers’ questions every week and shows whether you’re named – and who is named instead. The app is free, Pro is an optional subscription with a 7-day free trial, and you don’t need an account.