Enter the exact page
Sign in with Google or email, then use a public homepage, service page or article URL. Rules can differ by path, so a homepage check does not describe the whole site.
Find what could stop AI agents and search engines from reading your page. See what’s accessible, what’s blocked and what needs your attention.
Spot potential barriers to AI access, understand what needs attention on your page, and get clear findings you can share with your developer or team.
A focused technical check of one public page, with the evidence behind each finding. No unexplained AI visibility score.
See whether robots.txt permits each search, training and user-request agent to fetch the selected URL.
Find the winning Allow or Disallow directive and line number. Fetch failures remain unknown.
Check the final response and redirect count. A denied or challenged APR request is shown separately from robots policy.
See a warning when the page tells supporting crawlers not to index it, even if robots.txt allows access.
Count the text in the initial HTML. Follow up with Humans vs LLMs if the page may depend on rendering.
Keep your timestamped findings at a saved report URL. Copy the link to share with a developer or client. Sign in with Google or email to run your free check.
Sign in with Google or email, then use a public homepage, service page or article URL. Rules can differ by path, so a homepage check does not describe the whole site.
Review search crawler permissions first, then the page response, redirects and available text. Training permissions are a separate decision.
Find the matched rule, correct accidental restrictions, and share your saved report. Use logs and your search tools to verify actual crawler behavior.
Your page opens in your browser. Can a crawler get to it too? Start with Page Access Overview: check the page response, whether robots.txt is available, and how much text arrives before JavaScript runs.
A missing page, an access-denied response or a bot challenge gives you somewhere to investigate. Redirects show the route to the final page, including any stops on an insecure HTTP connection. If very little text is available in the initial HTML, check whether your main content depends on JavaScript. That alone does not mean every AI service is unable to read it.
You’ll also see a warning if we find a noindex instruction. A crawler that supports it may be allowed to read the page but told to leave it out of its index. The response we check is to APR’s request; your server or firewall logs can confirm what happens when a real vendor’s crawler visits.
If you want people to find your page through AI search, start with Search & Discovery. It checks the robots.txt rules for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot: the search crawlers used by OpenAI, Anthropic, Perplexity, Google and Microsoft.
Check the page you care about, rather than stopping at your homepage. Your homepage could be allowed while a rule for /services blocks your service pages. Each bot’s card shows the matching rule and its line number, so your developer knows where to look. An allowed result means the file permits that crawler to fetch the page; it does not promise a place in an AI answer.
You may want your pages found in search while limiting their use for training. Those are separate choices. OpenAI uses GPTBot for training and OAI-SearchBot for search. Anthropic separates ClaudeBot from Claude-SearchBot in a similar way.
Training & Other AI Uses also covers CCBot, which collects pages for Common Crawl’s public dataset, and Google-Extended. Google-Extended is a permission setting for certain Gemini training and grounding uses, rather than a separate visiting bot. Blocking it does not block Google Search or its AI features.
The AI Bots Allowed percentage counts robots.txt permissions across all three groups. It is not an AI readiness or visibility score. An intentional training block can lower the percentage without being an SEO problem.
Someone pastes your article into an assistant and asks for a summary. That can trigger a visit on their behalf, rather than a background search crawl. User-Requested Fetches checks the declared rules for ChatGPT-User, Claude-User and Perplexity-User.
Providers handle these requests differently. Anthropic says Claude-User follows robots.txt. OpenAI says those rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Use this section to understand your stated preferences, then check access logs if you need to confirm what was actually retrieved.
Start with an unexpected block on a page you want people to discover. Share the report with your developer using Email Report to Team or the copy-link button. The matched rule, page response and redirect chain give them specific evidence to work with, while intentional training restrictions can stay in place.
After a change, check the snapshot date: results may be reused for up to 24 hours. Use server logs to verify an urgent fix. To explore the content captured from your page, try Humans vs LLMs. For broader checks and recommendations, run the free AI SEO audit.
It checks robots.txt permissions for the exact page you enter, including search crawlers, training bots and user-requested fetches. Your report also shows the page response, redirects, text available before JavaScript runs, and a warning if we find a noindex instruction. It does not ask an AI model about your brand or measure citations.
No. The percentage counts the robots.txt permissions across search, training and user-requested fetches. If you intentionally block training bots, that can lower the percentage without being an SEO problem. Review each group against how you want your content used, rather than aiming for 100%.
GPTBot collects content for training. OAI-SearchBot is a separate crawler for search. You can choose to block training while permitting search. ChatGPT-User handles user-requested fetches and has a different purpose again.
Googlebot is the crawler for Google Search, including AI Overviews and AI Mode. Google-Extended controls certain training and grounding uses outside Search. Allowing Google-Extended is not a requirement for appearing in AI Overviews.
Yes. A CDN, firewall or bot challenge can prevent retrieval independently of robots.txt. This tool checks the response to an APR request; it does not impersonate a verified vendor IP. Confirm actual vendor access in your server or CDN logs.
No. An allowed policy is only one technical condition. Retrieval, indexing, relevance and citation selection are separate decisions. This checker does not prove a real crawler visited or guarantee a mention in any answer.
A 404 or 410 means no robots.txt rules were found at that location; it is not automatically a block. A timeout, server error, access-denied response or challenge is reported as unknown, rather than being treated as permission.
Google says you do not need new AI text files or special schema to appear in AI Overviews or AI Mode. This checker focuses on observable access and page signals rather than scoring an optional llms.txt file as a ranking requirement.
Yes. Sign in with Google or email to run a free check. You can start 3 new page scans per 24 hours per IP address. Successful results may be reused for up to 24 hours and show their check time. Saved reports remain available at their shareable URL. Anyone with the report link can view it without signing in. Large responses, slow sites and excessive redirects are bounded, and a service-wide limit protects availability.