Free AI Crawler Checker

Find what could stop AI agents and search engines from reading your page. See what’s accessible, what’s blocked and what needs your attention.

About your free AI Crawler Check

One page, a clearer picture

Check one public page for access barriers, page errors and readable text. Search and training permissions are shown separately. Your findings are saved at a report URL you can share with your developer.

Free checks and saved results

You can start 3 new page scans per 24 hours per IP address. Successful results may be reused for up to 24 hours without another scan. Check the timestamp after making changes. Saved reports remain available at their shareable URL.

Access doesn’t guarantee visibility

An allowed result doesn’t prove an AI agent visited or guarantee indexing, rankings or citations. It doesn’t measure whether your brand appears in AI answers.

Explore agency plans

See what your report reveals

Spot potential barriers to AI access, understand what needs attention on your page, and get clear findings you can share with your developer or team.

What Does the AI Crawler Checker Check?

A focused technical check of one public page, with the evidence behind each finding. No unexplained AI visibility score.

  • Exact-page crawler permissions

    See whether robots.txt permits each search, training and user-request agent to fetch the selected URL.

  • Matched rules, not a mystery score

    Find the winning Allow or Disallow directive and line number. Fetch failures remain unknown.

  • HTTP response & redirects

    Check the final response and redirect count. A denied or challenged APR request is shown separately from robots policy.

  • Warnings about indexing

    See a warning when the page tells supporting crawlers not to index it, even if robots.txt allows access.

  • Text without JavaScript

    Count the text in the initial HTML. Follow up with Humans vs LLMs if the page may depend on rendering.

  • A report you can share

    Keep your timestamped findings at a saved report URL. Copy the link to share with a developer or client. Sign in with Google or email to run your free check.

How to check AI crawler access

01

Enter the exact page

Sign in with Google or email, then use a public homepage, service page or article URL. Rules can differ by path, so a homepage check does not describe the whole site.

02

Read each signal separately

Review search crawler permissions first, then the page response, redirects and available text. Training permissions are a separate decision.

03

Review the fix and verify

Find the matched rule, correct accidental restrictions, and share your saved report. Use logs and your search tools to verify actual crawler behavior.

Can AI crawlers reach your page?

Your page opens in your browser. Can a crawler get to it too? Start with Page Access Overview: check the page response, whether robots.txt is available, and how much text arrives before JavaScript runs.

A missing page, an access-denied response or a bot challenge gives you somewhere to investigate. Redirects show the route to the final page, including any stops on an insecure HTTP connection. If very little text is available in the initial HTML, check whether your main content depends on JavaScript. That alone does not mean every AI service is unable to read it.

You’ll also see a warning if we find a noindex instruction. A crawler that supports it may be allowed to read the page but told to leave it out of its index. The response we check is to APR’s request; your server or firewall logs can confirm what happens when a real vendor’s crawler visits.

If you want people to find your page through AI search, start with Search & Discovery. It checks the robots.txt rules for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot: the search crawlers used by OpenAI, Anthropic, Perplexity, Google and Microsoft.

Check the page you care about, rather than stopping at your homepage. Your homepage could be allowed while a rule for /services blocks your service pages. Each bot’s card shows the matching rule and its line number, so your developer knows where to look. An allowed result means the file permits that crawler to fetch the page; it does not promise a place in an AI answer.

Set your preferences for AI training

You may want your pages found in search while limiting their use for training. Those are separate choices. OpenAI uses GPTBot for training and OAI-SearchBot for search. Anthropic separates ClaudeBot from Claude-SearchBot in a similar way.

Training & Other AI Uses also covers CCBot, which collects pages for Common Crawl’s public dataset, and Google-Extended. Google-Extended is a permission setting for certain Gemini training and grounding uses, rather than a separate visiting bot. Blocking it does not block Google Search or its AI features.

The AI Bots Allowed percentage counts robots.txt permissions across all three groups. It is not an AI readiness or visibility score. An intentional training block can lower the percentage without being an SEO problem.

What happens when someone asks an AI assistant to open your link?

Someone pastes your article into an assistant and asks for a summary. That can trigger a visit on their behalf, rather than a background search crawl. User-Requested Fetches checks the declared rules for ChatGPT-User, Claude-User and Perplexity-User.

Providers handle these requests differently. Anthropic says Claude-User follows robots.txt. OpenAI says those rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Use this section to understand your stated preferences, then check access logs if you need to confirm what was actually retrieved.

Use your report to decide what to fix

Start with an unexpected block on a page you want people to discover. Share the report with your developer using Email Report to Team or the copy-link button. The matched rule, page response and redirect chain give them specific evidence to work with, while intentional training restrictions can stay in place.

After a change, check the snapshot date: results may be reused for up to 24 hours. Use server logs to verify an urgent fix. To explore the content captured from your page, try Humans vs LLMs. For broader checks and recommendations, run the free AI SEO audit.

Frequently Asked Questions

It checks robots.txt permissions for the exact page you enter, including search crawlers, training bots and user-requested fetches. Your report also shows the page response, redirects, text available before JavaScript runs, and a warning if we find a noindex instruction. It does not ask an AI model about your brand or measure citations.

No. The percentage counts the robots.txt permissions across search, training and user-requested fetches. If you intentionally block training bots, that can lower the percentage without being an SEO problem. Review each group against how you want your content used, rather than aiming for 100%.

GPTBot collects content for training. OAI-SearchBot is a separate crawler for search. You can choose to block training while permitting search. ChatGPT-User handles user-requested fetches and has a different purpose again.

Googlebot is the crawler for Google Search, including AI Overviews and AI Mode. Google-Extended controls certain training and grounding uses outside Search. Allowing Google-Extended is not a requirement for appearing in AI Overviews.

Yes. A CDN, firewall or bot challenge can prevent retrieval independently of robots.txt. This tool checks the response to an APR request; it does not impersonate a verified vendor IP. Confirm actual vendor access in your server or CDN logs.

No. An allowed policy is only one technical condition. Retrieval, indexing, relevance and citation selection are separate decisions. This checker does not prove a real crawler visited or guarantee a mention in any answer.

A 404 or 410 means no robots.txt rules were found at that location; it is not automatically a block. A timeout, server error, access-denied response or challenge is reported as unknown, rather than being treated as permission.

Google says you do not need new AI text files or special schema to appear in AI Overviews or AI Mode. This checker focuses on observable access and page signals rather than scoring an optional llms.txt file as a ranking requirement.

Yes. Sign in with Google or email to run a free check. You can start 3 new page scans per 24 hours per IP address. Successful results may be reused for up to 24 hours and show their check time. Saved reports remain available at their shareable URL. Anyone with the report link can view it without signing in. Large responses, slow sites and excessive redirects are bounded, and a service-wide limit protects availability.

For SEO & Digital Marketing Agencies

Turn a technical finding into a client conversation

Add APR's white-label AEO audit widget to your agency website. Give prospects a broader readiness report and collect a lead your team can follow up.