Blog · AI crawler log analysis
Agent Analytics: How to Read AI Crawler Behavior Logs
Priya Bothra · September 10, 2026
AI crawler logs show which AI systems request your pages, which pages they fetch, how often, and what your server returns. Reading them well starts with one distinction: there are three kinds of AI bots. Training crawlers collect content for model training, search crawlers build indexes that AI answers retrieve from, and user-triggered fetchers visit a page because someone asked an assistant about it. Each tells you something different.
Logs are one of the few direct views you have into how AI systems interact with your site. They can reveal blocked crawlers, priority pages AI systems never fetch, errors served only to bots and which content users are sending assistants to read. This guide explains how to identify AI agents, verify they are real, and interpret what they are doing.
The three categories of AI agents
| Category | Examples | What a hit means | robots.txt |
|---|---|---|---|
| Training crawlers | GPTBot (OpenAI), ClaudeBot (Anthropic) | Content may be collected for model training | Respected per provider documentation |
| Search crawlers | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot | Page may be indexed for AI search answers | Respected per provider documentation |
| User-triggered fetchers | ChatGPT-User, Claude-User, Perplexity-User | A user's request caused the assistant to read the page | May not apply, per some providers |
Sources: OpenAI crawler documentation, Anthropic's crawler descriptions via Search Engine Land, Perplexity crawler documentation.
A few specifics from provider documentation:
- OpenAI states that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers," and that because ChatGPT-User actions are initiated by users, "robots.txt rules may not apply."
- Perplexity states that PerplexityBot is used to surface and link websites in search results and is not used to crawl content for training foundation models, and that Perplexity-User "generally ignores robots.txt rules" because a user triggered the request.
- Anthropic's three bots each respect robots.txt and need separate rules.
- Google's AI features use regular Googlebot crawling. Google-Extended is a robots.txt control token for Gemini training and grounding uses, not a separate crawler that appears in logs.
Step 1: Collect the right log data
You need raw request logs, not just analytics. JavaScript-based analytics tools generally do not see crawlers, because bots often do not execute tracking scripts.
Useful fields:
- Timestamp
- Requested URL
- HTTP status code
- User agent string
- Client IP address
- Response size and time
- Referrer, where present
Sources include your web server, load balancer, CDN logs or an edge worker. If your CDN serves cached pages, the origin server may not see every request, so CDN-level logs are often more complete.
Step 2: Identify AI agents by user agent
Filter for known AI user agent tokens, such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and Googlebot variants. Maintain the list over time, because providers add and rename agents.
Step 3: Verify that bots are real
User agent strings are easy to fake. Scrapers sometimes impersonate well-known bots. Verify before drawing conclusions:
- OpenAI publishes IP ranges for its bots, for example at
openai.com/searchbot.jsonfor OAI-SearchBot, as referenced in its ChatGPT search help documentation. - Perplexity publishes IP ranges for its crawlers in its documentation.
- Googlebot can be verified with reverse DNS lookup followed by a forward lookup, as described in Google's documentation.
Traffic that claims to be an AI bot but comes from unlisted IPs should be treated as unverified.
Step 4: Answer the key questions
Can AI search crawlers reach your priority pages?
List your 20 to 50 most important pages: pricing, product, comparison, security and core guides. Check whether verified search crawlers requested them in the last 30 to 90 days and what status code they received.
What to look for: priority pages with no search crawler hits, or hits returning 403, 429 or 5xx errors. A 403 served only to bots often points to CDN or firewall bot protection.
Are bots receiving different content from users?
Compare response sizes for bot requests and normal requests to the same URL. Very small responses to bots can indicate challenge pages, blocked resources or JavaScript-only content.
What do user-triggered fetches tell you?
A ChatGPT-User, Claude-User or Perplexity-User hit means an assistant fetched that page in response to a user's request. Patterns in these hits are a directional demand signal: which pages people, or the assistants working for them, are reading. Spikes on a pricing or comparison page can indicate active evaluation. These hits do not identify users and should not be treated as visits.
Is training crawl activity consistent with your policy?
If you have decided to block training crawlers, confirm verified GPTBot and ClaudeBot requests stop, allowing for caching of robots.txt. If you allow them, no action is needed.
Are crawlers wasting effort?
Heavy crawling of faceted URLs, parameter variants or low-value archives can indicate crawl waste. Consider canonicalization, internal linking and robots.txt rules for low-value paths, being careful not to block important content.
Step 5: Turn findings into actions
| Finding | Likely cause | Action |
|---|---|---|
| No OAI-SearchBot hits on any page | robots.txt disallow or CDN block | Review rules; allow the search crawler |
| 403 responses only for AI bots | Bot protection settings | Allowlist verified AI search crawler IPs |
| Priority pages never fetched | Weak internal linking or not in sitemap | Improve linking, update sitemap |
| Small responses to bots | JavaScript-dependent content | Server-side rendering for key content |
| Spike in ChatGPT-User hits on pricing | Active buyer research | Check pricing page accuracy and clarity |
| Unverified "GPTBot" traffic | Impersonating scraper | Block by IP or rate-limit |
Reporting cadence
A practical rhythm:
- Weekly: error rates for AI search crawlers, spikes in user-triggered fetches.
- Monthly: crawl coverage of priority pages by provider, top fetched pages.
- Quarterly: policy review for training crawlers, updated user agent list.
Common mistakes
Relying on JavaScript analytics. Most bots never appear there.
Trusting user agents without verification. Spoofing is common.
Blocking everything labeled AI. Blocking search crawlers can remove you from AI answers.
Reading too much into single hits. Look at patterns over weeks.
Ignoring CDN logs. Cached responses may never reach your origin.
A hypothetical example
A hypothetical SaaS company reviews 60 days of CDN logs. Verified OAI-SearchBot and PerplexityBot requests reach the blog but receive 403 responses on the pricing and security pages, which sit behind a stricter firewall rule set up after a scraping incident. ChatGPT-User hits on the pricing page show the same 403s, meaning assistants trying to read pricing for users were blocked. Allowlisting verified crawler IP ranges for those paths fixes the issue. The team then monitors whether assistants begin citing the pricing page.
How Bob Builds AI's Agent Analytics helps
Bob Builds AI's Agent Analytics shows how AI crawlers access your site. According to its documentation, crawl data can be collected through a Cloudflare Worker, a WordPress plugin or an nginx log shipper, which helps teams without direct log access get this view.
FAQ
What is an AI crawler?
An AI crawler is an automated agent that requests web pages on behalf of an AI company. Some collect content for model training, such as GPTBot and ClaudeBot. Others index pages for AI search, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. Others fetch pages when a user asks an assistant to.
How do I see AI crawlers in my logs?
Use raw server, load balancer or CDN logs, not JavaScript analytics. Filter by AI user agent tokens, then verify requests against the IP ranges providers publish or through reverse DNS where supported.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot indexes content for ChatGPT search. OpenAI states that sites opted out of OAI-SearchBot will not appear in ChatGPT search answers, so blocking GPTBot and blocking OAI-SearchBot have different effects.
What does a ChatGPT-User hit mean?
It means ChatGPT fetched the page because a user's request required it. OpenAI says robots.txt rules may not apply to these user-initiated requests. Patterns in these hits can indicate which pages people are asking ChatGPT to read.
Why are AI crawlers getting 403 errors on my site?
Common causes are robots.txt rules, CDN or firewall bot protection, rate limiting and geographic blocking. If verified AI search crawlers receive 403s on important pages, review bot protection settings and consider allowlisting published IP ranges.
Should I block AI training crawlers?
It is a business decision. Blocking training crawlers such as GPTBot, ClaudeBot or the Google-Extended token limits use of your content for training. Most providers separate training from search crawlers, so you can block training while still allowing AI search visibility.
How often do AI crawlers visit a site?
It varies widely by site size, popularity, update frequency and provider. Establish your own baseline from logs and watch for changes, especially on priority pages.
Conclusion
AI crawler logs show whether AI systems can reach your most important pages, what they receive when they do and which pages users send assistants to read. Separating training, search and user-triggered agents, verifying identities and focusing on priority pages turns raw log lines into clear fixes.
Start by pulling 30 days of CDN or server logs and checking one question: did verified AI search crawlers successfully fetch your pricing, product and comparison pages? Bob Builds AI's Agent Analytics can collect and report this view continuously.