AI bots & crawlers

Server SDK

Detect and track GPTBot, ClaudeBot, PerplexityBot and other AI crawlers and agents with a small server middleware.

GPTBot, ClaudeBot, PerplexityBot, Google-Extended and most AI agents never run JavaScript, so the browser SDK can't see them. The server integration looks at every request your server handles and reports the ones that come from bots.

No server code to touch? Send your platform's request logs instead with a log drain.

Get the server key

Copy the secret server key (al_srv_…) from Project settings → Setup → AI agents and crawlers (also under Project settings → General). Unlike the public key, it must stay on the server.

.env
AGENTSTATS_SERVER_KEY=al_srv_…

Install

npm i @agentstats/sdk

Add the middleware

import { createAgentStats } from '@agentstats/sdk/server'

const lens = createAgentStats({ serverKey: process.env.AGENTSTATS_SERVER_KEY })

Then hook it into your framework:

// proxy.ts (Next 16; on Next 15, middleware.ts exporting `middleware`)
import { NextResponse } from 'next/server'

export function proxy(request, event) {
  const response = NextResponse.next()
  lens.track(request, { response, waitUntil: p => event.waitUntil(p) })
  return response
}

::

Detect bots in your code

Every adapter also tells your app who's asking, so you can treat AI agents differently (serve Markdown, skip a cookie banner, log them):

FrameworkWhere
Next.jsdetectBot(await headers()) in server components, lens.track() returns it in proxy.ts
Nuxtevent.context.bot in server routes, useRequestEvent()?.context.bot in pages
SvelteKitevent.locals.bot
AstroAstro.locals.bot
Expressres.locals.bot
Honoc.get('bot')

detectBot() also works on its own, with no key: pass it a Request, a Headers object or a Node request.

import { detectBot } from '@agentstats/sdk/server'

const bot = detectBot(request)
// { isBot: true, isAI: true, id: 'gptbot', name: 'GPTBot', category: 'training',
//   product: 'ChatGPT', operator: 'OpenAI', browser: false, signed: false, referral: null }
isBot / isAI
boolean
Any automated client, and the AI ones (categories training, search, assistant, agent).
id / name / product / operator
string | null
Which bot, e.g. claude-user / Claude-User / Claude / Anthropic.
category
string | null
training, search (AI search indexers), assistant (fetched because someone asked an assistant), agent (browses and acts for a user), search_engine or other.
browser
boolean
The agent loads pages in a real browser, so it runs JavaScript and can call your WebMCP tools.
signed
boolean
The request carries Web Bot Auth signature headers.
referral
string | null
For people: the assistant that sent them, e.g. ChatGPT, from the referrer or utm_source.

This is the same bot list the dashboard uses, so your app and your dashboard always agree.

Detection reads the User-Agent, which anyone can fake, so don't use it to grant access. The dashboard verifies every hit against vendor IP ranges, reverse DNS and signatures, and flags spoofed ones.

Options

serverKey
string
Secret server key, al_srv_…. Never expose it to browsers. Without it (say, locally without the env var), the adapters still detect bots but nothing is sent.
apiKey
string
Secret API key, al_api_…, to also track the calls agents make to your API. See API tracking.
countHumans
boolean
Default true. Counts human page views as one number per day, nothing else, so the dashboard can show the AI share of your traffic.
visitCookie
boolean
Default true. Links visits from agents that browse with a real browser to their WebMCP tool calls. See Verification.
trustProxy
boolean
Node / Express only. Whether to read the client IP from proxy headers. Automatic by default, see below.
ipHeader
string
Read the client IP from this header only, e.g. cf-connecting-ip behind Cloudflare. See below.
ignore
RegExp | (path) => boolean
Skip matching paths, such as health checks or internal APIs.
maxBatch / flushIntervalMs
number
Batching for long-running servers. Defaults: 100 hits, 5000 ms.
debug / disabled
boolean
Log to the console, or turn the integration off.

Behind a proxy or load balancer

Behind nginx, Render, Railway, Fly or a cloud load balancer, the client IP comes from forwarding headers (CF-Connecting-IP, X-Real-IP, X-Forwarded-For).

The Node / Express adapter trusts those headers automatically when the connection comes from a private or loopback address, and ignores them on direct public connections so they can't be spoofed. Force either behaviour with trustProxy: true or false.

On Vercel, Netlify, Fly and Cloudflare Workers the SDK reads only the header that platform sets (X-Real-IP, X-Nf-Client-Connection-IP, Fly-Client-IP, CF-Connecting-IP), because the platform overwrites it and clients can't forge it. SvelteKit and Astro use the address the framework's adapter reports.

Anywhere else, if you're behind a CDN or proxy that always sets one header, name it with ipHeader (e.g. ipHeader: 'cf-connecting-ip' behind Cloudflare). Otherwise a client could send a vendor's IP in a forwarding header and pass as a verified crawler.

A hit that still arrives with an internal IP is shown as unverified, never as spoofed.

Pages served from a CDN cache never reach your server, so crawler hits on them aren't counted. Use a log drain from your CDN to cover those.