Integrations

Cloudflare Worker — AI Traffic for SearchMention

This guide walks you through connecting your storefront (on Cloudflare) to SearchMention AI Traffic, so you can see AI crawlers and human visits referred by AI apps in your dashboard.

What you need

Requirement	Details
Cloudflare	Your shop’s domain uses Cloudflare (DNS proxied / orange cloud).
SearchMention	A project whose domain matches the site this worker runs on.
Plan	Starter or Growth — AI Traffic API keys are not available on Free. Generate a key under Dashboard → Settings.
Tools	Node.js (for Wrangler CLI), or use the Cloudflare dashboard to create routes after deploy.

The worker never blocks, slows, or changes what shoppers see. It only sends small background requests to SearchMention when it spots AI-related traffic.

How it works

A request hits your site through Cloudflare.
The worker inspects User-Agent (AI bots) and, if needed, Referer (clicks from ChatGPT, Perplexity, etc.).
If nothing matches, the request continues with no extra work.
If it matches, your origin still responds as usual; the worker asynchronously POSTs a summary to SearchMention’s API.
Data appears under Dashboard → AI Traffic for that project.

Before you deploy (checklist)

Domain is on Cloudflare and traffic goes through the Worker route you will add.
SearchMention project domain equals your store hostname (e.g. www.yourstore.com vs yourstore.com — be consistent with how you track the project).
You have an API key (sm_live_…) from Settings on that project.
You created a folder on your computer for this worker (e.g. searchmention-ai-tracker/).

Step 1 — Create `wrangler.toml`

In your project folder, create wrangler.toml. You can copy this template and adjust the worker name if you like:

name = "searchmention-ai-tracker"
main = "worker.js"
compatibility_date = "2024-01-01"

# Optional: define routes here instead of in the dashboard (uncomment and edit):
# routes = [
#   { pattern = "www.yourstore.com/*", zone_name = "yourstore.com" }
# ]

Notes

name is the Worker name in Cloudflare.
main must point to worker.js (the script below / in this repo).
Routes can be set in this file or in Workers & Pages → your worker → Triggers (see Step 6).

Step 2 — Add `worker.js`

Place worker.js in the same folder as wrangler.toml.

On the SearchMention docs site: use the “Worker script” code block at the bottom of the integrations page — copy the full file.
From the Git repository: use cloudflare-worker/worker.js.

Do not edit the API URL unless you are self-hosting (see Self-hosted SearchMention).

Step 3 — Install Wrangler and log in

npm install -g wrangler
wrangler login

A browser window opens so you can authorize the CLI to your Cloudflare account.

Step 4 — Set your SearchMention API key (secret)

From the folder that contains wrangler.toml and worker.js:

wrangler secret put SEARCHMENTION_API_KEY

Paste your key when prompted (it starts with sm_live_). This value is not stored in wrangler.toml and is encrypted by Cloudflare.

Step 5 — Deploy the Worker

wrangler deploy

Wrangler prints your Worker URL and confirms the deployment. If something fails, check that you are in the correct directory and logged in (wrangler whoami).

Step 6 — Route traffic through the Worker

The Worker must run for requests to your storefront. Two common options:

Option A — Route in Cloudflare dashboard

Open Cloudflare dashboard → select your zone (domain).
Go to Workers Routes (or Workers & Pages → your worker → Triggers / Routes, depending on UI).
Add a route that covers your site, for example:
- yourstore.com/*
- www.yourstore.com/*
  Add both if you use both hostnames.
Save. Propagation is usually quick (often under a minute).

Option B — Route in `wrangler.toml`

Uncomment and set the routes block with your real domain and zone, then run wrangler deploy again.

Route setting — Fail open (required for storefronts)

After your route exists, open its settings in the Cloudflare dashboard (route details / edit route — exact labels depend on the current UI).

Find the failure / limit behavior and choose:

Setting	What it does
Fail open (proceed)	If the Worker cannot run (errors, limits, etc.), additional requests bypass the Worker and go straight to your origin. Shoppers still get your normal site.
Fail closed (block)	Requests may not reach your origin when the Worker fails — bad for a live store.

Turn on Fail open (proceed). SearchMention only needs a best-effort beacon; your checkout and catalog must keep working even if the Worker hiccups. The wording in the dashboard is often along the lines of: “Additional requests will bypass your Worker and proceed to your origin.”

If Cloudflare ever recommends “fail closed” for security-sensitive Workers, that does not apply here — this Worker does not enforce auth or block traffic; it only reports AI-related visits in the background.

Step 7 — Confirm in SearchMention

Open Dashboard → AI Traffic for the project whose API key you used.
Generate a bit of test traffic if needed (e.g. visit your site with a normal browser — non-AI traffic won’t show; AI bot or referral traffic will after it occurs).
If nothing appears, see Troubleshooting below.

Self-hosted SearchMention

If you run your own app URL (not searchmention.com), set the ingest URL as a secret:

wrangler secret put SEARCHMENTION_ENDPOINT

Enter the full URL to the visits endpoint, e.g. https://your-domain.com/api/v1/visits. The default in the worker script points at the hosted SearchMention API.

What gets detected (reference)

SearchMention stores visits when the User-Agent matches a known AI bot or the Referer matches a known AI app. The lists evolve; the server applies the canonical rules in production.

AI bots (examples)

Name	Company
GPTBot, ChatGPT-User, OAI-SearchBot	OpenAI
ClaudeBot, anthropic-ai	Anthropic
PerplexityBot	Perplexity
Google-Extended, Googlebot	Google
Bytespider	ByteDance
CCBot	Common Crawl
Applebot-Extended	Apple
Amazonbot	Amazon
Meta-ExternalAgent	Meta
cohere-ai	Cohere

AI referrals (human clicks from)

Examples include ChatGPT, Perplexity, Gemini, Claude, Copilot, You.com, Phind — matched by referrer host (e.g. chatgpt.com, perplexity.ai).

Troubleshooting

Problem	What to check
No data in AI Traffic	Worker route covers the hostname shoppers use; API key is for the same SearchMention project as that store domain.
`401` / invalid key	Regenerate key in Settings (paid plan), update secret: `wrangler secret put SEARCHMENTION_API_KEY`.
Worker not running	Route pattern matches zone; DNS proxied through Cloudflare; no conflicting Worker higher in the route list.
Only some visits missing	Traffic must match bot/referrer rules; normal visitors are not sent.
Free plan	API keys for AI Traffic require Starter or Growth.

Quick command summary

npm install -g wrangler && wrangler login
wrangler secret put SEARCHMENTION_API_KEY
wrangler deploy

Then add routes in Cloudflare so yourstore.com/* (and www if needed) invoke this Worker.

More detail

HTTP API (for custom integrations): see SearchMention’s docs/ai-traffic-api.md in the app repository.

Worker script — `worker.js`

Save as worker.js next to wrangler.toml, then run wrangler deploy.

/**
 * SearchMention AI Traffic Tracker — CloudFlare Worker
 *
 * Detects AI bot visits AND human visitors referred by AI platforms
 * (ChatGPT, Perplexity, Gemini, Claude, etc.) and reports them to
 * the SearchMention API. Runs on every request but only sends a
 * beacon when relevant traffic is detected.
 * Never blocks or modifies the response to the visitor.
 */

const AI_BOT_PATTERNS = [
  { name: "GPTBot",             pattern: /GPTBot/i },
  { name: "ChatGPT-User",      pattern: /ChatGPT-User/i },
  { name: "OAI-SearchBot",     pattern: /OAI-SearchBot/i },
  { name: "ClaudeBot",         pattern: /ClaudeBot/i },
  { name: "anthropic-ai",      pattern: /anthropic-ai/i },
  { name: "PerplexityBot",     pattern: /PerplexityBot/i },
  { name: "Google-Extended",   pattern: /Google-Extended/i },
  { name: "Bytespider",        pattern: /Bytespider/i },
  { name: "CCBot",             pattern: /CCBot/i },
  { name: "Applebot-Extended", pattern: /Applebot-Extended/i },
  { name: "Amazonbot",         pattern: /Amazonbot/i },
  { name: "Meta-ExternalAgent",pattern: /Meta-ExternalAgent/i },
  { name: "cohere-ai",         pattern: /cohere-ai/i },
];

const AI_REFERRER_DOMAINS = [
  { name: "ChatGPT",    domains: ["chatgpt.com", "chat.openai.com"] },
  { name: "Perplexity",  domains: ["perplexity.ai", "www.perplexity.ai"] },
  { name: "Gemini",      domains: ["gemini.google.com"] },
  { name: "Claude",      domains: ["claude.ai", "www.claude.ai"] },
  { name: "Copilot",     domains: ["copilot.microsoft.com"] },
  { name: "You.com",     domains: ["you.com", "www.you.com"] },
  { name: "Phind",       domains: ["phind.com", "www.phind.com"] },
];

function detectBot(userAgent) {
  if (!userAgent) return null;
  for (const bot of AI_BOT_PATTERNS) {
    if (bot.pattern.test(userAgent)) {
      return bot.name;
    }
  }
  return null;
}

function detectAiReferrer(referer) {
  if (!referer) return null;
  try {
    const host = new URL(referer).hostname.toLowerCase();
    for (const platform of AI_REFERRER_DOMAINS) {
      if (platform.domains.includes(host)) {
        return platform.name;
      }
    }
  } catch (_) {
    // Invalid URL
  }
  return null;
}

async function reportVisit(env, request, response, visitType, referrer) {
  const endpoint = env.SEARCHMENTION_ENDPOINT || "https://searchmention.com/api/v1/visits";
  const apiKey = env.SEARCHMENTION_API_KEY;
  if (!apiKey) return;

  const userAgent = request.headers.get("user-agent") || "";

  const payload = {
    visits: [
      {
        url: request.url,
        user_agent: userAgent,
        visit_type: visitType,
        referrer: referrer || null,
        method: request.method,
        status_code: response.status,
        ip_address: request.headers.get("cf-connecting-ip") || null,
        visited_at: new Date().toISOString(),
        source: "cloudflare",
      },
    ],
  };

  try {
    await fetch(endpoint, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${apiKey}`,
      },
      body: JSON.stringify(payload),
    });
  } catch (_) {
    // Silently ignore — never affect site performance
  }
}

export default {
  async fetch(request, env, ctx) {
    const userAgent = request.headers.get("user-agent") || "";
    const referer = request.headers.get("referer") || "";

    const bot = detectBot(userAgent);
    const aiReferrer = !bot ? detectAiReferrer(referer) : null;

    const response = await fetch(request);

    if (bot) {
      ctx.waitUntil(reportVisit(env, request, response, "bot", null));
    } else if (aiReferrer) {
      ctx.waitUntil(reportVisit(env, request, response, "referral", referer));
    }

    return response;
  },
};