Blocking AI Crawlers

AI crawlers have become a common presence across the web, scanning websites and collecting large volumes of content. While some serve legitimate purposes, others operate in ways that are overly aggressive, consuming resources and capturing information that site owners may want to keep private. To maintain control and protect valuable content, it is important to have effective measures in place for managing and blocking unwanted AI crawler access.

Prerequisites

• Contentstack account
• Access to Launch for your organization

Launch provides two ways to help you control access by AI crawlers:

1. Using a robots.txt file.
2. Using Contentstack Launch Edge Functions to block crawlers at runtime.

Using robots.txt to Disallow AI Crawlers

The robots.txt file provides crawl instructions for compliant bots. You can use it to disallow specific User-Agent strings from accessing certain parts of your site.

Here’s a sample robots.txt to block common AI crawlers:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: ai-crawler
Disallow: /

User-agent: *
Disallow: /private-directory/

Note: Some bots may ignore the robots.txt file and continue crawling your site. To strictly block AI crawlers, you can enforce the restriction at the edge using Launch Edge Functions.

How to Add robots.txt in Next.js

• Using App router
• Using Pages router

Blocking AI Crawlers Using Launch Edge Functions

As bots can ignore the robots.txt file, you can use Launch Edge Functions to detect and block suspicious User-Agent strings in real time.

Example Launch Edge Function

const KNOWN_BOTS = [
  'claudebot',
  'gptbot',
  'googlebot',
  'bingbot',
  'ahrefsbot',
  'yandexbot',
  'semrushbot',
  'mj12bot',
  'facebookexternalhit',
  'twitterbot',
  //more bots can be added here
];

export default function handler(request) {
  const userAgent = (request.headers.get('user-agent') || '').toLowerCase();

  const isBot = KNOWN_BOTS.some(bot => userAgent.includes(bot));

  if (isBot) {
    return new Response('Forbidden: AI crawlers are not allowed.', { status: 403 });
  }

  return fetch(request);
}

Deployment Instructions on Contentstack Launch

1. Refer to the Launch Edge Functions documentation for setup.
2. Add the edge function to your project’s edge runtime entry point.
3. Deploy using your Launch pipeline. The edge function will begin filtering requests before they reach your backend or frontend.

Best Practices & Recommendations

1. Regularly update the list of known AI crawler User-Agents to ensure ongoing effectiveness.
2. Use a robots.txt file to guide compliant bots and edge functions to block non-compliant or deceptive bots.
3. Review server logs and User-Agent headers periodically to refine detection and blocking rules.

Conclusion

While the robots.txt file helps communicate your site's crawling preferences, runtime protections like Launch Edge Functions offer more reliable control, especially in an AI-driven environment where bot behavior is increasingly unpredictable.