Methods of bot management and protection

To effectively manage bot traffic and protect against harmful bots, you can use various techniques, including:

• Signature-based detection: Inspecting HTTP requests for known bot signatures
• Rate limiting: Restricting how often certain actions can be performed to prevent abuse
• Challenges: Using JavaScript checks to verify human presence
• Behavioral analysis: Detecting unusual patterns in user activity that suggest automation

With Vercel, you can use:

• Managed rulesets to challenge specific bot traffic
• Rate limiting and challenge actions with WAF custom rules to prevent bot activity from reaching your application
• DDoS protection to defend your application against bot driven attacks
• Observability and Firewall to monitor bot patterns, traffic sources, and the effectiveness of your bot management strategies

Bot protection managed ruleset

all plans

With Vercel, you can use the bot protection managed ruleset to challenge non-browser traffic from accessing your applications. It filters out automated threats while allowing legitimate traffic.

• It identifies clients that violate browser-like behavior and serves a javascript challenge to them.
• It prevents requests that falsely claim to be from a browser such as a curl request identifying as Chrome.
• It automatically excludes verified bots, such as Google's crawler, from evaluation.

To learn more about how the ruleset works, review the Challenge section of Firewall actions. To understand the details of what get logged and how to monitor your traffic, review Firewall Observability.

custom WAF rules with bypass actions that will allow this traffic to skip the bot protection ruleset.

Enable the ruleset

The ruleset is inactive by default. In the dashboard this is labeled Off. Matching traffic is not evaluated and reaches your application.

You can apply the ruleset to your project in log or challenge mode. Learn how to configure the bot protection managed ruleset.

Bot protection ruleset with reverse proxies

Bot Protection doesn't work when a reverse proxy (e.g. Cloudflare, Azure, or other CDNs) is placed in front of your Vercel deployment. This setup significantly degrades detection accuracy and performance, leading to a suboptimal end-user experience.

Reverse proxies interfere with Vercel's ability to reliably identify bots:

• Obscured detection signals: Legitimate users may be incorrectly challenged because the proxy masks signals that Bot Protection relies on.
• Frequent re-challenges: Some proxies rotate their exit node IPs frequently, forcing Vercel to re-initiate the challenge on every IP change.

AI bots managed ruleset

all plans

Vercel's AI bots managed ruleset allows you to control traffic from AI bots that crawl your site for training data, search purposes, or user-generated fetches.

• It identifies and filters requests from known AI crawlers and bots.
• It provides options to log or deny these requests based on your preferences.
• The list of known AI bots is automatically maintained and updated by Vercel.

When new AI bots emerge, Vercel automatically adds them to its managed list and handles them according to your existing configured action without requiring any changes on your part.

Enable the ruleset

The ruleset is inactive by default. In the dashboard this is labeled Allow. Matching traffic is not evaluated and reaches your application.

You can apply the ruleset to your project in log or deny mode. Learn how to configure the AI bots managed ruleset.

Verified bots

Vercel maintains and continuously updates a comprehensive directory of known legitimate bots from across the internet. This directory is regularly updated to include new legitimate services as they emerge. Attack Mode and bot protection automatically recognize and allow these bots to pass through without being challenged. You can block access to some or all of these bots by writing WAF custom rules with the User Agent match condition or Signature-Agent header. To learn how to do this, review WAF Examples.

Bot verification methods

To prove that bots are legitimate and verify their claimed identity, several methods are used:

googleother

search_engine_crawler

Crawling preferences addressed to the GoogleOther user agent don't affect any specific product. GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. It has no effect on Google Search or other products.

View