How to Block Bots, Crawlers, Spiders using NGINX


block bots, crawlers, spiders using nginx

Sometimes, you may need to block search engines and crawlers from your website, if it contains sensitive information. You may also need to block malicious bots that artificially inflate your site traffic and spoil conversion rate. Here’s how to block bots, crawlers, spiders using NGINX.

 

How to Block Bots, Crawlers, Spiders using NGINX

Here are the steps to block bots, crawlers and spiders using NGINX.

 

1. Open NGINX config file

Open NGINX config file in a text editor. You will typically find it at /etc/nginx/nginx.conf


$ sudo vim /etc/nginx/nginx.conf

 

2. Block bots, crawlers and spiders

Let’s say you want to block googlebot, bingbot and baiduspider all over your website, you can simply add the following lines to your NGINX config file.


location / {
 if ($http_user_agent ~ (googlebot|bingbot|baiduspider) ) {
  return 403;
 }
}

In the above code, NGINX will check $http_user_agent variable’s values, and if it matches any of the blocked bots, then it will return a “403 : Access Forbidden” response.

The above matching is case insensitive. You can add more bots if you want using the ‘|’ operator.

 

If you want block a specific URL,


location /specific-url-to-page {
 if ($http_user_agent ~ (googlebot|bingbot|baiduspider) ) {
  return 403;
 }
}

 

 

Block bots from crawling a specific file type


location ~* (filename.pdf) {
 if ($http_user_agent ~ (googlebot|bingbot|baiduspider) ) {
  return 403;
 }
}

Block bots from crawling multiple file types


location ~* (.*\.pdf) {
 if ($http_user_agent ~ (googlebot|bingbot|baiduspider) ) {
  return 403;
 }
}

 

If you want to block bots from all .jpg files in a directory


location ~* /uploads/(.+\.jpg)$ {
 if ($http_user_agent ~ (googlebot|bingbot|baiduspider) ) {
  return 403;
 }
}

 

 

3. Reload NGINX web server

Reload NGINX server to apply changes.

$ sudo service nginx reload

 

 

About Sreeram Sreenivasan

Sreeram Sreenivasan is the Founder of Ubiq, a business dashboard & reporting platform for small & medium businesses. Ubiq makes it easy to build business dashboards & reports for your business. Try it for free today!