Installation

Installing Blackhole for Bad Bots

1. Upload the Blackhole plugin to your blog and activate
2. Visit the Blackhole Settings and copy the Robots Rules
3. Add the Robots Rules to your site’s robots.txt file (see note)*
4. Configure the Blackhole Settings as desired and done

Note: For the robots.txt rules, there are two scenarios:

1. Your site has a physical robots.txt file that you can see on the server. In this case, you need to add the required rules manually.
2. OR, your site is using the dynamic/virtual WP-generated robots.txt file, and there is no physical robots.txt file on your server. In this case, the plugin adds the required rules automatically. You do not need to add anything manually.

For complete documentation, click the “Help” tab in the upper-right corner of the Blackhole settings screen. Help tab also available on the “Bad Bots” screen.

More info on installing WP plugins

Verify Blackhole is working properly

Using a caching plugin? Check out the section below called “Caching Plugins” for important info.

No robots.txt?

For the robots.txt file, there are two possible scenarios:

1. You want to use your own physical robots.txt file that you can view and edit on the server. In this case, follow the steps below to create your site’s robots.txt file.
2. OR, you want to use the dynamic/virtual WP-generated robots.txt file, such that there is no physical robots.txt file on your server. In this case, you don’t need to do anything, because WordPress automatically generates a robots.txt file when requested.

If you go with option #1, here are the steps to create a robots.txt file for your site:

1. Add a blank plain-text file to the root directory of your site
2. Name the text file robots.txt and upload to your server

Done. Now you can add the Blackhole rules provided on the plugin settings page. See the next section to learn more and validate your robots.txt file.

To view your robots.txt file, visit the following URL (replace example.com with your domain):

https://example.com/robots.txt

Tip: you can find a link to your site’s robots.txt file on the plugin settings page.

Robots Tools & Info

• Learn more about robots.txt
• Validate your robots.txt file
• Validate robots.txt in Google Webmaster Tools
• Google Robots.txt Specifications
• How to Create a robots.txt file

Lots more great resources on the web to learn about and validate your robots.txt file. Read up, it’s important for SEO.

Caching Plugins

Blackhole works with any type of caching plugin where “page caching” is not enabled.

There are many types of cache plugins. They provide all sorts of different caching mechanisms and features. All caching features work great with Blackhole except for “page caching”. With page caching, the required WP init hook may not be fired, which means that plugins like Blackhole are not able to log and ban requests dynamically. Fortunately, some of the most popular caching plugins provide settings that enable full compatibility with Blackhole. For a complete list, check out this article. Note: that article was written for Blackhole Pro, but the compatibility list and general info apply also to Blackhole (free version).

Testing

To test that the Blackhole trap is working, view the source code of any web page on your site. Scroll down near the footer of the page until you locate a link that looks similar to the following:

<a rel="nofollow" style="display:none;" href="https://example.com/?blackhole=1234567890" title="Do NOT follow this link or you will be banned from the site!">Name of Your Website</a>

Click the link (the href value) to view the Warning Message. After visiting the Warning Message, refresh the page to view the Access Denied message. And/or visit any other page on the front-end of your site to verify that you have been banned. But don’t worry, you will never be banned from the WP Admin Area or the WP Login Page. So simply log in and remove your IP address from the Bad Bots list to restore front-end access.

More information on how to verify Blackhole is working »

Why no bots?

If you’re not seeing any bad bots getting blocked, there are several things to keep in mind:

• Make sure you’ve set up according to the docs above
• New(er) websites may not get a lot of bad bot traffic
• Sites with low traffic may not get a lot of bad bots
• Check if you are using any other bot-blocking plugins
• Not all websites (even popular ones) get tons of bots
• Blackhole will not work if you have page caching on site
• If in doubt, you can test if the plugin is working (see previous section above)

So keep those things in mind. In most cases it’s just a matter of time before some bad bots fall into the black hole.

Whitelisted Bots

Blackhole for Bad Bots is rigorously tested to ensure that the top search engine bots are NEVER BLOCKED. Any bots reporting a User Agent that contains any of the following strings will always have access to your site, even if they disobey robots.txt.

a6-indexer, adsbot-google, ahrefsbot, aolbuild, apis-google, baidu, bingbot, bingpreview, butterfly, cloudflare, chrome, duckduckgo, embedly, facebookexternalhit, facebot, google page speed, googlebot, ia_archiver, linkedinbot, mediapartners-google, msnbot, netcraftsurvey, outbrain, pinterest, quora, rogerbot, showyoubot, slackbot, slurp, sogou, teoma, tweetmemebot, twitterbot, uptimerobot, urlresolver, vkshare, w3c_validator, wordpress, wp rocket, yandex

Of course, this list is completely customizable via the plugin settings. Each added string is matched against the full user agent, so be careful. Learn more about user agents of the top search engines.

You can also whitelist bots by IP address. Visit the setting, “Whitelisted IPs”, and enter the IP address (separate multiple IPs with commas). You can also whitelist entire ranges of IPs. In the same plugin setting, add something like this:

123.456.

That will allow all bots reporting any IP that begins with 123.456.. You can also whitelist IP addresses using CIDR notation. Check out the Help tab on the plugin settings page for details.

Customizing

Blackhole provides plenty of hooks for customizing and extending:

blackhole_options
blackhole_badbots
blackhole_get_options
blackhole_get_badbots
blackhole_log_data
blackhole_trigger
blackhole_vars
blackhole_log
blackhole_ip_keys
blackhole_alert_name
blackhole_alert_subject
blackhole_alert_message
blackhole_alert_headers
blackhole_needle
blackhole_message_default
blackhole_message_custom
blackhole_message_nothing
blackhole_ignore_loggedin
blackhole_ignore_backend
blackhole_ignore_login
blackhole_block_status
blackhole_block_protocol
blackhole_block_connection
blackhole_ip_filter
blackhole_validate_ip_log
blackhole_settings_contextual_help
blackhole_badbots_contextual_help

If you need a hook added, drop me a line, will be glad to hook it up 😉

Custom Warning Template

The Blackhole displays two types of messages:

• Warning Message – Displayed when bots follow the blackhole trigger
• Blocked Message – Displayed for all requests made by blocked bots

The Blocked Message may be customized via the plugin settings. The Warning Message may be customized by setting up a custom template:

1. Copy blackhole-template.php from the plugin’s /inc/ directory
2. Paste the file into your theme template, for example: /wp-content/my-awesome-theme/blackhole-template.php
3. Customize any of the markup between “BEGIN TEMPLATE” and “END TEMPLATE”
4. Upload to your server and done

If the custom template exists in your theme directory, the plugin automatically will use it to display the Warning Message. If the custom template does not exist in your theme directory, the plugin will fallback to the default warning message.

Tip: Instead of including the custom template in your theme, you can include via /wp-content/ directory, like: /wp-content/blackhole/blackhole-template.php

More options available in the Pro version »

Uninstalling

Blackhole for Bad Bots cleans up after itself. All plugin settings and the bad bot list will be removed from your database when the plugin is uninstalled via the Plugins screen. After uninstalling, don’t forget to remove the blackhole rules from your robots.txt file. It’s fine to leave them in place, it will not hurt anything, but they serve no purpose without the plugin installed.

More specifically, Blackhole adds only two things to the database: options and bot list. When the plugin is uninstalled/deleted via the Plugins screen, both of those items are removed automatically via the following lines in uninstall.php:

delete_option('bbb_options');
delete_option('bbb_badbots');

So after uninstalling the plugin and deleting the robots.txt rules, there will be no trace of Blackhole for Bad Bots on your site.

Like the plugin?

If you like Blackhole for Bad Bots, please take a moment to give a 5-star rating. It helps to keep development and support going strong. Thank you!