Extension:SpamBlacklist
![]() Release status: stable |
|
---|---|
![]() |
|
Implementation | Page action |
Description | Provides a regex-based spam filter |
Author(s) | Tim Starlingtalk |
Latest version | Continuous updates |
Compatibility policy | Snapshots releases along with MediaWiki. Master is not backward compatible. |
MediaWiki | 1.31+ |
License | GNU General Public License 2.0 or later |
Download | README |
|
|
|
|
Translate the SpamBlacklist extension if it is available at translatewiki.net | |
Issues | Open tasks · Report a bug |
The SpamBlacklist extension prevents edits that contain URLs whose domains match regular expression patterns defined in specified files or wiki pages and registration by users using specified email addresses.
When someone tries to save a page, this extension checks the text against a (potentially very large) list of illegal host names. If there is a match, the extension displays an error message to the user and refuses to save the page.
Installation and setup[edit]
Installation[edit]
- Download and place the file(s) in a directory called
SpamBlacklist
in yourextensions/
folder. - Add the following code at the bottom of your LocalSettings.php:
wfLoadExtension( 'SpamBlacklist' );
- Configure the block list at your convenience
Done – Navigate to Special:Version on your wiki to verify that the extension is successfully installed.
Setting the block list[edit]
The following local pages are always used, whatever additional sources are listed:
- MediaWiki:Spam-blacklist
- MediaWiki:Spam-whitelist
- MediaWiki:Email-blacklist
- MediaWiki:Email-whitelist
The default additional source for a block list of forbidden URLs is the Wikimedia spam block list on Meta-Wiki, at m:Spam block list. By default, the extension uses this list, and reloads it once every 10-15 minutes. For many wikis, using this list will be enough to block most spamming attempts. However, since the Wikimedia block list is used by a diverse group of large wikis with hundreds of thousands of external links, it is comparatively conservative in the links it blocks.
The Wikimedia spam block list can only be edited by administrators; but you can suggest modifications to the block list at m:Talk:Spam blacklist.
You can add other bad URLs on your own wiki. List them in the global variable $wgBlacklistSettings
in LocalSettings.php . See examples below.
$wgBlacklistSettings
is an two level array. Top level key is spam
or email
. They take an array with each value containing either a URL, a filename or a database location.
If you use $wgBlacklistSettings
in "LocalSettings.php", the default value of "[[m:Spam blacklist]]" will no longer be used - if you want that block list to be accessed, you will have to add it in manually, see examples below.
Specifying a database location allows you to draw the block list from a page on your wiki.
The format of the database location specifier is ">DB: [db name] [title]". [db name] should exactly match the value of $wgDBname
in LocalSettings.php . You should create the required page name [title] in the default namespace of your wiki. If you do this, it is strongly recommended that you protect the page from general editing. Besides the obvious danger that someone may add a regex that matches everything, please note that an attacker with the ability to input arbitrary regular expressions may be able to generate segfaults in the PCRE library.
Examples[edit]
If you want to, for instance, use the English-language Wikipedia's spam block list in addition to the standard Meta-Wiki one, you could call the following in LocalSettings.php , AFTER wfLoadExtension( 'SpamBlacklist' );
call:
$wgBlacklistSettings = [
'spam' => [
'files' => [
"https://meta.wikimedia.org/w/index.php?title=Spam_blacklist&action=raw&sb_ver=1",
"https://en.wikipedia.org/w/index.php?title=MediaWiki:Spam-blacklist&action=raw&sb_ver=1"
],
],
];
Here is an example of an entirely local set of block lists: the administrator is using the update script to generate a local file called "wikimedia_blacklist" that holds a copy of the Meta-Wiki blacklist, and has an additional block list on the wiki page "My spam block list":
$wgBlacklistSettings = [
'spam' => [
'files' => [
"$IP/extensions/SpamBlacklist/wikimedia_blacklist", // Wikimedia's list
// database title
'DB: wikidb My_spam_block_list',
],
],
];
Logging[edit]
By default, the extension does not log hits into the spam blacklist log.
To enable logging set $wgLogSpamBlacklistHits = true;
.
You can use the spamblacklist
user right to control access to the logs.
Every signed-in user can view the logs by default.
Issues[edit]
Backtrack Limit[edit]
If you encounter issues with the block list, you may want to increase the backtrack limit. However on the other hand, this can reduce your security against DOS attacks, as the backtrack limit is a performance limit:
// Bump the Perl Compatible Regular Expressions backtrack memory limit
// (PHP 5.3.x default, 1000K, is too low for SpamBlacklist)
ini_set( 'pcre.backtrack_limit', '8M' );
Hardened Wikis[edit]
The SpamBlacklist will not allow editing if the wiki is hardened. Hardening includes limiting open_basedir
so that curl
is not on-path, and setting allow_url_fopen=Off
in php.ini
.
In the hardened case, SpamBlacklist will cause an exception when Guzzle attempts to make a network request. The Guzzle exception message is, GuzzleHttp requires cURL, the allow_url_fopen ini setting, or a custom HTTP handler.
User Filtering[edit]
The SpamBlacklist will filter all users by default, including Sysops, administrators and bots. Also see Phabricator issue T36928.
Safe list[edit]
A corresponding safe list can be maintained by editing the MediaWiki:Spam-whitelist page. This is useful if you would like to override certain entries from another wiki's block list that you are using. Wikimedia wikis, for instance, sometimes use the spam block list for purposes other than combating spam.
It is questionable how effective the Wikimedia spam block lists are at keeping spam off of third-party wikis. Some spam might be targeted only at Wikimedia wikis, or only at third-party wikis, which would make Wikimedia's blacklist of little help to said third-party wikis in those cases. Also, some third-party wikis might prefer that users be allowed to cite sources that are not considered reliable on Wikipedia, or that Wikipedia has considered so ideologically offensive as to warrant blocking. Sometimes what one wiki considers useless spam, another wiki might consider useful.
Users may not always realize that, when a link is rejected as spammy, it does not necessarily mean that the individual wiki they are editing has specifically chosen to ban that URL. Therefore, wiki system administrators may want to edit the Manual:System messages at MediaWiki:Spamprotectiontext and/or MediaWiki:Spamprotectionmatch on your wiki to invite users to make suggestions at MediaWiki talk:Spam-whitelist for pages that should be added by a Manual:Administrators to the safe list. For example, you could put, for MediaWiki:Spamprotectiontext:
- The text you wanted to save was blocked by the spam filter. This is probably caused by a link to a blacklisted external site. {{SITENAME}} maintains [[MediaWiki:Spam-blacklist|its own block list]]; however, most blocking is done by means of [[metawikimedia:Spam-blacklist|Meta-Wiki's block list]], so this block should not necessarily be construed as an indication that {{SITENAME}} made a decision to block this particular text (or URL). If you would like this text (or URL) to be added to [[MediaWiki:Spam-whitelist|the local spam safe list]], so that {{SITENAME}} users will not be blocked from adding it to pages, please make a request at [[MediaWiki talk:Spam-whitelist]]. A [[Project:Sysops|sysop]] will then respond on that page with a decision as to whether it should be listed as safe.
Notes[edit]
- This extension examines only new external links added by wiki editors. To check user agents, add Bad Behaviour or Akismet As the various tools for combating spam on MediaWiki use different methods to spot abuse, the safeguards are best used in combination.
- The Extension:SpamBlacklist/update script is a cron script that can automate updates from shared block lists. If you are using memcached, you will also have to delete the
spam_blacklist_regexes
key (for example, usingmaintenance/mcc.php
). - There're no way to let some users override spam block list. See bugzilla:34928.
Usage[edit]
Block list syntax[edit]
If you would like to create a block list of your own, or modify an existing one, here is the syntax:
Everything on a line after a '#' character is ignored (for comments). All other strings are regex fragments which will only match inside URLs.
- Notes
- Do not add "http://"; this would fail, since the regex will match after "http://" (or "https://") inside URLs.
- Furthermore "www" is unneeded, since the regex will match any subdomains. By giving "www\." explicitly one can match specific subdomains.
- The
(?<=//|\.)
and$
anchors match the beginning and end of the domain name, not the beginning and end of the URL. The regular anchor^
won't be of any use. - Slashes don't need to be escaped by backslashes, this will be done automatically by the script.
- Example
The following line will block all URLs that contain the string "example.com", except where it is immediately preceded or followed by a letter or a number.
\bexample\.com\b
These are blocked:
- http://www.example.com
- http://www.this-example.com
- http://www.google.de/search?q=example.com
These are not blocked:
- http://www.goodexample.com
- http://www.google.de/search?q=example.commodity
Performance[edit]
The extension creates a single regex statement which looks like /https?:\/\/[a-z0-9\-.]*(line 1|line 2|line 3|....)/Si
(where all slashes within the lines are escaped automatically).
It saves this in a small "loader" file to avoid loading all the code on every page view.
Page view performance will not be affected even if you're not using a bytecode cache although using a cache is strongly recommended for any MediaWiki installation.
The regex match itself generally adds an insignificant overhead to page saves (on the order of 100ms in our experience). However, loading the spam file from disk or the database, and constructing the regex, may take a significant amount of time depending on your hardware. If you find that enabling this extension slows down saves excessively, try installing a supported bytecode cache. This extension will cache the constructed regex if such a system is present.
If you're sharing a server and cache with several wikis, you may improve your cache performance by modifying getSharedBlacklists and clearCache in SpamBlacklist_body.php to use $wgSharedUploadDBname
(or a specific DB if you do not have a shared upload DB) rather than $wgDBname
. Be sure to get all references! The regexes from the separate MediaWiki:Spam-blacklist and MediaWiki:Spam-whitelist pages on each wiki will still be applied.
External block list servers (RBL's)[edit]
In its standard form, this extension requires that the block list be constructed manually. While regular expression wildcards are permitted, and a block list originated on one wiki may be re-used by many others, there is still some effort required to add new patterns in response to spam or remove patterns which generate false-positives.
Much of this effort may be reduced by supplementing the spam regex with lists of known domains advertised in spam email. The regex will catch common patterns (like "casino-" or "-viagra") while the external block list server will automatically update with names of specific sites being promoted through spam.
In the filter() function in SpamBlacklist_body.php, approximately halfway between the file start and end, are the lines:
# Do the match
wfDebugLog( 'SpamBlacklist', "Checking text against " . count( $blacklists ) .
" regexes: " . implode( ', ', $blacklists ) . "\n" );
Directly above this section (which does the actual regex test on the extracted links), one could add additional code to check the external RBL servers:
# Do RBL checks
$retVal = false;
$wgAreBelongToUs = ['l1.apews.org.', 'multi.surbl.org.', 'multi.uribl.com.'];
foreach( $addedLinks as $link ) {
$link_url=parse_url($link);
$link_url=$link_url['host'];
if ($link_url) {
foreach( $wgAreBelongToUs as $base ) {
$host = "$link_url.$base";
$ipList = gethostbynamel( $host );
if( $ipList ) {
wfDebug( "RBL match: Hostname $host is {$ipList[0]}, it's spam says $base!\n" );
$ip = wfGetIP();
wfDebugLog( 'SpamBlacklistHit', "$ip caught submitting spam: {$link_url} per RBL {$base}\n" );
$retVal = $link_url . ' (blacklisted by ' . $base .')';
wfProfileOut( $fname );
return $retVal;
}
}
}
}
# if no match found on RBL server, continue normally with regex tests...
This ensures that, if an edit contains URLs from already blocked spam domains, an error is returned to the user indicating which link cannot be saved due to its appearance on an external spam block list. If nothing is found, the remaining regex tests are allowed to run normally, so that any manually-specified 'suspicious pattern' in the URL may be identified and blocked.
Note that the RBL servers list just the base domain names - not the full URL path - so http://example.com/casino-viagra-lottery.html will trigger RBL only if "example.com" itself were blocked by name by the external server. The regex, however, would be able to block on any of the text in the URL and path, from "example" to "lottery" and everything in between. Both approaches carry some risk of false-positives - the regex because of the use of wildcard expressions, and the external RBL as these servers are often created for other purposes - such as control of abusive spam email - and may include domains which are not engaged in forum, wiki, blog or guestbook comment spam per se.
Other spam-fighting tools[edit]
There are various helpful manuals on mediawiki.org on combating spam and other vandalism:
- Anti-spam features - includes link to the built-in $wgSpamRegex anti-spam mechanism.
- Combating spam
- combating vandalism
Other anti-spam, anti-vandalism extensions include:
See also[edit]
- Compatible blocklist (this is just a tiny sampling; there are many more)
- Other resources
- Combating spam and combating vandalism .
![]() | This extension is being used on one or more Wikimedia projects. This probably means that the extension is stable and works well enough to be used by such high-traffic websites. Look for this extension's name in Wikimedia's CommonSettings.php and InitialiseSettings.php configuration files to see where it's installed. A full list of the extensions installed on a particular wiki can be seen on the wiki's Special:Version page. |
- Extensions bundled with MediaWiki 1.21
- Stable extensions
- Page action extensions
- GPL licensed extensions
- Extensions in Wikimedia version control
- EditFilter extensions
- EditFilterMergedContent extensions
- PageSaveComplete extensions
- ParserOutputStashForEdit extensions
- UploadVerifyUpload extensions
- UserCanSendEmail extensions
- All extensions
- Extensions used on Wikimedia
- API extensions
- Edit extensions
- Spam management extensions
- Extensions for data exchange with other wikis