MT-Blacklist is a plugin for Movable Type which provides for content-based blocking and one-step cleanup of comment and trackback spam. The tool utilizes an owner-edited blacklist of strings (or regular expressions) commonly used by spammers to block spam before it is posted to the website and to identify any which gets through the first line of defense for rapid deletion.
On the most basic level, MT-Blacklist is a general content filter. It can be used to enforce an obscenity policy or even for keeping an ex-girlfriend's name from ever appearing in the comments (these two things may in fact be considered synonymous), however when applied to the scourge of comment spam, it becomes a magnificent multi-layer line of defense.
The plugin's Search and De-spam feature also makes MT-Blacklist a powerful comment/trackback manager. It takes only two clicks of a button to wipe a massive spam attack completely off of your site and to bolster your defenses against future attacks.
MT-blacklist is not an IP address blacklist like that which Movable Type provides. IP address blacklists are extremely ineffective due to prevalence of dynamic IP addressing (e.g. from dialup accounts) and for the same reason tend to yield a high number of false positive identifications. In fact, any defense that relies on thwarting a spammer's techniques or methods of access are bound to fail as they will invariably find new avenues of exploitation.
The first line of defense is the spam blocking functionality. When a comment or trackback is submitted, MT-Blacklist is called by your Movable Type installation to check the content. Each of the input fields are compared to each of the strings in your blacklist. If any of the strings are found (regular expressions are evaluated if present), the submission is rejected.
The second line of defense, in case one or more spams are posted using content not yet on your blacklist, is Search & De-spam mode. If you have comment notifications turned on, you will see a link on the bottom of the email. On the resulting page, you can delete the submission, rebuild the entry and related pages and add all or any of the extracted URL strings to your blacklist. After execution, you can also search for other submissions matching your newly updated blacklist, an arbitrary text string or regexp expression, the last submitter's IP address, or even with no filter at all (which just shows you the last N comments/trackbacks). This feature makes cleaning spam out of your blog a quick and painless task.
Many blacklists fail because they focus on moving or mutable targets (e.g. IP addresss, keywords, SMTP relays, etc). MT-Blacklist works because it focuses on one that is far more difficult to change and is in fact, the most important part of the spam: the spamvertised URL.
This is the spammer's weak point. Unlike email spam, the motivation is not to sell you or your visitors anything or even to get you to click on their links. The most solid theory for why comment spamming exists — and what makes the battle against it very different from the previous scourges — is that by placing links all over the web, spammers increase the Google PageRank for the sites they are hawking.
In order for this scheme to work, the links must be published on as many sites as possible for as long of a time as possible, but at least long enough for the Googlebot to see them. This is where MT-Blacklist's Search and De-spam mode comes into play. MT-Blacklist makes it terribly easy to recover very quickly from even the most intense spam attack with almost no effort, rewarding the spammer with no extra Google juice. Furthermore, for efficiency reasons, spammers tend to put twenty to thirty URLs in each comment but it takes only one known URL to block it.
By using MT-Blacklist, you effectively shift the burden, in terms of both time and cost, onto the spammers which is the only way to win the war.
Although we certainly have a few versions of the plugin to go before we get to the distributed, peer-to-peer Nirvana that we're headed for, there is a new tool in the battle against comment spam which brings us a step closer:
While most of this is explained on the weblog (see entry entitled "MT-Blacklist/Comment Spam Clearinghouse"), I will say a centralized database, for all of its long-term flaws, is an excellent short-term measure for spreading the web of protection across all of our websites. What's more, for the first time, the protection of a common blacklist can be easily extended through RSS and other technologies to others in the weblog community who are not currently running Movable Type.
And that is a win-win situation for everyone.
As much as my altruistic side would love to continue to develop MT-Blacklist and maintain the Comment Spam Database for free, it puts a tremendous strain on me in terms of time and energy. Every minute I spend responding to technical support requests, poring over Clearinghouse spam submissions and upgrading the program itself is a minute stolen from my freelance work projects. Because I currently have a large level of personal debt, I do not have the luxury to spend those precious minutes on things which do not somehow contribute to its reduction.
Under the current licensing terms, MT-Blacklist is free. However, if you would like to see the program developed for future versions of Movable Type or with additional capabilities, you may want to consider donating.
Think of it this way: How much time and energy has MT-Blacklist saved you? You decide how much that is worth.
If your web hosting provider has the correct tools installed (in particular, the Storable perl module), there is nothing more required than the basic Movable Type installation. If not, then you can use filesystem-based persistence (as opposed to database persistence) of data via the YAML perl module which does not require help from your hosting provider to install.
These two options are detailed in the installation instructions.
If you find an incompatibility, please let me know (mt-blacklist at jayallen dot org) and I will list it here.
Note to upgraders: If you are upgrading MT-Blacklist from a version older than 1.63, you must follow the directions below, but make sure to read the Upgrade notes after you are finished upgrading. If you are upgrading from MT-Blacklist v1.63, you may use the upgrade installation (tar/gzip or .zip) instead.
Depending on your server environment, there are two different installtion paths for MT-Blacklist: the ridiculously easy and the less easy. Both paths set out in the same direction but the latter requires some further steps to finish the installation. If you know that your host does not have Storable perl module installed, you can guess which path you will be taking. It's okay. None of this is terribly difficult.
First, download, unpack the archive and upload to your webserver in ASCII mode. Put the files in the following locations relative to your main Movable Type directory (abbreviated MT_DIR below) where mt.cgi resides. You may have to create directories if they do not already exist.
MT_DIR/mt-blacklist.cgi <--- Should be in the same directory as mt.cgi MT_DIR/plugins/Blacklist.pl MT_DIR/extlib/jayallen/Blacklist.pm MT_DIR/extlib/jayallen/MTBlPing.pm MT_DIR/extlib/jayallen/MTBlPost.pm
Set the permissions for all files to 755 (-rwxr-xr-x). If you are using FTP software to the files to your server, you should be able to do this through that software. Look for an option called "Permissions" or "Chmod" related to file information.
Once you do that, go to http://YOURDOMAIN/MT_PATH/mt-blacklist.cgi (replacing the words in italics with the actual values. Hint: It should look very similar to the URL you use to go to MT's main menu) and configure the plugin. All of the options are documented on the configuration page. The plugin is NOT ACTIVE until you activate it by following this step.
This is where the installation paths diverge.
Note to upgraders: You can move on to the Upgrade notes now...
If you were able to successfully load the main configuration screen, you can breathe a sign of relief, for you are nearly finished. All you need to do now is grab the latest blacklist and immunize your Movable Type installation. Click on the List tab of the navbar and follow the directions.
That is it. There is nothing else you need to do. There are no template modifications or extra tags to insert. There are no template modules to make and there isn't need to shake chicken bones and sacrifice small animals. At this point you are equipped with a default blacklist of over 500 entries, which you can tailor to your liking.
It really is that simple.
Make sure to also read the Usage notes which will give you further information and preemptively answer some common questions other users have had.
If you encountered a nasty error on the first load of the plugin's web interface, please see the Troubleshooting section first to see if it can be easily fixed. You may be led back to this section, which means that you are in for just a little bit more work.
Alright... Back here again, eh? Wait, let me guess... You either don't have the Storable perl module installed or you're using PostgresDB as your MT database? Yep, there is a bug (a few actually) in PluginData.pm which creates an incompatibility between PostgresDB and Storable.
Lucky for you, MT-Blacklist supports an alternative method of data persistence via YAML. Using YAML, your blacklist configuration and data will be stored on the filesystem instead of in the database.
First, create a directory in MT_DIR/extlib/jayallen called Blacklist_Config. The new directory should be in the same directory as the Blacklist.pm file. Set the permissions on the new directory to 777 (-rwxrwxrwx).
Side note: You could actually create the directory in many different places, but this is a safe and out of the way place where Bad Things™ won't happen. Unless you know what you're doing and have a reason to put it elsewhere, stick to the recommendation.
Open up Blacklist.pm in a text editor (i.e. not Microsoft word...use Notepad if you have nothing better) and near the top of the file, you will see a block of text like this:
################################################## # # PATH TO FILESYSTEM-BASED BLACKLIST CONFIGURATION # # The following configuration option should only # be uncommented if you want to use a filesystem- # based configuration and blacklist instead of one # stored in your database. # # [OMITTED FOR BREVITY] # # $blacklist_config_directory = '/path/to/extlib/jayallen/Blacklist_Config/'; # ##################################################
Replace the string /path/to/extlib/jayallen/Blacklist_Config/ with the real path to your Blacklist_Config directory and then remove the # sign in front of the line. Make sure that the path is enclosed in single quotes with no spaces at the ends! Just like you found it...
Next, we need YAML (rhymes with camel, in case you were wondering...) YAML allows us to store data very smartly in files on the file system.
Note to advanced users: This is not the entire YAML archive. Just the files we need to make everything work...
You should put the entire contents of the tar archive (Yaml.pm and the YAML/ directory) into your MT_DIR/extlib/ directory.
Now, If you have followed my directions precisely and use the suggested names, the file hierarchy you have installed should look like this.
MT_DIR/mt-blacklist.cgi <--- Should be in the same directory as mt.cgi MT_DIR/plugins/Blacklist.pl MT_DIR/extlib/jayallen/Blacklist.pm MT_DIR/extlib/jayallen/MTBlPing.pm MT_DIR/extlib/jayallen/MTBlPost.pm MT_DIR/extlib/jayallen/Blacklist_Config/ MT_DIR/extlib/Yaml.pm MT_DIR/extlib/YAML/Error.pm MT_DIR/extlib/YAML/Family.pm MT_DIR/extlib/YAML/Node.pm MT_DIR/extlib/YAML/Transfer.pm
Whew! Still with me? Not as easy as the easy path, but that is what happens when we have to work around bugs in pre-installed software.
FINALLY, run mt-blacklist.cgi from your web browser. If you get the configuration screen, stand up, pat yourself on the back and rejoice! After that, go finish up with the Ridiculously Easy Path installation where we diverged...
But before you go, know this: You never have to take the hard road again...
If you were able to run MT-Blacklist previously (even with a few bugs), upgrading is a cinch. Simply follow the installation directions and you are finished! All of your configuration settings are saved from before the upgrade.
The plugin will automatically handle any changes in the configuration, if necessary.
For those who are upgrading from a previous version (i.e. successfully ran MT-Blacklist), there are only two major changes of note:
The addition of text string or regular expression search allows you to search comments or trackbacks on the basis of arbitrary terms. This may be useful outside of the realm of comment spam, for instance, if you are trying to find all comments by a certain user. Also, this feature allows you to test out regular expressions before adding them to your blacklist to check for false positives in your current comments/trackbacks.
The NO FILTER search option simply displays the last N comments/trackbacks (where N is the number specified at the top of the search page) which is useful for seeing your latest user submissions to your site.
The plugin will block incoming comments/trackbacks with content matched by any one of the entries in the blacklist. All comment and trackback input fields are scanned for matches. If you have selected the option, an entry will be submitted to the Activity Log listing the type of submission and the first blacklist entry that matched the content.
If you have comment and trackback notifications on, you will notice a new link at the bottom of your email. This link will bring you to a screen where you can, in one step, extract URLs from the entry, add them to the blacklist, delete the comment/trackback and rebuild the pages related to the commented on or pinged entry. This is equivalent to saving the entry, and not to rebuilding the blog.
You will then be given the opportunity to Search & De-spam the rest of your entries in all blogs. This feature is also available from the De-spam button on the toolbar.
MT-Blacklist uses Movable Type's native user permissions to determine which actions a user can take. This is essential in a multi-user environment with differeing levels of trust. The permissions, detailed below, are designed to reflect analogous actions using MT.
MT-Blacklist works perfectly in protecting you from spam that you have seen before, but, like virus scanning software, it must be regularly updated in order to protect you from future attacks with new spam domains.
Read Staying up to date for a few simple ways to keep your blacklist up to date.
Each entry you add to your blacklist makes commenting and trackbacks slightly but incrementally slower. See Avoiding blacklist bloat for information on keeping your blacklist healthy and speedy.
The Comment spam clearinghouse is a companion website to MT-Blacklist created for discussing the comment spam scourge, new techniques for combatting it as well information about the plugin itself.
In addition to that information, the Clearinghouse contains the master blacklist used in installation of the plugin. The master blacklist is collaboratively-compiled from spam reports by blog owners. As mentioned earlier, only through a community effort can we stay a step ahead of the spammers. By submitting all comment spam you receive to the Clearinghouse for inclusion into the master blacklist, you make the entire community stronger.
More information can be found from the links below:
IMPORTANT NOTE: Before submitting spam reports to the Comment Spam Clearinghouse, make sure to read the rules and check the master blacklist (use your browser's Find feature) to make sure that you are not submitting a duplicate.
On the de-spam page, you are presented with a checkbox at the bottom of the page, labelled "Rebuild the relevant entries after comment deletion". It is common, upon first install of MT-Blacklist, to find hundreds if not thousands of spam comments. Unfortunately, despamming and rebuilding a large number of entries may overload your server.
If you have a large number of comments to despam, you may first want to uncheck the box and submit the form which will delete the comments without rebuilding each entry. After you have finished despamming your blog(s), then rebuild your entire blog through the MT menu.
Check back here periodically as I will be adding some really useful information soon. I didn't want to hold up release of the new version of the plugin and the information is not essential for simply running the plugin. If you are curious about regular expressions see this solid tutorial.
There are no known bugs.
If you truly think you have found a bug (in most cases, you have not but instead did something wrong in installation/upgrade), please send a report to mt-blacklist-bugs at jayallen dot org with as much of the following information as you can provide:
Please send feature requests to mt-blacklist-request at jayallen dot org.
List viewI am receiving this error message:
An error occurred: Can't locate jayallen/Blacklist.pm in @INC (@INC contains: ./extlib ./lib /usr/lib/perl5/5.6.1/i686-linux /usr/lib/perl5/5.6.1 /usr/lib/perl5/site_perl/5.6.1/i686-linux /usr/lib/perl5/site_perl/5.6.1 /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at mt-blacklist.cgi line 29.
That probably means that Blacklist.pm was not installed in the correct location or with the correct permissions. Please read the Installation section again carefully making note of the folder hierarchy for the files.
I am receiving this error message:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
More information about this error may be available in the server error log.
If you are using the YAML method, perhaps you used a text editor which inserted unwanted carriage returns or control characters (e.g. Microsoft Word) in the Blacklist.pm file. Notepad or another basic text editor should work fine. Otherwise, this most likely means that you did not upload the file in ASCII mode or set the permissions correctly. Please read the Installation section again carefully.
I've done everything you said, but I get this error:
An error occurred: Can't locate Storable.pm in @INC (@INC contains: ./extlib ./lib /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at lib/MT/PluginData.pm line 9. BEGIN failebdcompilation aborted at lib/MT/PluginData.pm line 9. Compilation failed in require at extlib/jayallen/Blacklist.pm line 18. BEGIN failebdcompilation aborted at extlib/jayallen/Blacklist.pm line 18. Compilation failed in require at mt-blacklist.cgi line 29.
That means that you do not have the Storable perl module installed. Continue your installation following the steps outlined in the section entitled The Less Easy Path.
I've done everything you said, but I get this error:
An error occurred: Can't locate object method "load" via package "MT::PluginData" (perhaps you forgot to load "MT::PluginData"?) at extlib/jayallen/Blacklist.pm line 2878.
That means that you do not have the Storable perl module installed. Continue your installation following the steps outlined in the section entitled The Less Easy Path.
I installed MT-Blacklist and everything works as advertised, but now the comment notification emails don't contain the extra information that I added via hacking in MT/App/Comments.pm. What happened?
MT Blacklist, among other things, overrides MTs internal comment post and trackback ping routines. What that means is that if you hacked the MT source code for those two subroutines in (MT::App::Comments.pm and MT::App::Trackback.pm), you won't see your hacks because MT-Blacklist's subroutines are called in their place. If you like, you can apply your hacks into the MT-Blacklist source code (MTBlPost.pm and MTBlPing.pm repsectively) if you like. They are essentially exactly the same except for some small bits of code I injected to activate the blacklist filtering. See the Requirements section for more incompatibilities and ways to restore some hacks.
I installed MT-Blacklist but it doesn't stop comments/trackbacks containing domains on my blacklist and comment notifications don't have the de-spam link!
This problem often occurs when:
Before you do anything, please delete all MT-Blacklist files (there are five) and reinstall carefully with the correct version from scratch.
There is a spammer on my site right now! I put his IP address onto my blacklist, but he's still spamming my comments! What do I do!??
The best thing you can do is to re-read the instructions for MT-Blacklist. This plugin is not an IP banning plugin. Movable Type already has that and it would be an immense waste of my time to duplicate functionality. MT-Blacklist is a content-based filter. The blacklist entries are compared to the name, email, url and body fields of all comments and the source url, title and excerpt fields of all trackback pings. If you want to block a spammer, you enter into the blacklist the URLs he is trying to spamvertise.
The biggest thank you goes out to Patrick Nielsen-Hayden of Electrolite for his huge help in tracking down the mysterious and major de-spam bug found in version 1.5.
The MT-Blacklist Alpha Testers did an outstanding job finding holes that I never saw. Thank you Stepan Riha, Al-Muhajabah, Kevin Aylward, girlie, Yoz Grahame, Etan Kerner and Matt Warden
I would also like to especially thank Kevin Shay and Ben Trott without whose early help I could have never completed this plugin.
If you think you have found a bug, please see the Known Bugs section to make sure that it is new and for submission guidelines.
If you have a feature request, send it to mt-blacklist-request at jayallen dot org.
If you are having trouble installing, upgrading or running the plugin, please try posting your problem to this thread first (there are some very saavy MT-Blacklist user's who can probably help you). If however you can't solve your problem that way, send an email to mt-blacklist-support at jayallen dot org and I will try to get to it as soon as I can. (I apologize. It's temporary)
In either case, please provide as much of the following information as you can:
For all other comments and inquiries, you can reach me at mt-blacklist at jayallen dot org.