“The WP-SpamFree plugin virtually eliminates automated comment spam from bots, including trackback and pingback spam,” its author, Scott Allen, claims. “It takes a different approach than most and stops spam at the door.”

Indeed, everyone who’s tried it reports that their spam has dropped off to virtually zero and that they haven’t heard from anybody who had problems leaving comments. Sounds like the Holy Grail of spam prevention, right?

Not so fast.

WP-SpamFree, it turns out, uses JavaScript and cookies to verify that someone is using an actual web browser to access your site and leave a comment. These approaches are not that different from what other plugins have done in the past. What distinguishes WP-SpamFree in this respect is that it requires both JavaScript and cookies in order for someone to post a comment. This will certainly keep out virtually every spambot out there.

Unfortunately, it will also block most mobile web browsers and some disabled users. In both cases the browsers being used aren’t capable of JavaScript, cookies, or both. If your blog targets mobile web users or people with disabilities, WP-SpamFree might not be for you.

Then there is WP-SpamFree’s method of blocking trackback and pingback spam. These are always automated, so using JavaScript and cookies is impossible. WP-SpamFree, it turns out, uses several extensive internal lists of IP addresses, URL fragments, and keywords to block this type of spam.

This works fairly well; however, the way it’s implemented in the current version of WP-SpamFree (1.9.6.2) is quite strange. It appears the author didn’t want to use arrays and loops to iterate through his lists and instead unrolled all his loops, resulting in a huge plugin clocking in at over 3,700 lines. There’s no obvious good reason for this; it would seem in PHP that the plugin would be much slower than it would otherwise. The gain of not having the loops doesn’t seem nearly as much as the overhead of compiling thousands of extra lines of bytecode. In addition there are several other examples of duplicate code which could have been split into functions.

These technical implementation issues make me wonder at how much experience the programmer has. If they were intentionally done by an experienced programmer, I would have expected them to be mentioned in the README or release notes or a blog entry, but especially in the code comments.

Despite those issues, the plugin works pretty well for what it does. I hope that the author addresses those implementation issues for his next major version, though, to make the plugin even better.

And there are things that WP-SpamFree does not do. It does not block email harvesters, for instance. It also does not block spambots when they scrape your site looking for your comment forms, nor block denial of service attacks. Indeed, under a heavy spam attack, its size and CPU usage could cause limited web hosting resources to be exhausted.

That’s all just a long way of saying that WP-SpamFree has its pros and cons, and if you choose to use WP-SpamFree, you still should keep Bad Behavior around as part of your overall spam prevention strategy.

Running behind the scenes of Matt Mullenweg‘s new commercial WordPress project, WordPress.com, is of course WordPress, everyone’s favorite blogging platform. And running on WordPress.com is Bad Behavior, the premier solution for blog spam.

Mullenweg attended the Blog Business Summit in San Francisco this week to promote WordPress.com as a corporate blogging platform competing with Six Apart’s TypePad service. WordPress.com is currently in an invitation-only phase as it ramps up to full service.

At the summit Mullenweg said one of the main features of WordPress.com, which will cause people to choose it over TypePad, is better spam protection. WordPress.com uses Bad Behavior as its primary line of defense against blog spam.

Bad Behavior works by analyzing the entirety of incoming HTTP requests to ensure that they match profiles of legitimate browsers, and don’t match profiles of known spammers. In addition to WordPress, Bad Behavior also runs on MediaWiki, Geeklog, and Drupal, and can be integrated easily into any PHP-based software. Bad Behavior is the only software known to exist which uses this approach, and in practice eliminates virtually all incoming spam while remaining very fast.

I’m proud to have Bad Behavior running on WordPress.com, and I’m proud to support WordPress.com, and all WordPress bloggers, in the fight against blog spam. Like WordPress, Bad Behavior is released under the GNU General Public License, and development of Bad Behavior is funded through user contributions. If you would like to contribute to further development of Bad Behavior, click here.

Update August 19: Bad Behavior is now available for Drupal.

Bad Behavior 1.2 has been released. Bad Behavior stops link spam at the front door by denying spammers the ability to access your PHP-based web site at all.

Thanks to all of you who tested the release candidates, and actually found fewer bugs than I was expecting. Either I’m getting better at this, or you guys aren’t actually installing the software.

New in this release:

• Bad Behavior now has whitelisting capability. Edit the file bad-behavior-whitelist.php to add any IP address ranges or user agents you need to whitelist for your particular site. (Note that search engine bots should not be whitelisted by user agent, but by IP address range, because spammers pretend to be search engine bots. Bad Behavior already passes all major search engine bots which behave properly.)
• The specific reason for blocking is now logged in the database. This will help in determining whether new robots should be blocked by Bad Behavior or not.
• Several additional spammers have been identified and blocked in this release.
• When logging is turned on, Bad Behavior will identify spammers it has recently seen, even if their profile changes, and continue to block them.

Download Bad Behavior now!

Bad Behavior 1.2 Release Candidate 3 has been posted. Bad Behavior stops link spam at the front door by denying spammers the ability to access your PHP-based web site at all.

As I close in on a final 1.2 release, the reports I have gotten have been quite encouraging. Most testers have reported a complete elimination of link spam to their sites. So I’ve cleaned up a bit, fixed one problem, and this will probably be the final 1.2 release, or very close to it.

New in this release:

• A bug in Opera causes it to be incorrectly identified as a spambot by Bad Behavior. The check has been temporarily disabled until I can confirm that Opera has fixed its browser.
• Bad Behavior no longer sends any data to the Bad Behavior Blackhole. This proved entirely unnecessary, as Bad Behavior appears perfectly capable of catching spambots on its own without outside help.

New since version 1.1.4:

• Bad Behavior now has whitelisting capability. Edit the file bad-behavior-whitelist.php to add any IP address ranges or user agents you need to whitelist for your particular site. (Note that search engine bots should not be whitelisted by user agent, but by IP address range, because spammers pretend to be search engine bots. Bad Behavior already passes all major search engine bots which behave properly.)
• The specific reason for blocking is now logged in the database. This will help in determining whether new robots should be blocked by Bad Behavior or not.
• Several additional spammers have been identified and blocked in this release.
• When logging is turned on, Bad Behavior will identify spammers it has recently seen, even if their profile changes, and continue to block them.

Update 16 August: See Bad Behavior 1.2 release.

The second release candidate of version 1.2 of Bad Behavior is now available! Bad Behavior stops link spam at the front door by denying spammers the ability to access your PHP-based web site at all.

Surprisingly, no one reported any bugs in the first release candidate, but a very few spammers are still making it through. So I’ve made an update which attempts to address this and get that last 0.1% of the spam.

New from version 1.2 Release Candidate 1: When logging is turned on, Bad Behavior will identify spammers it has recently seen, even if their profile changes, and continue to block them. I believe this simple change should eliminate virtually all spam, even at the highest-traffic sites, while remaining fast.

Again, I still need reports of any spammers which escape Bad Behavior’s notice. Please contact me and include output from phpMyAdmin showing the relevant records for the spammer. Verbose logging has been turned on for this build so that the necessary records will be available if this happens.

Update August 11: Please see the newer version Bad Behavior 1.2 Release Candidate 3.

The first release candidate for Bad Behavior 1.2 is now available. Bad Behavior, the bane of link spammers everywhere, has been strong and stable. I’ve added some new features and need your feedback.

• Bad Behavior now has whitelisting capability. Edit the file bad-behavior-whitelist.php to add any IP address ranges or user agents you need to whitelist for your particular site. (Note that search engine bots should not be whitelisted by user agent, but by IP address range, because spammers pretend to be search engine bots. Bad Behavior already passes all major search engine bots which behave properly.)
• The specific reason for blocking is now logged in the database. This will help in determining whether new robots should be blocked by Bad Behavior or not.
• Several additional spammers have been identified and blocked in this release.
• Bad Behavior now sends copies of spam received automatically for use in Bad Behavior Blackhole. If you don’t want copies of your spam sent in, edit bad-behavior-blackhole.php.

The spammers are starting to use some new techniques and variations on old techniques, and so every so often I must update in order to keep them at bay. Ye olde arms race. This one, however, is about to get much more escalated; as the Bad Behavior Blackhole matures and becomes ready, you should see link spam start to disappear all over the Internet, and especially on your own site.

This release works with MediaWiki, WordPress and in generic mode with any PHP script. It has not yet been updated for Geeklog.

Please test this new release of Bad Behavior and leave your feedback here, or for private feedback or spambot reporting, contact me instead. To report a spammer or inappropriate blocking, please send the records in the bad_behavior_log table in the database relating to the issue, which you can retrieve using phpMyAdmin. Update: I especially need reports of any spammer which manages to get past Bad Behavior at this point. Please send in any that you see. Thanks!

Oh, and don’t contact me about the YahooSeeker bot. It is misbehaving. And you likely don’t need it anyway; it’s only used for Yahoo! Shopping. If you do need to be listed on Yahoo! Shopping, you can whitelist it. The main Yahoo bot is called Slurp, and this is the one that puts you in Yahoo! It passes just fine.

Update August 11: Please see the newer version Bad Behavior 1.2 Release Candidate 3.

Bad Behavior 1.1.4 has been released.

This release fixes a problem with the W3C Validator being blocked inadvertently. The downside is you’re going to get a few more spammers who were also being blocked. A more permanent solution is in the works, but this should get you XHTML freaks who revalidate your pages daily back in business.

If you don’t care at all about the W3C Validator then feel free to use 1.1.3 as it may block a few more spammers than 1.1.4. This will be resolved in a future release.

First, say “I Hate Perl” three times, and then Download Bad Behavior now!

Bad Behavior 1.1.3 has now been released. I’ve been holding this a little longer than some of you would like; because of a rise in spam attacks and some slightly smarter spammers, a few spams have been getting through to your sites, and because they’re getting smarter I’ve wanted to ensure that I could block the spammers and only the spammers. I’ve blocked all the spammers I can reasonably block and maintain no false positives.

I also fixed a (very uncommon) update service being recognized as a spambot. I have a policy of zero false positives, so if you see traffic that should be getting through and is being blocked, or if you are getting spam to your site, please report it immediately.

Changed in this release:

• Several additional spambots have been identified and blocked thanks to user contributions.
• Mozilla Blog Updates is no longer blocked.
• A typo causing a PHP warning in bad-behavior-http-headers.php has been fixed.

It’s that time again, so Download Bad Behavior now!

In case you somehow don’t know what I’m talking about, let me fill you in. Bad Behavior is PHP-based software which blocks automated link spam. And link spam is the growing problem of spammers taking advantage of blogs, wikis, forums, guestbooks, CMS, and similar software to post spam. Link spam has been a serious problem for a couple of years, and many people have tackled it with varying degrees of success.

On the evening of 31 December 2004, I suffered what every blogger experiences eventually: my first comment spam attack. A spammer using automated software and open proxy servers sent 764 spam comments to my site, only half of which were caught by WordPress. The other half were scattered all over my site. After deleting all of the junk, I responded by writing some code, and thus was born the WordPress SpamAssassin plugin, which filters blog comments through SpamAssassin. It actually proved to be useful at stopping a lot of spam, but wasn’t able to catch all of it. Throughout the life of wp-spamassassin, the main thing I learned is that email spam and blog spam are two quite different creatures. I finally wound up having WordPress moderate all first-time commenters, and gave up further development of wp-spamassassin around mid-March, recognizing it as not quite appropriate to the task.

At that point I began using the Spam Karma 2 WordPress plugin. It proved quite effective at keeping spam off my site, but it has two serious drawbacks: first, the spam is still there in your database and if you get a lot of spam you have to spend a lot of time managing it, and second, it invariably would catch legitimate comments and mark them as spam, making the spam management problem far worse than it would otherwise be.

I want to spend my time blogging. I don’t want to spend my time scouring through 1,000 or more spams a day for the three comments that were thrown in by mistake. But there simply was no other solution I could live with. They either required a captcha, which isn’t accessible to some people with disabilities, or required JavaScript, which many people turn off and also isn’t accessible to some people with disabilities, or had too high a false positive rate, or too high a false negative rate, or…

At some point I realized there was an approach it seemed no one had tried yet. I did several Google searches looking for any evidence that anyone had tried this approach with any software on any blog, forum, wiki or CMS, and came up empty. I began coding, and about a week later I put out the first release candidate of Bad Behavior.

The premise behind Bad Behavior is not to analyze the comment, but to analyze the visitor. The idea I had had was that if the spam is automated, the spambot software must be distinguishable from actual people reading your site. But spambots typically fake the User-Agent and Referer fields in the HTTP request. What else is there to work with? As it turns out, there are quite a few other fields in the HTTP header that can be analyzed, if only you know what they are and how to get at them. And it turns out that spambots do have a fingerprint that allows them to be distinguished from the Web browsers they pretend to be.

I designed Bad Behavior to be fast and portable to PHP-based software other than WordPress and to err on the side of caution, allowing a user through if there is doubt as to whether it is a spambot, so as to minimize false positives. Accordingly, Bad Behavior became wildly successful in a very short time, even beyond my initial expectations. So many people have downloaded it and used it that I can’t even count them all in any reasonable manner. People are even writing plugins for it and porting it to other platforms. It now runs on WordPress, MediaWiki and Geeklog, and I’ve received reports of people using it on Drupal, ExpressionEngine and custom PHP-based sites in its generic mode.

It hasn’t been all sweetness and light, though. I’ve had days where I had to release twice to fix some stupid error I should have caught the first time round. I’ve had Microsoft do things which caused their search engine bot to get blocked. I’ve seen a sharp increase in spam directed here, both of the blog variety and the email variety.

But I also get to see new link spammer techniques as they develop, because they seem to want to test them here. This gives me a window of opportunity in the event that something new needs to be added to Bad Behavior, or something needs to be changed. Surprisingly, spambot software is not getting much better overall. While the spammers are beginning to adapt to Bad Behavior, they still have serious weaknesses in their delivery methods that I am able to take advantage of to keep them blocked out. For now, I’m far ahead of the spammers. They have a lot of catching up to do, but updating spambot software takes time and costs money, and most spammers won’t bother, since (unfortunately) there are still far too many sites out there without adequate protection, such as Bad Behavior.

The sophisticated link spammer technique in common use now is to use some sort of script to harvest comment forms from a group of sites, then to fill in the fields appropriately, and a few hours or days later, to use a network of open proxy servers to relay the spam comments to thousands — or hundreds of thousands — of sites which use the same type of software. Repeatedly.

As spambot software continues to improve, I am seeing more instances of spambot software which closely matches the fingerprint of legitimate user agents such as Internet Explorer or Firefox. Bad Behavior must continue to improve to analyze the delivery method these spammers use, and the next step is to analyze the open proxy server. Accordingly the Bad Behavior Blackhole is a spin-off project which intends to do just that. Like Bad Behavior, it is designed to cause minimal or no inconvenience to actual humans by providing a fully automated, immediate removal process — but only for humans. (At the time of this writing, manual removal is implemented, and automated removal is in testing.) A lot of work has been done on the open proxy server problem already, and Bad Behavior Blackhole will build on this. When it is ready, Bad Behavior Blackhole will be integrated into Bad Behavior.

There are other ideas on the drawing board as well, but I don’t want to give the spammers (who actually do read my blog, but can’t seem to leave a comment) too much of a clue where I am going or how I will shut them down next. Like them, you will have to stay tuned.

I hope you found this little essay interesting, and if you haven’t installed Bad Behavior yet, what are you waiting for?

Bad Behavior 1.1.2, the latest version of the Web’s only portable link spam killer, has been released.

Fixed in this release:

• Due to recent changes made by Microsoft, MSNBot was being blocked about 70% of the time. This has been fixed.

Changed in this release:

• A very surprisingly large number of people wanted to have individual bad_behavior_log tables for each installation of WordPress, MediaWiki, Geeklog, etc., rather than a combined table. This is now supported and Bad Behavior will create a table using the table prefix provided by each individual software. This means, for instance, instead of a bad_behavior_log table, you will have a wp_bad_behavior_log table on WordPress, or a mw1_bad_behavior_log table on MediaWiki. The table prefix, of course, will vary depending on the settings of the software on which Bad Behavior is installed. The old combined bad_behavior_log table will be left in place; you will need to rename or remove it yourself if you desire.

I’m also moving closer to having the Geeklog port stabilized and included in the mainline Bad Behavior release. Currently it is built and distributed separately.

Thanks again to everyone who has written me, and written on their own sites, about their successes with Bad Behavior. It’s that time again, so Download Bad Behavior now!