Archive for the 'Spam' Category
Bad Behavior versions 2.0.37 and 2.1.3 have been released. For the 2.0 stable branch, this release is a maintenance release recommended for all users.
Please note: The 2.0 series of Bad Behavior is receiving limited updates, including unblocks, bug fixes and security fixes only. Future development is taking place in the 2.1 development tree.
Who should upgrade?
Users deploying Bad Behavior on Microsoft IIS should upgrade to ensure that all Bad Behavior functionality works as intended.
Users who receive a significant amount of traffic from proxied connections (e.g. small business and enterprise users) should upgrade to prevent a tiny minority of these users from being blocked.
Users following the development branch should upgrade to take advantage of support for the CloudFlare reverse proxy service.
What’s new?
New in the 2.0.37 stable release (since 2.0.36):
- In rare configurations, the Firefox and Safari web browsers may send the nonexistent “Proxy-Connection” HTTP header. Old versions of Internet Explorer may also send this header in their default configurations. This usually occurs when the web browser is configured to connect to an (obsolete) HTTP/1.0 proxy or has been explicitly configured to use HTTP/1.0 when talking to a proxy, even if the proxy understands HTTP/1.1. This header originated with a proposal made by (then) Netscape which was rejected for inclusion in HTTP in 1998 due to its causing interoperability problems. Bad Behavior checks for this header as it has historically made an excellent indicator of malicious activity if it is seen at the origin server, because proxy servers are expected to strip the header. Because of the slight possibility of blocking legitimate users, this check is now active only in strict mode. (Thanks to Mark Nottingham for reporting this issue.)
- A workaround for a problem with PHP on IIS servers has been implemented. This issue caused various parts of Bad Behavior’s functionality to fail on IIS. (Thanks to Michael Kingery for reporting this issue.)
New in the 2.1.3 development release (since 2.1.2):
- The changes listed above for 2.0.37 have also been implemented.
- New code which implements “round-trip DNS” for verifying that an IP address belongs to a specific entity is now being used to verify Googlebot and MSNbot. This code replaces the old hard-coded IP addresses.
- Support for the CloudFlare reverse proxy service has been added. Users of this service should now be able to use Bad Behavior successfully. (Thanks to Matthew Prince at Project Honey Pot for his assistance with this implementation.)
Download
Download Bad Behavior now!
The 2.1 development releases will not be offered through the WordPress automatic upgrade facility. Only stable releases will be offered through automatic upgrade.
Support
You’ve probably noticed that there hasn’t been a release of Bad Behavior in several months. This is due entirely to the fact that I can only spend time on it when incoming donations cover the cost of my time. Otherwise I have to engage in paying work to keep food on my table.
I happen to like giving spammers a hard time, and it’s frustrating that I don’t get to spend enough time on it. You can help me make Bad Behavior even better by setting up a recurring contribution, or making your most generous one-time contribution for any amount.
Thank you again for supporting Bad Behavior development!
Recently it was suggested to me that Bad Behavior could incorporate support for Stop Forum Spam.
Stop Forum Spam is meant to be a list of IP addresses, emails and usernames which spammers use when registering or posting spam to forums. It seems to work well, but it has some shortcomings.
First among them is it has no native support for DNSBL. Instead, it exports its data to a third party DNSBL where the data is commingled with other data from unknown sources, making it difficult to use effectively.
Second is that it has no clearly defined removal policy. It does provide a form where people can request manual removal, but it also implies that a “network administrator” has to request removal.
After much experimentation with blackhole lists over the years, Bad Behavior currently uses only the Project Honey Pot http:BL list (and it is disabled by default). This list works very well at catching actual spammers, and it provides instant automatic removal for the very few legitimate users who happen to get caught by it.
Bad Behavior is meant to provide as little inconvenience to legitimate users as possible. When it happens, the user must be given clear directions on how to resolve the problem and ideally must be able to restore their access as soon as possible, e.g., by removing the viruses from their computer, etc.
Because it lacks a removal policy and clear process, it will not be appropriate to incorporate Stop Forum Spam at this time. I will continue to monitor the service and if it changes to allow for easier removal by legitimate users, then it may be incorporated in the future.
Bad Behavior 2.0.29 has been released. It is a maintenance release and is recommended for all users.
MediaWiki and WordPress users who have not updated in the last year or so should take note of special upgrade instructions below.
Who should upgrade?
All users should upgrade to resolve issues with certain specialized web crawlers being blocked. Users who wish to use OpenID in conjunction with Bad Behavior should also upgrade to resolve authentication issues.
What’s new?
New in this release (since 2.0.28):
- Users authenticating to a Bad Behavior-protected site using a third party OpenID were blocked with a message stating that: “Data may not be posted from offsite forms.” In most circumstances, your site does not want to receive a POST which originated from another site; however, OpenID requires this. A new option, offsite_forms, has been added to Bad Behavior to permit data to be posted to your site from other sites. Enabling this option will allow OpenID to work but may expose your site to spam which was previously blocked. WordPress users will find the option on Bad Behavior’s options page; other platforms should check their platform-specific documentation for how to set options.
- A few specialized web crawlers use an unusual form of the Range: HTTP header in their requests, requesting a range starting with 0. This behavior, while technically permitted by the HTTP specification, is most often seen with malicious crawlers; web browsers and major search engines do not use it. Bad Behavior will now block these requests only when strict mode is enabled.
Support
Thank you to everyone who has chosen to make a financial contribution toward further development of Bad Behavior. Your contributions ensure that I can prioritize Bad Behavior development and make more frequent and timely releases, like this one.
Download
Download Bad Behavior now!
Special Upgrade Instructions
Users of MediaWiki and WordPress upgrading from version 2.0.20 or earlier should follow these special directions (from 2.0.21 or later, upgrade normally):
For MediaWiki: Before installing this version of Bad Behavior, manually remove (e.g. using FTP or ssh) any old versions you may have, including the lines added to LocalSettings.php. Then install the new version fresh, following the installation instructions for MediaWiki.
For WordPress: If updating to this version through the automatic updater fails, manually remove (e.g. using FTP or ssh) any old versions you may have installed. Then upload and install the new version fresh, following the installation instructions for WordPress. After doing so, future automatic updates should proceed normally.
For other platforms: No changes to your upgrade procedures should be necessary.
Bad Behavior 2.0.21 has been released. It is a maintenance release and is recommended for all users.
MediaWiki and WordPress users should take note of special upgrade instructions below.
Who should upgrade?
Users who receive significant traffic from the Ukraine should upgrade to fix an issue which may cause users in the Ukraine to be blocked.
All users should upgrade to take advantage of protection from newly identified spambots and malicious bots as well as a new method of spambot detection.
What’s new?
New in this release (since 2.0.20):
- Users who specified the Ukrainian language in their browser settings were mistakenly blocked. This issue has been fixed.
- Bad Behavior now incorporates data on harvesters and comment spammers compiled by Project Honey Pot and published through its http:BL service. In order to enable this feature, you must obtain an http:BL access key and provide this key to Bad Behavior in its settings. While the http:BL settings can be fine-tuned to block or allow requests based on the threat level and age of a harvester or comment spammer record, the default settings have been extensively tested and found to block virtually all spammers known to http:BL while allowing all legitimate users, even those that http:BL may have classified as suspicious. This feature obsoletes any other http:BL plugins you may have, and they can be removed.
- The Majestic-12 search engine crawler was mistakenly blocked. This block has been removed and a block placed for a malicious bot which pretends to be the Majestic-12 crawler.
- The bot used by Attributor, a service which looks for copyright infringement and sends takedown notices, has been identified and blocked.
- Several additional spambots have been identified and blocked by user agent.
Support
If Bad Behavior has helped you, please make a financial contribution toward further development. Your contribution ensures that I can prioritize Bad Behavior development. Otherwise I must spend most of my time on other projects which pay the bills. Which is a shame, because I really enjoy making spammers miserable and drying up their revenue streams until it’s more profitable for them to work at McDonald’s than to send spam.
Download
Download Bad Behavior now!
Special Upgrade Instructions
For MediaWiki: Before installing this version of Bad Behavior, manually remove (e.g. using FTP or ssh) any old versions you may have, including the lines added to LocalSettings.php. Then install the new version fresh, following the installation instructions for MediaWiki.
For WordPress: If updating to this version through the automatic updater fails, manually remove (e.g. using FTP or ssh) any old versions you may have installed. Then upload and install the new version fresh, following the installation instructions for WordPress. After doing so, future automatic updates should proceed normally.
For other platforms: No changes to your upgrade procedures should be necessary.
Bad Behavior 2.0.19 has been released. It is a maintenance release and is recommended for all users.
Warning: The minimum system requirements for WordPress have changed as of this version. Bad Behavior on WordPress now requires at least version 1.5. (It was previously version 1.2.) Users of WordPress versions prior to 1.5 should upgrade WordPress prior to updating to this version of Bad Behavior.
Who should upgrade?
All users should upgrade to take advantage of protection from newly identified bots.
WordPress users should upgrade to use the new administration page which allows for browsing and searching Bad Behavior’s log.
What’s new?
New in this release (since 2.0.18):
- The test for the spambot identified in version 2.0.18 was not functioning correctly. The test has been fixed
- A new administration page has been added for WordPress which allows for browsing through the Bad Behavior log. Click Manage > Bad Behavior to view the log files. This feature will be expanded in the future based on user feedback. WordPress version 1.5 or higher is required.
Support
If you find Bad Behavior useful, please consider making a financial contribution to its further development.
Download
Download Bad Behavior now!
There’s a whole lot of buzz about the newest WordPress spam-fighting plugin on the block, and so I decided to go take a look and see if WP-SpamFree lives up to its hype.
“The WP-SpamFree plugin virtually eliminates automated comment spam from bots, including trackback and pingback spam,” its author, Scott Allen, claims. “It takes a different approach than most and stops spam at the door.”
Indeed, everyone who’s tried it reports that their spam has dropped off to virtually zero and that they haven’t heard from anybody who had problems leaving comments. Sounds like the Holy Grail of spam prevention, right?
Not so fast.
WP-SpamFree, it turns out, uses JavaScript and cookies to verify that someone is using an actual web browser to access your site and leave a comment. These approaches are not that different from what other plugins have done in the past. What distinguishes WP-SpamFree in this respect is that it requires both JavaScript and cookies in order for someone to post a comment. This will certainly keep out virtually every spambot out there.
Unfortunately, it will also block most mobile web browsers and some disabled users. In both cases the browsers being used aren’t capable of JavaScript, cookies, or both. If your blog targets mobile web users or people with disabilities, WP-SpamFree might not be for you.
Then there is WP-SpamFree’s method of blocking trackback and pingback spam. These are always automated, so using JavaScript and cookies is impossible. WP-SpamFree, it turns out, uses several extensive internal lists of IP addresses, URL fragments, and keywords to block this type of spam.
This works fairly well; however, the way it’s implemented in the current version of WP-SpamFree (1.9.6.2) is quite strange. It appears the author didn’t want to use arrays and loops to iterate through his lists and instead unrolled all his loops, resulting in a huge plugin clocking in at over 3,700 lines. There’s no obvious good reason for this; it would seem in PHP that the plugin would be much slower than it would otherwise. The gain of not having the loops doesn’t seem nearly as much as the overhead of compiling thousands of extra lines of bytecode. In addition there are several other examples of duplicate code which could have been split into functions.
These technical implementation issues make me wonder at how much experience the programmer has. If they were intentionally done by an experienced programmer, I would have expected them to be mentioned in the README or release notes or a blog entry, but especially in the code comments.
Despite those issues, the plugin works pretty well for what it does. I hope that the author addresses those implementation issues for his next major version, though, to make the plugin even better.
And there are things that WP-SpamFree does not do. It does not block email harvesters, for instance. It also does not block spambots when they scrape your site looking for your comment forms, nor block denial of service attacks. Indeed, under a heavy spam attack, its size and CPU usage could cause limited web hosting resources to be exhausted.
That’s all just a long way of saying that WP-SpamFree has its pros and cons, and if you choose to use WP-SpamFree, you still should keep Bad Behavior around as part of your overall spam prevention strategy.
This article applies to the 2.x.x series of Bad Behavior. If you are using a 1.x.x version of Bad Behavior, please update as soon as possible.
One of the two topics I get most frequently is the assertion that Bad Behavior has blocked a legitimate request from an actual user, sometimes even the owner of the blog! Since this seems to come up every so often, I’m going to see if I can help out, and maybe eliminate the need for some of these folks to contact me.
(But before we get started, if you are an AOL user, do not use the built-in AOL browser. Use
Firefox or something else. And get a real ISP as soon as possible.)
Before doing anything else, ensure that you have the latest version of Bad Behavior. Do not leave a comment or contact me if you have failed to update to the latest version. Too many people have done exactly that. It is your responsibility to know how to install and update software on your own Web site.
The next thing to do is to determine why Bad Behavior blocked you. Bad Behavior will display a short message along with a technical support key and a link to “fix the problem yourself.” Make a note of the technical support key, and then click the link. You’ll be presented with more information on why the request was blocked and several suggestions on how to fix the problem.
If you’ve been blocked from a site, and you aren’t the site administrator, please contact that person first, as they will be able to access records on their web server which will be helpful in solving the problem. Be sure to provide them with the technical support key you received. (If you are trying to access a site from a corporate or government network, you may need to contact the network administrator for your company or government agency to resolve the problem.)
If you are the site administrator, and one of your users was blocked and has contacted you for help, you can go directly to the support page and look up their technical support key yourself. You can use either the 8-character key from your database entries, or the 16-character key shown to users, with or without hyphens. You’ll then see the page that would have been shown to that user.
But you should ensure that your user has already followed the suggestions given on the page. The support page is written with non-technical users in mind, and so those of you who really know what you’re doing probably won’t like it, but it’s been my experience that, excepting the occasional bug in Bad Behavior, almost every actual human being who sees the page is able to fix the problem themselves.
If you’re unable to fix the problem yourself, and you’re the site owner/administrator, get your IP address, or the user’s IP address, log in to your phpMyAdmin, and Search the wp_bad_behavior table for the IP address and the last half of the technical support key (without the hyphen). Export the records from phpMyAdmin in SQL format and send them to me. You do not need to zip them, but it’s OK if you do. Please do not export in any other format but SQL. If you send me a screenshot, a PDF, or even worse, an Excel file, I will curse your name until the end of days, and probably not respond.
Finally, if Bad Behavior has been valuable to you, please consider making a contribution to further Bad Behavior development.
Project Honey Pot made several announcements this week, the largest of them Thursday when it announced it had filed a $1 billion lawsuit against spammers on behalf of the members of Project Honey Pot. I’m proud to say I’ve been such a member for some time now, and will lend whatever assistance I can to efforts to stop spam.
Project Honey Pot has been targeting email spam for years. But now it has also quietly launched an initiative to target blog comment spam. I’m proud to say I’m also participating in that effort.
On Wednesday, the project announced http:BL, a DNS-based blacklist of IP addresses which have been seen harvesting email addresses and sending email and comment spam. This is just about exactly what I had in mind when I announced the Bad Behavior Blackhole almost two years ago; Project Honey Pot has actually built something better.
I’ve spent the last day or so evaluating http:BL and found that its design is unfortunately not amenable to adding directly in to Bad Behavior, as it has significant technical differences from other DNS-based blacklists.
Therefore, I’m writing a separate http:BL plugin for WordPress. I’m currently testing it here and I hope to make the first release in the next few days.
Project Honey Pot relies on webmasters who want to actively participate in stopping spam. But the project has only a few bloggers running honey pots, so it’s not yet catching a lot of comment spam bots.
You can help by signing up for Project Honey Pot and installing a honey pot on your blog, forum or wiki.
Your honey pot, along with millions of others, will trap spambots of all types and feed its data into http:BL, which will improve the service for everyone.
Bad Behavior 2.0.9 has been released. It is a strongly recommended upgrade for all users.
This release is likely the final release in the 2.0 series as I make a major change in the development process; see below for details on this change.
This release addresses a further set of “false positive” reports received from various users which affect some uncommon circumstances.
New in this release (since 2.0.8):
- A workaround has been placed for a problem with the Clearswift Web Policy Engine. Users behind this proxy server are no longer blocked.
- A workaround has been placed for a bug in the LiveJournal OpenID process which Six Apart refuses to fix. Logins using OpenID will no longer fail.
- A workaround has been placed for bugs in some versions of Internet Explorer and Safari web browsers which caused them to be blocked after leaving a comment on WordPress. These requests are no longer blocked.
- A spam prevention feature was causing users to be blocked from their own blogs when they also subscribed to their own feed, or when they accessed the site with multiple web browsers at the same time; it has been disabled for rework.
Download Bad Behavior now!
The 2.0 series of Bad Behavior will be maintained as a legacy branch, with only bug fixes, false positive fixes and security fixes applied to this branch, if any such fixes are needed. No new checks for spammers will be added.
Shortly I will introduce a “development” 2.1 series on a much shorter development cycle, with days or perhaps even hours between releases. In this branch I’ll be experimenting with new spam prevention features, rolling them out quickly and rolling back quickly in case of actual trouble. I’ll also be rolling out a new packaging method which I’ve discussed previously, that will make Bad Behavior even more platform-independent than it currently is, and allow for the “core” to be updated separately from the “glue” which connects it to your host platform.
Once features prove themselves through development and testing to be stable, they’ll be rolled forward into a “stable” 2.2 series, intended for those users who are averse to the risks of blocking legitimate users or having the occasional crash. While I work very hard to ensure that every release, however labeled, does not crash, and does not generate false positives, things occasionally happen which are outside my control.
This parallel development scheme will help balance the needs of the two primary groups of Bad Behavior users.
The first group needs enterprise-grade code which ideally never blocks a single legitimate request and can quickly be rolled into production environments with a high degree of confidence. The tradeoff is the same as it has always been: to prevent any chance of false positives, Bad Behavior’s stable branch will permit some spam, anywhere from 0.1% to 10%, to pass through, and will require a backup solution such as Akismet. Even so, it will drastically reduce the amount of time and money spent managing spam, especially for deployments of dozens or hundreds or thousands of sites.
To serve this class of users more effectively, I’m also studying the feasibility of offering support contracts for enterprise users of Bad Behavior. Services offered under such contracts might include installation assistance, on-call support, hotfix development and deployment, and per-incident support. If your organization may need such a service, stay tuned for more details in the near future.
The second group, I believe, is the majority of Web sites: those for whom a rare blocked user is merely an annoyance rather than a critical problem, and who have much lower tolerance for spam because they aren’t being paid to manage their own blogs, wikis and forums. As much as possible, Bad Behavior’s development branch will limit spam for this class of users to 0.5% missed. The tradeoff is that you will be asked to do what you already do: to report any problems you encounter, whether they be missed spam or blocked users or plain old crashes.
And for users who would like to have their cake and eat it too, the development and stable versions will be installable side-by-side on the same site, and you will be able to switch back and forth between them at the click of a button.
Finally, prior to the first stable 2.2 release, I will be reworking all of Bad Behavior’s documentation and moving Bad Behavior from its current home to a new site dedicated solely to Bad Behavior. So you all will have to update your feed URLs to the new location soon. (Mailing list readers won’t have to do anything.)
In the meantime, Bad Behavior remains a user-supported project, with all code released under the GNU General Public License. If you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my limited spare time, and every contribution means I can devote more time to its development.
Bad Behavior 2.0.8 has been released.
This version contains updates for various “false positive” reports and is recommended for all users.
Updated in this release (since 2.0.7):
- Verizon Wireless EV-DO users are no longer blocked.
- Blocked requests will be subject to a two-second delay before a response is sent. (See below.)
- Some blackhole lists previously used in Bad Behavior have been scaled back or removed.
- The address for the Bad Behavior Blackhole has been added. (See below.)
- Some new spambots have been identified and blocked.
In recent days spam attacks have been on the rise, with one especially obnoxious bot delivering requests so fast that some sites have been taken offline by them. While the requests aren’t especially numerous or resource-intensive, the most common software used by Web hosting providers is very inefficient at serving dynamic pages such as PHP-based Web sites. So even a moderate number of requests can take a whole server down, or lead the hosting provider to take the site down before the whole server goes down.
Bad Behavior now counters this by introducing a short two second delay to blocked requests, before the HTTP response is sent. Since most spambots wait for the response before going on to the next request, this should sufficiently slow down most of the overly aggressive spambots and give Web site operators some breathing room. While I would have liked to put in a delay of a minute or more, there remains the slight chance that an actual human being would be blocked, and they should be able to get a response back in a reasonable time.
With respect to realtime blackhole lists, all of the existing lists target e-mail spam, and since spambots who send link spam are almost always also sending e-mail spam through the same servers, these are a fairly effective means of blocking link spam. However, since they target e-mail spam, they also block legitimate users. The primary issue here is that while an IP address may be added to a blackhole list quickly, it is not removed quickly — or at all — once the spam stops. Thus, people with dynamic IP addresses are unfairly blocked because some other customer was sending spam.
Bad Behavior Blackhole, which should go online within the next few weeks, is designed specifically for link spam. It adds IP addresses to its database quickly when actual spam is received, and in addition, drops the IP addresses once the spam stops. This helps prevent dynamic IP customers from being blocked because another user’s computer was sending spam. Once Bad Behavior Blackhole is online, all other realtime blackhole lists will be dropped from Bad Behavior.
Download Bad Behavior now!
As always, if you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my spare time, and every little bit means I have more spare time to devote to its development.
And don’t forget to subscribe to the RSS feed or the mailing list. (They’re the same content.)