Tuesday :: May 3, 2005

Site Difficulties Update

by Mary

As Steve noted in his post, comments had been disabled on the site in response to a very bad spam attack today. Things should be back to normal now, but I thought I'd explain what happened and what we are doing about it (besides praying that the spammers get their just reward).

Also, if you encounter technical problems on the site, please copy me on any mail you send Steve so I can also look into it too.

First of all, comments and trackbacks for posts are susceptible to spam because spammers use it to increase their site ratings on the search engines. Google is fighting back by providing a "nofollow" directive that I will be adding to the html embedded in comments which tells search engines to not count the links and not to follow the links to the site referenced in comments. (Of course, we as human readers can follow the links provided in comments to our hearts content because this is really only a key word for search engine spiders that are following links to create a map of the internet space.) By adding the "nofollow" directive to comments posted on the site, we remove any value the spammer gets for adding spam because the new links won't get counted and they won't get paid for additional links.

Second, we are keeping up (or trying to keep up) with the latest technology from SixApart, the company that sells Moveable Type. We've been using mt-blacklist to trap and quarentine spam comments and trackbacks. Recently SixApart recommended Brad Choate's SpamLookup plugin which is an enhancement to the existing mt-blacklist utility and provides a centralized list of spam urls that are blocked from adding comments or trackbacks if the content or url contains sites that are on the blacklist. These utilities also block a posts that contains a lot of urls (more than 10) because 99.9% of the time, comments or trackbacks with lots of urls come from spammers. (If you ever have problems posting a comment with lots of urls, break up your comment into two or more comments so you don't trigger this trap.)

One thing I've learned about comment and trackback spam is that this type of spam is quite similiar to the problems people have seen with email spam. Things start out quite simply but steadily get more complicated. (And who says the evolution isn't a fact?)

Thus, the first comment spams were very simple. Then the spam bot (which is really just a little automatic program used to enter spam) got smarter. Next, spam blockers saw that lots of comments being added by the same IP address were easy to block by adding the IP address to MT's banned IP list. So then spammers started to use dynamic DNS addresses which are harder to block. In response, the spam blockers added functionality to ban a set of IP addresses from a particular dynamic dns provider. (Spammers use a particular dns provider until it gets blocked and then create another provider - just list crooks will use one cell phone number until that one gets too hot and then they get another one.) And on and on and on.... What SpamLookup does beyond mt-blacklist is to look up ALL the known spam dns providers instead of just one or two like the axillary utilities did that we use today.

So what happened today? The spammers figured out if they tried to add their comments anonymously, they could avoid one of the checks used by mt-blacklist to stop spam. When I got home from work tonight, I cleaned out over 200 spam comments which had all been added without a user name. The attack was arrested by our ISP detecting a pattern of too many comments being added which looked more like a program than humans and so they disabled our comments to prevent the attack from bringing down the entire server. Once I realized what happened, I reenabled the comments (hopefully long after the spammers gave up and the blacklist utility got updated with a fix to block the latest hole).

Just as we've found with email spam, spammers are always looking for a way to get past the roadblocks we've put up against them. In fact, they are like the HIV virus: when they encounter one barrier, they mutate and try another avenue to break through the blocks. Will we have better luck at stopping spam than viruses? Some experts in the field do believe that email spam is pretty much defanged and that blogs can solve the problem using the same blockades as the email spam blockers. But as a student of evolution, I don't know how confident I would be in predicting success against the spammers. Just like the battle humans have had with bacteria in the past 60 years, it looks like we once more have to face deadly bacteria that is impervious to our most advanced antibiotics. Certainly we will find ways to fight off the spammers, but probably not without suffering some of the consequences of a successful attack.

What else can we do? I'm planing to upgrade our site to using SpamLookup by this weekend which should give us some pretty good immunity for awhile. But, there are other things we could do today to stop the spammers. We could require users to register if they want to leave comments, or we could refuse anonymous comments. Do let me know what you think about these approaches by dropping a comment to this post or sending me email. And let me know if you have any better ideas about how to stop spammers in their tracks. We'll be working hard to stop them however we can.

PS: A special note for those who followed this post to the bitter end: Crooked Timber's take on the aliens that produce the spam. And thanks to you all for reading The Left Coaster.

Mary :: 10:06 PM :: Comments (10) :: Digg It!