I recently wrote something about fighting Emacs Wiki spam on Google+.
Lauren said “I’m hoping we can stay focused on practical processes and solutions that are workable at very large scales.” This post is about a wiki that is popular but not as popular as Wikipedia.
I’ve been running Emacs Wiki for a while now. It’s definitely not “at very large scales” but it was certainly a step up from all my previous wiki efforts. I got started back on the Portland Pattern Repository and Meatball Wiki and so I kept trying to live up to Soft Security. That’s why Emacs Wiki still doesn’t have logins and passwords for ordinary users. There are passwords for roles that allow you to lock pages and to edit locked pages, for example.
The first line of defense I added was rolling back edits. The first wiki I used allowed you to edit an old revision and save it (click history link, click good revision, scroll to bottom, click edit link, scroll to bottom, click save button). I wanted to speed up the clean up. Now you you click history link and click the rollback button.
The remaining features are reserved for administrators.
We are mostly getting link spam. Therefore I soon introduced a list of regular expressions (on a locked page that I could edit together with my co-administrators). In order to speed this up, I added some code to rollbacks. If the rollback removed URLs from the page text, those are listed and you automatically get a form where you can write a regular expression based on the list you’re seeing. Clicking the submit button will add this regular expression to the ban list.
This is a very crude measure. Luckily enough, it still works often enough. Perfect for defense in depth. I added more code to rollbacks. After a rollback, administrators are presented with a link to “ban contributors”. If you click on it, you’re presented with the editors of recent page revisions, together with a note indicating whether they have been banned or not. You can check the IP numbers of the contributors not yet banned and click a button to add an appropriate regular expression to the list.
This is also a very crude measure. It’s our last automatic defense. We’ve added a few regular expressions to this list such as the Russian word for porn because we were getting vandals that posted links to forums and the like, together with some keywords, and those forum posts would then contain the link to the material we wanted to ban. The indirection via forum, URL shortener and the like circumvented our earlier ban mechanisms. This was our solution.
If we’re under a large scale attack, we can always lock the wiki and wait. Once the damage is done, however, we can reset the wiki to a particular edit, generating the appropriate rollbacks for every page (i.e. these rollbacks are all regular edits and can again be undone by other administrators).
There are also other defense mechanisms unrelated to banning.
We only ask for a CAPTCHA once. Answering the question sets a cookie that bypasses the CAPTCHA. Clearly, this only works for a low profile site.
If you load too many pages in a short time window, the wiki will start responding with error messages, assuming that you’re a bot and not a human.
There is currently no way to automatically ban the entire IP range given an IP number.
There is currently a semi-automatic expiry process for bans. It would be better if this was automatic. These days I have to run a whois query, type the two IP numbers into my little Python programm, and type the result into my
There used to be a ban network sharing those regular expression lists but it was brittle and so I abandoned it.
Wow, what a spam storm! I rolled back all those changes and locked the wiki for the moment. There’s no fighting a bot manually.
I’ll look into what allowed this to happen later today and hopefully enable page editing and commenting again.
Update: IP number blocked, lock removed. That was weird. :vee:
Wow, spammers are learning about role playing games…
I still hate spammers, of course. But I have to confess, an adventure “on the most dangerous island of the world, R’lyeh, the resting place of the great old one, Cthulhu.” Just, wow! Or woe, I guess.
I’ve started to note a particularly nasty kind of spam for the Best Essays website. They don’t only add links to the wiki pages. At first they posted entire new pages that are on the topic of writing essays to CommunityWiki. But lately they’ve been using my username (!) to make edits. I assume they’re trying to sidestep peer review. I’ve just silently rolled back any changes they made, but today I saw another edit using my username, and when I investigated I found how they had experimented with the rollback function themselves and linked to an old revision containing a spam link to Best Term Papers. There must be real humans at work.
Here I am trying to send mail using some Perl module. But most of them seem to be written for the last millenium. They work best with a local SMTP host. For my own needs, that no longer works. The webhost doesn’t have a sendmail binary available. The SMTP hosts I can reach require SSL and TLS authentication. Now I’m slowly digging into MIME::Entity, Mail::Internet, Mail::Mailer, Net::SMTP, Net::SMTP::SSL, Net::SMTP::TLS… :sucks:
After experimenting with the four or five mail accounts I have access to, I was finally able to get the following to work:
my $from = 'firstname.lastname@example.org'; my $to = $from; my $host = 'mail.epfarms.org'; my $user = 'alex'; my $password = '*secret*'; use MIME::Entity; my $mail = new MIME::Entity->build(To => $to, From => $from, Subject => 'test', Path => '/Users/alex/test.html', Type => 'text/html'); use Net::SMTP::TLS; my $smtp = Net::SMTP::TLS->new($host, User => $user, Password => $password, Debug => 1); $smtp->mail($from); # sender $smtp->to($to); # recipient $smtp->data; $smtp->datasend($mail->stringify); $smtp->dataend; $smtp->quit;
Often I log on to my various instant messaging networks (→ Contact) and some dummy Yahoo account will send me spam. Something like the following:
Back at work… got over 10’000 spam emails in my Junk folder, and about 300 mails in my Inbox with a 50:50 mix of spam to ham. Not too shabbby and lots better than in previous years!
I switched to GMail a long time ago because I felt that GMail had superior anti spam measures in place. These days I’m not so sure. I have less mail in my Spam box (4065 right now), and yet I have to manually mark about a dozen mails as spam every day. Are other users poisoning the spam filters? Maybe inactive accounts gathering spam negatively affect the filters…
Well, I’m being hit by spam of the form
gdufty upibhmf fknwvl aixqf gumi pqrjxfb ntscbzoyp
-- [http://www.icxjrg.yamfw.com krwfuq qeysb] 2007-04-24 15:27 UTC
egzp supbvqnf qbgtfnyv bmilfzo ewfhl xospqz rtmxfbajo http://www.mvuh.meptrc.com
-- [http://www.google.com mvny vqahjngdu] 2007-04-24 15:28 UTC
How stupid is that?
Unfortunately, that means you’ll have to answer a question when you try to post to this site.
When an idiot comes along and spams twenty pages on my site, I can easily roll it back. But all changes are recorded for posterity in the log files, and therefore they appear on RecentChanges and thus also in the RSS feed. Too bad!
Unfortunatly simplistic schemes won’t work: I cannot skip all entries whose last edit was a rollback. I need to figure out what edits where undone by the rollback and then check whether there are any remaining edits in the current time window… Damn!
Define external redirect: CommunityWiki