# 2015-03-10 Fighting Wiki Spam

## Wikis

Lauren said “I’m hoping we can stay focused on practical processes and solutions that are workable at very large scales.” This post is about a wiki that is popular but not as popular as Wikipedia.

I’ve been running Emacs Wiki for a while now. It’s definitely not “at very large scales” but it was certainly a step up from all my previous wiki efforts. I got started back on the Portland Pattern Repository and Meatball Wiki and so I kept trying to live up to Soft Security. That’s why Emacs Wiki still doesn’t have logins and passwords for ordinary users. There are passwords for roles that allow you to lock pages and to edit locked pages, for example.

## Rollback

The first line of defense I added was rolling back edits. The first wiki I used allowed you to edit an old revision and save it (click history link, click good revision, scroll to bottom, click edit link, scroll to bottom, click save button). I wanted to speed up the clean up. Now you you click history link and click the rollback button.

The remaining features are reserved for administrators.

## Banning URLs by regexp

We are mostly getting link spam. Therefore I soon introduced a list of regular expressions (on a locked page that I could edit together with my co-administrators). In order to speed this up, I added some code to rollbacks. If the rollback removed URLs from the page text, those are listed and you automatically get a form where you can write a regular expression based on the list you’re seeing. Clicking the submit button will add this regular expression to the ban list.

## Banning IP numbers

This is a very crude measure. Luckily enough, it still works often enough. Perfect for defense in depth. I added more code to rollbacks. After a rollback, administrators are presented with a link to “ban contributors”. If you click on it, you’re presented with the editors of recent page revisions, together with a note indicating whether they have been banned or not. You can check the IP numbers of the contributors not yet banned and click a button to add an appropriate regular expression to the list.

## Banning key words

This is also a very crude measure. It’s our last automatic defense. We’ve added a few regular expressions to this list such as the Russian word for porn because we were getting vandals that posted links to forums and the like, together with some keywords, and those forum posts would then contain the link to the material we wanted to ban. The indirection via forum, URL shortener and the like circumvented our earlier ban mechanisms. This was our solution.

## Mass Rollback

If we’re under a large scale attack, we can always lock the wiki and wait. Once the damage is done, however, we can reset the wiki to a particular edit, generating the appropriate rollbacks for every page (i.e. these rollbacks are all regular edits and can again be undone by other administrators).

There are also other defense mechanisms unrelated to banning.

## Surge Protection

If you load too many pages in a short time window, the wiki will start responding with error messages, assuming that you’re a bot and not a human.

## Missing features

There is currently no way to automatically ban the entire IP range given an IP number.

There is currently a semi-automatic expiry process for bans. It would be better if this was automatic. These days I have to run a whois query, type the two IP numbers into my little Python programm, and type the result into my .htaccess file.

There used to be a ban network sharing those regular expression lists but it was brittle and so I abandoned it.

Tags:

Comments on 2015-03-10 Fighting Wiki Spam

Great post.

There is one thing about Soft Security that makes me feel worried. What if some insane person wants to terrorize the wiki? That is, a directional attack, without links, without keywords, from different IPs. And all that stuff is mixed with minor edits, impersonation and malicious rollbacks. The only way to undo this mess it to restore the content from a backup (which is rather easy to do with the git extension I use, but it is still a pain).

The only thing that prevents this is the lack of motivation (which is somewhat associated with the absence of competition), but I’m not sure if that applies to all other wikis out there.

AlexDaniel 2015-03-10 23:55 UTC

Here are my disorganized thoughts on the limits of Soft Security.

I think we can talk about two different kinds of attacks: a long term infiltration under the radar and a massive attack on multiple levels. Both attacks need to be stopped by fish-bowling the wiki: making it read-only.

In order to detect long term infiltration, you need constant peer review. I myself have bookmarks for my wikis, obviously. They all look like this: ...wiki?action=rc;showedit=1;days=3;rollback=1;lang=;css= or similar. I want to see rollbacks and minor edits! This is important.

Clearly, Soft Security does not work as well if the community is small, visits rarely, or if there are long periods of inactivity.

In order to defend against sudden onslaughts, the ability to lock the wiki needs to be available to a lot of people that are constantly watching. You also need the ability to mass revert the vandalism. If your site does not provide this, restoring from backup will be necessary.

1. Do you know how to lock the wiki? It must also be easy to unlock—easy enough for your friends to do it, hard enough to prevent the vandals from doing it.
2. Oddmuse has the ability to mass revert vandalism, but only up to the “keep days” window (14 days by default). This time window is important: Do you check at least once in this window? Including holidays and business trips?
3. Do you have backups from various points in time?

In a way, for small communities using Soft Security, defense in depth now means defense in time.

For Campaign Wiki, for example, there is a way for people to lock and unlock a wiki. The setting is easy enough to find (I hope) but hidden well enough to elude casual spammers.

There are limits to Soft Security, however. Looking at the abuse Anita Sarkeesian endures every week makes me think that perhaps Soft Security wouldn’t work for her. Then again, perhaps we would learn that many more people support her, defending her site against the trolls?

– Alex Schroeder 2015-03-11 09:47 UTC

# 2014-12-30 Emac Wiki Migration

OK, I guess Nic and I have a plan. We’ve started writing a backlog for the Emacs Wiki Migration. Hopefully this will make the current wiki easier to cache.

# 2014-12-28 Emacs Wiki Migration

Gaah!

So, I still think this is a problem with the hosting provider. Here’s what I think is evidence:

What could be causing this regular forks if not other virtual machines hogging CPU at regular intervals, or other virtual machines hogging disk access, causing CGI scripts to pile up? I wonder…

Things I care about: Diagnosing Disk I/O issues: swapping, high IO wait, congestion.

Let’s see what iotop -ao says right now:

Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
11774 be/4 www-data     36.00 K     20.00 K  0.00 %  0.02 % [/usr/sbin/apach]
11801 be/4 www-data      0.00 B     12.00 K  0.00 %  0.00 % apache2 -k start
11810 be/4 www-data      0.00 B      4.00 K  0.00 %  0.00 % apache2 -k start
29022 be/4 root          0.00 B     32.00 K  0.00 %  0.00 % monit -c /etc/monit/monitrc
20368 be/4 root          0.00 B     60.00 K  0.00 %  0.00 % rsyslogd -c5

I don’t think we’re using swap at all (munin shows us using less than 2G with 4G available). But who knows? I used top, pressed f to show the list of fields, and I used d (display) and s (sort) on SWAP.

top - 22:14:00 up 7 days, 21:11,  2 users,  load average: 0.51, 0.31, 0.22
Tasks:  24 total,   1 running,  23 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.6 us,  0.6 sy,  0.0 ni, 95.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   4194304 total,   894976 used,  3299328 free,        0 buffers
KiB Swap:  1048576 total,     5524 used,  1043052 free,   785880 cached

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND              SWAP
15874 root      20   0 71192 5224 1620 S   0.0  0.1   2:11.36 fail2ban-server      2296
27076 root      20   0 72564 1092  708 S   0.0  0.0   0:16.47 sendmail-mta         1384
21949 root      20   0 49892  684  568 S   0.0  0.0   0:02.23 sshd                  492
29022 root      20   0 42984 2688 2136 S   0.0  0.1   0:48.28 monit                 284
20367 root      20   0 49860 4504  712 S   0.0  0.1   0:24.56 rsyslogd              236
19427 root      30  10 12252  788  784 S   0.0  0.0   0:00.00 rsync                 160
15876 root      20   0 22260  896  672 S   0.0  0.0   0:08.86 gam_server            136
22815 root      20   0 20368  496  364 S   0.0  0.0   0:02.03 cron                  120 

I don’t think that’s suspicious. Those tasks probably ended up in swap because they have been running for so long?

I installed the apache_vhosts plugin for munin. I think there will be no surprises with emacswiki.org taking the brunt with a factor of 10:1. Right now, however, it seems to be misconfigured somehow, not recording any emacswiki.org traffic and missing most of the other traffic. Weird!

Ah, apache_vhosts munin plugin problem fixed. All my sites were using a CustomLog except for alexschroeder.ch:80 and campaignwiki.org:80—these just redirect to the HTTPS version of the site. Thus, these sites were being counted by the apache_vhosts plugin. But that is now fixed, and we have a graph!

I’m not quite sure how to compare this with the following log file analysis where I simply group access by site:

root@kallobombus:/var/log/apache2# perl /home/alex/bin/site-comparison < access.log
Site       hits bandw. hits% interv. status code distrib.
www.emacswiki.org     104499     6K  47%    2.1s  301 (29%), 501 (26%), 200 (21%
www.alexschroeder.ch      67396    31K  30%    3.2s  200 (94%), 304 (2%), 301 (1%),
www.communitywiki.org      24574    17K  11%    8.9s  200 (80%), 301 (7%), 404 (3%),
www.campaignwiki.org      13902    13K   6%   15.6s  200 (74%), 404 (17%), 302 (4%)
www.oddmuse.org       9498    23K   4%   22.9s  200 (66%), 404 (14%), 301 (12%
www.arabisch-lernen.org        845     7K   0%  256.4s  200 (94%), 404 (3%), 403 (1%),
www.orientalisch.info        404     4K   0%  536.5s  200 (80%), 301 (14%), 404 (2%)

Ah, of course. The access.log is can be up to seven days. Thus:

root@kallobombus:/var/log/apache2# tail -n 10000 access.log | /home/alex/bin/site-comparison
Site       hits bandw. hits% interv. status code distrib.
www.alexschroeder.ch       6430    24K  64%    1.9s  200 (88%), 301 (8%), 404 (1%),
www.emacswiki.org       1187    12K  11%   10.4s  301 (38%), 200 (36%), 404 (19%
www.communitywiki.org       1150    14K  11%   10.7s  200 (79%), 301 (7%), 404 (5%),
www.campaignwiki.org        604     9K   6%   20.3s  200 (58%), 301 (22%), 404 (12%
www.oddmuse.org        415    18K   4%   29.6s  200 (64%), 404 (18%), 301 (10%
www.arabisch-lernen.org        202     5K   2%   37.4s  404 (90%), 200 (9%), 501 (0%)
www.orientalisch.info         12     4K   0% 1010.2s  200 (91%), 304 (8%)

# 2014-12-27 Emacs Wiki Migration

Based on some information I got from Charlie McMackin—who had set up a virtual machine and got Emacs Wiki to run on it—I switched Apache from “worker” to “prefork” and I’m now using the following config:

<IfModule mpm_prefork_module>
StartServers          2
MinSpareServers       1
MaxSpareServers       2
MaxClients		  5
MaxRequestsPerChild 10000
</IfModule>

I checked http://localhost/server-info and can confirm that we’re running Prefork:

Server Settings

Server Version: Apache/2.2.22 (Debian) mod_ssl/2.2.22 OpenSSL/1.0.1e mod_perl/
2.0.7 Perl/v5.14.2
Server Built: Jul 24 2014 15:34:00
Compiled with APR Version: 1.4.6
Compiled with APU Version: 1.4.1
Module Magic Number: 20051115:30
Hostname/port: alexschroeder.ch:443
Timeouts: connection: 300    keep-alive: 5
MPM Name: Prefork
MPM Information: Max Daemons: 5 Threaded: no Forked: yes
Server Architecture: 64-bit

# 2014-12-26 Emacs Wiki Migration

I’m not happy. Today, multiple Apache restarts. Grepping syslog for restarts:

Dec 26 07:38:32 localhost monit[29022]: 'apache' trying to restart
Dec 26 11:38:53 localhost monit[29022]: 'apache' trying to restart
Dec 26 14:47:16 localhost monit[29022]: 'apache' trying to restart
Dec 26 16:49:46 localhost monit[29022]: 'apache' trying to restart
Dec 26 17:26:06 localhost monit[29022]: 'apache' trying to restart
Dec 26 18:06:29 localhost monit[29022]: 'apache' trying to restart
Dec 26 19:48:54 localhost monit[29022]: 'apache' trying to restart✎

I’m going to further reduce the number of threads.

<IfModule mpm_worker_module>
StartServers          1
ServerLimit           2
MaxClients           10
MaxRequestsPerChild  10000
</IfModule>✎

I also installed lsof and wrote a little plugin for munin. What I really want to see is the number of open files, or something like that. I remember checking that when we had load problems on Mirabel without a matching number of processes actually doing anything. Just in case the processes are all waiting for file access.

I’m not sure how well this is going to work, but this is my first stab at the files plugin. 😊

#!/bin/sh
#
# Plugin to monitor the number of open files in the system.
#
# Parameters:
#
# 	config   (required)
# 	autoconf (optional - used by munin-config)
#
# Magic markers (Used by munin-config and some installation scripts.
# Optional):
#
#%# family=contrib
#%# capabilities=autoconf

# PROC=${0##*/files_} PORTS="80 443" if [ "$1" = "autoconf" ]; then
if [ -x /usr/bin/lsof ]; then
echo yes
exit 0
else
echo "no (no lsof)"
exit 0
fi
fi

if [ "$1" = "config" ]; then echo graph_title "open files" echo graph_vlabel "number of files" echo graph_info "This graph shows the number of open files by port (80 is http, 443 is https)." echo graph_category "apache" for P in$PORTS; do
echo "$P.label port$P"
done
exit 0
fi

for P in $PORTS; do N=$(lsof -i :$P | tail -n +2 | wc -l) echo "$P.value $N" done✎ All I know is that load is climbing right now. Close to 6. Time for bed! Grrrrr. A few hours later, stats seem to be worse: Dec 26 22:47:16 localhost monit[29022]: 'apache' trying to restart Dec 26 23:23:34 localhost monit[29022]: 'apache' trying to restart Dec 26 23:58:01 localhost monit[29022]: 'apache' trying to restart Dec 27 00:10:19 localhost monit[29022]: 'apache' trying to restart Dec 27 00:38:40 localhost monit[29022]: 'apache' trying to restart Dec 27 01:06:58 localhost monit[29022]: 'apache' trying to restart Dec 27 01:17:15 localhost monit[29022]: 'apache' trying to restart Dec 27 03:51:40 localhost monit[29022]: 'apache' trying to restart Dec 27 04:23:59 localhost monit[29022]: 'apache' trying to restart Dec 27 04:42:21 localhost monit[29022]: 'apache' trying to restart Dec 27 05:30:47 localhost monit[29022]: 'apache' trying to restart Dec 27 06:59:09 localhost monit[29022]: 'apache' trying to restart Dec 27 08:05:33 localhost monit[29022]: 'apache' trying to restart✎ Let’s hope that things will improve with prefork! Add Comment # 2014-12-25 Emacs Wiki Migration I’ve been thinking about special-casing some caching: a mod_rewrite rule that says: • if the URL has the form /emacs/page • and there is no cookie (ie. bots, just readers) • and there is an if-none-match header (from nginx, I presume?) • and its value matches the “default” • return 304 without invoking the wiki So, what’s the “default”? sub DoBrowseRequest { ... print$q->header(-status=>'304 NOT MODIFIED') and return if PageFresh(); # return value is ignored
...
}

sub PageFresh { # pages can depend on other pages (ie. last update), admin status, and css
return 1 if $q->http('HTTP_IF_NONE_MATCH') and GetParam('cache',$UseCache) >= 2
and $q->http('HTTP_IF_NONE_MATCH') eq PageEtag(); } sub PageEtag { my ($changed, $visible, %params) = CookieData(); return UrlEncode(join($FS, $LastUpdate||$Now, sort(values %params))); # no CTL in field values
}✎

And what about $LastUpdate?  my$ts = (stat($IndexFile))[9]; # always stat for multiple server processes ...$LastUpdate = $ts;✎ So, I think that the “default” ETag should simply be the mtime of the pageidx file. We’d have to use the RewriteMap directive with an External Rewriting Program. Yikes! I’m not sure this is really faster… I guess it would be faster if the wiki itself was no longer using mod_perl (for whatever reasons). The external rewriting program would comparing the ETag header with the pageidx mtime and return a special value which would in turn be rewritten by a second rule to “redirect” to a 304 NOT MODIFIED response… Something to consider for later. As I’m writing this, I’m noticing that the server is slow. Check the current time, right, the five minutes before the full hour are a problem. Load reaching 6. top - 17:59:18 up 4 days, 16:57, 1 user, load average: 5.94, 5.41, 4.04 Tasks: 20 total, 1 running, 19 sleeping, 0 stopped, 0 zombie %Cpu(s): 98.7 us, 1.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 4194304 total, 1688940 used, 2505364 free, 0 buffers KiB Swap: 1048576 total, 4416 used, 1044160 free, 1331364 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2635 www-data 20 0 470m 122m 4704 S 101.9 3.0 42:43.65 /usr/sbin/apach 4902 www-data 20 0 434m 91m 5360 S 72.6 2.2 21:12.41 /usr/sbin/apach 1 root 20 0 10608 196 60 S 0.0 0.0 0:04.12 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd/2828 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper/2828 702 root 20 0 51560 9856 2068 S 0.0 0.2 0:11.56 munin-node 1790 messageb 20 0 29764 256 12 S 0.0 0.0 0:00.00 dbus-daemon 8703 root 20 0 71252 3664 2872 S 0.0 0.1 0:00.02 sshd 8707 alex 20 0 71252 1720 924 S 0.0 0.0 0:00.00 sshd 8708 alex 20 0 21256 4096 1712 S 0.0 0.1 0:00.05 bash 8723 alex 20 0 23120 1612 1188 R 0.0 0.0 0:00.00 top 15874 root 20 0 71192 5168 1620 S 0.0 0.1 1:20.36 fail2ban-server 15876 root 20 0 22260 908 672 S 0.0 0.0 0:06.63 gam_server 19427 root 30 10 12252 956 792 S 0.0 0.0 0:00.00 rsync 20367 root 20 0 49728 4624 712 S 0.0 0.1 0:19.49 rsyslogd 21949 root 20 0 49892 1236 628 S 0.0 0.0 0:00.81 sshd 22815 root 20 0 20368 496 364 S 0.0 0.0 0:01.23 cron 25118 root 20 0 122m 10m 3932 S 0.0 0.3 0:01.28 /usr/sbin/apach 27076 root 20 0 72564 2024 732 S 0.0 0.0 0:09.69 sendmail-mta 29022 root 20 0 42984 2976 2148 S 0.0 0.1 0:18.03 monit ✎ A few minutes later load reached 7. And my processes aren’t busy. top - 18:01:45 up 4 days, 16:59, 1 user, load average: 6.65, 5.91, 4.43 Tasks: 24 total, 2 running, 22 sleeping, 0 stopped, 0 zombie %Cpu(s): 97.2 us, 0.9 sy, 1.7 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.2 st KiB Mem: 4194304 total, 1726856 used, 2467448 free, 0 buffers KiB Swap: 1048576 total, 4416 used, 1044160 free, 1331376 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4902 www-data 20 0 437m 108m 5384 S 61.6 2.7 22:46.97 /usr/sbin/apach 9105 munin 30 10 140m 23m 5936 R 3.3 0.6 0:02.26 munin-graph 1 root 20 0 10608 196 60 S 0.0 0.0 0:04.12 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd/2828 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper/2828 702 root 20 0 51560 9856 2068 S 0.0 0.2 0:11.57 munin-node 1790 messageb 20 0 29764 256 12 S 0.0 0.0 0:00.00 dbus-daemon 8703 root 20 0 71252 3664 2872 S 0.0 0.1 0:00.02 sshd 8707 alex 20 0 71252 1720 924 S 0.0 0.0 0:00.00 sshd 8708 alex 20 0 21256 4096 1712 S 0.0 0.1 0:00.05 bash 8723 alex 20 0 23120 1612 1188 R 0.0 0.0 0:00.06 top 8734 root 20 0 33032 1160 836 S 0.0 0.0 0:00.00 cron 8738 munin 20 0 10708 1388 1200 S 0.0 0.0 0:00.00 sh 8742 munin 20 0 10716 1392 1192 S 0.0 0.0 0:00.00 munin-cron 15874 root 20 0 71192 5168 1620 S 0.0 0.1 1:20.38 fail2ban-server 15876 root 20 0 22260 908 672 S 0.0 0.0 0:06.64 gam_server 19427 root 30 10 12252 956 792 S 0.0 0.0 0:00.00 rsync 20367 root 20 0 49728 4624 712 S 0.0 0.1 0:19.49 rsyslogd 21949 root 20 0 49892 1236 628 S 0.0 0.0 0:00.81 sshd 22815 root 20 0 20368 496 364 S 0.0 0.0 0:01.23 cron 25118 root 20 0 122m 10m 3932 S 0.0 0.3 0:01.28 /usr/sbin/apach 27076 root 20 0 72564 2024 732 S 0.0 0.0 0:09.70 sendmail-mta 29022 root 20 0 42984 2976 2148 S 0.0 0.1 0:18.03 monit ✎ Comments on 2014-12-25 Emacs Wiki Migration So I’m thinking, why don’t you run some small benchmark (with apache turned off) to see how much of the CPU you’ve got? Or maybe even IO benchmark?– AlexDaniel 2014-12-26 11:00 UTC Yeah. I guess I was hoping that munin does this, effectively. I’ll have to check the plugins currently enabled and check whether they do what I think they do. AlexSchroeder 2014-12-26 20:19 UTC Add Comment # 2014-12-24 Emacs Wiki Migration (Continued from yesterday.) Current status: • Read-only server is up. • Monit restarts Apache practically every hour because of load. • As far as I am concerned, using nginx as a caching proxy isn’t as effective as some people have been suggesting. • I’m noticing monit restarting apache practically every hour. Is some other user on this shared host running an hourly job that borks the system? I must have misconfigured Apache such that the resulting performance dip results in a death spiral. Here’s the munin graph that makes me suspect that I’m not getting all of the CPU I need. The gaps happen when load is so high that munin doesn’t get to run. Yikes! Field Info system CPU time spent by the kernel in system activities user CPU time spent by normal programs and daemons nice CPU time spent by nice(1)d programs idle Idle CPU time iowait CPU time spent waiting for I/O operations to finish when there is nothing else to do. irq CPU time spent handling interrupts softirq CPU time spent handling “batched” interrupts steal The time that a virtual CPU had runnable tasks, but the virtual CPU itself was not running I think what happens is that somebody else on this server is taking up %CPU and this results in Apache processes piling up. Here’s from the log files: Dec 24 09:46:37 localhost monit[29022]: 'apache' trying to restart Dec 24 09:46:37 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2 Dec 24 09:46:38 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 71 ... Dec 24 13:02:55 localhost monit[29022]: 'apache' trying to restart Dec 24 13:02:55 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2 Dec 24 13:03:01 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 37 ... Dec 24 15:55:35 localhost monit[29022]: 'apache' trying to restart Dec 24 15:55:35 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2 Dec 24 15:55:43 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 27 ... Dec 24 16:54:16 localhost monit[29022]: 'apache' trying to restart Dec 24 16:54:16 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2 Dec 24 16:54:25 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 34 ... Dec 24 17:56:52 localhost monit[29022]: 'apache' trying to restart Dec 24 17:56:52 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2 ... Dec 24 21:56:02 localhost monit[29022]: 'apache' trying to restart Dec 24 21:56:02 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2 Dec 24 21:56:03 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 37✎ OK, so how do I make Apache more robust against those spikes? The current settings: # StartServers: initial number of server processes to start # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadLimit: ThreadsPerChild can be changed to this maximum value during a # graceful restart. ThreadLimit can only be changed by stopping # and starting Apache. # ThreadsPerChild: constant number of worker threads in each server process # MaxClients: maximum number of simultaneous client connections # MaxRequestsPerChild: maximum number of requests a server process serves # <IfModule mpm_worker_module> # StartServers 2 # MinSpareThreads 25 # MaxSpareThreads 75 # ThreadLimit 64 # ThreadsPerChild 25 # MaxClients 150 # MaxRequestsPerChild 0 # </IfModule> <IfModule mpm_worker_module> StartServers 2 ServerLimit 3 MinSpareThreads 25 MaxSpareThreads 75 ThreadLimit 100 ThreadsPerChild 100 MaxClients 300 MaxRequestsPerChild 10000 </IfModule>✎ Nic said, I should drop those numbers to better match the cores I have available. Each thread actually running will do IO, so that’s an additional issue. Two cores → no more than 2 servers. When a thread waits for the disk, another thread can run. How many more threads? Not many, because they will also need to use the disk. How about a 90% reduction: 10 threads per child. Looking at this German blog post again. • MaxClients = ServerLimit x ThreadsPerChild. OK. • MinSpareThreads = ThreadsPerChild * 3 to handle spikes – but I think more threads jumping in will lead us to a death spiral. Keeping this lower than 30! • MaxSpareThreads ≥ MinSpareThreads + ThreadsPerChild. <IfModule mpm_worker_module> StartServers 1 ServerLimit 2 ThreadsPerChild 10 MaxClients 20 MinSpareThreads 10 MaxSpareThreads 20 MaxRequestsPerChild 10000 </IfModule>✎ Hours later. It seems to have worked? Load stable, hovering around 2 – this makes sense. All is not perfect, unfortunately. Monit says Apache has 6h uptime. That means somebody restarted Apache during the night. Damn. Add Comment # 2014-12-23 Emacs Wiki Migration (Continued from yesterday.) Current status: 1. Nic Ferrier set up a dedicated machine with nginx to act as a cache for emacswiki.org 2. I’ve switched emacswiki.org to read-only for the moment (since you can only reach the front page anyway) 3. I’ve pointed emacswiki.org to Nic’s machine instead of mine—as soon as this change reaches you, you should be using Nic’s cache 4. Tomorrow, I’ll remove emergency URL rewrite which redirects all wiki requests except for the main page to http://emacswiki.org/load.html Ideally, this would mean that we’ll see a read-only copy of emacswiki.org back up, soon. Once Nic and I figure out how to handle the cookie and the ETag stuff for Emacs Wiki, we can make the wiki editable again. Note to self: (Continued tomorrow.) Add Comment # 2014-12-22 Emacs Wiki Migration (Continued from yesterday.) Well… I considered making this website mod_perl in order to reduce load. This morning it was incredibly high again. When I tried it, however, I ran into problems. Error messages indicated that my script wouldn’t compile. Something about ModPerl::Registry not being available (eventhough it seemed to work for Emacs Wiki), or links on my site pointing to Emacs Wiki. Something was getting mixed up? Does mod_perl not work when used for more than one script in a virtual hosting environment? I rolled back my changes and went to do whatever one does a few years before the year. As I came back, the situation doesn’t seem too bad: I’d prefer load around 2, but at least the site responds, most of the time. Well, service continues to be lousy and I’m practically out of ideas. The next step looks like hacking Oddmuse to try and reduce file accesses and hits. In the mean time, here’s the top of Apache’s server-info. Server Version: Apache/2.2.22 (Debian) mod_ssl/2.2.22 OpenSSL/1.0.1e mod_perl/2.0.7 Perl/v5.14.2 Server Built: Jul 24 2014 15:34:03 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Current Time: Tuesday, 23-Dec-2014 00:25:10 CET Restart Time: Tuesday, 23-Dec-2014 00:23:07 CET Parent Server Generation: 0 Server uptime: 2 minutes 3 seconds Total accesses: 482 - Total Traffic: 2.2 MB CPU Usage: u83.27 s1.06 cu18.66 cs0 - 83.7% CPU load 3.92 requests/sec - 18.0 kB/second - 4705 B/request 9 requests currently being processed, 91 idle workers W________________K_____________R_W_______K_K_W__________________ ________________R__________________K............................ ................................................W............... ................................................................ ............................................ Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process✎ That doesn’t really look very busy. So perhaps file access is the new bottle neck? If you know some Perl, and if you know mod_perl, and you feel like you could help, please contact me! The current setup uses a Linux OpenVZ / SolusVM VPS in Switzerland with 4096 MB RAM and 2 vCores (shared Dual Intel Xeon), 500 GB HDD (practically unused) Speicherplatz, with Debian Wheezy. The Apache setup I mentioned previously: <IfModule mpm_worker_module> StartServers 2 ServerLimit 3 MinSpareThreads 25 MaxSpareThreads 75 ThreadLimit 100 ThreadsPerChild 100 MaxClients 300 MaxRequestsPerChild 10000 </IfModule>✎ The virtual server setup: <VirtualHost *:80> ServerAdmin alex@emacswiki.org <Directory /> Options None AllowOverride None Order Deny,Allow Deny from all </Directory> ServerName www.emacswiki.org ServerAlias emacswiki.org DocumentRoot /home/alex/emacswiki.org <Directory /home/alex/emacswiki.org> Options ExecCGI Includes Indexes SymLinksIfOwnerMatch AddHandler cgi-script .pl AllowOverride All Order Allow,Deny Allow from all </Directory> <Files emacs.pl> Options ExecCGI SymLinksIfOwnerMatch SetHandler perl-script PerlHandler ModPerl::Registry </Files> RewriteLog "/home/alex/logs/emacswiki.org/rewrite.log" RewriteLogLevel 0 CustomLog /home/alex/logs/emacswiki.org/access.log combined ErrorLog /home/alex/logs/emacswiki.org/error.log # LogLevel debug CustomLog /var/log/apache2/access.log vhost_combined ServerSignature On </VirtualHost>✎ I run monit and munin for monitoring. And now we’re getting into Oddmuse details. If you need more information in order to setup up a test environment, I’ll be happy to provide anything you need. #! /usr/bin/perl package OddMuse; use CGI;$q = new CGI;
$RunCGI = 0;$UseConfig = 0; # no config!
$DataDir = '/home/alex/emacswiki'; if (not$config_loaded) {
do '/home/alex/emacswiki.org/current.pl';
do '/home/alex/emacswiki/modules/concat.pl';
do '/home/alex/emacswiki/config';
$config_loaded = 1; } # make sure traces of other language configs disappear %Translate=();$HomePage    = "SiteMap";
$RCName = 'RecentChanges'; @UserGotoBarPages = ($HomePage, 'Search',
'ElispArea', 'HowTo', 'Glossary',
$RCName, 'News', 'Problems', 'Suggestions');$EditNote = '<p>Please make sure you contribute only your own work. IP numbers are made available via the page history. If you want to keep it a secret, you need to use <a href="http://www.torproject.org/">Tor</a>. See <a href="/emacs/TextFormattingRules">TextFormattingRules</a> for how to format text. See <a href="/emacs/StyleGuide">StyleGuide</a> for the suggested writing style on this wiki.</p>';

# language specific files should only set variables, never redefine subroutines
$ConfigFile = "$DataDir/config-en";
# my $requested_language = "da, en-gb;q=0.8, en;q=0.7"; my$requested_language = $q->http('Accept-language'); my @languages = split(/ *, */,$requested_language);
my %Lang = ();
foreach $_ (@languages) { my$qual = 1;
$qual =$1 if (/q=([0-9.]+)/);
$Lang{$qual} = $1 if (/^([-a-z]+[a-z])/); } my @prefs = sort {$b <=> $a } keys %Lang; # my$html = "input: $requested_language" . "\nResult: " . join(', ', map { "$_ ($Lang{$_})" } @prefs);
# print header, start_html, pre($html), end_html; exit; foreach$_ (@prefs) {
if (-r "$DataDir/config-$Lang{$_}") {$ConfigFile = "$DataDir/config-$Lang{$_}"; last; } } do$ConfigFile;
# Do the wiki
DoWikiRequest();✎

(Continued tomorrow.)

Comments on 2014-12-22 Emacs Wiki Migration

Completely unrelated, but if the majority of the traffic is anonymous, I would suggest installing Varnish

– Anonymous 2014-12-23 19:06 UTC

I’m not sure. In my .htaccess, I already provide for long browser-side caching of static files:

ExpiresActive On
ExpiresDefault "access plus 1 hours"
ExpiresByType image/png "access plus 3 days"
ExpiresByType image/jpeg "access plus 3 days"
ExpiresByType image/svg+xml "access plus 3 days"
ExpiresByType text/css "access plus 3 days"

And the wiki itself uses ETags the way they were meant to be used, I think. See the wiki page on caching, if you’re interested. It’s true that returning the 304 code requires the wiki code to run, but I’m still saving on all the rendering. It’s worth a try, I guess. A short skimming of an intro page I found made me think that using it for wiki pages is going to involve some more work. Do you have some detailed instructions somewhere, or can you help in configuring Varnish for Oddmuse? The Varnish wiki page on ETags looks intimidating.

– Alex Schroeder 2014-12-23 20:33 UTC

# 2013-12-01 Anonymizing the Oddmuse log files

I’ve just implemented a new non-optional Oddmuse feature. I’m removing all hostnames and IP numbers of older log entries. The log entries older than 90 days are stored in a different log file in order to speed up the generation of RecentChanges. During maintenance, these log entries are copied from one file to the other and I’m now taking advantage of this copying to remove the hostname or IP number.

Basically I find that as a person, I dislike invasions of privacy and I feel that in some small form, software engineers are inviting it because often it’s easier to do. We often model things to never forget, e.g. version control.

One of the important pages on Meatball was ForgiveAndForget. Forgetting is human.

At the same time, with Snowden and the NSA, I feel that as a hoster I’m more comfortable if I cannot provide the logs an agency is looking for.

Furthermore, I’ve had a very small number of emails from users asking me to remove their hostnames from the log files because they had accidentally edited the wiki from work. Pages containing their hostname will eventually be deleted but log entries were not. Now they’re anonymized and people can feel safer knowing that the traces will eventually disappear again.

The idea is that you would only need hostnames or IP numbers to fight spam and vandalism: Add regular expressions matching either hostname or IP number of spammers or vandals to your list of banned hosts and prevent the attack from continuing. After a few days, however, this information is no longer required. In this day and age of privacy invasion, I think software should take a pro-active stance. The log entries must be anonymized.

The existing log file for the older entries is not changed. If you want to do the right thing, there’s a script called anonymize.pl in the contrib directory to do just that.

Just call it in your data directory. Example:

alex@psithyrus:~/oddmuse\$ perl ~/src/oddmuse/contrib/anonymize.pl
Wrote anonymized 'oldrc.log'.
Saved a backup as 'oldrc.log~'

See Oddmuse:Upgrading Issues for a more technical explanation of what’s going on.

Tags: