Emacs

Emacs Feed The pages where I write about Emacs. It doesn’t happen very often. I think you’re probably best off following Planet Emacsen.

2014-12-30 Emac Wiki Migration

OK, I guess Nic and I have a plan. We’ve started writing a backlog for the Emacs Wiki Migration. Hopefully this will make the current wiki easier to cache.

Tags: RSS RSS RSS RSS RSS RSS RSS

Add Comment

2014-12-28 Emacs Wiki Migration

Gaah! :(

So, I still think this is a problem with the hosting provider. Here’s what I think is evidence:

https://farm8.staticflickr.com/7575/15509223034_8ee49a110d_o.png

What could be causing this regular forks if not other virtual machines hogging CPU at regular intervals, or other virtual machines hogging disk access, causing CGI scripts to pile up? I wonder…

Things I care about: Diagnosing Disk I/O issues: swapping, high IO wait, congestion.

Let’s see what iotop -ao says right now:

Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                    
11774 be/4 www-data     36.00 K     20.00 K  0.00 %  0.02 % [/usr/sbin/apach]
11801 be/4 www-data      0.00 B     12.00 K  0.00 %  0.00 % apache2 -k start
11810 be/4 www-data      0.00 B      4.00 K  0.00 %  0.00 % apache2 -k start 
29022 be/4 root          0.00 B     32.00 K  0.00 %  0.00 % monit -c /etc/monit/monitrc
20368 be/4 root          0.00 B     60.00 K  0.00 %  0.00 % rsyslogd -c5

I don’t think we’re using swap at all (munin shows us using less than 2G with 4G available). But who knows? I used top, pressed f to show the list of fields, and I used d (display) and s (sort) on SWAP.

top - 22:14:00 up 7 days, 21:11,  2 users,  load average: 0.51, 0.31, 0.22
Tasks:  24 total,   1 running,  23 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.6 us,  0.6 sy,  0.0 ni, 95.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   4194304 total,   894976 used,  3299328 free,        0 buffers
KiB Swap:  1048576 total,     5524 used,  1043052 free,   785880 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND              SWAP 
15874 root      20   0 71192 5224 1620 S   0.0  0.1   2:11.36 fail2ban-server      2296 
27076 root      20   0 72564 1092  708 S   0.0  0.0   0:16.47 sendmail-mta         1384 
21949 root      20   0 49892  684  568 S   0.0  0.0   0:02.23 sshd                  492 
29022 root      20   0 42984 2688 2136 S   0.0  0.1   0:48.28 monit                 284 
20367 root      20   0 49860 4504  712 S   0.0  0.1   0:24.56 rsyslogd              236 
19427 root      30  10 12252  788  784 S   0.0  0.0   0:00.00 rsync                 160 
15876 root      20   0 22260  896  672 S   0.0  0.0   0:08.86 gam_server            136 
22815 root      20   0 20368  496  364 S   0.0  0.0   0:02.03 cron                  120 

I don’t think that’s suspicious. Those tasks probably ended up in swap because they have been running for so long?

I installed the apache_vhosts plugin for munin. I think there will be no surprises with emacswiki.org taking the brunt with a factor of 10:1. Right now, however, it seems to be misconfigured somehow, not recording any emacswiki.org traffic and missing most of the other traffic. Weird!

Ah, apache_vhosts munin plugin problem fixed. All my sites were using a CustomLog except for alexschroeder.ch:80 and campaignwiki.org:80—these just redirect to the HTTPS version of the site. Thus, these sites were being counted by the apache_vhosts plugin. But that is now fixed, and we have a graph!

https://farm8.staticflickr.com/7527/15960447830_a102ae6df7_o.png

https://farm9.staticflickr.com/8667/15528022193_78df3fc37d_o.png

I’m not quite sure how to compare this with the following log file analysis where I simply group access by site:

root@kallobombus:/var/log/apache2# perl /home/alex/bin/site-comparison < access.log
                          Site       hits bandw. hits% interv. status code distrib.
             www.emacswiki.org     104499     6K  47%    2.1s  301 (29%), 501 (26%), 200 (21%
          www.alexschroeder.ch      67396    31K  30%    3.2s  200 (94%), 304 (2%), 301 (1%),
         www.communitywiki.org      24574    17K  11%    8.9s  200 (80%), 301 (7%), 404 (3%),
          www.campaignwiki.org      13902    13K   6%   15.6s  200 (74%), 404 (17%), 302 (4%)
               www.oddmuse.org       9498    23K   4%   22.9s  200 (66%), 404 (14%), 301 (12%
       www.arabisch-lernen.org        845     7K   0%  256.4s  200 (94%), 404 (3%), 403 (1%),
         www.orientalisch.info        404     4K   0%  536.5s  200 (80%), 301 (14%), 404 (2%)

Ah, of course. The access.log is can be up to seven days. Thus:

root@kallobombus:/var/log/apache2# tail -n 10000 access.log | /home/alex/bin/site-comparison 
                          Site       hits bandw. hits% interv. status code distrib.
          www.alexschroeder.ch       6430    24K  64%    1.9s  200 (88%), 301 (8%), 404 (1%),
             www.emacswiki.org       1187    12K  11%   10.4s  301 (38%), 200 (36%), 404 (19%
         www.communitywiki.org       1150    14K  11%   10.7s  200 (79%), 301 (7%), 404 (5%),
          www.campaignwiki.org        604     9K   6%   20.3s  200 (58%), 301 (22%), 404 (12%
               www.oddmuse.org        415    18K   4%   29.6s  200 (64%), 404 (18%), 301 (10%
       www.arabisch-lernen.org        202     5K   2%   37.4s  404 (90%), 200 (9%), 501 (0%)
         www.orientalisch.info         12     4K   0% 1010.2s  200 (91%), 304 (8%)

Tags: RSS RSS RSS RSS RSS RSS RSS

Add Comment

2014-12-27 Emacs Wiki Migration

Based on some information I got from Charlie McMackin—who had set up a virtual machine and got Emacs Wiki to run on it—I switched Apache from “worker” to “prefork” and I’m now using the following config:

<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       1
    MaxSpareServers       2
    MaxClients		  5
    MaxRequestsPerChild 10000
</IfModule>

I checked http://localhost/server-info and can confirm that we’re running Prefork:

Server Settings

Server Version: Apache/2.2.22 (Debian) mod_ssl/2.2.22 OpenSSL/1.0.1e mod_perl/
    2.0.7 Perl/v5.14.2
Server Built: Jul 24 2014 15:34:00
Server loaded APR Version: 1.4.6
Compiled with APR Version: 1.4.6
Server loaded APU Version: 1.4.1
Compiled with APU Version: 1.4.1
Module Magic Number: 20051115:30
Hostname/port: alexschroeder.ch:443
Timeouts: connection: 300    keep-alive: 5
MPM Name: Prefork
MPM Information: Max Daemons: 5 Threaded: no Forked: yes
Server Architecture: 64-bit

Tags: RSS RSS RSS RSS RSS RSS RSS

Add Comment

2014-12-26 Emacs Wiki Migration

I’m not happy. Today, multiple Apache restarts. Grepping syslog for restarts:

Dec 26 07:38:32 localhost monit[29022]: 'apache' trying to restart
Dec 26 11:38:53 localhost monit[29022]: 'apache' trying to restart
Dec 26 14:47:16 localhost monit[29022]: 'apache' trying to restart
Dec 26 16:49:46 localhost monit[29022]: 'apache' trying to restart
Dec 26 17:26:06 localhost monit[29022]: 'apache' trying to restart
Dec 26 18:06:29 localhost monit[29022]: 'apache' trying to restart
Dec 26 19:48:54 localhost monit[29022]: 'apache' trying to restart

https://farm8.staticflickr.com/7566/15494392223_95ee7bf147_o.png

I’m going to further reduce the number of threads.

<IfModule mpm_worker_module>
    StartServers          1
    ServerLimit           2
    ThreadsPerChild       5
    MaxClients           10
    MinSpareThreads      10
    MaxSpareThreads      20
    MaxRequestsPerChild  10000
</IfModule>

I also installed lsof and wrote a little plugin for munin. What I really want to see is the number of open files, or something like that. I remember checking that when we had load problems on Mirabel without a matching number of processes actually doing anything. Just in case the processes are all waiting for file access.

I’m not sure how well this is going to work, but this is my first stab at the files plugin. 😊

#!/bin/sh
# 
# Plugin to monitor the number of open files in the system.
#
# Parameters:
# 	
# 	config   (required)
# 	autoconf (optional - used by munin-config)
#
# Magic markers (Used by munin-config and some installation scripts.
# Optional):
#
#%# family=contrib
#%# capabilities=autoconf

# PROC=${0##*/files_}
PORTS="80 443"

if [ "$1" = "autoconf" ]; then
	if [ -x /usr/bin/lsof ]; then
		echo yes
		exit 0
	else
		echo "no (no lsof)"
		exit 0
	fi
fi

if [ "$1" = "config" ]; then
	echo graph_title "open files"
	echo graph_vlabel "number of files"
	echo graph_info "This graph shows the number of open files by port (80 is http, 443 is https)."
	echo graph_category "apache"
	for P in $PORTS; do
	    echo "$P.label port $P"
	done
	exit 0
fi

for P in $PORTS; do 
    N=$(lsof -i :$P | tail -n +2 | wc -l)
    echo "$P.value $N"
done

All I know is that load is climbing right now. Close to 6. Time for bed! Grrrrr.

A few hours later, stats seem to be worse:

Dec 26 22:47:16 localhost monit[29022]: 'apache' trying to restart
Dec 26 23:23:34 localhost monit[29022]: 'apache' trying to restart
Dec 26 23:58:01 localhost monit[29022]: 'apache' trying to restart
Dec 27 00:10:19 localhost monit[29022]: 'apache' trying to restart
Dec 27 00:38:40 localhost monit[29022]: 'apache' trying to restart
Dec 27 01:06:58 localhost monit[29022]: 'apache' trying to restart
Dec 27 01:17:15 localhost monit[29022]: 'apache' trying to restart
Dec 27 03:51:40 localhost monit[29022]: 'apache' trying to restart
Dec 27 04:23:59 localhost monit[29022]: 'apache' trying to restart
Dec 27 04:42:21 localhost monit[29022]: 'apache' trying to restart
Dec 27 05:30:47 localhost monit[29022]: 'apache' trying to restart
Dec 27 06:59:09 localhost monit[29022]: 'apache' trying to restart
Dec 27 08:05:33 localhost monit[29022]: 'apache' trying to restart

Let’s hope that things will improve with prefork!

Tags: RSS RSS RSS RSS RSS RSS RSS

Add Comment

2014-12-25 Emacs Wiki Migration

I’ve been thinking about special-casing some caching: a mod_rewrite rule that says:

  • if the URL has the form /emacs/page
  • and there is no cookie (ie. bots, just readers)
  • and there is an if-none-match header (from nginx, I presume?)
  • and its value matches the “default”
  • return 304 without invoking the wiki

So, what’s the “default”?

sub DoBrowseRequest {
  ...
  print $q->header(-status=>'304 NOT MODIFIED') and return if PageFresh(); # return value is ignored
  ...
}

sub PageFresh { # pages can depend on other pages (ie. last update), admin status, and css
  return 1 if $q->http('HTTP_IF_NONE_MATCH') and GetParam('cache', $UseCache) >= 2
    and $q->http('HTTP_IF_NONE_MATCH') eq PageEtag();
}

sub PageEtag {
  my ($changed, $visible, %params) = CookieData();
  return UrlEncode(join($FS, $LastUpdate||$Now, sort(values %params))); # no CTL in field values
}

And what about $LastUpdate?

  my $ts = (stat($IndexFile))[9]; # always stat for multiple server processes
  ...
  $LastUpdate = $ts;

So, I think that the “default” ETag should simply be the mtime of the pageidx file. We’d have to use the RewriteMap directive with an External Rewriting Program.

Yikes! I’m not sure this is really faster… I guess it would be faster if the wiki itself was no longer using mod_perl (for whatever reasons). The external rewriting program would comparing the ETag header with the pageidx mtime and return a special value which would in turn be rewritten by a second rule to “redirect” to a 304 NOT MODIFIED response…

Something to consider for later.

As I’m writing this, I’m noticing that the server is slow. Check the current time, right, the five minutes before the full hour are a problem. Load reaching 6.

top - 17:59:18 up 4 days, 16:57,  1 user,  load average: 5.94, 5.41, 4.04
Tasks:  20 total,   1 running,  19 sleeping,   0 stopped,   0 zombie
%Cpu(s): 98.7 us,  1.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   4194304 total,  1688940 used,  2505364 free,        0 buffers
KiB Swap:  1048576 total,     4416 used,  1044160 free,  1331364 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
 2635 www-data  20   0  470m 122m 4704 S 101.9  3.0  42:43.65 /usr/sbin/apach   
 4902 www-data  20   0  434m  91m 5360 S  72.6  2.2  21:12.41 /usr/sbin/apach   
    1 root      20   0 10608  196   60 S   0.0  0.0   0:04.12 init              
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kthreadd/2828     
    3 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khelper/2828      
  702 root      20   0 51560 9856 2068 S   0.0  0.2   0:11.56 munin-node        
 1790 messageb  20   0 29764  256   12 S   0.0  0.0   0:00.00 dbus-daemon       
 8703 root      20   0 71252 3664 2872 S   0.0  0.1   0:00.02 sshd              
 8707 alex      20   0 71252 1720  924 S   0.0  0.0   0:00.00 sshd              
 8708 alex      20   0 21256 4096 1712 S   0.0  0.1   0:00.05 bash              
 8723 alex      20   0 23120 1612 1188 R   0.0  0.0   0:00.00 top               
15874 root      20   0 71192 5168 1620 S   0.0  0.1   1:20.36 fail2ban-server   
15876 root      20   0 22260  908  672 S   0.0  0.0   0:06.63 gam_server        
19427 root      30  10 12252  956  792 S   0.0  0.0   0:00.00 rsync             
20367 root      20   0 49728 4624  712 S   0.0  0.1   0:19.49 rsyslogd          
21949 root      20   0 49892 1236  628 S   0.0  0.0   0:00.81 sshd              
22815 root      20   0 20368  496  364 S   0.0  0.0   0:01.23 cron              
25118 root      20   0  122m  10m 3932 S   0.0  0.3   0:01.28 /usr/sbin/apach   
27076 root      20   0 72564 2024  732 S   0.0  0.0   0:09.69 sendmail-mta      
29022 root      20   0 42984 2976 2148 S   0.0  0.1   0:18.03 monit             

A few minutes later load reached 7. And my processes aren’t busy.

top - 18:01:45 up 4 days, 16:59,  1 user,  load average: 6.65, 5.91, 4.43
Tasks:  24 total,   2 running,  22 sleeping,   0 stopped,   0 zombie
%Cpu(s): 97.2 us,  0.9 sy,  1.7 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem:   4194304 total,  1726856 used,  2467448 free,        0 buffers
KiB Swap:  1048576 total,     4416 used,  1044160 free,  1331376 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
 4902 www-data  20   0  437m 108m 5384 S  61.6  2.7  22:46.97 /usr/sbin/apach   
 9105 munin     30  10  140m  23m 5936 R   3.3  0.6   0:02.26 munin-graph       
    1 root      20   0 10608  196   60 S   0.0  0.0   0:04.12 init              
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kthreadd/2828     
    3 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khelper/2828      
  702 root      20   0 51560 9856 2068 S   0.0  0.2   0:11.57 munin-node        
 1790 messageb  20   0 29764  256   12 S   0.0  0.0   0:00.00 dbus-daemon       
 8703 root      20   0 71252 3664 2872 S   0.0  0.1   0:00.02 sshd              
 8707 alex      20   0 71252 1720  924 S   0.0  0.0   0:00.00 sshd              
 8708 alex      20   0 21256 4096 1712 S   0.0  0.1   0:00.05 bash              
 8723 alex      20   0 23120 1612 1188 R   0.0  0.0   0:00.06 top               
 8734 root      20   0 33032 1160  836 S   0.0  0.0   0:00.00 cron              
 8738 munin     20   0 10708 1388 1200 S   0.0  0.0   0:00.00 sh                
 8742 munin     20   0 10716 1392 1192 S   0.0  0.0   0:00.00 munin-cron        
15874 root      20   0 71192 5168 1620 S   0.0  0.1   1:20.38 fail2ban-server   
15876 root      20   0 22260  908  672 S   0.0  0.0   0:06.64 gam_server        
19427 root      30  10 12252  956  792 S   0.0  0.0   0:00.00 rsync             
20367 root      20   0 49728 4624  712 S   0.0  0.1   0:19.49 rsyslogd          
21949 root      20   0 49892 1236  628 S   0.0  0.0   0:00.81 sshd              
22815 root      20   0 20368  496  364 S   0.0  0.0   0:01.23 cron              
25118 root      20   0  122m  10m 3932 S   0.0  0.3   0:01.28 /usr/sbin/apach   
27076 root      20   0 72564 2024  732 S   0.0  0.0   0:09.70 sendmail-mta      
29022 root      20   0 42984 2976 2148 S   0.0  0.1   0:18.03 monit             

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-25 Emacs Wiki Migration

So I’m thinking, why don’t you run some small benchmark (with apache turned off) to see how much of the CPU you’ve got? Or maybe even IO benchmark?– AlexDaniel 2014-12-26 11:00 UTC



AlexSchroeder
Yeah. I guess I was hoping that munin does this, effectively. I’ll have to check the plugins currently enabled and check whether they do what I think they do.

AlexSchroeder 2014-12-26 20:19 UTC

Add Comment

2014-12-24 Emacs Wiki Migration

(Continued from yesterday.)

Current status:

  • Read-only server is up.
  • Monit restarts Apache practically every hour because of load.
  • As far as I am concerned, using nginx as a caching proxy isn’t as effective as some people have been suggesting.
  • I’m noticing monit restarting apache practically every hour. Is some other user on this shared host running an hourly job that borks the system? I must have misconfigured Apache such that the resulting performance dip results in a death spiral.

Here’s the munin graph that makes me suspect that I’m not getting all of the CPU I need. The gaps happen when load is so high that munin doesn’t get to run. Yikes!

https://farm8.staticflickr.com/7537/15911905077_b82b4056a0_o.png

Field
Info
system
CPU time spent by the kernel in system activities
user
CPU time spent by normal programs and daemons
nice
CPU time spent by nice(1)d programs
idle
Idle CPU time
iowait
CPU time spent waiting for I/O operations to finish when there is nothing else to do.
irq
CPU time spent handling interrupts
softirq
CPU time spent handling “batched” interrupts
steal
The time that a virtual CPU had runnable tasks, but the virtual CPU itself was not running

I think what happens is that somebody else on this server is taking up %CPU and this results in Apache processes piling up.

Here’s from the log files:

Dec 24 09:46:37 localhost monit[29022]: 'apache' trying to restart
Dec 24 09:46:37 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 09:46:38 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 71
...
Dec 24 13:02:55 localhost monit[29022]: 'apache' trying to restart
Dec 24 13:02:55 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 13:03:01 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 37
...
Dec 24 15:55:35 localhost monit[29022]: 'apache' trying to restart
Dec 24 15:55:35 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 15:55:43 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 27
...
Dec 24 16:54:16 localhost monit[29022]: 'apache' trying to restart
Dec 24 16:54:16 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 16:54:25 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 34
...
Dec 24 17:56:52 localhost monit[29022]: 'apache' trying to restart
Dec 24 17:56:52 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
...
Dec 24 21:56:02 localhost monit[29022]: 'apache' trying to restart
Dec 24 21:56:02 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 21:56:03 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 37

OK, so how do I make Apache more robust against those spikes?

The current settings:

# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadLimit: ThreadsPerChild can be changed to this maximum value during a
#              graceful restart. ThreadLimit can only be changed by stopping
#              and starting Apache.
# ThreadsPerChild: constant number of worker threads in each server process
# MaxClients: maximum number of simultaneous client connections
# MaxRequestsPerChild: maximum number of requests a server process serves
# <IfModule mpm_worker_module>
#     StartServers          2
#     MinSpareThreads      25
#     MaxSpareThreads      75 
#     ThreadLimit          64
#     ThreadsPerChild      25
#     MaxClients          150
#     MaxRequestsPerChild   0
# </IfModule>
<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit           3
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit         100
    ThreadsPerChild     100
    MaxClients          300
    MaxRequestsPerChild  10000
</IfModule>

Nic said, I should drop those numbers to better match the cores I have available. Each thread actually running will do IO, so that’s an additional issue.

Two cores → no more than 2 servers. When a thread waits for the disk, another thread can run. How many more threads? Not many, because they will also need to use the disk. How about a 90% reduction: 10 threads per child. Looking at this German blog post again.

  • MaxClients = ServerLimit x ThreadsPerChild. OK.
  • MinSpareThreads = ThreadsPerChild * 3 to handle spikes – but I think more threads jumping in will lead us to a death spiral. Keeping this lower than 30!
  • MaxSpareThreads ≥ MinSpareThreads + ThreadsPerChild.
<IfModule mpm_worker_module>
    StartServers          1
    ServerLimit           2
    ThreadsPerChild      10
    MaxClients           20
    MinSpareThreads      10
    MaxSpareThreads      20
    MaxRequestsPerChild  10000
</IfModule>

Hours later. It seems to have worked? Load stable, hovering around 2 – this makes sense.

https://farm8.staticflickr.com/7532/16100535652_84b896b86a_o.png

All is not perfect, unfortunately. Monit says Apache has 6h uptime. That means somebody restarted Apache during the night. Damn.

Tags: RSS RSS RSS RSS RSS RSS RSS

Add Comment

2014-12-23 Emacs Wiki Migration

(Continued from yesterday.)

Current status:

  1. Nic Ferrier set up a dedicated machine with nginx to act as a cache for emacswiki.org
  2. I’ve switched emacswiki.org to read-only for the moment (since you can only reach the front page anyway)
  3. I’ve pointed emacswiki.org to Nic’s machine instead of mine—as soon as this change reaches you, you should be using Nic’s cache
  4. Tomorrow, I’ll remove emergency URL rewrite which redirects all wiki requests except for the main page to http://emacswiki.org/load.html

Ideally, this would mean that we’ll see a read-only copy of emacswiki.org back up, soon. Once Nic and I figure out how to handle the cookie and the ETag stuff for Emacs Wiki, we can make the wiki editable again.

Note to self:

(Continued tomorrow.)

Tags: RSS RSS RSS RSS RSS RSS RSS

Add Comment

2014-12-22 Emacs Wiki Migration

(Continued from yesterday.)

Well… I considered making this website mod_perl in order to reduce load. This morning it was incredibly high again. When I tried it, however, I ran into problems. Error messages indicated that my script wouldn’t compile. Something about ModPerl::Registry not being available (eventhough it seemed to work for Emacs Wiki), or links on my site pointing to Emacs Wiki. Something was getting mixed up? Does mod_perl not work when used for more than one script in a virtual hosting environment? I rolled back my changes and went to do whatever one does a few years before the year. As I came back, the situation doesn’t seem too bad:

https://farm8.staticflickr.com/7483/15459696234_a9351a5831_o.png

I’d prefer load around 2, but at least the site responds, most of the time.

Well, service continues to be lousy and I’m practically out of ideas. The next step looks like hacking Oddmuse to try and reduce file accesses and hits.

In the mean time, here’s the top of Apache’s server-info.

Server Version: Apache/2.2.22 (Debian) mod_ssl/2.2.22 OpenSSL/1.0.1e mod_perl/2.0.7 Perl/v5.14.2
Server Built: Jul 24 2014 15:34:03

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Current Time: Tuesday, 23-Dec-2014 00:25:10 CET
Restart Time: Tuesday, 23-Dec-2014 00:23:07 CET
Parent Server Generation: 0
Server uptime: 2 minutes 3 seconds
Total accesses: 482 - Total Traffic: 2.2 MB
CPU Usage: u83.27 s1.06 cu18.66 cs0 - 83.7% CPU load
3.92 requests/sec - 18.0 kB/second - 4705 B/request
9 requests currently being processed, 91 idle workers

W________________K_____________R_W_______K_K_W__________________
________________R__________________K............................
................................................W...............
................................................................
............................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

That doesn’t really look very busy. So perhaps file access is the new bottle neck?

If you know some Perl, and if you know mod_perl, and you feel like you could help, please contact me!

The current setup uses a Linux OpenVZ / SolusVM VPS in Switzerland with 4096 MB RAM and 2 vCores (shared Dual Intel Xeon), 500 GB HDD (practically unused) Speicherplatz, with Debian Wheezy.

The Apache setup I mentioned previously:

<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit           3
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit         100
    ThreadsPerChild     100
    MaxClients          300
    MaxRequestsPerChild  10000
</IfModule>

The virtual server setup:

<VirtualHost *:80>
    ServerAdmin alex@emacswiki.org
    <Directory />
        Options None
        AllowOverride None
        Order Deny,Allow
        Deny from all
    </Directory>
    ServerName www.emacswiki.org
    ServerAlias emacswiki.org
    DocumentRoot /home/alex/emacswiki.org
    <Directory /home/alex/emacswiki.org>
        Options ExecCGI Includes Indexes SymLinksIfOwnerMatch
	AddHandler cgi-script .pl
        AllowOverride All
        Order Allow,Deny
        Allow from all
    </Directory>
    <Files emacs.pl>
       Options ExecCGI SymLinksIfOwnerMatch
       SetHandler perl-script
       PerlHandler ModPerl::Registry
    </Files>
    RewriteLog "/home/alex/logs/emacswiki.org/rewrite.log"
    RewriteLogLevel 0
    CustomLog /home/alex/logs/emacswiki.org/access.log combined
    ErrorLog  /home/alex/logs/emacswiki.org/error.log
    # LogLevel debug
    CustomLog /var/log/apache2/access.log vhost_combined
    ServerSignature On
</VirtualHost>

I run monit and munin for monitoring.

And now we’re getting into Oddmuse details. If you need more information in order to setup up a test environment, I’ll be happy to provide anything you need.

#! /usr/bin/perl
package OddMuse;
use CGI;
$q = new CGI;
$RunCGI = 0;
$UseConfig = 0; # no config!
$DataDir = '/home/alex/emacswiki';
if (not $config_loaded) {
  do '/home/alex/emacswiki.org/current.pl';
  do '/home/alex/emacswiki/modules/concat.pl';
  do '/home/alex/emacswiki/config';
  $config_loaded = 1;
}

# make sure traces of other language configs disappear
%Translate=();
$HomePage    = "SiteMap";
$RCName      = 'RecentChanges';
@UserGotoBarPages = ($HomePage, 'Search',
		     'ElispArea', 'HowTo', 'Glossary',
		     $RCName, 'News', 'Problems', 'Suggestions');
$EditNote = '<p>Please make sure you contribute only your own work. IP numbers are made available via the page history. If you want to keep it a secret, you need to use <a href="http://www.torproject.org/">Tor</a>. See <a href="/emacs/TextFormattingRules">TextFormattingRules</a> for how to format text. See <a href="/emacs/StyleGuide">StyleGuide</a> for the suggested writing style on this wiki.</p>';

# language specific files should only set variables, never redefine subroutines
$ConfigFile = "$DataDir/config-en";
# my $requested_language = "da, en-gb;q=0.8, en;q=0.7";
my $requested_language = $q->http('Accept-language');
my @languages = split(/ *, */, $requested_language);
my %Lang = ();
foreach $_ (@languages) {
  my $qual = 1;
  $qual = $1 if (/q=([0-9.]+)/);
  $Lang{$qual} = $1 if (/^([-a-z]+[a-z])/);
}
my @prefs = sort { $b <=> $a } keys %Lang;
# my $html = "input: $requested_language" . "\nResult: " . join(', ', map { "$_ ($Lang{$_})" } @prefs);
# print header, start_html, pre($html), end_html; exit;
foreach $_ (@prefs) {
  if (-r "$DataDir/config-$Lang{$_}") {
    $ConfigFile = "$DataDir/config-$Lang{$_}";
    last;
  }
}
do $ConfigFile;
# Do the wiki
DoWikiRequest();

(Continued tomorrow.)

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-22 Emacs Wiki Migration

Completely unrelated, but if the majority of the traffic is anonymous, I would suggest installing Varnish

– Anonymous 2014-12-23 19:06 UTC



Alex Schroeder
I’m not sure. In my .htaccess, I already provide for long browser-side caching of static files:

ExpiresActive On
ExpiresDefault "access plus 1 hours"
ExpiresByType image/png "access plus 3 days"
ExpiresByType image/jpeg "access plus 3 days"
ExpiresByType image/svg+xml "access plus 3 days"
ExpiresByType text/css "access plus 3 days"

And the wiki itself uses ETags the way they were meant to be used, I think. See the wiki page on caching, if you’re interested. It’s true that returning the 304 code requires the wiki code to run, but I’m still saving on all the rendering. It’s worth a try, I guess. A short skimming of an intro page I found made me think that using it for wiki pages is going to involve some more work. Do you have some detailed instructions somewhere, or can you help in configuring Varnish for Oddmuse? The Varnish wiki page on ETags looks intimidating.

– Alex Schroeder 2014-12-23 20:33 UTC

Add Comment

2014-12-19 Emacs Wiki Migration

(Continued from yesterday.)

Right now I’m trying to get a mod_perl setup going. Unfortunately, it still isn’t working. For the moment, I have disabled most of the rewrite rules, such that the main page loads but everything else does not.

Mood: frustrated.

There are many things that still don’t work:

  1. git integration (some issues regarding a cleanup that was planned)
  2. backup (due to permission issues which I haven’t been able to look into)
  3. frequent outages (the page comes up empty) – I’m not sure what is happening, here: processes crashing? taking too long?
  4. broken reloads after editing a page (but trying again solves the problem)
  5. we lost foreign language support (at the top it will always say SiteMap even if your browser preferences are for a different language)
  6. various mod_rewrite rules that I had to disable (not sure whether I want them back!)

And there are probably many more. I haven’t looked at the Elisp Area, RSS feeds, all the cron jobs, etc. Yikes!

memory today CPU today

Hm, the situation seems to be much better right now. I think the most important thing I changed was that I prevent monit from killing Apache.

Syslog had been showing this pattern a lot:

Dec 20 08:26:34 kallobombus monit[27415]: 'apache' total mem amount of 1267108kB matches resource limit [total mem amount>512001kB]
Dec 20 08:26:34 kallobombus monit[27415]: 'apache' trying to restart
Dec 20 08:26:34 kallobombus monit[27415]: 'apache' stop: /etc/init.d/apache2
Dec 20 08:26:42 kallobombus monit[27415]: 'apache' start: /etc/init.d/apache2
...
Dec 20 08:28:42 kallobombus monit[27415]: 'apache' total mem amount of 571704kB matches resource limit [total mem amount>512001kB]
Dec 20 08:28:42 kallobombus monit[27415]: 'apache' trying to restart
Dec 20 08:28:42 kallobombus monit[27415]: 'apache' stop: /etc/init.d/apache2
Dec 20 08:28:54 kallobombus monit[27415]: 'apache' start: /etc/init.d/apache2

Basically this was telling me that Apache wasn’t running out of memory – monit was killing Apache when it thought Apache was using too much memory.

I made some changes to the new rules regarding Apache in /etc/monit/monitrc.

check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host alexschroeder.ch port 80 protocol http
      and request "/monit/token" then restart
#  if cpu is greater than 60% for 2 cycles then alert
#  if cpu > 80% for 5 cycles then restart
#  if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
#  if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

Basically, since almost everything I’m doing on this server involves Apache, I’m no longer interested in holding it back. Use all the CPU if you want, Apache. Use all the memory if you want, Apache. If mod_perl means big memory footprint, then so be it. We’ve got a lot of requests to serve.

https://farm9.staticflickr.com/8588/16062737735_3034fd5cd7_o.png https://farm8.staticflickr.com/7568/15443081663_d5a398c249_o.png

(Continued here.)

Tags: RSS RSS RSS RSS RSS

Add Comment

2014-12-18 Emacs Wiki Migration

I’m having problems with the git integration. The problem is that the user running git commands on the new server will be both www-data and alex.

On #git, I was told make sure all the permissions for the files in the .git are set up correctly.

sudo find .git -type f -exec chmod g+rw {} \;
git config core.sharedRepository true

And it worked! Thanks, Seveas.

Wow, what an exercise in humility this has been! 😨 I had switched the DNS entry and was awaiting the slow increase in traffic, wondering how this system would handle it. Thomas Waldmann of Waldmann had hosted the Emacs Wiki since the very early days and in recent years he had kept asking for me to switch from CGI to FastCGI. It just never seemed that urgent. I was about to learn how damn urgent it was!

Soon, load climbed to 40 and 50. It was getting hard to use Emacs. Load kept on climbing. When it reached 198 I gave up fiddling with the rewrite rules and just killed Apache. There was no point.

When the system had recovered, I started working on FCGI support. I was starting with some old info on the Oddmuse wiki and learning as a I went. The result is on the Using mod_fastcgi page. Rewrite rules weren’t working correctly. The language stuff was interfering. Holy cow and I just kept on hacking.

The status right now:

  1. The other sites are up (such as this one) and that’s already a win. 😊
  2. I can read the pages on Emacs Wiki. It is slow but it works. 👍
  3. Sometimes I get a 503 error. 👎
  4. When I’m on the server, running Emacs and stuff, these apps will sometimes get killed for no apparent reason. Out of memory errors? 😢
  5. I realized that syslog had been uninstalled and the replacement rsyslog wasn’t running—that’s why I still don’t know whether this is a memory problem. 😣
  6. I upgraded the hosting package but I’m not sure what I need to increase to solve that problem. Perhaps it can be resolved using some of the many fcgid parameters? 😪

I’m currently using:

FcgidMaxProcesses 20
FcgidProcessLifeTime 300
FcgidMaxRequestsPerProcess 100

top lists a lot of zombies. The default for the FcgidZombieScanInterval directive is supposed to be 3s. I wonder why they keep hanging around? Oh well.

# ps aux | grep fcgi
www-data 17386  0.0  0.0      0     0 ?        Z    19:55   0:05 [_emacs.fcgi] <defunct>
www-data 17394  0.0  0.0      0     0 ?        Z    19:55   0:05 [_emacs.fcgi] <defunct>
www-data 20349  0.0  0.0      0     0 ?        Z    20:16   0:04 [_emacs.fcgi] <defunct>
www-data 20420  0.0  0.0      0     0 ?        Z    20:17   0:03 [_emacs.fcgi] <defunct>
www-data 20438  0.1  0.0      0     0 ?        Z    20:17   0:05 [_emacs.fcgi] <defunct>
www-data 20969  0.0  0.0      0     0 ?        Z    20:22   0:00 [_emacs.fcgi] <defunct>
www-data 20970  0.0  0.0      0     0 ?        Z    20:22   0:00 [_emacs.fcgi] <defunct>
www-data 21394  0.0  0.0      0     0 ?        Z    20:26   0:02 [_emacs.fcgi] <defunct>
www-data 21446  0.1  0.0      0     0 ?        Z    20:27   0:05 [_emacs.fcgi] <defunct>
www-data 23966  0.1  0.0      0     0 ?        Z    20:47   0:04 [_emacs.fcgi] <defunct>
www-data 24036  0.7  0.0      0     0 ?        Z    20:48   0:19 [_emacs.fcgi] <defunct>
www-data 24050  0.2  0.0      0     0 ?        Z    20:49   0:06 [_emacs.fcgi] <defunct>
www-data 24055  0.0  0.0      0     0 ?        Z    20:49   0:00 [_emacs.fcgi] <defunct>
www-data 24056  0.0  0.0      0     0 ?        Z    20:49   0:00 [_emacs.fcgi] <defunct>
www-data 24475  0.1  0.0      0     0 ?        Z    20:50   0:03 [_emacs.fcgi] <defunct>
www-data 24482  0.9  0.0      0     0 ?        Z    20:51   0:23 [_emacs.fcgi] <defunct>
www-data 24483  1.2  0.0      0     0 ?        Z    20:51   0:32 [_emacs.fcgi] <defunct>
www-data 24486  2.7  0.0      0     0 ?        Z    20:51   1:08 [_emacs.fcgi] <defunct>
www-data 24581  0.2  0.0      0     0 ?        Z    20:54   0:05 [_emacs.fcgi] <defunct>
www-data 26669  0.3  0.0      0     0 ?        Z    21:02   0:05 [_emacs.fcgi] <defunct>

I wonder how Nic Ferrier and friends will handle this load once they take over Emacs Wiki. After all, that’s their long term plan. See this page from January 2013, for example.

This unexpected quitting while I’m on the server is disconcerting. I’m going to try this, now:

FcgidMaxProcesses 5
FcgidProcessLifeTime 300
FcgidMaxRequestsPerProcess 100
FcgidZombieScanInterval 3

More issues from the Apache error log:

[Fri Dec 19 00:15:48 2014] [warn] [client 68.180.228.96] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 00:15:49 2014] [error] [client 162.243.99.58] (13)Permission denied: exec of '/home/alex/emacswiki.org/emacs.pl' failed
[Fri Dec 19 00:15:49 2014] [error] [client 162.243.99.58] Premature end of script headers: emacs.pl
[Fri Dec 19 00:15:53 2014] [warn] [client 91.39.71.18] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 00:16:04 2014] [error] [client 65.49.14.147] File does not exist: /home/alex/emacswiki.org/cgi-bin, referer: http://www.emacswiki.org/test/?action=browse;diff=2;id=Comments_on_HomePage
[Fri Dec 19 00:16:05 2014] [error] [client 174.36.228.156] (13)Permission denied: exec of '/home/alex/emacswiki.org/emacs.pl' failed
[Fri Dec 19 00:16:05 2014] [error] [client 174.36.228.156] Premature end of script headers: emacs.pl

😲

“mod_fcgid: can’t apply process slot” → “This warning tells you that the FastCGI process pool is exhausted and it has a global limit of FcgidMaxProcesses and a per-script limit of FcgidMaxProcessesPerClass.” [1]

I guess 5 is not a good number?

Recommendation found elsewhere [2], but I need to think about this.

I keep starting at the error.log. Looking at a single IP address:

[Fri Dec 19 08:42:37 2014] [warn] [client XXX] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 08:42:43 2014] [warn] [client XXX] mod_fcgid: error reading data, FastCGI server closed connection
[Fri Dec 19 08:42:43 2014] [error] [client XXX] Premature end of script headers: _emacs.fcgi

I have two cores and 2GB RAM in this openvz. Experimenting some more.

# This determines how many processes each user can run:
FcgidMinProcessesPerClass 0
FcgidMaxProcessesPerClass 2
# The following depends on the RAM we have on the machine:
FcgidMaxProcesses 10
# Lifetime control:
FcgidIdleTimeout 60
FcgidIdleScanInterval 30
FcgidProcessLifeTime 120
# Restart after a while
FcgidMaxRequestsPerProcess 1000
FcgidZombieScanInterval 3

The result is always the same: After restarting Apache, the site works for a while. I still have the feeling that eventually it stops working.

Let’s look at the processes over time:

alex@kallobombus:~$ ps aux | grep _emacs
www-data 10594  0.0  0.0  50160  1672 ?        S    Dec18   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
www-data 26812  0.7  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 28892  112 77.2 1680376 1619920 ?     R    08:50   0:03 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ ps aux | grep _emacs
www-data 10594  0.0  0.0  50160  1672 ?        S    Dec18   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
www-data 26812  0.6  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 28898  118 85.9 1867532 1801836 ?     R    08:50   0:03 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ ps aux | grep _emacs
www-data 10594  0.0  0.0  50160  1672 ?        S    Dec18   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
www-data 26812  0.6  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 28902 74.3 97.4 2234656 2044004 ?     R    08:50   0:04 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ ps aux | grep _emacs
www-data 10594  0.0  0.0  50160  1672 ?        S    Dec18   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
www-data 26812  0.6  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 28911 84.6 51.7 1142412 1084660 ?     R    08:51   0:02 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ ps aux | grep _emacs
www-data 10594  0.0  0.0  50160  1672 ?        S    Dec18   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
www-data 26812  0.6  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 28916 67.5 97.5 2375680 2046584 ?     R    08:51   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi

So, what am I seeing?

  1. PID 10594 keeps sleeping.
  2. PID 26812 keeps being a zombie.
  3. the third slot keeps doing all the work

I’m going to kill them.

alex@kallobombus:~$ sudo kill 10594
alex@kallobombus:~$ ps -U www-data u | grep _emacs
www-data 26812  0.5  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 29457 73.5 95.8 2251504 2010956 ?     R    08:56   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ sudo kill 26812
alex@kallobombus:~$ ps -U www-data u | grep _emacs
www-data 26812  0.5  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 29468 65.1 97.6 2375688 2048860 ?     R    08:56   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ sudo kill -9 26812
alex@kallobombus:~$ ps -U www-data u | grep _emacs
www-data 26812  0.5  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 29475 71.7 46.2 1010184 970136 ?      R    08:56   0:02 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ ps -U www-data u | grep _emacs
www-data 26812  0.5  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 29475 64.8 97.5 2248324 2046564 ?     D    08:56   0:05 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ ps -U www-data u | grep _emacs
www-data 26812  0.5  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 29489 66.8 97.0 2251764 2034580 ?     D    08:57   0:06 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ ps -U www-data u | grep _emacs
www-data 26812  0.5  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 29502 37.6  0.0      0     0 ?        Z    08:57   0:01 [_emacs.fcgi] <defunct>
alex@kallobombus:~$ ps -U www-data u | grep _emacs
www-data 26812  0.5  0.0      0     0 ?        Z    08:33   0:07 [_emacs.fcgi] <defunct>
www-data 29507 55.5  0.8  47356 18444 ?        R    08:57   0:01 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi

What am I seeing?

  1. the zombie stays and isn’t cleaned up
  2. no new fcgi process was started

I’m going to determine the zombie’s parent and kill it, too.

alex@kallobombus:~$ ps -U www-data ao pid,ppid,comm | head
  PID  PPID COMMAND
24999 24998 bash
25019 25018 bash
26746 12667 /usr/sbin/apach
26747 12667 /usr/sbin/apach
26812 26747 _emacs.fcgi <defunct>
26814 12667 /usr/sbin/apach
28394 25019 sudo
28395 28394 emacs
28486 12667 /usr/sbin/apach
alex@kallobombus:~$ sudo kill 26747
alex@kallobombus:~$ ps -U www-data u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data 31118  0.0  0.2  99856  4480 ?        S    09:03   0:00 /usr/sbin/apache2 -k start
www-data 31119  0.0  0.2 101008  4460 ?        S    09:03   0:00 /usr/sbin/apache2 -k start
www-data 31120  0.2  0.4 382116  8700 ?        Sl   09:03   0:00 /usr/sbin/apache2 -k start
www-data 31121  0.6  0.5 581604 11648 ?        Sl   09:03   0:00 /usr/sbin/apache2 -k start
www-data 31180 17.8  0.9  48168 19256 ?        S    09:03   0:02 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
www-data 31184 39.2 84.1 1833296 1765244 ?     R    09:03   0:03 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
alex@kallobombus:~$ ps -U www-data u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data 31118  0.0  0.0  99856  1844 ?        S    09:03   0:00 /usr/sbin/apache2 -k start
www-data 31119  0.0  0.0 101008  1988 ?        S    09:03   0:00 /usr/sbin/apache2 -k start
www-data 31120  0.2  0.1 382116  4120 ?        Sl   09:03   0:00 /usr/sbin/apache2 -k start
www-data 31121  0.4  0.1 581604  3840 ?        Sl   09:03   0:00 /usr/sbin/apache2 -k start
www-data 31180 21.3 75.1 1647300 1575160 ?     R    09:03   0:04 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
www-data 31184 31.6  0.0      0     0 ?        Z    09:03   0:06 [_emacs.fcgi] <defunct>
alex@kallobombus:~$ ps -U www-data u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data 31118  0.0  0.0  99856   392 ?        S    09:03   0:00 /usr/sbin/apache2 -k start
www-data 31119  0.0  0.0 101008   624 ?        S    09:03   0:00 /usr/sbin/apache2 -k start
www-data 31120  0.2  0.0 382116  1736 ?        Sl   09:03   0:00 /usr/sbin/apache2 -k start
www-data 31121  0.4  0.1 581628  2156 ?        Sl   09:03   0:00 /usr/sbin/apache2 -k start
www-data 31180 24.0 96.9 2303540 2033992 ?     R    09:03   0:06 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi
www-data 31197 32.0  0.8  44444 17060 ?        R    09:03   0:00 /usr/bin/perl /home/alex/emacswiki.org/_emacs.fcgi

What am I seeing?

  1. the two fcgi processes are running
  2. the two fcgi processes are sharing %CPU

Back to the error log. I still see several of these per minute. But the error frequency seems to be much lower.

[Fri Dec 19 09:09:09 2014] [warn] [client XXX] mod_fcgid: error reading data, FastCGI server closed connection
[Fri Dec 19 09:09:09 2014] [error] [client XXX] Premature end of script headers: _emacs.fcgi

Let’s look at the output of top:

top - 09:11:29 up 56 days,  1:49,  1 user,  load average: 2.21, 1.55, 1.18
Tasks:  26 total,   3 running,  23 sleeping,   0 stopped,   0 zombie
%Cpu(s): 24.5 us, 10.8 sy,  0.0 ni, 64.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.4 st
KiB Mem:   2097152 total,  2093688 used,     3464 free,        0 buffers
KiB Swap:   524288 total,   421636 used,   102652 free,      292 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
32061 www-data  20   0  128m  97m 1672 R  30.7  4.8   0:00.93 _emacs.fcgi
32053 www-data  20   0 2323m 1.9g 1672 R  26.4 93.3   0:04.98 _emacs.fcgi
31593 www-data  20   0  374m 2348 1508 S   1.3  0.1   0:00.66 /usr/sbin/apach
31121 www-data  20   0  569m 1896 1488 S   1.0  0.1   0:01.45 /usr/sbin/apach
31120 www-data  20   0  440m 2864 1640 S   0.7  0.1   0:01.24 /usr/sbin/apach
13266 root      20   0 51560 1892 1724 S   0.3  0.1   0:02.76 munin-node

What am I seeing?

  1. I have two cores and load is around 2, I guess that’s good
  2. I have very little memory free – an upgrade might be worth it
  3. I still have two fcgi processes running, sharing %CPU, so that’s good

The site is still incredibly slow.

I got rid of the “Negotiation: discovered file(s) matching request: /home/alex/emacswiki.org/emacs (None could be negotiated).” error that had shown up after another tweak of my .htaccess file by disabling MultiViews. That’s the part that will run “emacs.pl” when requesting “emacs”, I guess.

I also noticed that after a sudo service apache graceful I’m getting a lot of those “mod_fcgid: can’t apply process slot” errors again. Then I thought that a sudo service apache restart seemed to fix it. But no. After a while:

[Fri Dec 19 09:36:22 2014] [warn] [client XXX] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 09:36:23 2014] [warn] [client XXX] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 09:36:24 2014] [warn] [client XXX] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 09:36:25 2014] [warn] [client XXX] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 09:36:27 2014] [warn] [client XXX] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 09:36:30 2014] [warn] [client XXX] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi
[Fri Dec 19 09:36:32 2014] [warn] [client XXX] mod_fcgid: can't apply process slot for /home/alex/emacswiki.org/_emacs.fcgi

What are the fcgi processes doing?

alex@kallobombus:~$ ps -U www-data u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data  1819  0.0  0.0  99856   388 ?        S    09:31   0:00 /usr/sbin/apache2 -k start
www-data  1820  0.1  0.0 101008   572 ?        S    09:31   0:00 /usr/sbin/apache2 -k start
www-data  1821  0.1  0.3 385656  7504 ?        Sl   09:31   0:00 /usr/sbin/apache2 -k start
www-data  1880  0.1  0.3 384996  8060 ?        Sl   09:31   0:00 /usr/sbin/apache2 -k start
www-data  1965  7.0  0.0      0     0 ?        Z    09:33   0:20 [_emacs.fcgi] <defunct>
www-data  1968  1.9  0.0      0     0 ?        Z    09:33   0:05 [_emacs.fcgi] <defunct>
www-data  1973  0.1  0.3 383852  8024 ?        Sl   09:33   0:00 /usr/sbin/apache2 -k start
www-data  2013  0.1  0.3 383692  8152 ?        Sl   09:34   0:00 /usr/sbin/apache2 -k start

Apparently, only a complete sudo apachectl stop will do what is required. The apache2 parent process isn’t reaping the zombies, and that’s why they hang around. It’s only if the parent is itself totally stopped that reaping happens and new fcgi processes will get started. How annoying!

The reason for the zombie epidemic remains unknown. Restart:

alex@kallobombus:~$ ps -U www-data ao pid,ppid,comm
  PID  PPID COMMAND
 2941  2940 bash
 3150  2491 /usr/sbin/apach
 3151  2491 /usr/sbin/apach
 3152  2491 /usr/sbin/apach
 3153  2491 /usr/sbin/apach
 3237  3151 _emacs.fcgi
 3264  3151 _emacs.fcgi
 3268  3152 cw-en.pl
 3269  2941 ps

Wait for a bit:

alex@kallobombus:~$ ps -U www-data ao pid,ppid,comm
  PID  PPID COMMAND
 3150  2491 /usr/sbin/apach
 3151  2491 /usr/sbin/apach
 3153  2491 /usr/sbin/apach
 3660  3151 _emacs.fcgi <defunct>
 3675  3151 _emacs.fcgi <defunct>
 3679  2491 /usr/sbin/apach
 3714  2491 /usr/sbin/apach
 3759  3758 bash
 3798  3153 cw-en.pl <defunct>
 3799  3759 ps

Why?

Well… Running sudo service apache2 restart and waiting for it to happen again. If you have any ideas, let me know!

alex@kallobombus:~/emacswiki/git$ ps -U www-data ao pid,ppid,comm
  PID  PPID COMMAND
 3759  3758 bash
 7229  7225 /usr/sbin/apach
 7230  7225 /usr/sbin/apach
 7231  7225 /usr/sbin/apach
 7235  7225 /usr/sbin/apach
 7289  7230 _emacs.fcgi
 7290  7230 _emacs.fcgi
 7291  3759 ps

Soon enough:

alex@kallobombus:~/emacswiki/git$ ps -U www-data ao pid,ppid,comm
  PID  PPID COMMAND
 3759  3758 bash
 7229  7225 /usr/sbin/apach
 7230  7225 /usr/sbin/apach
 7235  7225 /usr/sbin/apach
 7626  7230 _emacs.fcgi <defunct>
 7633  7225 /usr/sbin/apach
 7666  7230 _emacs.fcgi <defunct>
 7691  7225 /usr/sbin/apach
 7734  3759 ps

Gaaah! Also note this thread: Unreaped Zombie Children Of Mod_fcgid. Sounds like my problem. I’m running Apache 2.2.22-13+deb7u3 and mod_fcgid 2.3.6-1.2+deb7u1.

Another theory: I restarted Apache, waited for a few seconds, got booted off my server (out of memory?), reconnected, and found the fcgi processes defunct. Perhaps this is related?

I just wonder whether upgrading from 2GB of RAM to 4GB of RAM will “magically” fix this.

Continued 2014-12-19 Emacs Wiki Migration

Tags: RSS RSS RSS RSS RSS

Comments on 2014-12-18 Emacs Wiki Migration


Denis Evsyukov
Hello, why do you use apache, instead of nginx? For these tasks, nginx is best suited. And memory consumes much less and works faster.

Denis Evsyukov 2014-12-21 06:31 UTC



Alex Schroeder
I would have to relearn everything I know about Apache.

– Alex Schroeder 2014-12-21 08:52 UTC


May the Force be with you!

AlexDaniel 2014-12-22 08:33 UTC



Alex Schroeder
Thanks! 😊

To quote from 30 Most Memorable ‘Star Wars’ Quotes: “What a piece of junk!”

– Alex Schroeder 2014-12-22 12:25 UTC

Add Comment

More...

Define external redirect: LastUpdate