Oddmuse

Difference between revision 12 and current revision

Summary: http://emacswiki.org/alex → https://alexschroeder.ch/wiki

Changed:

< [[image/right/inline external:rss-big.png|RSS Feed|http://emacswiki.org/alex/feed/full/oddmuse]]

to

> [[image/right/inline external:rss-big.png|RSS Feed|https://alexschroeder.ch/wiki/feed/full/oddmuse]]


RSS Feed Oddmuse is the wiki engine running all the wikis at emacswiki.org – including EmacsWiki itself. My main interests are:

See OddmuseRoadmap for my thoughts about the design of Oddmuse.

2015-06-14 Go Oddmuse

Sometimes I wonder whether I should move Oddmuse away from Perl 5 to something else. Something I would like to learn. An opportunity to redo everything, from scratch. Not many people are using Oddmuse, so I might as well write something else, for myself.

I recently read Writing Web Applications for Go, where they develop a simple wiki. I wondered about storing the data in a git repository. I found git2go. Hm…

Tags: RSS RSS RSS

Comments on 2015-06-14 Go Oddmuse

Last edit

Summary: fixing links

Changed:

< -- AlexDaniel 2015-06-14 14:05 UTC

to

> -- [http://alexdaniel.org/About AlexDaniel] 2015-06-14 14:05 UTC

Changed:

< -- AlexDaniel 2015-06-16 00:22 UTC

to

> -- [http://alexdaniel.org/About AlexDaniel] 2015-06-16 00:22 UTC

Changed:

< -- AlexDaniel 2015-06-16 00:31 UTC

to

> -- [http://alexdaniel.org/About AlexDaniel] 2015-06-16 00:31 UTC

Changed:

< Hehe, I think you and [http://oddmuse.org/wiki/Revolutionary_Changes Oddmuse: Revolutionary Changes] is what started this all! I looked at Oddmuse as a "mission accomplished" project. With the migration of Emacs Wiki I've realized that a big site might want to serve its own pages without running a web server. I tried to understand [http://plackperl.org/ PSGI/Plack] in order to have Perl run Oddmuse like a web application. When I saw the wiki tutorial for Go, and how I got a web server up and running in no time, and how easy it was to include some basic markup, I was thrilled! Sure, [https://github.com/kensanata/oddmu the code] doesn't have a lot of features. But it beats the SmallestWiki candidates. :)

to

> Hehe, I think you and [http://oddmuse.org/wiki/Revolutionary_Changes Oddmuse: Revolutionary Changes] is what started this all! I looked at Oddmuse as a "mission accomplished" project. With the migration of Emacs Wiki I've realized that a big site might want to serve its own pages without running a web server. I tried to understand [http://plackperl.org/ PSGI/Plack] in order to have Perl run Oddmuse like a web application. When I saw the wiki tutorial for Go, and how I got a web server up and running in no time, and how easy it was to include some basic markup, I was thrilled! Sure, [https://github.com/kensanata/oddmu the code] doesn't have a lot of features. But it beats the [[SmallestWiki]] candidates. :)


Perl 6 is an obvious candidate. And yes, they did not just change print ‘x’ to print(’x’) like they did in Python, it is a completely new language that you have to learn.

Speaking of Go, there is a great video called Perl 6 for Mere Mortals, and here is a direct link to a part of it which is somehow :) related to Go. That’s not a meaningful comparison, but it should give you the idea. You can also watch that video from the beginning, it is very interesting if you have not investigated into Perl 6 yet.

Now, if we think about all of the languages out there, what are the actual features that are required for Oddmuse?

Well, since Oddmuse is all about parsing the wiki text, you need some tools for parsing. Perl 5 has regexes (um, okay), Perl 6 has grammars (exactly what we need!), and Go has something as well (like this, which is, again, verbose as hell), but I’m not an expert. You can also use libs, but having a built-in support is probably advantageous.

Somehow I can’t really think of any other criteria, everything else probably exists in every other language out there. You might also say “performance” but the only website suffering from that is probably emacswiki, and it all depends on your actual goals – for example, better execution time can be achieved by using concurrency, which does not really lead to better CPU time. Concurrency in Perl 6 – yes (but maybe just a bit flaky at the moment), performance in Perl 6 – well, not now, but the potential is there (i.e. gradual typing).

In other words: what is the motivation? “Just to learn” is not going to get you anywhere, not even make you learn something (learning to find reasons for doing stuff (i.e. causation) is part of the learning, doing stuff purposelessly is harmful to the thinking process).

AlexDaniel 2015-06-14 14:05 UTC



Alex Schroeder
Perhaps Go just makes me want to delve into the low level stuff. I like the post about Handwritten Parsers & Lexers in Go. :)

I keep thinking about all the design decisions. Filenames are pagenames. Namespaces are subdirectories. Can we switch to “git first”? No more log files. We no longer need keep files. How would modules work? Would you simply recompile the wiki? Dynamic linking? Would I want to use a templating architecture? People seem to like that. Was “printing as we go” a good decision? Perhaps it no longer matters much.

I need to think about extensibility of Cajun, the Wiki Creole lexer and parser.

– Alex Schroeder


These are very good questions. Maybe you should write down these ideas on Oddmuse: Revolutionary Changes.

  • “git first” – Yea, but… This would effectively break the goal of having one script file that just works (and I think that this is more important than having no recent changes and keep files logic in the core).
  • “How would modules work? Would you simply recompile the wiki?” – this would effectively throw away all of the nice things like Module Updater or Module Bisect (or at least make them much harder to implement). It will also break the potential of a few other ideas that keep floating in my head. In other words, “simply recompile the wiki” is not that simple.
  • “Would I want to use a templating architecture? People seem to like that.” Yea, but… It seems like if we switch to Perl6 then the core will collapse into a small piece of code with all of the irrelevant stuff separated into grammars or other classes, or maybe just tiny subroutines. Which brings us to the next point…
  • “Was “printing as we go” a good decision? Perhaps it no longer matters much.” – Good decision, but it really depends on your understanding of “as we go”. For example, what if we parse the wiki text first, before printing anything except the header? Then, once we start processing the parsed text, we can print stuff as we go. That’s exactly how it will work in perl6 (at least, that’s the easiest way) – first you slam the wiki text into your grammar and wait for it to be processed (oh by the way, while this is happening you can actually do other stuff asynchronously, like start reading the tag index or whatever). Then, when your wikitext is parsed, you can start deciding what to print now. And this is so much better than what we have now. Want to print table of contents? Just take the required information from the match object, no need for dirty hacks. In other words, in perl6 it is very easy to have parsing and actions separately, which is beneficial in many ways. But at the same time, once you start processing the actions you can start printing as you go (which makes sense, what if there is some complex search stuff included on the page, you’d probably want to see at least half of the page while this is going).

AlexDaniel 2015-06-16 00:22 UTC


And yeah, if you are seriously thinking about Go, what if we make a comparison table with Perl6 and Go, and the relevant language features? Both of the languages have some interesting stuff to offer, for example Perl6 to JS and Go to JS. And there are definitely some winning points for Go, especially associated with execution time and maturity.

AlexDaniel 2015-06-16 00:31 UTC



Alex Schroeder
Hehe, I think you and Oddmuse: Revolutionary Changes is what started this all! I looked at Oddmuse as a “mission accomplished” project. With the migration of Emacs Wiki I’ve realized that a big site might want to serve its own pages without running a web server. I tried to understand PSGI/Plack in order to have Perl run Oddmuse like a web application. When I saw the wiki tutorial for Go, and how I got a web server up and running in no time, and how easy it was to include some basic markup, I was thrilled! Sure, the code doesn’t have a lot of features. But it beats the SmallestWiki candidates. :)

You arguments in favor of some of the core architecture decisions made for Oddmuse are good – and your hopes for Perl 6 and improved parsing are infectious! I’m very much looking forward to Perl 6.

– Alex Schroeder 2015-06-16 06:02 UTC



Alex Schroeder
It’s also cool to look at the TinyWiki source code again and rediscover why I rewrote the Usemod Wiki style search and replace code with the extensible state machine code we have right now.

– Alex Schroeder 2015-06-16 08:00 UTC

Add Comment

2014-12-30 Emac Wiki Migration

OK, I guess Nic and I have a plan. We’ve started writing a backlog for the Emacs Wiki Migration. Hopefully this will make the current wiki easier to cache.

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-30 Emac Wiki Migration

Last edit

No diff available.

Add Comment

2014-12-28 Emacs Wiki Migration

Gaah! :(

So, I still think this is a problem with the hosting provider. Here’s what I think is evidence:

https://farm8.staticflickr.com/7575/15509223034_8ee49a110d_o.png

What could be causing this regular forks if not other virtual machines hogging CPU at regular intervals, or other virtual machines hogging disk access, causing CGI scripts to pile up? I wonder…

Things I care about: Diagnosing Disk I/O issues: swapping, high IO wait, congestion.

Let’s see what iotop -ao says right now:

Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                    
11774 be/4 www-data     36.00 K     20.00 K  0.00 %  0.02 % [/usr/sbin/apach]
11801 be/4 www-data      0.00 B     12.00 K  0.00 %  0.00 % apache2 -k start
11810 be/4 www-data      0.00 B      4.00 K  0.00 %  0.00 % apache2 -k start 
29022 be/4 root          0.00 B     32.00 K  0.00 %  0.00 % monit -c /etc/monit/monitrc
20368 be/4 root          0.00 B     60.00 K  0.00 %  0.00 % rsyslogd -c5

I don’t think we’re using swap at all (munin shows us using less than 2G with 4G available). But who knows? I used top, pressed f to show the list of fields, and I used d (display) and s (sort) on SWAP.

top - 22:14:00 up 7 days, 21:11,  2 users,  load average: 0.51, 0.31, 0.22
Tasks:  24 total,   1 running,  23 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.6 us,  0.6 sy,  0.0 ni, 95.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   4194304 total,   894976 used,  3299328 free,        0 buffers
KiB Swap:  1048576 total,     5524 used,  1043052 free,   785880 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND              SWAP 
15874 root      20   0 71192 5224 1620 S   0.0  0.1   2:11.36 fail2ban-server      2296 
27076 root      20   0 72564 1092  708 S   0.0  0.0   0:16.47 sendmail-mta         1384 
21949 root      20   0 49892  684  568 S   0.0  0.0   0:02.23 sshd                  492 
29022 root      20   0 42984 2688 2136 S   0.0  0.1   0:48.28 monit                 284 
20367 root      20   0 49860 4504  712 S   0.0  0.1   0:24.56 rsyslogd              236 
19427 root      30  10 12252  788  784 S   0.0  0.0   0:00.00 rsync                 160 
15876 root      20   0 22260  896  672 S   0.0  0.0   0:08.86 gam_server            136 
22815 root      20   0 20368  496  364 S   0.0  0.0   0:02.03 cron                  120 

I don’t think that’s suspicious. Those tasks probably ended up in swap because they have been running for so long?

I installed the apache_vhosts plugin for munin. I think there will be no surprises with emacswiki.org taking the brunt with a factor of 10:1. Right now, however, it seems to be misconfigured somehow, not recording any emacswiki.org traffic and missing most of the other traffic. Weird!

Ah, apache_vhosts munin plugin problem fixed. All my sites were using a CustomLog except for alexschroeder.ch:80 and campaignwiki.org:80—these just redirect to the HTTPS version of the site. Thus, these sites were being counted by the apache_vhosts plugin. But that is now fixed, and we have a graph!

https://farm8.staticflickr.com/7527/15960447830_a102ae6df7_o.png

https://farm9.staticflickr.com/8667/15528022193_78df3fc37d_o.png

I’m not quite sure how to compare this with the following log file analysis where I simply group access by site:

root@kallobombus:/var/log/apache2# perl /home/alex/bin/site-comparison < access.log
                          Site       hits bandw. hits% interv. status code distrib.
             www.emacswiki.org     104499     6K  47%    2.1s  301 (29%), 501 (26%), 200 (21%
          www.alexschroeder.ch      67396    31K  30%    3.2s  200 (94%), 304 (2%), 301 (1%),
         www.communitywiki.org      24574    17K  11%    8.9s  200 (80%), 301 (7%), 404 (3%),
          www.campaignwiki.org      13902    13K   6%   15.6s  200 (74%), 404 (17%), 302 (4%)
               www.oddmuse.org       9498    23K   4%   22.9s  200 (66%), 404 (14%), 301 (12%
       www.arabisch-lernen.org        845     7K   0%  256.4s  200 (94%), 404 (3%), 403 (1%),
         www.orientalisch.info        404     4K   0%  536.5s  200 (80%), 301 (14%), 404 (2%)

Ah, of course. The access.log is can be up to seven days. Thus:

root@kallobombus:/var/log/apache2# tail -n 10000 access.log | /home/alex/bin/site-comparison 
                          Site       hits bandw. hits% interv. status code distrib.
          www.alexschroeder.ch       6430    24K  64%    1.9s  200 (88%), 301 (8%), 404 (1%),
             www.emacswiki.org       1187    12K  11%   10.4s  301 (38%), 200 (36%), 404 (19%
         www.communitywiki.org       1150    14K  11%   10.7s  200 (79%), 301 (7%), 404 (5%),
          www.campaignwiki.org        604     9K   6%   20.3s  200 (58%), 301 (22%), 404 (12%
               www.oddmuse.org        415    18K   4%   29.6s  200 (64%), 404 (18%), 301 (10%
       www.arabisch-lernen.org        202     5K   2%   37.4s  404 (90%), 200 (9%), 501 (0%)
         www.orientalisch.info         12     4K   0% 1010.2s  200 (91%), 304 (8%)

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-28 Emacs Wiki Migration

Last edit

No diff available.

Add Comment

2014-12-27 Emacs Wiki Migration

Based on some information I got from Charlie !McMackin—who had set up a virtual machine and got Emacs Wiki to run on it—I switched Apache from “worker” to “prefork” and I’m now using the following config:

<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       1
    MaxSpareServers       2
    MaxClients		  5
    MaxRequestsPerChild 10000
</IfModule>

I checked http://localhost/server-info and can confirm that we’re running Prefork:

Server Settings

Server Version: Apache/2.2.22 (Debian) mod_ssl/2.2.22 OpenSSL/1.0.1e mod_perl/
    2.0.7 Perl/v5.14.2
Server Built: Jul 24 2014 15:34:00
Server loaded APR Version: 1.4.6
Compiled with APR Version: 1.4.6
Server loaded APU Version: 1.4.1
Compiled with APU Version: 1.4.1
Module Magic Number: 20051115:30
Hostname/port: alexschroeder.ch:443
Timeouts: connection: 300    keep-alive: 5
MPM Name: Prefork
MPM Information: Max Daemons: 5 Threaded: no Forked: yes
Server Architecture: 64-bit

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-27 Emacs Wiki Migration

Last edit

No diff available.

Add Comment

2014-12-26 Emacs Wiki Migration

I’m not happy. Today, multiple Apache restarts. Grepping syslog for restarts:

Dec 26 07:38:32 localhost monit[29022]: 'apache' trying to restart
Dec 26 11:38:53 localhost monit[29022]: 'apache' trying to restart
Dec 26 14:47:16 localhost monit[29022]: 'apache' trying to restart
Dec 26 16:49:46 localhost monit[29022]: 'apache' trying to restart
Dec 26 17:26:06 localhost monit[29022]: 'apache' trying to restart
Dec 26 18:06:29 localhost monit[29022]: 'apache' trying to restart
Dec 26 19:48:54 localhost monit[29022]: 'apache' trying to restart

https://farm8.staticflickr.com/7566/15494392223_95ee7bf147_o.png

I’m going to further reduce the number of threads.

<IfModule mpm_worker_module>
    StartServers          1
    ServerLimit           2
    ThreadsPerChild       5
    MaxClients           10
    MinSpareThreads      10
    MaxSpareThreads      20
    MaxRequestsPerChild  10000
</IfModule>

I also installed lsof and wrote a little plugin for munin. What I really want to see is the number of open files, or something like that. I remember checking that when we had load problems on Mirabel without a matching number of processes actually doing anything. Just in case the processes are all waiting for file access.

I’m not sure how well this is going to work, but this is my first stab at the files plugin. 😊

#!/bin/sh
# 
# Plugin to monitor the number of open files in the system.
#
# Parameters:
# 	
# 	config   (required)
# 	autoconf (optional - used by munin-config)
#
# Magic markers (Used by munin-config and some installation scripts.
# Optional):
#
#%# family=contrib
#%# capabilities=autoconf

# PROC=${0##*/files_}
PORTS="80 443"

if [ "$1" = "autoconf" ]; then
	if [ -x /usr/bin/lsof ]; then
		echo yes
		exit 0
	else
		echo "no (no lsof)"
		exit 0
	fi
fi

if [ "$1" = "config" ]; then
	echo graph_title "open files"
	echo graph_vlabel "number of files"
	echo graph_info "This graph shows the number of open files by port (80 is http, 443 is https)."
	echo graph_category "apache"
	for P in $PORTS; do
	    echo "$P.label port $P"
	done
	exit 0
fi

for P in $PORTS; do 
    N=$(lsof -i :$P | tail -n +2 | wc -l)
    echo "$P.value $N"
done

All I know is that load is climbing right now. Close to 6. Time for bed! Grrrrr.

A few hours later, stats seem to be worse:

Dec 26 22:47:16 localhost monit[29022]: 'apache' trying to restart
Dec 26 23:23:34 localhost monit[29022]: 'apache' trying to restart
Dec 26 23:58:01 localhost monit[29022]: 'apache' trying to restart
Dec 27 00:10:19 localhost monit[29022]: 'apache' trying to restart
Dec 27 00:38:40 localhost monit[29022]: 'apache' trying to restart
Dec 27 01:06:58 localhost monit[29022]: 'apache' trying to restart
Dec 27 01:17:15 localhost monit[29022]: 'apache' trying to restart
Dec 27 03:51:40 localhost monit[29022]: 'apache' trying to restart
Dec 27 04:23:59 localhost monit[29022]: 'apache' trying to restart
Dec 27 04:42:21 localhost monit[29022]: 'apache' trying to restart
Dec 27 05:30:47 localhost monit[29022]: 'apache' trying to restart
Dec 27 06:59:09 localhost monit[29022]: 'apache' trying to restart
Dec 27 08:05:33 localhost monit[29022]: 'apache' trying to restart

Let’s hope that things will improve with prefork!

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-26 Emacs Wiki Migration

Last edit

No diff available.

Add Comment

2014-12-25 Emacs Wiki Migration

I’ve been thinking about special-casing some caching: a mod_rewrite rule that says:

  • if the URL has the form /emacs/page
  • and there is no cookie (ie. bots, just readers)
  • and there is an if-none-match header (from nginx, I presume?)
  • and its value matches the “default”
  • return 304 without invoking the wiki

So, what’s the “default”?

sub DoBrowseRequest {
  ...
  print $q->header(-status=>'304 NOT MODIFIED') and return if PageFresh(); # return value is ignored
  ...
}

sub PageFresh { # pages can depend on other pages (ie. last update), admin status, and css
  return 1 if $q->http('HTTP_IF_NONE_MATCH') and GetParam('cache', $UseCache) >= 2
    and $q->http('HTTP_IF_NONE_MATCH') eq PageEtag();
}

sub PageEtag {
  my ($changed, $visible, %params) = CookieData();
  return UrlEncode(join($FS, $LastUpdate||$Now, sort(values %params))); # no CTL in field values
}

And what about $LastUpdate?

  my $ts = (stat($IndexFile))[9]; # always stat for multiple server processes
  ...
  $LastUpdate = $ts;

So, I think that the “default” ETag should simply be the mtime of the pageidx file. We’d have to use the RewriteMap directive with an External Rewriting Program.

Yikes! I’m not sure this is really faster… I guess it would be faster if the wiki itself was no longer using mod_perl (for whatever reasons). The external rewriting program would comparing the ETag header with the pageidx mtime and return a special value which would in turn be rewritten by a second rule to “redirect” to a 304 NOT MODIFIED response…

Something to consider for later.

As I’m writing this, I’m noticing that the server is slow. Check the current time, right, the five minutes before the full hour are a problem. Load reaching 6.

top - 17:59:18 up 4 days, 16:57,  1 user,  load average: 5.94, 5.41, 4.04
Tasks:  20 total,   1 running,  19 sleeping,   0 stopped,   0 zombie
%Cpu(s): 98.7 us,  1.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   4194304 total,  1688940 used,  2505364 free,        0 buffers
KiB Swap:  1048576 total,     4416 used,  1044160 free,  1331364 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
 2635 www-data  20   0  470m 122m 4704 S 101.9  3.0  42:43.65 /usr/sbin/apach   
 4902 www-data  20   0  434m  91m 5360 S  72.6  2.2  21:12.41 /usr/sbin/apach   
    1 root      20   0 10608  196   60 S   0.0  0.0   0:04.12 init              
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kthreadd/2828     
    3 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khelper/2828      
  702 root      20   0 51560 9856 2068 S   0.0  0.2   0:11.56 munin-node        
 1790 messageb  20   0 29764  256   12 S   0.0  0.0   0:00.00 dbus-daemon       
 8703 root      20   0 71252 3664 2872 S   0.0  0.1   0:00.02 sshd              
 8707 alex      20   0 71252 1720  924 S   0.0  0.0   0:00.00 sshd              
 8708 alex      20   0 21256 4096 1712 S   0.0  0.1   0:00.05 bash              
 8723 alex      20   0 23120 1612 1188 R   0.0  0.0   0:00.00 top               
15874 root      20   0 71192 5168 1620 S   0.0  0.1   1:20.36 fail2ban-server   
15876 root      20   0 22260  908  672 S   0.0  0.0   0:06.63 gam_server        
19427 root      30  10 12252  956  792 S   0.0  0.0   0:00.00 rsync             
20367 root      20   0 49728 4624  712 S   0.0  0.1   0:19.49 rsyslogd          
21949 root      20   0 49892 1236  628 S   0.0  0.0   0:00.81 sshd              
22815 root      20   0 20368  496  364 S   0.0  0.0   0:01.23 cron              
25118 root      20   0  122m  10m 3932 S   0.0  0.3   0:01.28 /usr/sbin/apach   
27076 root      20   0 72564 2024  732 S   0.0  0.0   0:09.69 sendmail-mta      
29022 root      20   0 42984 2976 2148 S   0.0  0.1   0:18.03 monit             

A few minutes later load reached 7. And my processes aren’t busy.

top - 18:01:45 up 4 days, 16:59,  1 user,  load average: 6.65, 5.91, 4.43
Tasks:  24 total,   2 running,  22 sleeping,   0 stopped,   0 zombie
%Cpu(s): 97.2 us,  0.9 sy,  1.7 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem:   4194304 total,  1726856 used,  2467448 free,        0 buffers
KiB Swap:  1048576 total,     4416 used,  1044160 free,  1331376 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
 4902 www-data  20   0  437m 108m 5384 S  61.6  2.7  22:46.97 /usr/sbin/apach   
 9105 munin     30  10  140m  23m 5936 R   3.3  0.6   0:02.26 munin-graph       
    1 root      20   0 10608  196   60 S   0.0  0.0   0:04.12 init              
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kthreadd/2828     
    3 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khelper/2828      
  702 root      20   0 51560 9856 2068 S   0.0  0.2   0:11.57 munin-node        
 1790 messageb  20   0 29764  256   12 S   0.0  0.0   0:00.00 dbus-daemon       
 8703 root      20   0 71252 3664 2872 S   0.0  0.1   0:00.02 sshd              
 8707 alex      20   0 71252 1720  924 S   0.0  0.0   0:00.00 sshd              
 8708 alex      20   0 21256 4096 1712 S   0.0  0.1   0:00.05 bash              
 8723 alex      20   0 23120 1612 1188 R   0.0  0.0   0:00.06 top               
 8734 root      20   0 33032 1160  836 S   0.0  0.0   0:00.00 cron              
 8738 munin     20   0 10708 1388 1200 S   0.0  0.0   0:00.00 sh                
 8742 munin     20   0 10716 1392 1192 S   0.0  0.0   0:00.00 munin-cron        
15874 root      20   0 71192 5168 1620 S   0.0  0.1   1:20.38 fail2ban-server   
15876 root      20   0 22260  908  672 S   0.0  0.0   0:06.64 gam_server        
19427 root      30  10 12252  956  792 S   0.0  0.0   0:00.00 rsync             
20367 root      20   0 49728 4624  712 S   0.0  0.1   0:19.49 rsyslogd          
21949 root      20   0 49892 1236  628 S   0.0  0.0   0:00.81 sshd              
22815 root      20   0 20368  496  364 S   0.0  0.0   0:01.23 cron              
25118 root      20   0  122m  10m 3932 S   0.0  0.3   0:01.28 /usr/sbin/apach   
27076 root      20   0 72564 2024  732 S   0.0  0.0   0:09.70 sendmail-mta      
29022 root      20   0 42984 2976 2148 S   0.0  0.1   0:18.03 monit             

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-25 Emacs Wiki Migration

Last edit

Summary: fixing links

Changed:

< So I'm thinking, why don't you run some small benchmark (with apache turned off) to see how much of the CPU you've got? Or maybe even IO benchmark?-- AlexDaniel 2014-12-26 11:00 UTC

to

> So I'm thinking, why don't you run some small benchmark (with apache turned off) to see how much of the CPU you've got? Or maybe even IO benchmark?-- [http://alexdaniel.org/About AlexDaniel] 2014-12-26 11:00 UTC

Changed:

< -- AlexSchroeder 2014-12-26 20:19 UTC

to

> -- [[AlexSchroeder]] 2014-12-26 20:19 UTC


So I’m thinking, why don’t you run some small benchmark (with apache turned off) to see how much of the CPU you’ve got? Or maybe even IO benchmark?– AlexDaniel 2014-12-26 11:00 UTC



AlexSchroeder
Yeah. I guess I was hoping that munin does this, effectively. I’ll have to check the plugins currently enabled and check whether they do what I think they do.

AlexSchroeder 2014-12-26 20:19 UTC

Add Comment

2014-12-24 Emacs Wiki Migration

(Continued from yesterday.)

Current status:

  • Read-only server is up.
  • Monit restarts Apache practically every hour because of load.
  • As far as I am concerned, using nginx as a caching proxy isn’t as effective as some people have been suggesting.
  • I’m noticing monit restarting apache practically every hour. Is some other user on this shared host running an hourly job that borks the system? I must have misconfigured Apache such that the resulting performance dip results in a death spiral.

Here’s the munin graph that makes me suspect that I’m not getting all of the CPU I need. The gaps happen when load is so high that munin doesn’t get to run. Yikes!

https://farm8.staticflickr.com/7537/15911905077_b82b4056a0_o.png

Field
Info
system
CPU time spent by the kernel in system activities
user
CPU time spent by normal programs and daemons
nice
CPU time spent by nice(1)d programs
idle
Idle CPU time
iowait
CPU time spent waiting for I/O operations to finish when there is nothing else to do.
irq
CPU time spent handling interrupts
softirq
CPU time spent handling “batched” interrupts
steal
The time that a virtual CPU had runnable tasks, but the virtual CPU itself was not running

I think what happens is that somebody else on this server is taking up %CPU and this results in Apache processes piling up.

Here’s from the log files:

Dec 24 09:46:37 localhost monit[29022]: 'apache' trying to restart
Dec 24 09:46:37 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 09:46:38 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 71
...
Dec 24 13:02:55 localhost monit[29022]: 'apache' trying to restart
Dec 24 13:02:55 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 13:03:01 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 37
...
Dec 24 15:55:35 localhost monit[29022]: 'apache' trying to restart
Dec 24 15:55:35 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 15:55:43 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 27
...
Dec 24 16:54:16 localhost monit[29022]: 'apache' trying to restart
Dec 24 16:54:16 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 16:54:25 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 34
...
Dec 24 17:56:52 localhost monit[29022]: 'apache' trying to restart
Dec 24 17:56:52 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
...
Dec 24 21:56:02 localhost monit[29022]: 'apache' trying to restart
Dec 24 21:56:02 localhost monit[29022]: 'apache' stop: /etc/init.d/apache2
Dec 24 21:56:03 localhost sm-mta[27076]: rejecting connections on daemon MTA-v4: load average: 37

OK, so how do I make Apache more robust against those spikes?

The current settings:

# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadLimit: ThreadsPerChild can be changed to this maximum value during a
#              graceful restart. ThreadLimit can only be changed by stopping
#              and starting Apache.
# ThreadsPerChild: constant number of worker threads in each server process
# MaxClients: maximum number of simultaneous client connections
# MaxRequestsPerChild: maximum number of requests a server process serves
# <IfModule mpm_worker_module>
#     StartServers          2
#     MinSpareThreads      25
#     MaxSpareThreads      75 
#     ThreadLimit          64
#     ThreadsPerChild      25
#     MaxClients          150
#     MaxRequestsPerChild   0
# </IfModule>
<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit           3
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit         100
    ThreadsPerChild     100
    MaxClients          300
    MaxRequestsPerChild  10000
</IfModule>

Nic said, I should drop those numbers to better match the cores I have available. Each thread actually running will do IO, so that’s an additional issue.

Two cores → no more than 2 servers. When a thread waits for the disk, another thread can run. How many more threads? Not many, because they will also need to use the disk. How about a 90% reduction: 10 threads per child. Looking at this German blog post again.

  • MaxClients = ServerLimit x ThreadsPerChild. OK.
  • MinSpareThreads = ThreadsPerChild * 3 to handle spikes – but I think more threads jumping in will lead us to a death spiral. Keeping this lower than 30!
  • MaxSpareThreads ≥ MinSpareThreads + ThreadsPerChild.
<IfModule mpm_worker_module>
    StartServers          1
    ServerLimit           2
    ThreadsPerChild      10
    MaxClients           20
    MinSpareThreads      10
    MaxSpareThreads      20
    MaxRequestsPerChild  10000
</IfModule>

Hours later. It seems to have worked? Load stable, hovering around 2 – this makes sense.

https://farm8.staticflickr.com/7532/16100535652_84b896b86a_o.png

All is not perfect, unfortunately. Monit says Apache has 6h uptime. That means somebody restarted Apache during the night. Damn.

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-24 Emacs Wiki Migration

Last edit

No diff available.

Add Comment

2014-12-23 Emacs Wiki Migration

(Continued from yesterday.)

Current status:

  1. Nic Ferrier set up a dedicated machine with nginx to act as a cache for emacswiki.org
  2. I’ve switched emacswiki.org to read-only for the moment (since you can only reach the front page anyway)
  3. I’ve pointed emacswiki.org to Nic’s machine instead of mine—as soon as this change reaches you, you should be using Nic’s cache
  4. Tomorrow, I’ll remove emergency URL rewrite which redirects all wiki requests except for the main page to http://emacswiki.org/load.html

Ideally, this would mean that we’ll see a read-only copy of emacswiki.org back up, soon. Once Nic and I figure out how to handle the cookie and the ETag stuff for Emacs Wiki, we can make the wiki editable again.

Note to self:

(Continued tomorrow.)

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-23 Emacs Wiki Migration

Last edit

No diff available.

Add Comment

2014-12-22 Emacs Wiki Migration

(Continued from yesterday.)

Well… I considered making this website mod_perl in order to reduce load. This morning it was incredibly high again. When I tried it, however, I ran into problems. Error messages indicated that my script wouldn’t compile. Something about ModPerl::Registry not being available (eventhough it seemed to work for Emacs Wiki), or links on my site pointing to Emacs Wiki. Something was getting mixed up? Does mod_perl not work when used for more than one script in a virtual hosting environment? I rolled back my changes and went to do whatever one does a few years before the year. As I came back, the situation doesn’t seem too bad:

https://farm8.staticflickr.com/7483/15459696234_a9351a5831_o.png

I’d prefer load around 2, but at least the site responds, most of the time.

Well, service continues to be lousy and I’m practically out of ideas. The next step looks like hacking Oddmuse to try and reduce file accesses and hits.

In the mean time, here’s the top of Apache’s server-info.

Server Version: Apache/2.2.22 (Debian) mod_ssl/2.2.22 OpenSSL/1.0.1e mod_perl/2.0.7 Perl/v5.14.2
Server Built: Jul 24 2014 15:34:03

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Current Time: Tuesday, 23-Dec-2014 00:25:10 CET
Restart Time: Tuesday, 23-Dec-2014 00:23:07 CET
Parent Server Generation: 0
Server uptime: 2 minutes 3 seconds
Total accesses: 482 - Total Traffic: 2.2 MB
CPU Usage: u83.27 s1.06 cu18.66 cs0 - 83.7% CPU load
3.92 requests/sec - 18.0 kB/second - 4705 B/request
9 requests currently being processed, 91 idle workers

W________________K_____________R_W_______K_K_W__________________
________________R__________________K............................
................................................W...............
................................................................
............................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

That doesn’t really look very busy. So perhaps file access is the new bottle neck?

If you know some Perl, and if you know mod_perl, and you feel like you could help, please contact me!

The current setup uses a Linux OpenVZ / SolusVM VPS in Switzerland with 4096 MB RAM and 2 vCores (shared Dual Intel Xeon), 500 GB HDD (practically unused) Speicherplatz, with Debian Wheezy.

The Apache setup I mentioned previously:

<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit           3
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit         100
    ThreadsPerChild     100
    MaxClients          300
    MaxRequestsPerChild  10000
</IfModule>

The virtual server setup:

<VirtualHost *:80>
    ServerAdmin alex@emacswiki.org
    <Directory />
        Options None
        AllowOverride None
        Order Deny,Allow
        Deny from all
    </Directory>
    ServerName www.emacswiki.org
    ServerAlias emacswiki.org
    DocumentRoot /home/alex/emacswiki.org
    <Directory /home/alex/emacswiki.org>
        Options ExecCGI Includes Indexes SymLinksIfOwnerMatch
	AddHandler cgi-script .pl
        AllowOverride All
        Order Allow,Deny
        Allow from all
    </Directory>
    <Files emacs.pl>
       Options ExecCGI SymLinksIfOwnerMatch
       SetHandler perl-script
       PerlHandler ModPerl::Registry
    </Files>
    RewriteLog "/home/alex/logs/emacswiki.org/rewrite.log"
    RewriteLogLevel 0
    CustomLog /home/alex/logs/emacswiki.org/access.log combined
    ErrorLog  /home/alex/logs/emacswiki.org/error.log
    # LogLevel debug
    CustomLog /var/log/apache2/access.log vhost_combined
    ServerSignature On
</VirtualHost>

I run monit and munin for monitoring.

And now we’re getting into Oddmuse details. If you need more information in order to setup up a test environment, I’ll be happy to provide anything you need.

#! /usr/bin/perl
package OddMuse;
use CGI;
$q = new CGI;
$RunCGI = 0;
$UseConfig = 0; # no config!
$DataDir = '/home/alex/emacswiki';
if (not $config_loaded) {
  do '/home/alex/emacswiki.org/current.pl';
  do '/home/alex/emacswiki/modules/concat.pl';
  do '/home/alex/emacswiki/config';
  $config_loaded = 1;
}

# make sure traces of other language configs disappear
%Translate=();
$HomePage    = "SiteMap";
$RCName      = 'RecentChanges';
@UserGotoBarPages = ($HomePage, 'Search',
		     'ElispArea', 'HowTo', 'Glossary',
		     $RCName, 'News', 'Problems', 'Suggestions');
$EditNote = '<p>Please make sure you contribute only your own work. IP numbers are made available via the page history. If you want to keep it a secret, you need to use <a href="http://www.torproject.org/">Tor</a>. See <a href="/emacs/TextFormattingRules">TextFormattingRules</a> for how to format text. See <a href="/emacs/StyleGuide">StyleGuide</a> for the suggested writing style on this wiki.</p>';

# language specific files should only set variables, never redefine subroutines
$ConfigFile = "$DataDir/config-en";
# my $requested_language = "da, en-gb;q=0.8, en;q=0.7";
my $requested_language = $q->http('Accept-language');
my @languages = split(/ *, */, $requested_language);
my %Lang = ();
foreach $_ (@languages) {
  my $qual = 1;
  $qual = $1 if (/q=([0-9.]+)/);
  $Lang{$qual} = $1 if (/^([-a-z]+[a-z])/);
}
my @prefs = sort { $b <=> $a } keys %Lang;
# my $html = "input: $requested_language" . "\nResult: " . join(', ', map { "$_ ($Lang{$_})" } @prefs);
# print header, start_html, pre($html), end_html; exit;
foreach $_ (@prefs) {
  if (-r "$DataDir/config-$Lang{$_}") {
    $ConfigFile = "$DataDir/config-$Lang{$_}";
    last;
  }
}
do $ConfigFile;
# Do the wiki
DoWikiRequest();

(Continued tomorrow.)

Tags: RSS RSS RSS RSS RSS RSS RSS

Comments on 2014-12-22 Emacs Wiki Migration

Last edit

Summary: I'm not sure. In my .htaccess, I already provide for long browser-side caching of static files: {{{ ExpiresActive On ExpiresDefault "access plus 1 hours" ExpiresByType imagepng "access plus 3 days" ExpiresByType imagejpeg "access plus 3 days" ExpiresByType imagesvg+xml "access plus 3 days"…

Changed:

< It's worth a try, I guess. A short skimming of [http://www.thomas-krenn.com/de/wiki/Varnish_Web_Beschleuniger an intro page] I found made me think that using it for wiki pages is going to involve some more work. Do you have some detailed instructions somewhere, or can you help in configuring Varnish for Oddmuse? The [https://www.varnish-cache.org/trac/wiki/ETags Varnish wiki page on ETags] doesn't look too convincing.

to

> It's worth a try, I guess. A short skimming of [http://www.thomas-krenn.com/de/wiki/Varnish_Web_Beschleuniger an intro page] I found made me think that using it for wiki pages is going to involve some more work. Do you have some detailed instructions somewhere, or can you help in configuring Varnish for Oddmuse? The [https://www.varnish-cache.org/trac/wiki/ETags Varnish wiki page on ETags] looks intimidating.


Completely unrelated, but if the majority of the traffic is anonymous, I would suggest installing Varnish

– Anonymous 2014-12-23 19:06 UTC



Alex Schroeder
I’m not sure. In my .htaccess, I already provide for long browser-side caching of static files:

ExpiresActive On
ExpiresDefault "access plus 1 hours"
ExpiresByType image/png "access plus 3 days"
ExpiresByType image/jpeg "access plus 3 days"
ExpiresByType image/svg+xml "access plus 3 days"
ExpiresByType text/css "access plus 3 days"

And the wiki itself uses ETags the way they were meant to be used, I think. See the wiki page on caching, if you’re interested. It’s true that returning the 304 code requires the wiki code to run, but I’m still saving on all the rendering. It’s worth a try, I guess. A short skimming of an intro page I found made me think that using it for wiki pages is going to involve some more work. Do you have some detailed instructions somewhere, or can you help in configuring Varnish for Oddmuse? The Varnish wiki page on ETags looks intimidating.

– Alex Schroeder 2014-12-23 20:33 UTC

Add Comment

2014-12-21 Emacs Wiki Migration

(Continued from here.)

Memory problems continue. The VPS control panel says “Memory Usage: 98%. 1.96 GB of 2 GB Used / 38.52 MB Free” for example. The numbers change all the time. I thought perhaps reducing the number of Apache processes would help.

I changed /etc/apache2/apache2.conf to say:

<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       2
    MaxSpareServers       4
    MaxClients          150
    MaxRequestsPerChild 10000
</IfModule>

The numbers used to be 5, 5, 10, I think. When I use ps aux|grep sbin/apache I see four processes, now. So, did things improve? No. 😭

When I’m running top, I can watch as one of the apache processes starts with around 8% of memory and grows and grows and within about ten seconds, %MEM approaches 100% and then the process disappears. Killed, I presume. As far as I can tell, this would happen even if I had just one process running.

top - 12:15:57 up 11:13,  1 user,  load average: 2.71, 3.33, 3.16
Tasks:  25 total,   1 running,  21 sleeping,   2 stopped,   1 zombie
%Cpu(s): 57.1 us, 25.5 sy,  0.0 ni, 16.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.5 st
KiB Mem:   2097152 total,  2088764 used,     8388 free,        0 buffers
KiB Swap:   524288 total,   496028 used,    28260 free,       80 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 1550 www-data  20   0 2589m 1.9g 1704 S  89.7 97.0   0:06.34 /usr/sbin/apach
 1583 www-data  20   0  485m  18m 1704 S   8.6  0.9   0:00.26 /usr/sbin/apach
    1 root      20   0 10608  144   40 S   0.0  0.0   0:00.64 init
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kthreadd/2828
    3 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khelper/2828
 1612 www-data  20   0     0    0    0 Z   0.0  0.0   0:00.14 wiki.pl
 1790 messageb  20   0 29764  244    4 S   0.0  0.0   0:00.00 dbus-daemon

How can this be? It looks like a memory leak to me! I need to figure out how to debug this for my mod_perl setup. Perhaps I can start by setting MaxRequestsPerChild to a really low number? This morning, this parameter wasn’t set at all. Then I read Tuning an Apache server in 5 minutes and set it to 10000. I think I’ll set it to 100 and see how it goes.

top - 12:27:09 up 11:25,  1 user,  load average: 3.48, 2.99, 2.99
Tasks:  24 total,   1 running,  21 sleeping,   2 stopped,   0 zombie
%Cpu(s): 35.2 us, 14.4 sy,  0.0 ni, 50.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   2097152 total,  2087168 used,     9984 free,        0 buffers
KiB Swap:   524288 total,   509824 used,    14464 free,       72 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 4547 www-data  20   0 2527m 1.9g 1708 S  66.8 96.9   0:04.95 /usr/sbin/apach
 4575 www-data  20   0  420m  17m 1708 S   5.7  0.9   0:00.17 /usr/sbin/apach
 4363 alex      20   0 23124  724  596 R   0.3  0.0   0:00.05 top

Did things improve? No. 😳

OK, I rewrote the initialization of the wiki. I had an idea…

top - 12:57:43 up 11:55,  1 user,  load average: 9.53, 5.91, 3.88
Tasks:  30 total,   2 running,  26 sleeping,   2 stopped,   0 zombie
%Cpu(s): 98.7 us,  0.0 sy,  1.3 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   2097152 total,   470128 used,  1627024 free,        0 buffers
KiB Swap:   524288 total,   404140 used,   120148 free,   106960 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
11517 www-data  20   0  580m 120m 4640 S  49.2  5.9   4:08.00 /usr/sbin/apach
11547 www-data  20   0  567m 108m 4532 S  49.2  5.3   1:48.72 /usr/sbin/apach

Wow, memory consumption is really low, now. But load problems are back. 😥

Since memory consumption is low, I’m trying to use more processes again.

<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       2
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild 10000
</IfModule>

The output of top seems to indicate that the number of processes has no effect on %CPU. Basically I have two CPUs available, and each is running an apache process… There’s not much else to be gained unless these processes spend their time waiting for I/O. We’ll see.

An hour later:

top - 14:44:16 up 13:42,  1 user,  load average: 1.97, 1.92, 1.97
Tasks:  25 total,   1 running,  22 sleeping,   2 stopped,   0 zombie
%Cpu(s): 40.5 us, 14.1 sy,  0.0 ni, 45.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem:   2097152 total,  1457144 used,   640008 free,        0 buffers
KiB Swap:   524288 total,   416852 used,   107436 free,     4400 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
14628 www-data  20   0 1858m 1.3g 3956 S  87.2 65.6   0:02.78 /usr/sbin/apach
14656 www-data  20   0  445m  43m 3468 S  10.0  2.1   0:00.30 /usr/sbin/apach
16857 www-data  20   0  939m 1532  140 S   0.0  0.1   0:00.00 /usr/sbin/apach
21618 www-data  20   0  494m 2936  132 S   0.0  0.1   0:00.00 /usr/sbin/apach
21619 www-data  20   0  494m 2952  132 S   0.0  0.1   0:00.00 /usr/sbin/apach

Load seems to be under control. Memory, not so much. It keeps fluctuating. I guess whatever I thought I had fixed, wasn’t actually helping? 😞

Another fix, another moment of hope.

top - 15:00:48 up 13:58,  1 user,  load average: 1.14, 1.10, 1.42
Tasks:  25 total,   1 running,  22 sleeping,   2 stopped,   0 zombie
%Cpu(s): 91.2 us,  1.8 sy,  0.0 ni,  6.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem:   2097152 total,   331668 used,  1765484 free,        0 buffers
KiB Swap:   524288 total,   404936 used,   119352 free,    85024 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
16085 www-data  20   0  568m 101m 4588 S  75.2  5.0   5:32.24 /usr/sbin/apach
17497 www-data  20   0  574m  52m 4276 S  43.9  2.6   0:03.15 /usr/sbin/apach
16114 www-data  20   0  527m  52m 4832 S   0.3  2.6   0:31.14 /usr/sbin/apach
    1 root      20   0 10608  196   60 S   0.0  0.0   0:00.81 init

Gaaaah. It dpkg-query -l apache2 indicates that I’m not using the prefork server but the worker! All those times I thought I was changing the parameters, I guess I wasn’t.

alex@kallobombus:~$ dpkg-query -l apache2*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name              Version       Architecture  Description
+++-=================-=============-=============-========================================
ii  apache2           2.2.22-13+deb amd64         Apache HTTP Server metapackage
un  apache2-common    <none>                      (no description available)
un  apache2-doc       <none>                      (no description available)
un  apache2-mpm       <none>                      (no description available)
un  apache2-mpm-event <none>                      (no description available)
un  apache2-mpm-itk   <none>                      (no description available)
un  apache2-mpm-prefo <none>                      (no description available)
ii  apache2-mpm-worke 2.2.22-13+deb amd64         Apache HTTP Server - high speed threaded
un  apache2-suexec    <none>                      (no description available)
un  apache2-suexec-cu <none>                      (no description available)
ii  apache2-utils     2.2.22-13+deb amd64         utility programs for webservers
ii  apache2.2-bin     2.2.22-13+deb amd64         Apache HTTP Server common binary files
ii  apache2.2-common  2.2.22-13+deb amd64         Apache HTTP Server common files

Given that currently load is hovering between 45 and 50, what should I do? I changed MaxRequestsPerChild from 0 to 10000.

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild  10000
</IfModule>

When I restarted apache2 just now, I noticed how load has dropped below 10. Weird! Why was it so high?

Out of curiosity, I wanted to know whether it is indeed emacswiki.org that is smashing my server. And I guess it is. But other sites, such as my homepage, aren’t doing too badly, either. Perhaps I need to mod_perl them, too!

bin/site-comparison < /var/log/apache2/access.log
                          Site       hits bandw. hits% interv. status code distrib.
             www.emacswiki.org      65092    10K  74%    1.0s  200 (61%), 301 (19%), 500 (7%)
          www.alexschroeder.ch       7877    30K   9%    8.3s  200 (90%), 304 (3%), 404 (1%),
         www.communitywiki.org       7531    15K   8%    8.7s  200 (81%), 301 (8%), 404 (4%),
               www.oddmuse.org       3389    21K   3%   19.4s  200 (69%), 301 (12%), 404 (8%)
          www.campaignwiki.org       3302    11K   3%   19.9s  200 (78%), 404 (14%), 302 (5%)
         www.orientalisch.info        136     4K   0%  481.8s  200 (73%), 301 (14%), 304 (8%)
       www.arabisch-lernen.org         60    22K   0% 1102.9s  200 (90%), 404 (5%), 304 (3%),

I’ve been reading a German blog post about Apache tuning and decided to add a ServerLimit:

<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit           3
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild	 50
    MaxClients          150
    MaxRequestsPerChild  10000
</IfModule>

Less processes would seem to be the way to reduce load? I’m not sure…

Right now, my homepage is incredibly slow and Emacs Wiki is—sometimes—very fast. That’s cool, I guess. mod_perl really seems to help. Perhaps it’s time to start investigating all the remaining Emacs Wiki issues?

As I’m looking at the output of ps, I noticed something else that’s weird: There are several apache processes which don’t do anything (apparently) and they were started a long time ago. The last apache2 restart I ran was around 23:00. I guess I’ll send them a kill signal. Let’s see where that takes us.

root@kallobombus:/home/alex# ps -u www-data ao pid,ppid,%cpu,%mem,vsz,rss,stat,start,time,cmd
  PID  PPID %CPU %MEM    VSZ   RSS STAT  STARTED     TIME CMD
16857     1  0.0  0.0 962520  1536 S    04:20:18 00:00:00 /home/alex/emacsw -k start
21618     1  0.0  0.1 506716  2940 S    02:19:01 00:00:00 /usr/sbin/apache2 -k start
21619     1  0.0  0.1 506724  2956 S    02:19:01 00:00:00 /usr/sbin/apache2 -k start
22110 22109  0.0  0.2  21340  4312 Ss   21:42:45 00:00:00 -bash
26046 22110  0.0  0.0  33584  1872 S    22:23:58 00:00:00 sudo su
26051 26046  0.0  0.0  37112  1368 S    22:24:08 00:00:00 su
26052 26051  0.0  0.1  20488  3484 S    22:24:09 00:00:00 bash
28681 28668 78.5  5.8 973236 123176 Sl  23:00:34 00:04:36 /usr/sbin/apache2 -k start
29401     1  0.0  0.1  55720  2216 S    16:37:12 00:00:01 /usr/bin/perl /home/alex/alexschroeder.ch/wiki.pl
29423     1  0.0  0.0 486056   192 S    16:37:46 00:00:00 /home/alex/emacsw -k start
30429 26052  0.0  0.0  16796  1264 R+   23:06:26 00:00:00 ps -u www-data ao pid,ppid,%cpu,%mem,vsz,rss,stat,sta

I wonder why load is so high. A disk latency issue?

(Continued tomorrow.)

Tags: RSS RSS RSS RSS

Comments on 2014-12-21 Emacs Wiki Migration

Last edit

No diff available.

Add Comment

More...

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.