2015-10-09 Oddmuse and Mojolicious

I’m trying to run Oddmuse within a Perl web framework: Mojolicious using Mojolicious::Plugin::CGI. It won’t be as perfect as a true Mojolicious app, but it will still be much faster than a simple CGI script. When running as a CGI script, every request loads Perl and compiles all the modules – including the CGI module itself – and Oddmuse and the config files. When running under Mojolicious, we no longer load Perl and we compile Oddmuse just once. Oddmuse itself will keep loading the config file and all that, but it’s still much better than before. The Mojolicious app itself is then started by Toadfarm. And in order to force myself to test it, I’ve switched this wiki over to the new setup!

First, tell Apache to act as a reverse proxy and pass all /wiki requests to the new server. The two ProxyPass instructions at the bottom are the important bits:

<VirtualHost *:80>
    ServerName alexschroeder.ch
    ServerAlias www.alexschroeder.ch
    Redirect permanent / https://alexschroeder.ch/
</VirtualHost>
<VirtualHost *:443>
    ServerAdmin alex@alexschroeder.ch
    <Directory />
        Options None
        AllowOverride None
        Order Deny,Allow
        Deny from all
    </Directory>
    ServerName alexschroeder.ch
    ServerAlias www.alexschroeder.ch
    ServerAlias rpg.alexschroeder.ch
    DocumentRoot /home/alex/alexschroeder.ch
    <Directory /home/alex/alexschroeder.ch>
        Options ExecCGI Includes Indexes MultiViews SymLinksIfOwnerMatch
        AddHandler cgi-script .pl
        AllowOverride All
        Order Allow,Deny
        Allow from all
    </Directory>

    SSLEngine on
    SSLCertificateFile      /home/alex/ssl/alexschroeder.crt
    SSLCertificateKeyFile   /home/alex/ssl/alexschroeder.key
    SSLCertificateChainFile /home/alex/ssl/GandiStandardSSLCA2.pem
    SSLVerifyClient None

    ProxyPass /wiki             https://alexschroeder.ch:8080/wiki
    ProxyPass /mojo             https://alexschroeder.ch:8080/mojo

</VirtualHost>

Toadfarm setup in ~/farm/farm:

#!/usr/bin/env perl
use Toadfarm -init;

logging {
  combined => 1,
  file     => "farm.log",
  level    => "info",
};

1. other apps go here...

mount "$farm/alexschroeder.pl" => {
  "Host" => qr{^(www.)?alexschroeder\.ch$},
  mount_point => '/wiki',
};

plugin "Toadfarm::Plugin::AccessLog";

start; # needs to be at the last line

The above means that all requests to htpp://alexschroeder.ch:8080/wiki and htpp://www.alexschroeder.ch:8080/wiki will be handled by Oddmuse.

Mojolicious wrapper in ~/farm/alexschroeder.pl:

#! /usr/bin/env perl

use Mojolicious::Lite;

plugin CGI => {
  support_semicolon_in_query_string => 1,
};

plugin CGI => {
  route => '/',
  script => 'wiki.pl', # ~/farm/wiki.pl
  env => {WikiDataDir => '/home/alex/alexschroeder'},
  errlog => 'alexschroeder.log', # path to where STDERR from cgi script goes
};

app->start;

In this case, wiki.pl is simply the original Oddmuse script, unchanged. :)

Tags:

Comments

Cool! That’s interesting.

What about side effects? Like, if you forget to initialize some variable somewhere, what is going to happen? (Thinking about mod_perl and all stuff related to it)

Also, I miss one important thing from your report... What about page load time? How much faster is it right now, comparing to regular cgi script?

– AlexDaniel 2015-10-09 11:21 UTC


I did not measure page load time. I also think that it keeps forking to run the script, so globals start empty every time? Not sure about the details. I haven’t seen any side effects, yet. Let me know if you see anything!

– Alex Schroeder 2015-10-09 11:39 UTC


Could you be so kind to measure the load time?

– AlexDaniel 2015-10-09 11:50 UTC


I think I was under the illusion that it was working. Now it is working. Hah! I had to rename my old wrapper script to realize that my ProxyPass instructions had been wrong.

Things to remember:

Check the config file for weird stuff.

1. if ($ENV{SERVER_PORT} == 8080) {
1.   $ScriptName  = 'https://alexschroeder.ch:8080/wiki';
1.   $FullUrl     = 'https://alexschroeder.ch:8080/wiki';
1. }

Make sure the user running toadfarm can write alle the files in your data directory. In my case, the data directory belongs to www-data.alex but there were temp files that alex could not write.

– Alex Schroeder 2015-10-09 13:27 UTC


Hm. Current setup, I get more or less the same time when I look at network analysis with my browser when I’m loading /wiki/About and when I’m loading /wiki2/About with wiki2.pl being my old wiki.pl wrapper script. I get numbers between 200ms and 700ms. I do note that /wiki/About keeps getting me a 200 OK response instead of a 304 NOT MODIFIED. I’m not quite sure what the problem is.

– Alex Schroeder 2015-10-09 13:40 UTC


Oh, and the code shows that our script is being exec’d. I’m not sure about the headers. I think if our script prints HTTP headers, we should be fine.

– Alex Schroeder 2015-10-09 13:47 UTC


Well, at least it seems to work right now. I think I got confused about the various restarts required.

The missing 304 NOT MODIFIED remains a mistery, however. I’ve tried $q = new CGI; $q->nph(1) in the config file, $CGI::NPH = 1 in the config file, CGI::nph(1) in the config file, use CGI qw/-utf8 -nph/ in the core script, and print $q->header(-nph=>1, -status=>'304 NOT MODIFIED') and return if PageFresh(); in the core script, reloading my toadfarm between edits and found no change.

When I tried to export MOJO_PLUGIN_CGI_DEBUG=1 before starting the toadfarm, I got a 503 error when visiting the site.

This is very annoying.

– AlexSchroeder 2015-10-09 14:32 UTC


OK, so the question is: How often is 304 NOT MODIFIED actually used?

alex@kallobombus:~/farm$ sudo wc -l /var/log/apache2/access.log.1
259852 /var/log/apache2/access.log.1
alex@kallobombus:~/farm$ sudo grep ^oddmuse /var/log/apache2/access.log.1 | wc -l
32867
alex@kallobombus:~/farm$ sudo grep "^oddmuse.*GET /wiki.* 304 " /var/log/apache2/access.log.1 | grep -v feed | wc -l
270

Filtering out feed request we’re down to 8‰.

For my own site:

alex@kallobombus:~/farm$ sudo grep ^alexschroeder /var/log/apache2/access.log.1 | wc -l
79286
alex@kallobombus:~/farm$ sudo grep "^alexschroeder.*GET /wiki.* 304 " /var/log/apache2/access.log.1 | grep -v feed | wc -l
702

This reduces the percentage to 9‰.

So, if we redirect feed requests to files and use a cron job to produce some of them, the 304 NOT MODIFIED response seems to be something we can do without, even if I think it ought to work.

– AlexSchroeder 2015-10-09 14:54 UTC


In other words, no noticeable performance boost? Do I read it right? But then why bother?

– AlexDaniel 2015-10-09 21:26 UTC


Hehe, indeed. The point is, however, that I want some sort of plan that will allow me to incrementally develop a solution. Mojolicious::Plugin::CGI does a lot of what I want. It integrates into Mojolicious and thus it can do logging, forking, and so on. It handles input and output. The only thing that’s missing is that it exec’s the script – loading Perl and compiling all the modules on every request.

I think I’ll try and modify the Plugin for my purposes. Then it’ll require Oddmuse once and just run DoWikiRequest on every request, mod_perl style.

– AlexSchroeder 2015-10-10 07:40 UTC


These are the changes I had to make to Mojolicious::Plugin::CGI: Add run option for a code reference and Make sure the status code is actually used and Add HTTP_IF_NONE_MATCH to environment.

This is the new server script I used:

#! /usr/bin/env perl

use lib '/Users/alex/src/mojolicious-plugin-cgi/lib';

use Mojolicious::Lite;

plugin CGI => {
  support_semicolon_in_query_string => 1,
};
 
plugin CGI => {
  route => '/wiki',
  script => 'wiki.pl',
  run => \&OddMuse::DoWikiRequest,
  before => sub {
    $OddMuse::RunCGI = 0;
    $OddMuse::DataDir = '/tmp/oddmuse';
    require 'wiki.pl' unless defined &OddMuse::DoWikiRequest;
  },
  env => {},
  errlog => 'wiki.log', # path to where STDERR from cgi script goes
};

get '/' => sub {
  my $self = shift;
  $self->redirect_to('/wiki');
};
 
app->start;

– AlexSchroeder 2015-10-11 06:51 UTC


With the above setup now installed on alexschroeder.ch, I’m getting response times of 150–230ms when looking at /wiki/About (and a 304 Not Modified response). The old CGI setup under /wiki2/About is giving response times of 250–720ms.

So, much better?

I’ve also started noticing the first signs of trouble: Edit page, save, look at it (looks good), edit page again and notice that it’s showing the old text, not the new text.

– AlexSchroeder 2015-10-11 10:28 UTC


When I tried it with this page, I saw my browser requesting the page using the following:

GET /wiki?action=edit;id=Comments_on_2015-10-09_Oddmuse_and_Mojolicious HTTP/1.1
...
If-None-Match: 1444395217%1e1%1eAlexSchroeder%1ekensanata%40gmail.com

Oddmuse replied with a 304 Not Modified and the content was wrong. As I checked, however, the timestamp of the index file does in fact match:

alex@kallobombus:~/alexschroeder$ date -r pageidx +%s
1444395217

Looking at the log of recent changes, however:

alex@kallobombus:~/alexschroeder$ tail -n 1 rc.log
1444559479Comments_on_2015-10-09_Oddmuse_and_Mojolicious1With the above setup now installed on alexschroeder.ch, I'm getting response times of 150–230ms when looking at wikiAbout (and a 304 Not Modified response). So, much better? 127.0.0.1AlexSchroeder23en

I guess something about TouchIndexFile isn’t working.

sub TouchIndexFile {
  my $ts = time;
  utime $ts, $ts, $IndexFile;
  $LastUpdate = $Now = $ts;
}

I’ve made sure that all the files belong to alex.alex instead of www-data.alex. And that seems to help. After saving:

alex@kallobombus:~/alexschroeder$ date -r pageidx +%s
1444560771
alex@kallobombus:~/alexschroeder$ tail -n 1 rc.log
1444560771Comments_on_2015-10-09_Oddmuse_and_MojoliciousWhen I tried it with this page, I saw my browser requesting the page using the following: {{{ GET wiki?action=edit;id=Commentson2015-10-09OddmuseandMojolicious HTTP1.1 … If-None-Match: 1444395217%1e1%1eAlexSchroeder%1ekensanata%40gmail.com }}} Oddmuse replied with a 304 Not Modified and the content…127.0.0.1AlexSchroeder24en

And the next request:

GET /wiki?action=edit;id=Comments_on_2015-10-09_Oddmuse_and_Mojolicious HTTP/1.1
...
If-None-Match: 1444395217%1e1%1eAlexSchroeder%1ekensanata%40gmail.com

HTTP/1.1 200 OK
...
Etag: 1444560771%1e1%1eAlexSchroeder%1ekensanata%40gmail.com
...

Yay?

– AlexSchroeder 2015-10-11 10:52 UTC


OK, added one final patch to make it work without having to use the -nph option for the CGI library and submitted a pull request. And I installed it for this site as well. “Eat your own dog food,” and all that.

– AlexSchroeder 2015-10-11 12:41 UTC


Wow, many more commits after talking to the maintainer.

– AlexSchroeder 2015-10-11 16:14 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.