2017-05-18 Emacs Wiki Down

Sadly it seems that Emacs Wiki is down. Currently Nic Ferrier is paying for the servers, so I sent him an email but haven’t heard back. So I’m thinking of resurrecting the site on my own servers.

I still remember the fiasco that started with 2014-12-18 Emacs Wiki Migration.

These are my notes. Perhaps they’re useful in case I have to restore another backup, or they might be useful to you if you want to fork Emacs Wiki.

First, restore the backups from Chile provided by zeus. Thanks, man!

rsync -az kensanata@theshire.emacs.cl:emacswiki.org .
rsync -az kensanata@theshire.emacs.cl:emacswiki .

I want to run the script from my Mojolicious Toadfarm. I added the following lines to my Toadfarm setup:

# emacswiki.org

mount "$farm/emacswiki.pl" => {
  "Host" => qr{^emacswiki\.org:8080$},
  mount_point => '/wiki',
};

And this is the Mojolicious CGI Plugin wrapper, emacswiki.pl:

#! /usr/bin/env perl

use Mojolicious::Lite;

plugin CGI => {
  support_semicolon_in_query_string => 1,
};

plugin CGI => {
  route => '/',
  script => '/home/alex/farm/wiki.pl', # not necessary
  errlog => '/home/alex/farm/emacswiki.log',
  run => \&OddMuse::DoWikiRequest,
  before => sub {
    no warnings;
    $OddMuse::RunCGI = 0;
    $OddMuse::DataDir = '/home/alex/emacswiki';
    require '/home/alex/farm/wiki.pl';
  },
};

app->start;

In order for this to work, I need an Apache site. I created /etc/apache2/sites-available/500-emacswiki.org.conf with the following:

<VirtualHost *:80>
    ServerName emacswiki.org
    ServerAlias www.emacswiki.org
    Redirect permanent / https://emacswiki.org/
</VirtualHost>
<VirtualHost *:443>
    ServerAdmin alex@emacswiki.org
    ServerName emacswiki.org
    ServerAlias www.emacswiki.org
    DocumentRoot /home/alex/emacswiki.org
    <Directory /home/alex/emacswiki.org>
        Options ExecCGI Includes Indexes MultiViews SymLinksIfOwnerMatch
	# legacy CGI scripts like info-ref.pl                                                                                                                
        AddHandler cgi-script .pl
        AllowOverride All
        Require all granted
    </Directory>

    SSLEngine on
    SSLCertificateFile      /etc/letsencrypt.sh/certs/emacswiki.org/cert.pem
    SSLCertificateKeyFile   /etc/letsencrypt.sh/certs/emacswiki.org/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt.sh/certs/emacswiki.org/chain.pem
    SSLVerifyClient None

    ProxyPass /wiki             http://emacswiki.org:8080/wiki
    ProxyPass /mojo             http://emacswiki.org:8080/mojo

</VirtualHost>

I remove all the *.pl files except for info-ref.pl in the emacswiki.org directory.

Reloded the farm using ./farm reload. Check the log file for Mounting emacswiki with conditions.

Activating the site using sudo a2ensite 500-emacswiki.org.

Check the config using sudo apachectl configtest. Oops! This is an obvious erorr, of course: SSLCertificateFile: file '/etc/letsencrypt.sh/certs/emacswiki.org/cert.pem' does not exist or is empty.

I need to get the SSL certificates, too.

I added emacswiki.org www.emacswiki.org to /etc/letsencrypt.sh/domains.txt and ran /etc/letsencrypt.sh/letsencrypt.sh -c but that doesn’t work. I guess it doesn’t work because the name emacswiki.org still points to the old server. I guess for the moment I’ll try to do without HTTPS.

So this is what I’ll be using instead for the site:

<VirtualHost *:80>
    ServerName emacswiki.org
    ServerAlias www.emacswiki.org
    ServerAdmin alex@emacswiki.org
    DocumentRoot /home/alex/emacswiki.org
    <Directory /home/alex/emacswiki.org>
        Options ExecCGI Includes Indexes MultiViews SymLinksIfOwnerMatch
        # legacy CGI scripts like info-ref.pl                                                                                                                
        AddHandler cgi-script .pl
        AllowOverride All
        Require all granted
    </Directory>

    ProxyPass /emacs            http://emacswiki.org:8080/wiki
    ProxyPass /wiki             http://emacswiki.org:8080/wiki
    ProxyPass /mojo             http://emacswiki.org:8080/mojo

</VirtualHost>

Now sudo apachectl configtest says Syntax OK.

Reloaded Apache using sudo service apache2 reload.

Added a line to my /etc/hosts file:

192.71.233.105  emacswiki.org

Testing w3m http://emacswiki.org/wiki and w3m http://emacswiki.org/wiki seems to work!

Better make the wiki read-only: touch ~/emacswiki/noedit.

Following links doesn’t work. w3m http://www.emacswiki.org/emacs/HowToNavigate tells me: Can't load http://www.emacswiki.org/emacs/HowToNavigate. The problem is that Apache has www.emacswiki.org as a server alias, but the Toadfarm only listens for emacswiki.org.

Change that:

# emacswiki.org                                                                                                                                              

mount "$farm/emacswiki.pl" => {
  "Host" => qr{^(www\.)?emacswiki\.org:8080$},
  mount_point => '/wiki',
};

And reload: ./farm reload.

That didn’t work. Hah, of course not. I need to add www.emacswiki.org to my /etc/hosts, of course!

Now it works.

OK, next problem: Why does w3m http://www.emacswiki.org/ give me the directory listing? Surely I’m missing my .htaccess file. Is it not being read? The /var/log/apache2/error.log file has not suspicious. Well, it does mention something about the emacswiki.org/rss directory but I just deleted it. Are the permissions wrong? I did a chmod g-w .htaccess just to be sure and now it says:

-rw-r--r-- 1 alex alex 1955 May 29  2016 emacswiki.org/.htaccess

This looks correct to me.

In there, it says DirectoryIndex emacs.pl. Ah. That might be a problem because I removed that script. Changing that to DirectoryIndex emacs did the job!

OK, so anybody who has access to their own /etc/hosts file can now access a read-only copy of the site.

Here’s what I have planned:

  1. change the DNS entry ✓
  2. see how the site explodes 🔥🔥🔥
  3. add HTTPS

When I tried to add a News page, I noticed that I was unable to get the wiki back into writeable mode. I had to remove the noedit file I had created earlier using rm ~/emacswiki/noedit.

Then, when I tried to save, the wiki complained about some page that looked like spam not being readable and I figured that the page index must have been out of sync so I simply removed it using rm ~/emacswiki/pageidx.

And finally I recreated the lock using touch ~/emacswiki/noedit.

OK, now I’m waiting for the DNS change to spread and watching my Munin graphs.

Also, all the people with HTTPS bookmarks will get errors like the following: Bad cert ident from emacswiki.org: dNSName=alexschroeder.ch rpg.alexschroeder.ch www.alexschroeder.ch : accept? (y/n). That’s because emacswiki.org is currently longer listening on port 443 and the default site is alexschroeder.ch. Oh well! In a few hours I’m hoping that Let’s Encrypt will allow me to regenerate certificates for Emacs Wiki and then we’ll move to HTTPS.

Hours later, I checked again and HTTP access was working. So I ran sudo /etc/letsencrypt.sh/letsencrypt.sh -c to get the certificates and this time it worked. I reverted the changes to the site config file /etc/apache2/sites-available/500-emacswiki.org.conf and we’re no using this:

<VirtualHost *:80>
    ServerName emacswiki.org
    ServerAlias www.emacswiki.org
    Redirect permanent / https://emacswiki.org/
</VirtualHost>
<VirtualHost *:443>
    ServerAdmin alex@emacswiki.org
    ServerName emacswiki.org
    ServerAlias www.emacswiki.org
    DocumentRoot /home/alex/emacswiki.org
    <Directory /home/alex/emacswiki.org>
        Options ExecCGI Includes Indexes MultiViews SymLinksIfOwnerMatch
	# legacy CGI scripts like info-ref.pl
	AddHandler cgi-script .pl
        AllowOverride All
	Require all granted
    </Directory>

    SSLEngine on
    SSLCertificateFile      /etc/letsencrypt.sh/certs/emacswiki.org/cert.pem
    SSLCertificateKeyFile   /etc/letsencrypt.sh/certs/emacswiki.org/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt.sh/certs/emacswiki.org/chain.pem
    SSLVerifyClient None

    ProxyPass /emacs            http://emacswiki.org:8080/wiki
    ProxyPass /wiki             http://emacswiki.org:8080/wiki
    ProxyPass /mojo             http://emacswiki.org:8080/mojo

</VirtualHost>

Notice that both /emacs and /wiki will work. Is this a bad idea? sudo apachectl configtest says the changes are good and so I ran sudo service apache2 reload. Everthing seems to be working!

What about load? It’s definitely going up! :(

OK, time to read up on mod_cache. I think I want something like the following:

# Turn on caching
CacheSocache shmcb
CacheSocacheMaxSize 102400
<Location "/emacs">
    CacheEnable socache
</Location>

Well, before diving into this, I think we should just monitor how load develops over the next few hours.

A few hours later it would seem to me that there are no open issues so there is no need for extra caching.

And that also means, I can try and make the website editable again.

Let’s see, what else do we need to check?

  1. does git work?
  2. what about cronjobs?

As for git, this is simple. I created the page 2017-05-18 and I expect to see it on the emacsmirror master branch. Sadly, it isn’t there. Why not?

Let’s take a look:

alex@sibirocobombus:~/emacswiki/git$ git log
commit a08f867084896e9892d148f76a54976166cd75db
Author: Alex Schroeder <anybody@emacswiki.org>
Date:   Thu May 18 13:56:01 2017 +0200

    Backup site!

Oops! Apparently, the git repository wasn’t checked out. It makes sense, actually. But now I need to fix this. git remote -v shows no remotes. Let’s add it, and fetch data. This works because my public key is already part of the emacswirror org on GitHub.

git remote add origin git@github.com:emacsmirror/emacswiki.org.git
git fetch
git branch -m master temp
git checkout master

At this point it should tell you Branch master set up to track remote branch master from origin.

git cherry-pick a08f86
git push
git branch -D temp

OK. Time to test it! In order to be able to save, I now have to change the site URL in the config file back to HTTPS. It should read

my $root = "https://www.emacswiki.org";

I made the page edit and that seems to do the trick. git log in the git directory lists the new edit.

This brings me to the next part: cron jobs. Somebody has to push those commits, right?

  1. I added emacswiki and emacswiki.org to the shell script that uses rsync to store daily backups in Chile. Thanks again, zeus!
  2. I found an old emacs-git-update in my bin directory and added an appropriate entry to my crontab using crontab -e.
  3. I found an old maintain-emacswiki in my bin directory, fixed it, and also added it to crontab. I definitely need to check the maintenance page a few times over the next few days.
  4. I found an old update-ell in my bin directory and decided to check the XML file referenced. The timestamp says Wed 24 Dec 2014 11:36:00 GMT so I think it’s safe to say that this won’t be required anymore.
  5. I did not find a copy of the emacs-elisp-area script. I checked the code in my config file and now I remember: this job used to call the expensive Elisp Area URLs and save the result to disk, and then URL rewriting made sure that the Elisp Area URLs called from the web would read those files instead. I just tried those links (”Alphabetical list of all Elisp libraries on this Wiki” with and without context, and “Chronological list of all Elisp libraries on this Wiki”) and it seems to work just fine. It takes a few seconds, but nothing terrible. I’ll say that this won’t be required anymore.
  6. I found an old copy of emacs-rss in my bin directory. That one precomputes some resource intensive RSS feeds. I should definitely get those back into working condition. When I run it, the four files are generated, and they’re the four RSS feed advertised in the HTML of the wiki, so that’s great.

And that’s all the jobs I found in an old crontab file!

Current status, one day later:

The only suspicious thing is the spike around 2:30 in the morning. But the explanation might be simple enough, looking at my crontab:

#m   h  dom mon dow   command
 02  5  *   *   *     /home/alex/bin/maintain-campaignwiki
 47 4,16 *  *   *     /home/alex/bin/backup
 28  4  *   *   *     /home/alex/bin/subscriptions
 14  3  *   *   *     /home/alex/bin/emacs-git-update
 32  2  *   *   *     /home/alex/bin/maintain-emacswiki

At 2:32, the maintenance job runs. The curl output is available in the maintenance directory:

--2017-05-19 02:32:01--  https://www.emacswiki.org/emacs
Resolving www.emacswiki.org (www.emacswiki.org)... 192.71.233.105
Connecting to www.emacswiki.org (www.emacswiki.org)|192.71.233.105|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/home/alex/emacswiki.org/maintenance/last.html’

     0K .......... .......... .......... .......... .......... 43.1K
    50K .......... .......... .......... .......... .......... 11.3K
   100K .......... .......... .......... .......... .......... 3.66K
   150K .......... .......... .......... .......... .......... 21.3K
   200K .......... .......... .......... .......... .......... 35.7K
   250K .......... .......... .......... .......... .......... 7.86K
   300K .......... .......... .......... .......... .......... 12.4K
   350K .......... .......... .......... .......... .......... 22.8K
   400K .......... .......... .......... .......... .......... 19.3K
   450K .......... .......... .......... .......... .......... 3.64K
   500K .......... .......... .......... .......... .......... 5.91K
   550K .......... .......... .......... .......... .......... 14.6K
   600K .......... .......... .......... .......... .......... 10.3K
   650K .......... .......... .......... .......... .......... 15.4K
   700K .......... .......... .......... .......... .......... 20.5K
   750K .......... .......... .......... .......... .......... 15.3K
   800K .......... .......... .......... .......... .......... 15.3K
   850K .......... .......... .......... .......... .......... 15.4K
   900K .......... .......... .......... ......                18.5K=86s

2017-05-19 02:34:03 (10.9 KB/s) - ‘/home/alex/emacswiki.org/maintenance/last.html’ saved [959329]

I’m guessing that these two minutes are causing the spike.

When I did some testing with the CSS, I ran into problems. If you choose a different theme via the CSS page, it gets stored in a cookie. Cookies are specific to a site, and so cookies set on http://emacswiki.org, https://emacswiki.org, http://www.emacswiki.org and https://www.emacswiki.org are separate from each other. This will not do. I’ve now changed the Apache config file to create the appropriate redirections. At the same time, I wanted to clean up the https://www.emacswiki.org/wiki vs. https://www.emacswiki.org/emacs situation.

<VirtualHost *:80>
    ServerName www.emacswiki.org
    ServerAlias emacswiki.org
    Redirect permanent / https://www.emacswiki.org/
</VirtualHost>
<VirtualHost *:443>
    ServerName emacswiki.org
    Redirect permanent / https://www.emacswiki.org/
    SSLEngine on
    SSLCertificateFile      /etc/letsencrypt.sh/certs/emacswiki.org/cert.pem
    SSLCertificateKeyFile   /etc/letsencrypt.sh/certs/emacswiki.org/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt.sh/certs/emacswiki.org/chain.pem
    SSLVerifyClient None
</VirtualHost>
<VirtualHost *:443>
    ServerAdmin alex@emacswiki.org
    ServerName www.emacswiki.org
    DocumentRoot /home/alex/emacswiki.org
    <Directory /home/alex/emacswiki.org>
        Options ExecCGI Includes Indexes MultiViews SymLinksIfOwnerMatch
        # legacy CGI scripts like info-ref.pl                                                               
        AddHandler cgi-script .pl
        AllowOverride All
        Require all granted
    </Directory>
    SSLEngine on
    SSLCertificateFile      /etc/letsencrypt.sh/certs/emacswiki.org/cert.pem
    SSLCertificateKeyFile   /etc/letsencrypt.sh/certs/emacswiki.org/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt.sh/certs/emacswiki.org/chain.pem
    SSLVerifyClient None
    Redirect permanent /wiki    https://www.emacswiki.org/emacs
    ProxyPass /emacs            http://emacswiki.org:8080/wiki
    ProxyPass /mojo             http://emacswiki.org:8080/mojo
</VirtualHost>

I wonder whether it’s important to prevent outside access to http://emacswiki.org:8080/wiki. I see no problem?

Tags:

Show Google +1

Comments

Heroic effort, and amazing notes! Thanks for Emacs Wiki!

AaronHawley 2017-05-19 01:27 UTC


Thanks!

– Alex Schroeder 2017-05-19 07:04 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.

Referrers: #emacs Planet Emacsen