# 2015-10-09 Oddmuse and Mojolicious

I’m trying to run Oddmuse within a Perl web framework: Mojolicious using Mojolicious::Plugin::CGI. It won’t be as perfect as a true Mojolicious app, but it will still be much faster than a simple CGI script. When running as a CGI script, every request loads Perl and compiles all the modules – including the CGI module itself – and Oddmuse and the config files. When running under Mojolicious, we no longer load Perl and we compile Oddmuse just once. Oddmuse itself will keep loading the config file and all that, but it’s still much better than before. The Mojolicious app itself is then started by Toadfarm. And in order to force myself to test it, I’ve switched this wiki over to the new setup!

First, tell Apache to act as a reverse proxy and pass all /wiki requests to the new server. The two ProxyPass instructions at the bottom are the important bits:

<VirtualHost *:80>
ServerName alexschroeder.ch
ServerAlias www.alexschroeder.ch
Redirect permanent / https://alexschroeder.ch/
</VirtualHost>
<VirtualHost *:443>
<Directory />
Options None
AllowOverride None
Order Deny,Allow
Deny from all
</Directory>
ServerName alexschroeder.ch
ServerAlias www.alexschroeder.ch
ServerAlias rpg.alexschroeder.ch
DocumentRoot /home/alex/alexschroeder.ch
<Directory /home/alex/alexschroeder.ch>
Options ExecCGI Includes Indexes MultiViews SymLinksIfOwnerMatch
AllowOverride All
Order Allow,Deny
Allow from all
</Directory>

SSLEngine on
SSLCertificateFile      /home/alex/ssl/alexschroeder.crt
SSLCertificateKeyFile   /home/alex/ssl/alexschroeder.key
SSLCertificateChainFile /home/alex/ssl/GandiStandardSSLCA2.pem
SSLVerifyClient None

ProxyPass /wiki             http://alexschroeder.ch:8080/wiki
ProxyPass /mojo             http://alexschroeder.ch:8080/mojo

</VirtualHost>

Toadfarm setup in ~/farm/farm:

#!/usr/bin/env perl

logging {
combined => 1,
file     => "farm.log",
level    => "info",
};

# other apps go here...

mount "$farm/alexschroeder.pl" => { "Host" => qr{^(www.)?alexschroeder\.ch$},
mount_point => '/wiki',
};

start; # needs to be at the last line

The above means that all requests to htpp://alexschroeder.ch:8080/wiki and htpp://www.alexschroeder.ch:8080/wiki will be handled by Oddmuse.

Mojolicious wrapper in ~/farm/alexschroeder.pl:

#! /usr/bin/env perl

use Mojolicious::Lite;

plugin CGI => {
support_semicolon_in_query_string => 1,
};

plugin CGI => {
route => '/',
script => 'wiki.pl', # ~/farm/wiki.pl
errlog => 'alexschroeder.log', # path to where STDERR from cgi script goes
};

app->start;

In this case, wiki.pl is simply the original Oddmuse script, unchanged.

Comments on 2015-10-09 Oddmuse and Mojolicious

Cool! That’s interesting.

What about side effects? Like, if you forget to initialize some variable somewhere, what is going to happen? (Thinking about mod_perl and all stuff related to it)

Also, I miss one important thing from your report… What about page load time? How much faster is it right now, comparing to regular cgi script?

– AlexDaniel 2015-10-09 11:21 UTC

I did not measure page load time. I also think that it keeps forking to run the script, so globals start empty every time? Not sure about the details. I haven’t seen any side effects, yet. Let me know if you see anything!

– Alex Schroeder 2015-10-09 11:39 UTC

Could you be so kind to measure the load time?

– AlexDaniel 2015-10-09 11:50 UTC

I think I was under the illusion that it was working. Now it is working. Hah! I had to rename my old wrapper script to realize that my ProxyPass instructions had been wrong.

Things to remember:

Check the config file for weird stuff.

# if ($ENV{SERVER_PORT} == 8080) { #$ScriptName  = 'http://alexschroeder.ch:8080/wiki';
#   $FullUrl = 'http://alexschroeder.ch:8080/wiki'; # } Make sure the user running toadfarm can write alle the files in your data directory. In my case, the data directory belongs to www-data.alex but there were temp files that alex could not write. – Alex Schroeder 2015-10-09 13:27 UTC Hm. Current setup, I get more or less the same time when I look at network analysis with my browser when I’m loading /wiki/About and when I’m loading /wiki2/About with wiki2.pl being my old wiki.pl wrapper script. I get numbers between 200ms and 700ms. I do note that /wiki/About keeps getting me a 200 OK response instead of a 304 NOT MODIFIED. I’m not quite sure what the problem is. – Alex Schroeder 2015-10-09 13:40 UTC Oh, and the code shows that our script is being exec’d. I’m not sure about the headers. I think if our script prints HTTP headers, we should be fine. – Alex Schroeder 2015-10-09 13:47 UTC Well, at least it seems to work right now. I think I got confused about the various restarts required. The missing 304 NOT MODIFIED remains a mistery, however. I’ve tried $q = new CGI; $q->nph(1) in the config file, $CGI::NPH = 1 in the config file, CGI::nph(1) in the config file, use CGI qw/-utf8 -nph/ in the core script, and print $q->header(-nph=>1, -status=>'304 NOT MODIFIED') and return if PageFresh(); in the core script, reloading my toadfarm between edits and found no change. When I tried to export MOJO_PLUGIN_CGI_DEBUG=1 before starting the toadfarm, I got a 503 error when visiting the site. This is very annoying. – AlexSchroeder 2015-10-09 14:32 UTC OK, so the question is: How often is 304 NOT MODIFIED actually used? alex@kallobombus:~/farm$ sudo wc -l /var/log/apache2/access.log.1
259852 /var/log/apache2/access.log.1
alex@kallobombus:~/farm$sudo grep ^oddmuse /var/log/apache2/access.log.1 | wc -l 32867 alex@kallobombus:~/farm$ sudo grep "^oddmuse.*GET /wiki.* 304 " /var/log/apache2/access.log.1 | grep -v feed | wc -l
270

Filtering out feed request we’re down to 8‰.

For my own site:

alex@kallobombus:~/farm$sudo grep ^alexschroeder /var/log/apache2/access.log.1 | wc -l 79286 alex@kallobombus:~/farm$ sudo grep "^alexschroeder.*GET /wiki.* 304 " /var/log/apache2/access.log.1 | grep -v feed | wc -l
702

This reduces the percentage to 9‰.

So, if we redirect feed requests to files and use a cron job to produce some of them, the 304 NOT MODIFIED response seems to be something we can do without, even if I think it ought to work.

– AlexSchroeder 2015-10-09 14:54 UTC

In other words, no noticeable performance boost? Do I read it right? But then why bother?

– AlexDaniel 2015-10-09 21:26 UTC

Hehe, indeed. The point is, however, that I want some sort of plan that will allow me to incrementally develop a solution. Mojolicious::Plugin::CGI does a lot of what I want. It integrates into Mojolicious and thus it can do logging, forking, and so on. It handles input and output. The only thing that’s missing is that it exec’s the script – loading Perl and compiling all the modules on every request.

I think I’ll try and modify the Plugin for my purposes. Then it’ll require Oddmuse once and just run DoWikiRequest on every request, mod_perl style.

– AlexSchroeder 2015-10-10 07:40 UTC

These are the changes I had to make to Mojolicious::Plugin::CGI: Add run option for a code reference and Make sure the status code is actually used and Add HTTP_IF_NONE_MATCH to environment.

This is the new server script I used:

#! /usr/bin/env perl

use lib '/Users/alex/src/mojolicious-plugin-cgi/lib';

use Mojolicious::Lite;

plugin CGI => {
support_semicolon_in_query_string => 1,
};

plugin CGI => {
route => '/wiki',
script => 'wiki.pl',
run => \&OddMuse::DoWikiRequest,
before => sub {
$OddMuse::RunCGI = 0;$OddMuse::DataDir = '/tmp/oddmuse';
require 'wiki.pl' unless defined &OddMuse::DoWikiRequest;
},
env => {},
errlog => 'wiki.log', # path to where STDERR from cgi script goes
};

get '/' => sub {
my $self = shift;$self->redirect_to('/wiki');
};

app->start;

– AlexSchroeder 2015-10-11 06:51 UTC

With the above setup now installed on alexschroeder.ch, I’m getting response times of 150–230ms when looking at /wiki/About (and a 304 Not Modified response). The old CGI setup under /wiki2/About is giving response times of 250–720ms.

So, much better?

I’ve also started noticing the first signs of trouble: Edit page, save, look at it (looks good), edit page again and notice that it’s showing the old text, not the new text.

– AlexSchroeder 2015-10-11 10:28 UTC

When I tried it with this page, I saw my browser requesting the page using the following:

GET /wiki?action=edit;id=Comments_on_2015-10-09_Oddmuse_and_Mojolicious HTTP/1.1
...
If-None-Match: 1444395217%1e1%1eAlexSchroeder%1ekensanata%40gmail.com

Oddmuse replied with a 304 Not Modified and the content was wrong. As I checked, however, the timestamp of the index file does in fact match:

alex@kallobombus:~/alexschroeder$date -r pageidx +%s 1444395217 Looking at the log of recent changes, however: alex@kallobombus:~/alexschroeder$ tail -n 1 rc.log
1444559479Comments_on_2015-10-09_Oddmuse_and_Mojolicious1With the above setup now installed on alexschroeder.ch, I'm getting response times of 150–230ms when looking at wikiAbout (and a 304 Not Modified response). So, much better? 127.0.0.1AlexSchroeder23en

I guess something about TouchIndexFile isn’t working.

sub TouchIndexFile {
my $ts = time; utime$ts, $ts,$IndexFile;
$LastUpdate =$Now = $ts; } I’ve made sure that all the files belong to alex.alex instead of www-data.alex. And that seems to help. After saving: alex@kallobombus:~/alexschroeder$ date -r pageidx +%s
1444560771
alex@kallobombus:~/alexschroeder$tail -n 1 rc.log 1444560771Comments_on_2015-10-09_Oddmuse_and_MojoliciousWhen I tried it with this page, I saw my browser requesting the page using the following: {{{ GET wiki?action=edit;id=Commentson2015-10-09OddmuseandMojolicious HTTP1.1 … If-None-Match: 1444395217%1e1%1eAlexSchroeder%1ekensanata%40gmail.com }}} Oddmuse replied with a 304 Not Modified and the content…127.0.0.1AlexSchroeder24en And the next request: GET /wiki?action=edit;id=Comments_on_2015-10-09_Oddmuse_and_Mojolicious HTTP/1.1 ... If-None-Match: 1444395217%1e1%1eAlexSchroeder%1ekensanata%40gmail.com HTTP/1.1 200 OK ... Etag: 1444560771%1e1%1eAlexSchroeder%1ekensanata%40gmail.com ... Yay? – AlexSchroeder 2015-10-11 10:52 UTC OK, added one final patch to make it work without having to use the -nph option for the CGI library and submitted a pull request. And I installed it for this site as well. “Eat your own dog food,” and all that. – AlexSchroeder 2015-10-11 12:41 UTC Wow, many more commits after talking to the maintainer. – AlexSchroeder 2015-10-11 16:14 UTC Add Comment # 2015-10-06 Pink Bliss Pink Bliss has improved somewhat. Mostly, I just added some magit faces. You know you want it! Download from GitHub! Emacs, in it’s maximized glory: Tags: Add Comment # 2015-10-05 Character Sheet Generator Using New Face Generator My Character Sheet Generator now uses my Face Generator. The rewrite is done and it’s time to go back to drawing some face elements! Elves! Comments on 2015-10-05 Character Sheet Generator Using New Face Generator Now drawing: dwarf beards. – Alex Schroeder 2015-10-07 09:39 UTC Add Comment # 2015-10-02 RPG Web Apps I Have Written I’m trying to make a list of web apps I have written for role-playing games… If you like what I’m doing, you can sponsor future web apps via Patreon or just send some money to kensanata@gmail.com using PayPal. Tags: Comments on 2015-10-02 RPG Web Apps I Have Written I think that you forgot to mention Gridmapper. – AlexDaniel 2015-10-03 16:26 UTC Oops! Thanks, added. – Alex Schroeder 2015-10-03 22:16 UTC And I have finished rewriting the Character Generator as a Mojolicious::Lite application and both the Face Generator are now running as two apps within a Toadfarm, and Toadfarm is beeing monitored by Monit. I think I’m happy now! – Alex Schroeder 2015-10-05 07:35 UTC Add Comment # 2015-09-28 Better Everything is better! My Face Generator is progressing nicely. I think the galleries for men and women are already pretty good. The entire app is running on Hypnotoad behind Apache/mod_proxy. It’s fast! And I’m going to support multiple artists. Please help me if you can! I’m not great with faces. I have barely drawn any until this project started. I had to read How to Draw a Face on WikiHow, that’s how bad I was! Here’s how I’m doing it right now. The basic instructions are on GitHub. First, print a copy of my empty.pdf. If you need a thinner face, take my elf.pdf. If you need a rounder face, take my dwarf.pdf. If you printed the default file, this is what it would look like: Download the PDF! Make sure you don’t scale the image to fit the page. Try to use 100% of the image. Then you draw 25 images of a particular element: hair, eyes, nose, ears, mouth, chin. 10% of all faces also get an extra element. Good for scars. My experience is that you need around 50 components for every element to provide enough variety – two sheets for each element is what I suggest. Then you scan it. You should get something like the following: There are more examples on Flickr. If you have something like the above your work is done. Send the images to me and I will do the rest! What does the rest involve? You crop it and resize it to 2250×3000 pixels. My scanner is notoriously bad about these proportions. I have scripts to extract whatever you draw from the helping lines of your scan. I have three variants ready to go: 1. The standard reduces the image to a few shades of blue: convert -blur 0x1 +dither -remap tintenblau.png scan1.jpg source1.png 2. This is good if you draw using a pencil and want to keep that pencil look: convert -blur 0x1 +dither -level 30%,80% scan1.jpg source1.png 3. This is good if you draw using a pencil and want to have it look like it was inked without going through the trouble of inking it yourself: convert -blur 0x1 +dither -level 50%,80% -grayscale rec709luma scan1.jpg source1.png I think you need to think about speed. Two sheet per element with six elements means 12 sheets. Save yourself the time of inking! Fiddle with the -level argument instead. The first percentage means that every pixels darker than this much is going to end up black. The second percentage means that every pixel lighter than this much is going to end up white. Raise the first number until you’re happy with the darkness of your image. Lower the second number until the helping lines disappear. You should end up with something like the following: Now’s the time to use my cutter script. It splits the image into 25 images of 450 x 600 pixels each. You must tell it the artist (alex), the element (hair, eyes, nose, ears, mouth, chin, extra) and the type (man, woman, elf, dwarf) you want to save it under: perl helpers/cutter.pl source1.png alex eyes_all. Eventually, you’ll end up with a large number of files. And that’s it! I commit it to my repository on GitHub and deploy it on my server. If you’re running the web application, there’s even the possibility to debug faces. I’m going to link to the debug interface so that you can take a look – but in the end, you won’t be able to save any changes unless you’re the artist of one the collections. 1. Choose an element to debug 2. List all the components for this element (in this case, eyes) 3. Edit one component (in this case, eyes_all_59.png) 4. When you click any of the zones, you should get an error… Comments on 2015-09-28 Better Airi Ayame has been adding manga style face elements. Thank you!! Not yet complete, but already recognizable: → Women's gallery by Airi! – Alex Schroeder 2015-09-28 12:30 UTC So, if I want to get digital, a single pic with 450*600 pixels, maybe even with a blue palette would be great. I just found a book on my harddrive with some art from early 19th century - So no scanning but copying and pasting… I was worried that it is a little different stylewise, but, hey Manga… rorschachhamster 2015-09-28 15:56 UTC I’ve drawn face elements for elves and dwarves and noticed that having a different head shape was essential for me. That’s why the repository now contains elf.pdf and dwarf.pdf. Another thing I noticed was that having opaque elements is tricky. This is most relevant if you plan on having white or blond hair cover parts of the face. As you’ll be using a lot of white filling, and since white is considered to be transparent, the face will will shine through your hair. In order to handle this, I have seen two solutions. Perhaps there are more? 1. Don’t do it – anything that hangs “in front” of the face needs to be inked. This is what I do. 2. Use some other color to fill areas that should be opaque and white. Use #FFFFFF for the transparent background and use #FEFEFF for the white-looking opaque foreground. This usually involves flood-filling those areas in Gimp or Photoashop and therefore requires well defined, enclosed areas. This is not always easy to achieve and takes a lot of time. 3. Use colors. I haven’t seen this in actual use but it should work. If you need something light and opaque, use a light color as you draw. A very light blue, or beige, or grey. Just make sure it’s different from the white background and make sure it’s not interspersed whith white pixels. Using a light pencil might leave too many white pixels, for example. – AlexSchroeder 2015-10-07 21:15 UTC Add Comment # 2015-09-18 Emacs and TTY I’m connecting to Kallobombus via SSH. It runs Debian 7.9 aka Wheezy. Recently a new problem appeared. I can no longer use Emacs unless I’m root. alex@kallobombus:~$ emacs
emacs: Could not open file: /dev/tty
alex@kallobombus:~$sudo emacs alex@kallobombus:~$ ls -l /dev/tty
crw------- 1 root root 5, 0 Sep 18 08:28 /dev/tty
alex@kallobombus:~$cat /etc/debian_version 7.9 alex@kallobombus:~$ which emacs
/usr/bin/emacs

What’s the problem?

On a different server, running Ubuntu 14.04.1 LTS:

kensanata@po6:~$cat /etc/debian_version jessie/sid kensanata@po6:~$ ls -l /dev/tty
crw-rw-rw- 1 root tty 5, 0 Sep 15 11:42 /dev/tty
kensanata@po6:~$emacs So, is the solution to chmod o+rw /dev/tty? Oh well, I ran sudo chmod go+rw /dev/tty and that seemed to fix the problem. I just hope it didn’t create any security issues. Tags: Add Comment # 2015-09-16 Mojolicious Yesterday I needed to write a little prototype of a site that would allow users to spell check their documents. The idea being that we want to add new dictionaries to the site, demo it to the right owners and thus secure an open license for their word lists. This is for an association interested in providing tech support for minority languages called Korero. I used the Text::Hunspell library and the Hunspell dictionaries available from Debian. Yay! Oddmuse is my Perl 5 project and has been since 2003. At the time, all we needed was a CGI script. These days, however, people often seem to use their own web servers that are part of their application. And so I decided to take a look at Mojolicious. And it was simple to use and easy to get started. I like it! More info on Github. Plans: 1. add mobile support, the CSS currently depends on the :hover attribute 2. better menu placement, the CSS currently just uses position: absolute which places the menu at the left edge (this might require Javascript) 3. the menu should replace the text in question 4. a JSON interface to allow other apps to do thrir spell checking here At the same time, we’re working on a Hunspell dictionary for one of the many Rumantsch idioms in Switzerland. Tags: Add Comment # 2015-08-03 Switching to Postfix What I want to do is read all my email using Google. I like Gmail, I like Inbox, I like their spam filtering. Cool. I also want to receive email for my various domains. Thus, all these domains have an MX record pointing to my Raspberry Pi. It runs Postfix and Dovecot, and my accounts have .forward files pointing to kensanata@gmail.com. This ought to be cool, too. If I remove that .forward file, I can read my email on the Raspberry Pi using IMAP. Sadly, I’m getting a lot of these: <kensanata@gmail.com>: host gmail-smtp-in.l.google.com[173.194.78.26] said: 550-5.7.1 [77.57.27.11 12] Our system has detected that this message is 550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail, 550-5.7.1 this message has been blocked. Please visit 550 5.7.1 https://support.google.com/mail/answer/188131 for more information. eu1si12802336wic.49 - gsmtp (in reply to end of DATA command) So now I’m trying to setup up things to fix this. I’ve tried doing these things using Exim but I find the official documentation to be hard to understand for newbies like me. So now I’m trying to switch to Postfix. ## SPF In order to reduce spam, I have a TXT record in my zone file with some SPF1 information. There is some info by Google on what they suggest (include:_spf.google.com ~all) and there is the SPF Record Syntax documentation which shows you how to indicate who else can send mail for your domain (mx means “the MX records” and a means “the current domain”). @ 10800 IN TXT "v=spf1 ip6:2001:4830:134:3::10 mx a include:_spf.google.com ~all" The IP6 number is for egg.gnu.org. When I send mail from my Raspberry Pi to my gnu.org address without this information, it arrives on Gmail with the following headers: Received: from eggs.gnu.org (eggs.gnu.org. [2001:4830:134:3::10]) by mx.google.com with ESMTPS id g4si5396462igt.36.2015.08.03.02.14.24 for <kensanata@gmail.com> (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 03 Aug 2015 02:14:24 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning alex@alexschroeder.ch does not designate 2001:4830:134:3::10 as permitted sender) client-ip=2001:4830:134:3::10; Authentication-Results: mx.google.com; spf=softfail (google.com: domain of transitioning alex@alexschroeder.ch does not designate 2001:4830:134:3::10 as permitted sender) smtp.mail=alex@alexschroeder.ch Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <alex@alexschroeder.ch>) id 1ZMBp9-0002rf-2D for kensanata@gmail.com; Mon, 03 Aug 2015 05:14:23 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:37492) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <alex@alexschroeder.ch>) id 1ZMBp8-0002rb-VG for kensanata@gmail.com; Mon, 03 Aug 2015 05:14:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45863) by fencepost.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <alex@alexschroeder.ch>) id 1ZMBp8-0005RP-Lv for alex@gnu.org; Mon, 03 Aug 2015 05:14:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <alex@alexschroeder.ch>) id 1ZMBp5-0002r8-Ar for alex@gnu.org; Mon, 03 Aug 2015 05:14:22 -0400 Received: from 77-57-27-11.dclient.hispeed.ch ([77.57.27.11]:48751 helo=mail.alexschroeder.ch) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <alex@alexschroeder.ch>) id 1ZMBp5-0002qy-3w for alex@gnu.org; Mon, 03 Aug 2015 05:14:19 -0400 Received: by mail.alexschroeder.ch (Postfix, from userid 1001) id BA59610FC; Mon, 3 Aug 2015 10:35:13 +0200 (CEST) So, the mail went from mail.alexschroeder.ch to eggs.gnu.org, was scanned for spam and forwarded to fencepost.gnu.org which is where my account is, and my .forward file there tells the Gnu mail server to send it on to gmail.com. There, they find that eggs.gnu.org is not allowed to send mail in the name of alexschroeder.ch. Adding the IP6 number 2001:4830:134:3::10 to my SPF record changes that. ## Greylisting sudo apt-get install postfix postgrey That should install Greylisting. ## TLS Transport Layer Security makes sure nobody can eavesdrop on your emails. This used to be a separate step when I was using Exim. When I switched to Postfix, it was the standard. pi@raspberrypi ~$ grep -i tls /etc/postfix/main.cf
# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

## Authentication

I’m not sure how this is supposed to work. Looking at the Postfix SASL Howto it would seem that Postfix can use my Dovecot installation. But to be honest, I’m not quite sure I need this.

pi@raspberrypi ~ $sudo apt-get install opendkim opendkim-tools I then edited /etc/opendkim.conf and added the following: # How To Install and Configure DKIM with Postfix on Debian Wheezy # https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-dkim-with-postfix-on-debian-wheezy AutoRestart Yes AutoRestartRate 10/1h UMask 002 Syslog yes SyslogSuccess Yes LogWhy Yes Canonicalization relaxed/simple ExternalIgnoreList refile:/etc/opendkim/TrustedHosts InternalHosts refile:/etc/opendkim/TrustedHosts KeyTable refile:/etc/opendkim/KeyTable SigningTable refile:/etc/opendkim/SigningTable Mode sv PidFile /var/run/opendkim/opendkim.pid SignatureAlgorithm rsa-sha256 UserID opendkim:opendkim Socket inet:12301@localhost The instructions mentioned something about editing /etc/default/opendkim and adding SOCKET="inet:12301@localhost" but this seems unnecessary. A comment at the top says Command-line options specified here will override the contents of /etc/opendkim.conf. I’m not doing anything for the moment since I see the «Socket» setting in /etc/opendkim.conf. I added the following lines to /etc/postfix/main.cf: milter_protocol = 2 milter_default_action = accept smtpd_milters = inet:localhost:12301 non_smtpd_milters = inet:localhost:12301 I created the necessary directories: sudo mkdir -p /etc/opendkim/keys And I moved the key pair I had created there. These I had created using the instructions elsewhere instead of using opendkim-genkey. openssl genrsa -out dkim.private 1024 openssl rsa -in dkim.private -out dkim.public -pubout -outform PEM As far as I can tell, the key cannot be more than 1024 because the TXT record in the zone (see below) is limited in size. Change ownership and permissions of the private key: sudo chown opendkim:opendkim dkim.private sudo chmod 440 keys/dkim.private  Previously, my permission for the private key had been -rw-r--r-- and I was getting key data is not secure errors in my /var/log/mail.err. as they say, if anyone on the system can change the key, or generate signatures, or steal the key, that’s not good. When I tried to use -r-------- I was getting a different error: can't load key from /etc/opendkim/keys/dkim.private: Permission denied. So now I’m using -r--r-----. I also added the following line to my zone file: dkim._domainkey 10800 IN TXT "k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGMDmbDQEk62j5n8pcHsWNAkrnvhes2g+c2O/gXEytr40E7nqX21YuIDj1e6S07+A8aM/0TSV5nIiWEhOocJtG91wNeM7ypqAgR5zihYoolkXk4FWsdIPbuwZbIEbLBLkgJdD2qbgLLOe/OFMIF3/MQROrfVh+rrxHun2RFJLKDwIDAQAB" Test it using dig -t TXT dkim._domainkey.alexschroeder.ch. This should tell you whether the record is correct. In addition to that, Check a DKIM Core Key. Then I created the following files. Note that I’m using the selector dkim instead of mail because of the guide I had followed in an earlier attempt, linked above. /etc/opendkim/TrustedHosts 127.0.0.1 localhost 192.168.0.1/24 *.alexschroeder.ch *.communitywiki.org *.orientalisch.info *.campaignwiki.org *.arabisch-lernen.net *.oddmuse.org *.emacswiki.org /etc/opendkim/KeyTable dkim._domainkey.alexschroeder.ch alexschroeder.ch:dkim:/etc/opendkim/keys/dkim.private dkim._domainkey.communitywiki.org communitywiki.org:dkim:/etc/opendkim/keys/dkim.private dkim._domainkey.orientalisch.info orientalisch.info:dkim:/etc/opendkim/keys/dkim.private dkim._domainkey.campaignwiki.org campaignwiki.org:dkim:/etc/opendkim/keys/dkim.private dkim._domainkey.arabisch-lernen.org arabisch-lernen.org:dkim:/etc/opendkim/keys/dkim.private dkim._domainkey.oddmuse.org oddmuse.org:dkim:/etc/opendkim/keys/dkim.private dkim._domainkey.emacswiki.org emacswiki.org:dkim:/etc/opendkim/keys/dkim.private /etc/opendkim/SigningTable *@alexschroeder.ch dkim._domainkey.alexschroeder.ch *@communitywiki.org dkim._domainkey.communitywiki.org *@orientalisch.info dkim._domainkey.orientalisch.info *@campaignwiki.org dkim._domainkey.campaignwiki.org *@arabisch-lernen.org dkim._domainkey.arabisch-lernen.org *@oddmuse.org dkim._domainkey.oddmuse.org *@emacswiki.org dkim._domainkey.emacswiki.org With all of this done, it’s time to restart our services. sudo service postfix restart sudo service opendkim restart ## Getting Off Blacklists I had to request the removal of my Raspberry Pi’s IP number from some blacklists. ## Accepting Mail Sending mail to my various domains resulted in “Relay access denied” messages: 554 5.7.1 <admin@campaignwiki.org>: Relay access denied I realized what the problem was: What domains to receive mail for needs to list all of them! In /etc/postfix/main.cf I changed this line: mydestination = alexschroeder.ch, mail.alexschroeder.ch, communitywiki.org, orientalisch.info, campaignwiki.org, arabisch-lernen.org, oddmuse.org, emacswiki.org, localhost.alexschroeder.ch, localhost ## Aliases This is how most of my mail is handled: pi@raspberrypi ~$ cat /etc/aliases
# /etc/aliases
mailer-daemon: postmaster
postmaster: root
nobody: root
hostmaster: root
usenet: root
news: root
webmaster: root
www: root
ftp: root
abuse: root
noc: root
security: root
root: alex

## Fighting Spam

I’m following Timo’s blog:

apt-get install spamassassin spamc

This pulls a ton of dependencies… And then edit /etc/default/spamassassin, setting ENABLED=1.

Then edit /etc/spamassassin/local.cf and add my contact email. I also experimented with rewrite_header but found that it made scanning my Junk folder harder than necessary. That’s also the reason why I disabled report_safe.

# rewrite_header Subject *****SPAM*****
report_contact kensanata@gmail.com

#   Save spam messages as a message/rfc822 MIME attachment instead of
#   modifying the original message (0: off, 2: use text/plain instead)
#
report_safe 0

No Bayes auto-learning because I’m planning to forward all my mails to Gmail eventually. At that point, no more learning would occur.

Now edit /etc/postfix/master.cf and add a line to smtp as follows:

smtp      inet  n       -       -       -       -       smtpd
-o content_filter=spamassassin

Also add the following at the end:

spamassassin unix -     n       n       -       -       pipe
user=debian-spamd argv=/usr/bin/spamc -f -e
/usr/sbin/sendmail -oi -f ${sender}${recipient}

Restart everything.

sudo service spamassassin restart
sudo service postfix restart

## Moving Spam to Junk Folder

Sadly, this doesn’t work as intended. It should be easy but I have permission problems. I get errors like these in my mail log: dovecot: lda(alex): Error: setegid(privileged) failed: Operation not permitted. I think I’m only getting this error when my mail is not spam and should therefore end in /var/mail/alex. Perhaps I have to move from LDA to LMTP.

This is what I had in /etc/postfix/main.cf:

# mailbox_command = procmail -a "$EXTENSION" mailbox_command = /usr/lib/dovecot/deliver deliver is linked to deliver-lda and somehow deliver-lda is not allowed to do its job? So I switched to dovecot-lmtp. I followed the instructions on the Dovecot wiki. Running apt-get install dovecot-lmtp already added the protocol via /usr/share/dovecot/protocols.d/lmtpd.protocol. In /etc/dovecot/conf.d/10-master.conf I made the required change: service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } No changes were required for /etc/dovecot/conf.d/20-lmtp.conf. I told Postfix about the change in /etc/postfix/main.cf. # mailbox_command = procmail -a "$EXTENSION"
# mailbox_command = /usr/lib/dovecot/deliver

# Using Dovecot LMTP
# http://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP
mailbox_transport = lmtp:unix:private/dovecot-lmtp

At this point I restarted both services.

sudo service dovecot restart
sudo service postfix restart

But now I was getting a different error when I sent myself a test email:

Aug  9 09:36:40 raspberrypi postfix/lmtp[29502]: 61A5618B6: to=<alex@alexschroeder.ch>, orig_to=<alex>, relay=mail.alexschroeder.ch[private/dovecot-lmtp], delay=0.57, delays=0.19/0.07/0.13/0.18, dsn=5.1.1, status=bounced (host mail.alexschroeder.ch[private/dovecot-lmtp] said: 550 5.1.1 <alex@alexschroeder.ch> User doesn't exist: alex@alexschroeder.ch (in reply to RCPT TO command))

The problem was that I’m using existing system accounts on my server, not virtual accounts. That’s why the user lookup needs to ignore the domain. I made the following change in /etc/dovecot/conf.d/10-auth.conf:

# auth_username_format = %Lu
# http://www.0xf8.org/2012/11/trouble-with-postfix-and-dovecotpam-unknown-user/
auth_username_format = %Ln

The rest of the file remained unchanged. Which is why it still says at the end:

!include auth-system.conf.ext

It seems to work!

My users now all have the following ~/.dovecot.sieve file:

require "fileinto";
if header :contains "X-Spam-Flag" "YES" {
fileinto "Junk";
}

I wanted to set up server-side email filtering with Dovecot Sieve but I couldn’t find the /etc/dovecot/sieve/default.sieve file it told me to edit. Oh well, I just have to users, so copying the tiny file was no problem.

## Postfix Queue Management

The commands I’ve used the most often:

mailq
sudo postcat -q 1EF1841F
sudo postsuper -d 1EF1841F

(From here.)

# 2015-07-24 Gmail, Gnus and GPG

I wrote some instructions for setting up Gmail, Gnus and GPG. I used Emacs 24.5 and GPG 2.0 for this tutorial.

## If you’re on Windows

I installed gnupg 1.4 using Cygwin and it worked just as well.

## If you’re on a Mac

I tried this using my Mac and a Guest account. A Guest account has all its files deleted when you log out. It’s as “clean” as I could get.

You can install Emacs in various ways. You could use Emacs For Mac OS X. “Pure Emacs! No Extras! No Nonsense!” This is what I used to run. I’ve also built Emacs from source many times. Just remember to ./configure --with-ns before you start. You can install it from Homebrew: You could brew install emacs or you could install Yamamoto Mitsuharu’s Emacs Mac Port. This last variant is what I’m using right now.

brew tap railwaycat/emacsmacport
brew install emacs-mac --with-gnutls --with-imagemagick --with-official-icon --with-xml2
brew linkapps emacs-mac

As for GPG, you have two options. You can install GPG via GPG Suite. This is what I used at first. gpg --version said gpg (GnuPG/MacGPG2) 2.0.27 (the current, stable version is 2.0.28). But I wanted to install GPG via Homebrew. If you want to try it as well, make sure you uninstall the GPG Suite if you had it installed.

If you just install gpg2, you won’t be able to enter your passphrase from within Emacs with a GUI. You’ll see a simple, cut off message saying epa-file--find-file-not-found-function: Opening input file: Decryption failed,. If you only use Emacs within terminal windows, no problem. No need to do anything. But if you like to use Emacs with fonts and colors, menus and scrollbars, then you need to install a special variant of pinentry in order to get a separate Pinentry window:

brew install gpg2
brew install pinentry-mac

Once you have done that, change your ~/.gnupg/gpg-agent.conf file so that it has only this one line:

pinentry-program /usr/local/bin/pinentry-mac

When you install gpg2, it tells you to put use-standard-socket in your ~/.gnupg/gpg-agent.conf file. Don’t do that unless you only want to use Emacs from a terminal. Here’s a sample session in a terminal, showing you what you’ll get, if you don’t use pinentry-mac:

alex@Megabombus:~$echo "This is a test." > test alex@Megabombus:~$ gpg --recipient kensanata --encrypt test
alex@Megabombus:~$ls test* test test.gpg alex@Megabombus:~$ gpg --output test2 test.gpg

You need a passphrase to unlock the secret key for
user: "Alex Schroeder <kensanata@keybase.io>"
8192-bit RSA key, ID 4529A45C, created 2015-03-01 (main key ID ACECFEAE)

gpg: encrypted with 8192-bit RSA key, ID 4529A45C, created 2015-03-01
"Alex Schroeder <kensanata@keybase.io>"
alex@Megabombus:~$ls test* test test.gpg test2 alex@Megabombus:~$ diff test test2

               ┌────────────────────────────────────────────────────────────────────────────────────┐
│ Please enter the passphrase to unlock the secret key for the OpenPGP certificate:  │
│ "Alex Schroeder <kensanata@keybase.io>"                                            │
│ 8192-bit RSA key, ID 4529A45C,                                                     │
│ created 2015-03-01 (main key ID ACECFEAE).                                         │
│                                                                                    │
│                                                                                    │
│ Passphrase *****************************************************************______ │
│                                                                                    │
│            <OK>                                                  <Cancel>          │
└────────────────────────────────────────────────────────────────────────────────────┘

This pinentry works in a terminal, but it won’t show up when GPG is called by GUI Emacs! Install pinentry-mac and you’ll be able to provide your passphrase when using Emacs.

If you already tinkered with gpg before changing ~/.gnupg/gpg-agent.conf, you’ll have to reload gpg-agent.

alex@Megabombus:~\$ echo RELOADAGENT | gpg-connect-agent
OK

Comments on 2015-07-24 Gmail, Gnus and GPG

Today I was able to copy that config file for my Emacs installation at work, use the public keys of both my work account and my private account for mml2015-signers, and it worked! Yay.

– Alex Schroeder 2015-11-23 20:24 UTC

# 2015-06-21 Jessie

I upgraded my old Apple iBook G4 (PowerPC architecture) from Debian Wheezy to Jessie. And I was greeted with a black screen. I rebooted and watched the text scrolling by. Something failed about Load Kernel Modules. Something else failed about LSB. Nothing serious? The text console login prompt, and then a black screen and the ventilator blowing.

I had never figured out how to switch from X11 to the consoles on my iBook. But now I was determined to figure it out! I had seen the prompt, after all. It turns out that I need to press fn + ctrl + alt + cmd + F1.

OK. Looks like no GDM3? I’m going to install the display manager slim and hope that this is enough. Time passes. Wow, I am greeted by a display that looks like a CGA display with four colors? Maybe six colors? It’s terrible. But I see a Debian logo, and a user name prompt. I’m logging in!

And… All the colors are wrong. I feel like my entire desktop has been resampled. Ough! My eyes! I’m going to install xdm… Rebooting… And the graphics are unchanged. Damn!

When I switch to the console, run Emacs, and call M-x list-colors-display I see the colors just fine even if the entire screen is terribly dark. Answering Linux nouveau.noaccel=1 to the second boot prompt also didn’t solve my problem.

For the moment I uninstalled all display managers (no xdm, no gdm3) and so the system just shows the console login prompts when I boot. Too bad the damn stuff is so dark I can’t use the laptop during the daylight.

Tags:

Claudia is using her laptop and so I’m rebooting the old iBook G4. Upgrading the system. Who knows, perhaps something has been fixed?

Also, I’m using lxde.

This is what fixed my color problem: enable KMS in /etc/yaboot.conf. It really works. I am amazed. I had given up on this machine.

image=/boot/vmlinux
label=Linux
initrd=/boot/initrd.img

...

Don’t forget to run sudo ybin -v when you’re done.

I found a solution to the super dark console on SuperUser: Zapping the PRAM.

1. Restart the Mac and before the grey screen appears, hold down Command + Option + P + R
2. Hold the keys down until the computer restarts and you hear the startup sound for the second time.
3. Release the keys.

– Alex Schroeder 2015-10-01 13:02 UTC

Uh, today I returned to this laptop and it didn’t work. During the boot process the screen turns black and stays black. Time for some experiments:

Boot Parameter Console X11Notes
none ok 8bit
none ok 8bit
video=1024x768-32ok 8bit
radeon.modeset=1ok good locks up after a few moments
??

– Alex Schroeder 2015-10-09 12:09 UTC

I’ve been looking at the systemctl status reports. I’ve got systemd-modules-load: Failed to find module 'apm_emu' and pommed: Failed to start LSB: Apple laptops hotkeys event handler. I’m guessing these two are not related to X11 freezing.

– Alex Schroeder 2015-10-09 12:53 UTC