These are the pages for all my system administration woes. I rarely post happy things in this section. You have been warned.

2019-03-31 Email Server, Again

Do I want to do it again? I ran email on my server for a while, the. I ran email on a Raspberry Pi for a while. But at the same time I am annoyed by my reliance on Gmail, and on the unwillingness of Protonmail and others to provide IMAP access to my mail.

Recently, @algernon posted How To Run Your Own Mail Server and said it was “a detailed HOWTO that isn’t completely obsolete, and is easy enough to adapt to my needs. It goes step by step, instead of setting up everything in one sitting.” Perhaps my kind of thing?

I mean, I’ve said in the past that email is the new snail mail, i.e. only used for spam and bills; and that I had more to fear from criminals that from state actors and big corporations as far as email went, but I still don’t feel at ease.

So perhaps one day I will try again!


Comments on 2019-03-31 Email Server, Again

Day 1: Change the DNS records. This is what I added:

@ 10800 IN MX 10
@ 10800 IN TXT "v=spf1 mx -all"
mail 10800 IN A
mail 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1

Next step: reverse DNS. I wasted precious time trying to figure out how to do that until I discovered that I had to send a mail to the hosting provider of my server, apparently. Oh well.

Slowly things are coming back to me. Kallobombus Mail, for example. Or 2015-08-03 Switching to Postfix. And Raspberry Pi als privater Email Server. Uuuugh!

– Alex Schroeder 2019-03-31 11:49 UTC

Day 2: OK, so the hosting company says that the domain name service company should do it (Gandi). I don’t see anything in the menus but it seems to me that it should involve the PTR record type from what I can read. But apparently I cannot type it into the same file I mentioned above. This appears to be an error: 10800 IN PTR IN PTR

Where does this stuff go? Somewhere on Gandi?

Apparently something is already configured somehwere, somehow: host gives me domain name pointer but host 2a02:418:6a04:178:209:50:237:1 gives me Host not found: 3(NXDOMAIN) – whatever that means.

And where do I do this? Some sort of new “zone” file – on the Gandi site I see no way to do it, and on my own server I don’t think I have a name server installed. Should I?

– Alex Schroeder 2019-04-04 21:28 UTC

It sounds like your hosting provider is confused - maybe they think you’re just talking about regular DNS vs reverse DNS?

– Anonymous 2019-04-05 01:07 UTC

Yeah, I got the same advice on Mastodon (@fitheach, @freakazoid, @Frotz, all of them very helpful) suggesting the same thing. I sent them another message.

– Alex Schroeder 2019-04-05 08:20 UTC

Add Comment

2019-03-17 I borked my system

I was unhappy about my Perl installation. I was getting a warning regarding XS. Some of the libraries I had installed were compiled for a different version of Perl. Sure, I had installed Perl modules using cpan and cpanm. Apparently I had received a new Perl via an upgrade of PureOS. Sadly, I didn’t know how to quickly find all the modules which included XS and reinstall and recompile them.

Here’s a start:

find $(perl -e 'print join " ", @INC') -name

But then again... Perhaps it would make more sense to use perlbrew. It’s what I use on my server. Just ignore the system Perl!

But it also involves a lot of downloading and installing of Perl modules... That’s not cool. So perhaps I should continue using the system Perl and just try and replace all the packages I had installed with system packages? Let the package manager handle it?

Sadly, I immediately ran into the missing package Mastodon::Client. And once you try to install it, it pulls in all the dependencies. Some of them I might have installed using the system package manager, but cpanm doesn’t know about that.

So back to perlbrew, right? And I can delete all the system Perl modules...

Get the list:

apt list --installed "*-perl" | grep -v automatic | sed 's/\/.*//'

And run the command:

sudo apt remove libcaptcha-recaptcha-perl libcapture-tiny-perl \
libcrypt-random-seed-perl libcrypt-rijndael-perl libdatetime-perl \
libdatetime-timezone-perl libhtml-template-perl libhttp-date-perl \
libhttp-server-simple-perl libjson-perl liblist-allutils-perl \
liblocale-gettext-perl libmce-perl libmldbm-perl libmodern-perl-perl \ libmojolicious-perl libnet-server-perl libnet-whois-parser-perl \
libpod-strip-perl librpc-xml-perl libtext-charwidth-perl libtext-iconv-perl \
 libtext-markdown-perl libtext-wrapi18n-perl libtime-parsedate-perl \
libxml-atom-perl libxml-libxml-perl

Whatever, right? Enter! Enter!

But wait... what’s this? This is all wrong!

Removing gnome-shell-extensions (3.30.1-1) ...
Removing gdm3 (3.30.2-1pureos1) ...
Removing gnome-getting-started-docs (3.30.0-1) ...
Removing gnome-user-docs (3.30.2-1) ...
Removing yelp (3.31.90-1) ...
Removing inkscape (0.92.4-2) ...
Removing libgtkspell0:amd64 (2.0.16-1.2) ...
Removing aspell-de (20161207-7) ...
Removing aspell-en (2018.04.16-0-1) ...
Removing aspell (0.60.7~20110707-6) ...
Removing blends-tasks (0.7.2) ...
Removing chrome-gnome-shell (10.1-5) ...
Removing pureos-standard (0.9.4) ...
Removing pureos-minimal (0.9.4) ...
Removing console-setup (1.188) ...
Removing console-setup-linux (1.188) ...
Removing debconf-i18n (1.5.71) ...
Removing devscripts (2.19.3) ...
Removing dhelp (0.6.25) ...
Removing docbook2x (0.8.8-17) ...
Removing enchant (1.6.0-11.1+b1) ...
Removing evince (3.30.2-3) ...
Removing gnome-todo (3.28.1-2) ...
Removing gnome-contacts (3.30.2-1) ...
Removing libfolks-eds25:amd64 (0.11.4-1+b2) ...
Removing gedit (3.30.2-2) ...
Removing gnome-sushi (3.30.0-2) ...
Removing gir1.2-evince-3.0:amd64 (3.30.2-3) ...
Removing gnome-maps (3.30.3-1) ...
Removing gir1.2-webkit2-4.0:amd64 (2.22.6-1) ...
Removing gnome-calendar (3.30.1-2) ...
Removing gnome-control-center (1:3.30.3-1) ...
Removing gnome-initial-setup (3.30.0-1pureos2) ...
Removing gnome-online-accounts (3.30.1-2) ...
Removing gnome-session (3.30.1-2pureos1) ...
Removing gnome-software (3.28.0-1pureos1) ...
Removing gnulib (20140202+stable-3.1) ...
Removing xorg (1:7.7+19) ...
Removing xserver-xorg (1:7.7+19) ...
Removing xserver-xorg-video-all (1:7.7+19) ...
Removing xserver-xorg-video-vmware (1:13.3.0-2) ...
Removing xserver-xorg-video-vesa (1:2.4.0-1) ...
Removing xserver-xorg-input-all (1:7.7+19) ...
Removing xserver-xorg-input-libinput (0.28.2-1) ...
Removing xserver-xorg-video-qxl (0.1.5-2+b1) ...
Removing xserver-xorg-video-nouveau (1:1.0.16-1) ...
Removing xserver-xorg-input-wacom ( ...
Removing libaudio-scrobbler-perl (0.01-2.3) ...
Removing libcaptcha-recaptcha-perl (0.98+ds-1) ...
Removing perlbrew (0.86-1) ...
Removing lintian (2.9.1) ...
Removing libcapture-tiny-perl (0.48-1) ...
Removing libcrypt-random-seed-perl (0.03-1) ...
Removing libcrypt-rijndael-perl (1.13-1+b5) ...
Removing librpc-xml-perl (0.80-2) ...
Removing libdatetime-format-iso8601-perl (0.08-2) ...
Removing libdatetime-format-builder-perl (0.8100-2) ...
Removing libdatetime-format-strptime-perl (1.7600-1) ...
Removing libxml-atom-perl (0.42-2) ...
Removing libdatetime-perl:amd64 (2:1.50-1+b1) ...
Removing libdatetime-timezone-perl (1:2.23-1+2018i) ...
Removing libevview3-3:amd64 (3.30.2-3) ...
Removing libxml-xpath-perl (1.44-1) ...
Removing libnet-dbus-perl (1.1.0-5+b1) ...
Removing libxml-twig-perl (1:3.50-1) ...
Removing libxmlrpc-lite-perl (0.717-1) ...
Removing libsoap-lite-perl (1.27-1) ...
Removing libgitlab-api-v4-perl (0.16-1) ...
Removing libgoa-backend-1.0-1:amd64 (3.30.1-2) ...
Removing libgspell-1-1:amd64 (1.6.1-2) ...
Removing libhtml-form-perl (6.03-1) ...
Removing libhtml-template-perl (2.97-1) ...
Removing libhttp-daemon-perl (6.01-3) ...
Removing libhttp-server-simple-perl (0.52-1) ...
Removing libjson-perl (4.02000-1) ...
Removing liblist-allutils-perl (0.15-1) ...
^[Removing libparse-debianchangelog-perl (1.2.0-13) ...
Removing libmce-perl (1.838-1) ...
Removing libmldbm-perl (2.05-2) ...
^[Removing libmodern-perl-perl (1.20180901-1) ...
Removing libmojo-server-fastcgi-perl (0.50-1) ...
Removing libmojolicious-perl (8.12+dfsg-1) ...
Removing libnet-server-perl (2.009-1) ...
Removing libnet-whois-parser-perl (0.08-1) ...
Removing libpod-strip-perl (1.02-2) ...
Removing libtext-wrapi18n-perl (0.06-7.1) ...
Removing libtext-charwidth-perl (0.04-7.1+b1) ...
Removing libtext-markdown-perl (1.000031-2) ...
Removing libtime-parsedate-perl (2015.103-3) ...
Removing texinfo (6.5.0.dfsg.1-4+b1) ...
Removing libxml-simple-perl (2.25-1) ...
Removing libxml-libxslt-perl (1.96-1+b1) ...
Removing libxml-sax-expat-perl (0.51-1) ...
Removing libyelp0:amd64 (3.31.90-1) ...
Removing xserver-xorg-video-amdgpu (18.1.99+git20190207-1) ...
Removing xserver-xorg-video-ati (1:18.1.99+git20190207-1) ...
Removing xserver-xorg-video-fbdev (1:0.5.0-1) ...
Removing xserver-xorg-video-intel (2:2.99.917+git20180925-2) ...
Removing xserver-xorg-video-radeon (1:18.1.99+git20190207-1) ...
Removing xserver-xorg-core (2:1.20.3-1) ...
Removing keyboard-configuration (1.188) ...
Removing libxml-parser-perl (2.44-4) ...
Removing libxml-libxml-perl (2.0134+dfsg-1) ...
Removing libwww-perl (6.36-1) ...
Removing libfile-listing-perl (6.04-1) ...
Removing libhttp-cookies-perl (6.04-1) ...
Removing libhttp-negotiate-perl (6.01-1) ...
Removing libhttp-message-perl (6.18-1) ...
Removing libhttp-date-perl (6.02-1) ...
Removing tasksel-data (3.50) ...
Removing liblwp-protocol-https-perl (6.07-2) ...
Removing tasksel (3.50) ...
Removing liblocale-gettext-perl (1.07-3+b4) ...
Removing mutter (3.30.2-6) ...
Removing zenity (3.30.0-2) ...
Removing libwebkit2gtk-4.0-37:amd64 (2.22.6-1) ...
Removing libenchant1c2a:amd64 (1.6.0-11.1+b1) ...
Removing hunspell-en-us (1:2018.04.16-1) ...
Removing dictionaries-common (1.28.1) ...
Removing 'diversion of /usr/share/dict/words to /usr/share/dict/words.pre-dictionaries-common by dictionaries-common'
Removing evolution-data-server (3.30.5-1) ...
Removing libedataserverui-1.2-2:amd64 (3.30.5-1) ...
Removing libtext-iconv-perl (1.7-5+b7) ...
Removing gnome-shell (3.30.2-3) ...

Nooooooo! It took so long to abort the operation, now it’s all borked, surely.

OK, how to I recover from this?

apt install evince inkscape gnome pureos-standard perlbrew aspell-de

I hope? Wish me luck!

Fist, switch to a stable Perl using perlbrew and get the modules installed. At this point I’m just trying to finish the task I set out to do, hours ago.


Comments on 2019-03-17 I borked my system

And I think I’m back! Restarted the system and no problem. Emacs works. Browser works. Phew! 😅

– Alex Schroeder 2019-03-17 21:53 UTC


For what it’s worth, I prefer to package any missing libraries as .deb packages myself and install them. dh-make-perl --cpan MODULE makes this easy, usually.

(If you need much newer versions of modules, especially “low level” ones, it might not be feasible to go this way, though.)

Adam 2019-03-17 22:11 UTC

Oh, interesting! Thank you for the suggestion.

– Alex Schroeder 2019-03-18 06:10 UTC

Add Comment

2019-03-13 Debian forever?

I loved A (Partial) Defense of Debian by John Goerzen. “We recognize the value of something that just works, that is so stable that things like unattended-upgrades are safe and reliable.” I did not appreciate this for many years. But now I’m back to using Debian for my servers and that’s exactly why. The years on my laptop have taught me this.

And I still love ❤ Slackware. 🙂

The only reason I’m using PureOS on my laptop is that it came pre-installed and hasn’t failed me, yet. It’s basically a repackaging of Debian, with changes.

I do acknowledge the beauty of a web based workflow, but my aesthetics actually point me the other way, the way John Goerzen defends, based on local git, local review tools, local email clients. Perhaps these days it’s harder to set these up locally? This is something I have noticed at the office, for example: many developers don’t like to use git directly. They try to never go to the command line, preferring to use the Eclipse git plugin EGit instead. I am torn. On Windows, using git and and magit inside Emacs is slow. I only drop to the command line when I’m confused and need to make sure I know what’s going on. And we use GitLab to review code using merge requests. So yes, I understand the lure and the convenience of using an all-in-one website.

I still want to like the old school workflow using local tools and email.

Github constantly forces me to their website. I can’t very well work on bug reports, etc. without a strong Internet connection. And it’s designed to push people into using their tools and their interface, which is inferior in a lot of ways to a local interface – but then the process to pull down someone else’s set of patches involves a lot of typing and clicking, much more that would be involved from a simple git format-patch. In short, I don’t have my shortcut keys, my environment, etc. for reviewing things – the roadblocks are there to make me use theirs. – John Goerzen

I know this is true, and yet I’ve also enjoyed contributing code to projects hosted on GitHub and GitLab and other “software forges”. It’s smooth sailing if you’re all-in or all-out. Crossing over is what breaks your workflow.


Comments on 2019-03-13 Debian forever?

Glad to find more people making the point. Github was tolerable but the long term effects are starting to show.

orbifx 2019-03-18 15:07 UTC

Add Comment

2019-03-01 Podcast Numbers

I was wondering about the download statistics for my tiny little podcast. How would I figure this out? On my server, I keep four days of access logs. (See Privacy Policy for more information.) I posted the last episode three days ago and I verified that the last of my log files does not mention it. That means I didn’t miss any of the downloads.

So what I did is I grepped through the logs for the MP3 file, saving those lines for me to look through.

wc -l 20-halberds-and-helmets.log tells me that there are 48 lines. It’s not a lot but it’s what I’ve got.

A quick inspection shows that I have to discard HEAD requests. I should just be counting GET requests!

grep GET 20-halberds-and-helmets.log | wc -l gives me 37 lines.

Further inspection shows that quite a few of these requests have the status 206 (”partial content”) so that’s a single application downloading various parts of the file. But how to figure out which of them belong together? I’m trying to figure this out without looking at IP numbers.

Visually, it looks like I can determine what’s going on by looking at the minutes and the user agent for all the 206 results. Let’s try this. (And yes, I did have a HEAD request with a 206 result!)

grep 'GET.* 206 ' 20-halberds-and-helmets.log | perl -e '
while (<STDIN>) {
  chomp; my ($ts, $ua) =
  print "$ts $ua\n";

And here’s the result, with an arrow indicating the rows I consider to be “duplicates.”

20 Mozilla
44 AppleCoreMedia
44 AppleCoreMedia ←
39 iTMS
44 AppleCoreMedia
44 AppleCoreMedia ←
44 AppleCoreMedia ←
46 Mozilla
46 Mozilla ←
46 Mozilla ←
19 Mozilla

Manually counting them, I think we could get away by saying that we need to discount three AppleCoreMedia and two Mozilla results, right?

So let’s count the hits per user agent and then we’ll correct for the partial content results above.

grep GET 20-halberds-and-helmets.log | perl -e '
my %count;
while (<STDIN>) {
  chomp; my ($ua) = /"([^"\/]*)[^"]*"$/;
for my $ua (sort {$count{$b} <=> $count{$a}} keys %count) {
  print sprintf("%5d %s\n", $count{$ua}, $ua);
print "---- --------------------\n";
print sprintf("%5d total\n", $total);

This would be the result without correcting for the partial content:

   10 Mozilla
    5 AppleCoreMedia
    4 Pocket Casts
    3 Overcast
    3 PodcastAddict
    2 Dalvik
    2 okhttp
    2 Googlebot-Video
    2 stagefright
    1 AndroidDownloadManager
    1 iTMS
    1 Player FM
    1 iCatcher!
---- --------------------
   37 total

Making the correction I mentioned above:

    7 Mozilla ←
    4 Pocket Casts
    3 Overcast
    3 PodcastAddict
    2 AppleCoreMedia ←
    2 Googlebot-Video
    2 stagefright
    2 okhttp
    2 Dalvik
    1 AndroidDownloadManager
    1 iCatcher!
    1 iTMS
    1 Player FM
---- --------------------
   32 total ←

The result shows that Pocket Casts is popular. I guess it’s a podcatcher. iCatcher! is the one I use. 🙂

I’m surprised that Mozilla is up there. When I looked at the details of the user agent strings, I noticed that they mostly belong to bots:

  • two are Yandex bots
  • three ended up being Overcast (via their website?), but these are the three Mozilla results with 206 “partial content” so basically it’s just one listener from Overcast; the rest look legit

What the hell is Googlebot-Video doing, here? Is google offering audio search results somewhere? Using podcasts to train their AI overlords?

All the other user agents look like legitimate tools, frameworks, programming languages, libraries, etc.

That’s why I think that 32 people listened to the podcast episode 20, and a few probably didn’t listen to all of it.


Comments on 2019-03-01 Podcast Numbers

I use PodBean as my catcher, but I’m not sure what engine it uses for download.

Shelby 2019-03-01 18:28 UTC

Add Comment

2019-02-03 Network Issues

I seem to have some sort of network problem. I can reach my own web sites (and post this, for example). I can reach But I cannot reach and a bunch of other sites I’m trying to reach. The iPad has the same problem as the laptop, so it’s not the operating system but modem, ISP, or the Internet is down.


Comments on 2019-02-03 Network Issues

Whatever it was, it got fixed. Thanks, kamis of the Internet!

– Alex Schroeder 2019-02-03 21:59 UTC

Add Comment

2019-01-20 fail2ban to watch over my sites

I’m hosting my sites on a tiny server. At the same time, it’s all dynamic web apps and wikis and CGI scripts that take CPU and memory resources. Not a problem, if humans are browsing the site. If people try to download the entire site using automated tools that don’t wait between requests (leeches), or don’t check the meta information included in HTTP headers and HTML tags, then they overload my sites, and they get lost in the maze of links: page history, recent changes, old page revisions, you can download for ever if you’re not careful.

Enter fail2ban. This tool watches log files for regular expressions (filters) and if it find matches, it adds IP numbers to the firewall. You can then tell it which filters to apply to which log files and how many hits you’ll allow (jail).

When writing the rules, I just need to be careful: it’s OK to download a lot of static files. I just don’t want leeches or spammers (trying to brute force the questions I sometimes ask before people get to edit their first page on my sites).

Here’s my setup:



This is for the Apache web server with virtual hosts. The comment shows an example entry.

Notice the ignoreregex to make sure that some of the apps and directories don’t count.

Note that only newer versions of fail2ban will be able to match IPv6 hosts.

# Author: Alex Schroeder <>

# ANY match in the logfile counts!
# - - [24/Aug/2018:16:59:55 +0200] "GET /wiki/BannedHosts HTTP/1.1" 200 7180 "" "Pcore-HTTP/v0.44.0"
failregex = ^[^:]+:[0-9]+ <HOST> 

# Except cgit, css files, images...
# 0:0:0:0:0:0:0:0 - - [28/Aug/2018:09:14:39 +0200] "GET /cgit/bitlbee-mastodon/objects/9b/ff0c237ace5569aa348f6b12b3c2f95e07fd0d HTTP/1.1" 200 3308 "-" "git/2.18.0"
ignoreregex = ^[^"]*"GET /(robots\.txt |favicon\.ico |[^/ ]+.(css|js) |cgit/|css/|fonts/|pics/|1pdc/|gallery/|static/|munin/|osr/|indie/|face/|traveller/|hex-describe/|text-mapper/)



Yeah, I also make the wiki available via gopher...

# Author: Alex Schroeder <>

# 2018/08/25-09:08:55 CONNECT TCP Peer: "[]:56281" Local: "[]:70"
datepattern = ^%%Y/%%m/%%d-%%H:%%M:%%S

# ANY match in the logfile counts!
failregex = CONNECT TCP Peer: "\[<HOST>\]:\d+"



Now I need to tell fail2ban which log files to watch and which filters to use.

Note how I assume a human will basically click a link every 2s. Bursts are OK, but 20 hits in 40s are the limit.

Notice that the third jail just reuses the filter of the second jail.

enabled = true
port    = http,https
logpath = %(apache_access_log)s
findtime = 40
maxretry = 20

enabled = true
port    = 70
logpath = /home/alex/farm/gopher-server.log
findtime = 40
maxretry = 20

enabled = true
filter  = alex-gopher
port    = 7443
logpath = /home/alex/farm/gopher-server-ssl.log
findtime = 40
maxretry = 20


Comments on 2019-01-20 fail2ban to watch over my sites

Blimey. I don’t remember the last time anyone did anything gophery I noticed.

Blue Tyson 2019-01-30 10:47 UTC

The #gopher discussion is alive and well on Mastodon... 😊

– Alex Schroeder 2019-01-30 13:11 UTC

Something to review when I have a bit of time: Web Server Security by @infosechandbook.

– Alex Schroeder 2019-02-01 18:11 UTC

Add Comment

2018-11-11 ssh config

How to set up your SSH config file: Simplify Your Life With an SSH Config File, a blog post from 2011. I’m 45 and have been using GNU/Linux since way back before people had Internet at home here in Switzerland and I’ve only learned about this today. 😮

@danyspin97 recommended autossh which keeps connections alive. “Autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.” I don’t often run into this problem, though.

@zladuric recommended multiplexing which speeds up reconnecting to a server. “OpenSSH can reuse an existing TCP connection for multiple concurrent SSH sessions. This results into reduction of the overhead of creating new TCP connections.” I never saw this as problem, though. Probably because the host I connect to most often is in the same city as I am and I have fibre to the home. But if you’re a heavy duty system administrator perhaps?

@oliof recommended sshshuttle for people that forward many ports and don’t have a virtual private network (VPN). “Transparent proxy server that works as a poor man’s VPN. Forwards over ssh. Doesn’t require admin.” I don’t need that, either. Maybe if I had more need to circumventing geoblocking. 😁

Since we’re talking about SSH, here’s the most important server-side advice I have for you:

  1. use fail2ban to temporarily ban IP numbers that cause too many failed login attempts
  2. change the default port to something else in orther to reduce the number of login attempts
  3. if you need to allow root access on your system (which I do if I want to make backups using rsync) then use the prohibit-password option

The last two are changes you make to the /etc/ssh/sshd_config file. Uncomment #Port 22 and change the port, and set the PermitRootLogin option to prohibit-password.


Comments on 2018-11-11 ssh config

One thing I like to do is to configure a group, which a user has to be in, to be allowed to ssh in. So the default is that a user can’t.

This avoids e.g. the problem where you create a dummy user with a weak password to test something, and forget to delete the user. Not that I’ve ever done that. Noooo.

I’ve split the users allowed to ssh in up into “sshusers” and “sshkeyusers”, the first category is allowed to use a password (basically, me), and the second category can only login with a key (the handful of friends that have access). In /etc/ssh/sshd_config I have:

# Only allow sshusers/sshkeyusers to ssh in:
AllowGroups sshusers sshkeyusers
Match Group sshkeyusers
  PasswordAuthentication no

Adam 2018-11-11 15:22 UTC

Hahaha! 😎

Luckily, that server should only have me on it. But you’re right. Better be safe than sorry!

– Alex Schroeder 2018-11-11 18:32 UTC

I currently have the following:

# List of user names allowed to log in.
# We need root for our backup script.
# We need git for git-via-SSH repos.
AllowUsers alex git root

PermitRootLogin prohibit-password

So let me think this through:

  1. I want alex to connect using a password just in case.
  2. I don’t want git to connect using a password (not sure why it seems to have a password in /etc/shadow and /usr/bin/git-shell as its shell in /etc/passwd and I don’t know whether any of these obviate a setting in /etc/sshd_config) 🤔❓
  3. I don’t want root to connect using a password.

So I added the following:

Match User git
    PasswordAuthentication no

Seems to work as intended, for now.

– Alex Schroeder 2018-11-12 13:26 UTC

“The rage that one felt was an abstract, undirected emotion which could be switched from one object to another like the flame of a blowlamp.” – George Orwell, 1984, chapter 1

– Alex Schroeder 2018-11-12 13:30 UTC

Add Comment

2018-11-10 Manual Pages

I like people who write their own man pages with the most important information they keep forgetting about existing commands. Keep this info in text files and use a little script like mman to access those text files; keep using man for the original. I found this via tfurrows’s gopher site.

On Mastodon, I got some more suggestions:

@raucao suggested tldr. I installed the Perl variant via cpanm tldr.

alex@melanobombus:~$ tldr emacs

  The extensible, customizable, self-documenting, real-time display editor.

  ✓ Open emacs in console mode (without X window):

    🍣  emacs -nw

  ✓ Open a file in emacs:

    🍣  emacs {{filename}}

  ✓ Exit emacs:

    🍣  C-x C-c

Know about ed I can tell you that knowing how to quit is essential information. But surely there is more to it?

@coffe suggested cheat. I installed it via pip3 install cheat. And I set some options in my ~/bashrc:

export EDITOR=emacsclient
export CHEATCOLORS=true

You need to provide an editor because you can edit your cheat sheets. That’s the spirit!

The colourisation for the Emacs page is wrong. I guess it isn’t handled as Markdown? But at least there’s more information available. Perhaps it’s too much?

alex@melanobombus:~$ cheat emacs
# Running emacs

  GUI mode            $ emacs
  Terminal mode       $ emacs -nw

# Basic usage

  Indent              Select text then press TAB
  Cut                 CTRL-w
  Copy                ALT-w
  Paste ("yank")      CTRL-y
  Begin selection     CTRL-SPACE
  Search/Find         CTRL-s
  Replace             ALT-% (ALT-SHIFT-5)
  Save                CTRL-x CTRL-s
  Save as             CTRL-x CTRL-w
  Load/Open           CTRL-x CTRL-f
  Undo                CTRL-x u
  Highlight all text  CTRL-x h
  Directory listing   CTRL-x d
  Cancel a command    CTRL-g
  Font size bigger    CTRL-x CTRL-+
  Font size smaller   CTRL-x CTRL--

# Buffers

  Split screen vertically                         CTRL-x 2
  Split screen vertically with 5 row height       CTRL-u 5 CTRL-x 2
  Split screen horizontally                       CTRL-x 3
  Split screen horizontally with 24 column width  CTRL-u 24 CTRL-x 3
  Revert to single screen                         CTRL-x 1
  Hide the current screen                         CTRL-x 0
  Move to the next screen                         CTRL-x o
  Kill the current buffer                         CTRL-x k
  Select a buffer                                 CTRL-x b
  Run command in the scratch buffer               CTRL-x CTRL-e

# Navigation ( backward / forward )
  Character-wise                                  CTRL-b , CTRL-f
  Word-wise                                       ALT-b  , ALT-f
  Line-wise                                       CTRL-p , CTRL-n
  Sentence-wise                                   ALT-a  , ALT-e
  Paragraph-wise                                  ALT-{ , ALT-}
  Function-wise                                   CTRL-ALT-a , CTRL-ALT-e
  Line beginning / end                            CTRL-a , CTRL-e

# Other stuff

  Open a shell         ALT-x eshell
  Goto a line number   ALT-x goto-line
  Word wrap            ALT-x toggle-word-wrap
  Spell checking       ALT-x flyspell-mode
  Line numbers         ALT-x linum-mode
  Toggle line wrap     ALT-x visual-line-mode
  Compile some code    ALT-x compile
  List packages        ALT-x package-list-packages

# Line numbers

  To add line numbers and enable moving to a line with CTRL-l:

    (global-set-key "\C-l" 'goto-line)
    (add-hook 'find-file-hook (lambda () (linum-mode 1)))

I am reminded of my own recommendation regarding Emacs. I wrote this in 2006. Reference Sheet by Alex Schroeder (for Emacs).


Comments on 2018-11-10 Manual Pages

I’ve done something similar using BASH functions. I first saw these in a comment on hackernews, and have simply taken them wholesale.

mdless() {
      pandoc -s -f markdown -t man $ | groff -T utf8 -man | less
umedit() { mkdir -p ~/docs/linux_stuff/knowledge; vim ~/docs/linux_stuff/knowledge/$; }
um() { mdless ~/docs/linux_stuff/knowledge/"$1"; }
umls() { ls ~/docs/linux_stuff/knowledge; }

However, these leave a little to be desired in terms of completeness. I’ve been mostly satisfied with them so far though!

– reed 2018-11-12 20:37 UTC

Hah, to translate from Markdown to man is genius. :)

– Alex 2018-11-13 06:22 UTC

Add Comment

2018-11-06 Memory Mystery

The mysteries of problems regarding my website continues. What the hell happened?

Sudden Spike

Memory for “apps” went up to 16G at around 21:00 and then almost everything was killed.

Let’s see what grep oom-killer /var/log/syslog has to say:

Nov  6 21:05:03 sibirocobombus kernel: [18425702.447121] /home/alex/farm invoked oom-killer: ...
Nov  6 21:05:09 sibirocobombus kernel: [18425708.244062] munin-update invoked oom-killer: ...
Nov  6 21:05:09 sibirocobombus kernel: [18425708.925423] /home/alex/farm invoked oom-killer: ...
Nov  6 21:05:13 sibirocobombus kernel: [18425712.660969] apache2 invoked oom-killer: ...
Nov  6 21:05:14 sibirocobombus kernel: [18425713.857753] fail2ban-server invoked oom-killer: ...
Nov  6 21:05:15 sibirocobombus kernel: [18425714.904769] fw_conntrack invoked oom-killer: ...
Nov  6 21:05:18 sibirocobombus kernel: [18425717.168963] apache_accesses invoked oom-killer: ...
Nov  6 21:05:18 sibirocobombus kernel: [18425717.985385] apache_accesses invoked oom-killer: ...
Nov  6 21:05:20 sibirocobombus kernel: [18425719.907638] vmstat invoked oom-killer: ...
Nov  6 21:05:23 sibirocobombus kernel: [18425722.207184] jbd2/vda1-8 invoked oom-killer: ...
Nov  6 21:05:24 sibirocobombus kernel: [18425723.800850] memory invoked oom-killer: ...
Nov  6 21:05:27 sibirocobombus kernel: [18425726.582062] exiqgrep invoked oom-killer: ...
Nov  6 21:05:29 sibirocobombus kernel: [18425728.479563] fail2ban-server invoked oom-killer: ...
Nov  6 21:05:31 sibirocobombus kernel: [18425730.862151] fail2ban-client invoked oom-killer: ...
Nov  6 21:05:31 sibirocobombus kernel: [18425731.049454] /home/alex/camp invoked oom-killer: ...

OK, and what happened back then?

Grepping /var/log/apache2/access.log for this minute does not reveal anything interesting.

My gopher logs show just two hits, no issues.

Checking the toadfarm log file shows the first problem at 21:05:00: “fork failed: Cannot allocate memory”. Hm. The other problem is at 21:05:11: “Worker 4166 has no heartbeat (30 seconds), restarting”. Could be anything? Grepping for “Inactivity timeout” gives me 36 hits from 21:04:56 to 21:05:12. The end must have come quickly.

Hm. Looking at the toadfarm log file, the number of hits to the website of my Arabic teacher in just two minutes is pretty amazing.        388    56%        273    40%         14     2%
                              6     0%

This is grep '21:0[45]:.*' /home/alex/farm/farm*.log | /home/alex/bin/site-grouping and the second script is very simple:

#!/usr/bin/env perl
while (<STDIN>) {
  m!GET https?://([^:]+):!;
@result = sort {$site{$b} <=> $site{$a}} keys %site;
foreach $label (@result) {
  printf "%20s %10d   %3d%%\n", $label, $site{$label}, 100* $site{$label} / $total;

OK, so who’s calling these two sites?

Note that this is toadfarm reporting. At the same time, Apache isn’t reporting as much. grep "21:0[45]" /var/log/apache2/access.log | wc -l says only 183 requests. When I look at the IP numbers, I don’t see a single IP number doing anything wrong. After a while I do think that there’s a lot of IP numbers starting with a common prefix and so I’m going to count them. 62 hits in two minutes from ChinaNetCenter Ltd. Gaaaah. Looking at the URLs these IP numbers are hitting it’s full of admin actions, diffs, edit action. Clearly, a misbehaving spider. I think I’m going to ban them all, for now.

Soon, I’m going follow these instructions. Grrrrr! Unexpected DDOS: Blocking China with ipset and iptables. (Just kidding. I hate that!)

Gah. Problem. ipset add blacklist ended with an error. The list is full! And I cannot just ipset del blacklist because that gives me another error saying the element isn’t added. Not even for ip in $(ipset list blacklist|grep 220); do ipset del blacklist $ip; done seems to work. Or at least, judging from a few calls to ipset list blacklist | wc -l it’s not very fast. What now?

I guess I will have to look at my setup again. I probably need a different type? It’s type is hash:ip and it’s maxelem is 65536. That would have to increase, of course. Oh well. Perhaps I’ll leave it for some other time. We’ll see how far we’ll get by not doing anything.


Comments on 2018-11-06 Memory Mystery

How is 388 requests in 2 minutes a problem? That’s no more than 4 requests per second - shouldn’t the webserver be able to handle that?

Is there some kind of memory leak or pathological case that the chinese crawlers happen to hit? The sudden spike in memory use is not just because of 4 hits/s, is it?

Can you reproduce the memory use, maybe on a local machine replaying the requests from the log around the time of the problem (or just bombarding it with requests)?

Just random questions from my armchair :-)

Adam 2018-11-09 21:46 UTC

Heh. :) The main problem is that we’re not talking about static HTML pages. If somebody does a search on this wiki, there’s a grep over 7000 pages, and then the wiki code looks at the matches, renders the pages, highlights the phrases, and so on. That means the process takes a while. If it takes 10 seconds, and I get 4 searches per second (I doubt this was the explanation, just trying to explain how the server is prone to accidental denial of service attacks), then I have 40 processes grepping through 7000 files per second. And then it dies, because this is a tiny 2G virtual machine I’m somewhere, managed by myself and nobody else. At least that’s what I suspect.

That’s why my /robots.txt says Crawl-delay: 20.

Replaying the requests from the log is a good idea. I should do it the next time this happens. And I should make sure to compare the memory used with the number of processes running at the time.

– Alex Schroeder 2018-11-10 00:30 UTC

More armchair comments: Could you exclude the search URL in robots.txt?

A more laborous solution, if searching turns out to be the problem, is to index the pages using, say, Xapian (the Perl-bindings work nicely).

Anyway, good luck in finding the cause!

Adam 2018-11-10 13:49 UTC

As they are already ignoring the crawl delay I think it is safe to say that they either don’t know or don’t care. My robot.txt Contains the line Disallow: /wiki? which should take care of anything that is not a page view. But it’s being ignored.

A long time ago I used a Perl module that indexed the text and allowed scoring of results - but eventually it turned out that brute forcing search using grep was actually faster. But that was years ago and on a different setup. Perhaps I should revisit this.

These days I am more interested in blocking misbehaving agents. The old solution limits visits from the same IP number (surge protection, and I have fail2ban watching my log files, to watch over all the other services (cgit and the like). But if the request come from all over a subnet, these defenses don’t work, unfortunately.

– Alex Schroeder 2018-11-10 18:09 UTC

Add Comment

2018-10-13 cgit

@aminb asked my about my cgit setup.

This is /etc/cgitrc:

# cgit config
# see cgitrc(5) for details

root-title=Git repositories
root-desc=Code repositories hosted by Alex Schroeder




This means that the /cgit path from all my virtual servers is going to end up serving cgit. I don’t mind.

The /home/git directory has all my repositories:

drwxr-xr-x 24 git  git  4096 Sep 13 23:14 .
drwxr-xr-x  4 root root 4096 Apr 20 15:20 ..
-rw-------  1 git  git   146 Apr 20 15:27 .bash_history
-rw-r--r--  1 git  git   220 Apr 20 15:20 .bash_logout
-rw-r--r--  1 git  git  3526 Apr 20 15:20 .bashrc
drwxrwsr-x  7 git  git  4096 Sep 10 10:38 bitlbee-mastodon.git
drwxrwsr-x  7 git  git  4096 Jun 21 11:10 character-sheet-generator.git
drwxrwsr-x  7 git  git  4096 Sep 14 23:41 emacs-setup.git
drwxrwsr-x  7 git  git  4096 Jun 21 11:10 face-generator.git
-rw-r--r--  1 git  git    39 Jun 21 11:07 .gitconfig
drwxrwsr-x  7 git  git  4096 Jun 21 11:10 gridmapper.git
drwxrwsr-x  7 git  git  4096 Jul 20 00:12 halberds-and-helmets.git
drwxrwsr-x  7 git  git  4096 Jun 21 11:10 hellebarden-und-helme.git
drwxr-xr-x  7 git  git  4096 Jun 25 14:50 hex-mapping.git
drwxrwsr-x  7 git  git  4096 Oct  8 23:26 mastodon-archive.git
drwxrwsr-x  7 git  git  4096 Sep 17 07:42 mastodon-blocker.git
drwxrwsr-x  7 git  git  4096 Jul 16 15:34 moku-pona.git
drwxrwsr-x  7 git  git  4096 Jul 24 15:50 mu.git
drwxrwsr-x  7 git  git  4096 Sep 13 18:08 oddmu.git
drwxrwsr-x  7 git  git  4096 Oct  7 20:54 oddmuse6.git
drwxrwsr-x  7 git  git  4096 Oct  7 18:15 oddmuse.git
-rw-r--r--  1 git  git   675 Apr 20 15:20 .profile
drwxrwsr-x  7 git  git  4096 Aug 30 23:52 qmk_firmware.git
-rw-r--r--  1 root root  122 Jun  4 08:44 README
-rw-r--r--  1 root root   49 Jun  4 08:43 README~
-rw-r--r--  1 root root  494 Jun  9 00:36 root.html
-rw-r--r--  1 root root   14 Jun  9 00:30 root.html~
drwxrwsr-x  7 git  git  4096 Jul 17 13:57 sitelen-mute.git
drwxrwsr-x  7 git  git  4096 Jul 31 11:47 spellcasters.git
drwx------  2 git  git  4096 Jun 13 09:13 .ssh
drwxrwsr-x  7 git  git  4096 Sep 11 09:08 text-mapper-mastodon-bot.git
drwxrwsr-x  7 git  git  4096 Sep 18 11:59 trunk.git
drwxrwsr-x  7 git  git  4096 Aug 31 00:04 typing.git

Because I’m evil, the .gitconfig says:

	denynonfastforwards = false

The README file says:

mkdir project.git
cd project.git
git init --bare

My root user has a little script called add-repo which I have to run to create new projects:


if [ -z $1 ]; then
    echo Must provide the name of a new repo
    exit 1
elif [ ! -z $2 ]; then
    echo Must provide just the name a new repo
    exit 1
elif [ -e $1 ]; then
    echo This file exists already
    exit 1


mkdir "$dir"
chown git.git "$dir"
(cd "$dir" && sudo -u git git init --bare --shared)
echo Provide description
read desc &&  echo $desc > "$dir/description"

Remember the /etc/cgitrc file also mentioned some scripts for formatting. /usr/lib/cgit/filters/ is what I frankensteined together. See 2018-06-11 Highlighting Code and Rendering Markdown for cgit for more.


Add Comment



Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.