These are the pages for all my system administration woes. I rarely post happy things in this section. You have been warned.

2020-12-22 Website down, disk full, logs crazy

This little website of mine was down because the logs had gone crazy. The reason, I think, is that somebody has been using Phoebe to download all of Wikipedia (more or less) and the table rendering code in there issued a warning for every table row it encountered…

Comments on 2020-12-22 Website down, disk full, logs crazy

I definitely need to integrate Phoebe in my fail2ban setup – which I was hoping to avoid because that requires me to log IP numbers.

– 2020-12-22 10:40 UTC

Add Comment

2020-12-22 Apache config file to block user agents

This is a long one… My “/etc/apache2/conf-enabled/blocklist.conf” has grown over the years. It all started with a blog post I had seen entitled “stop fake user agents with htaccess”. The URL is somewhere in the config below, if you’re interested.

RewriteEngine on

# fediverse instances asking for previews
RewriteCond "%{HTTP_USER_AGENT}" "Mastodon|Friendica|Pleroma" [nocase]
# except for Epicyon on the domain
# RewriteCond "%{HTTP_HOST}" !""
# except for Webfinger stuff
# RewriteCond "%{REQUEST_URI}" !"\.well-known/webfinger"
# except for Oddmuse ActivityPub
# RewriteCond "%{QUERY_STRING}" !"action=(webfinger|actor|inbox|outbox)"
RewriteRule ^(.*)$ - [forbidden,last]

# misbehaving bots
RewriteCond "%{HTTP_USER_AGENT}" "pcore|megaindex|semrushbot|wiederfrei" [nocase]
# forbidden!
RewriteRule ^(.*)$ - [forbidden,last]


## Detect abnormal user-agents by
## Version 1.656, 2018-12-03
## Part 1 - basic rules
# Add your informative page to prevent redirection loops
# RewriteCond %{REQUEST_URI} !strangebrowser\.html
## Exclusions
# Covenant Eyes parental monitoring
# RewriteCond %{REMOTE_ADDR} !^69\.41\.14\.
# VirusTotal Cloud uses MSIE 9.0; Windows NT 9.0 user agent
# RewriteCond %{HTTP_USER_AGENT} !virustotalcloud\)$
# Google-SearchByImage uses mismatching version numbers
RewriteCond %{HTTP_USER_AGENT} !\ Google-SearchByImage\)
## UA blacklist
# Known bad bots ignoring or not reading robots.txt
RewriteCond %{HTTP_USER_AGENT} "centurybot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "cognitiveseo\.com" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "DnyzBot/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "evc-batch/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Facebot\ Twitterbot/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Gluten\ Free\ Crawler" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Gowikibot/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "GrapeshotCrawler/" [OR]
RewriteCond %{HTTP_USER_AGENT} "IndeedBot\ " [OR]
RewriteCond %{HTTP_USER_AGENT} "linkdexbot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MS\ Search\ 6\.0\ Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "PaperLiBot/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "PowerMapper\.com" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "raventools\.com" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "RukiCrawler" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "SemrushBot" [OR]
RewriteCond %{HTTP_USER_AGENT} "SeoBotM6" [OR]
RewriteCond %{HTTP_USER_AGENT} "seocharger" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "SEOkicks-Robot" [OR]
RewriteCond %{HTTP_USER_AGENT} "SMTBot/" [OR]
RewriteCond %{HTTP_USER_AGENT} "XoviBot/" [OR]
RewriteCond %{HTTP_USER_AGENT} "ZoomBot" [OR]
# Known bad WordPress login page bot; Firefox does not use full version numbers anymore
RewriteCond %{HTTP_USER_AGENT} "rv:40\.0\)\ Gecko/20100101\ Firefox/40\.1$" [NC,OR]
# Known site analysis bots that try to reveal sensitive data about your server
RewriteCond %{HTTP_USER_AGENT} "Wappalyzer" [OR]
# Non-standard beginnings
# RewriteCond %{HTTP_USER_AGENT} ^-?$ [OR]
RewriteCond %{HTTP_USER_AGENT} "^\(" [OR]
RewriteCond %{HTTP_USER_AGENT} ^\' [OR]
RewriteCond %{HTTP_USER_AGENT} "^\ " [OR]
RewriteCond %{HTTP_USER_AGENT} ^\" [OR]
# RewriteCond %{HTTP_USER_AGENT} ^- [OR]
RewriteCond %{HTTP_USER_AGENT} ^= [OR]
RewriteCond %{HTTP_USER_AGENT} ^\.$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^\\$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^\d [OR]
RewriteCond %{HTTP_USER_AGENT} "^Chrome" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Empty" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Firefox" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^IE" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Internet\ Explorer" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^MSIE" [NC,OR]
# Non-standard endings
RewriteCond %{HTTP_USER_AGENT} "Chrome$" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Firefox$" [NC,OR]
# Way too short user agent strings
RewriteCond %{HTTP_USER_AGENT} "^Mozilla$" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Mozilla/\d\.\d$" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Mozilla\ compatible$" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Mozilla/\d\.\d\ \(compatible\)$" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Mozilla/\d\.\d\ \(compatible;\)$" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Opera$" [NC,OR]
# Repeating same stuff
RewriteCond %{HTTP_USER_AGENT} "compatible.*compatible" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Gecko.*Gecko" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Mozilla.*Mozilla" [NC,OR]
# Missing space after closing parenthesis
RewriteCond %{HTTP_USER_AGENT} "\)([a-z|A-Z])" [NC,OR]
# Fake mixtures of browsers
RewriteCond %{HTTP_USER_AGENT} "Firefox.*Netscape" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Firefox.*Opera" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE.*Chrome/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE.*Firefox" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE.*Edge/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE.*rv:" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Opera.*Trident/" [NC,OR]
# Letters instead of version numbers
RewriteCond %{HTTP_USER_AGENT} "Chrome/([a-z]|[A-Z])\." [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Firefox/([a-z]|[A-Z])\." [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Gecko/([a-z]|[A-Z])\." [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/([a-z]|[A-Z]) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ ([a-z]|[A-Z])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Opera/([a-z]|[A-Z])\." [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "\ rv:([a-z]|[A-Z])\." [NC,OR]
# Impossible Mozilla versions
RewriteCond %{HTTP_USER_AGENT} "Mozilla/([0-3]|[6-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]|6[0-9]|7[0-9]|8[0-9]|9[0-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Mozilla/\d\.([1-9])" [NC,OR]
# Impossible MSIE versions
RewriteCond %{HTTP_USER_AGENT} "MSIE\ \d\.([1-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ \d\d\.([1-9])" [NC,OR]
# Impossible MSIE versions on certain Windows versions
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 5\..*\ NT\ 6\." [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 5\..*\ NT\ 10" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 6\..*\ NT\ 6\." [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 6\..*\ NT\ 10" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 8\..*\ NT\ 6\.2" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 8\..*\ NT\ 6\.3" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 8\..*\ NT\ 10" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 9\..*\ NT\ 6\.2" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 9\..*\ NT\ 6\.3" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 9\..*\ NT\ 10" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 10\..*\ NT\ 6\.3" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 10\..*\ NT\ 10" [NC,OR]
# Impossible MSIE and Trident combinations
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 8\..*\ Trident/([0-3]|[5-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 9\..*\ Trident/([0-4]|[6-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ 10\..*\ Trident/([0-5]|[7-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ".*\ Trident/([0-6]|[8-9]);\ rv:11\.0" [NC,OR]
# Other impossible browser or engine version numbers
RewriteCond %{HTTP_USER_AGENT} "Chrome/99" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Chrome/\d\d\d" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Firefox/99" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Firefox/\d\d\d" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\ \d\d\d" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Opera/9\.99" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Opera/\d\d\d" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Presto/9" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Trident/\d\d" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Trident/([0-3]|[8-9])" [NC,OR]
# Mismatching Firefox and rv versions
RewriteCond %{HTTP_USER_AGENT} ".*rv:1\..*Firefox/(0|[2-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ".*rv:2\..*Firefox/([0-1]|[2-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ".*rv:3\..*Firefox/([0-2]|[4-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ".*rv:4\..*Firefox/([0-3]|[5-9])" [NC,OR]
# Fake user agents used while testing programs and apps
RewriteCond %{HTTP_USER_AGENT} "^MyApp$" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Synapse\)" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Test\ Certificate\ Info" [NC,OR]
# Unblocking because of false positives like "powerpc64le-unknown-linux-gnu" and "Tiny Tiny RSS/UNKNOWN"
# RewriteCond %{HTTP_USER_AGENT} "Unknown" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "WinHTTP" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "WinHttpRequest" [NC,OR]
# Other standards-breaking fake user agents
RewriteCond %{HTTP_USER_AGENT} "\ \(Chrome\)" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "\(Win/$" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "compatible\ ;" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "CAIMEO\ Artificial\ Intelligence" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Gecko/\ " [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Gecko/20([2-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "-IE\d" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "\ IE\d" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Mozilla\ ([0-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Mozilla/\d\.\d\(" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Mozilla/\d\.\d\ \(\ " [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "MSIE\d" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^QuickTime/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "SuperCleaner" [OR]
RewriteCond %{HTTP_USER_AGENT} "User\ Agent" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows/" [NC,OR]
# Fake Windows strings and versions
RewriteCond %{HTTP_USER_AGENT} "Windows/" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ ([0-2]|[4-8])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ NT\)" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ NT\ 0" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ NT\ 1\." [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ NT\ ([2-3]|[7-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ NT\ 5\.([3-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ NT\ 6\.([4-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ NT\ 10\.([1-9])" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Windows\ NT\ (1[1-9]|4[0-9]|5[0-9]|6[0-9])" [NC,OR]
# Suspicious user agents
RewriteCond %{HTTP_USER_AGENT} "TO-Browser/TOB" [NC]
# Redirect during the testing period
# RewriteRule .* /strangebrowser.html? [R=307,L]
# Comment out the line above and remove the comment mark below to block fake browsers after the testing period
RewriteRule ^.*$ - [F,L]

## Part 2 - additional bad browser rules with conditions
# 1. Double Trident is used by MSNBot
# Add your informative page to prevent redirection loops
# RewriteCond %{REQUEST_URI} !strangebrowser\.html
# RewriteCond %{REMOTE_ADDR} !^131\.253\.25\.
RewriteCond %{HTTP_USER_AGENT} "Trident/.*Trident" [NC]
# Redirect during the testing period
# RewriteRule .* /strangebrowser.html? [R=307,L]
# Comment out the line above and remove the comment mark below to block fake browsers after the testing period
RewriteRule ^.*$ - [F,L]

# 2. Firefox full version numbers are used by WaterFox
# Add your informative page to prevent redirection loops
# RewriteCond %{REQUEST_URI} !strangebrowser\.html
RewriteCond %{HTTP_USER_AGENT} !Waterfox/\d\d\.
RewriteCond %{HTTP_USER_AGENT} !Waterfox\)
RewriteCond %{HTTP_USER_AGENT} ".*Firefox/\d\d\.\d\."
# Redirect during the testing period
# RewriteRule .* /strangebrowser.html? [R=307,L]
# Comment out the line above and remove the comment mark below to block fake browsers after the testing period
RewriteRule ^.*$ - [F,L]

# 3. MSIE user string must also have a Trident version
# Add your informative page to prevent redirection loops
# RewriteCond %{REQUEST_URI} !strangebrowser\.html
# Exclusions
# VirusTotal Cloud uses MSIE 9.0; Windows NT 9.0 user agent
# RewriteCond %{HTTP_USER_AGENT} !virustotalcloud\)$
## The rule
RewriteCond %{HTTP_USER_AGENT} !Trident/
RewriteCond %{HTTP_USER_AGENT} MSIE\ .*Windows
# Redirect during the testing period
# RewriteRule .* /strangebrowser.html? [R=307,L]
# Comment out the line above and remove the comment mark below to block fake browsers after the testing period
RewriteRule ^.*$ - [F,L]

# 4. PaleMoon has higher Gecko version numbers than other browsers
# Add your informative page to prevent redirection loops
# RewriteCond %{REQUEST_URI} !strangebrowser\.html
RewriteCond %{HTTP_USER_AGENT} !PaleMoon/
RewriteCond %{HTTP_USER_AGENT} "Gecko/201([1-9])"
# Redirect during the testing period
# RewriteCond %{REQUEST_URI} !strangebrowser\.html
# RewriteCond %{REMOTE_ADDR} !^131\.253\.25\.
# Redirect during the testing period
# RewriteRule .* /strangebrowser.html? [R=307,L]
# Comment out the line above and remove the comment mark below to block fake browsers after the testing period
RewriteRule ^.*$ - [F,L]

Add Comment

2020-08-29 Restarting Phoebe

I have trouble restarting Phoebe and other services I run. I guess I don’t understand how these forking processes work. These processes fork for every request they get, and that works. When they get a lot of requests, however, they enter some sort of failed state. And by “a lot” I mean more than a hundred or so. I know that it’s not a lot but I want these services to work for the small net and so I don’t want to spend any effort in trying to make them handle more. I don’t want to start caching HTTP requests, for example. What irks me most of all is that this doesn’t happen because hundreds of visitors want to know about my stuff. No, I share a link on Mastodon, and my post gets federated, and then every single server on the fediverse tries to get a preview image to display.

I’ve solved this issue for all my services behind Apache by blocking all the fediverse user agents, but Phoebe runs without a web server front-end. I’ve tried to abort as soon as possible, using the same regular expression, but that doesn’t seem to work.

But I have a second layer of defence: Monit watches over my processes. Please forgive the huge start program option. Maybe it’s time to move some of that into a config file. Please just scroll down. I also didn’t want to shorten it, because I think it’s an interesting snapshot of a non-trivial Phoebe setup.

Let’s go through this.

First we have a PID file, where the process ID is going to be. This is how Monit identifies the parent process responsible for the service. The --pid_file option is what tells Phoebe to write the same file. So far so good.

check process phoebe with pidfile /home/alex/farm/
    start program = "/usr/bin/perl -I/home/alex/phoebe/lib /home/alex/farm/phoebe
 --setsid --user=alex --group=alex
 --log_level=3 --log_file=/home/alex/farm/phoebe.log
 --wiki_dir=/home/alex/phoebe --cert_file=/var/lib/dehydrated/certs/ --key_file=/var/lib/dehydrated/certs/ --cert_file=/var/lib/dehydrated/certs/ --key_file=/var/lib/dehydrated/certs/ --cert_file=/var/lib/dehydrated/certs/ --key_file=/var/lib/dehydrated/certs/ --cert_file=/var/lib/dehydrated/certs/ --key_file=/var/lib/dehydrated/certs/ --cert_file=/var/lib/dehydrated/certs/ --key_file=/var/lib/dehydrated/certs/ --cert_file=/var/lib/dehydrated/certs/ --key_file=/var/lib/dehydrated/certs/
 --wiki_main_page=Welcome --wiki_pages=About
 --wiki_mime_type=image/png --wiki_mime_type=image/jpeg

OK, with that out of the way, let’s talk about the important stuff: stopping and restarting the process, and determining when to restart the process.

    # leave enough time after a stop for the server to recover before starting
    stop program = "/bin/bash -c 'kill -s SIGKILL `cat /home/alex/farm/`; sleep 120'"
    if failed
	port 1965
	type tcpssl
        send "gemini://\r\n"
	expect "20 .*"
	for 5 cycles
	then restart
    if totalmem > 100 MB for 5 cycles then restart
    if 6 restarts within 15 cycles then stop

Monit checks the service using a regular request, once every cycle (5min). If it fails five times in a row (25min), it restarts. It also restarts when total memory is more than 100MB five times in a row. And when it had to restart six times in 15 cycles (75min), then the process gets stopped.

What happens on a restart? First the program is stopped and then it is started. But yesterday for example:

[CEST Aug 29 00:39:05] error    : 'phoebe' total mem amount of 190.0 MB matches resource limit [total mem amount>100 MB]
[CEST Aug 29 00:39:05] info     : 'phoebe' trying to restart
[CEST Aug 29 00:39:05] info     : 'phoebe' stop: '/bin/bash -c kill -s SIGKILL `cat /home/alex/farm/`; sleep 120'
[CEST Aug 29 00:39:35] info     : 'phoebe' start: '/usr/bin/perl -I/home/alex/phoebe/lib /home/alex/farm/phoebe --setsid --user=alex --group=alex --log_level=3 --log_file=/home/alex/farm/phoebe.log --pid_file=/home/alex/farm/ --wiki_dir=/home/alex/phoebe --cert_file=/va...'
[CEST Aug 29 00:44:42] error    : 'phoebe' process is not running
[CEST Aug 29 00:44:42] info     : 'phoebe' trying to restart
[CEST Aug 29 00:44:42] info     : 'phoebe' start: '/usr/bin/perl -I/home/alex/phoebe/lib /home/alex/farm/phoebe --setsid --user=alex --group=alex --log_level=3 --log_file=/home/alex/farm/phoebe.log --pid_file=/home/alex/farm/ --wiki_dir=/home/alex/phoebe --cert_file=/va...'
[CEST Aug 29 00:49:57] error    : 'phoebe' process is not running
[CEST Aug 29 00:49:57] info     : 'phoebe' trying to restart
[CEST Aug 29 00:49:57] info     : 'phoebe' start: '/usr/bin/perl -I/home/alex/phoebe/lib /home/alex/farm/phoebe --setsid --user=alex --group=alex --log_level=3 --log_file=/home/alex/farm/phoebe.log --pid_file=/home/alex/farm/ --wiki_dir=/home/alex/phoebe --cert_file=/va...'
[CEST Aug 29 00:55:02] error    : 'phoebe' process is not running
[CEST Aug 29 00:55:02] info     : 'phoebe' trying to restart
[CEST Aug 29 00:55:02] info     : 'phoebe' start: '/usr/bin/perl -I/home/alex/phoebe/lib /home/alex/farm/phoebe --setsid --user=alex --group=alex --log_level=3 --log_file=/home/alex/farm/phoebe.log --pid_file=/home/alex/farm/ --wiki_dir=/home/alex/phoebe --cert_file=/va...'
[CEST Aug 29 01:00:12] error    : 'phoebe' process is not running
[CEST Aug 29 01:00:12] info     : 'phoebe' trying to restart
[CEST Aug 29 01:00:12] info     : 'phoebe' start: '/usr/bin/perl -I/home/alex/phoebe/lib /home/alex/farm/phoebe --setsid --user=alex --group=alex --log_level=3 --log_file=/home/alex/farm/phoebe.log --pid_file=/home/alex/farm/ --wiki_dir=/home/alex/phoebe --cert_file=/va...'
[CEST Aug 29 01:05:19] error    : 'phoebe' process is not running
[CEST Aug 29 01:05:19] info     : 'phoebe' trying to restart
[CEST Aug 29 01:05:19] info     : 'phoebe' start: '/usr/bin/perl -I/home/alex/phoebe/lib /home/alex/farm/phoebe --setsid --user=alex --group=alex --log_level=3 --log_file=/home/alex/farm/phoebe.log --pid_file=/home/alex/farm/ --wiki_dir=/home/alex/phoebe --cert_file=/va...'
[CEST Aug 29 01:10:25] error    : 'phoebe' service restarted 6 times within 6 cycles(s) - stop

Why isn’t the process running? Here’s a selection from the other log:

2020/08/29-00:39:38 App::Phoebe (type Net::Server::Fork) starting! pid(19496)
2020/08/29-00:39:38 Cannot connect to SSL port 1965 on [Address already in use]
2020/08/29-00:44:43 App::Phoebe (type Net::Server::Fork) starting! pid(20808)
2020/08/29-00:44:43 Cannot connect to SSL port 1965 on [Address already in use]
2020/08/29-00:49:59 App::Phoebe (type Net::Server::Fork) starting! pid(22125)
2020/08/29-00:49:59 Cannot connect to SSL port 1965 on [Address already in use]
2020/08/29-00:55:05 App::Phoebe (type Net::Server::Fork) starting! pid(23449)
2020/08/29-00:55:05 Cannot connect to SSL port 1965 on [Address already in use]
2020/08/29-01:00:15 App::Phoebe (type Net::Server::Fork) starting! pid(27704)
2020/08/29-01:00:15 Cannot connect to SSL port 1965 on [Address already in use]
2020/08/29-01:05:21 App::Phoebe (type Net::Server::Fork) starting! pid(15897)
2020/08/29-01:05:21 Cannot connect to SSL port 1965 on [Address already in use]

When I returned to the server this morning, that was the state it was in, and when I tried to restart it, same problem. There was a process still running, but the PID file was gone, and so Monit couldn’t stop the process, but the process was also not serving the port it was using.

ps aux | grep phoebe

So what’s the best solution, here? I’m thinking of a variant of “killall”, perhaps? The example below uses “[p]hoebe” to avoid the grep command from listing itself.

ps aux | grep '[p]hoebe' | awk '{print $2}' | xargs kill

What do you think?

Add Comment

2020-08-05 Abusing Soweli Lukin

A long time ago, I wrote Soweli Lukin, a web proxy for Gopher and Gemini. That allowed me to do some munging of the Gopher texts and render it as HTML, which meant reflowing long lines. That’s the main reason Gopher clients don’t work for me on mobile phones.

Now, if somebody wants to crawl all of Gopher space, they can crawl Gopher… or they can use a web proxy. That’d be pretty aggravating, of course, but not inconceivable. There are, after all, lots of people not wearing masks and endangering others, I bet they’re also willing to write such crawlers.

Now look at that, for the logfiles of today and the day before:

root@sibirocobombus:~# grep soweli-lukin /var/log/apache2/access.log /var/log/apache2/access.log.1 | /home/alex/bin/leech-detector
                  ip       hits bandw. hits% interv. status code distrib.      14956     5K  91%   -0.0s  200 (99%), 500 (0%), 502 (0%)
       XX.XX.XXX.XXX        629     2K   3%   -5.3s  200 (97%), 502 (1%), 500 (0%)
      XXX.XXX.XX.XXX        433     4K   2%   -1.4s  200 (99%), 502 (0%)
         XX.XX.X.XXX        221     7K   1%   92.9s  502 (68%), 200 (31%)
…XXXX:XXXX:XXXX:XXXX         16     5K   0%   25.2s  200 (56%), 500 (25%), 304 (18%)
…XX:XXX:XXX:XX:XXX:X         12     4K   0% 1756.0s  200 (100%)

Fifteen thousand hits.

Reminds me of who was slamming Gemini space a few days ago.

Well… I used ipset to drop their packets at the firewall, for now. Let’s see whether they change it, trying to make a comeback.

While I was at it, I also checked the IP numbers with requests in the hundreds. One of them was my monitor that checks whether the site is up, the other two were spiders with such promising names as “” and “SEOkicks”. Ugh. Think of all the CO₂ wasted because of this SEO-💩.

Comments on 2020-08-05 Abusing Soweli Lukin

I took Soweli Lukin offline because some search spider was going crazy in it. The idiots running PetalBot did not deem it worthy to honour the “disallow” statement for Soweli Lukin.

– Alex 2021-03-14 19:02 UTC

Add Comment

2020-07-28 What to do if I die?

I’m trying to write a document for my wife describing all the IT stuff I run so that she knows what to do or who to contact in case I can’t do it any more. My problem is that I don’t have any friends interested in these things. Nobody knows Perl around here. The two people that do live in Estonia and Canada and they don’t share my hobbies.

What is going to happen to a site like Sure, my wife can keep them up for a while. But she’s not going to turn into a system admin? All these wikis and apps. Somebody should be running this stuff. How does one find people willing to take over? Damn. I should organise better.

How does one archive these things for the future? The problem is that people can export their data but most of them will not, I’m sure. Upload to the Web Archive? Upload to Git Hub? Both? Other stuff? Are there other archives interested in this kind of thing? I mean, in a way I feel it is part of our collective history and how is that going to work, aren’t the national archives interested in keeping copies of all these things? They usually want copies of all the printed books… but websites? Apps? I think this necessarily must change – unless the apocalypse catches up, first.

Makes me think about resilience to capitalism and the apocalypse, both. I recently read the following by @yaaps, in an unrelated thread:

You’re an artisan. Occupying niches too small for economies of scale is how this precapitalist pattern survives and how it spawns new businesses under capitalism. It relies on a certain amount of privilege in that you need free time, skills, and personal/professional connections to succeed. in addition, the artisan is vulnerable to their own success. Replacing artisans with commodity labor is how capitalism grows

To scale up while maintaining autonomy, you need to establish a worker’s collective. That provides a model to distribute income for skilled labor, as needed, without selling equity.

Yeah. I reminded me of the fact that in order to get anything done, we need to organize. I should create an association according to Swiss law to run my sites and find people interested in keeping this stuff up. And then these people can continue after I’m gone? Let me know if you’re interested and we can write some bylaws together! It’s not hard, there are templates.


Comments on 2020-07-28 What to do if I die?

An interesting idea: some sort of Sysop Buddy System.

– Alex Schroeder

That’s a tricky one. So far, things I’ve done include making sure my sites are easy to crawl and/or re-host (mainly by making them static for the most part), and making sure trusted friends have admin rights on social media groups I manage. Not sure what to do about my domain names, since I don’t have anyone who’s both trusted enough and technical enough. Your solution is interesting.

Felix 2020-07-29 15:32 UTC

Reply by Cos.

I’ve also been part of an association offering hosting … many of our members have much less interest in helping to keep it going than they once did. A few faithful plug away, but each year things feel a little more grim.

Yes, that’s what I fear. In the past, I used Eggplant Farms. Super nice one-person operation. They used to be more than one but then it all shrank down to just one, sadly. We’d have to prepare for that, too.

I think bank accounts and passwords are ok. Gotta check those instructions again, though. And perhaps find a better way to access my passwords other than using Emacs...

– Alex Schroeder 2020-07-30 05:56 UTC

I can gladly take over your websites and manage them in the way how you wish it. You would need only to pay your domains and hosting for the number of years you wish and need. Since 20 years I am managing servers.

– Anonymous 2020-10-18 16:49 UTC

Document your art. Archive your art. and Document Your Art: How to Archive, by @ajroach42.

– Alex 2021-02-19 18:13 UTC

Add Comment

2020-06-10 Radicale for Calendar and Contacts

So, in an attempt to further degoogle my life, I installed Radicale. It’s a small but powerful CalDAV (calendars, to-do lists) and CardDAV (contacts) server written in Python. It comes with Debian.

Sadly, things aren’t as smooth as I would want them.

The first problem was that authentication via Apache didn’t work. It was weird. When I didn’t have the password file, Apache complained. When I had it, with a user and a password, Apache would let me in but I’d get to the Radicale login box, and there no login worked.

I guess what happened is that somehow Radicale either did not receive the username from Apache, or looked it up and didn’t find it, so I don’t know.

I decided to use Radicale directly, based on the page on the Debian Wiki. That seemed to work.

The second problem is that I could add the CalDAV and CardDAV accounts to my iOS device, but when I used the Contacts or Calendar app, these new accounts still did not show up. All I got is my work account (Exchange) and Gmail. None of that self-hosted stuff.

OK, next morning, checking the debug log, I see some activity from my iOS devices – and Calendar integration seems to work! When I set it up, I used and that allowed me to set it up but not to use it. Then I tried and that seems to be working, with alex being my username, or my collection name, or both. Oh dear.

The documentation has the following to say:

Many clients do not support the creation of new calendars and address books. You can use Radicale’s web interface (e.g. http://localhost:5232) to create and manage address books and calendars.

In some clients you can just enter the URL of the Radicale server (e.g. http://localhost:5232) and your user name. In others, you have to enter the URL of the collection directly (e.g. http://localhost:5232/user/calendar10min).

So now I can choose my new calendar in the Calendar app.

It still doesn’t seem to work for Contacts, though. But to be honest, I don’t think I need those as much. I’ve had a text file with ten addresses for a year or two…

I’ll keep looking for the correct setup info.


Comments on 2020-06-10 Radicale for Calendar and Contacts

I’ve been using Radicale instead of iCould on my iDevices as well - I can confirm that the collection URL setup worked for both contacts and calendars.

Creating repeating events on Radicale calendar is one of the things that I have varying degrees of success depending on the client I use: it works fine on iOS, but rather finicky in evolution seemingly due to part of CalDav spec being unimplemented.

shimmy1996 2020-06-10 12:24 UTC

Ah! There’s hope! Thank you. 🙂

– Alex Schroeder 2020-06-10 16:24 UTC

After what feels like endless re-configuring, trying to get the right URL, the right account, waiting before trying to save, setting the port again, setting https again, and on and on... it finally worked! 👍

– Alex Schroeder 2020-06-10 16:40 UTC

When I added a second user to radicale, I was unable to log in.

htpasswd /etc/radicale/users somebody
New password: 
Re-type new password: 
Adding password for user somebody

But tail /var/log/radicale/radicale.log says:

2020-07-03 11:48:21,963 - [7fe996f01700] ERROR: An exception occurred during PROPFIND request on '/': Invalid htpasswd file '/etc/radicale/users': not a valid bcrypt hash

Ah! grep encrypt /etc/radicale/config says:

# Htpasswd encryption method
htpasswd_encryption = bcrypt

So now I wonder what htpasswd uses... Here’s what htpasswd --help 2>&1 | grep encrypt says:

 -m  Force MD5 encryption of the password (default).
 -B  Force bcrypt encryption of the password (very secure).
 -d  Force CRYPT encryption of the password (8 chars max, insecure).
 -s  Force SHA encryption of the password (insecure).
 -p  Do not encrypt the password (plaintext, insecure).

I guess I forgot the -B option! Going to retry with htpasswd -B /etc/radicale/users somebody. Now it works! 😀

– Alex Schroeder 2020-07-03 09:55 UTC

Add Comment

2020-06-04 DMARC an Mailing Lists

A while ago I moved my main mail account to my own domain, using They have an excellent setup guide, and a diagnostics page. There’s MX delegation, DKIM public keys, SPF policy, DMARC policy, and so on.

Recently, I subscribed to a mailing list using Mailman. Today I got a friendly email from a mailing list member telling me about a setup problem: the DMARC policy for my domain was causing them problems, as they were getting messages like the following:

Your membership in the mailing list … has been disabled due to excessive bounces

Oops! Apparently my config causes servers to send bounces when my emails get sent via the mailing list, and those bounces are triggering Mailman’s bounce protection for other members. Not good!

But relaxing those rules is also not good. Some searching on the internet found this article: Actually, DMARC works fine with mailing lists. I don’t know what to think about it. The explanation seems to be that mailing lists that are configured to change the subject line of my emails are invalidating the signatures, which causes the conforming mail servers to bounce my mails. Just telling them to no longer reject my emails but to deliver them into Junk folders would have fixed half the problem (the bounces the mailing list members were getting) but it wouldn’t have fixed the problem that my mails look like spoofed emails to many people.

Even the mailing list archive says so:

  • SPOOFED: Re: authorship tag Alex Schroeder

This is not good, and I’m way out of my league. I decided to unsubscribe from the mailing list. Once again, the propensity of every human activity on the Internet to be spammed to death makes every open tool super complicated and the technology required to master harder to understand with every passing year.

I think I’m going to do fine without the mailing list.


Comments on 2020-06-04 DMARC an Mailing Lists

Odd, because your previous messages came through to the mailing list without incident, and looking over the headers of all your messages, I don’t see why that last one would get the SPOOFED bit, because it’s no different from the others one (other than the subject line and dates).

Sean Conner 2020-06-05 07:02 UTC

Well, sadly I have no idea – and I don’t think I’ll be investigating. If there is an easy fix out there, I’ll make it. Perhaps I have my mail stuff misconfigured? But I’m afraid I don’t have the know-how to figure out what’s wrong.

– Alex Schroeder 2020-06-05 07:39 UTC

Add Comment

2020-05-26 Certificate problems

Today I was informed by various sources that Emacs Wiki was unreachable. Other sites as well. How strange! I use dehydrated via a cron job to keep my certificates from Let’s Encrypt up to date. I checked. They were up to date! The Apache config files point at the correct certificates. The certificates have expiry dates in July.

I wondered: is it possible that Apache caches these certificates and doesn’t notice when they update? After all, dehydrated sets up symbolic links that I refer to in my Apache config, and these always point to the current certificate files.

I reloaded the Apache config file using apachectl graceful and that seemed to work. Later today I got more reports, however and checked it on another laptop and sure enough, same problem. This time I stopped and restarted Apache completely. Once again, everything seems to be fine. Let’s see whether that was good enough.

If not, let me know. 🙂

Then again, if it doesn’t work, you won’t be able to read this... So here I am, hoping for the best.


Add Comment

2020-04-05 Jitsi Considerations

Here are some things to look out for:

  1. Does your Jitsi instance use things that enable tracking, such as Google Analytics, Google Fonts, Google STUN servers?
  2. Does your Jitsi instance have a decent privacy policy, such as minimal logging, no sharing with third parties, and so on?
  3. If you’re self-hosting, have you considered these issues?
  4. Make sure people only use the apps, or Chrome – apparently Safari doesn’t work and Firefox ruins the experience for everybody else in the meeting
  5. Understand that all the apps except for the one on the F-Droid store for Android come with trackers

This is based on the following sources:

  1. Good info by Freifunk München on how to use Jitsi, and a link to their instance
  2. Also, an app for the desktop based on Electron, so at least you don’t have to run Google Chrome
  3. Privacy Tools on Zoom vs. Jitsi, listing some of the issues with Jitsi


Comments on 2020-04-05 Jitsi Considerations

I installed the Debian package for the Freifunk Jitsi app and that worked just fine.

alex@melanobombus:~$ sudo dpkg -i Downloads/ffmuc-meet-amd64.deb 
[sudo] password for alex: 
Selecting previously unselected package jitsi-meet-electron.
(Reading database ... 498341 files and directories currently installed.)
Preparing to unpack Downloads/ffmuc-meet-amd64.deb ...
Unpacking jitsi-meet-electron (2.0.4-31) ...
dpkg: dependency problems prevent configuration of jitsi-meet-electron:
 jitsi-meet-electron depends on libappindicator3-1; however:
  Package libappindicator3-1 is not installed.

dpkg: error processing package jitsi-meet-electron (--install):
 dependency problems - leaving unconfigured
Processing triggers for gnome-menus (3.31.4-3) ...
Processing triggers for mime-support (3.62) ...
Processing triggers for desktop-file-utils (0.23-4) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Errors were encountered while processing:
alex@melanobombus:~$ apt search libappindicator3-1
Sorting... Done
Full Text Search... Done
gir1.2-appindicator3-0.1/amber 0.4.92-7 amd64
  Typelib files for libappindicator3-1

libappindicator3-1/amber 0.4.92-7 amd64
  allow applications to export a menu into the panel -- GTK3 version

alex@melanobombus:~$ sudo apt install libappindicator3-1
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libappindicator3-1 : Depends: libindicator3-7 but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
alex@melanobombus:~$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libappindicator3-1 libindicator3-7
The following NEW packages will be installed:
  libappindicator3-1 libindicator3-7
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 106 kB of archives.
After this operation, 217 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 amber/main amd64 libindicator3-7 amd64 0.5.0-4 [52.9 kB]
Get:2 amber/main amd64 libappindicator3-1 amd64 0.4.92-7 [53.5 kB]
Fetched 106 kB in 1s (187 kB/s)               
Selecting previously unselected package libindicator3-7:amd64.
(Reading database ... 498429 files and directories currently installed.)
Preparing to unpack .../libindicator3-7_0.5.0-4_amd64.deb ...
Unpacking libindicator3-7:amd64 (0.5.0-4) ...
Selecting previously unselected package libappindicator3-1:amd64.
Preparing to unpack .../libappindicator3-1_0.4.92-7_amd64.deb ...
Unpacking libappindicator3-1:amd64 (0.4.92-7) ...
Setting up libindicator3-7:amd64 (0.5.0-4) ...
Setting up libappindicator3-1:amd64 (0.4.92-7) ...
Setting up jitsi-meet-electron (2.0.4-31) ...
Processing triggers for libc-bin (2.28-10) ...

OK, there was this little problem because I didn’t know how to install a package from a file and satisfy all its dependency from the regular repos.

Also, I’m starting to feel like installing all this stuff from packages I download myself is a security nightmare. These packages are never going to get fixed! 😟

– Alex Schroeder 2020-04-05 12:14 UTC

Add Comment

2020-03-26 Jitsi and the CPU

Yesterday I ran my RPG game on the self-hosted Jitsi installation. All participants need to have the right bandwidth, of course. If you’re close to Zürich, you probably have glass fibre, so no problem. As it turns out, CPU seems to be the limiting factor. The machine I’m renting has two cores and 6GB RAM available. Here are the Munin reports:

Firewall throughput CPU usageLoad average Memory usage


Comments on 2020-03-26 Jitsi and the CPU

Other people: Jitsi Meet and the Number of Video Participants – 「… 2 virtual cores and 4GB of memory … both CPUs ran at around 70% with 10 clients and memory use was around 1.4 GBs at the beginning and 800 MB a little while later when everyone had joined and settled in … I was far away from saturating the CPU with 10 simultaneous video streams in the conference.」 – Nice! 🙂

Other software: Big Blue Button: Requirements – 「… we recommend starting with a ‘clean’ Ubuntu 16.04 64-bit server dedicated for BigBlueButton. … 4 GB of memory with swap enabled (8 GB of memory is better) … 4 CPU cores (8 is better)」 – but as noted I only have 2 GB of memory, only 2 CPU cores, it’s not a dedicated server, and it’s Debian Buster, not Ubuntu 16. Oh well!

– Alex Schroeder 2020-03-28 11:54 UTC

Had another session yesterday and when the conference ended it seemed to me that some of the other services on this machine had been killed. Out of memory? I don’t think so. I need to investigate. Perhaps the response times for requests had grown so long that Monit decided the services were down and started restarting them?

– Alex Schroeder 2020-03-29 10:06 UTC

Switching to Twitch TV...

– Alex Schroeder 2020-04-17 12:27 UTC

Add Comment



Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Just say HELLO