Administration

These are the pages for all my system administration woes. I rarely post happy things in this section. You have been warned.

2019-07-24 Alternative multips plugins for Munin

I learned about multips and multips_memory today but found that I couldn’t use them for my purposes.

The main problem is that both of them depend on the process name. Let me illustrate the problem:

# ps -eo rss,comm
...
29908 /home/alex/farm
30836 /home/alex/farm
 7596 /home/alex/farm
24156 /home/alex/farm
...

But I want to match on the command name:

# ps -eo rss,command
...
29908 /home/alex/farm/communitywiki2.pl
30836 /home/alex/farm/campaignwiki2.pl
 7596 /home/alex/farm/emacswiki2.pl
24156 /home/alex/farm/oddmuse2.pl

Furthermore, I can’t rely on awk splitting and looking at $2 because I want to match on the arguments as well. These three processes are distinct:

# ps -eo rss,command|grep gopher|grep -v grep
 4008 /home/alex/perl5/perlbrew/perls/perl-5.26.1/bin/perl /home/alex/farm/gopher-server.pl --setsid --user=alex --group=alex --host=alexschroeder.ch --port=79 --log_level=3 --log_file=/home/alex/farm/finger-server.log --pid_file=/home/alex/farm/finger-server.pid --wiki=/home/alex/farm/wiki.pl --wiki_dir=/home/alex/alexschroeder --wiki_pages=alex --wiki_pages=About --wiki_pages=Contact
26240 /home/alex/perl5/perlbrew/perls/perl-5.26.1/bin/perl /home/alex/farm/gopher-server.pl --setsid --user=alex --group=alex --host=alexschroeder.ch --port=7443 --log_level=3 --log_file=/home/alex/farm/gopher-server-ssl.log --pid_file=/home/alex/farm/gopher-server-ssl.pid --wiki=/home/alex/farm/wiki.pl --wiki_dir=/home/alex/alexschroeder --wiki_pages=SiteMap --wiki_pages=About --wiki_pages=Gopher --menu=Moku_Pona_Updates --menu_file=/home/alex/.moku-pona/updates.txt --menu=Moku_Pona_Sites --menu_file=/home/alex/.moku-pona/sites.txt --wiki_cert_file=/var/lib/dehydrated/certs/alexschroeder.ch/fullchain.pem --wiki_key_file=/var/lib/dehydrated/certs/alexschroeder.ch/privkey.pem
19776 /home/alex/perl5/perlbrew/perls/perl-5.26.1/bin/perl /home/alex/farm/gopher-server.pl --setsid --user=alex --group=alex --host=alexschroeder.ch --port=70 --log_level=3 --log_file=/home/alex/farm/gopher-server.log --pid_file=/home/alex/farm/gopher-server.pid --wiki=/home/alex/farm/wiki.pl --wiki_dir=/home/alex/alexschroeder --wiki_pages=SiteMap --wiki_pages=About --wiki_pages=Gopher --menu=Moku_Pona_Updates --menu_file=/home/alex/.moku-pona/updates.txt --menu=Moku_Pona_Sites --menu_file=/home/alex/.moku-pona/sites.txt

That’s a Gopher server listening on ports 70 (Gohper), 79 (Finger), and 7443 (TLS Gopher).

Anyway, I made some changes.

For multips I got rid of the overly restricted regular expressions:

# diff /usr/share/munin/plugins/multips /etc/munin/plugins/alex_multips
8,9c8,9
< multips - Munin plugin to monitor number of processes. Which processes
< are configured in client-conf.d
---
> alex_multips - Munin plugin to monitor number of processes. Which
> processes are configured in client-conf.d
19,25c19,20
<   [multips]
<      env.names pop3d imapd sslwrap
<      env.regex_imapd ^[0-9]* imapd:
<      env.regex_pop3d ^[0-9]* pop3d:
< 
< The regex parts are not needed if the name given in "names" can be
< used to grep with directly.
---
>   [alex_multips]
>      env.names perl6
26a22,23
> The regexp environment variables have been removed.
>      
30,31c27
< configured regular expressions.  The regular expressions are
< interpreted by "grep" (and not egrep or perl).
---
> command name.
80d75
< 		eval REGEX='"${regex_'$name'-\<'$name'\>}"'
84c79
< 		echo "$fieldname.info Processes matching this regular expression: /$REGEX/"
---
> 		echo "$fieldname.info Processes matching $name"
95d89
< 	eval REGEX='"${regex_'$name'-\<'$name'\>}"'
98c92
< 		$PGREP -f -l "$name" | grep "$REGEX" | wc -l
---
> 		$PGREP -f -l "$name" | wc -l
101c95
< 		/usr/ucb/ps auxwww | grep "$REGEX" | grep -v grep | wc -l
---
> 		/usr/ucb/ps auxwww | grep -v grep | wc -l
103c97
< 		ps auxwww | grep "$REGEX" | grep -v grep | wc -l
---
> 		ps auxwww | grep -v grep | wc -l

For multips_memory I got rid of the overly restricted regular expressions, but this time in the awk script at the end, and I use command instead of comm in the ps format.

# diff /usr/share/munin/plugins/multips_memory /etc/munin/plugins/alex_multips_memory
8c8
< multips_memory - Munin plugin to monitor memory usage of processes. Which
---
> alex_multips_memory - Munin plugin to monitor memory usage of processes. Which
15c15
<   ps -eo rss,comm
---
>   ps -eo rss,command
19c19
< You must specify what process names to monitor:
---
> You must specify what commands to monitor:
21c21
<   [multips_memory]
---
>   [alex_multips_memory]
28d27
< 
42,43c41,43
< /etc/munin/plugins or /etc/opt/munin/plugins), eg. multips_memory_rss and
< multips_memory_vsize as symlinks to multips_memory and configure them thus:
---
> /etc/munin/plugins or /etc/opt/munin/plugins), eg. multips_memory_rss
> and multips_memory_vsize as symlinks to alex_multips_memory and
> configure them thus:
58d57
< 
77,81d75
< Only the executable name is matched against (ps -eo comm)1, and it must
< be a full string match to the executable base name, not substring,
< unless you enter a name such as ".*apache" since RE meta characters in
< the names are active.
< 
95a90,91
> Switching from comm to command by Alex Schroeder <alex@gnu.org>
> 
129,130d124
< 		eval REGEX='^$name$';
< 
132c126
< 		echo "$fieldname.info For /$REGEX/"
---
> 		echo "$fieldname.info For /$name/"
140c134
< 	ps -eo $monitor,comm | gawk '
---
> 	ps -eo $monitor,command | gawk '
143c137
< $2 ~ /^'"$name"'$/ { total = total + ($1*1024); }
---
> '"/$name/"'        { total = total + ($1*1024); }

For configuration, I use the same regular expressions for both plugins:

[alex_multips*]
env.names gridmapper-server alexschroeder campaignwiki communitywiki emacswiki face food halberdsnhelmets helmut hex-describe hug linearb mark megadungeon monones names oddmuse paper small-sites software soweli-lukin tarballs text-mapper traveller trunk gopher-server.*port=79 gopher-server.*port=70 gopher-server.*port=7443 perl6

Tags:

Add Comment

2019-07-21 PureOS, Debian, and updates

I use PureOS, which is derived from Debian. Debian recently had a new release, so I was interested in learning how they handled it. Apparently not too well.

I had already added the following line to my /etc/apt/sources.list because I needed the GNU manuals:

deb http://http.us.debian.org/debian testing non-free

That’s why I wasn’t surprised when apt told me that something or other had changed from testing to something else. Whatever, I accept. For the purposes of this blog post I commented that line.

But something still isn’t right:

$ sudo apt update
Hit:1 https://repo.puri.sm/pureos green InRelease
Reading package lists... Done                               
Building dependency tree       
Reading state information... Done
All packages are up to date.
N: Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'https://repo.puri.sm/pureos green InRelease' doesn't support architecture 'i386'

I wonder what that means.

Tags:

Comments on 2019-07-21 PureOS, Debian, and updates

The problem remains unsolved. I deleted /etc/apt/sources.list and recreated it using Software & Updates.

Software & Updates

New content:

deb https://repo.pureos.net/pureos/ green main

I used to have a line saying puri.sm instead of pureos.net but it appears to make no difference:

$ sudo apt update
Hit:1 https://repo.pureos.net/pureos green InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
N: Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'https://repo.pureos.net/pureos green InRelease' doesn't support architecture 'i386'

– Alex Schroeder 2019-07-27 20:53 UTC


What architecture am I using?

$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386

Perhaps that is the problem? I think I don’t have any i386 stuff installed:

$ dpkg --get-selections | grep :i386

No output.

Let’s remove it!

$ sudo dpkg --remove-architecture i386

Problem solved!

$ sudo apt update
Hit:1 https://repo.pureos.net/pureos green InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.

– Alex Schroeder 2019-07-27 20:59 UTC


Add this to the Purism forum, just in case you find more info in the replies over there.

– Alex Schroeder 2019-07-27 21:05 UTC


For future reference, you shouldn’t have a line like

deb http://http.us.debian.org/debian testing non-free

because you might accidently upgrade to the next testing release when you don’t want to. Instead you should refer to the releases by codename, ie:

deb http://http.us.debian.org/debian buster non-free

That way, you can upgrade to the next release at your own discretion.

– matthew 2019-07-30 05:38 UTC


Thanks!

– Alex Schroeder 2019-07-30 06:02 UTC

Add Comment

2019-07-21 Not trusting a Mac

Here I am, sitting next to my wife’s unattended Mac. Suddenly the Mac’s fan is spinning up. What the hell?

I open a terminal and run top. Apparently load was up to 6, slowly going down but still around 3. How strange. I use the o key to change the sort order and use cpu as the primary key. The process using about 50% of the CPU is photoanalys. It’s shortened. I assume it’s photoanalysisid. Other people have reported something like it back in 2016.

Today, I had loaded some pictures onto the external disk. That would explain it? So I’m trying to eject the external disk, but that doesn’t work because it’s “being used by another process”. I still have top running. So now I’m force-ejecting the external drive. In the top window I see ReportCrash for a moment. What the hell is it doing?

This blog post says “it’s designed to saves the application state to aid developers in working out why the app crashed” (ReportCrash High CPU & How to Disable reportcrash in Mac OSX). OK, I guess?

But I think my main problem is I don’t trust systems that have a ton of processes starting up and doing stuff and shutting down and maybe all of that is required, and perhaps it’s “modular design”, but I also get a vague feeling of dread as the design of our machines complicates.

Tags:

Comments on 2019-07-21 Not trusting a Mac

Just so you know I dislike this complexity in all systems, here’s what happened to me today with my GNU/Linux laptop: I pulled out the SD card including the adapter from my camera, marvelled at it, pulled the micro (?) card out of the adapter, showed it to my wife, put it back, and plugged it into my laptop. I heard the typical beep-bop sound, but the drive didn’t mount. I removed the card and heard the typical bop-beep sound. Repeated it a few times, but it didn’t mount. I asked for help on Mastodon, got some good replies, with ideas ranging from lsblk to mount to fdisk. But the thing that fixed it was trying the same thing on the Mac, seeing that it failed, pulling the micro card out of the adapter and inserting it again, and then trying it all over again. And then it worked.

So, the problem was that something about the contacts in the card adapter was good enough for the laptop to recognize a card but not good enough to recognize a filesystem on it? Or something? And the suggestions revealed the abyss of layers upon layers of architecture required to make external drives and plug-and-play and USB all work. And for a short moment, I wanted it all to just go away. What have we done?

– Alex Schroeder 2019-07-21 21:22 UTC


Every problem in computing can be solved by adding another layer of indirection, except the problem of having too many layers of indirection.

Ed Davies 2019-07-22 10:49 UTC


Hahaha!

– Alex Schroeder 2019-07-22 14:10 UTC

Add Comment

2019-07-09 Web Requests

I have dozens of little scripts, bash functions and aliases...

This one, for example, shows me the distribution of HTTP codes Apache is returning. These are the codes you’re going to see in the list below:

codemessage
200OK, page was served
301redirect (permanent, this should go down)
302redirect (temporary, this might change in the future)
304not modified (no need to serve page, this is the best)
404forbidden (this is a suspected bot or spammer)
404not found (should be changed into a redirect)
408timeout (this is a problem)
502bad gateway (this is a problem)
root@sibirocobombus:~# time-grouping 
   09/Jul/2019:12:40        524     5%  403 (41%), 200 (40%), 301 (7%), 404 (4%), 304 (3%), 408 (1%), 302 (0%), 400 (0%)
   09/Jul/2019:12:30        677     6%  200 (69%), 301 (14%), 408 (4%), 404 (4%), 304 (3%), 403 (2%), 302 (0%)
   09/Jul/2019:12:20        628     6%  200 (69%), 301 (10%), 304 (5%), 408 (4%), 302 (3%), 404 (3%), 403 (2%)
   09/Jul/2019:12:10        860     8%  200 (57%), 403 (17%), 301 (13%), 304 (3%), 408 (3%), 404 (3%), 400 (0%), 302 (0%)
   09/Jul/2019:12:00        936     9%  403 (42%), 200 (40%), 301 (8%), 408 (3%), 304 (2%), 404 (1%), 302 (0%)
   09/Jul/2019:11:50        578     5%  200 (69%), 301 (12%), 304 (7%), 408 (3%), 403 (2%), 302 (1%), 404 (1%)
   09/Jul/2019:11:40        705     7%  200 (65%), 301 (10%), 503 (9%), 304 (4%), 408 (3%), 404 (2%), 403 (2%), 502 (1%), 302 (0%)
   09/Jul/2019:11:30        719     7%  200 (72%), 301 (9%), 304 (6%), 408 (5%), 404 (3%), 403 (1%), 302 (0%)
   09/Jul/2019:11:20        740     7%  200 (71%), 301 (10%), 304 (6%), 403 (4%), 408 (4%), 404 (1%), 302 (0%)
   09/Jul/2019:11:10        673     6%  200 (68%), 301 (13%), 408 (6%), 404 (4%), 304 (3%), 403 (2%), 302 (0%)
   09/Jul/2019:11:00        694     6%  200 (70%), 301 (12%), 304 (6%), 408 (4%), 404 (3%), 403 (1%), 302 (0%), 502 (0%)
   09/Jul/2019:10:50        580     5%  200 (72%), 301 (14%), 304 (5%), 404 (3%), 408 (2%), 403 (1%), 302 (0%), 400 (0%)
   09/Jul/2019:10:40        707     7%  200 (76%), 301 (9%), 304 (4%), 408 (3%), 404 (3%), 403 (1%), 302 (1%)
   09/Jul/2019:10:30        805     8%  200 (71%), 301 (14%), 408 (4%), 403 (3%), 404 (2%), 304 (2%), 302 (0%)
   09/Jul/2019:10:20        174     1%  200 (70%), 301 (12%), 304 (6%), 408 (6%), 404 (1%), 403 (1%), 302 (0%)

The time-grouping is an alias for:

tail -n 10000 /var/log/apache2/access.log | /home/alex/bin/time-grouping 10

And /home/alex/bin/time-grouping is a Perl script to parse access.log:

#!/usr/bin/env perl
use Modern::Perl;
use Time::ParseDate;

die "Argument '10min' to use smaller bucket size\n" if grep { $_ eq '--help' } @ARGV;

my $bucket = '1h';
$bucket = '10min' if grep { $_ eq '10' } @ARGV;

my $ESC = "\x1b";

sub red {
  my $text = shift;
  return '' unless $text;
  return "$ESC\[31m$text$ESC\[0m";
}

sub bold_red {
  my $text = shift;
  return '' unless $text;
  return "$ESC\[1;31m$text$ESC\[0m";
}

sub green {
  my $text = shift;
  return '' unless $text;
  return "$ESC\[32m$text$ESC\[0m";
}

sub yellow {
  my $text = shift;
  return '' unless $text;
  return "$ESC\[33m$text$ESC\[0m";
}

sub color_code {
  my $code = shift;
  return '' unless $code;
  return green($code) if substr($code,0,1) eq '2' or $code eq '304'; # or not modified
  return red($code) if substr($code,0,1) eq '5' or $code eq '408'; # server errors or timeout
  # return yellow($code) if substr($code,0,1) eq '3'; # redirects
  return yellow($code) if substr($code,0,1) eq '4'; # error messages
  return $code;
}

my %latest;
my %status;
my %count;
my $total;
while (<STDIN>) {
  m/^(?:(\S+):\S+ )?(\S+) \S+ \S+ \[(.*?)\] "(.*?)" (\d+)/ or die "Cannot parse:\n$_";
  my $host = $1;
  my $ip = $2;
  my $ts = $3;
  my $url = $4;
  my $code = $5;
  my $time = parsedate($ts);
  my $label;
  $label = substr($ts,0,14) if $bucket eq '1h';
  $label = substr($ts,0,16) . '0' if $bucket eq '10min';
  $total++;
  $latest{$label}=$time;
  $status{$label} = () unless exists $status{$label};
  $status{$label}{$code}++;
  $count{$label}++;
}
my @result = sort {$latest{$b} <=> $latest{$a}} keys %count;
foreach my $label (@result) {
  printf "%20s %10d   %3d%%  %s\n", $label, $count{$label}, 100* $count{$label} / $total,
      join(', ', map { sprintf("%3s (%d%%)",
			       color_code($_),
			       100 * $status{$label}{$_} / $count{$label})
	   } sort { $status{$label}{$b} <=> $status{$label}{$a} } keys %{$status{$label}});
}

Tags:

Add Comment

2019-07-04 Lost contact with the machine

A few minutes ago I noticed that I couldn’t reach my website. I tried to connect via SSH but got a timeout. I tried to start a VNC server via the admin console but got an error. I figured it must have gone totally bonkers and rebooted it, after more than 450 days of uptime. Now that it’s back up, I’m checking stuff but it looks like that wasn’t the problem. Time to look at the log files…

CPU looks OK Load looks OK

Time to figure out when this happened! Let’s zoom into the very end, when Apache reports no more hits:

Apache logs showing a drop to zero after 9:00

Let’s look at the time between 9:00 and 9:10 in the logfile!

Jul  4 09:00:01 sibirocobombus CRON[13112]: (alex) CMD (/home/alex/bin/trunk-bot)
Jul  4 09:00:01 sibirocobombus CRON[13113]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jul  4 09:00:01 sibirocobombus CRON[13114]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/
null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Jul  4 09:00:01 sibirocobombus CRON[13115]: (root) CMD (   test -x /usr/sbin/tigercron && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; nice -n$NICETIGER /
usr/sbin/tigercron -q ; })
Jul  4 09:02:01 sibirocobombus CRON[13592]: (logcheck) CMD (   if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi)
Jul  4 09:05:01 sibirocobombus CRON[13627]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jul  4 09:05:01 sibirocobombus CRON[13628]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/
null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Jul  4 09:06:01 sibirocobombus CRON[13938]: (alex) CMD (/home/alex/bin/trunk-bot)
Jul  4 09:07:01 sibirocobombus CRON[13940]: (alex) CMD (/home/alex/bin/planet-indie)
Jul  4 09:10:01 sibirocobombus CRON[13961]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jul  4 09:10:01 sibirocobombus CRON[13962]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)

This is extremely and reassuringly boring. Nothing at all seems to have happened. Happy cron jobs!

I rebooted about ten minutes later:

Jul  4 09:18:49 sibirocobombus kernel: [    0.000000] ...

@fitheach asked about that spike around midnight. I remember seeing those on a regular basis and never bothered to look into it. Let’s check now!

Jul  3 23:55:01 sibirocobombus CRON[24281]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/
null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Jul  3 23:55:01 sibirocobombus CRON[24282]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jul  3 23:57:25 sibirocobombus systemd[1]: Created slice User Slice of git.
Jul  3 23:57:25 sibirocobombus systemd[1]: Starting User Manager for UID 1001...
Jul  3 23:57:25 sibirocobombus systemd[1]: Started Session 394503 of user git.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Reached target Timers.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Reached target Paths.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Listening on GnuPG cryptographic agent (access for web browsers).
Jul  3 23:57:25 sibirocobombus systemd[24851]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Jul  3 23:57:25 sibirocobombus systemd[24851]: Listening on GnuPG cryptographic agent and passphrase cache.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jul  3 23:57:25 sibirocobombus systemd[24851]: Listening on GnuPG network certificate management daemon.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Reached target Sockets.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Reached target Basic System.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Reached target Default.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Startup finished in 50ms.
Jul  3 23:57:25 sibirocobombus systemd[1]: Started User Manager for UID 1001.
Jul  3 23:57:25 sibirocobombus systemd[1]: Stopping User Manager for UID 1001...
Jul  3 23:57:25 sibirocobombus systemd[24851]: Stopped target Default.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Stopped target Basic System.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Stopped target Timers.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Stopped target Paths.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Stopped target Sockets.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Closed GnuPG cryptographic agent and passphrase cache.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Jul  3 23:57:25 sibirocobombus systemd[24851]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Jul  3 23:57:25 sibirocobombus systemd[24851]: Closed GnuPG cryptographic agent (access for web browsers).
Jul  3 23:57:25 sibirocobombus systemd[24851]: Closed GnuPG network certificate management daemon.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Reached target Shutdown.
Jul  3 23:57:25 sibirocobombus systemd[24851]: Starting Exit the Session...
Jul  3 23:57:25 sibirocobombus systemd[24851]: Received SIGRTMIN+24 from PID 24867 (kill).
Jul  3 23:57:25 sibirocobombus systemd[1]: Stopped User Manager for UID 1001.
Jul  3 23:57:25 sibirocobombus systemd[1]: Removed slice User Slice of git.
Jul  3 23:57:51 sibirocobombus systemd[1]: Started Session 394505 of user alex.
Jul  3 23:57:59 sibirocobombus systemd[1]: Created slice User Slice of git.
Jul  3 23:57:59 sibirocobombus systemd[1]: Starting User Manager for UID 1001...
Jul  3 23:57:59 sibirocobombus systemd[24942]: Listening on GnuPG cryptographic agent and passphrase cache.
Jul  3 23:57:59 sibirocobombus systemd[24942]: Listening on GnuPG network certificate management daemon.
Jul  3 23:57:59 sibirocobombus systemd[24942]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Jul  3 23:57:59 sibirocobombus systemd[24942]: Reached target Timers.
Jul  3 23:57:59 sibirocobombus systemd[24942]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jul  3 23:57:59 sibirocobombus systemd[24942]: Listening on GnuPG cryptographic agent (access for web browsers).
Jul  3 23:57:59 sibirocobombus systemd[24942]: Reached target Sockets.
Jul  3 23:57:59 sibirocobombus systemd[24942]: Reached target Paths.
Jul  3 23:57:59 sibirocobombus systemd[24942]: Reached target Basic System.
Jul  3 23:57:59 sibirocobombus systemd[24942]: Reached target Default.
Jul  3 23:57:59 sibirocobombus systemd[24942]: Startup finished in 25ms.
Jul  3 23:57:59 sibirocobombus systemd[1]: Started User Manager for UID 1001.
Jul  3 23:57:59 sibirocobombus systemd[1]: Started Session 394506 of user git.
Jul  3 23:58:00 sibirocobombus systemd[1]: Stopping User Manager for UID 1001...
Jul  3 23:58:00 sibirocobombus systemd[24942]: Stopped target Default.
Jul  3 23:58:00 sibirocobombus systemd[24942]: Stopped target Basic System.
Jul  3 23:58:00 sibirocobombus systemd[24942]: Stopped target Timers.
Jul  3 23:58:00 sibirocobombus systemd[24942]: Stopped target Paths.
Jul  3 23:58:00 sibirocobombus systemd[24942]: Stopped target Sockets.
Jul  3 23:58:00 sibirocobombus systemd[24942]: Closed GnuPG cryptographic agent and passphrase cache.
Jul  3 23:58:00 sibirocobombus systemd[24942]: Closed GnuPG cryptographic agent (access for web browsers).
Jul  3 23:58:00 sibirocobombus systemd[24942]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Jul  3 23:58:00 sibirocobombus systemd[24942]: Closed GnuPG network certificate management daemon.
Jul  3 23:58:00 sibirocobombus systemd[24942]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Jul  3 23:58:00 sibirocobombus systemd[24942]: Reached target Shutdown.
Jul  3 23:58:00 sibirocobombus systemd[24942]: Starting Exit the Session...
Jul  3 23:58:00 sibirocobombus systemd[24942]: Received SIGRTMIN+24 from PID 24957 (kill).
Jul  3 23:58:00 sibirocobombus systemd[1]: Stopped User Manager for UID 1001.
Jul  3 23:58:00 sibirocobombus systemd[1]: Removed slice User Slice of git.
Jul  4 00:00:01 sibirocobombus CRON[25130]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Jul  4 00:00:01 sibirocobombus CRON[25131]: (alex) CMD (/home/alex/bin/trunk-bot)
Jul  4 00:00:01 sibirocobombus CRON[25133]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jul  4 00:00:01 sibirocobombus CRON[25132]: (root) CMD (   test -x /usr/sbin/tigercron && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; nice -n$NICETIGER /usr/sbin/tigercron -q ; })
Jul  4 00:00:08 sibirocobombus crontab[26499]: (root) LIST (nobody)
Jul  4 00:02:01 sibirocobombus CRON[29160]: (logcheck) CMD (   if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi)
Jul  4 00:05:01 sibirocobombus CRON[29266]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Jul  4 00:05:01 sibirocobombus CRON[29267]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jul  4 00:06:01 sibirocobombus CRON[29598]: (alex) CMD (/home/alex/bin/trunk-bot)
Jul  4 00:07:01 sibirocobombus CRON[29644]: (alex) CMD (/home/alex/bin/planet-osr)

OK, so shortly before midnight, there’s some systemd activity, but nothing stands out, I think?

For the moment, there doesn’t seem to be anything I can learn from the experience.

  1. Apache was unreachable
  2. SSH daemon was unreachable

I didn’t check ping, nor the Gopher services. Perhaps I should have, before rebooting.

Log files are looking good. Monitoring seems to have worked just fine.

I have no idea what happened.

As you can see in the logs above I have some bots running via cron jobs. Both of them mailed me error messages.

At 09:08 and at 09:14, when the system was unreachable, I got the error “socket.gaierror: [Errno -3] Temporary failure in name resolution.”

At 10:00, after the reboot, I got a message from Tiger:

# Checking listening processes 
OLD: --WARN-- [lin003w] The process `systemd-resolve' is listening on socket 53 (UDP on 127.0.0.53 interface) is run by systemd-resolve. 
OLD: --WARN-- [lin003w] The process `systemd-resolve' is listening on socket 5355 (TCP on every interface) is run by systemd-resolve. 
OLD: --WARN-- [lin003w] The process `systemd-resolve' is listening on socket 5355 (UDP on every interface) is run by systemd-resolve. 

So... what happened? I’m skimming journalctl -r but that just starts with the boot and journalctl --list-boots shows I have just this one to go by. Sad! I guess we’ll never know...

Tags:

Comments on 2019-07-04 Lost contact with the machine

I’m not sure we’re done, here. I just tried contacting websites and was getting proxy errors: Apache was unable to contact the Hypnotoad services!

Monit reported all of them up and running. Munin reported load average around 0.6. What the hell?

  1. I added a ton of IP numbers with suspicious activity in my access log to the black list
  2. I restarted all the Hypnotoads

Let’s see what happens.

– Alex Schroeder 2019-07-07 14:14 UTC


Same thing, today! I just had to reload all the Hypnotoads. When it’s happening, I’m getting 502 Proxy errors and cannot connect to my sites via Apache, and I can’t connect to the same sites directly via HTTP. This includes the Mojolicious::Plugin::Status page. Thus, the plugin is currently useless for debugging this.

  1. CPU% is not suspicious
  2. Load is not suspicious
  3. Apache hits are not suspicious

Let’s take a look at the Apache result codes in 10min time groupings:

tail -n 10000 /var/log/apache2/access.log | /home/alex/bin/time-grouping 10 | cut -c 1-80
   08/Jul/2019:11:30        581     5%  200 (62%), 403 (21%), 301 (6%), 408 (3%)
   08/Jul/2019:11:20        885     8%  200 (67%), 301 (16%), 304 (6%), 404 (5%)
   08/Jul/2019:11:10       1265    12%  200 (36%), 502 (29%), 403 (19%), 301 (8%
   08/Jul/2019:11:00        721     7%  200 (75%), 301 (11%), 408 (3%), 304 (3%)
   08/Jul/2019:10:50        798     7%  200 (79%), 301 (11%), 304 (3%), 404 (2%)
   08/Jul/2019:10:40        780     7%  200 (74%), 301 (15%), 404 (3%), 304 (2%)
   08/Jul/2019:10:30        701     7%  200 (72%), 301 (13%), 408 (5%), 304 (3%)
   08/Jul/2019:10:20        757     7%  200 (72%), 301 (16%), 304 (4%), 408 (3%)
   08/Jul/2019:10:10        622     6%  200 (68%), 301 (17%), 408 (4%), 404 (4%)
   08/Jul/2019:10:00        876     8%  200 (75%), 301 (9%), 403 (5%), 408 (4%),
   08/Jul/2019:09:50        701     7%  200 (76%), 301 (11%), 404 (3%), 304 (3%)
   08/Jul/2019:09:40        790     7%  200 (76%), 301 (10%), 408 (4%), 304 (4%)
   08/Jul/2019:09:30        523     5%  200 (78%), 301 (10%), 408 (4%), 404 (3%)

Clearly, at 11:10 the requests nearly double and about a third (29%) result in 502 (proxy error) codes. But there are also 19% resulting in 403 (forbidden). The typical 200 (ok) results drop significantly.

New approach!

I’m going to use the rules listed in Stop fake and bad user agents with htaccess. Let’s hope this works! 🙂

– Alex Schroeder 2019-07-08 10:28 UTC


And it just happened again:

time-grouping | cut -c 1-80
   08/Jul/2019:13:20        122     1%  200 (60%), 403 (18%), 301 (13%), 404 (3%
   08/Jul/2019:13:10        737     7%  200 (75%), 301 (12%), 403 (4%), 404 (3%)
→  08/Jul/2019:13:00        615     6%  502 (44%), 200 (29%), 301 (16%), 403 (3%
→  08/Jul/2019:12:50        642     6%  200 (37%), 502 (34%), 301 (15%), 403 (5%
   08/Jul/2019:12:40        719     7%  200 (68%), 301 (14%), 403 (5%), 408 (4%)
   08/Jul/2019:12:30        774     7%  200 (70%), 301 (13%), 403 (6%), 408 (3%)
   08/Jul/2019:12:20        665     6%  200 (70%), 301 (12%), 304 (7%), 404 (3%)
   08/Jul/2019:12:10        763     7%  200 (73%), 301 (12%), 304 (6%), 404 (3%)
   08/Jul/2019:12:00        848     8%  200 (76%), 301 (12%), 304 (4%), 404 (2%)
   08/Jul/2019:11:50        887     8%  200 (75%), 301 (6%), 304 (6%), 404 (3%),
   08/Jul/2019:11:40        862     8%  200 (78%), 301 (8%), 304 (6%), 408 (3%),
   08/Jul/2019:11:30        809     8%  200 (66%), 403 (15%), 301 (7%), 304 (4%)
   08/Jul/2019:11:20        885     8%  200 (67%), 301 (16%), 304 (6%), 404 (5%)
   08/Jul/2019:11:10        672     6%  200 (41%), 403 (22%), 502 (21%), 301 (8%

I’m out of ideas... I’ll try changing all the Hypnotoad configurations and increase the number of workers per application from 1 to 2.

– Alex Schroeder 2019-07-08 11:20 UTC

Add Comment

2019-06-26 Privacy vs. fail2ban

I’m using fail2ban to watch over my sites. Or at least I thought I did!

CPU went up Load went up

Apparently, fail2ban didn’t notice a thing.

fail2ban is not busy

But zoom out and you’ll notice that the rules for Apache and Gopher aren’t seeing any matches.

fail2ban used to be busy

What happened? I don’t know. But I suspect this is due to me removing IP numbers from the web server logs. This meant that the fail2ban rules no longer matched. Not once. Somebody started unleashing another bot and here we are, once again fighting the leeches.

Well, I enabled the logging of IP numbers again and I see fail2ban has started banning some IP numbers again. But unfortunately, load remains an issue.

So now I’ve started the process of tearing the farm apart.

Toadfarm is a single Mojolicious application which mounted all of the 27 Mojolicious applications I had as one “farm”. This resulted in two problems:

  1. the processes were all named farm so I couldn’t tell them apart
  2. Mojolicious offers a status plugin which doesn’t work with Toadfarm

Now now I’m removing the applications one by one from Toadfarm, using Hypnotoad to run them.

  1. uncomment the application in the Toadfarm script
  2. assign them a separate port in the Apache config
  3. create a separate Monit config for each application
  4. create a separate Hypnotoad wrapper script for each application

See if it works.

Current status: 4/27 done. I have a bad feeling about this.

Tags:

Comments on 2019-06-26 Privacy vs. fail2ban

Status: still fucked.

CPU at 100% for 100% of the time

– Alex Schroeder 2019-06-27 07:02 UTC


Migrated more applications from Toadfarm to Hypnotoad. Looked at the log files and banned a bot via UserAgent because their webpage said they only looked at the directives for all bots. Witnessed a dozens of requests for the same page in a short time, from different IP numbers, with different user agents, checked it, and saw that it was spammed. They had made 307 edits in four days, starting June 23. Hm. Look at the graph above... Perhaps they are the problem?

log file matches
access.log 125
access.log.1 584
access.log.2.gz 726
access.log.3.gz 870
access.log.4.gz 556

My wikis don’t react well when lots of hits arrive in bursts, sadly. That’s what fail2ban is supposed to stop. We’ll see how it goes. I’m not too hopeful because I don’t think these are actually a lot of hits.

Looking at the current log file:

hour hits rel.
27/Jun/2019:13 17 13%
27/Jun/2019:12 21 16%
27/Jun/2019:11 18 14%
27/Jun/2019:10 15 11%
27/Jun/2019:09 17 13%
27/Jun/2019:08 16 12%
27/Jun/2019:07 11 8%
27/Jun/2019:06 13 10%

The data for Apache accesses also makes me think that the number of hits isn’t the problem, here.

Hits don't grow like CPU% does

As for the conversion from Toadfarm to Hypnotoad: just four applications left to do! 🙂

– Alex Schroeder 2019-06-27 11:24 UTC


Also reducing the number of preforked worked threads to 1 for almost all apps. The only exceptions are Emacs Wiki (3) and Campaign Wiki (2), as these seem to be the most popular ones, based on domain names.

domain hits rel.
www.emacswiki.org 17831 51%
campaignwiki.org 6424 18%
alexschroeder.ch 5969 17%
communitywiki.org 2069 6%

I think that helps? That was also a benefit of the Toadfarm. You didn’t have workers sleeping for every app! Now I do. Can’t go below zero, though. 🙂

Number of processes going down

– Alex Schroeder 2019-06-27 11:37 UTC


Well, right now it doesn’t look so bad!

Load dropped below 2

– Alex Schroeder 2019-06-27 11:59 UTC


Yes, looks like I’m back from the brink.

Load is back under control

But to be honest I’m not sure what did it. Perhaps moving from Toadfarm to Hypnotoad allowed me to see that a lot of processes were serving the Food wiki even though it wasn’t getting a lot of hits? Or perhaps that was simply half the story and the other half was fail2ban not working? I’m not sure. Nothing helps against distributed denial of service attacks, basically.

I think I’d need a “shields up” mode. Something that I could trigger from a job when load reaches goes over twenty or thirty? Perhaps there’s a way to write a job that just runs once a day and starts all the Monit services that are no longer running as currently I only attempt to restart them a handful of times.

check process alexschroeder with pidfile ...
    start program = "..." as uid alex gid alex
    stop program = "..." as uid alex gid alex
    if failed host ... port ... type tcp protocol http
      and request "..." for 5 cycles then restart
    if totalmem > 500 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then timeout

I’ve just set set daemon 300 (up from 120) so now it checks services every five minutes.

If it can’t reach the site five times, i.e. 25min (up from 10m) it restarts the services and if there were already 3 restarts in 75min (up from 30min) the service goes down and I have to restart it manually after investigating.

Sadly my impression is that “timeout” doesn’t actually stop services. It’s just Monit that goes stops monitoring them. The website continues to struggle and eat up resources. I think that’s what I would need: I should use stop instead of timeout/unmonitor.

– Alex Schroeder 2019-06-27 14:47 UTC


At least fail2ban seems to be working again:

one or two IP numbers banned

– Alex Schroeder 2019-07-04 08:18 UTC


Ok, I decided to look at the /var/log/fail2ban.log file.

2019-07-04 09:37:41,759 fail2ban.actions        [459]: NOTICE  [alex-apache] Ban 64.62.202.71
2019-07-04 09:40:39,990 fail2ban.actions        [459]: NOTICE  [alex-apache] Ban 62.210.139.12
2019-07-04 09:47:42,540 fail2ban.actions        [459]: NOTICE  [alex-apache] Unban 64.62.202.71
2019-07-04 09:50:40,779 fail2ban.actions        [459]: NOTICE  [alex-apache] Unban 62.210.139.12

Hm... Let’s see who offends the most. Let’s write fail2ban-report.pl.

#!/usr/bin/env perl
use Time::ParseDate;

while (<STDIN>) {
  next if /Unban|already banned/;
  m/^([0-9: -]+).*Ban (\S+)$/ or warn "Cannot parse:\n$_" and next;
  my ($date, $ip) = ($1, $2);
  $time = parsedate($date);
  $count{$ip}++;
  $first{$ip} = $time unless $first{$ip};
  $last{$ip} = $time;
  $total++;
}
@result = sort {$count{$b} <=> $count{$a}} keys %count;
print "                                      ip      #bans bans% interval\n";
foreach my $ip (@result) {
  $avg = 0;
  if ($first{$ip} and $last{$ip} and $count{$ip} > 1) {
    $avg = ($last{$ip} - $first{$ip}) / ($count{$ip} -1) / 3600;
  }
  printf ("% 40s %10d %3d%% %7s\n",
	  $ip,
	  $count{$ip},
	  100 * $count{$ip} / $total,
	  $avg ? sprintf('%3.2fh', $avg) : '');
}

And call it:

/home/alex/bin/fail2ban-report.pl < /var/log/fail2ban.log | head
                                      ip      #bans bans% interval
                          62.210.180.146         69  10%   1.04h
                                     ***         60   8%   1.68h
                           62.210.139.12         54   7%   1.33h
                          62.210.180.164         52   7%   1.18h
                           62.210.83.206         47   6%   1.54h
                           62.210.177.44         32   4%   2.22h
                                     ***         28   4%   2.61h
                                     ***         23   3%   2.53h
                                     ***         16   2%   4.56h

This 62.210.*.* number sure shows up a lot! whois 62.210.180.146 tells me that 62.210.128.0 - 62.210.255.255 is owned by a French hosting and services provider. Oh well.

Let’s see what they do:

alexschroeder.ch:443 62.210.180.146 - - [04/Jul/2019:08:07:22 +0200] "POST /software HTTP/1.0" 403 5976 "https://alexschroeder.ch/software/Comments_on_Merge_Venus_dev_branch" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"

That’s right, they’re trying to spam my wiki! They’re already getting a 403 response, but I really hate those spammers.

I recently rebooted the server so all my blacklists are gone. But we can get them back. Let’s start with the following:

ipset create blacklist hash:ip hashsize 4096
iptables -I INPUT -m set --match-set blacklist src -j DROP
iptables -I FORWARD -m set --match-set blacklist src -j DROP
ipset add blacklist 62.210.180.146
ipset add blacklist 62.210.180.164
ipset add blacklist 62.210.177.44
ipset add blacklist 62.210.139.12
ipset add blacklist 62.210.83.206

That’s right, I added the last one as well. I checked and the 62.210.0.0 - 62.210.127.255 range belongs to the same provider.

Check the setup:

iptables -L
ipset list blacklist

– Alex Schroeder 2019-07-04 09:03 UTC


I also checked #2 in the list above. I checked the logs and noticed that they make two HEAD and a GET request for all of the new page on my site. Looks like they’re checking all the links in the RSS feed, perhaps?

grep *** /var/log/apache2/access.log | /home/alex/bin/leech-detector
 ip       hits bandw. hits% interv. status code distrib.
***         43     9K 100%  128.8s  200 (93%), 503 (6%)

43 hits doesn’t look like much for today, but they did get banned 60 times by fail2ban. Remember the data up there? Once every 2h at least! That must be the 503 result. Let’s look at these patterns.

grep *** /var/log/apache2/access.log | cut -c 50-69,77,79- | sed -e 's/Friendica.*/Friendica.../' -e 's/".*"/ /' -e 's/HTTP\/1.1 //'
04/Jul/2019:07:05:03 GET /wiki/feed/full/ Friendica...
04/Jul/2019:07:05:04 HEAD /wiki/2019-06-25_Reflowing_arbitrary_text Friendica...
04/Jul/2019:07:05:04 HEAD /wiki/2019-06-25_Reflowing_arbitrary_text Friendica...
04/Jul/2019:07:05:04 GET /wiki/2019-06-25_Reflowing_arbitrary_text Friendica...
04/Jul/2019:07:05:05 HEAD /wiki/2019-06-26_Gopher_to_HTTP_and_HTML_Proxy Friendica...
04/Jul/2019:07:05:05 HEAD /wiki/2019-06-26_Gopher_to_HTTP_and_HTML_Proxy Friendica...
04/Jul/2019:07:05:06 GET /wiki/2019-06-26_Gopher_to_HTTP_and_HTML_Proxy Friendica...
04/Jul/2019:07:05:06 HEAD /wiki/2019-06-26_Hiding_your_address Friendica...
04/Jul/2019:07:05:07 HEAD /wiki/2019-06-26_Hiding_your_address Friendica...
04/Jul/2019:07:05:07 GET /wiki/2019-06-26_Hiding_your_address Friendica...
04/Jul/2019:07:05:08 HEAD /wiki/2019-06-26_Privacy_vs._fail2ban Friendica...
04/Jul/2019:07:05:08 HEAD /wiki/2019-06-26_Privacy_vs._fail2ban Friendica...
04/Jul/2019:07:05:09 GET /wiki/2019-06-26_Privacy_vs._fail2ban Friendica...
04/Jul/2019:07:05:09 HEAD /wiki/2019-06-29_Microsoft_illustrates_why_DRM_is_shit Friendica...
04/Jul/2019:07:05:10 HEAD /wiki/2019-06-29_Microsoft_illustrates_why_DRM_is_shit Friendica...
04/Jul/2019:07:05:11 GET /wiki/2019-06-29_Microsoft_illustrates_why_DRM_is_shit Friendica...
04/Jul/2019:07:05:11 HEAD /wiki/2019-06-29_Property Friendica...
04/Jul/2019:07:05:12 HEAD /wiki/2019-06-29_Property Friendica...
04/Jul/2019:07:05:12 GET /wiki/2019-06-29_Property Friendica...
04/Jul/2019:07:05:12 HEAD /wiki/2019-07-01_Orcs Friendica...
04/Jul/2019:07:05:13 HEAD /wiki/2019-07-01_Orcs Friendica...
04/Jul/2019:08:35:03 GET /wiki/feed/full/ Friendica...
04/Jul/2019:08:35:03 HEAD /wiki/2019-06-25_Reflowing_arbitrary_text Friendica...
04/Jul/2019:08:35:04 HEAD /wiki/2019-06-25_Reflowing_arbitrary_text Friendica...
04/Jul/2019:08:35:04 GET /wiki/2019-06-25_Reflowing_arbitrary_text Friendica...
04/Jul/2019:08:35:04 HEAD /wiki/2019-06-26_Gopher_to_HTTP_and_HTML_Proxy Friendica...
04/Jul/2019:08:35:05 HEAD /wiki/2019-06-26_Gopher_to_HTTP_and_HTML_Proxy Friendica...
04/Jul/2019:08:35:05 GET /wiki/2019-06-26_Gopher_to_HTTP_and_HTML_Proxy Friendica...
04/Jul/2019:08:35:06 HEAD /wiki/2019-06-26_Hiding_your_address Friendica...
04/Jul/2019:08:35:06 HEAD /wiki/2019-06-26_Hiding_your_address Friendica...
04/Jul/2019:08:35:06 GET /wiki/2019-06-26_Hiding_your_address Friendica...
04/Jul/2019:08:35:07 HEAD /wiki/2019-06-26_Privacy_vs._fail2ban Friendica...
04/Jul/2019:08:35:07 HEAD /wiki/2019-06-26_Privacy_vs._fail2ban Friendica...
04/Jul/2019:08:35:07 GET /wiki/2019-06-26_Privacy_vs._fail2ban Friendica...
04/Jul/2019:08:35:08 HEAD /wiki/2019-06-29_Microsoft_illustrates_why_DRM_is_shit Friendica...
04/Jul/2019:08:35:08 HEAD /wiki/2019-06-29_Microsoft_illustrates_why_DRM_is_shit Friendica...
04/Jul/2019:08:35:09 GET /wiki/2019-06-29_Microsoft_illustrates_why_DRM_is_shit Friendica...
04/Jul/2019:08:35:09 HEAD /wiki/2019-06-29_Property Friendica...
04/Jul/2019:08:35:09 HEAD /wiki/2019-06-29_Property Friendica...
04/Jul/2019:08:35:10 GET /wiki/2019-06-29_Property Friendica...
04/Jul/2019:08:35:10 HEAD /wiki/2019-07-01_Orcs Friendica...
04/Jul/2019:08:35:10 HEAD /wiki/2019-07-01_Orcs Friendica...
04/Jul/2019:08:35:11 GET /wiki/2019-07-01_Orcs Friendica...

Let’s list the things I am unhappy with:

  1. two HEAD requests when one would have done
  2. it sends all the requests until it gets a 503
  3. the next time it shows up, it hasn’t slowed down

So what I think I’m going to do is simply add them to the User Agent based rules in my Apache site configuration:

RewriteEngine on
RewriteCond "%{HTTP_USER_AGENT}" "Mastodon|Pcore|MegaIndex|Friendica"
RewriteRule ".*" "-" [redirect=403,last]

There you go. Sorry, Friendica! I’m suspecting somebody has added my blog’s RSS feed to their Friendica instance, and that’s a very flattering thing, but I don’t think this is how feed aggregators ought to work.

– Alex Schroeder 2019-07-04 09:22 UTC

Add Comment

2019-06-25 Bots rule the web

I looked at my web logs again. I have a script that looks at the user agent entry in my logs and uses the following regular expression to figure out what sort of bot we’re looking at:

/([a-z0-9@+.]*bot[a-z0-9@+.]*)/i

Thus, some sort of word or email address containing the word “bot”. Given a user agent like the following, the script counts that as a hit for “serpstatbot”.

"serpstatbot/1.0 (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)"

Here’s data showing that 21% of my hits are bots (18253 / 88862). Of these, 20% are by the Google bot, 19% are by the Bing bot, 10% are by the Yandex bot, 5% are by the Apple bot, and so on. And that is considering a long robots.txt file!

Stupid bots! 😠

I’m going to add serpstatbot to my robots.txt files.

Notice the entry that just says bot. How stupid is that? One possible culprit is this one, as “bot” is the first word in the user agent matching the string “bot”. Thanks for nothing, Quant.

Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)

And now for the data:

# /home/alex/bin/bot-detector < /var/log/apache2/access.log
    ----------------------------Bandwidth-------Hits-------Actions--Delay
                     Everybody      3417M      88862
                      All Bots       767M      18253   100%     9%
    ---------------------------------------------------------------------
                     Googlebot     62588K       3818    20%    21%    15s
                       bingbot    493730K       3540    19%     6%    17s
                   serpstatbot     15398K       2965    16%     3%    20s
                     YandexBot    148204K       1949    10%     0%    30s
                      Applebot      5840K        916     5%     0%    66s
                         CCBot      6880K        769     4%    21%    78s
                           bot     14858K        755     4%     5%    79s
        +centurybot9@gmail.com      5086K        567     3%     2%   106s
                        DotBot       959K        351     1%     0%   171s
                      chimebot     14805K        266     1%     0%   228s
                       Gigabot      2246K        245     1%     0%   248s
                        Exabot      1994K        238     1%    29%   254s
                    SemrushBot       633K        196     1%     0%   304s
                      Slackbot      1203K        179     0%    95%   338s
                        robots      1643K        146     0%     0%   408s
                   ZoominfoBot      1110K        134     0%     0%   447s
                      Cliqzbot       577K        133     0%     0%   436s
                       BLEXBot       189K        131     0%     0%   435s
                    robot.html       947K        111     0%     0%   417s
                           Bot       843K        104     0%     0%   579s
                    istellabot       283K         88     0%    14%   285s
           trendictionbot0.5.0       301K         61     0%    39%   982s
                    PaperLiBot       455K         52     0%    50%   1087s
                DomainStatsBot      1199K         47     0%    14%   137s
                       MagiBot       400K         46     0%     0%   1309s
                    Twitterbot       256K         43     0%     0%   1264s
                     MojeekBot        88K         41     0%     2%   1183s
                      rogerbot       197K         40     0%     0%   1314s
                          bots       207K         40     0%     0%   1396s
                    SEMrushBot       105K         36     0%     0%   1594s
                     coccocbot       149K         27     0%     0%   1748s
                       yacybot        78K         20     0%     5%   2276s
                     RSSingBot       131K         19     0%     0%   2561s
                       MJ12bot        43K         16     0%     0%   3706s
                     BoxcarBot         6K         14     0%   100%   4479s
                        SMTBot       129K         13     0%     0%     8s
                         ICBot        40K         12     0%     0%   2256s
           bot@linkfluence.com        85K         11     0%     0%   4936s
                     Uptimebot        45K         10     0%     0%   5505s
                   SurdotlyBot        30K          8     0%     0%     0s
        YandexAccessibilityBot        63K          7     0%     0%   5470s
                  TweetmemeBot        37K          6     0%     0%   4566s
                       feedbot       639K          6     0%    33%   7430s
                  Laserlikebot        19K          5     0%     0%   9075s
                        ZumBot        72K          5     0%     0%     1s
                          oBot        15K          5     0%     0%   3495s
               Mediatoolkitbot        49K          4     0%     0%   10673s
                    startmebot        38K          4     0%     0%   1991s
                    toot.robot        16K          4     0%     0%     1s
               YandexMobileBot        46K          4     0%     0%   13973s
                     AhrefsBot         9K          4     0%     0%   4874s
                   DuckDuckBot        17K          4     0%     0%     1s
                     SabsimBot        60K          4     0%    25%   3773s
                       ZoomBot        17K          4     0%     0%     0s
                 wiederfreibot        36K          3     0%    66%   21665s
                  OutclicksBot         1K          3     0%     0%   6324s
                   TelegramBot        28K          3     0%     0%   9789s
                      bot.html        46K          3     0%     0%   15600s
                      bitlybot        11K          2     0%     0%     0s
                  botsin.space         7K          2     0%     0%   2114s
                     BublupBot         8K          2     0%     0%   23512s
                    robots.txt         9K          2     0%     0%     0s
                  AwarioRssBot        28K          2     0%   100%   3831s
                    Discordbot        50K          2     0%     0%   9774s
                     redditbot         4K          2     0%     0%     1s
                   newsbots.eu         4K          1     0%     0%     0s
                  OnalyticaBot         4K          1     0%     0%     0s
                    LivelapBot        10K          1     0%     0%     0s
                       Facebot       219K          1     0%     0%     0s

Remember, that already takes into account all the bots that don’t crawl my sites because of robots.txt.

“Actions” are those URLs that contain a query parameter called “action” as this is an indication for misbehaving bots that follow links they shouldn’t follow.

“Delay” is there to show whether the bot observes the crawl delay I specify in my robots.txt.

If you look at the source code, you’ll see that my log files no longer contain any IP numbers. That’s how I try to protect the privacy of my visitors, even against myself. 🙄

Source code:

#!/usr/bin/perl
use strict;
use Time::ParseDate;

my $all = grep /--all/, @ARGV;
my %agent;
my %action;
my %bandwidth;
my $actions;
my $hits;
my $bandwidth;
my $bot_bandwidth;
my $bot_hits;
my %first;
my %last;
while (<STDIN>) {
  # example line from my log file
  # www.emacswiki.org:443 - [25/Jun/2019:14:01:04 +0200] "GET /images/logo218x38.png HTTP/1.1" 200 3919 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/75.0.3770.90 Chrome/75.0.3770.90 Safari/537.36"

  #  parse line
  m/^(\S+:\d+) (-|admin) \[(.*?)\] "(.*?)" (\d+) (\d+|-) "(.*?)" "(.*?)"/ or warn "Cannot parse:\n$_" and next;
  my ($host, $user, $time, $request, $code, $bytes, $referrer, $agent) = ($1, $2, $3, $4, $5, $6, $7, $8, $9);

  # determine the value of $uri
  my ($method, $uri, $junk) = split(' ', $request, 3);

  warn "Cannot parse: $_\n" unless $host;
  $hits++;
  $bandwidth += $bytes;
  my $domain;
  next unless $all or ($domain) = $agent =~ /([a-z0-9@+.]*bot[a-z0-9@+.]*)/i;
  my $key = $domain;
  $key = $1 if not $key and $agent =~ /https?:\/\/([^ \/()]+)/; # prefer just the domain of the bot
  $key ||= $agent; # fallback: everything
  $agent{$key}++;
  $bandwidth{$key} += $bytes;
  $bot_hits++;
  $bot_bandwidth += $bytes;

  my $date = parsedate($time);
  $first{$key} = $date unless $first{$key};
  $last{$key} = $date;

  if ($uri =~ /action=/i) {
    $actions++;
    $action{$key}++;
  }
}
my @result = sort {$agent{$b} <=> $agent{$a}} keys %agent;

print "    ----------------------------Bandwidth-------Hits-------Actions--Delay\n";
printf "%30s %9dM %10d\n", 'Everybody',
  $bandwidth / 1024 / 1024, $hits;
printf "%30s %9dM %10d   %3d%%   %3d%%\n", 'All Bots',
  $bot_bandwidth / 1024 / 1024, $bot_hits, 100, 100 * $actions / $bot_hits;
print "    ---------------------------------------------------------------------\n";
foreach my $key (@result) {
  my $avg = "";
  if ($first{$key} and $last{$key} and $agent{$key} > 1) {
    $avg = ($last{$key} - $first{$key}) / ($agent{$key} -1);
  }
  printf "%30s %9dK %10d   %3d%%   %3d%%   %3ds\n", $key,
      $bandwidth{$key} / 1024,
      $agent{$key},
      100 * $agent{$key} / $bot_hits,
      100 * $action{$key} / $agent{$key},
      $avg;
}

Tags:

Comments on 2019-06-25 Bots rule the web

@wion pointed me to Blocking robots on your web page – the list of 1800 bad bots which is a regular expression to match user agents and block them from your .htaccess file.

One the one hand, wow! So much block! On the other hand, reading the comments it is obvious that there will be the occasional false positive in that list, and that make me wary, and weary.

– Alex Schroeder 2019-06-26 06:45 UTC

Add Comment

2019-03-31 Email Server, Again

Do I want to do it again? I ran email on my server for a while, the. I ran email on a Raspberry Pi for a while. But at the same time I am annoyed by my reliance on Gmail, and on the unwillingness of Protonmail and others to provide IMAP access to my mail.

Recently, @algernon posted How To Run Your Own Mail Server and said it was “a detailed HOWTO that isn’t completely obsolete, and is easy enough to adapt to my needs. It goes step by step, instead of setting up everything in one sitting.” Perhaps my kind of thing?

I mean, I’ve said in the past that email is the new snail mail, i.e. only used for spam and bills; and that I had more to fear from criminals that from state actors and big corporations as far as email went, but I still don’t feel at ease.

So perhaps one day I will try again!

Tags:

Comments on 2019-03-31 Email Server, Again

Day 1: Change the DNS records. This is what I added:

@ 10800 IN MX 10 mail.alexschroeder.ch
@ 10800 IN TXT "v=spf1 mx -all"
mail 10800 IN A 178.209.50.237
mail 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1

Next step: reverse DNS. I wasted precious time trying to figure out how to do that until I discovered that I had to send a mail to the hosting provider of my server, apparently. Oh well.

Slowly things are coming back to me. Kallobombus Mail, for example. Or 2015-08-03 Switching to Postfix. And Raspberry Pi als privater Email Server. Uuuugh!

– Alex Schroeder 2019-03-31 11:49 UTC


Day 2: OK, so the hosting company says that the domain name service company should do it (Gandi). I don’t see anything in the menus but it seems to me that it should involve the PTR record type from what I can read. But apparently I cannot type it into the same file I mentioned above. This appears to be an error:

237.50.209.178.in-addr.arpa. 10800 IN PTR mail.alexschroeder.ch.
1.0.0.0.7.3.2.0.0.5.0.0.9.0.2.0.8.7.1.0.4.0.a.6.8.1.4.0.2.0.a.2.ip6.arpa. IN PTR mail.alexschroeder.ch.

Where does this stuff go? Somewhere on Gandi?

Apparently something is already configured somehwere, somehow: host 178.209.50.237 gives me 237.50.209.178.in-addr.arpa domain name pointer mx8.virtnode.com. but host 2a02:418:6a04:178:209:50:237:1 gives me Host 1.0.0.0.7.3.2.0.0.5.0.0.9.0.2.0.8.7.1.0.4.0.a.6.8.1.4.0.2.0.a.2.ip6.arpa not found: 3(NXDOMAIN) – whatever that means.

And where do I do this? Some sort of new “zone” file – on the Gandi site I see no way to do it, and on my own server I don’t think I have a name server installed. Should I?

– Alex Schroeder 2019-04-04 21:28 UTC


It sounds like your hosting provider is confused - maybe they think you’re just talking about regular DNS vs reverse DNS?

– Anonymous 2019-04-05 01:07 UTC


Yeah, I got the same advice on Mastodon (@fitheach, @freakazoid, @Frotz, all of them very helpful) suggesting the same thing. I sent them another message.

– Alex Schroeder 2019-04-05 08:20 UTC

Add Comment

2019-03-17 I borked my system

I was unhappy about my Perl installation. I was getting a warning regarding XS. Some of the libraries I had installed were compiled for a different version of Perl. Sure, I had installed Perl modules using cpan and cpanm. Apparently I had received a new Perl via an upgrade of PureOS. Sadly, I didn’t know how to quickly find all the modules which included XS and reinstall and recompile them.

Here’s a start:

find $(perl -e 'print join " ", @INC') -name XS.so

But then again... Perhaps it would make more sense to use perlbrew. It’s what I use on my server. Just ignore the system Perl!

But it also involves a lot of downloading and installing of Perl modules... That’s not cool. So perhaps I should continue using the system Perl and just try and replace all the packages I had installed with system packages? Let the package manager handle it?

Sadly, I immediately ran into the missing package Mastodon::Client. And once you try to install it, it pulls in all the dependencies. Some of them I might have installed using the system package manager, but cpanm doesn’t know about that.

So back to perlbrew, right? And I can delete all the system Perl modules...

Get the list:

apt list --installed "*-perl" | grep -v automatic | sed 's/\/.*//'

And run the command:

sudo apt remove libcaptcha-recaptcha-perl libcapture-tiny-perl \
libcrypt-random-seed-perl libcrypt-rijndael-perl libdatetime-perl \
libdatetime-timezone-perl libhtml-template-perl libhttp-date-perl \
libhttp-server-simple-perl libjson-perl liblist-allutils-perl \
liblocale-gettext-perl libmce-perl libmldbm-perl libmodern-perl-perl \ libmojolicious-perl libnet-server-perl libnet-whois-parser-perl \
libpod-strip-perl librpc-xml-perl libtext-charwidth-perl libtext-iconv-perl \
 libtext-markdown-perl libtext-wrapi18n-perl libtime-parsedate-perl \
libxml-atom-perl libxml-libxml-perl

Whatever, right? Enter! Enter!

But wait... what’s this? This is all wrong!

Removing gnome-shell-extensions (3.30.1-1) ...
Removing gdm3 (3.30.2-1pureos1) ...
Removing gnome-getting-started-docs (3.30.0-1) ...
Removing gnome-user-docs (3.30.2-1) ...
Removing yelp (3.31.90-1) ...
Removing inkscape (0.92.4-2) ...
Removing libgtkspell0:amd64 (2.0.16-1.2) ...
Removing aspell-de (20161207-7) ...
Removing aspell-en (2018.04.16-0-1) ...
Removing aspell (0.60.7~20110707-6) ...
Removing blends-tasks (0.7.2) ...
Removing chrome-gnome-shell (10.1-5) ...
Removing pureos-standard (0.9.4) ...
Removing pureos-minimal (0.9.4) ...
Removing console-setup (1.188) ...
Removing console-setup-linux (1.188) ...
Removing debconf-i18n (1.5.71) ...
Removing devscripts (2.19.3) ...
Removing dhelp (0.6.25) ...
Removing docbook2x (0.8.8-17) ...
Removing enchant (1.6.0-11.1+b1) ...
Removing evince (3.30.2-3) ...
Removing gnome-todo (3.28.1-2) ...
Removing gnome-contacts (3.30.2-1) ...
Removing libfolks-eds25:amd64 (0.11.4-1+b2) ...
Removing gedit (3.30.2-2) ...
Removing gnome-sushi (3.30.0-2) ...
Removing gir1.2-evince-3.0:amd64 (3.30.2-3) ...
Removing gnome-maps (3.30.3-1) ...
Removing gir1.2-webkit2-4.0:amd64 (2.22.6-1) ...
Removing gnome-calendar (3.30.1-2) ...
Removing gnome-control-center (1:3.30.3-1) ...
Removing gnome-initial-setup (3.30.0-1pureos2) ...
Removing gnome-online-accounts (3.30.1-2) ...
Removing gnome-session (3.30.1-2pureos1) ...
Removing gnome-software (3.28.0-1pureos1) ...
Removing gnulib (20140202+stable-3.1) ...
Removing xorg (1:7.7+19) ...
Removing xserver-xorg (1:7.7+19) ...
Removing xserver-xorg-video-all (1:7.7+19) ...
Removing xserver-xorg-video-vmware (1:13.3.0-2) ...
Removing xserver-xorg-video-vesa (1:2.4.0-1) ...
Removing xserver-xorg-input-all (1:7.7+19) ...
Removing xserver-xorg-input-libinput (0.28.2-1) ...
Removing xserver-xorg-video-qxl (0.1.5-2+b1) ...
Removing xserver-xorg-video-nouveau (1:1.0.16-1) ...
Removing xserver-xorg-input-wacom (0.34.99.1-1) ...
Removing libaudio-scrobbler-perl (0.01-2.3) ...
Removing libcaptcha-recaptcha-perl (0.98+ds-1) ...
Removing perlbrew (0.86-1) ...
Removing lintian (2.9.1) ...
Removing libcapture-tiny-perl (0.48-1) ...
Removing libcrypt-random-seed-perl (0.03-1) ...
Removing libcrypt-rijndael-perl (1.13-1+b5) ...
Removing librpc-xml-perl (0.80-2) ...
Removing libdatetime-format-iso8601-perl (0.08-2) ...
Removing libdatetime-format-builder-perl (0.8100-2) ...
Removing libdatetime-format-strptime-perl (1.7600-1) ...
Removing libxml-atom-perl (0.42-2) ...
Removing libdatetime-perl:amd64 (2:1.50-1+b1) ...
Removing libdatetime-timezone-perl (1:2.23-1+2018i) ...
Removing libevview3-3:amd64 (3.30.2-3) ...
Removing libxml-xpath-perl (1.44-1) ...
Removing libnet-dbus-perl (1.1.0-5+b1) ...
Removing libxml-twig-perl (1:3.50-1) ...
Removing libxmlrpc-lite-perl (0.717-1) ...
Removing libsoap-lite-perl (1.27-1) ...
Removing libgitlab-api-v4-perl (0.16-1) ...
Removing libgoa-backend-1.0-1:amd64 (3.30.1-2) ...
Removing libgspell-1-1:amd64 (1.6.1-2) ...
Removing libhtml-form-perl (6.03-1) ...
Removing libhtml-template-perl (2.97-1) ...
Removing libhttp-daemon-perl (6.01-3) ...
Removing libhttp-server-simple-perl (0.52-1) ...
Removing libjson-perl (4.02000-1) ...
Removing liblist-allutils-perl (0.15-1) ...
^[Removing libparse-debianchangelog-perl (1.2.0-13) ...
Removing libmce-perl (1.838-1) ...
Removing libmldbm-perl (2.05-2) ...
^[Removing libmodern-perl-perl (1.20180901-1) ...
Removing libmojo-server-fastcgi-perl (0.50-1) ...
Removing libmojolicious-perl (8.12+dfsg-1) ...
Removing libnet-server-perl (2.009-1) ...
Removing libnet-whois-parser-perl (0.08-1) ...
Removing libpod-strip-perl (1.02-2) ...
Removing libtext-wrapi18n-perl (0.06-7.1) ...
Removing libtext-charwidth-perl (0.04-7.1+b1) ...
Removing libtext-markdown-perl (1.000031-2) ...
Removing libtime-parsedate-perl (2015.103-3) ...
Removing texinfo (6.5.0.dfsg.1-4+b1) ...
Removing libxml-simple-perl (2.25-1) ...
Removing libxml-libxslt-perl (1.96-1+b1) ...
Removing libxml-sax-expat-perl (0.51-1) ...
Removing libyelp0:amd64 (3.31.90-1) ...
Removing xserver-xorg-video-amdgpu (18.1.99+git20190207-1) ...
Removing xserver-xorg-video-ati (1:18.1.99+git20190207-1) ...
Removing xserver-xorg-video-fbdev (1:0.5.0-1) ...
Removing xserver-xorg-video-intel (2:2.99.917+git20180925-2) ...
Removing xserver-xorg-video-radeon (1:18.1.99+git20190207-1) ...
Removing xserver-xorg-core (2:1.20.3-1) ...
Removing keyboard-configuration (1.188) ...
Removing libxml-parser-perl (2.44-4) ...
Removing libxml-libxml-perl (2.0134+dfsg-1) ...
Removing libwww-perl (6.36-1) ...
Removing libfile-listing-perl (6.04-1) ...
Removing libhttp-cookies-perl (6.04-1) ...
Removing libhttp-negotiate-perl (6.01-1) ...
Removing libhttp-message-perl (6.18-1) ...
Removing libhttp-date-perl (6.02-1) ...
Removing tasksel-data (3.50) ...
Removing liblwp-protocol-https-perl (6.07-2) ...
Removing tasksel (3.50) ...
Removing liblocale-gettext-perl (1.07-3+b4) ...
Removing mutter (3.30.2-6) ...
Removing zenity (3.30.0-2) ...
Removing libwebkit2gtk-4.0-37:amd64 (2.22.6-1) ...
Removing libenchant1c2a:amd64 (1.6.0-11.1+b1) ...
Removing hunspell-en-us (1:2018.04.16-1) ...
Removing dictionaries-common (1.28.1) ...
Removing 'diversion of /usr/share/dict/words to /usr/share/dict/words.pre-dictionaries-common by dictionaries-common'
Removing evolution-data-server (3.30.5-1) ...
Removing libedataserverui-1.2-2:amd64 (3.30.5-1) ...
Removing libtext-iconv-perl (1.7-5+b7) ...
Removing gnome-shell (3.30.2-3) ...

Nooooooo! It took so long to abort the operation, now it’s all borked, surely.

OK, how to I recover from this?

apt install evince inkscape gnome pureos-standard perlbrew aspell-de

I hope? Wish me luck!

Fist, switch to a stable Perl using perlbrew and get the modules installed. At this point I’m just trying to finish the task I set out to do, hours ago.

Tags:

Comments on 2019-03-17 I borked my system

And I think I’m back! Restarted the system and no problem. Emacs works. Browser works. Phew! 😅

– Alex Schroeder 2019-03-17 21:53 UTC


Oops!

For what it’s worth, I prefer to package any missing libraries as .deb packages myself and install them. dh-make-perl --cpan MODULE makes this easy, usually.

(If you need much newer versions of modules, especially “low level” ones, it might not be feasible to go this way, though.)

Adam 2019-03-17 22:11 UTC


Oh, interesting! Thank you for the suggestion.

– Alex Schroeder 2019-03-18 06:10 UTC

Add Comment

2019-03-13 Debian forever?

I loved A (Partial) Defense of Debian by John Goerzen. “We recognize the value of something that just works, that is so stable that things like unattended-upgrades are safe and reliable.” I did not appreciate this for many years. But now I’m back to using Debian for my servers and that’s exactly why. The years on my laptop have taught me this.

And I still love ❤ Slackware. 🙂

The only reason I’m using PureOS on my laptop is that it came pre-installed and hasn’t failed me, yet. It’s basically a repackaging of Debian, with changes.

I do acknowledge the beauty of a web based workflow, but my aesthetics actually point me the other way, the way John Goerzen defends, based on local git, local review tools, local email clients. Perhaps these days it’s harder to set these up locally? This is something I have noticed at the office, for example: many developers don’t like to use git directly. They try to never go to the command line, preferring to use the Eclipse git plugin EGit instead. I am torn. On Windows, using git and and magit inside Emacs is slow. I only drop to the command line when I’m confused and need to make sure I know what’s going on. And we use GitLab to review code using merge requests. So yes, I understand the lure and the convenience of using an all-in-one website.

I still want to like the old school workflow using local tools and email.

Github constantly forces me to their website. I can’t very well work on bug reports, etc. without a strong Internet connection. And it’s designed to push people into using their tools and their interface, which is inferior in a lot of ways to a local interface – but then the process to pull down someone else’s set of patches involves a lot of typing and clicking, much more that would be involved from a simple git format-patch. In short, I don’t have my shortcut keys, my environment, etc. for reviewing things – the roadblocks are there to make me use theirs. – John Goerzen

I know this is true, and yet I’ve also enjoyed contributing code to projects hosted on GitHub and GitLab and other “software forges”. It’s smooth sailing if you’re all-in or all-out. Crossing over is what breaks your workflow.

Tags:

Comments on 2019-03-13 Debian forever?

Glad to find more people making the point. Github was tolerable but the long term effects are starting to show.

orbifx 2019-03-18 15:07 UTC

Add Comment

More...

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.