Administration

These are the pages for all my system administration woes. I rarely post happy things in this section. You have been warned.

2019-03-17 I borked my system

I was unhappy about my Perl installation. I was getting a warning regarding XS. Some of the libraries I had installed were compiled for a different version of Perl. Sure, I had installed Perl modules using cpan and cpanm. Apparently I had received a new Perl via an upgrade of PureOS. Sadly, I didn’t know how to quickly find all the modules which included XS and reinstall and recompile them.

Here’s a start:

find $(perl -e 'print join " ", @INC') -name XS.so

But then again... Perhaps it would make more sense to use perlbrew. It’s what I use on my server. Just ignore the system Perl!

But it also involves a lot of downloading and installing of Perl modules... That’s not cool. So perhaps I should continue using the system Perl and just try and replace all the packages I had installed with system packages? Let the package manager handle it?

Sadly, I immediately ran into the missing package Mastodon::Client. And once you try to install it, it pulls in all the dependencies. Some of them I might have installed using the system package manager, but cpanm doesn’t know about that.

So back to perlbrew, right? And I can delete all the system Perl modules...

Get the list:

apt list --installed "*-perl" | grep -v automatic | sed 's/\/.*//'

And run the command:

sudo apt remove libcaptcha-recaptcha-perl libcapture-tiny-perl \
libcrypt-random-seed-perl libcrypt-rijndael-perl libdatetime-perl \
libdatetime-timezone-perl libhtml-template-perl libhttp-date-perl \
libhttp-server-simple-perl libjson-perl liblist-allutils-perl \
liblocale-gettext-perl libmce-perl libmldbm-perl libmodern-perl-perl \ libmojolicious-perl libnet-server-perl libnet-whois-parser-perl \
libpod-strip-perl librpc-xml-perl libtext-charwidth-perl libtext-iconv-perl \
 libtext-markdown-perl libtext-wrapi18n-perl libtime-parsedate-perl \
libxml-atom-perl libxml-libxml-perl

Whatever, right? Enter! Enter!

But wait... what’s this? This is all wrong!

Removing gnome-shell-extensions (3.30.1-1) ...
Removing gdm3 (3.30.2-1pureos1) ...
Removing gnome-getting-started-docs (3.30.0-1) ...
Removing gnome-user-docs (3.30.2-1) ...
Removing yelp (3.31.90-1) ...
Removing inkscape (0.92.4-2) ...
Removing libgtkspell0:amd64 (2.0.16-1.2) ...
Removing aspell-de (20161207-7) ...
Removing aspell-en (2018.04.16-0-1) ...
Removing aspell (0.60.7~20110707-6) ...
Removing blends-tasks (0.7.2) ...
Removing chrome-gnome-shell (10.1-5) ...
Removing pureos-standard (0.9.4) ...
Removing pureos-minimal (0.9.4) ...
Removing console-setup (1.188) ...
Removing console-setup-linux (1.188) ...
Removing debconf-i18n (1.5.71) ...
Removing devscripts (2.19.3) ...
Removing dhelp (0.6.25) ...
Removing docbook2x (0.8.8-17) ...
Removing enchant (1.6.0-11.1+b1) ...
Removing evince (3.30.2-3) ...
Removing gnome-todo (3.28.1-2) ...
Removing gnome-contacts (3.30.2-1) ...
Removing libfolks-eds25:amd64 (0.11.4-1+b2) ...
Removing gedit (3.30.2-2) ...
Removing gnome-sushi (3.30.0-2) ...
Removing gir1.2-evince-3.0:amd64 (3.30.2-3) ...
Removing gnome-maps (3.30.3-1) ...
Removing gir1.2-webkit2-4.0:amd64 (2.22.6-1) ...
Removing gnome-calendar (3.30.1-2) ...
Removing gnome-control-center (1:3.30.3-1) ...
Removing gnome-initial-setup (3.30.0-1pureos2) ...
Removing gnome-online-accounts (3.30.1-2) ...
Removing gnome-session (3.30.1-2pureos1) ...
Removing gnome-software (3.28.0-1pureos1) ...
Removing gnulib (20140202+stable-3.1) ...
Removing xorg (1:7.7+19) ...
Removing xserver-xorg (1:7.7+19) ...
Removing xserver-xorg-video-all (1:7.7+19) ...
Removing xserver-xorg-video-vmware (1:13.3.0-2) ...
Removing xserver-xorg-video-vesa (1:2.4.0-1) ...
Removing xserver-xorg-input-all (1:7.7+19) ...
Removing xserver-xorg-input-libinput (0.28.2-1) ...
Removing xserver-xorg-video-qxl (0.1.5-2+b1) ...
Removing xserver-xorg-video-nouveau (1:1.0.16-1) ...
Removing xserver-xorg-input-wacom (0.34.99.1-1) ...
Removing libaudio-scrobbler-perl (0.01-2.3) ...
Removing libcaptcha-recaptcha-perl (0.98+ds-1) ...
Removing perlbrew (0.86-1) ...
Removing lintian (2.9.1) ...
Removing libcapture-tiny-perl (0.48-1) ...
Removing libcrypt-random-seed-perl (0.03-1) ...
Removing libcrypt-rijndael-perl (1.13-1+b5) ...
Removing librpc-xml-perl (0.80-2) ...
Removing libdatetime-format-iso8601-perl (0.08-2) ...
Removing libdatetime-format-builder-perl (0.8100-2) ...
Removing libdatetime-format-strptime-perl (1.7600-1) ...
Removing libxml-atom-perl (0.42-2) ...
Removing libdatetime-perl:amd64 (2:1.50-1+b1) ...
Removing libdatetime-timezone-perl (1:2.23-1+2018i) ...
Removing libevview3-3:amd64 (3.30.2-3) ...
Removing libxml-xpath-perl (1.44-1) ...
Removing libnet-dbus-perl (1.1.0-5+b1) ...
Removing libxml-twig-perl (1:3.50-1) ...
Removing libxmlrpc-lite-perl (0.717-1) ...
Removing libsoap-lite-perl (1.27-1) ...
Removing libgitlab-api-v4-perl (0.16-1) ...
Removing libgoa-backend-1.0-1:amd64 (3.30.1-2) ...
Removing libgspell-1-1:amd64 (1.6.1-2) ...
Removing libhtml-form-perl (6.03-1) ...
Removing libhtml-template-perl (2.97-1) ...
Removing libhttp-daemon-perl (6.01-3) ...
Removing libhttp-server-simple-perl (0.52-1) ...
Removing libjson-perl (4.02000-1) ...
Removing liblist-allutils-perl (0.15-1) ...
^[Removing libparse-debianchangelog-perl (1.2.0-13) ...
Removing libmce-perl (1.838-1) ...
Removing libmldbm-perl (2.05-2) ...
^[Removing libmodern-perl-perl (1.20180901-1) ...
Removing libmojo-server-fastcgi-perl (0.50-1) ...
Removing libmojolicious-perl (8.12+dfsg-1) ...
Removing libnet-server-perl (2.009-1) ...
Removing libnet-whois-parser-perl (0.08-1) ...
Removing libpod-strip-perl (1.02-2) ...
Removing libtext-wrapi18n-perl (0.06-7.1) ...
Removing libtext-charwidth-perl (0.04-7.1+b1) ...
Removing libtext-markdown-perl (1.000031-2) ...
Removing libtime-parsedate-perl (2015.103-3) ...
Removing texinfo (6.5.0.dfsg.1-4+b1) ...
Removing libxml-simple-perl (2.25-1) ...
Removing libxml-libxslt-perl (1.96-1+b1) ...
Removing libxml-sax-expat-perl (0.51-1) ...
Removing libyelp0:amd64 (3.31.90-1) ...
Removing xserver-xorg-video-amdgpu (18.1.99+git20190207-1) ...
Removing xserver-xorg-video-ati (1:18.1.99+git20190207-1) ...
Removing xserver-xorg-video-fbdev (1:0.5.0-1) ...
Removing xserver-xorg-video-intel (2:2.99.917+git20180925-2) ...
Removing xserver-xorg-video-radeon (1:18.1.99+git20190207-1) ...
Removing xserver-xorg-core (2:1.20.3-1) ...
Removing keyboard-configuration (1.188) ...
Removing libxml-parser-perl (2.44-4) ...
Removing libxml-libxml-perl (2.0134+dfsg-1) ...
Removing libwww-perl (6.36-1) ...
Removing libfile-listing-perl (6.04-1) ...
Removing libhttp-cookies-perl (6.04-1) ...
Removing libhttp-negotiate-perl (6.01-1) ...
Removing libhttp-message-perl (6.18-1) ...
Removing libhttp-date-perl (6.02-1) ...
Removing tasksel-data (3.50) ...
Removing liblwp-protocol-https-perl (6.07-2) ...
Removing tasksel (3.50) ...
Removing liblocale-gettext-perl (1.07-3+b4) ...
Removing mutter (3.30.2-6) ...
Removing zenity (3.30.0-2) ...
Removing libwebkit2gtk-4.0-37:amd64 (2.22.6-1) ...
Removing libenchant1c2a:amd64 (1.6.0-11.1+b1) ...
Removing hunspell-en-us (1:2018.04.16-1) ...
Removing dictionaries-common (1.28.1) ...
Removing 'diversion of /usr/share/dict/words to /usr/share/dict/words.pre-dictionaries-common by dictionaries-common'
Removing evolution-data-server (3.30.5-1) ...
Removing libedataserverui-1.2-2:amd64 (3.30.5-1) ...
Removing libtext-iconv-perl (1.7-5+b7) ...
Removing gnome-shell (3.30.2-3) ...

Nooooooo! It took so long to abort the operation, now it’s all borked, surely.

OK, how to I recover from this?

apt install evince inkscape gnome pureos-standard perlbrew aspell-de

I hope? Wish me luck!

Fist, switch to a stable Perl using perlbrew and get the modules installed. At this point I’m just trying to finish the task I set out to do, hours ago.

Tags:

Comments on 2019-03-17 I borked my system

And I think I’m back! Restarted the system and no problem. Emacs works. Browser works. Phew! 😅

– Alex Schroeder 2019-03-17 21:53 UTC


Oops!

For what it’s worth, I prefer to package any missing libraries as .deb packages myself and install them. dh-make-perl --cpan MODULE makes this easy, usually.

(If you need much newer versions of modules, especially “low level” ones, it might not be feasible to go this way, though.)

Adam 2019-03-17 22:11 UTC


Oh, interesting! Thank you for the suggestion.

– Alex Schroeder 2019-03-18 06:10 UTC

Add Comment

2019-03-13 Debian forever?

I loved A (Partial) Defense of Debian by John Goerzen. “We recognize the value of something that just works, that is so stable that things like unattended-upgrades are safe and reliable.” I did not appreciate this for many years. But now I’m back to using Debian for my servers and that’s exactly why. The years on my laptop have taught me this.

And I still love ❤ Slackware. 🙂

The only reason I’m using PureOS on my laptop is that it came pre-installed and hasn’t failed me, yet. It’s basically a repackaging of Debian, with changes.

I do acknowledge the beauty of a web based workflow, but my aesthetics actually point me the other way, the way John Goerzen defends, based on local git, local review tools, local email clients. Perhaps these days it’s harder to set these up locally? This is something I have noticed at the office, for example: many developers don’t like to use git directly. They try to never go to the command line, preferring to use the Eclipse git plugin EGit instead. I am torn. On Windows, using git and and magit inside Emacs is slow. I only drop to the command line when I’m confused and need to make sure I know what’s going on. And we use GitLab to review code using merge requests. So yes, I understand the lure and the convenience of using an all-in-one website.

I still want to like the old school workflow using local tools and email.

Github constantly forces me to their website. I can’t very well work on bug reports, etc. without a strong Internet connection. And it’s designed to push people into using their tools and their interface, which is inferior in a lot of ways to a local interface – but then the process to pull down someone else’s set of patches involves a lot of typing and clicking, much more that would be involved from a simple git format-patch. In short, I don’t have my shortcut keys, my environment, etc. for reviewing things – the roadblocks are there to make me use theirs. – John Goerzen

I know this is true, and yet I’ve also enjoyed contributing code to projects hosted on GitHub and GitLab and other “software forges”. It’s smooth sailing if you’re all-in or all-out. Crossing over is what breaks your workflow.

Tags:

Comments on 2019-03-13 Debian forever?

Glad to find more people making the point. Github was tolerable but the long term effects are starting to show.

orbifx 2019-03-18 15:07 UTC

Add Comment

2019-03-01 Podcast Numbers

I was wondering about the download statistics for my tiny little podcast. How would I figure this out? On my server, I keep four days of access logs. (See Privacy Policy for more information.) I posted the last episode three days ago and I verified that the last of my log files does not mention it. That means I didn’t miss any of the downloads.

So what I did is I grepped through the logs for the MP3 file, saving those lines for me to look through.

wc -l 20-halberds-and-helmets.log tells me that there are 48 lines. It’s not a lot but it’s what I’ve got.

A quick inspection shows that I have to discard HEAD requests. I should just be counting GET requests!

grep GET 20-halberds-and-helmets.log | wc -l gives me 37 lines.

Further inspection shows that quite a few of these requests have the status 206 (”partial content”) so that’s a single application downloading various parts of the file. But how to figure out which of them belong together? I’m trying to figure this out without looking at IP numbers.

Visually, it looks like I can determine what’s going on by looking at the minutes and the user agent for all the 206 results. Let’s try this. (And yes, I did have a HEAD request with a 206 result!)

grep 'GET.* 206 ' 20-halberds-and-helmets.log | perl -e '
while (<STDIN>) {
  chomp; my ($ts, $ua) =
    /\[\d\d\/\w+\/\d\d\d\d:\d\d:(\d\d).*"([^"\/]*)[^"]*"$/;
  print "$ts $ua\n";
}
'

And here’s the result, with an arrow indicating the rows I consider to be “duplicates.”

20 Mozilla
44 AppleCoreMedia
44 AppleCoreMedia ←
39 iTMS
44 AppleCoreMedia
44 AppleCoreMedia ←
44 AppleCoreMedia ←
46 Mozilla
46 Mozilla ←
46 Mozilla ←
19 Mozilla

Manually counting them, I think we could get away by saying that we need to discount three AppleCoreMedia and two Mozilla results, right?

So let’s count the hits per user agent and then we’ll correct for the partial content results above.

grep GET 20-halberds-and-helmets.log | perl -e '
my %count;
while (<STDIN>) {
  chomp; my ($ua) = /"([^"\/]*)[^"]*"$/;
  $count{$ua}++;
  $total++;
}
for my $ua (sort {$count{$b} <=> $count{$a}} keys %count) {
  print sprintf("%5d %s\n", $count{$ua}, $ua);
}
print "---- --------------------\n";
print sprintf("%5d total\n", $total);
'

This would be the result without correcting for the partial content:

   10 Mozilla
    5 AppleCoreMedia
    4 Pocket Casts
    3 Overcast
    3 PodcastAddict
    2 Dalvik
    2 okhttp
    2 Googlebot-Video
    2 stagefright
    1 AndroidDownloadManager
    1 iTMS
    1 Player FM
    1 iCatcher!
---- --------------------
   37 total

Making the correction I mentioned above:

    7 Mozilla ←
    4 Pocket Casts
    3 Overcast
    3 PodcastAddict
    2 AppleCoreMedia ←
    2 Googlebot-Video
    2 stagefright
    2 okhttp
    2 Dalvik
    1 AndroidDownloadManager
    1 iCatcher!
    1 iTMS
    1 Player FM
---- --------------------
   32 total ←

The result shows that Pocket Casts is popular. I guess it’s a podcatcher. iCatcher! is the one I use. 🙂

I’m surprised that Mozilla is up there. When I looked at the details of the user agent strings, I noticed that they mostly belong to bots:

  • two are Yandex bots
  • three ended up being Overcast (via their website?), but these are the three Mozilla results with 206 “partial content” so basically it’s just one listener from Overcast; the rest look legit

What the hell is Googlebot-Video doing, here? Is google offering audio search results somewhere? Using podcasts to train their AI overlords?

All the other user agents look like legitimate tools, frameworks, programming languages, libraries, etc.

That’s why I think that 32 people listened to the podcast episode 20, and a few probably didn’t listen to all of it.

Tags:

Comments on 2019-03-01 Podcast Numbers

I use PodBean as my catcher, but I’m not sure what engine it uses for download.

Shelby 2019-03-01 18:28 UTC

Add Comment

2019-02-03 Network Issues

I seem to have some sort of network problem. I can reach my own web sites (and post this, for example). I can reach google.ch. But I cannot reach DuckDuckGo.com and a bunch of other sites I’m trying to reach. The iPad has the same problem as the laptop, so it’s not the operating system but modem, ISP, or the Internet is down.

Tags:

Comments on 2019-02-03 Network Issues

Whatever it was, it got fixed. Thanks, kamis of the Internet!

– Alex Schroeder 2019-02-03 21:59 UTC

Add Comment

2019-01-20 fail2ban to watch over my sites

I’m hosting my sites on a tiny server. At the same time, it’s all dynamic web apps and wikis and CGI scripts that take CPU and memory resources. Not a problem, if humans are browsing the site. If people try to download the entire site using automated tools that don’t wait between requests (leeches), or don’t check the meta information included in HTTP headers and HTML tags, then they overload my sites, and they get lost in the maze of links: page history, recent changes, old page revisions, you can download for ever if you’re not careful.

Enter fail2ban. This tool watches log files for regular expressions (filters) and if it find matches, it adds IP numbers to the firewall. You can then tell it which filters to apply to which log files and how many hits you’ll allow (jail).

When writing the rules, I just need to be careful: it’s OK to download a lot of static files. I just don’t want leeches or spammers (trying to brute force the questions I sometimes ask before people get to edit their first page on my sites).

Here’s my setup:

alex-apache.conf

/etc/fail2ban/filter.d/alex-apache.conf

This is for the Apache web server with virtual hosts. The comment shows an example entry.

Notice the ignoreregex to make sure that some of the apps and directories don’t count.

Note that only newer versions of fail2ban will be able to match IPv6 hosts.

# Author: Alex Schroeder <alex@gnu.org>

[Definition]
# ANY match in the logfile counts!
# communitywiki.org:443 000.000.000.000 - - [24/Aug/2018:16:59:55 +0200] "GET /wiki/BannedHosts HTTP/1.1" 200 7180 "https://communitywiki.org/wiki/BannedHosts" "Pcore-HTTP/v0.44.0"
failregex = ^[^:]+:[0-9]+ <HOST> 

# Except cgit, css files, images...
# alexschroeder.ch:443 0:0:0:0:0:0:0:0 - - [28/Aug/2018:09:14:39 +0200] "GET /cgit/bitlbee-mastodon/objects/9b/ff0c237ace5569aa348f6b12b3c2f95e07fd0d HTTP/1.1" 200 3308 "-" "git/2.18.0"
ignoreregex = ^[^"]*"GET /(robots\.txt |favicon\.ico |[^/ ]+.(css|js) |cgit/|css/|fonts/|pics/|1pdc/|gallery/|static/|munin/|osr/|indie/|face/|traveller/|hex-describe/|text-mapper/)

alex-gopher.conf

/etc/fail2ban/filter.d/alex-gopher.conf

Yeah, I also make the wiki available via gopher...

# Author: Alex Schroeder <alex@gnu.org>

[Init]
# 2018/08/25-09:08:55 CONNECT TCP Peer: "[000.000.000.000]:56281" Local: "[000.000.000.000]:70"
datepattern = ^%%Y/%%m/%%d-%%H:%%M:%%S

[Definition]
# ANY match in the logfile counts!
failregex = CONNECT TCP Peer: "\[<HOST>\]:\d+"

alex.conf

/etc/fail2ban/jail.d/alex.conf

Now I need to tell fail2ban which log files to watch and which filters to use.

Note how I assume a human will basically click a link every 2s. Bursts are OK, but 20 hits in 40s are the limit.

Notice that the third jail just reuses the filter of the second jail.

[alex-apache]
enabled = true
port    = http,https
logpath = %(apache_access_log)s
findtime = 40
maxretry = 20

[alex-gopher]
enabled = true
port    = 70
logpath = /home/alex/farm/gopher-server.log
findtime = 40
maxretry = 20

[alex-gopher-ssl]
enabled = true
filter  = alex-gopher
port    = 7443
logpath = /home/alex/farm/gopher-server-ssl.log
findtime = 40
maxretry = 20

Tags:

Comments on 2019-01-20 fail2ban to watch over my sites

Blimey. I don’t remember the last time anyone did anything gophery I noticed.

Blue Tyson 2019-01-30 10:47 UTC


The #gopher discussion is alive and well on Mastodon... 😊

– Alex Schroeder 2019-01-30 13:11 UTC


Something to review when I have a bit of time: Web Server Security by @infosechandbook.

– Alex Schroeder 2019-02-01 18:11 UTC

Add Comment

2018-11-11 ssh config

How to set up your SSH config file: Simplify Your Life With an SSH Config File, a blog post from 2011. I’m 45 and have been using GNU/Linux since way back before people had Internet at home here in Switzerland and I’ve only learned about this today. 😮

@danyspin97 recommended autossh which keeps connections alive. “Autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.” I don’t often run into this problem, though.

@zladuric recommended multiplexing which speeds up reconnecting to a server. “OpenSSH can reuse an existing TCP connection for multiple concurrent SSH sessions. This results into reduction of the overhead of creating new TCP connections.” I never saw this as problem, though. Probably because the host I connect to most often is in the same city as I am and I have fibre to the home. But if you’re a heavy duty system administrator perhaps?

@oliof recommended sshshuttle for people that forward many ports and don’t have a virtual private network (VPN). “Transparent proxy server that works as a poor man’s VPN. Forwards over ssh. Doesn’t require admin.” I don’t need that, either. Maybe if I had more need to circumventing geoblocking. 😁

Since we’re talking about SSH, here’s the most important server-side advice I have for you:

  1. use fail2ban to temporarily ban IP numbers that cause too many failed login attempts
  2. change the default port to something else in orther to reduce the number of login attempts
  3. if you need to allow root access on your system (which I do if I want to make backups using rsync) then use the prohibit-password option

The last two are changes you make to the /etc/ssh/sshd_config file. Uncomment #Port 22 and change the port, and set the PermitRootLogin option to prohibit-password.

Tags:

Comments on 2018-11-11 ssh config

One thing I like to do is to configure a group, which a user has to be in, to be allowed to ssh in. So the default is that a user can’t.

This avoids e.g. the problem where you create a dummy user with a weak password to test something, and forget to delete the user. Not that I’ve ever done that. Noooo.

I’ve split the users allowed to ssh in up into “sshusers” and “sshkeyusers”, the first category is allowed to use a password (basically, me), and the second category can only login with a key (the handful of friends that have access). In /etc/ssh/sshd_config I have:

# Only allow sshusers/sshkeyusers to ssh in:
AllowGroups sshusers sshkeyusers
Match Group sshkeyusers
  PasswordAuthentication no

Adam 2018-11-11 15:22 UTC


Hahaha! 😎

Luckily, that server should only have me on it. But you’re right. Better be safe than sorry!

– Alex Schroeder 2018-11-11 18:32 UTC


I currently have the following:

# List of user names allowed to log in.
# We need root for our backup script.
# We need git for git-via-SSH repos.
AllowUsers alex git root

PermitRootLogin prohibit-password

So let me think this through:

  1. I want alex to connect using a password just in case.
  2. I don’t want git to connect using a password (not sure why it seems to have a password in /etc/shadow and /usr/bin/git-shell as its shell in /etc/passwd and I don’t know whether any of these obviate a setting in /etc/sshd_config) 🤔❓
  3. I don’t want root to connect using a password.

So I added the following:

Match User git
    PasswordAuthentication no

Seems to work as intended, for now.

– Alex Schroeder 2018-11-12 13:26 UTC


“The rage that one felt was an abstract, undirected emotion which could be switched from one object to another like the flame of a blowlamp.” – George Orwell, 1984, chapter 1

– Alex Schroeder 2018-11-12 13:30 UTC

Add Comment

2018-11-10 Manual Pages

I like people who write their own man pages with the most important information they keep forgetting about existing commands. Keep this info in text files and use a little script like mman to access those text files; keep using man for the original. I found this via tfurrows’s gopher site.

On Mastodon, I got some more suggestions:

@raucao suggested tldr. I installed the Perl variant via cpanm tldr.

alex@melanobombus:~$ tldr emacs

  The extensible, customizable, self-documenting, real-time display editor.

  ✓ Open emacs in console mode (without X window):

    🍣  emacs -nw

  ✓ Open a file in emacs:

    🍣  emacs {{filename}}

  ✓ Exit emacs:

    🍣  C-x C-c

Know about ed I can tell you that knowing how to quit is essential information. But surely there is more to it?

@coffe suggested cheat. I installed it via pip3 install cheat. And I set some options in my ~/bashrc:

export EDITOR=emacsclient
export CHEATCOLORS=true

You need to provide an editor because you can edit your cheat sheets. That’s the spirit!

The colourisation for the Emacs page is wrong. I guess it isn’t handled as Markdown? But at least there’s more information available. Perhaps it’s too much?

alex@melanobombus:~$ cheat emacs
# Running emacs

  GUI mode            $ emacs
  Terminal mode       $ emacs -nw

# Basic usage

  Indent              Select text then press TAB
  Cut                 CTRL-w
  Copy                ALT-w
  Paste ("yank")      CTRL-y
  Begin selection     CTRL-SPACE
  Search/Find         CTRL-s
  Replace             ALT-% (ALT-SHIFT-5)
  Save                CTRL-x CTRL-s
  Save as             CTRL-x CTRL-w
  Load/Open           CTRL-x CTRL-f
  Undo                CTRL-x u
  Highlight all text  CTRL-x h
  Directory listing   CTRL-x d
  Cancel a command    CTRL-g
  Font size bigger    CTRL-x CTRL-+
  Font size smaller   CTRL-x CTRL--

# Buffers

  Split screen vertically                         CTRL-x 2
  Split screen vertically with 5 row height       CTRL-u 5 CTRL-x 2
  Split screen horizontally                       CTRL-x 3
  Split screen horizontally with 24 column width  CTRL-u 24 CTRL-x 3
  Revert to single screen                         CTRL-x 1
  Hide the current screen                         CTRL-x 0
  Move to the next screen                         CTRL-x o
  Kill the current buffer                         CTRL-x k
  Select a buffer                                 CTRL-x b
  Run command in the scratch buffer               CTRL-x CTRL-e

# Navigation ( backward / forward )
  
  Character-wise                                  CTRL-b , CTRL-f
  Word-wise                                       ALT-b  , ALT-f
  Line-wise                                       CTRL-p , CTRL-n
  Sentence-wise                                   ALT-a  , ALT-e
  Paragraph-wise                                  ALT-{ , ALT-}
  Function-wise                                   CTRL-ALT-a , CTRL-ALT-e
  Line beginning / end                            CTRL-a , CTRL-e

# Other stuff

  Open a shell         ALT-x eshell
  Goto a line number   ALT-x goto-line
  Word wrap            ALT-x toggle-word-wrap
  Spell checking       ALT-x flyspell-mode
  Line numbers         ALT-x linum-mode
  Toggle line wrap     ALT-x visual-line-mode
  Compile some code    ALT-x compile
  List packages        ALT-x package-list-packages

# Line numbers

  To add line numbers and enable moving to a line with CTRL-l:

    (global-set-key "\C-l" 'goto-line)
    (add-hook 'find-file-hook (lambda () (linum-mode 1)))

I am reminded of my own recommendation regarding Emacs. I wrote this in 2006. Reference Sheet by Alex Schroeder (for Emacs).

Tags:

Comments on 2018-11-10 Manual Pages

I’ve done something similar using BASH functions. I first saw these in a comment on hackernews, and have simply taken them wholesale.

mdless() {
      pandoc -s -f markdown -t man $1.md | groff -T utf8 -man | less
        }
umedit() { mkdir -p ~/docs/linux_stuff/knowledge; vim ~/docs/linux_stuff/knowledge/$1.md; }
um() { mdless ~/docs/linux_stuff/knowledge/"$1"; }
umls() { ls ~/docs/linux_stuff/knowledge; }

However, these leave a little to be desired in terms of completeness. I’ve been mostly satisfied with them so far though!

– reed 2018-11-12 20:37 UTC


Hah, to translate from Markdown to man is genius. :)

– Alex 2018-11-13 06:22 UTC

Add Comment

2018-11-06 Memory Mystery

The mysteries of problems regarding my website continues. What the hell happened?

Sudden Spike

Memory for “apps” went up to 16G at around 21:00 and then almost everything was killed.

Let’s see what grep oom-killer /var/log/syslog has to say:

Nov  6 21:05:03 sibirocobombus kernel: [18425702.447121] /home/alex/farm invoked oom-killer: ...
Nov  6 21:05:09 sibirocobombus kernel: [18425708.244062] munin-update invoked oom-killer: ...
Nov  6 21:05:09 sibirocobombus kernel: [18425708.925423] /home/alex/farm invoked oom-killer: ...
Nov  6 21:05:13 sibirocobombus kernel: [18425712.660969] apache2 invoked oom-killer: ...
Nov  6 21:05:14 sibirocobombus kernel: [18425713.857753] fail2ban-server invoked oom-killer: ...
Nov  6 21:05:15 sibirocobombus kernel: [18425714.904769] fw_conntrack invoked oom-killer: ...
Nov  6 21:05:18 sibirocobombus kernel: [18425717.168963] apache_accesses invoked oom-killer: ...
Nov  6 21:05:18 sibirocobombus kernel: [18425717.985385] apache_accesses invoked oom-killer: ...
Nov  6 21:05:20 sibirocobombus kernel: [18425719.907638] vmstat invoked oom-killer: ...
Nov  6 21:05:23 sibirocobombus kernel: [18425722.207184] jbd2/vda1-8 invoked oom-killer: ...
Nov  6 21:05:24 sibirocobombus kernel: [18425723.800850] memory invoked oom-killer: ...
Nov  6 21:05:27 sibirocobombus kernel: [18425726.582062] exiqgrep invoked oom-killer: ...
Nov  6 21:05:29 sibirocobombus kernel: [18425728.479563] fail2ban-server invoked oom-killer: ...
Nov  6 21:05:31 sibirocobombus kernel: [18425730.862151] fail2ban-client invoked oom-killer: ...
Nov  6 21:05:31 sibirocobombus kernel: [18425731.049454] /home/alex/camp invoked oom-killer: ...

OK, and what happened back then?

Grepping /var/log/apache2/access.log for this minute does not reveal anything interesting.

My gopher logs show just two hits, no issues.

Checking the toadfarm log file shows the first problem at 21:05:00: “fork failed: Cannot allocate memory”. Hm. The other problem is at 21:05:11: “Worker 4166 has no heartbeat (30 seconds), restarting”. Could be anything? Grepping for “Inactivity timeout” gives me 36 hits from 21:04:56 to 21:05:12. The end must have come quickly.

Hm. Looking at the toadfarm log file, the number of hits to the website of my Arabic teacher in just two minutes is pretty amazing.

 arabisch-lernen.org        388    56%
       emacswiki.org        273    40%
    campaignwiki.org         14     2%
                              6     0%

This is grep '21:0[45]:.*' /home/alex/farm/farm*.log | /home/alex/bin/site-grouping and the second script is very simple:

#!/usr/bin/env perl
while (<STDIN>) {
  m!GET https?://([^:]+):!;
  $site{$1}++;
  $total++;
}
@result = sort {$site{$b} <=> $site{$a}} keys %site;
foreach $label (@result) {
  printf "%20s %10d   %3d%%\n", $label, $site{$label}, 100* $site{$label} / $total;
}

OK, so who’s calling these two sites?

Note that this is toadfarm reporting. At the same time, Apache isn’t reporting as much. grep "21:0[45]" /var/log/apache2/access.log | wc -l says only 183 requests. When I look at the IP numbers, I don’t see a single IP number doing anything wrong. After a while I do think that there’s a lot of IP numbers starting with a common prefix and so I’m going to count them. 62 hits in two minutes from ChinaNetCenter Ltd. Gaaaah. Looking at the URLs these IP numbers are hitting it’s full of admin actions, diffs, edit action. Clearly, a misbehaving spider. I think I’m going to ban them all, for now.

Soon, I’m going follow these instructions. Grrrrr! Unexpected DDOS: Blocking China with ipset and iptables. (Just kidding. I hate that!)

Gah. Problem. ipset add blacklist 220.243.0.0/16 ended with an error. The list is full! And I cannot just ipset del blacklist 220.243.0.0/16 because that gives me another error saying the element isn’t added. Not even for ip in $(ipset list blacklist|grep 220); do ipset del blacklist $ip; done seems to work. Or at least, judging from a few calls to ipset list blacklist | wc -l it’s not very fast. What now?

I guess I will have to look at my setup again. I probably need a different type? It’s type is hash:ip and it’s maxelem is 65536. That would have to increase, of course. Oh well. Perhaps I’ll leave it for some other time. We’ll see how far we’ll get by not doing anything.

Tags:

Comments on 2018-11-06 Memory Mystery

How is 388 requests in 2 minutes a problem? That’s no more than 4 requests per second - shouldn’t the webserver be able to handle that?

Is there some kind of memory leak or pathological case that the chinese crawlers happen to hit? The sudden spike in memory use is not just because of 4 hits/s, is it?

Can you reproduce the memory use, maybe on a local machine replaying the requests from the log around the time of the problem (or just bombarding it with requests)?

Just random questions from my armchair :-)

Adam 2018-11-09 21:46 UTC


Heh. :) The main problem is that we’re not talking about static HTML pages. If somebody does a search on this wiki, there’s a grep over 7000 pages, and then the wiki code looks at the matches, renders the pages, highlights the phrases, and so on. That means the process takes a while. If it takes 10 seconds, and I get 4 searches per second (I doubt this was the explanation, just trying to explain how the server is prone to accidental denial of service attacks), then I have 40 processes grepping through 7000 files per second. And then it dies, because this is a tiny 2G virtual machine I’m somewhere, managed by myself and nobody else. At least that’s what I suspect.

That’s why my /robots.txt says Crawl-delay: 20.

Replaying the requests from the log is a good idea. I should do it the next time this happens. And I should make sure to compare the memory used with the number of processes running at the time.

– Alex Schroeder 2018-11-10 00:30 UTC


More armchair comments: Could you exclude the search URL in robots.txt?

A more laborous solution, if searching turns out to be the problem, is to index the pages using, say, Xapian (the Perl-bindings work nicely).

Anyway, good luck in finding the cause!

Adam 2018-11-10 13:49 UTC


As they are already ignoring the crawl delay I think it is safe to say that they either don’t know or don’t care. My robot.txt Contains the line Disallow: /wiki? which should take care of anything that is not a page view. But it’s being ignored.

A long time ago I used a Perl module that indexed the text and allowed scoring of results - but eventually it turned out that brute forcing search using grep was actually faster. But that was years ago and on a different setup. Perhaps I should revisit this.

These days I am more interested in blocking misbehaving agents. The old solution limits visits from the same IP number (surge protection, and I have fail2ban watching my log files, to watch over all the other services (cgit and the like). But if the request come from all over a subnet, these defenses don’t work, unfortunately.

– Alex Schroeder 2018-11-10 18:09 UTC

Add Comment

2018-10-13 cgit

@aminb asked my about my cgit setup.

This is /etc/cgitrc:

# cgit config
# see cgitrc(5) for details

css=/cgit-css/cgit.css
logo=https://alexschroeder.ch/pics/alex-and-beard-80x80.jpg
root-title=Git repositories
root-desc=Code repositories hosted by Alex Schroeder
root-readme=/home/git/root.html
readme=master:README.md
readme=master:README
about-filter=/usr/lib/cgit/filters/about-formatting.sh

source-filter=/usr/lib/cgit/filters/syntax-highlighting-or-markdown.py

clone-prefix=https://alexschroeder.ch/cgit

remove-suffix=1
enable-index-owner=0
scan-path=/home/git

This means that the /cgit path from all my virtual servers is going to end up serving cgit. I don’t mind.

The /home/git directory has all my repositories:

drwxr-xr-x 24 git  git  4096 Sep 13 23:14 .
drwxr-xr-x  4 root root 4096 Apr 20 15:20 ..
-rw-------  1 git  git   146 Apr 20 15:27 .bash_history
-rw-r--r--  1 git  git   220 Apr 20 15:20 .bash_logout
-rw-r--r--  1 git  git  3526 Apr 20 15:20 .bashrc
drwxrwsr-x  7 git  git  4096 Sep 10 10:38 bitlbee-mastodon.git
drwxrwsr-x  7 git  git  4096 Jun 21 11:10 character-sheet-generator.git
drwxrwsr-x  7 git  git  4096 Sep 14 23:41 emacs-setup.git
drwxrwsr-x  7 git  git  4096 Jun 21 11:10 face-generator.git
-rw-r--r--  1 git  git    39 Jun 21 11:07 .gitconfig
drwxrwsr-x  7 git  git  4096 Jun 21 11:10 gridmapper.git
drwxrwsr-x  7 git  git  4096 Jul 20 00:12 halberds-and-helmets.git
drwxrwsr-x  7 git  git  4096 Jun 21 11:10 hellebarden-und-helme.git
drwxr-xr-x  7 git  git  4096 Jun 25 14:50 hex-mapping.git
drwxrwsr-x  7 git  git  4096 Oct  8 23:26 mastodon-archive.git
drwxrwsr-x  7 git  git  4096 Sep 17 07:42 mastodon-blocker.git
drwxrwsr-x  7 git  git  4096 Jul 16 15:34 moku-pona.git
drwxrwsr-x  7 git  git  4096 Jul 24 15:50 mu.git
drwxrwsr-x  7 git  git  4096 Sep 13 18:08 oddmu.git
drwxrwsr-x  7 git  git  4096 Oct  7 20:54 oddmuse6.git
drwxrwsr-x  7 git  git  4096 Oct  7 18:15 oddmuse.git
-rw-r--r--  1 git  git   675 Apr 20 15:20 .profile
drwxrwsr-x  7 git  git  4096 Aug 30 23:52 qmk_firmware.git
-rw-r--r--  1 root root  122 Jun  4 08:44 README
-rw-r--r--  1 root root   49 Jun  4 08:43 README~
-rw-r--r--  1 root root  494 Jun  9 00:36 root.html
-rw-r--r--  1 root root   14 Jun  9 00:30 root.html~
drwxrwsr-x  7 git  git  4096 Jul 17 13:57 sitelen-mute.git
drwxrwsr-x  7 git  git  4096 Jul 31 11:47 spellcasters.git
drwx------  2 git  git  4096 Jun 13 09:13 .ssh
drwxrwsr-x  7 git  git  4096 Sep 11 09:08 text-mapper-mastodon-bot.git
drwxrwsr-x  7 git  git  4096 Sep 18 11:59 trunk.git
drwxrwsr-x  7 git  git  4096 Aug 31 00:04 typing.git

Because I’m evil, the .gitconfig says:

[receive]
	denynonfastforwards = false

The README file says:

# https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server
mkdir project.git
cd project.git
git init --bare

My root user has a little script called add-repo which I have to run to create new projects:

#!/bin/bash

if [ -z $1 ]; then
    echo Must provide the name of a new repo
    exit 1
elif [ ! -z $2 ]; then
    echo Must provide just the name a new repo
    exit 1
elif [ -e $1 ]; then
    echo This file exists already
    exit 1
fi

# https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server

dir="/home/git/$1.git"
mkdir "$dir"
chown git.git "$dir"
(cd "$dir" && sudo -u git git init --bare --shared)
echo Provide description
read desc &&  echo $desc > "$dir/description"

Remember the /etc/cgitrc file also mentioned some scripts for formatting. /usr/lib/cgit/filters/syntax-highlighting-or-markdown.py is what I frankensteined together. See 2018-06-11 Highlighting Code and Rendering Markdown for cgit for more.

Tags:

Add Comment

2018-09-05 Toadfarm and Monit

Every now and then I wake up to a website that’s dead. When I try to restart the Toadfarm using Monit, nothing happens. I connect to the box using ssh and try to restart it from the command line:

$ ./farm reload
 * Reloading the process farm
Can't create listen socket: Address already in use at /home/alex/perl5/perlbrew/perls/perl-5.26.1/lib/site_perl/5.26.1/Mojo/IOLoop.pm line 126.
   ...fail!

So the farm is up and listening to the port but it’s not actually doing anything. Now what? They’re all asleep!

top - 07:24:55 up 150 days, 15:34,  2 users,  load average: 0.00, 0.03, 0.00
Tasks: 124 total,   2 running, 122 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.7 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.3 st
KiB Mem :  3084768 total,   336608 free,   737820 used,  2010340 buff/cache
KiB Swap:  1022972 total,       16 free,  1022956 used.  2119480 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                     
16448 alex      20   0  331468 125888   3760 S  0.0  4.1   0:00.33 /home/alex/farm             
16563 alex      20   0  378652 104660   3668 S  0.0  3.4   0:00.38 /home/alex/farm             
 6681 alex      20   0  358828  92952   3940 S  0.0  3.0   0:00.89 /home/alex/farm             
  709 alex      20   0  347792  84888   5484 S  0.0  2.8   0:01.06 /home/alex/farm             
 6730 alex      20   0  342380  74416   3940 S  0.0  2.4   0:01.13 /home/alex/farm             
  710 alex      20   0  335376  65868   5424 S  0.0  2.1   0:00.99 /home/alex/farm             
22714 alex      30  10  118320  45080   7772 S  0.0  1.5   0:00.53 perl                        
18640 alex      20   0   85272  25896   6612 S  0.0  0.8   0:00.43 perl                        
 9579 alex      20   0  343312  25568   3848 S  0.0  0.8   0:00.45 /home/alex/farm             
13534 alex      20   0  331080  25548   3848 S  0.0  0.8   0:00.09 /home/alex/farm             
 8731 alex      20   0  362272  25520   3848 S  0.0  0.8   0:00.10 /home/alex/farm             
10399 alex      20   0  330736  25500   3812 S  0.0  0.8   0:00.35 /home/alex/farm             
31772 alex      20   0  327720  25468   3848 S  0.0  0.8   0:01.46 /home/alex/farm             
 1931 alex      20   0  328808  25396   3848 S  0.0  0.8   0:00.19 /home/alex/farm             
  974 alex      20   0  329248  25380   3848 S  0.0  0.8   0:00.27 /home/alex/farm             

And so I’m doing what I usually do, without actually solving the underlying problem:

for pid in $(ps -u alex | grep /farm | cut -c 1-5); do kill $pid; done

Do you have a better idea? What’s going on, here?

Tags:

Comments on 2018-09-05 Toadfarm and Monit

Gah. Had to kill it all, again. Why‽

I fixed my kill-farm script:

#! /bin/bash
for pid in $(ps -xo pid,command|perl -ne '@s=split; print $s[0] . "\n" if $s[1] eq "/home/alex/farm/farm"'); do
    kill -9 $pid
done

From /var/log/syslog:

Linux sibirocobombus 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Oct  1 19:40:53 2018 from 2a02:168:4823:0:9c0b:e651:b5d7:2f4f
root@sibirocobombus:~# cat /tmp/log
Oct  1 16:51:54 sibirocobombus kernel: [15296489.567141] /home/alex/farm: page allocation stalls for 10472ms, order:0, mode:0x24200ca(GFP_HIGHUSER_MOVABLE)
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567150] CPU: 0 PID: 13804 Comm: /home/alex/farm Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u5
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567153]  0000000000000000 ffffffffb9f285b4 ffffffffba5febb0 ffffb9ccc10cbb68
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567156]  ffffffffb9d84f3a 024200ca00000002 ffffffffba5febb0 ffffb9ccc10cbb08
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567159]  ffff8f1a00000010 ffffb9ccc10cbb78 ffffb9ccc10cbb28 98dccf6c149eda9d
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567161] Call Trace:
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567168]  [<ffffffffb9f285b4>] ? dump_stack+0x5c/0x78
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567171]  [<ffffffffb9d84f3a>] ? warn_alloc+0x13a/0x160
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567173]  [<ffffffffb9d8592d>] ? __alloc_pages_slowpath+0x95d/0xbc0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567175]  [<ffffffffb9d85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567177]  [<ffffffffb9dd7c3e>] ? alloc_pages_vma+0xae/0x260
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567181]  [<ffffffffb9daf069>] ? wp_page_copy+0x89/0x700
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567183]  [<ffffffffb9db0361>] ? do_wp_page+0x161/0x7d0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567186]  [<ffffffffb9db3170>] ? handle_mm_fault+0x8d0/0x1350
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567188]  [<ffffffffb9c24701>] ? __switch_to+0x2c1/0x6c0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567191]  [<ffffffffb9c5fd84>] ? __do_page_fault+0x2a4/0x510
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567194]  [<ffffffffba207788>] ? async_page_fault+0x28/0x30
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567195] Mem-Info:
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199] active_anon:563083 inactive_anon:141043 isolated_anon:813
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  active_file:3237 inactive_file:1620 isolated_file:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  unevictable:0 dirty:21 writeback:7962 unstable:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  slab_reclaimable:8695 slab_unreclaimable:14080
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  mapped:5449 shmem:8956 pagetables:15088 bounce:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  free:14229 free_pcp:0 free_cma:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567209] Node 0 active_anon:2252332kB inactive_anon:564172kB active_file:12948kB inactive_file:6480kB unevictable:0kB isolated(anon):3252kB isolated(file):0kB mapped:21796kB dirty:84kB writeback:31848kB shmem:35824kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:2 all_unreclaimable? no
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567210] Node 0 DMA free:12124kB min:232kB low:288kB high:344kB active_anon:3492kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:12kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:184kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567214] lowmem_reserve[]: 0 2975 2975 2975 2975
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567217] Node 0 DMA32 free:44792kB min:44820kB low:56024kB high:67228kB active_anon:2248840kB inactive_anon:564172kB active_file:12948kB inactive_file:6480kB unevictable:0kB writepending:31932kB present:3129216kB managed:3068860kB mlocked:0kB slab_reclaimable:34768kB slab_unreclaimable:56232kB kernel_stack:5392kB pagetables:60168kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567221] lowmem_reserve[]: 0 0 0 0 0
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567224] Node 0 DMA: 5*4kB (MEH) 5*8kB (UME) 4*16kB (ME) 5*32kB (UEH) 9*64kB (UMEH) 6*128kB (UEH) 5*256kB (UEH) 2*512kB (ME) 2*1024kB (UH) 3*2048kB (UME) 0*4096kB = 12124kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567236] Node 0 DMA32: 2599*4kB (UMEH) 1538*8kB (UMEH) 672*16kB (UMEH) 265*32kB (UME) 23*64kB (MEH) 1*128kB (H) 1*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB 0*4096kB = 44812kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567248] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567249] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567249] 26825 total pagecache pages
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567251] 13006 pages in swap cache
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567252] Swap cache stats: add 3064172, delete 3051166, find 484392409/484925615
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567253] Free swap  = 147028kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567253] Total swap = 1022972kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567254] 786302 pages RAM
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567255] 0 pages HighMem/MovableOnly
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567255] 15110 pages reserved
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567256] 0 pages hwpoisoned
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538179] /home/alex/farm invoked oom-killer: gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE), nodemask=0, order=0, oom_score_adj=0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538181] /home/alex/farm cpuset=/ mems_allowed=0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538186] CPU: 0 PID: 13764 Comm: /home/alex/farm Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u5
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538189]  0000000000000000 ffffffffb9f285b4 ffffb9ccc0e9bc20 ffff8f1a4e8f8100
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538193]  ffffffffb9dfe020 0000000000000000 0000000000000000 0000000000000001
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538195]  ffffffffb9d844e7 0000004252982a80 ffffffffc0454695 0000000000000001
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538197] Call Trace:
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538204]  [<ffffffffb9f285b4>] ? dump_stack+0x5c/0x78
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538206]  [<ffffffffb9dfe020>] ? dump_header+0x78/0x1fd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538209]  [<ffffffffb9d844e7>] ? get_page_from_freelist+0x3f7/0xb40
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538214]  [<ffffffffc0454695>] ? virtballoon_oom_notify+0x25/0x70 [virtio_balloon]
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538217]  [<ffffffffb9d8047a>] ? oom_kill_process+0x21a/0x3e0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538220]  [<ffffffffb9d800fd>] ? oom_badness+0xed/0x170
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538221]  [<ffffffffb9d80911>] ? out_of_memory+0x111/0x470
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538223]  [<ffffffffb9d85b4f>] ? __alloc_pages_slowpath+0xb7f/0xbc0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538225]  [<ffffffffb9d85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538227]  [<ffffffffb9dd7c3e>] ? alloc_pages_vma+0xae/0x260
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538230]  [<ffffffffb9daf069>] ? wp_page_copy+0x89/0x700
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538233]  [<ffffffffb9db0361>] ? do_wp_page+0x161/0x7d0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538235]  [<ffffffffb9db3170>] ? handle_mm_fault+0x8d0/0x1350
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538238]  [<ffffffffb9c24701>] ? __switch_to+0x2c1/0x6c0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538240]  [<ffffffffb9c5fd84>] ? __do_page_fault+0x2a4/0x510
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538243]  [<ffffffffba207788>] ? async_page_fault+0x28/0x30
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538244] Mem-Info:
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248] active_anon:571476 inactive_anon:143154 isolated_anon:532
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  active_file:204 inactive_file:222 isolated_file:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  unevictable:0 dirty:0 writeback:3178 unstable:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  slab_reclaimable:3157 slab_unreclaimable:13740
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  mapped:1215 shmem:7913 pagetables:15023 bounce:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  free:14207 free_pcp:37 free_cma:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538252] Node 0 active_anon:2285904kB inactive_anon:572616kB active_file:816kB inactive_file:888kB unevictable:0kB isolated(anon):2128kB isolated(file):0kB mapped:4860kB dirty:0kB writeback:12712kB shmem:31652kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:5391 all_unreclaimable? yes
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538252] Node 0 DMA free:12124kB min:232kB low:288kB high:344kB active_anon:3484kB inactive_anon:4kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:12kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:188kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538257] lowmem_reserve[]: 0 2975 2975 2975 2975
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538260] Node 0 DMA32 free:44704kB min:44820kB low:56024kB high:67228kB active_anon:2282420kB inactive_anon:572612kB active_file:816kB inactive_file:888kB unevictable:0kB writepending:12712kB present:3129216kB managed:3068860kB mlocked:0kB slab_reclaimable:12616kB slab_unreclaimable:54872kB kernel_stack:5376kB pagetables:59904kB bounce:0kB free_pcp:148kB local_pcp:148kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538264] lowmem_reserve[]: 0 0 0 0 0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538266] Node 0 DMA: 4*4kB (UEH) 2*8kB (E) 2*16kB (E) 5*32kB (UMEH) 8*64kB (UMEH) 5*128kB (UEH) 4*256kB (UEH) 1*512kB (M) 3*1024kB (UEH) 3*2048kB (UME) 0*4096kB = 12128kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538278] Node 0 DMA32: 2108*4kB (UEH) 1402*8kB (UMEH) 746*16kB (UMEH) 286*32kB (UMEH) 40*64kB (MEH) 1*128kB (H) 1*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB 0*4096kB = 44704kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538289] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538291] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538291] 18160 total pagecache pages
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538293] 9813 pages in swap cache
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538294] Swap cache stats: add 3100981, delete 3091168, find 484392412/484925620
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538294] Free swap  = 0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538295] Total swap = 1022972kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538296] 786302 pages RAM
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538296] 0 pages HighMem/MovableOnly
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538297] 15110 pages reserved
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538298] 0 pages hwpoisoned
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538298] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538302] [  188]     0   188    15502      833      27       3       80             0 systemd-journal
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538304] [  376]     0   376    11625       55      26       3      104             0 systemd-logind
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538306] [  381]     0   381     7416       35      20       3       50             0 cron
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538308] [  387]     0   387    62560      163      30       3      351             0 rsyslogd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538309] [  388]     0   388     3152        9      12       3       37             0 rsync
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538311] [  393]     0   393     1054        0       8       3       35             0 acpid
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538312] [  394]   105   394    11283       60      27       3       78          -900 dbus-daemon
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538314] [  450]     0   450     3634        0      12       3       38             0 agetty
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538316] [  458]     0   458    28698      198      26       3       93             0 monit
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538317] [  464]     0   464    17486       41      37       3      164         -1000 sshd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538319] [  614]     0   614    12949      767      30       3     1563             0 munin-node
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538329] [  823]   107   823    27941       23      50       3      365             0 exim4
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538331] [10368]   100 10368    31821       19      32       3      119             0 systemd-timesyn
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538333] [10648]     0 10648    11308       12      24       3       91         -1000 systemd-udevd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538335] [25971]  1000 25971    27465      378      54       3     6922             0 /home/alex/camp
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538337] [22812]     0 22812    26416      842      58       3       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538338] [ 2794]     0  2794   169401     3422      61       4      455             0 fail2ban-server
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538340] [29327]    33 29327   216079     2688     100       4      530             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538342] [23538]  1000 23538    81856      167     143       3    29692             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538343] [14140]  1000 14140    81929      249     144       3    29760             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538345] [15321]  1000 15321    85002      249     149       3    31614             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538346] [15742]  1000 15742    82686      247     144       3    29465             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538348] [15822]  1000 15822    82672      253     144       3    29467             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538349] [18583]  1000 18583    85088      250     149       3    31258             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538351] [16180]  1000 16180    83100      164     148       3    30518             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538353] [16240]  1000 16240    84600      170     151       3    31838             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538354] [17135]  1000 17135    84455      169     150       3    31723             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538356] [18081]  1000 18081    83276      169     147       3    29866             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538357] [18089]  1000 18089    82757      167     146       3    29665             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538359] [18097]  1000 18097    83000      167     146       3    29722             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538360] [28873]  1000 28873    83946      167     148       3    30644             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538362] [29477]  1000 29477    84020      168     148       3    30825             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538364] [17703]  1000 17703    85534    10436     152       3    22151             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538366] [25630]  1000 25630    84044    17581     147       3    13484             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538367] [23799]  1000 23799    83909    16024     149       4    15467             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538369] [23831]  1000 23831    88924    19813     159       4    16552             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538370] [17148]  1000 17148    27465     3262      52       3     4112             0 /home/alex/camp
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538372] [24680] 65534 24680    12608     2506      27       3        0             0 python
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538374] [ 6121]  1000  6121    84608    32664     151       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538376] [ 9051]    33  9051    26290      742      54       3       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538377] [ 9372]  1000  9372    79677    26765     141       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538379] [ 9393]  1000  9393    88087    36094     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538380] [ 9395]  1000  9395   104793    53630     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538382] [ 9396]  1000  9396    94890    43125     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538392] [ 9418]  1000  9418    21312     4988      45       3        0             0 perl
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538394] [ 9989]  1000  9989    17541     3956      37       3        0             0 perl
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538395] [11782]    33 11782   215922     2843      99       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538397] [11833]    33 11833   214475     2459      97       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538398] [26741]  1000 26741    92116    38897     165       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538400] [17934]  1000 17934    87620    35407     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538402] [18489]  1000 18489    85855    32863     152       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538403] [19093]    33 19093   215694     2608      99       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538405] [13755]  1000 13755    94098    41334     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538407] [13758]  1000 13758   106413    54375     195       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538408] [13759]  1000 13759    94098    41361     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538410] [13763]  1000 13763    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538411] [13764]  1000 13764    89787    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538413] [13765]  1000 13765    93158    38962     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538414] [13766]  1000 13766    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538415] [13767]  1000 13767    94098    41334     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538417] [13768]  1000 13768   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538419] [13769]  1000 13769    93158    38962     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538420] [13771]  1000 13771    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538422] [13772]  1000 13772   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538423] [13773]  1000 13773    93158    38961     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538425] [13774]  1000 13774    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538427] [13775]    33 13775   214080      930      95       4       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538428] [13804]  1000 13804    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538430] [13805]  1000 13805    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538431] [13806]  1000 13806    95932    43127     174       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538433] [13807]  1000 13807   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538435] [13808]  1000 13808    92634    38961     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538436] [13809]  1000 13809    89786    36815     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538438] [13810]  1000 13810    95932    43127     174       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538439] [13812]  1000 13812    92634    38961     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538441] [13813]  1000 13813   105709    54206     190       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538442] [13814]  1000 13814    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538444] [13815]    33 13815   213919      868      95       4       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538446] [13817]  1000 13817    89164    36681     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538447] [13844]  1000 13844    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538449] [13845]  1000 13845   105609    54032     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538450] [13846]  1000 13846    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538452] [13847]  1000 13847   105789    54335     190       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538453] [13848]  1000 13848    89263    36789     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538455] [13849]  1000 13849    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538456] [13850]  1000 13850    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538458] [13851]  1000 13851    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538459] [13852]  1000 13852   105311    53674     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538461] [13853]  1000 13853    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538462] [13854]  1000 13854    92634    38940     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538463] [13855]  1000 13855    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538465] [13856]  1000 13856   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538466] [13857]  1000 13857    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538468] [13858]  1000 13858    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538469] [13859]  1000 13859    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538471] [13860]  1000 13860   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538472] [13861]  1000 13861    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538474] [13862]  1000 13862    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538475] [13863]  1000 13863    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538477] [13864]  1000 13864   105311    53737     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538478] [13865]  1000 13865    92116    38883     162       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538480] [13866]  1000 13866    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538481] [13867]  1000 13867    88087    36080     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538483] [13868]  1000 13868   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538484] [13869]  1000 13869    94890    43125     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538486] [13870]  1000 13870    92116    38883     162       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538488] [13871]  1000 13871    88087    36080     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538489] [13872]  1000 13872   104793    53616     188       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538491] [13873]  1000 13873    94890    43111     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538492] [13875]  1000 13875    88087    36070     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538494] Out of memory: Kill process 13768 (/home/alex/farm) score 53 or sacrifice child
Oct  1 16:51:56 sibirocobombus kernel: [15296512.867096] Killed process 13768 (/home/alex/farm) total-vm:425652kB, anon-rss:217024kB, file-rss:668kB, shmem-rss:0kB

– Alex Schroeder 2018-10-01 17:02 UTC


OK, I wrote myself a cull-farm script:

#! /home/alex/perl5/perlbrew/perls/perl-5.26.1/bin/perl
use Modern::Perl;
my %data = map { $_->[0] => $_->[1] }
           grep { $_->[2] eq "/home/alex/farm/farm" and $_->[3] ne "1" }
           map { [ split ] }
           (qx(ps -xo pid,etimes,command,ppid));
my @ids = sort { $data{$a} <=> $data{$b} } keys %data;
# say "pids: @ids";
my @old_ids = @ids[4 .. $#ids];
if (@old_ids > 0) {
  say "SIGTERM for @old_ids";
  kill 'TERM', @old_ids;
} else {
  say "The only child processes we have are @ids";
}

It runs a ps which prints the process id, the elapsed time in seconds, the command name (so that I can filter for my processes), and the parent process id. I don’t want to kill the top level process that keeps spawning the children!

– Alex Schroeder 2018-10-07 17:17 UTC


I fiddled with that script again because I noticed something. Here’s an example output when sorted by etimes, the elapsed time in seconds:

alex@sibirocobombus:~$ ps -x -k etimes -o pid,etimes,command,ppid | grep /home/alex/farm/farm
22634       0 grep /home/alex/farm/farm   16815
22541      75 /home/alex/farm/farm        32495
22542      75 /home/alex/farm/farm        32495
22539      76 /home/alex/farm/farm        32495
22540      76 /home/alex/farm/farm        32495
32495   27081 /home/alex/farm/farm            1
 4376  217327 /home/alex/farm/farm            1
 4352  217356 /home/alex/farm/farm            1
 4285  217497 /home/alex/farm/farm            1
 4238  217538 /home/alex/farm/farm            1

In this situation we want to start killing from the end. Look at all those old processes that are still hanging around, started by 1. These need to go!

#! /home/alex/perl5/perlbrew/perls/perl-5.26.1/bin/perl
use Modern::Perl;

if (grep /^(-h|--help)$/, @ARGV) {
  say "Use -d for debugging";
  exit;
}

my $debug = 0;
$debug = 1 if (grep /^(-d|--debug)$/, @ARGV);

my %data =
    map { $_->[0] => { etimes => $_->[1], ppid => $_->[3] }}
    grep { $_->[2] eq "/home/alex/farm/farm" }
    map { [ split ] }
    # -o is output; 0: pid; 1: etimes; 2: command; 3: ppid
    # -k is sorting; etimes is elapsed time
    (qx(ps -x -k etimes -o pid,etimes,command,ppid));

# Example result:
# 22634       0 grep /home/alex/farm/farm   16815
# 22541      75 /home/alex/farm/farm        32495
# 22542      75 /home/alex/farm/farm        32495
# 22539      76 /home/alex/farm/farm        32495
# 22540      76 /home/alex/farm/farm        32495
# 32495   27081 /home/alex/farm/farm            1
#  4376  217327 /home/alex/farm/farm            1
#  4352  217356 /home/alex/farm/farm            1
#  4285  217497 /home/alex/farm/farm            1
#  4238  217538 /home/alex/farm/farm            1
# In this situation, the process that we do not want to kill is 32495 and we
# want to start killing from the back: 4376, 4352, 4285, 4238.

for my $pid (keys %data) {
  if ($data{$pid} and $data{$data{$pid}->{ppid}}) {
    say "protected pid: $data{$pid}->{ppid}" if $debug;
    delete $data{$data{$pid}->{ppid}};
  }
}
my @ids = sort { $data{$b} <=> $data{$a} } keys %data;
say "pids: @ids" if $debug;
my @old_ids = @ids[4 .. $#ids];
if (@old_ids > 0) {
  say "SIGTERM for @old_ids";
  kill 'TERM', @old_ids unless $debug;
} else {
  say "The only child processes we have are @ids";
}

– Alex Schroeder 2018-10-26 12:04 UTC

Add Comment

More...

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.