Monit

See Sibirocobombus. It’s the tool I use to restart processes when I’m away from the laptop.

2018-09-05 Toadfarm and Monit

Every now and then I wake up to a website that’s dead. When I try to restart the Toadfarm using Monit, nothing happens. I connect to the box using ssh and try to restart it from the command line:

$ ./farm reload
 * Reloading the process farm
Can't create listen socket: Address already in use at /home/alex/perl5/perlbrew/perls/perl-5.26.1/lib/site_perl/5.26.1/Mojo/IOLoop.pm line 126.
   ...fail!

So the farm is up and listening to the port but it’s not actually doing anything. Now what? They’re all asleep!

top - 07:24:55 up 150 days, 15:34,  2 users,  load average: 0.00, 0.03, 0.00
Tasks: 124 total,   2 running, 122 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.7 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.3 st
KiB Mem :  3084768 total,   336608 free,   737820 used,  2010340 buff/cache
KiB Swap:  1022972 total,       16 free,  1022956 used.  2119480 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                     
16448 alex      20   0  331468 125888   3760 S  0.0  4.1   0:00.33 /home/alex/farm             
16563 alex      20   0  378652 104660   3668 S  0.0  3.4   0:00.38 /home/alex/farm             
 6681 alex      20   0  358828  92952   3940 S  0.0  3.0   0:00.89 /home/alex/farm             
  709 alex      20   0  347792  84888   5484 S  0.0  2.8   0:01.06 /home/alex/farm             
 6730 alex      20   0  342380  74416   3940 S  0.0  2.4   0:01.13 /home/alex/farm             
  710 alex      20   0  335376  65868   5424 S  0.0  2.1   0:00.99 /home/alex/farm             
22714 alex      30  10  118320  45080   7772 S  0.0  1.5   0:00.53 perl                        
18640 alex      20   0   85272  25896   6612 S  0.0  0.8   0:00.43 perl                        
 9579 alex      20   0  343312  25568   3848 S  0.0  0.8   0:00.45 /home/alex/farm             
13534 alex      20   0  331080  25548   3848 S  0.0  0.8   0:00.09 /home/alex/farm             
 8731 alex      20   0  362272  25520   3848 S  0.0  0.8   0:00.10 /home/alex/farm             
10399 alex      20   0  330736  25500   3812 S  0.0  0.8   0:00.35 /home/alex/farm             
31772 alex      20   0  327720  25468   3848 S  0.0  0.8   0:01.46 /home/alex/farm             
 1931 alex      20   0  328808  25396   3848 S  0.0  0.8   0:00.19 /home/alex/farm             
  974 alex      20   0  329248  25380   3848 S  0.0  0.8   0:00.27 /home/alex/farm             

And so I’m doing what I usually do, without actually solving the underlying problem:

for pid in $(ps -u alex | grep /farm | cut -c 1-5); do kill $pid; done

Do you have a better idea? What’s going on, here?

Tags:

Comments on 2018-09-05 Toadfarm and Monit

Gah. Had to kill it all, again. Why‽

I fixed my kill-farm script:

#! /bin/bash
for pid in $(ps -xo pid,command|perl -ne '@s=split; print $s[0] . "\n" if $s[1] eq "/home/alex/farm/farm"'); do
    kill -9 $pid
done

From /var/log/syslog:

Linux sibirocobombus 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Oct  1 19:40:53 2018 from 2a02:168:4823:0:9c0b:e651:b5d7:2f4f
root@sibirocobombus:~# cat /tmp/log
Oct  1 16:51:54 sibirocobombus kernel: [15296489.567141] /home/alex/farm: page allocation stalls for 10472ms, order:0, mode:0x24200ca(GFP_HIGHUSER_MOVABLE)
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567150] CPU: 0 PID: 13804 Comm: /home/alex/farm Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u5
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567153]  0000000000000000 ffffffffb9f285b4 ffffffffba5febb0 ffffb9ccc10cbb68
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567156]  ffffffffb9d84f3a 024200ca00000002 ffffffffba5febb0 ffffb9ccc10cbb08
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567159]  ffff8f1a00000010 ffffb9ccc10cbb78 ffffb9ccc10cbb28 98dccf6c149eda9d
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567161] Call Trace:
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567168]  [<ffffffffb9f285b4>] ? dump_stack+0x5c/0x78
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567171]  [<ffffffffb9d84f3a>] ? warn_alloc+0x13a/0x160
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567173]  [<ffffffffb9d8592d>] ? __alloc_pages_slowpath+0x95d/0xbc0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567175]  [<ffffffffb9d85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567177]  [<ffffffffb9dd7c3e>] ? alloc_pages_vma+0xae/0x260
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567181]  [<ffffffffb9daf069>] ? wp_page_copy+0x89/0x700
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567183]  [<ffffffffb9db0361>] ? do_wp_page+0x161/0x7d0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567186]  [<ffffffffb9db3170>] ? handle_mm_fault+0x8d0/0x1350
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567188]  [<ffffffffb9c24701>] ? __switch_to+0x2c1/0x6c0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567191]  [<ffffffffb9c5fd84>] ? __do_page_fault+0x2a4/0x510
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567194]  [<ffffffffba207788>] ? async_page_fault+0x28/0x30
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567195] Mem-Info:
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199] active_anon:563083 inactive_anon:141043 isolated_anon:813
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  active_file:3237 inactive_file:1620 isolated_file:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  unevictable:0 dirty:21 writeback:7962 unstable:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  slab_reclaimable:8695 slab_unreclaimable:14080
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  mapped:5449 shmem:8956 pagetables:15088 bounce:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  free:14229 free_pcp:0 free_cma:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567209] Node 0 active_anon:2252332kB inactive_anon:564172kB active_file:12948kB inactive_file:6480kB unevictable:0kB isolated(anon):3252kB isolated(file):0kB mapped:21796kB dirty:84kB writeback:31848kB shmem:35824kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:2 all_unreclaimable? no
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567210] Node 0 DMA free:12124kB min:232kB low:288kB high:344kB active_anon:3492kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:12kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:184kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567214] lowmem_reserve[]: 0 2975 2975 2975 2975
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567217] Node 0 DMA32 free:44792kB min:44820kB low:56024kB high:67228kB active_anon:2248840kB inactive_anon:564172kB active_file:12948kB inactive_file:6480kB unevictable:0kB writepending:31932kB present:3129216kB managed:3068860kB mlocked:0kB slab_reclaimable:34768kB slab_unreclaimable:56232kB kernel_stack:5392kB pagetables:60168kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567221] lowmem_reserve[]: 0 0 0 0 0
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567224] Node 0 DMA: 5*4kB (MEH) 5*8kB (UME) 4*16kB (ME) 5*32kB (UEH) 9*64kB (UMEH) 6*128kB (UEH) 5*256kB (UEH) 2*512kB (ME) 2*1024kB (UH) 3*2048kB (UME) 0*4096kB = 12124kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567236] Node 0 DMA32: 2599*4kB (UMEH) 1538*8kB (UMEH) 672*16kB (UMEH) 265*32kB (UME) 23*64kB (MEH) 1*128kB (H) 1*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB 0*4096kB = 44812kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567248] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567249] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567249] 26825 total pagecache pages
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567251] 13006 pages in swap cache
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567252] Swap cache stats: add 3064172, delete 3051166, find 484392409/484925615
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567253] Free swap  = 147028kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567253] Total swap = 1022972kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567254] 786302 pages RAM
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567255] 0 pages HighMem/MovableOnly
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567255] 15110 pages reserved
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567256] 0 pages hwpoisoned
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538179] /home/alex/farm invoked oom-killer: gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE), nodemask=0, order=0, oom_score_adj=0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538181] /home/alex/farm cpuset=/ mems_allowed=0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538186] CPU: 0 PID: 13764 Comm: /home/alex/farm Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u5
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538189]  0000000000000000 ffffffffb9f285b4 ffffb9ccc0e9bc20 ffff8f1a4e8f8100
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538193]  ffffffffb9dfe020 0000000000000000 0000000000000000 0000000000000001
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538195]  ffffffffb9d844e7 0000004252982a80 ffffffffc0454695 0000000000000001
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538197] Call Trace:
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538204]  [<ffffffffb9f285b4>] ? dump_stack+0x5c/0x78
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538206]  [<ffffffffb9dfe020>] ? dump_header+0x78/0x1fd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538209]  [<ffffffffb9d844e7>] ? get_page_from_freelist+0x3f7/0xb40
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538214]  [<ffffffffc0454695>] ? virtballoon_oom_notify+0x25/0x70 [virtio_balloon]
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538217]  [<ffffffffb9d8047a>] ? oom_kill_process+0x21a/0x3e0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538220]  [<ffffffffb9d800fd>] ? oom_badness+0xed/0x170
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538221]  [<ffffffffb9d80911>] ? out_of_memory+0x111/0x470
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538223]  [<ffffffffb9d85b4f>] ? __alloc_pages_slowpath+0xb7f/0xbc0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538225]  [<ffffffffb9d85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538227]  [<ffffffffb9dd7c3e>] ? alloc_pages_vma+0xae/0x260
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538230]  [<ffffffffb9daf069>] ? wp_page_copy+0x89/0x700
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538233]  [<ffffffffb9db0361>] ? do_wp_page+0x161/0x7d0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538235]  [<ffffffffb9db3170>] ? handle_mm_fault+0x8d0/0x1350
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538238]  [<ffffffffb9c24701>] ? __switch_to+0x2c1/0x6c0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538240]  [<ffffffffb9c5fd84>] ? __do_page_fault+0x2a4/0x510
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538243]  [<ffffffffba207788>] ? async_page_fault+0x28/0x30
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538244] Mem-Info:
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248] active_anon:571476 inactive_anon:143154 isolated_anon:532
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  active_file:204 inactive_file:222 isolated_file:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  unevictable:0 dirty:0 writeback:3178 unstable:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  slab_reclaimable:3157 slab_unreclaimable:13740
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  mapped:1215 shmem:7913 pagetables:15023 bounce:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  free:14207 free_pcp:37 free_cma:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538252] Node 0 active_anon:2285904kB inactive_anon:572616kB active_file:816kB inactive_file:888kB unevictable:0kB isolated(anon):2128kB isolated(file):0kB mapped:4860kB dirty:0kB writeback:12712kB shmem:31652kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:5391 all_unreclaimable? yes
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538252] Node 0 DMA free:12124kB min:232kB low:288kB high:344kB active_anon:3484kB inactive_anon:4kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:12kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:188kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538257] lowmem_reserve[]: 0 2975 2975 2975 2975
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538260] Node 0 DMA32 free:44704kB min:44820kB low:56024kB high:67228kB active_anon:2282420kB inactive_anon:572612kB active_file:816kB inactive_file:888kB unevictable:0kB writepending:12712kB present:3129216kB managed:3068860kB mlocked:0kB slab_reclaimable:12616kB slab_unreclaimable:54872kB kernel_stack:5376kB pagetables:59904kB bounce:0kB free_pcp:148kB local_pcp:148kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538264] lowmem_reserve[]: 0 0 0 0 0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538266] Node 0 DMA: 4*4kB (UEH) 2*8kB (E) 2*16kB (E) 5*32kB (UMEH) 8*64kB (UMEH) 5*128kB (UEH) 4*256kB (UEH) 1*512kB (M) 3*1024kB (UEH) 3*2048kB (UME) 0*4096kB = 12128kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538278] Node 0 DMA32: 2108*4kB (UEH) 1402*8kB (UMEH) 746*16kB (UMEH) 286*32kB (UMEH) 40*64kB (MEH) 1*128kB (H) 1*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB 0*4096kB = 44704kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538289] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538291] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538291] 18160 total pagecache pages
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538293] 9813 pages in swap cache
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538294] Swap cache stats: add 3100981, delete 3091168, find 484392412/484925620
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538294] Free swap  = 0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538295] Total swap = 1022972kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538296] 786302 pages RAM
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538296] 0 pages HighMem/MovableOnly
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538297] 15110 pages reserved
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538298] 0 pages hwpoisoned
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538298] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538302] [  188]     0   188    15502      833      27       3       80             0 systemd-journal
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538304] [  376]     0   376    11625       55      26       3      104             0 systemd-logind
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538306] [  381]     0   381     7416       35      20       3       50             0 cron
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538308] [  387]     0   387    62560      163      30       3      351             0 rsyslogd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538309] [  388]     0   388     3152        9      12       3       37             0 rsync
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538311] [  393]     0   393     1054        0       8       3       35             0 acpid
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538312] [  394]   105   394    11283       60      27       3       78          -900 dbus-daemon
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538314] [  450]     0   450     3634        0      12       3       38             0 agetty
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538316] [  458]     0   458    28698      198      26       3       93             0 monit
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538317] [  464]     0   464    17486       41      37       3      164         -1000 sshd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538319] [  614]     0   614    12949      767      30       3     1563             0 munin-node
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538329] [  823]   107   823    27941       23      50       3      365             0 exim4
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538331] [10368]   100 10368    31821       19      32       3      119             0 systemd-timesyn
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538333] [10648]     0 10648    11308       12      24       3       91         -1000 systemd-udevd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538335] [25971]  1000 25971    27465      378      54       3     6922             0 /home/alex/camp
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538337] [22812]     0 22812    26416      842      58       3       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538338] [ 2794]     0  2794   169401     3422      61       4      455             0 fail2ban-server
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538340] [29327]    33 29327   216079     2688     100       4      530             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538342] [23538]  1000 23538    81856      167     143       3    29692             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538343] [14140]  1000 14140    81929      249     144       3    29760             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538345] [15321]  1000 15321    85002      249     149       3    31614             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538346] [15742]  1000 15742    82686      247     144       3    29465             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538348] [15822]  1000 15822    82672      253     144       3    29467             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538349] [18583]  1000 18583    85088      250     149       3    31258             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538351] [16180]  1000 16180    83100      164     148       3    30518             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538353] [16240]  1000 16240    84600      170     151       3    31838             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538354] [17135]  1000 17135    84455      169     150       3    31723             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538356] [18081]  1000 18081    83276      169     147       3    29866             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538357] [18089]  1000 18089    82757      167     146       3    29665             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538359] [18097]  1000 18097    83000      167     146       3    29722             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538360] [28873]  1000 28873    83946      167     148       3    30644             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538362] [29477]  1000 29477    84020      168     148       3    30825             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538364] [17703]  1000 17703    85534    10436     152       3    22151             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538366] [25630]  1000 25630    84044    17581     147       3    13484             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538367] [23799]  1000 23799    83909    16024     149       4    15467             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538369] [23831]  1000 23831    88924    19813     159       4    16552             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538370] [17148]  1000 17148    27465     3262      52       3     4112             0 /home/alex/camp
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538372] [24680] 65534 24680    12608     2506      27       3        0             0 python
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538374] [ 6121]  1000  6121    84608    32664     151       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538376] [ 9051]    33  9051    26290      742      54       3       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538377] [ 9372]  1000  9372    79677    26765     141       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538379] [ 9393]  1000  9393    88087    36094     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538380] [ 9395]  1000  9395   104793    53630     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538382] [ 9396]  1000  9396    94890    43125     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538392] [ 9418]  1000  9418    21312     4988      45       3        0             0 perl
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538394] [ 9989]  1000  9989    17541     3956      37       3        0             0 perl
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538395] [11782]    33 11782   215922     2843      99       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538397] [11833]    33 11833   214475     2459      97       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538398] [26741]  1000 26741    92116    38897     165       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538400] [17934]  1000 17934    87620    35407     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538402] [18489]  1000 18489    85855    32863     152       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538403] [19093]    33 19093   215694     2608      99       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538405] [13755]  1000 13755    94098    41334     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538407] [13758]  1000 13758   106413    54375     195       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538408] [13759]  1000 13759    94098    41361     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538410] [13763]  1000 13763    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538411] [13764]  1000 13764    89787    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538413] [13765]  1000 13765    93158    38962     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538414] [13766]  1000 13766    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538415] [13767]  1000 13767    94098    41334     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538417] [13768]  1000 13768   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538419] [13769]  1000 13769    93158    38962     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538420] [13771]  1000 13771    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538422] [13772]  1000 13772   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538423] [13773]  1000 13773    93158    38961     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538425] [13774]  1000 13774    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538427] [13775]    33 13775   214080      930      95       4       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538428] [13804]  1000 13804    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538430] [13805]  1000 13805    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538431] [13806]  1000 13806    95932    43127     174       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538433] [13807]  1000 13807   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538435] [13808]  1000 13808    92634    38961     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538436] [13809]  1000 13809    89786    36815     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538438] [13810]  1000 13810    95932    43127     174       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538439] [13812]  1000 13812    92634    38961     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538441] [13813]  1000 13813   105709    54206     190       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538442] [13814]  1000 13814    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538444] [13815]    33 13815   213919      868      95       4       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538446] [13817]  1000 13817    89164    36681     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538447] [13844]  1000 13844    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538449] [13845]  1000 13845   105609    54032     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538450] [13846]  1000 13846    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538452] [13847]  1000 13847   105789    54335     190       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538453] [13848]  1000 13848    89263    36789     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538455] [13849]  1000 13849    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538456] [13850]  1000 13850    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538458] [13851]  1000 13851    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538459] [13852]  1000 13852   105311    53674     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538461] [13853]  1000 13853    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538462] [13854]  1000 13854    92634    38940     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538463] [13855]  1000 13855    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538465] [13856]  1000 13856   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538466] [13857]  1000 13857    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538468] [13858]  1000 13858    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538469] [13859]  1000 13859    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538471] [13860]  1000 13860   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538472] [13861]  1000 13861    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538474] [13862]  1000 13862    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538475] [13863]  1000 13863    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538477] [13864]  1000 13864   105311    53737     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538478] [13865]  1000 13865    92116    38883     162       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538480] [13866]  1000 13866    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538481] [13867]  1000 13867    88087    36080     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538483] [13868]  1000 13868   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538484] [13869]  1000 13869    94890    43125     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538486] [13870]  1000 13870    92116    38883     162       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538488] [13871]  1000 13871    88087    36080     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538489] [13872]  1000 13872   104793    53616     188       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538491] [13873]  1000 13873    94890    43111     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538492] [13875]  1000 13875    88087    36070     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538494] Out of memory: Kill process 13768 (/home/alex/farm) score 53 or sacrifice child
Oct  1 16:51:56 sibirocobombus kernel: [15296512.867096] Killed process 13768 (/home/alex/farm) total-vm:425652kB, anon-rss:217024kB, file-rss:668kB, shmem-rss:0kB

– Alex Schroeder 2018-10-01 17:02 UTC


OK, I wrote myself a cull-farm script:

#! /home/alex/perl5/perlbrew/perls/perl-5.26.1/bin/perl
use Modern::Perl;
my %data = map { $_->[0] => $_->[1] }
           grep { $_->[2] eq "/home/alex/farm/farm" and $_->[3] ne "1" }
           map { [ split ] }
           (qx(ps -xo pid,etimes,command,ppid));
my @ids = sort { $data{$a} <=> $data{$b} } keys %data;
# say "pids: @ids";
my @old_ids = @ids[4 .. $#ids];
if (@old_ids > 0) {
  say "SIGTERM for @old_ids";
  kill 'TERM', @old_ids;
} else {
  say "The only child processes we have are @ids";
}

It runs a ps which prints the process id, the elapsed time in seconds, the command name (so that I can filter for my processes), and the parent process id. I don’t want to kill the top level process that keeps spawning the children!

– Alex Schroeder 2018-10-07 17:17 UTC

Add Comment

2017-12-27 Gopher Server

The wiki is available as a Gopher server. This is weird, but it works.

Try it. lynx is a good Gopher client:

lynx gopher://alexschroeder.ch

or:

lynx gopher://alexschroeder.ch/02017-12-27_Gopher_Server

Does anybody know why I need to start the pagename with the gopher type character? (or any character at all)?

As for monitoring, I use monit and have set up the following:

check process gopher-server with pidfile /home/alex/alexschroeder-gopher-server.pid
      start program = "/home/alex/perl5/perlbrew/perls/perl-5.24.0/bin/perl /home/alex/farm/gopher-server.pl --setsid --user=alex --group=alex --port=70 --log_level=2 --log_file=/home/alex/farm/gopher-server.log --pid_file=/home/alex/farm/gopher-server.pid --wiki=/home/alex/farm/wiki.pl --wiki_dir=/home/alex/alexschroeder --wiki_pages=SiteMap --wiki_pages=About"
    stop program = "/bin/bash -c 'kill -s SIGTERM `cat /home/alex/alexschroeder-gopher-server.pid`'"
    if failed
	host alexschroeder.ch
	port 70
	type tcp
        send "Alex\r\n"
	expect "#REDIRECT.*"
	for 5 cycles
	then restart
    if totalmem > 100 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then timeout

The Net::Server is where you find stuff regarding all the command line options. --wiki, --wiki_pages and --wiki_dir are the ones I added.

Why gopher? Well...

“It’s actually rather nice to have a small ecosystem because no one’s running annoying ads in Gopherspace or trying to track your browsing habits,” he says. “The protocol makes the former hard and the latter almost impossible.” – Cameron Kaiser being quoted in The rise and fall of the Gopher protocol.
«The modern web is an ugly, massive, broken mess. This isn’t a secret or a new take or a controversial opinion. We stuff our webpages so full of tracking cookies and bitcoin mining advertisements and javascript apps that monitor your every move that it’s no wonder that modern web browsers are less effecient than ever.» — Gopher: Remembering the web that wasn't

This is nothing we can’t fix using HTML and HTTPS, but Gopher forces us to go without JavaScript and CSS and all that.

Tags:

Comments on 2017-12-27 Gopher Server

RFC 1436 remains relevant, I guess.

– Alex 2017-12-28 16:44 UTC


Work continues. I’ve added menus for page filtering, for search, for recent changes both major changes only and including minor changes.

Next up: page history.

Is there much else? I think soon enough the lingering question will be: how do we edit pages via gopher?

– Alex 2017-12-29 14:31 UTC


Writing unit tests makes me feel so much better.

– Alex 2017-12-30 04:56 UTC


On the Gopher web (what’s the name to use here, do we like the -verse suffix?), all the encoding problems of the early web are coming back. Latin 1 instead of UTF 8? No content negotiation? Not even encoding labeling? What are we going to do? Surely limiting ourselves to Latin 1 for menu texts is not good enough.

From RFC 1436: “It is highly recommended that the User_Name field contain only printable characters, since many different clients will be using it. However if eight bit characters are used, the characters should conform with the ISO Latin1 Character Set.”

It’s a different world out there. We need to talk about indicating encoding to clients.

Menu type m for metadata? e.g.

mEncoding\tUTF-8

I’ve been using a Gopher client on iOS for a bit and being German speaker, I soon stumbled over encoding issues. If I search Gopherpedia for „Zürich“, the client crashes. If I search for „Zurich“ and click on the first link, „Zürich“, it also crashes.

– Alex 2017-12-30 09:51 UTC

Add Comment

2017-12-01 Monit Self-Signed Certificate Expired

Monit runs a web server on a separate port so it doesn’t depend on your “real” web server. That also means it uses its own certificate. I just discovered that mine had expired.

Kallobombus Monit explains the basic setup. The main config file is this: /etc/monit/conf.d/monit.conf.

set httpd port 2812 and
    SSL ENABLE
    PEMFILE /etc/ssl/localcerts/monit.pem
    allow admin:*secret*

I wrote a note to myself in /etc/ssl/localcerts/README.

https://wiki.debian.org/Self-Signed_Certificate
https://www.howtoforge.com/server-monitoring-with-munin-and-monit-on-debian-wheezy-p2

sudo openssl req -new -x509 -days 365 -nodes -config monit.cnf -out monit.pem -keyout monit.pem
sudo chown root.root monit.pem
sudo chmod 0700 monit.pem
sudo service monit restart
sudo openssl x509 -noout -in monit.pem -fingerprint -sha256

The content of my monit.cnf file just provides some defaults.

# http://www.howtoforge.com/server-monitoring-with-munin-and-monit-on-debian-wheezy-p2
# create RSA certs - Server

RANDFILE = ./openssl.rnd

[ req ]
default_bits = 2048
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = CH

stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Zürich

localityName                    = Locality Name (eg, city)
localityName_default            = Zürich

organizationName                = Organization Name (eg, company)
organizationName_default        = Alex Schroeder

organizationalUnitName          = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Head Desk

commonName                      = Common Name (FQDN of your server)
commonName_default              = alexschroeder.ch

emailAddress                    = Email Address
emailAddress_default            = alex@gnu.org

[ cert_type ]
nsCertType = server

The last command gives me the new fingerprint of the service. When I reconnect to my monit, I’ll still get an error telling me that the cert authority is invalid, which it obviously is, since I self-signed this certificate.

Firefox uses it’s own certificate store. Thus, click on Advanced, Add Exception, View Certificate, and compare the SHA-256 fingerprint with what I just got on the server itself.

SHA256 Fingerprint=51:C8:A6:5E:2F:FB:F6:06:07:FE:B7:62:C3:03:4B:B8:BC:0D:43:AC:25:3C:40:B3:06:75:2F:70:5E:93:AC:78

This looks ok, so close this window and Permanently Accept the exception. Yay!

Tags:

Add Comment

2017-01-24 Monit, Toadfarm, Perlbrew

I’m using Perlbrew, which means that my Perl scripts depend on the right Perl version being used because the modules are all installed per version. If I don’t, then I’ll see errors like the following:

Can't load application from file "/home/alex/farm/face/face.pl": Can't locate GD.pm in @INC (you may need to install the GD module) (@INC contains: ...

When I run scripts from cron or monit, their environment is not as great as it usually is when I’m in an interactive session. Thus, I was able to run my scripts from the command line, but I couldn’t restart my webserver from the Monit page.

I run my sites using Toadfarm from /home/alex/farm/farm and thus I’d see the following in /var/log/monit.log once I ran into a problem:

[CET Jan 23 21:38:31] error    : 'toadfarm' total mem amount of 589.9 MB matches resource limit [total mem amount>500.0 MB]
[CET Jan 23 21:38:31] info     : 'toadfarm' trying to restart
[CET Jan 23 21:38:31] info     : 'toadfarm' stop: /home/alex/farm/farm
[CET Jan 23 21:39:01] error    : 'toadfarm' failed to stop (exit status 2) -- /home/alex/farm/farm: Can't locate Toadfarm.pm in @INC (you may need to install the Toadfarm module) (@INC contains: /home
/alex/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0/x86_64-linux /home/alex/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.
[CET Jan 23 21:41:01] info     : 'toadfarm' process is running after previous exec error (slow starting or manually recovered?)

So now I wrote a little wrapper script called /home/alex/farm/farm-wrapper.

I use it from my Monit config:

check process toadfarm with pidfile /home/alex/farm/farm.pid
    start program = "/home/alex/farm/farm-wrapper start" as uid alex gid alex
    stop program = "/home/alex/farm/farm-wrapper stop" as uid alex gid alex
    # http://alexschroeder.ch:8080/wiki
    if failed host alexschroeder.ch port 8080 type tcp protocol http
      and request "/wiki/Monit" for 5 cycles then restart
    if totalmem > 1000 MB for 5 cycles then restart
    if 6 restarts within 15 cycles then timeout

The wrapper script is chmod +x, just like the actual farm script, /home/alex/farm/farm.

#!/bin/bash

1. This wrapper sets up the Perlbrew environment we need to restart the
1. farm from Monit.
source /home/alex/perl5/perlbrew/etc/bashrc

1. Setup local::lib
eval "$(perl -I/home/alex/perl5/lib/perl5 -Mlocal::lib)"

1. Run farm, depending on its shebang
/home/alex/farm/farm $*

And the actual farm script:

#!/home/alex/perl5/perlbrew/perls/perl-5.24.0/bin/perl
use Toadfarm -init;

$ENV{PATH} .= ":/usr/local/bin";

...

Tags:

Add Comment

2017-01-21 Monit Summary

How weird. I just restarted Monit and it doesn’t get back up. What happened? It looks a lot like this issue.

In the log:

[CET Jan 21 13:58:14] info     : 'sibirocobombus' Monit started
[CET Jan 21 13:58:14] error    : Cannot initialize SSL server certificate handler -- error:140A90A1:SSL routines:func(169):reason(161)
[CET Jan 21 13:58:14] error    : HTTP server: Could not initialize SSL engine
[CET Jan 21 13:58:14] error    : Monit HTTP server not available

From the command line:

alex@sibirocobombus:~$ sudo monit summary
SSL read timeout error
Error connecting to the monit daemon

But then:

alex@sibirocobombus:~$ monit -V
This is Monit version 5.9
Copyright (C) 2001-2014 Tildeslash Ltd. All Rights Reserved.

Also, this hangs after a screen of output:

alex@sibirocobombus:~$ openssl s_client -debug -connect 127.0.0.1:2812
CONNECTED(00000003)
write to 0x73d350 [0x73ded0] (289 bytes => 289 (0x121))
...
0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01   ................
0120 - 01                                                .
^C

Uhm... Wait. I think I confused two directories. In fact what happened was that I moved the port my SSH server is listening on from the default (22) to a different one. But I had forgotten to fix my monit config file. And when I tried to fix the config file, I changed a config file in /etc/monit/monitrc.d/ instead of the real config file in /etc/monit/conf.d/.

But it’s fixed, now (with a security-by-obscurity new port):

check process sshd with pidfile /var/run/sshd.pid
  group system
  group sshd
  start program = "/etc/init.d/ssh start"
  stop  program = "/etc/init.d/ssh stop"
  if failed host localhost port 12345 with proto ssh then restart
  if 5 restarts with 5 cycles then timeout
  depend on sshd_bin
  depend on sftp_bin
  depend on sshd_rc
  depend on sshd_rsa_key
  depend on sshd_dsa_key

...

Restarting munin now works as intended. :)

Tags:

Add Comment

2016-05-31 Monit strangeness

I’m using Toadfarm on two machines. I can start the farm using my own username, and monitor it using Monit, but I can’t start or stop the farm using Monit – even though I use “as uid” and “as gid”:

check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/home/kensanata/farm/farm start" as uid "kensanata" and gid "kensanata"
    stop program = "/home/kensanata/farm/farm stop"
    ...

Weird.

At first, I thought that this actually worked on the older machine, but as it turns out, it never worked. I wonder why.

I rewrote it as follows:

check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/bin/bash -c '/home/kensanata/farm/farm start &> /home/kensanata/farm/start.log'" as uid kensanata
    stop program = "/bin/bash -c '/home/kensanata/farm/farm stop &> /home/kensanata/farm/stop.log'" as uid kensanata
    ...

The log file revealed that the hypnotoad executable was not on the path. So I added the following to my farm file:

$ENV{PATH} .= ":/usr/local/bin";

Now it will start – but not serve! I’m assuming this has to do with permissions somehow?

I think I’m giving up...

kensanata@po6:~/farm$ cat start.log
Server available at http://www.emacswiki.org:8081
farm (17832) started.

Testing:

alex@Megabombus:~$ curl http://www.emacswiki.org:8081/wiki
curl: (52) Empty reply from server

It takes a bit longer than 15s... Looks like an Oddmuse timeout?

I give up. I even added a call to id:

kensanata@po6:~/farm$ cat start.log
uid=1001(kensanata) gid=1001(kensanata) groups=1001(kensanata),0(root)
Server available at http://www.emacswiki.org:8081
farm (18294) started.

It looks good, but it doesn’t work.

I tried to make sure that I’m not using any relative paths. Log file, script, everything is getting absolute paths. And I’ve made some progress. I’m finally seeing some Mojolicious logs:

kensanata@po6:~/farm$ cat farm.log 
[Wed Jun  1 00:14:25 2016] [info] Mounting emacs with conditions
[Wed Jun  1 00:14:25 2016] [info] Loading plugin Toadfarm::Plugin::AccessLog
[Wed Jun  1 00:14:26 2016] [info] Mounting emacs with conditions
[Wed Jun  1 00:14:26 2016] [info] Loading plugin Toadfarm::Plugin::AccessLog
[Wed Jun  1 00:14:26 2016] [info] Listening at "http://www.emacswiki.org:8081"
[Wed Jun  1 00:14:26 2016] [info] Manager 19291 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19292 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19293 started
[Wed Jun  1 00:14:26 2016] [info] Creating process id file "/home/kensanata/farm/farm.pid"
[Wed Jun  1 00:14:26 2016] [debug] Worker 19295 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19294 started
[Wed Jun  1 00:14:36 2016] [debug] GET "/wiki"
[Wed Jun  1 00:14:36 2016] [debug] Routing to application "Mojolicious::Lite"
[Wed Jun  1 00:14:36 2016] [debug] Routing to a callback
[Wed Jun  1 00:14:36 2016] [debug] [CGI:CODE(0x2740360):19299] START /home/kensanata/farm/wiki.pl
[Wed Jun  1 00:14:36 2016] [error] STDERR: : Permission denied at /usr/local/share/perl/5.18.2/Mojolicious/Plugin/CGI.pm line 62.
...

I wonder what this is.

The code in question:

 open STDERR, $STDERR[0], $STDERR[1] or die "STDERR: @$stderr: $!";

Ah! There was an error logfile with not absolute path... Good, fixed, works!

So now, this is how Monit starts the Toadfarm:

kensanata@po6:~/farm$ sudo cat /etc/monit/conf.d/toadfarm.conf
check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/home/kensanata/farm/farm start" as uid www-data and gid www-data
    stop program = "/home/kensanata/farm/farm stop" as uid www-data and gid www-data
    if failed host www.emacswiki.org port 8081 type tcp protocol http
      and request "/wiki" for 5 cycles then restart
    if totalmem > 500 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then timeout

This defines the Toadfarm:

kensanata@po6:~/farm$ cat farm
#! /usr/bin/perl
use Toadfarm -init;

$ENV{PATH} .= ":/usr/local/bin";

logging {
  combined => 1,
  file     => "/home/kensanata/farm/farm.log",
  level    => "debug",
};

my $farm = '/home/kensanata/farm';

mount "$farm/emacs.pl" => {
  "Host" => qr{www.emacswiki.org:8081$},
  mount_point => '/wiki',
};

plugin "Toadfarm::Plugin::AccessLog";

start ["http://www.emacswiki.org:8081"], {proxy => 1}; # needs to be at the last line

(Must change “debug” back to “warn”!)

The wiki itself:

kensanata@po6:~/farm$ cat emacs.pl
#! /usr/bin/env perl
use Mojolicious::Lite;

plugin CGI => {
  support_semicolon_in_query_string => 1,
};

plugin CGI => {
  route => '/',
  script => '/home/kensanata/farm/wiki.pl',
  run => \&OddMuse::DoWikiRequest,
  before => sub {
      my $c = shift;
      $OddMuse::RunCGI = 0;
      $OddMuse::DataDir = '/home/kensanata/emacswiki';
      require '/home/kensanata/farm/wiki.pl' unless defined &OddMuse::DoWikiRequest;
  },
  env => {},
  errlog => '/home/kensanata/farm/emacs.log', # path to where STDERR from cgi script goes
};

app->start;

Note all the absolute paths. :)

This server is now also running a Toadfarm started by Monit. Yay!

Don’t forget permissions: key files and directories need to belong to www-data.kensanata (my user’s group so that I can also edit these files).

  1. the farm directory needs to be writable by www-data so that the pid file can be written; without it, Monit can start the toadfarm but will still claim it failed (pid file is missing) and stopping it will also fail (use ps fax to figure out which process to sudo kill)
  2. the farm log needs to be writable by www-data so that the toadfarm can log
  3. the application log file needs to be writable by www-data so that the wiki itself can log

To wit:

kensanata@po6:~$ ls -la farm
total 1368
drwxrwxr-x  2 www-data  kensanata    4096 Jun  2 17:00 .
drwxr-xr-x 12 kensanata kensanata    4096 Jun  1 00:24 ..
-rw-r--r--  1 www-data  kensanata   27935 Jun  2 16:34 emacs.log
-rw-rw-r--  1 kensanata kensanata     549 Jun  1 00:22 emacs.pl
-rw-rw-r--  1 kensanata kensanata     528 Jun  1 00:12 emacs.pl~
-rwxrwxr-x  1 kensanata kensanata     432 Jun  2 16:32 farm
-rwxrwxr-x  1 kensanata kensanata     433 Jun  1 00:13 farm~
-rw-r--r--  1 www-data  kensanata 1002606 Jun  2 16:34 farm.log
-rw-r--r--  1 www-data  www-data        5 Jun  2 17:00 farm.pid
-rw-rw-r--  1 kensanata kensanata     322 May 29 12:01 Makefile
-rw-rw-r--  1 kensanata kensanata     384 May 29 11:59 Makefile~
-rwxrwxr-x  1 kensanata kensanata  159959 Jun  1 00:10 wiki.pl
-rwxrwxr-x  1 kensanata kensanata  159988 Jun  1 00:07 wiki.pl~

And why is that? Because I want the process to have few permissions elsewhere, and the existing wiki is being run as a CGI script by Apache using www-data which is why the data directory has the same ownership – owned by user www-data and writable by a group I belong to.

Tags:

Add Comment

2014-12-20 Monit

Some monit configurations for my server, from /etc/monit/conf.d:

cron

check process crond with pidfile /var/run/crond.pid
   group system
   group crond
   start program = "/etc/init.d/cron start"
   stop  program = "/etc/init.d/cron stop"
   if 5 restarts with 5 cycles then timeout
   depend cron_bin
   depend cron_rc
   depend cron_spool

 check file cron_bin with path /usr/sbin/cron
   group crond
   include /etc/monit/templates/rootbin

 check file cron_rc with path "/etc/init.d/cron"
   group crond
   include /etc/monit/templates/rootbin

 check directory cron_spool with path /var/spool/cron/crontabs
   group crond
   if failed permission 1730 then unmonitor
   if failed uid root        then unmonitor
   if failed gid crontab     then unmonitor

fail2ban

check process fail2ban with pidfile /var/run/fail2ban/fail2ban.pid
   group monitoring
   start program = "/etc/init.d/fail2ban start"
   stop  program = "/etc/init.d/fail2ban stop"
   if failed unixsocket /var/run/fail2ban/fail2ban.sock then restart
   if 5 restarts within 5 cycles then timeout
   depends on fail2ban_bin
   depends on fail2ban_rc

check file fail2ban_bin with path /usr/bin/fail2ban-client
   group monitoring
   include /etc/monit/templates/rootbin

check file fail2ban_rc with path /etc/init.d/fail2ban
   group monitoring
   include /etc/monit/templates/rootbin

munin

check process munin-node with pidfile /var/run/munin/munin-node.pid
   group monitoring
   start program = "/etc/init.d/munin-node start"
   stop  program = "/etc/init.d/munin-node stop"
   if 5 restarts within 5 cycles then timeout
   depends on munin-node_bin
   depends on munin-node_rc

check file munin-node_bin with path /usr/sbin/munin-node
   group monitoring
   include /etc/monit/templates/rootbin

check file munin-node_rc with path /etc/init.d/munin-node
   group monitoring
   include /etc/monit/templates/rootbin

openssh-server

 check process sshd with pidfile /var/run/sshd.pid
   group system
   group sshd
   start program = "/etc/init.d/ssh start"
   stop  program = "/etc/init.d/ssh stop"
   if failed host localhost port 22 with proto ssh then restart
   if 5 restarts with 5 cycles then timeout
   depend on sshd_bin
   depend on sftp_bin
   depend on sshd_rc
   depend on sshd_rsa_key
   depend on sshd_dsa_key

 check file sshd_bin with path /usr/sbin/sshd
   group sshd
   include /etc/monit/templates/rootbin

 check file sftp_bin with path /usr/lib/openssh/sftp-server
   group sshd
   include /etc/monit/templates/rootbin

 check file sshd_rsa_key with path /etc/ssh/ssh_host_rsa_key
   group sshd
   include /etc/monit/templates/rootstrict

 check file sshd_dsa_key with path /etc/ssh/ssh_host_dsa_key
   group sshd
   include /etc/monit/templates/rootstrict

 check file sshd_rc with path /etc/ssh/sshd_config
   group sshd
   include /etc/monit/templates/rootrc

rsyslog

check process rsyslog with pidfile /var/run/rsyslogd.pid
   group monitoring
   start program = "/etc/init.d/rsyslog start"
   stop  program = "/etc/init.d/rsyslog stop"
   if 5 restarts within 5 cycles then timeout
   depends on rsyslog_bin
   depends on rsyslog_rc

check file rsyslog_bin with path /usr/sbin/rsyslogd
   group monitoring
   include /etc/monit/templates/rootbin

check file rsyslog_rc with path /etc/init.d/rsyslog
   group monitoring
   include /etc/monit/templates/rootbin

sendmail

check process sendmail with pidfile /var/run/sendmail/mta/sendmail.pid
   group mail
   start program = "/etc/init.d/sendmail start"
   stop  program = "/etc/init.d/sendmail stop"
1.  if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout
   depends on sendmail_bin
   depends on sendmail_rc

1. /usr/lib/sendmail points to /etc/alternatives/lib.sendmail which in
1. turn points to the file tested:

check file sendmail_bin with path /usr/lib/sm.bin/sendmail
   group mail
   if failed checksum        then unmonitor
   if failed permission 2755 then unmonitor
   if failed uid root        then unmonitor
   if failed gid smmsp       then unmonitor

check file sendmail_rc with path /etc/init.d/sendmail
   group mail
   include /etc/monit/templates/rootbin

System

There’s some stuff in /etc/monit/monitrc which I adapted from the defaults (the admin password and all that).

I changed the defaults for the system check:

check system Kallobombus
  if loadavg (5min) > 4 then alert
1. if loadavg (10min) > 2 then alert
1. if loadavg (15min) > 1 then alert
  if memory usage > 75% then alert
1. if swap usage > 25% then alert
1. if cpu usage (user) > 98% then alert
  if cpu usage (user) > 98% for 5 cycles then alert
  if cpu usage (system) > 50% then alert
  if cpu usage (wait) > 20% then alert

Apache

This is also changed from the defaults in /etc/monit/monitrc.

check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host alexschroeder.ch port 80 protocol http
      and request "/monit/token" then restart
1.  if cpu is greater than 60% for 2 cycles then alert
1.  if cpu > 80% for 5 cycles then restart
1.  if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
1.  if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

Rsync

Needed to create the file /etc/rsyncd.conf in order to actually share something. And I needed to set RSYNC_ENABLE=true in /etc/default/rsync in order to have rsync start up as a demon. And sudo service rsync start, of course.

For monit, however:

 check process rsyncd with pidfile /var/run/rsync.pid
   start program = "/etc/init.d/rsync start"
   stop  program = "/etc/init.d/rsync stop"
   if 5 restarts with 5 cycles then timeout
   depend rsync_bin
   depend rsync_rc
   depend rsync_conf

 check file rsync_bin with path /usr/bin/rsync
   include /etc/monit/templates/rootbin

 check file rsync_rc with path "/etc/init.d/rsync"
   include /etc/monit/templates/rootbin

 check file rsync_conf with path "/etc/rsyncd.conf"
   include /etc/monit/templates/rootrc

If you have nothing to share, disable it in /etc/default/rsync and stop monitoring it. Works for me!

Tags:

Add Comment

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.