2018-09-05 Toadfarm and Monit

Every now and then I wake up to a website that’s dead. When I try to restart the Toadfarm using Monit, nothing happens. I connect to the box using ssh and try to restart it from the command line:

$ ./farm reload
 * Reloading the process farm
Can't create listen socket: Address already in use at /home/alex/perl5/perlbrew/perls/perl-5.26.1/lib/site_perl/5.26.1/Mojo/IOLoop.pm line 126.
   ...fail!

So the farm is up and listening to the port but it’s not actually doing anything. Now what? They’re all asleep!

top - 07:24:55 up 150 days, 15:34,  2 users,  load average: 0.00, 0.03, 0.00
Tasks: 124 total,   2 running, 122 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.7 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.3 st
KiB Mem :  3084768 total,   336608 free,   737820 used,  2010340 buff/cache
KiB Swap:  1022972 total,       16 free,  1022956 used.  2119480 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                     
16448 alex      20   0  331468 125888   3760 S  0.0  4.1   0:00.33 /home/alex/farm             
16563 alex      20   0  378652 104660   3668 S  0.0  3.4   0:00.38 /home/alex/farm             
 6681 alex      20   0  358828  92952   3940 S  0.0  3.0   0:00.89 /home/alex/farm             
  709 alex      20   0  347792  84888   5484 S  0.0  2.8   0:01.06 /home/alex/farm             
 6730 alex      20   0  342380  74416   3940 S  0.0  2.4   0:01.13 /home/alex/farm             
  710 alex      20   0  335376  65868   5424 S  0.0  2.1   0:00.99 /home/alex/farm             
22714 alex      30  10  118320  45080   7772 S  0.0  1.5   0:00.53 perl                        
18640 alex      20   0   85272  25896   6612 S  0.0  0.8   0:00.43 perl                        
 9579 alex      20   0  343312  25568   3848 S  0.0  0.8   0:00.45 /home/alex/farm             
13534 alex      20   0  331080  25548   3848 S  0.0  0.8   0:00.09 /home/alex/farm             
 8731 alex      20   0  362272  25520   3848 S  0.0  0.8   0:00.10 /home/alex/farm             
10399 alex      20   0  330736  25500   3812 S  0.0  0.8   0:00.35 /home/alex/farm             
31772 alex      20   0  327720  25468   3848 S  0.0  0.8   0:01.46 /home/alex/farm             
 1931 alex      20   0  328808  25396   3848 S  0.0  0.8   0:00.19 /home/alex/farm             
  974 alex      20   0  329248  25380   3848 S  0.0  0.8   0:00.27 /home/alex/farm             

And so I’m doing what I usually do, without actually solving the underlying problem:

for pid in $(ps -u alex | grep /farm | cut -c 1-5); do kill $pid; done

Do you have a better idea? What’s going on, here?

Tags:

Comments

Gah. Had to kill it all, again. Why‽

I fixed my kill-farm script:

#! /bin/bash
for pid in $(ps -xo pid,command|perl -ne '@s=split; print $s[0] . "\n" if $s[1] eq "/home/alex/farm/farm"'); do
    kill -9 $pid
done

From /var/log/syslog:

Linux sibirocobombus 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Oct  1 19:40:53 2018 from 2a02:168:4823:0:9c0b:e651:b5d7:2f4f
root@sibirocobombus:~# cat /tmp/log
Oct  1 16:51:54 sibirocobombus kernel: [15296489.567141] /home/alex/farm: page allocation stalls for 10472ms, order:0, mode:0x24200ca(GFP_HIGHUSER_MOVABLE)
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567150] CPU: 0 PID: 13804 Comm: /home/alex/farm Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u5
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567153]  0000000000000000 ffffffffb9f285b4 ffffffffba5febb0 ffffb9ccc10cbb68
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567156]  ffffffffb9d84f3a 024200ca00000002 ffffffffba5febb0 ffffb9ccc10cbb08
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567159]  ffff8f1a00000010 ffffb9ccc10cbb78 ffffb9ccc10cbb28 98dccf6c149eda9d
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567161] Call Trace:
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567168]  [<ffffffffb9f285b4>] ? dump_stack+0x5c/0x78
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567171]  [<ffffffffb9d84f3a>] ? warn_alloc+0x13a/0x160
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567173]  [<ffffffffb9d8592d>] ? __alloc_pages_slowpath+0x95d/0xbc0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567175]  [<ffffffffb9d85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567177]  [<ffffffffb9dd7c3e>] ? alloc_pages_vma+0xae/0x260
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567181]  [<ffffffffb9daf069>] ? wp_page_copy+0x89/0x700
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567183]  [<ffffffffb9db0361>] ? do_wp_page+0x161/0x7d0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567186]  [<ffffffffb9db3170>] ? handle_mm_fault+0x8d0/0x1350
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567188]  [<ffffffffb9c24701>] ? __switch_to+0x2c1/0x6c0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567191]  [<ffffffffb9c5fd84>] ? __do_page_fault+0x2a4/0x510
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567194]  [<ffffffffba207788>] ? async_page_fault+0x28/0x30
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567195] Mem-Info:
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199] active_anon:563083 inactive_anon:141043 isolated_anon:813
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  active_file:3237 inactive_file:1620 isolated_file:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  unevictable:0 dirty:21 writeback:7962 unstable:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  slab_reclaimable:8695 slab_unreclaimable:14080
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  mapped:5449 shmem:8956 pagetables:15088 bounce:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567199]  free:14229 free_pcp:0 free_cma:0
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567209] Node 0 active_anon:2252332kB inactive_anon:564172kB active_file:12948kB inactive_file:6480kB unevictable:0kB isolated(anon):3252kB isolated(file):0kB mapped:21796kB dirty:84kB writeback:31848kB shmem:35824kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:2 all_unreclaimable? no
Oct  1 16:51:55 sibirocobombus kernel: [15296489.567210] Node 0 DMA free:12124kB min:232kB low:288kB high:344kB active_anon:3492kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:12kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:184kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567214] lowmem_reserve[]: 0 2975 2975 2975 2975
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567217] Node 0 DMA32 free:44792kB min:44820kB low:56024kB high:67228kB active_anon:2248840kB inactive_anon:564172kB active_file:12948kB inactive_file:6480kB unevictable:0kB writepending:31932kB present:3129216kB managed:3068860kB mlocked:0kB slab_reclaimable:34768kB slab_unreclaimable:56232kB kernel_stack:5392kB pagetables:60168kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567221] lowmem_reserve[]: 0 0 0 0 0
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567224] Node 0 DMA: 5*4kB (MEH) 5*8kB (UME) 4*16kB (ME) 5*32kB (UEH) 9*64kB (UMEH) 6*128kB (UEH) 5*256kB (UEH) 2*512kB (ME) 2*1024kB (UH) 3*2048kB (UME) 0*4096kB = 12124kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567236] Node 0 DMA32: 2599*4kB (UMEH) 1538*8kB (UMEH) 672*16kB (UMEH) 265*32kB (UME) 23*64kB (MEH) 1*128kB (H) 1*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB 0*4096kB = 44812kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567248] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567249] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567249] 26825 total pagecache pages
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567251] 13006 pages in swap cache
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567252] Swap cache stats: add 3064172, delete 3051166, find 484392409/484925615
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567253] Free swap  = 147028kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567253] Total swap = 1022972kB
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567254] 786302 pages RAM
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567255] 0 pages HighMem/MovableOnly
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567255] 15110 pages reserved
Oct  1 16:51:56 sibirocobombus kernel: [15296489.567256] 0 pages hwpoisoned
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538179] /home/alex/farm invoked oom-killer: gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE), nodemask=0, order=0, oom_score_adj=0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538181] /home/alex/farm cpuset=/ mems_allowed=0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538186] CPU: 0 PID: 13764 Comm: /home/alex/farm Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u5
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538189]  0000000000000000 ffffffffb9f285b4 ffffb9ccc0e9bc20 ffff8f1a4e8f8100
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538193]  ffffffffb9dfe020 0000000000000000 0000000000000000 0000000000000001
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538195]  ffffffffb9d844e7 0000004252982a80 ffffffffc0454695 0000000000000001
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538197] Call Trace:
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538204]  [<ffffffffb9f285b4>] ? dump_stack+0x5c/0x78
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538206]  [<ffffffffb9dfe020>] ? dump_header+0x78/0x1fd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538209]  [<ffffffffb9d844e7>] ? get_page_from_freelist+0x3f7/0xb40
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538214]  [<ffffffffc0454695>] ? virtballoon_oom_notify+0x25/0x70 [virtio_balloon]
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538217]  [<ffffffffb9d8047a>] ? oom_kill_process+0x21a/0x3e0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538220]  [<ffffffffb9d800fd>] ? oom_badness+0xed/0x170
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538221]  [<ffffffffb9d80911>] ? out_of_memory+0x111/0x470
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538223]  [<ffffffffb9d85b4f>] ? __alloc_pages_slowpath+0xb7f/0xbc0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538225]  [<ffffffffb9d85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538227]  [<ffffffffb9dd7c3e>] ? alloc_pages_vma+0xae/0x260
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538230]  [<ffffffffb9daf069>] ? wp_page_copy+0x89/0x700
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538233]  [<ffffffffb9db0361>] ? do_wp_page+0x161/0x7d0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538235]  [<ffffffffb9db3170>] ? handle_mm_fault+0x8d0/0x1350
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538238]  [<ffffffffb9c24701>] ? __switch_to+0x2c1/0x6c0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538240]  [<ffffffffb9c5fd84>] ? __do_page_fault+0x2a4/0x510
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538243]  [<ffffffffba207788>] ? async_page_fault+0x28/0x30
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538244] Mem-Info:
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248] active_anon:571476 inactive_anon:143154 isolated_anon:532
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  active_file:204 inactive_file:222 isolated_file:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  unevictable:0 dirty:0 writeback:3178 unstable:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  slab_reclaimable:3157 slab_unreclaimable:13740
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  mapped:1215 shmem:7913 pagetables:15023 bounce:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538248]  free:14207 free_pcp:37 free_cma:0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538252] Node 0 active_anon:2285904kB inactive_anon:572616kB active_file:816kB inactive_file:888kB unevictable:0kB isolated(anon):2128kB isolated(file):0kB mapped:4860kB dirty:0kB writeback:12712kB shmem:31652kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:5391 all_unreclaimable? yes
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538252] Node 0 DMA free:12124kB min:232kB low:288kB high:344kB active_anon:3484kB inactive_anon:4kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:12kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:188kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538257] lowmem_reserve[]: 0 2975 2975 2975 2975
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538260] Node 0 DMA32 free:44704kB min:44820kB low:56024kB high:67228kB active_anon:2282420kB inactive_anon:572612kB active_file:816kB inactive_file:888kB unevictable:0kB writepending:12712kB present:3129216kB managed:3068860kB mlocked:0kB slab_reclaimable:12616kB slab_unreclaimable:54872kB kernel_stack:5376kB pagetables:59904kB bounce:0kB free_pcp:148kB local_pcp:148kB free_cma:0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538264] lowmem_reserve[]: 0 0 0 0 0
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538266] Node 0 DMA: 4*4kB (UEH) 2*8kB (E) 2*16kB (E) 5*32kB (UMEH) 8*64kB (UMEH) 5*128kB (UEH) 4*256kB (UEH) 1*512kB (M) 3*1024kB (UEH) 3*2048kB (UME) 0*4096kB = 12128kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538278] Node 0 DMA32: 2108*4kB (UEH) 1402*8kB (UMEH) 746*16kB (UMEH) 286*32kB (UMEH) 40*64kB (MEH) 1*128kB (H) 1*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB 0*4096kB = 44704kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538289] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538291] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538291] 18160 total pagecache pages
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538293] 9813 pages in swap cache
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538294] Swap cache stats: add 3100981, delete 3091168, find 484392412/484925620
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538294] Free swap  = 0kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538295] Total swap = 1022972kB
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538296] 786302 pages RAM
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538296] 0 pages HighMem/MovableOnly
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538297] 15110 pages reserved
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538298] 0 pages hwpoisoned
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538298] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538302] [  188]     0   188    15502      833      27       3       80             0 systemd-journal
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538304] [  376]     0   376    11625       55      26       3      104             0 systemd-logind
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538306] [  381]     0   381     7416       35      20       3       50             0 cron
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538308] [  387]     0   387    62560      163      30       3      351             0 rsyslogd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538309] [  388]     0   388     3152        9      12       3       37             0 rsync
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538311] [  393]     0   393     1054        0       8       3       35             0 acpid
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538312] [  394]   105   394    11283       60      27       3       78          -900 dbus-daemon
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538314] [  450]     0   450     3634        0      12       3       38             0 agetty
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538316] [  458]     0   458    28698      198      26       3       93             0 monit
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538317] [  464]     0   464    17486       41      37       3      164         -1000 sshd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538319] [  614]     0   614    12949      767      30       3     1563             0 munin-node
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538329] [  823]   107   823    27941       23      50       3      365             0 exim4
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538331] [10368]   100 10368    31821       19      32       3      119             0 systemd-timesyn
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538333] [10648]     0 10648    11308       12      24       3       91         -1000 systemd-udevd
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538335] [25971]  1000 25971    27465      378      54       3     6922             0 /home/alex/camp
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538337] [22812]     0 22812    26416      842      58       3       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538338] [ 2794]     0  2794   169401     3422      61       4      455             0 fail2ban-server
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538340] [29327]    33 29327   216079     2688     100       4      530             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538342] [23538]  1000 23538    81856      167     143       3    29692             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538343] [14140]  1000 14140    81929      249     144       3    29760             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538345] [15321]  1000 15321    85002      249     149       3    31614             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538346] [15742]  1000 15742    82686      247     144       3    29465             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538348] [15822]  1000 15822    82672      253     144       3    29467             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538349] [18583]  1000 18583    85088      250     149       3    31258             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538351] [16180]  1000 16180    83100      164     148       3    30518             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538353] [16240]  1000 16240    84600      170     151       3    31838             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538354] [17135]  1000 17135    84455      169     150       3    31723             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538356] [18081]  1000 18081    83276      169     147       3    29866             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538357] [18089]  1000 18089    82757      167     146       3    29665             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538359] [18097]  1000 18097    83000      167     146       3    29722             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538360] [28873]  1000 28873    83946      167     148       3    30644             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538362] [29477]  1000 29477    84020      168     148       3    30825             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538364] [17703]  1000 17703    85534    10436     152       3    22151             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538366] [25630]  1000 25630    84044    17581     147       3    13484             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538367] [23799]  1000 23799    83909    16024     149       4    15467             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538369] [23831]  1000 23831    88924    19813     159       4    16552             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538370] [17148]  1000 17148    27465     3262      52       3     4112             0 /home/alex/camp
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538372] [24680] 65534 24680    12608     2506      27       3        0             0 python
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538374] [ 6121]  1000  6121    84608    32664     151       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538376] [ 9051]    33  9051    26290      742      54       3       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538377] [ 9372]  1000  9372    79677    26765     141       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538379] [ 9393]  1000  9393    88087    36094     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538380] [ 9395]  1000  9395   104793    53630     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538382] [ 9396]  1000  9396    94890    43125     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538392] [ 9418]  1000  9418    21312     4988      45       3        0             0 perl
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538394] [ 9989]  1000  9989    17541     3956      37       3        0             0 perl
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538395] [11782]    33 11782   215922     2843      99       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538397] [11833]    33 11833   214475     2459      97       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538398] [26741]  1000 26741    92116    38897     165       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538400] [17934]  1000 17934    87620    35407     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538402] [18489]  1000 18489    85855    32863     152       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538403] [19093]    33 19093   215694     2608      99       4       45             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538405] [13755]  1000 13755    94098    41334     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538407] [13758]  1000 13758   106413    54375     195       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538408] [13759]  1000 13759    94098    41361     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538410] [13763]  1000 13763    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538411] [13764]  1000 13764    89787    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538413] [13765]  1000 13765    93158    38962     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538414] [13766]  1000 13766    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538415] [13767]  1000 13767    94098    41334     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538417] [13768]  1000 13768   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538419] [13769]  1000 13769    93158    38962     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538420] [13771]  1000 13771    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538422] [13772]  1000 13772   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538423] [13773]  1000 13773    93158    38961     166       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538425] [13774]  1000 13774    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538427] [13775]    33 13775   214080      930      95       4       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538428] [13804]  1000 13804    94098    41501     170       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538430] [13805]  1000 13805    89786    36814     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538431] [13806]  1000 13806    95932    43127     174       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538433] [13807]  1000 13807   106413    54423     193       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538435] [13808]  1000 13808    92634    38961     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538436] [13809]  1000 13809    89786    36815     160       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538438] [13810]  1000 13810    95932    43127     174       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538439] [13812]  1000 13812    92634    38961     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538441] [13813]  1000 13813   105709    54206     190       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538442] [13814]  1000 13814    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538444] [13815]    33 13815   213919      868      95       4       46             0 apache2
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538446] [13817]  1000 13817    89164    36681     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538447] [13844]  1000 13844    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538449] [13845]  1000 13845   105609    54032     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538450] [13846]  1000 13846    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538452] [13847]  1000 13847   105789    54335     190       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538453] [13848]  1000 13848    89263    36789     158       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538455] [13849]  1000 13849    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538456] [13850]  1000 13850    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538458] [13851]  1000 13851    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538459] [13852]  1000 13852   105311    53674     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538461] [13853]  1000 13853    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538462] [13854]  1000 13854    92634    38940     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538463] [13855]  1000 13855    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538465] [13856]  1000 13856   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538466] [13857]  1000 13857    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538468] [13858]  1000 13858    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538469] [13859]  1000 13859    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538471] [13860]  1000 13860   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538472] [13861]  1000 13861    92634    38898     164       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538474] [13862]  1000 13862    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538475] [13863]  1000 13863    88605    36095     157       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538477] [13864]  1000 13864   105311    53737     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538478] [13865]  1000 13865    92116    38883     162       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538480] [13866]  1000 13866    95408    43126     172       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538481] [13867]  1000 13867    88087    36080     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538483] [13868]  1000 13868   105311    53631     189       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538484] [13869]  1000 13869    94890    43125     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538486] [13870]  1000 13870    92116    38883     162       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538488] [13871]  1000 13871    88087    36080     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538489] [13872]  1000 13872   104793    53616     188       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538491] [13873]  1000 13873    94890    43111     171       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538492] [13875]  1000 13875    88087    36070     155       3        0             0 /home/alex/farm
Oct  1 16:51:56 sibirocobombus kernel: [15296512.538494] Out of memory: Kill process 13768 (/home/alex/farm) score 53 or sacrifice child
Oct  1 16:51:56 sibirocobombus kernel: [15296512.867096] Killed process 13768 (/home/alex/farm) total-vm:425652kB, anon-rss:217024kB, file-rss:668kB, shmem-rss:0kB

– Alex Schroeder 2018-10-01 17:02 UTC


OK, I wrote myself a cull-farm script:

#! /home/alex/perl5/perlbrew/perls/perl-5.26.1/bin/perl
use Modern::Perl;
my %data = map { $_->[0] => $_->[1] }
           grep { $_->[2] eq "/home/alex/farm/farm" and $_->[3] ne "1" }
           map { [ split ] }
           (qx(ps -xo pid,etimes,command,ppid));
my @ids = sort { $data{$a} <=> $data{$b} } keys %data;
# say "pids: @ids";
my @old_ids = @ids[4 .. $#ids];
if (@old_ids > 0) {
  say "SIGTERM for @old_ids";
  kill 'TERM', @old_ids;
} else {
  say "The only child processes we have are @ids";
}

It runs a ps which prints the process id, the elapsed time in seconds, the command name (so that I can filter for my processes), and the parent process id. I don’t want to kill the top level process that keeps spawning the children!

– Alex Schroeder 2018-10-07 17:17 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.