2018-06-08 Out of Memory

I just noticed the following in my Munin:

Image 1

What had happened between 11:25 and 11:35? I looked at /var/log/syslog and found the following:

Jun  8 11:30:25 sibirocobombus kernel: [5341224.320404] python3 invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320407] python3 cpuset=/ mems_allowed=0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320414] CPU: 0 PID: 6857 Comm: python3 Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u5
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320418]  0000000000000000 ffffffffb9f285b4 ffffb9ccc195fc38 ffff8f1a486d1080
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320421]  ffffffffb9dfe020 0000000000000000 0000000000000000 0000000000000001
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320424]  ffffffffb9d844e7 0000004294fa8a80 ffffffffc0454695 0000000000000001
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320427] Call Trace:
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320436]  [<ffffffffb9f285b4>] ? dump_stack+0x5c/0x78
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320439]  [<ffffffffb9dfe020>] ? dump_header+0x78/0x1fd
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320443]  [<ffffffffb9d844e7>] ? get_page_from_freelist+0x3f7/0xb40
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320449]  [<ffffffffc0454695>] ? virtballoon_oom_notify+0x25/0x70 [virtio_balloon]
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320453]  [<ffffffffb9d8047a>] ? oom_kill_process+0x21a/0x3e0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320455]  [<ffffffffb9d800fd>] ? oom_badness+0xed/0x170
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320456]  [<ffffffffb9d80911>] ? out_of_memory+0x111/0x470
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320458]  [<ffffffffb9d85b4f>] ? __alloc_pages_slowpath+0xb7f/0xbc0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320460]  [<ffffffffb9d85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320465]  [<ffffffffb9dd5ec1>] ? alloc_pages_current+0x91/0x140
Jun  8 11:30:26 sibirocobombus kernel: [5341224.320467]  [<ffffffffb9d7e940>] ? filemap_fault+0x300/0x5a0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.324016]  [<ffffffffc0383791>] ? ext4_filemap_fault+0x31/0x50 [ext4]
Jun  8 11:30:26 sibirocobombus kernel: [5341224.324016]  [<ffffffffb9daeef1>] ? __do_fault+0x81/0x170
Jun  8 11:30:26 sibirocobombus kernel: [5341224.324016]  [<ffffffffb9db3671>] ? handle_mm_fault+0xdd1/0x1350
Jun  8 11:30:26 sibirocobombus kernel: [5341224.324016]  [<ffffffffb9c24701>] ? __switch_to+0x2c1/0x6c0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.324016]  [<ffffffffb9c5fd84>] ? __do_page_fault+0x2a4/0x510
Jun  8 11:30:26 sibirocobombus kernel: [5341224.324016]  [<ffffffffba207788>] ? async_page_fault+0x28/0x30
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402865] Mem-Info:
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402875] active_anon:573808 inactive_anon:143778 isolated_anon:0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402875]  active_file:186 inactive_file:96 isolated_file:37
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402875]  unevictable:0 dirty:0 writeback:0 unstable:0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402875]  slab_reclaimable:3009 slab_unreclaimable:11993
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402875]  mapped:1216 shmem:8885 pagetables:12225 bounce:0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402875]  free:14221 free_pcp:152 free_cma:0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402883] Node 0 active_anon:2295232kB inactive_anon:575112kB active_file:744kB inactive_file:384kB unevictable:0kB isolated(anon):0kB isolated(file):148kB mapped:4864kB dirty:0kB writeback:0kB shmem:35540kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:2335 all_unreclaimable? yes
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402884] Node 0 DMA free:12124kB min:232kB low:288kB high:344kB active_anon:3688kB inactive_anon:68kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402890] lowmem_reserve[]: 0 2975 2975 2975 2975
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402894] Node 0 DMA32 free:44760kB min:44820kB low:56024kB high:67228kB active_anon:2291544kB inactive_anon:575044kB active_file:744kB inactive_file:384kB unevictable:0kB writepending:0kB present:3129216kB managed:3068860kB mlocked:0kB slab_reclaimable:12036kB slab_unreclaimable:47960kB kernel_stack:8352kB pagetables:48896kB bounce:0kB free_pcp:608kB local_pcp:608kB free_cma:0kB
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402899] lowmem_reserve[]: 0 0 0 0 0
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402902] Node 0 DMA: 3*4kB (UH) 4*8kB (UMH) 1*16kB (H) 3*32kB (UH) 5*64kB (UMH) 3*128kB (UH) 2*256kB (UM) 1*512kB (U) 2*1024kB (UH) 2*2048kB (UM) 1*4096kB (E) = 12124kB
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402915] Node 0 DMA32: 6682*4kB (UMEH) 2054*8kB (UEH) 4*16kB (H) 2*32kB (H) 1*64kB (H) 1*128kB (H) 1*256kB (H) 2*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 44760kB
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402927] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402928] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402929] 14918 total pagecache pages
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402931] 5715 pages in swap cache
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402932] Swap cache stats: add 1066964, delete 1061249, find 206625603/206789816
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402933] Free swap  = 0kB
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402934] Total swap = 1022972kB
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402934] 786302 pages RAM
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402935] 0 pages HighMem/MovableOnly
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402936] 15110 pages reserved
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402937] 0 pages hwpoisoned
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402938] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402954] [  188]     0   188    15502      780      27       3       77             0 systemd-journal
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402958] [  376]     0   376    11625       55      26       3       86             0 systemd-logind
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402961] [  381]     0   381     7416       47      20       3       28             0 cron
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402963] [  387]     0   387    62529      478      30       3       77             0 rsyslogd
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402966] [  388]     0   388     3152        9      12       3       37             0 rsync
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402969] [  393]     0   393     1054        0       8       3       35             0 acpid
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402972] [  394]   105   394    11283       60      27       3       78          -900 dbus-daemon
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402974] [  450]     0   450     3634        0      12       3       38             0 agetty
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402976] [  458]     0   458    28698      227      26       3       59             0 monit
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402979] [  464]     0   464    17486       31      37       3      164         -1000 sshd
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402982] [  614]     0   614    12949      927      30       3     1385             0 munin-node
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402984] [  823]   107   823    27941       18      50       3      365             0 exim4
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402987] [  826]     0   826    73633     1101      44       3     1522             0 fail2ban-server
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402990] [ 1059]  1000  1059    27474      376      55       3     6919             0 /home/alex/camp
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402994] [22985]  1000 22985    74242        2     130       3    23696             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.402997] [23038]  1000 23038    72562        1     126       3    22595             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403000] [22174]  1000 22174    75475        1     135       3    25314             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403003] [10368]   100 10368    31821       17      32       3      116             0 systemd-timesyn
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403005] [10648]     0 10648    11308       11      24       3       92         -1000 systemd-udevd
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403008] [12215]     0 12215    26437      794      56       3       47             0 apache2
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403011] [15822]  1000 15822    77511        1     135       3    26522             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403014] [ 5025]  1000  5025    76563     2636     134       4    22902             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403017] [ 5955]  1000  5955    79129     2718     139       4    25356             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403020] [ 6554]  1000  6554    87614     3839     156       4    32240             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403022] [ 6845]  1000  6845    87597     7559     156       4    28512             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403026] [ 6930]  1000  6930    81413     6399     144       4    23554             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403029] [ 7143]  1000  7143    87659     7600     156       4    28482             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403031] [ 7564]  1000  7564    75261     1351     132       4    22884             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403034] [ 7610]  1000  7610    78205     4910     138       4    21774             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403037] [ 8548]  1000  8548    73994      624     129       3    23285             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403039] [ 8960]  1000  8960    74356      658     131       3    22646             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403043] [ 9075]  1000  9075    72920      686     127       3    22198             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403045] [ 9174]  1000  9174    72909      635     127       3    22213             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403048] [10473]  1000 10473    73076    11977     126       3    11083             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403052] [10295]  1000 10295    76489    21414     134       4     4042             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403054] [10296]  1000 10296    78132    24544     137       4     2560             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403057] [10300]  1000 10300    78133    24292     138       4     2812             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403060] [10436]  1000 10436    76499    20586     134       4     4887             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403062] [18588]  1000 18588    75233    24172     129       3        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403065] [14367]  1000 14367    96991    45941     170       3        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403067] [ 3040]  1000  3040    27474     3870      54       3     3562             0 /home/alex/camp
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403069] [ 4378]  1000  4378    69698    18695     125       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403072] [15022] 65534 15022    12598     2534      27       4        0             0 python
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403074] [16776]  1000 16776    76163    25107     136       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403076] [13061]  1000 13061    79048    28907     143       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403079] [ 2928]  1000  2928    72276    22294     123       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403082] [ 3415]    33  3415    26311      733      53       3       47             0 apache2
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403084] [ 3526]    33  3526   216362     3392     100       4       46             0 apache2
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403086] [ 3661]  1000  3661    20782     4802      43       3        0             0 perl
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403089] [ 4308]  1000  4308    17017     3773      37       3        0             0 perl
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403091] [23738]  1000 23738    72298    22266     125       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403094] [ 5556]    33  5556   214419     1979      95       4       46             0 apache2
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403096] [ 5588]    33  5588   214589     2148      95       4       46             0 apache2
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403099] [ 6431]    33  6431    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403101] [ 6432]    33  6432    17320     5307      40       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403103] [ 6437]    33  6437    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403105] [ 6438]    33  6438    17090     5076      38       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403107] [ 6462]    33  6462    35599      106      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403110] [ 6464]    33  6464   214782     1522      95       4       47             0 apache2
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403112] [ 6493]    33  6493    12016     4372      30       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403114] [ 6495]    33  6495    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403117] [ 6500]    33  6500    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403119] [ 6507]    33  6507    12016     4378      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403121] [ 6508]    33  6508    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403123] [ 6509]    33  6509    35599      106      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403126] [ 6513]    33  6513    12016     4371      29       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403128] [ 6517]    33  6517   215340     2016      97       4       46             0 apache2
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403131] [ 6521]    33  6521    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403133] [ 6524]    33  6524    11994     4306      29       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403135] [ 6555]    33  6555    12016     4339      29       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403137] [ 6556]    33  6556    35599      106      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403139] [ 6562]    33  6562    35599      106      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403142] [ 6568]    33  6568    12019     4338      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403144] [ 6570]    33  6570    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403146] [ 6574]    33  6574    12017     4337      29       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403148] [ 6575]    33  6575    11917     4264      29       4        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403151] [ 6591]    33  6591    11917     4236      29       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403153] [ 6599]    33  6599    35599      106      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403155] [ 6600]    33  6600    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403157] [ 6602]    33  6602   215282     1961      97       4       47             0 apache2
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403160] [ 6603]    33  6603    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403162] [ 6611]    33  6611    35599      106      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403164] [ 6634]    33  6634    11833     4170      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403166] [ 6637]    33  6637    11821     4380      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403169] [ 6643]    33  6643    35599      106      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403171] [ 6645]    33  6645    11856     4174      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403173] [ 6649]    33  6649    11833     4168      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403176] [ 6663]    33  6663    11853     4235      29       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403178] [ 6672]    33  6672    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403180] [ 6678]    33  6678    11854     4170      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403183] [ 6690]    33  6690    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403185] [ 6693]    33  6693    35599      106      24       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403187] [ 6696]    33  6696    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403190] [ 6698]    33  6698    11692     4062      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403192] [ 6703]    33  6703    11660     4040      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403194] [ 6709]    33  6709    11596     3984      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403196] [ 6722]    33  6722    35599      106      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403199] [ 6727]    33  6727    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403201] [ 6730]    33  6730    11560     3906      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403204] [ 6734]    33  6734    11560     3906      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403206] [ 6744]    33  6744    35599      107      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403208] [ 6750]    33  6750    11204     3558      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403210] [ 6751]    33  6751    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403212] [ 6752]    33  6752    35599      106      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403214] [ 6753]    33  6753    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403217] [ 6756]    33  6756    11305     3638      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403219] [ 6758]    33  6758    35599      106      21       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403221] [ 6760]    33  6760    11564     3903      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403223] [ 6763]    33  6763    11304     3650      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403225] [ 6764]    33  6764    35599      107      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403227] [ 6768]    33  6768    35599      106      21       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403238] [ 6769]    33  6769    35599      107      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403240] [ 6773]    33  6773    11139     3464      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403242] [ 6776]    33  6776    35599      105      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403244] [ 6778]    33  6778    11172     3530      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403246] [ 6779]    33  6779    11109     3498      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403248] [ 6781]    33  6781    35599      106      25       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403251] [ 6787]    33  6787    35599      106      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403254] [ 6788]    33  6788    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403256] [ 6792]    33  6792    11139     3463      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403258] [ 6793]    33  6793    11108     3463      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403261] [ 6796]    33  6796    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403263] [ 6797]    33  6797    11139     3464      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403265] [ 6802]    33  6802    11108     3464      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403267] [ 6803]    33  6803    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403269] [ 6804]    33  6804    35599      105      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403271] [ 6805]    33  6805    11140     3464      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403273] [ 6815]    33  6815    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403275] [ 6821]    33  6821    35599      105      21       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403278] [ 6822]    33  6822    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403281] [ 6823]    33  6823    11139     3475      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403284] [ 6826]    33  6826    11044     3458      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403287] [ 6829]    33  6829    11075     3455      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403290] [ 6833]    33  6833    35599      105      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403294] [ 6839]    33  6839    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403296] [ 6842]    33  6842    11075     3456      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403298] [ 6843]    33  6843    11075     3399      26       4        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403301] [ 6849]    33  6849    35599      107      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403303] [ 6850]    33  6850    35599      105      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403305] [ 6851]    33  6851    11139     3465      27       4        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403307] [ 6853]    33  6853    11075     3457      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403310] [ 6857]    33  6857    11075     3457      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403312] [ 6862]    33  6862    35599      106      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403314] [ 6872]    33  6872    35599      107      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403316] [ 6876]    33  6876    11044     3457      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403319] [ 6877]    33  6877    11075     3427      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403321] [ 6881]    33  6881    35599      106      21       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403324] [ 6882]    33  6882    35599      106      23       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403326] [ 6884]    33  6884    11075     3399      28       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403328] [ 6890]    33  6890    35599      105      21       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403331] [ 6891]    33  6891    35599      105      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403333] [ 6895]    33  6895    11011     3352      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403335] [ 6898]    33  6898    35599      107      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403337] [ 6900]    33  6900    35599      106      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403340] [ 6901]    33  6901    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403342] [ 6902]    33  6902    11142     3467      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403345] [ 6903]    33  6903    11075     3399      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403347] [ 6905]    33  6905    11075     3400      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403349] [ 6907]    33  6907    10980     3397      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403352] [ 6911]    33  6911    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403354] [ 6912]    33  6912    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403356] [ 6918]    33  6918    11015     3401      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403359] [ 6920]    33  6920    11011     3349      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403361] [ 6925]    33  6925    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403363] [ 6926]    33  6926    35599      107      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403365] [ 6933]    33  6933    10980     3351      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403368] [ 6935]    33  6935    11014     3351      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403369] [ 6940]    33  6940    11011     3351      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403372] [ 6942]    33  6942    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403374] [ 6943]    33  6943    35599      106      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403376] [ 6944]    33  6944    35599      105      21       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403379] [ 6952]    33  6952    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403381] [ 6953]    33  6953    35599      107      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403383] [ 6959]    33  6959    10980     3352      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403386] [ 6960]    33  6960    10768     3138      25       4        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403388] [ 6964]    33  6964    11075     3427      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403391] [ 6968]    33  6968    35599      105      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403393] [ 6969]    33  6969    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403396] [ 6970]    33  6970    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403398] [ 6972]    33  6972    11012     3351      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403400] [ 6975]    33  6975    10768     3132      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403402] [ 6977]    33  6977    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403405] [ 6984]    33  6984    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403407] [ 6985]    33  6985    10912     3254      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403409] [ 6989]    33  6989    10767     3087      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403411] [ 6990]    33  6990    10771     3091      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403414] [ 6992]    33  6992    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403417] [ 6993]    33  6993    35599      107      24       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403419] [ 6994]    33  6994    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403421] [ 7000]    33  7000    10767     3131      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403423] [ 7004]    33  7004    10814     3155      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403426] [ 7008]    33  7008    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403428] [ 7009]    33  7009    10814     3150      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403431] [ 7014]    33  7014    10768     3087      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403433] [ 7019]    33  7019    10751     3087      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403435] [ 7023]    33  7023    10915     3245      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403437] [ 7024]    33  7024    10984     3353      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403439] [ 7025]    33  7025    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403442] [ 7026]    33  7026    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403445] [ 7027]    33  7027    35599      105      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403447] [ 7028]    33  7028    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403449] [ 7029]    33  7029    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403451] [ 7030]    33  7030    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403454] [ 7031]    33  7031    35599      106      24       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403456] [ 7041]    33  7041    10735     3102      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403458] [ 7043]    33  7043    10768     3139      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403460] [ 7046]    33  7046    10687     3081      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403463] [ 7052]    33  7052    10688     3081      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403466] [ 7059]    33  7059    35599      107      20       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403469] [ 7060]    33  7060    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403472] [ 7066]    33  7066    10671     3086      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403475] [ 7069]    33  7069    10735     3086      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403479] [ 7070]    33  7070    10703     3071      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403482] [ 7072]    33  7072    35599      107      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403485] [ 7073]    33  7073    35599      107      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403487] [ 7082]    33  7082    10674     3075      24       4        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403490] [ 7083]    33  7083    10704     3022      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403492] [ 7087]    33  7087    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403494] [ 7088]    33  7088    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403497] [ 7089]    33  7089    35599      106      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403499] [ 7090]    33  7090    10691     3023      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403501] [ 7098]    33  7098    10704     3081      26       4        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403503] [ 7099]    33  7099    10687     3054      24       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403505] [ 7105]    33  7105    35599      107      21       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403508] [ 7106]    33  7106    35599      106      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403510] [ 7107]    33  7107    10688     3048      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403513] [ 7114]    33  7114    10639     3007      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403515] [ 7116]    33  7116    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403517] [ 7117]    33  7117    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403519] [ 7124]    33  7124    10623     2954      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403521] [ 7125]    33  7125    10690     3058      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403523] [ 7126]    33  7126    10688     3059      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403525] [ 7130]    33  7130    10704     3022      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403527] [ 7134]    33  7134    35599      106      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403530] [ 7135]    33  7135    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403532] [ 7140]    33  7140    10704     3021      27       4        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403535] [ 7145]    33  7145    35599      106      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403537] [ 7146]    33  7146    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403539] [ 7152]    33  7152    10607     3006      24       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403541] [ 7156]    33  7156    35599      107      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403544] [ 7157]    33  7157    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403546] [ 7160]    33  7160    10687     3020      24       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403547] [ 7165]  1000  7165    21621     6821      46       3        0             0 perl
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403550] [ 7171]    33  7171    10703     3042      26       4        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403552] [ 7172]    33  7172    35599      106      24       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403555] [ 7173]    33  7173    35599      107      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403557] [ 7180]    33  7180    10607     3006      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403559] [ 7184]    33  7184    35599      105      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403561] [ 7185]    33  7185    35599      106      24       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403564] [ 7186]    33  7186    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403566] [ 7188]     0  7188    12206       96      29       3        9             0 cron
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403569] [ 7195]    33  7195    10469     2825      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403571] [ 7198]    33  7198    10639     2954      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403574] [ 7199]    33  7199    10557     2885      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403576] [ 7200]    33  7200    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403579] [ 7201]    33  7201    35599      105      20       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403582] [ 7205]    33  7205    10595     2958      27       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403584] [ 7207]    33  7207    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403586] [ 7208]    33  7208    35599      107      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403589] [ 7209]   108  7209     1072       18       8       3        0             0 sh
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403591] [ 7210]    33  7210    10525     2880      24       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403593] [ 7212]    33  7212    10590     2952      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403595] [ 7214]    33  7214    10557     2888      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403597] [ 7215]   108  7215     1072       18       8       3        0             0 munin-cron
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403599] [ 7220]    33  7220    10571     2888      24       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403601] [ 7225]   108  7225    10477     2549      24       3        0             0 munin-update
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403604] [ 7226]    33  7226    10571     2940      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403605] [ 7227]    33  7227    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403607] [ 7228]    33  7228    35599      105      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403609] [ 7229]    33  7229    35599      107      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403612] [ 7231]    33  7231    10557     2889      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403615] [ 7240]    33  7240    10433     2758      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403617] [ 7246]    33  7246    35599      106      21       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403620] [ 7255]    33  7255    10227     2558      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403622] [ 7265]    33  7265    10460     2757      25       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403624] [ 7268]    33  7268    10229     2564      24       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403626] [ 7275]    33  7275    35599      105      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403629] [ 7276]    33  7276    35599      106      22       4        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403631] [ 7277]    33  7277    10227     2587      22       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403633] [ 7289]    33  7289    10227     2558      24       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403634] [ 7291]    33  7291    10227     2563      26       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403637] [ 7299]    33  7299    35599      107      21       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403639] [ 7300]    33  7300    10032     2364      23       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403641] [ 7302]    33  7302    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403643] [ 7306]    33  7306     9830     2201      22       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403646] [ 7311]  1000  7311    73390    22892     129       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403648] [ 7313]  1000  7313    77513    25432     136       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403650] [ 7314]  1000  7314    79566    28908     141       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403652] [ 7315]  1000  7315    76681    25114     135       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403654] [ 7316]  1000  7316    79566    28909     141       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403657] [ 7317]    33  7317    35599      107      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403659] [ 7318]    33  7318     8943     1308      21       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403662] [ 7320]    33  7320    35599      106      23       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403664] [ 7324]    33  7324     8940     1302      22       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403666] [ 7329]  1000  7329    72816    22268     128       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403669] [ 7330]  1000  7330    72794    22296     128       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403671] [ 7331]  1000  7331    18462     4678      39       3        0             0 perl
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403673] [ 7332]  1000  7332    76681    25109     135       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403676] [ 7333]  1000  7333    18428     4665      39       3        0             0 perl
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403678] [ 7334]  1000  7334    72816    22268     128       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403680] [ 7335]    33  7335    35599      106      22       3        0             0 cgit.cgi
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403682] [ 7336]  1000  7336    76163    25107     130       4        0             0 /home/alex/farm
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403684] [ 7337]    33  7337     5848      283      16       3        0             0 python3
Jun  8 11:30:26 sibirocobombus kernel: [5341224.403686] Out of memory: Kill process 14367 (/home/alex/farm) score 44 or sacrifice child
Jun  8 11:30:26 sibirocobombus kernel: [5341224.406828] Killed process 14367 (/home/alex/farm) total-vm:387964kB, anon-rss:183760kB, file-rss:4kB, shmem-rss:0kB

Grepping for “Out of memory” gives me the following:

Jun  8 11:31:10 sibirocobombus kernel: [5341254.428514] Out of memory: Kill process 7143 (/home/alex/farm) score 35 or sacrifice child
Jun  8 11:31:13 sibirocobombus kernel: [5341254.487951] Out of memory: Kill process 6554 (/home/alex/farm) score 35 or sacrifice child
Jun  8 11:31:22 sibirocobombus kernel: [5341281.999273] Out of memory: Kill process 6845 (/home/alex/farm) score 35 or sacrifice child
Jun  8 11:31:40 sibirocobombus kernel: [5341298.779134] Out of memory: Kill process 6930 (/home/alex/farm) score 29 or sacrifice child
Jun  8 11:31:44 sibirocobombus kernel: [5341303.361810] Out of memory: Kill process 7316 (/home/alex/farm) score 28 or sacrifice child
Jun  8 11:31:47 sibirocobombus kernel: [5341306.283764] Out of memory: Kill process 13061 (/home/alex/farm) score 28 or sacrifice child
Jun  8 11:31:51 sibirocobombus kernel: [5341310.597420] Out of memory: Kill process 5955 (/home/alex/farm) score 27 or sacrifice child
Jun  8 11:31:58 sibirocobombus kernel: [5341317.593756] Out of memory: Kill process 10300 (/home/alex/farm) score 26 or sacrifice child
Jun  8 11:32:03 sibirocobombus kernel: [5341322.250094] Out of memory: Kill process 10296 (/home/alex/farm) score 26 or sacrifice child
Jun  8 11:32:10 sibirocobombus kernel: [5341328.457769] Out of memory: Kill process 7610 (/home/alex/farm) score 26 or sacrifice child
Jun  8 11:32:10 sibirocobombus kernel: [5341328.803566] Out of memory: Kill process 15822 (/home/alex/farm) score 25 or sacrifice child
Jun  8 11:32:10 sibirocobombus kernel: [5341329.535406] Out of memory: Kill process 5025 (/home/alex/farm) score 25 or sacrifice child
Jun  8 11:32:36 sibirocobombus kernel: [5341354.585875] Out of memory: Kill process 10436 (/home/alex/farm) score 24 or sacrifice child
Jun  8 11:32:46 sibirocobombus kernel: [5341364.953989] Out of memory: Kill process 10295 (/home/alex/farm) score 24 or sacrifice child
Jun  8 11:33:08 sibirocobombus kernel: [5341386.209522] Out of memory: Kill process 7344 (/home/alex/farm) score 25 or sacrifice child
Jun  8 11:33:10 sibirocobombus kernel: [5341389.336958] Out of memory: Kill process 7336 (/home/alex/farm) score 24 or sacrifice child
Jun  8 11:33:15 sibirocobombus kernel: [5341394.201015] Out of memory: Kill process 22174 (/home/alex/farm) score 24 or sacrifice child

OK, so something was definitely hammering the site. Some idiot web spider, perhaps? I mean, mostly my Toadfarm processes were killed. These are the ones serving my wikis. But when I look at the syslog, I see that there is a surprising number of python3 and cgit processes in the list. I just installed cgit yesterday as part of my move away from GitHub. Did somebody try to spider these?

I definitely need to do the following:

  1. ban the idiot
  2. make sure cgit only lists to one hostname
  3. make sure robots.txt prevents spiders from indexing cgit output

Then again, there seems to be no obvious idiot, here. Just small numbers of hits in these minutes.

grep "2018:11:[23]" /var/log/apache2/access.log | /home/alex/bin/leech-detector | head

But here’s an interesting finding:

cat /var/log/apache2/access.log | grep cgit | /home/alex/bin/time-grouping
      08/Jun/2018:14          2     0%
      08/Jun/2018:11        186    91%
      08/Jun/2018:09          1     0%
      08/Jun/2018:08          9     4%
      08/Jun/2018:07          6     2%

At the hour I’m interested in, I got 186 hits for cgit. A visual inspection showed a lot of Mastodon instances!

cat /var/log/apache2/access.log | grep "cgit.*Mastodon" | wc -l
177

I had tooted something about self-hosting git and provided an example URL, and every instance that got the toot probably tried to verify that the link worked, or tried to fetch a preview window, or something. And since cgit installed as CGI script is slow, that caused the problem!

CGI scripts are optimized for not running at all: if you don’t call them, they don’t use any CPU cycles nor memory, unlike Toadfarm and similar setups.

This also shows that editing robots.txt won’t help:

cat /var/log/apache2/access.log | grep "robots.*Mastodon" | wc -l
0

So... some sort of User Agent test which will block access? Let’s edit /etc/apache2/conf-enabled/cgit.conf and add a rewrite rule:

ScriptAlias /cgit/ "/usr/lib/cgit/cgit.cgi/"
RedirectMatch ^/cgit$ /cgit/
Alias /cgit-css "/usr/share/cgit/"
<Directory "/usr/lib/cgit/">
   AllowOverride None
   Options ExecCGI FollowSymlinks
   Require all granted
   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} Mastodon
   RewriteRule .* - [redirect=403,last]
</Directory>

Or, to make links in the README.md of the about section work (if your links to directories end with a slash, in any case):

ScriptAlias /cgit/ "/usr/lib/cgit/cgit.cgi/"
RedirectMatch ^/cgit$ /cgit/
Alias /cgit-css "/usr/share/cgit/"
<Directory "/usr/lib/cgit/">
   AllowOverride None
   Options ExecCGI FollowSymlinks
   Require all granted
   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} Mastodon
   RewriteRule .* - [redirect=403,last]
   RewriteRule /(.*)/about/(..*)/$ /cgit/$1/tree/$2 [redirect,last]
   RewriteRule /(.*)/about/(..*)$ /cgit/$1/plain/$2 [redirect,last]
</Directory>

Tags:

Comments

This has been reported https://github.com/tootsuite/mastodon/issues/4486 but it is not obvious how to address it.

Stéphane Bortzmeyer 2018-06-08 13:54 UTC


Yeah, I think I will have to protect cgit from Mastodon or I’ll never be able to link to cgit. :)

– Alex Schroeder 2018-06-08 15:17 UTC


FYI, this post shows up as “out of memory” in your RSS feed.

Andreas Gohr 2018-06-08 18:37 UTC


Yeah, makes sense. :)

– Alex Schroeder 2018-06-08 20:47 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.

Referrers: Conversation