Monit

See Sibirocobombus.

2017-12-01 Monit Self-Signed Certificate Expired

Monit runs a web server on a separate port so it doesn’t depend on your “real” web server. That also means it uses its own certificate. I just discovered that mine had expired.

Kallobombus Monit explains the basic setup. The main config file is this: /etc/monit/conf.d/monit.conf.

set httpd port 2812 and
    SSL ENABLE
    PEMFILE /etc/ssl/localcerts/monit.pem
    allow admin:*secret*

I wrote a note to myself in /etc/ssl/localcerts/README.

https://wiki.debian.org/Self-Signed_Certificate
https://www.howtoforge.com/server-monitoring-with-munin-and-monit-on-debian-wheezy-p2

sudo openssl req -new -x509 -days 365 -nodes -config monit.cnf -out monit.pem -keyout monit.pem
sudo chown root.root monit.pem
sudo chmod 0700 monit.pem
sudo service monit restart
sudo openssl x509 -noout -in monit.pem -fingerprint -sha256

The content of my monit.cnf file just provides some defaults.

# http://www.howtoforge.com/server-monitoring-with-munin-and-monit-on-debian-wheezy-p2
# create RSA certs - Server

RANDFILE = ./openssl.rnd

[ req ]
default_bits = 2048
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = CH

stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Zürich

localityName                    = Locality Name (eg, city)
localityName_default            = Zürich

organizationName                = Organization Name (eg, company)
organizationName_default        = Alex Schroeder

organizationalUnitName          = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Head Desk

commonName                      = Common Name (FQDN of your server)
commonName_default              = alexschroeder.ch

emailAddress                    = Email Address
emailAddress_default            = alex@gnu.org

[ cert_type ]
nsCertType = server

The last command gives me the new fingerprint of the service. When I reconnect to my monit, I’ll still get an error telling me that the cert authority is invalid, which it obviously is, since I self-signed this certificate.

Firefox uses it’s own certificate store. Thus, click on Advanced, Add Exception, View Certificate, and compare the SHA-256 fingerprint with what I just got on the server itself.

SHA256 Fingerprint=51:C8:A6:5E:2F:FB:F6:06:07:FE:B7:62:C3:03:4B:B8:BC:0D:43:AC:25:3C:40:B3:06:75:2F:70:5E:93:AC:78

This looks ok, so close this window and Permanently Accept the exception. Yay!

Tags:

Add Comment

2017-01-24 Monit, Toadfarm, Perlbrew

I’m using Perlbrew, which means that my Perl scripts depend on the right Perl version being used because the modules are all installed per version. If I don’t, then I’ll see errors like the following:

Can't load application from file "/home/alex/farm/face/face.pl": Can't locate GD.pm in @INC (you may need to install the GD module) (@INC contains: ...

When I run scripts from cron or monit, their environment is not as great as it usually is when I’m in an interactive session. Thus, I was able to run my scripts from the command line, but I couldn’t restart my webserver from the Monit page.

I run my sites using Toadfarm from /home/alex/farm/farm and thus I’d see the following in /var/log/monit.log once I ran into a problem:

[CET Jan 23 21:38:31] error    : 'toadfarm' total mem amount of 589.9 MB matches resource limit [total mem amount>500.0 MB]
[CET Jan 23 21:38:31] info     : 'toadfarm' trying to restart
[CET Jan 23 21:38:31] info     : 'toadfarm' stop: /home/alex/farm/farm
[CET Jan 23 21:39:01] error    : 'toadfarm' failed to stop (exit status 2) -- /home/alex/farm/farm: Can't locate Toadfarm.pm in @INC (you may need to install the Toadfarm module) (@INC contains: /home
/alex/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0/x86_64-linux /home/alex/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.
[CET Jan 23 21:41:01] info     : 'toadfarm' process is running after previous exec error (slow starting or manually recovered?)

So now I wrote a little wrapper script called /home/alex/farm/farm-wrapper.

I use it from my Monit config:

check process toadfarm with pidfile /home/alex/farm/farm.pid
    start program = "/home/alex/farm/farm-wrapper start" as uid alex gid alex
    stop program = "/home/alex/farm/farm-wrapper stop" as uid alex gid alex
    # http://alexschroeder.ch:8080/wiki
    if failed host alexschroeder.ch port 8080 type tcp protocol http
      and request "/wiki/Monit" for 5 cycles then restart
    if totalmem > 1000 MB for 5 cycles then restart
    if 6 restarts within 15 cycles then timeout

The wrapper script is chmod +x, just like the actual farm script, /home/alex/farm/farm.

#!/bin/bash

1. This wrapper sets up the Perlbrew environment we need to restart the
1. farm from Monit.
source /home/alex/perl5/perlbrew/etc/bashrc

1. Setup local::lib
eval "$(perl -I/home/alex/perl5/lib/perl5 -Mlocal::lib)"

1. Run farm, depending on its shebang
/home/alex/farm/farm $*

And the actual farm script:

#!/home/alex/perl5/perlbrew/perls/perl-5.24.0/bin/perl
use Toadfarm -init;

$ENV{PATH} .= ":/usr/local/bin";

...

Tags:

Add Comment

2017-01-21 Monit Summary

How weird. I just restarted Monit and it doesn’t get back up. What happened? It looks a lot like this issue.

In the log:

[CET Jan 21 13:58:14] info     : 'sibirocobombus' Monit started
[CET Jan 21 13:58:14] error    : Cannot initialize SSL server certificate handler -- error:140A90A1:SSL routines:func(169):reason(161)
[CET Jan 21 13:58:14] error    : HTTP server: Could not initialize SSL engine
[CET Jan 21 13:58:14] error    : Monit HTTP server not available

From the command line:

alex@sibirocobombus:~$ sudo monit summary
SSL read timeout error
Error connecting to the monit daemon

But then:

alex@sibirocobombus:~$ monit -V
This is Monit version 5.9
Copyright (C) 2001-2014 Tildeslash Ltd. All Rights Reserved.

Also, this hangs after a screen of output:

alex@sibirocobombus:~$ openssl s_client -debug -connect 127.0.0.1:2812
CONNECTED(00000003)
write to 0x73d350 [0x73ded0] (289 bytes => 289 (0x121))
...
0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01   ................
0120 - 01                                                .
^C

Uhm... Wait. I think I confused two directories. In fact what happened was that I moved the port my SSH server is listening on from the default (22) to a different one. But I had forgotten to fix my monit config file. And when I tried to fix the config file, I changed a config file in /etc/monit/monitrc.d/ instead of the real config file in /etc/monit/conf.d/.

But it’s fixed, now (with a security-by-obscurity new port):

check process sshd with pidfile /var/run/sshd.pid
  group system
  group sshd
  start program = "/etc/init.d/ssh start"
  stop  program = "/etc/init.d/ssh stop"
  if failed host localhost port 12345 with proto ssh then restart
  if 5 restarts with 5 cycles then timeout
  depend on sshd_bin
  depend on sftp_bin
  depend on sshd_rc
  depend on sshd_rsa_key
  depend on sshd_dsa_key

...

Restarting munin now works as intended. :)

Tags:

Add Comment

2016-05-31 Monit strangeness

I’m using Toadfarm on two machines. I can start the farm using my own username, and monitor it using Monit, but I can’t start or stop the farm using Monit – even though I use “as uid” and “as gid”:

check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/home/kensanata/farm/farm start" as uid "kensanata" and gid "kensanata"
    stop program = "/home/kensanata/farm/farm stop"
    ...

Weird.

At first, I thought that this actually worked on the older machine, but as it turns out, it never worked. I wonder why.

I rewrote it as follows:

check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/bin/bash -c '/home/kensanata/farm/farm start &> /home/kensanata/farm/start.log'" as uid kensanata
    stop program = "/bin/bash -c '/home/kensanata/farm/farm stop &> /home/kensanata/farm/stop.log'" as uid kensanata
    ...

The log file revealed that the hypnotoad executable was not on the path. So I added the following to my farm file:

$ENV{PATH} .= ":/usr/local/bin";

Now it will start – but not serve! I’m assuming this has to do with permissions somehow?

I think I’m giving up...

kensanata@po6:~/farm$ cat start.log
Server available at http://www.emacswiki.org:8081
farm (17832) started.

Testing:

alex@Megabombus:~$ curl http://www.emacswiki.org:8081/wiki
curl: (52) Empty reply from server

It takes a bit longer than 15s... Looks like an Oddmuse timeout?

I give up. I even added a call to id:

kensanata@po6:~/farm$ cat start.log
uid=1001(kensanata) gid=1001(kensanata) groups=1001(kensanata),0(root)
Server available at http://www.emacswiki.org:8081
farm (18294) started.

It looks good, but it doesn’t work.

I tried to make sure that I’m not using any relative paths. Log file, script, everything is getting absolute paths. And I’ve made some progress. I’m finally seeing some Mojolicious logs:

kensanata@po6:~/farm$ cat farm.log 
[Wed Jun  1 00:14:25 2016] [info] Mounting emacs with conditions
[Wed Jun  1 00:14:25 2016] [info] Loading plugin Toadfarm::Plugin::AccessLog
[Wed Jun  1 00:14:26 2016] [info] Mounting emacs with conditions
[Wed Jun  1 00:14:26 2016] [info] Loading plugin Toadfarm::Plugin::AccessLog
[Wed Jun  1 00:14:26 2016] [info] Listening at "http://www.emacswiki.org:8081"
[Wed Jun  1 00:14:26 2016] [info] Manager 19291 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19292 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19293 started
[Wed Jun  1 00:14:26 2016] [info] Creating process id file "/home/kensanata/farm/farm.pid"
[Wed Jun  1 00:14:26 2016] [debug] Worker 19295 started
[Wed Jun  1 00:14:26 2016] [debug] Worker 19294 started
[Wed Jun  1 00:14:36 2016] [debug] GET "/wiki"
[Wed Jun  1 00:14:36 2016] [debug] Routing to application "Mojolicious::Lite"
[Wed Jun  1 00:14:36 2016] [debug] Routing to a callback
[Wed Jun  1 00:14:36 2016] [debug] [CGI:CODE(0x2740360):19299] START /home/kensanata/farm/wiki.pl
[Wed Jun  1 00:14:36 2016] [error] STDERR: : Permission denied at /usr/local/share/perl/5.18.2/Mojolicious/Plugin/CGI.pm line 62.
...

I wonder what this is.

The code in question:

 open STDERR, $STDERR[0], $STDERR[1] or die "STDERR: @$stderr: $!";

Ah! There was an error logfile with not absolute path... Good, fixed, works!

So now, this is how Monit starts the Toadfarm:

kensanata@po6:~/farm$ sudo cat /etc/monit/conf.d/toadfarm.conf
check process toadfarm with pidfile /home/kensanata/farm/farm.pid
    start program = "/home/kensanata/farm/farm start" as uid www-data and gid www-data
    stop program = "/home/kensanata/farm/farm stop" as uid www-data and gid www-data
    if failed host www.emacswiki.org port 8081 type tcp protocol http
      and request "/wiki" for 5 cycles then restart
    if totalmem > 500 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then timeout

This defines the Toadfarm:

kensanata@po6:~/farm$ cat farm
#! /usr/bin/perl
use Toadfarm -init;

$ENV{PATH} .= ":/usr/local/bin";

logging {
  combined => 1,
  file     => "/home/kensanata/farm/farm.log",
  level    => "debug",
};

my $farm = '/home/kensanata/farm';

mount "$farm/emacs.pl" => {
  "Host" => qr{www.emacswiki.org:8081$},
  mount_point => '/wiki',
};

plugin "Toadfarm::Plugin::AccessLog";

start ["http://www.emacswiki.org:8081"], {proxy => 1}; # needs to be at the last line

(Must change “debug” back to “warn”!)

The wiki itself:

kensanata@po6:~/farm$ cat emacs.pl
#! /usr/bin/env perl
use Mojolicious::Lite;

plugin CGI => {
  support_semicolon_in_query_string => 1,
};

plugin CGI => {
  route => '/',
  script => '/home/kensanata/farm/wiki.pl',
  run => \&OddMuse::DoWikiRequest,
  before => sub {
      my $c = shift;
      $OddMuse::RunCGI = 0;
      $OddMuse::DataDir = '/home/kensanata/emacswiki';
      require '/home/kensanata/farm/wiki.pl' unless defined &OddMuse::DoWikiRequest;
  },
  env => {},
  errlog => '/home/kensanata/farm/emacs.log', # path to where STDERR from cgi script goes
};

app->start;

Note all the absolute paths. :)

This server is now also running a Toadfarm started by Monit. Yay!

Don’t forget permissions: key files and directories need to belong to www-data.kensanata (my user’s group so that I can also edit these files).

  1. the farm directory needs to be writable by www-data so that the pid file can be written; without it, Monit can start the toadfarm but will still claim it failed (pid file is missing) and stopping it will also fail (use ps fax to figure out which process to sudo kill)
  2. the farm log needs to be writable by www-data so that the toadfarm can log
  3. the application log file needs to be writable by www-data so that the wiki itself can log

To wit:

kensanata@po6:~$ ls -la farm
total 1368
drwxrwxr-x  2 www-data  kensanata    4096 Jun  2 17:00 .
drwxr-xr-x 12 kensanata kensanata    4096 Jun  1 00:24 ..
-rw-r--r--  1 www-data  kensanata   27935 Jun  2 16:34 emacs.log
-rw-rw-r--  1 kensanata kensanata     549 Jun  1 00:22 emacs.pl
-rw-rw-r--  1 kensanata kensanata     528 Jun  1 00:12 emacs.pl~
-rwxrwxr-x  1 kensanata kensanata     432 Jun  2 16:32 farm
-rwxrwxr-x  1 kensanata kensanata     433 Jun  1 00:13 farm~
-rw-r--r--  1 www-data  kensanata 1002606 Jun  2 16:34 farm.log
-rw-r--r--  1 www-data  www-data        5 Jun  2 17:00 farm.pid
-rw-rw-r--  1 kensanata kensanata     322 May 29 12:01 Makefile
-rw-rw-r--  1 kensanata kensanata     384 May 29 11:59 Makefile~
-rwxrwxr-x  1 kensanata kensanata  159959 Jun  1 00:10 wiki.pl
-rwxrwxr-x  1 kensanata kensanata  159988 Jun  1 00:07 wiki.pl~

And why is that? Because I want the process to have few permissions elsewhere, and the existing wiki is being run as a CGI script by Apache using www-data which is why the data directory has the same ownership – owned by user www-data and writable by a group I belong to.

Tags:

Add Comment

2014-12-20 Monit

Some monit configurations for my server, from /etc/monit/conf.d:

cron

check process crond with pidfile /var/run/crond.pid
   group system
   group crond
   start program = "/etc/init.d/cron start"
   stop  program = "/etc/init.d/cron stop"
   if 5 restarts with 5 cycles then timeout
   depend cron_bin
   depend cron_rc
   depend cron_spool

 check file cron_bin with path /usr/sbin/cron
   group crond
   include /etc/monit/templates/rootbin

 check file cron_rc with path "/etc/init.d/cron"
   group crond
   include /etc/monit/templates/rootbin

 check directory cron_spool with path /var/spool/cron/crontabs
   group crond
   if failed permission 1730 then unmonitor
   if failed uid root        then unmonitor
   if failed gid crontab     then unmonitor

fail2ban

check process fail2ban with pidfile /var/run/fail2ban/fail2ban.pid
   group monitoring
   start program = "/etc/init.d/fail2ban start"
   stop  program = "/etc/init.d/fail2ban stop"
   if failed unixsocket /var/run/fail2ban/fail2ban.sock then restart
   if 5 restarts within 5 cycles then timeout
   depends on fail2ban_bin
   depends on fail2ban_rc

check file fail2ban_bin with path /usr/bin/fail2ban-client
   group monitoring
   include /etc/monit/templates/rootbin

check file fail2ban_rc with path /etc/init.d/fail2ban
   group monitoring
   include /etc/monit/templates/rootbin

munin

check process munin-node with pidfile /var/run/munin/munin-node.pid
   group monitoring
   start program = "/etc/init.d/munin-node start"
   stop  program = "/etc/init.d/munin-node stop"
   if 5 restarts within 5 cycles then timeout
   depends on munin-node_bin
   depends on munin-node_rc

check file munin-node_bin with path /usr/sbin/munin-node
   group monitoring
   include /etc/monit/templates/rootbin

check file munin-node_rc with path /etc/init.d/munin-node
   group monitoring
   include /etc/monit/templates/rootbin

openssh-server

 check process sshd with pidfile /var/run/sshd.pid
   group system
   group sshd
   start program = "/etc/init.d/ssh start"
   stop  program = "/etc/init.d/ssh stop"
   if failed host localhost port 22 with proto ssh then restart
   if 5 restarts with 5 cycles then timeout
   depend on sshd_bin
   depend on sftp_bin
   depend on sshd_rc
   depend on sshd_rsa_key
   depend on sshd_dsa_key

 check file sshd_bin with path /usr/sbin/sshd
   group sshd
   include /etc/monit/templates/rootbin

 check file sftp_bin with path /usr/lib/openssh/sftp-server
   group sshd
   include /etc/monit/templates/rootbin

 check file sshd_rsa_key with path /etc/ssh/ssh_host_rsa_key
   group sshd
   include /etc/monit/templates/rootstrict

 check file sshd_dsa_key with path /etc/ssh/ssh_host_dsa_key
   group sshd
   include /etc/monit/templates/rootstrict

 check file sshd_rc with path /etc/ssh/sshd_config
   group sshd
   include /etc/monit/templates/rootrc

rsyslog

check process rsyslog with pidfile /var/run/rsyslogd.pid
   group monitoring
   start program = "/etc/init.d/rsyslog start"
   stop  program = "/etc/init.d/rsyslog stop"
   if 5 restarts within 5 cycles then timeout
   depends on rsyslog_bin
   depends on rsyslog_rc

check file rsyslog_bin with path /usr/sbin/rsyslogd
   group monitoring
   include /etc/monit/templates/rootbin

check file rsyslog_rc with path /etc/init.d/rsyslog
   group monitoring
   include /etc/monit/templates/rootbin

sendmail

check process sendmail with pidfile /var/run/sendmail/mta/sendmail.pid
   group mail
   start program = "/etc/init.d/sendmail start"
   stop  program = "/etc/init.d/sendmail stop"
1.  if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout
   depends on sendmail_bin
   depends on sendmail_rc

1. /usr/lib/sendmail points to /etc/alternatives/lib.sendmail which in
1. turn points to the file tested:

check file sendmail_bin with path /usr/lib/sm.bin/sendmail
   group mail
   if failed checksum        then unmonitor
   if failed permission 2755 then unmonitor
   if failed uid root        then unmonitor
   if failed gid smmsp       then unmonitor

check file sendmail_rc with path /etc/init.d/sendmail
   group mail
   include /etc/monit/templates/rootbin

System

There’s some stuff in /etc/monit/monitrc which I adapted from the defaults (the admin password and all that).

I changed the defaults for the system check:

check system Kallobombus
  if loadavg (5min) > 4 then alert
1. if loadavg (10min) > 2 then alert
1. if loadavg (15min) > 1 then alert
  if memory usage > 75% then alert
1. if swap usage > 25% then alert
1. if cpu usage (user) > 98% then alert
  if cpu usage (user) > 98% for 5 cycles then alert
  if cpu usage (system) > 50% then alert
  if cpu usage (wait) > 20% then alert

Apache

This is also changed from the defaults in /etc/monit/monitrc.

check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host alexschroeder.ch port 80 protocol http
      and request "/monit/token" then restart
1.  if cpu is greater than 60% for 2 cycles then alert
1.  if cpu > 80% for 5 cycles then restart
1.  if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
1.  if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

Rsync

Needed to create the file /etc/rsyncd.conf in order to actually share something. And I needed to set RSYNC_ENABLE=true in /etc/default/rsync in order to have rsync start up as a demon. And sudo service rsync start, of course.

For monit, however:

 check process rsyncd with pidfile /var/run/rsync.pid
   start program = "/etc/init.d/rsync start"
   stop  program = "/etc/init.d/rsync stop"
   if 5 restarts with 5 cycles then timeout
   depend rsync_bin
   depend rsync_rc
   depend rsync_conf

 check file rsync_bin with path /usr/bin/rsync
   include /etc/monit/templates/rootbin

 check file rsync_rc with path "/etc/init.d/rsync"
   include /etc/monit/templates/rootbin

 check file rsync_conf with path "/etc/rsyncd.conf"
   include /etc/monit/templates/rootrc

If you have nothing to share, disable it in /etc/default/rsync and stop monitoring it. Works for me!

Tags:

Add Comment

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.