Comments on 2019-07-04 Lost contact with the machine

I’m not sure we’re done, here. I just tried contacting websites and was getting proxy errors: Apache was unable to contact the Hypnotoad services!

Monit reported all of them up and running. Munin reported load average around 0.6. What the hell?

  1. I added a ton of IP numbers with suspicious activity in my access log to the black list
  2. I restarted all the Hypnotoads

Let’s see what happens.

– Alex Schroeder 2019-07-07 14:14 UTC


Same thing, today! I just had to reload all the Hypnotoads. When it’s happening, I’m getting 502 Proxy errors and cannot connect to my sites via Apache, and I can’t connect to the same sites directly via HTTP. This includes the Mojolicious::Plugin::Status page. Thus, the plugin is currently useless for debugging this.

  1. CPU% is not suspicious
  2. Load is not suspicious
  3. Apache hits are not suspicious

Let’s take a look at the Apache result codes in 10min time groupings:

tail -n 10000 /var/log/apache2/access.log | /home/alex/bin/time-grouping 10 | cut -c 1-80
   08/Jul/2019:11:30        581     5%  200 (62%), 403 (21%), 301 (6%), 408 (3%)
   08/Jul/2019:11:20        885     8%  200 (67%), 301 (16%), 304 (6%), 404 (5%)
   08/Jul/2019:11:10       1265    12%  200 (36%), 502 (29%), 403 (19%), 301 (8%
   08/Jul/2019:11:00        721     7%  200 (75%), 301 (11%), 408 (3%), 304 (3%)
   08/Jul/2019:10:50        798     7%  200 (79%), 301 (11%), 304 (3%), 404 (2%)
   08/Jul/2019:10:40        780     7%  200 (74%), 301 (15%), 404 (3%), 304 (2%)
   08/Jul/2019:10:30        701     7%  200 (72%), 301 (13%), 408 (5%), 304 (3%)
   08/Jul/2019:10:20        757     7%  200 (72%), 301 (16%), 304 (4%), 408 (3%)
   08/Jul/2019:10:10        622     6%  200 (68%), 301 (17%), 408 (4%), 404 (4%)
   08/Jul/2019:10:00        876     8%  200 (75%), 301 (9%), 403 (5%), 408 (4%),
   08/Jul/2019:09:50        701     7%  200 (76%), 301 (11%), 404 (3%), 304 (3%)
   08/Jul/2019:09:40        790     7%  200 (76%), 301 (10%), 408 (4%), 304 (4%)
   08/Jul/2019:09:30        523     5%  200 (78%), 301 (10%), 408 (4%), 404 (3%)

Clearly, at 11:10 the requests nearly double and about a third (29%) result in 502 (proxy error) codes. But there are also 19% resulting in 403 (forbidden). The typical 200 (ok) results drop significantly.

New approach!

I’m going to use the rules listed in Stop fake and bad user agents with htaccess. Let’s hope this works! 🙂

– Alex Schroeder 2019-07-08 10:28 UTC


And it just happened again:

time-grouping | cut -c 1-80
   08/Jul/2019:13:20        122     1%  200 (60%), 403 (18%), 301 (13%), 404 (3%
   08/Jul/2019:13:10        737     7%  200 (75%), 301 (12%), 403 (4%), 404 (3%)
→  08/Jul/2019:13:00        615     6%  502 (44%), 200 (29%), 301 (16%), 403 (3%
→  08/Jul/2019:12:50        642     6%  200 (37%), 502 (34%), 301 (15%), 403 (5%
   08/Jul/2019:12:40        719     7%  200 (68%), 301 (14%), 403 (5%), 408 (4%)
   08/Jul/2019:12:30        774     7%  200 (70%), 301 (13%), 403 (6%), 408 (3%)
   08/Jul/2019:12:20        665     6%  200 (70%), 301 (12%), 304 (7%), 404 (3%)
   08/Jul/2019:12:10        763     7%  200 (73%), 301 (12%), 304 (6%), 404 (3%)
   08/Jul/2019:12:00        848     8%  200 (76%), 301 (12%), 304 (4%), 404 (2%)
   08/Jul/2019:11:50        887     8%  200 (75%), 301 (6%), 304 (6%), 404 (3%),
   08/Jul/2019:11:40        862     8%  200 (78%), 301 (8%), 304 (6%), 408 (3%),
   08/Jul/2019:11:30        809     8%  200 (66%), 403 (15%), 301 (7%), 304 (4%)
   08/Jul/2019:11:20        885     8%  200 (67%), 301 (16%), 304 (6%), 404 (5%)
   08/Jul/2019:11:10        672     6%  200 (41%), 403 (22%), 502 (21%), 301 (8%

I’m out of ideas... I’ll try changing all the Hypnotoad configurations and increase the number of workers per application from 1 to 2.

– Alex Schroeder 2019-07-08 11:20 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit this page if you need to fix typos. You can subscribe to updates by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.