Comments on 2017-02-15 Load

OK, I started disabling sites in my Toadfarm configuration until I only had two sites left: campaignwiki.org and oddmuse.org. A few hours I checked back and I had ten farm processes running using 10% CPU each. Still a problem!

Then I disabled campaignwiki.org and tried calling it from the command line. Mojolicious allows me to do that, just like the CGI module does, too. It turns out that I can run a few requests, but when I load RecentChanges, I get a timeout.

Here’s me getting a regular page:

alex@sibirocobombus:~/farm$ perl campaignwiki.pl get '/?action=browse&id=Hauptseite'
[Wed Feb 15 22:39:27 2017] [debug] GET "/"
[Wed Feb 15 22:39:27 2017] [debug] Routing to a callback
[Wed Feb 15 22:39:27 2017] [debug] [CGI:CODE(0x1506388):11783] START /home/alex/farm/wiki.pl
[Wed Feb 15 22:39:28 2017] [debug] 200 OK (0.32674s, 3.061/s)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
...
</body>
</html>

Here’s me getting RecentChanges:

alex@sibirocobombus:~/farm$ perl campaignwiki.pl get '/RecentChanges'
[Wed Feb 15 22:39:42 2017] [debug] GET "/RecentChanges"
[Wed Feb 15 22:39:42 2017] [debug] Routing to a callback
[Wed Feb 15 22:39:43 2017] [debug] [CGI:CODE(0x1552388):11786] START /home/alex/farm/wiki.pl
[Wed Feb 15 22:39:43 2017] [debug] 200 OK (0.350602s, 2.852/s)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
...
<div class="rc"><hr />[Wed Feb 15 22:39:58 2017] [debug] Inactivity timeout
[Wed Feb 15 22:40:03 2017] campaignwiki.pl: Problem loading URL "http://127.0.0.1:40810/RecentChanges": Inactivity timeout

When I check top, I see a perl process running eating all available CPU for every time I try to view RecentChanges and got an inactivity timeout. That’s suspiciously like the problem I’ve been experiencing.

So here’s the current status: All the web apps are back online except for campaignwiki.org.

– Alex Schroeder 2017-02-15 21:50 UTC


I added some debug warning messages and discovered that the infinite loop was somehow caused by StripRollbacks for the Gridmapper namespace. I looked at the rc.log file and found a single rollback entry. I removed the entire section and everything seems to work. So campaignwiki.org is up again and the bug remains a mystery.

It took me a while to get load back down

– Alex Schroeder 2017-02-15 22:16 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. See Info for text formatting rules. You can edit this page if you need to fix typos. You can subscribe to updates by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.