segfault after upgrade

I installed this plugin a few months ago, before any actual release, and things generally worked well. After noticing new releases here, I performed an upgrade from source (from 0dbae61 to 1.4.0+fb31b60), but now the plugin is crashing bitlbee with a segfault after a while:

11:40:04 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)

the kernel agrees with my guess:

[2635932.507453] traps: bitlbee[24205] trap stack segment ip:7f62b6876556 sp:7ffe4d693490 error:0
[2635932.507472]  in mastodon.so[7f62b6866000+16000]

Here's a backtrace I was able to get by running bitlbee under gdb:

bitlbee@marcos:/var/lib/bitlbee$ gdb /usr/sbin/bitlbee
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/bitlbee...(no debugging symbols found)...done.
(gdb) run -nvD
Starting program: /usr/sbin/bitlbee -nvD
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Warning: Warning: Couldn't write PID to `/var/run/bitlbee.pid'



Program received signal SIGSEGV, Segmentation fault.
mastodon_http_get_filters (req=<optimized out>) at mastodon-lib.c:4080
4080            md->flags |= MASTODON_GOT_FILTERS;
(gdb) bt full
#0  mastodon_http_get_filters (req=<optimized out>) at mastodon-lib.c:4080
        ic = 0x55555584a7e0
        md = 0x0
#1  0x000055555557e1af in ?? ()
No symbol table info available.
#2  0x000055555557cfd1 in ?? ()
No symbol table info available.
#3  0x00007ffff73845a0 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#4  0x000055555557cb50 in b_main_run ()
No symbol table info available.
#5  0x0000555555566d3d in main ()
No symbol table info available.

I tried to bisect the bug by recompiling, reinstalling and restarting, then waiting 40 seconds (as the crash happens anytime between 2 and 30 seconds after restart). I've been able to identify this commit as the source of the crash:

commit a3bc155b03bec0862835549c37ce3a69d3d39fd0
Author: Alex Schroeder <alex@gnu.org>
Date:   Wed Aug 29 11:07:55 2018 +0200

    Load filters together with the initial timeline.

More informally, I can confirm that v1.3.1 does not have this bug.

The plugin was compiled and is running on Debian stretch using bitlbee 3.5.1-1. I'm connected to two different Mastodon servers, one of which is failing with a 502 error (when bitlbee doesn't crash of course):

12:13:29 <@root> mastodon - Login error: Stream closed (502 Bad Gateway)

I hope that helps!


I was hoping that this issue was fixed with commit cb329b96 because I've noticed a similar problem with an instance that doesn't support filters. But fb31b60 includes that fix so... I guess not. Hm. – Alex

Tags: Closed

Reopen issue

Comments

for what it's worth, i believe this bug is fixed in the latest 1.4.0 release, as I have upgraded and can't reproduce the crash anymore. so this should probably be closed! :)

– Anonymous 2018-12-14 14:44 UTC


OK, cool! :)

– Alex Schroeder 2018-12-14 17:17 UTC


actually, i was wrong - the bug actually triggers every 5 or 15 minutes now. maybe i just didn't notice because everything restarts more or less normally afterwards? or maybe it's the way the Debian package installs the plugin (e.g. it doesn't ship the .la file).

[déc18 12:06] traps: bitlbee[26135] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000012]  in mastodon.so[7fb61022a000+d000]
[déc18 12:11] traps: bitlbee[623] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000012]  in mastodon.so[7fb61022a000+d000]
[déc18 12:31] traps: bitlbee[27873] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000009]  in mastodon.so[7fb61022a000+d000]
[déc18 12:36] traps: bitlbee[2507] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000012]  in mastodon.so[7fb61022a000+d000]
[déc18 13:01] traps: bitlbee[4326] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000012]  in mastodon.so[7fb61022a000+d000]
[déc18 13:21] traps: bitlbee[31383] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000011]  in mastodon.so[7fb61022a000+d000]
[déc18 13:26] traps: bitlbee[5863] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000012]  in mastodon.so[7fb61022a000+d000]
[déc18 13:41] traps: bitlbee[26093] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000010]  in mastodon.so[7fb61022a000+d000]
[déc18 13:56] traps: bitlbee[14224] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000012]  in mastodon.so[7fb61022a000+d000]
[déc18 14:01] traps: bitlbee[20919] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000012]  in mastodon.so[7fb61022a000+d000]
[déc18 14:31] traps: bitlbee[29737] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000010]  in mastodon.so[7fb61022a000+d000]
[déc18 14:41] traps: bitlbee[10984] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000011]  in mastodon.so[7fb61022a000+d000]
[déc18 14:51] traps: bitlbee[24458] trap stack segment ip:7fb610236276 sp:7ffee756d070 error:0
[  +0,000011]  in mastodon.so[7fb61022a000+d000]

I've attached a debugger to see if the backtrace is the same now.

– Anonymous 2018-12-18 20:07 UTC


SIGBUS after a while:

Program received signal SIGBUS, Bus error.
0x00007ffff7fec276 in mastodon_http_get_filters () from /usr/lib/bitlbee/mastodon.so
(gdb) bt
#0  0x00007ffff7fec276 in mastodon_http_get_filters () from /usr/lib/bitlbee/mastodon.so
#1  0x000055555557e1af in ?? ()
#2  0x000055555557cfd1 in ?? ()
#3  0x00007ffff73845a0 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#4  0x000055555557cb50 in b_main_run ()
#5  0x0000555555566d3d in main ()

my gdb-fu is rusty so i can't get more line information than that:

(gdb) list
79      in ../sysdeps/unix/syscall-template.S

... which i find somewhat strange...

i hope that helps!

anarcat 2018-12-18 20:44 UTC


here's where the crash actually happens, on git v1.4.0:

#0  0x00007ffff7fec276 in mastodon_http_get_filters (req=<optimized out>) at mastodon-lib.c:4063

That is:

4063            md->flags |= MASTODON_GOT_FILTERS;

it looks like the md pointer is garbage there:

(gdb) p md
$1 = (struct mastodon_data *) 0x7265762f73746e75
(gdb) p *md
Cannot access memory at address 0x7265762f73746e75

The rest of the im_connection structure is okay though:

(gdb) p *ic
$4 = {acc = 0x6970612f20544547, flags = 791770671, proto_data = 0x7265762f73746e75, inpa = 1601791593, keepalive = 1684370019, permit = 0x20736c6169746e65, 
  deny = 0x312e312f50545448, permdeny = 1866992141, away = 0x772e6c6169636f73 <error: Cannot access memory at address 0x772e6c6169636f73>, bee = 0xa0d74732e6f6865, 
  groupchats = 0x6567412d72657355, chatlist = 0x6c746942203a746e}

It's possible there's problem with my compilation process here: I'm using the plugin as built from Debian unstable on a Debian stable machine. Both bitlbe versions match, but there's been a "binNMU" (binary rebuild) of Bitlbee after the stable update, so it's possible there's been changes in libc or some other lib that causes those problems. After rebuilding a backport, it seems to fix the problem...

also, i wonder if this patch (not yet part of a release) might fix the problem pre-emptively.

anarcat 2018-12-18 21:27 UTC


Hm... Get filters. That reminds me of something. There used to be a problem with instances that didn't support filters in the first place.

This commit checks the status code, for example:

And of course the commit you pointed to seems interesting as well. In all these cases, though, I wonder about filters. What version does the instance run? Do you have filters defined? Filters require Mastodon v2.4.3 or newer.

– Alex Schroeder 2018-12-20 08:21 UTC


the instance I'm using is v2.5.2. i do not have filters defined. cb329b9 probably does not fix the problem as it's shipped in 1.4.0 which still suffers from the occasional segfault, even after a backport rebuild. it's just less frequent. i suspect some posts might trip the bug because i can go on for days without a problem and then all of a sudden it crashes regularly.

for example, it started crashing again today:

10:34:26 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
10:36:37 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
10:41:36 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
10:46:37 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
10:51:38 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
10:56:37 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
11:01:36 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
11:06:37 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
11:13:41 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
11:16:37 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
11:21:38 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
11:26:37 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
11:36:37 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)
11:46:37 [bitlbee] -!- ERROR BitlBee crashed! (SIGSEGV received)

this looks like a regular occurence: every 5 minutes or so. of course now I restarted bitlbee in a debugger and it stopped crashing. and for some reason I can't find that stupid core file anymore, so I don't have a backtrace for you this time around...

i'll try a rebuild with teh latest git see what that gives me.

anarcat 2019-01-15 17:00 UTC


Gah, how frustrating!

If things start working under the debugger, isn't that usually a sign of pointers pointing to random locations?

– Alex Schroeder 2019-01-15 22:08 UTC


To save this page you must answer this question:

What is the first letter of this question?