View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0005387GNUnetcadet servicepublic2018-07-03 17:092019-04-07 14:48
Reporterch3 Assigned To 
PrioritynormalSeveritycrashReproducibilitysometimes
Status acknowledgedResolutionopen 
Product VersionGit master 
Target Version0.11.4 
Summary0005387: Cadet crash
DescriptionObserved this during a gnunet-rps-profiler run:

Reading symbols from /home/ch3/prefix_gn/lib//gnunet/libexec/gnunet-service-cadet...done.

warning: core file may not match specified executable file.
[New LWP 1574]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/ch3/prefix_gn/lib//gnunet/libexec/gnunet-service-cadet -c /tmp/testbedwJ7'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:1512
1512 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory.
(gdb) bt
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:1512
#1 0x00007f5406092d82 in GNUNET_CONTAINER_multihashmap_get_multiple (map=0x1353810, key=0x135eb90, it=0x7f5405a5c871 <process_client_result>, it_cls=0x135eb70) at container_multihashmap.c:837
#2 0x00007f5405a5cc9f in handle_client_result (cls=0x133ca80, msg=0x135eb70) at dht_api.c:811
#3 0x00007f54060b1b47 in GNUNET_MQ_inject_message (mq=0x133ce10, mh=0x135eb70) at mq.c:250
#4 0x00007f54060839d4 in recv_message (cls=0x133ccf0, msg=0x135eb70) at client.c:334
#5 0x00007f54060b0ffb in GNUNET_MST_from_buffer (mst=0x133b390, buf=0x0, size=0, purge=0, one_shot=0) at mst.c:231
#6 0x00007f54060b18ac in GNUNET_MST_read (mst=0x133b390, sock=0x133f470, purge=0, one_shot=0) at mst.c:374
#7 0x00007f5406083f09 in receive_ready (cls=0x133ccf0) at client.c:421
#8 0x00007f54060c8f3d in GNUNET_SCHEDULER_do_work (sh=0x13436e0) at scheduler.c:2104
#9 0x00007f54060c9df3 in select_loop (sh=0x13436e0, context=0x7ffd2a70b240) at scheduler.c:2405
#10 0x00007f54060c42f3 in GNUNET_SCHEDULER_run (task=0x7f54060ca926 <service_main>, task_cls=0x7ffd2a70b560) at scheduler.c:725
#11 0x00007f54060cfad5 in GNUNET_SERVICE_run_ (argc=3, argv=0x7ffd2a70bae8, service_name=0x426c87 "cadet", options=GNUNET_SERVICE_OPTION_NONE, service_init_cb=0x4071d2 <run>, connect_cb=0x406488 <client_connect_cb>,
    disconnect_cb=0x406d5f <client_disconnect_cb>, cls=0x0, handlers=0x7ffd2a70b6c0) at service.c:1875
#12 0x0000000000407c7b in main (argc=3, argv=0x7ffd2a70bae8) at gnunet-service-cadet.c:1468
Additional Information(gdb) bt f
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:1512
No locals.
#1 0x00007f5406092d82 in GNUNET_CONTAINER_multihashmap_get_multiple (map=0x1353810, key=0x135eb90, it=0x7f5405a5c871 <process_client_result>, it_cls=0x135eb70) at container_multihashmap.c:837
        sme = 0x13665b0
        nxt = 0xdf0adba0df00074
        count = 1
        me = {sme = 0x1366340, bme = 0x1366340}
#2 0x00007f5405a5cc9f in handle_client_result (cls=0x133ca80, msg=0x135eb70) at dht_api.c:811
        handle = 0x133ca80
#3 0x00007f54060b1b47 in GNUNET_MQ_inject_message (mq=0x133ce10, mh=0x135eb70) at mq.c:250
        handler = 0x133cf10
        handled = 1
        msize = 456
        mtype = 145
        __FUNCTION__ = "GNUNET_MQ_inject_message"
#4 0x00007f54060839d4 in recv_message (cls=0x133ccf0, msg=0x135eb70) at client.c:334
        cstate = 0x133ccf0
        __FUNCTION__ = "recv_message"
#5 0x00007f54060b0ffb in GNUNET_MST_from_buffer (mst=0x133b390, buf=0x0, size=0, purge=0, one_shot=0) at mst.c:231
        hdr = 0x135eb70
        delta = 1530630319057661
        want = 456
        ibuf = 0x135eb70 <incomplete sequence \310>
        need_align = 32596
        offset = 20345184
        ret = 1
        cbret = 20290624
        __FUNCTION__ = "GNUNET_MST_from_buffer"
#6 0x00007f54060b18ac in GNUNET_MST_read (mst=0x133b390, sock=0x133f470, purge=0, one_shot=0) at mst.c:374
        ret = 457
        left = 457
        buf = 0x135eb70 <incomplete sequence \310>
        __FUNCTION__ = "GNUNET_MST_read"
#7 0x00007f5406083f09 in receive_ready (cls=0x133ccf0) at client.c:421
        cstate = 0x133ccf0
        ret = 8
#8 0x00007f54060c8f3d in GNUNET_SCHEDULER_do_work (sh=0x13436e0) at scheduler.c:2104
        p = GNUNET_SCHEDULER_PRIORITY_DEFAULT
        pos = 0x133f570
        now = {abs_value_us = 1530630319057651}
        __FUNCTION__ = "GNUNET_SCHEDULER_do_work"
#9 0x00007f54060c9df3 in select_loop (sh=0x13436e0, context=0x7ffd2a70b240) at scheduler.c:2405
        time_remaining = {rel_value_us = 0}
        rs = 0x13431a0
        ws = 0x1343230
        select_result = 5
        __FUNCTION__ = "select_loop"
#10 0x00007f54060c42f3 in GNUNET_SCHEDULER_run (task=0x7f54060ca926 <service_main>, task_cls=0x7ffd2a70b560) at scheduler.c:725
        sh = 0x13436e0
        driver = 0x1341ad0
        context = {scheduled_head = 0x1361bf0, scheduled_tail = 0x1341b00, timeout = {abs_value_us = 1530630319004971}}
#11 0x00007f54060cfad5 in GNUNET_SERVICE_run_ (argc=3, argv=0x7ffd2a70bae8, service_name=0x426c87 "cadet", options=GNUNET_SERVICE_OPTION_NONE, service_init_cb=0x4071d2 <run>, connect_cb=0x406488 <client_connect_cb>,
    disconnect_cb=0x406d5f <client_disconnect_cb>, cls=0x0, handlers=0x7ffd2a70b6c0) at service.c:1875
        sh = {cfg = 0x1339070, service_name = 0x426c87 "cadet", service_init_cb = 0x4071d2 <run>, connect_cb = 0x406488 <client_connect_cb>, disconnect_cb = 0x406d5f <client_disconnect_cb>, cb_cls = 0x0, slc_head = 0x1355970,
          slc_tail = 0x1355970, clients_head = 0x133f610, clients_tail = 0x133f610, handlers = 0x1339090, task_cls = 0x0, v4_denied = 0x0, v6_denied = 0x0, v4_allowed = 0x1343960, v6_allowed = 0x1345fa0, match_uid = 1, match_gid = 1,
          got_shutdown = 0, options = GNUNET_SERVICE_OPTION_NONE, ready_confirm_fd = -1, ret = 0, require_found = 1}
        cfg_filename = 0x1339050 "~/.config/gnunet.conf"
        opt_cfg_filename = 0x1339350 "/tmp/testbedwJ78VK/119/config"
        loglev = 0x0
        xdg = 0x0
        logfile = 0x0
        do_daemonize = 0
        skew_offset = 0
        skew_variance = 0
        clock_offset = 0
        cfg = 0x1339070
        ret = 3
        err = 0
        service_options = {{shortName = 99 'c', name = 0x7f54060dcc44 "config", argumentHelp = 0x7f54060dcc0a "FILENAME", description = 0x7f54060dcc50 "use configuration file FILENAME", require_argument = 1, option_mandatory = 0,
            processor = 0x7f54060ad34f <set_filename>, cleaner = 0x0, scls = 0x7ffd2a70b550}, {shortName = 100 'd', name = 0x7f54060dfbe4 "daemonize", argumentHelp = 0x0,
            description = 0x7f54060dfbc0 "do daemonize (detach from terminal)", require_argument = 0, option_mandatory = 0, processor = 0x7f54060ad014 <set_one>, cleaner = 0x0, scls = 0x7ffd2a70b53c}, {shortName = 104 'h',
---Type <return> to continue, or q <return> to quit---
            name = 0x7f54060dcb7e "help", argumentHelp = 0x0, description = 0x7f54060dcb83 "print this help", require_argument = 0, option_mandatory = 0, processor = 0x7f54060ac932 <format_help>, cleaner = 0x0, scls = 0x0}, {
            shortName = 76 'L', name = 0x7f54060dcbce "log", argumentHelp = 0x7f54060dcbd2 "LOGLEVEL", description = 0x7f54060dcbe0 "configure logging to use LOGLEVEL", require_argument = 1, option_mandatory = 0,
            processor = 0x7f54060ad0e1 <set_string>, cleaner = 0x0, scls = 0x7ffd2a70b548}, {shortName = 108 'l', name = 0x7f54060dcc02 "logfile", argumentHelp = 0x7f54060dcc0a "FILENAME",
            description = 0x7f54060dcc18 "configure logging to write logs to FILENAME", require_argument = 1, option_mandatory = 0, processor = 0x7f54060ad34f <set_filename>, cleaner = 0x0, scls = 0x7ffd2a70b540}, {shortName = 118 'v',
            name = 0x7f54060dca60 "version", argumentHelp = 0x0, description = 0x7f54060dca68 "print the version number", require_argument = 0, option_mandatory = 0, processor = 0x7f54060ac856 <print_version>, cleaner = 0x0,
            scls = 0x7f54060dfbee}, {shortName = 0 '\000', name = 0x0, argumentHelp = 0x0, description = 0x0, require_argument = 0, option_mandatory = 0, processor = 0x0, cleaner = 0x0, scls = 0x0}}
        __FUNCTION__ = "GNUNET_SERVICE_run_"
#12 0x0000000000407c7b in main (argc=3, argv=0x7ffd2a70bae8) at gnunet-service-cadet.c:1468
        mh = {{mv = 0x0, cb = 0x404582 <handle_port_open>, cls = 0x0, type = 1022, expected_size = 68}, {mv = 0x0, cb = 0x4047fd <handle_port_close>, cls = 0x0, type = 1023, expected_size = 68}, {mv = 0x0,
            cb = 0x404bad <handle_channel_create>, cls = 0x0, type = 1024, expected_size = 108}, {mv = 0x0, cb = 0x404fac <handle_channel_destroy>, cls = 0x0, type = 1025, expected_size = 8}, {mv = 0x405209 <check_local_data>,
            cb = 0x405492 <handle_local_data>, cls = 0x0, type = 1020, expected_size = 8}, {mv = 0x0, cb = 0x405714 <handle_local_ack>, cls = 0x0, type = 1021, expected_size = 8}, {mv = 0x0, cb = 0x405971 <handle_get_peers>, cls = 0x0,
            type = 1036, expected_size = 4}, {mv = 0x0, cb = 0x405b51 <handle_show_peer>, cls = 0x0, type = 1037, expected_size = 40}, {mv = 0x0, cb = 0x405cf9 <handle_info_tunnels>, cls = 0x0, type = 1032, expected_size = 4}, {mv = 0x0,
            cb = 0x405e64 <handle_info_tunnel>, cls = 0x0, type = 1033, expected_size = 40}, {mv = 0x0, cb = 0x406190 <handle_info_dump>, cls = 0x0, type = 1038, expected_size = 4}, {mv = 0x0, cb = 0x0, cls = 0x0, type = 0,
            expected_size = 0}}
TagsNo tags attached.

Activities

Christian Grothoff

2019-01-24 19:58

manager   ~0013462

Ugh, this one looks like general memory corruption. Can this be reproduced with [cadet]
PREFIX = valgrind

enabled?

dvn

2019-01-28 04:34

developer   ~0013487

ch3 can you give instructions to reproduce?

i.e. how would one run the gnunet-rps-profiler?

dvn

2019-01-28 04:51

developer   ~0013489

Last edited: 2019-01-28 04:52

 As no-body has reported a clone of this ticket, I think it is safe to say that it is probably an edge-case related to RPS. Since, to my knowledge, RPS is to remain experimental for release 0.11.0, I think we could move this issue outside of the release roadmap.

Christian Grothoff

2019-01-28 06:01

manager   ~0013493

No, CADET crashes are never 'edge cases' from RPS. This is likely a serious bug, just RPS is the only thing we have that really seriously exercises the CADET logic.

ch3

2019-01-29 11:32

developer   ~0013510

What I currently use to run the profiler is something like this:
`.libs/gnunet-rps-profiler -c profiler_rps.conf -l gnunet_rps_profiler.log -d 30m -n 10 -r 1000` (duration of 30 Mins - can be shorter, 10 peers - can be fewer, requesting 1000 IDs - just a high number that it keeps requesting)

(Was not able to generate valgrind reports yet.)

Christian Grothoff

2019-02-09 20:01

manager   ~0013633

I've run the test for 50 minutes now, but it did neither crash nor terminate. Is the -d switch working? Does it crash every time for you?

Christian Grothoff

2019-02-12 12:27

manager   ~0013680

ch3 says this only happens "rarely". So need to do more runs. Party.

Christian Grothoff

2019-02-13 18:12

manager   ~0013708

I've been running this for a day now in a loop, no dice. So if it is still real, it's so rare that I don't think it is inherently RC, or at least we need to put in more effort to reproduce it more easily.

Issue History

Date Modified Username Field Change
2018-07-03 17:09 ch3 New Issue
2018-07-03 17:09 ch3 Status new => assigned
2018-07-03 17:09 ch3 Assigned To => Bart Polot
2018-07-07 00:26 Christian Grothoff Target Version => 0.11.0
2019-01-24 19:58 Christian Grothoff Note Added: 0013462
2019-01-24 19:58 Christian Grothoff Assigned To Bart Polot => Christian Grothoff
2019-01-28 04:34 dvn Note Added: 0013487
2019-01-28 04:51 dvn Note Added: 0013489
2019-01-28 04:52 dvn Note Edited: 0013489
2019-01-28 06:01 Christian Grothoff Note Added: 0013493
2019-01-29 11:32 ch3 Note Added: 0013510
2019-02-09 20:01 Christian Grothoff Note Added: 0013633
2019-02-09 23:32 Christian Grothoff Assigned To Christian Grothoff => ch3
2019-02-09 23:32 Christian Grothoff Status assigned => feedback
2019-02-12 12:27 Christian Grothoff Assigned To ch3 => Christian Grothoff
2019-02-12 12:27 Christian Grothoff Status feedback => assigned
2019-02-12 12:27 Christian Grothoff Note Added: 0013680
2019-02-13 18:12 Christian Grothoff Target Version 0.11.0 => 0.11.1
2019-02-13 18:12 Christian Grothoff Note Added: 0013708
2019-02-13 23:53 Christian Grothoff Assigned To Christian Grothoff =>
2019-02-13 23:53 Christian Grothoff Status assigned => acknowledged
2019-04-03 12:15 Christian Grothoff Target Version 0.11.1 => 0.11.2
2019-04-04 12:44 Christian Grothoff Target Version 0.11.2 => 0.11.3
2019-04-07 14:48 Christian Grothoff Target Version 0.11.3 => 0.11.4