View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0004588GNUnetcadet servicepublic2016-06-23 21:452018-06-07 00:24
Reporterch3 Assigned ToBart Polot  
PrioritynormalSeveritycrashReproducibilitysometimes
Status closedResolutionfixed 
Product VersionGit master 
Target Version0.11.0pre66Fixed in Version0.11.0pre66 
Summary0004588: Assertion failed at mq.c:295
Description(gdb) bt
#0 0x00007f3c1a538295 in raise () from /usr/lib/libc.so.6
#1 0x00007f3c1a5396da in abort () from /usr/lib/libc.so.6
#2 0x00007f3c1c8b5c7d in GNUNET_abort_ () at common_logging.c:291
#3 0x00007f3c1c8e2e61 in GNUNET_MQ_send (mq=0x0, ev=0x832410) at mq.c:295
#4 0x00007f3c1c27df9a in GNUNET_CORE_notify_transmit_ready (handle=0x819cc0, cork=0,
    priority=GNUNET_CORE_PRIO_CRITICAL_CONTROL, maxdelay=..., target=0x81eab0, notify_size=140,
    notify=0x42e0bd <queue_send>, notify_cls=0x831b40) at core_api.c:1014
#5 0x000000000042fa1d in GCP_queue_add (peer=0x831b40, cls=0x832660, type=258, payload_type=65535,
    payload_id=0, size=100, c=0x833280, fwd=0, cont=0x41351c <conn_message_sent>, cont_cls=0x82aa60)
    at gnunet-service-cadet_peer.c:1559
#6 0x000000000041d6cd in GCC_send_prebuilt_message (message=0x7ffe37a5f180, payload_type=65535,
    payload_id=0, c=0x833280, fwd=0, force=1, cont=0x0, cont_cls=0x0)
    at gnunet-service-cadet_connection.c:3606
#7 0x00000000004149d1 in send_broken (c=0x833280, id1=0x642b80 <my_full_id>, id2=0x81ede0, fwd=0)
    at gnunet-service-cadet_connection.c:1033
#8 0x000000000041c566 in GCC_neighbor_disconnected (c=0x833280, peer=0x8300d0)
    at gnunet-service-cadet_connection.c:3350
#9 0x000000000042b9a8 in notify_broken (cls=0x8300d0, key=0x7ffe37a5f2e0, value=0x833280)
    at gnunet-service-cadet_peer.c:381
#10 0x00007f3c1c8c6c94 in GNUNET_CONTAINER_multihashmap_iterate (map=0x816f10,
    it=0x42b8db <notify_broken>, it_cls=0x8300d0) at container_multihashmap.c:358
#11 0x000000000042c1d3 in core_disconnect (cls=0x0, peer=0x81b6d0) at gnunet-service-cadet_peer.c:504
#12 0x00007f3c1c27acce in disconnect_and_free_peer_entry (cls=0x819cc0, key=0x7ffe37a5f440,
    value=0x81b690) at core_api.c:273
#13 0x00007f3c1c8c8377 in GNUNET_CONTAINER_multipeermap_iterate (map=0x819d60,
    it=0x7f3c1c27ac7d <disconnect_and_free_peer_entry>, it_cls=0x819cc0) at container_multipeermap.c:361
#14 0x00007f3c1c27b0d6 in reconnect_later (h=0x819cc0) at core_api.c:314
#15 0x00007f3c1c27b153 in handle_mq_error (cls=0x819cc0, error=GNUNET_MQ_ERROR_READ) at core_api.c:335
#16 0x00007f3c1c8e2cdf in GNUNET_MQ_inject_error (mq=0x81acf0, error=GNUNET_MQ_ERROR_READ) at mq.c:272
#17 0x00007f3c1c8e3f1f in handle_client_message (cls=0x81acf0, msg=0x0) at mq.c:605
#18 0x00007f3c1c8b00dc in receive_helper (cls=0x81ac50, buf=0x7ffe37a5f5f0, available=0, addr=0x81aaf0,
    addrlen=110, errCode=0) at client.c:577
#19 0x00007f3c1c8bede7 in receive_ready (cls=0x81ab70) at connection.c:1189
#20 0x00007f3c1c8f33e9 in run_ready (rs=0x81d920, ws=0x81d9b0) at scheduler.c:620
#21 0x00007f3c1c8f3d30 in GNUNET_SCHEDULER_run (task=0x7f3c1c900e3c <service_task>,
    task_cls=0x7ffe37a6fa00) at scheduler.c:887
#22 0x00007f3c1c902b6f in GNUNET_SERVICE_run (argc=3, argv=0x7ffe37a6fc98,
    service_name=0x43cc62 "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x435c36 <run>,
    task_cls=0x0) at service.c:1497
#23 0x0000000000435f9e in main (argc=3, argv=0x7ffe37a6fc98) at gnunet-service-cadet.c:173
Steps To Reproducerun rps testcases
Additional Information(gdb) bt f
#0 0x00007f3c1a538295 in raise () from /usr/lib/libc.so.6
No symbol table info available.
#1 0x00007f3c1a5396da in abort () from /usr/lib/libc.so.6
No symbol table info available.
#2 0x00007f3c1c8b5c7d in GNUNET_abort_ () at common_logging.c:291
No locals.
#3 0x00007f3c1c8e2e61 in GNUNET_MQ_send (mq=0x0, ev=0x832410) at mq.c:295
        __FUNCTION__ = "GNUNET_MQ_send"
#4 0x00007f3c1c27df9a in GNUNET_CORE_notify_transmit_ready (handle=0x819cc0, cork=0,
    priority=GNUNET_CORE_PRIO_CRITICAL_CONTROL, maxdelay=..., target=0x81eab0, notify_size=140,
    notify=0x42e0bd <queue_send>, notify_cls=0x831b40) at core_api.c:1014
        pr = 0x832a20
        th = 0x832a28
        smr = 0x832440
        env = 0x832410
        __FUNCTION__ = "GNUNET_CORE_notify_transmit_ready"
#5 0x000000000042fa1d in GCP_queue_add (peer=0x831b40, cls=0x832660, type=258, payload_type=65535,
    payload_id=0, size=100, c=0x833280, fwd=0, cont=0x41351c <conn_message_sent>, cont_cls=0x82aa60)
    at gnunet-service-cadet_peer.c:1559
        q = 0x833010
        priority = 0
        call_core = 1
        __FUNCTION__ = "GCP_queue_add"
#6 0x000000000041d6cd in GCC_send_prebuilt_message (message=0x7ffe37a5f180, payload_type=65535,
    payload_id=0, c=0x833280, fwd=0, force=1, cont=0x0, cont_cls=0x0)
    at gnunet-service-cadet_connection.c:3606
        fc = 0x8332d0
        q = 0x82aa60
        data = 0x832660
        size = 100
        type = 258
        droppable = 0
        __FUNCTION__ = "GCC_send_prebuilt_message"
#7 0x00000000004149d1 in send_broken (c=0x833280, id1=0x642b80 <my_full_id>, id2=0x81ede0, fwd=0)
    at gnunet-service-cadet_connection.c:1033
        msg = {header = {size = 25600, type = 513}, cid = {
            bits = "&D+\237\331\307\037\227\234\376\270\306Q\342\f\032\224T~y3\346\362\214\233\277`.\257|A\035"}, peer1 = {public_key = {
              q_y = "\327\213!\005\tFI\377\vK\245\222\370\340\377\336(\254\335\366\376aO\005K\343\341D\347\337\065\002"}}, peer2 = {public_key = {
              q_y = "\327\213!\005\tFI\377\vK\245\222\370\340\377\336(\254\335\366\376aO\005K\343\341D\347\337\065\002"}}}
        __FUNCTION__ = "send_broken"
#8 0x000000000041c566 in GCC_neighbor_disconnected (c=0x833280, peer=0x8300d0)
    at gnunet-service-cadet_connection.c:3350
        fc = 0x833288
        hop = 0x831b40
        peer_name = "TY5J", '\000' <repeats 11 times>
        fwd = 0
        __FUNCTION__ = "GCC_neighbor_disconnected"
#9 0x000000000042b9a8 in notify_broken (cls=0x8300d0, key=0x7ffe37a5f2e0, value=0x833280)
    at gnunet-service-cadet_peer.c:381
        peer = 0x8300d0
        c = 0x833280
        __FUNCTION__ = "notify_broken"
#10 0x00007f3c1c8c6c94 in GNUNET_CONTAINER_multihashmap_iterate (map=0x816f10,
    it=0x42b8db <notify_broken>, it_cls=0x8300d0) at container_multihashmap.c:358
        bme = 0x833110
---Type <return> to continue, or q <return> to quit---
        nxt = 0x0
        count = 0
        i = 6
        me = {sme = 0x833110, bme = 0x833110}
        kc = {bits = {2670412838, 2535442393, 3334012572, 437051985, 2038322324, 2364728883, 778092443,
            490831023, 0, 0, 0, 0, 0, 0, 0, 0}}
        __FUNCTION__ = "GNUNET_CONTAINER_multihashmap_iterate"
#11 0x000000000042c1d3 in core_disconnect (cls=0x0, peer=0x81b6d0) at gnunet-service-cadet_peer.c:504
        p = 0x8300d0
        direct_path = 0x81f9e0
        own_id = "TY5J", '\000' <repeats 11 times>
        __FUNCTION__ = "core_disconnect"
#12 0x00007f3c1c27acce in disconnect_and_free_peer_entry (cls=0x819cc0, key=0x7ffe37a5f440,
    value=0x81b690) at core_api.c:273
        h = 0x819cc0
        th = 0x832668
        pr = 0x81b690
        __FUNCTION__ = "disconnect_and_free_peer_entry"
#13 0x00007f3c1c8c8377 in GNUNET_CONTAINER_multipeermap_iterate (map=0x819d60,
    it=0x7f3c1c27ac7d <disconnect_and_free_peer_entry>, it_cls=0x819cc0) at container_multipeermap.c:361
        bme = 0x8218d0
        nxt = 0x0
        count = 2
        i = 87
        me = {sme = 0x8218d0, bme = 0x8218d0}
        kc = {public_key = {
            q_y = "\327\213!\005\tFI\377\vK\245\222\370\340\377\336(\254\335\366\376aO\005K\343\341D\347\337\065\002"}}
        __FUNCTION__ = "GNUNET_CONTAINER_multipeermap_iterate"
#14 0x00007f3c1c27b0d6 in reconnect_later (h=0x819cc0) at core_api.c:314
        __FUNCTION__ = "reconnect_later"
#15 0x00007f3c1c27b153 in handle_mq_error (cls=0x819cc0, error=GNUNET_MQ_ERROR_READ) at core_api.c:335
        h = 0x819cc0
#16 0x00007f3c1c8e2cdf in GNUNET_MQ_inject_error (mq=0x81acf0, error=GNUNET_MQ_ERROR_READ) at mq.c:272
        __FUNCTION__ = "GNUNET_MQ_inject_error"
#17 0x00007f3c1c8e3f1f in handle_client_message (cls=0x81acf0, msg=0x0) at mq.c:605
        mq = 0x81acf0
        state = 0x81ae30
#18 0x00007f3c1c8b00dc in receive_helper (cls=0x81ac50, buf=0x7ffe37a5f5f0, available=0, addr=0x81aaf0,
    addrlen=110, errCode=0) at client.c:577
        client = 0x81ac50
        remaining = {rel_value_us = 18446744073709551615}
        receive_handler = 0x7f3c1c8e3ee3 <handle_client_message>
        receive_handler_cls = 0x81acf0
        __FUNCTION__ = "receive_helper"
#19 0x00007f3c1c8bede7 in receive_ready (cls=0x81ab70) at connection.c:1189
        connection = 0x81ab70
        tc = 0x7f3c1cb216b0 <tc>
        buffer = "\000(\000K\000\224\000(\200\337\061i\n\245\244\b\224\246\277]\355\351\334 \022\273\016q\263 \224\353f6\321_+!\233\271\000\000zi\000\000\000\002\224\226\061L\001%\257\002/Gcc\f\261\262\327", '\000' <repeats 32 times>, "\067\252\271\320\332\324\314G\252\tk\300$\211\\,\224\226\061L\001%\257\002/Gcc\f\261\262\327\000h\000\021\000\000\000\000\067\252\271\320\332\324\314G\252\tk\300$\211\\,\224\226\061L\001%\257\002/Gcc\f\261\262\327unix\000\000\061\000\005\065\376\r\370\304\004\000\000\000\000\000\000\000)/tmp/test"...
        ret = 0
        receiver = 0x7f3c1c8afe44 <receive_helper>
        __FUNCTION__ = "receive_ready"
#20 0x00007f3c1c8f33e9 in run_ready (rs=0x81d920, ws=0x81d9b0) at scheduler.c:620
---Type <return> to continue, or q <return> to quit---
        p = GNUNET_SCHEDULER_PRIORITY_DEFAULT
        pos = 0x8341a0
        __FUNCTION__ = "run_ready"
#21 0x00007f3c1c8f3d30 in GNUNET_SCHEDULER_run (task=0x7f3c1c900e3c <service_task>,
    task_cls=0x7ffe37a6fa00) at scheduler.c:887
        rs = 0x81d920
        ws = 0x81d9b0
        timeout = {rel_value_us = 190043}
        ret = 5
        shc_int = 0x81e350
        shc_term = 0x81e410
        shc_quit = 0x81e590
        shc_hup = 0x81e650
        shc_pipe = 0x81e4d0
        last_tr = 789
        busy_wait_warning = 0
        pr = 0x815dc0
        c = 91 '['
        __FUNCTION__ = "GNUNET_SCHEDULER_run"
#22 0x00007f3c1c902b6f in GNUNET_SERVICE_run (argc=3, argv=0x7ffe37a6fc98,
    service_name=0x43cc62 "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x435c36 <run>,
    task_cls=0x0) at service.c:1497
        err = 0
        ret = 3
        cfg_fn = 0x813700 "~/.config/gnunet.conf"
        opt_cfg_fn = 0x813850 "/tmp/testbedCAkKiJ/4/config"
        loglev = 0x0
        logfile = 0x0
        do_daemonize = 0
        i = 0
        skew_offset = 0
        skew_variance = 0
        clock_offset = 3
        sctx = {cfg = 0x813720, server = 0x81e900, addrs = 0x0, service_name = 0x43cc62 "cadet",
          task = 0x435c36 <run>, task_cls = 0x0, v4_denied = 0x0, v6_denied = 0x0,
          v4_allowed = 0x824c10, v6_allowed = 0x81c500, my_handlers = 0x815af0, addrlens = 0x0,
          lsocks = 0x82cd60, shutdown_task = 0x81e990, timeout = {rel_value_us = 18446744073709551615},
          ret = 1, ready_confirm_fd = -1, require_found = 1, match_uid = 1, match_gid = 1,
          options = GNUNET_SERVICE_OPTION_NONE}
        cfg = 0x813720
        xdg = 0x0
        service_options = {{shortName = 99 'c', name = 0x7f3c1c91318d "config",
            argumentHelp = 0x7f3c1c913194 "FILENAME",
            description = 0x7f3c1c9131a0 "use configuration file FILENAME", require_argument = 1,
            processor = 0x7f3c1c8dffb9 <GNUNET_GETOPT_set_string>, scls = 0x7ffe37a6fab8}, {
            shortName = 100 'd', name = 0x7f3c1c9131c0 "daemonize", argumentHelp = 0x0,
            description = 0x7f3c1c9131d0 "do daemonize (detach from terminal)", require_argument = 0,
            processor = 0x7f3c1c8dff8c <GNUNET_GETOPT_set_one>, scls = 0x7ffe37a6faa4}, {
            shortName = 104 'h', name = 0x7f3c1c9131f4 "help", argumentHelp = 0x0,
            description = 0x7f3c1c9131f9 "print this help", require_argument = 0,
            processor = 0x7f3c1c8dfa22 <GNUNET_GETOPT_format_help_>, scls = 0x0}, {shortName = 76 'L',
            name = 0x7f3c1c913209 "log", argumentHelp = 0x7f3c1c91320d "LOGLEVEL",
            description = 0x7f3c1c913218 "configure logging to use LOGLEVEL", require_argument = 1,
            processor = 0x7f3c1c8dffb9 <GNUNET_GETOPT_set_string>, scls = 0x7ffe37a6fab0}, {
            shortName = 108 'l', name = 0x7f3c1c91323a "logfile",
            argumentHelp = 0x7f3c1c913242 "LOGFILE",
            description = 0x7f3c1c913250 "configure logging to write logs to LOGFILE",
            require_argument = 1, processor = 0x7f3c1c8dffb9 <GNUNET_GETOPT_set_string>,
---Type <return> to continue, or q <return> to quit---
            scls = 0x7ffe37a6faa8}, {shortName = 118 'v', name = 0x7f3c1c91327b "version",
            argumentHelp = 0x0, description = 0x7f3c1c913283 "print the version number",
            require_argument = 0, processor = 0x7f3c1c8df9dc <GNUNET_GETOPT_print_version_>,
            scls = 0x7f3c1c91329c}, {shortName = 0 '\000', name = 0x0, argumentHelp = 0x0,
            description = 0x0, require_argument = 0, processor = 0x0, scls = 0x0}}
        __FUNCTION__ = "GNUNET_SERVICE_run"
#23 0x0000000000435f9e in main (argc=3, argv=0x7ffe37a6fc98) at gnunet-service-cadet.c:173
        r = 0
TagsNo tags attached.

Activities

Christian Grothoff

2016-06-23 22:01

manager   ~0010928

21:57 <grothoff> Yes, core_api tells cadet: we got disconnected, all peers are now no longer connected.
21:58 <grothoff> while we are doing that iteration, cadet tries to queue another message -- to a peer that core knows is gone, but cadet not yet because we're currently notifying about disconnects!
21:58 <grothoff> Before, CORE would have queued the message and then told cadet 1ns later that the peer disconnected.
21:58 <grothoff> Right now, CORE crashes.
21:59 <grothoff> So CADET is not nice in that it does call CORE within a callback from CORE, and CORE is not nice in that it crashes this way ;-)

Christian Grothoff

2016-06-23 22:03

manager   ~0010929

As of SVN 37340, we give a GNUNET_break() if this happens and NTR returns NULL
(which may or may not be liked by clients...).

CADET should be changed to _not_ do anything immediately during the 'disconnect' notification callback. CADET should start a new task and in that new task it can try NTR. That way, CORE can tell CADET that all connections are down and we'll survive this way more nicely...

Christian Grothoff

2016-06-29 21:22

manager   ~0010947

Hacky fix in SVN 37421.

Issue History

Date Modified Username Field Change
2016-06-23 21:45 ch3 New Issue
2016-06-23 21:45 ch3 Status new => assigned
2016-06-23 21:45 ch3 Assigned To => Bart Polot
2016-06-23 22:01 Christian Grothoff Note Added: 0010928
2016-06-23 22:03 Christian Grothoff Note Added: 0010929
2016-06-29 21:22 Christian Grothoff Note Added: 0010947
2016-06-29 21:22 Christian Grothoff Fixed in Version => 0.11.0pre66
2016-06-29 21:22 Christian Grothoff Target Version => 0.11.0pre66
2016-06-29 23:20 Christian Grothoff Status assigned => resolved
2016-06-29 23:20 Christian Grothoff Resolution open => fixed
2018-06-07 00:24 Christian Grothoff Status resolved => closed