View Issue Details

IDProjectCategoryView StatusLast Update
0004035GNUnettestbed servicepublic2018-06-07 00:24
ReporterSree Harsha TotakuraAssigned ToChristian Grothoff 
PrioritynormalSeveritycrashReproducibilitysometimes
Status closedResolutionfixed 
Product VersionSVN HEAD 
Target Version0.11.0pre66Fixed in Version0.11.0pre66 
Summary0004035: Crash in Testbed
DescriptionAnother Testbed crash. Here, it is configured as a slave controller. May be related to 0004034.
Additional InformationCore was generated by `gnunet-service-testbed -c /tmp/totakura/testbed-helper7FdPnG/0/config'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd9123ac5f8 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0 0x00007fd9123ac5f8 in raise () from /usr/lib/libc.so.6
#1 0x00007fd9123ada7a in abort () from /usr/lib/libc.so.6
#2 0x00007fd9147179c9 in GNUNET_abort_ () at common_logging.c:290
#3 0x000000000041171a in occ_cache_get_handle_ats_occ_cb (cls=<optimized out>,
    ch=<optimized out>, th=<optimized out>, ac=<optimized out>,
    my_identity=<optimized out>) at gnunet-service-testbed_oc.c:680
#4 0x00007fd9147489a6 in run_ready (ws=0x225f6c0, rs=0x225f630) at scheduler.c:587
#5 GNUNET_SCHEDULER_run (task=task@entry=0x7fd91474fb60 <service_task>,
    task_cls=task_cls@entry=0x7fff17780230) at scheduler.c:868
#6 0x00007fd914752f6c in GNUNET_SERVICE_run (argc=<optimized out>,
    argv=<optimized out>, service_name=service_name@entry=0x41a892 "testbed",
    options=options@entry=GNUNET_SERVICE_OPTION_NONE,
    task=task@entry=0x4049e0 <testbed_run>, task_cls=task_cls@entry=0x0)
    at service.c:1503
#7 0x0000000000404119 in main (argc=<optimized out>, argv=<optimized out>)
    at gnunet-service-testbed.c:963
(gdb) bt full
#0 0x00007fd9123ac5f8 in raise () from /usr/lib/libc.so.6
No symbol table info available.
#1 0x00007fd9123ada7a in abort () from /usr/lib/libc.so.6
No symbol table info available.
#2 0x00007fd9147179c9 in GNUNET_abort_ () at common_logging.c:290
No locals.
#3 0x000000000041171a in occ_cache_get_handle_ats_occ_cb (cls=<optimized out>,
    ch=<optimized out>, th=<optimized out>, ac=<optimized out>,
    my_identity=<optimized out>) at gnunet-service-testbed_oc.c:680
        occ = <optimized out>
        lp2c = <optimized out>
        __FUNCTION__ = "occ_cache_get_handle_ats_occ_cb"
#4 0x00007fd9147489a6 in run_ready (ws=0x225f6c0, rs=0x225f630) at scheduler.c:587
        p = GNUNET_SCHEDULER_PRIORITY_DEFAULT
        pos = 0x23ebef0
        tc = {reason = GNUNET_SCHEDULER_REASON_TIMEOUT, read_ready = 0x225f630,
          write_ready = 0x225f6c0}
#5 GNUNET_SCHEDULER_run (task=task@entry=0x7fd91474fb60 <service_task>,
    task_cls=task_cls@entry=0x7fff17780230) at scheduler.c:868
        rs = 0x225f630
        ws = 0x225f6c0
        timeout = <optimized out>
        ret = <optimized out>
        shc_int = 0x225f770
        shc_term = 0x225f830
        shc_quit = 0x225f9b0
        shc_hup = 0x225fa70
        shc_pipe = 0x225f8f0
        last_tr = 5280
        busy_wait_warning = 0
        pr = 0x225f610
        c = 0 '\000'
        __FUNCTION__ = "GNUNET_SCHEDULER_run"
#6 0x00007fd914752f6c in GNUNET_SERVICE_run (argc=<optimized out>,
    argv=<optimized out>, service_name=service_name@entry=0x41a892 "testbed",
    options=options@entry=GNUNET_SERVICE_OPTION_NONE,
    task=task@entry=0x4049e0 <testbed_run>, task_cls=task_cls@entry=0x0)
    at service.c:1503
        err = 0
        ret = <optimized out>
        cfg_fn = 0x2255700 "~/.config/gnunet.conf"
        opt_cfg_fn = 0x2255850 "/tmp/totakura/testbed-helper7FdPnG/0/config"
        loglev = 0x0
        logfile = 0x0
        do_daemonize = 0
        i = <optimized out>
        skew_offset = 140570327512632
---Type <return> to continue, or q <return> to quit---
        skew_variance = 140570327534184
        clock_offset = <optimized out>
        sctx = {cfg = 0x2255720, server = 0x225fe60, addrs = 0x225f510,
          service_name = 0x41a892 "testbed", task = 0x4049e0 <testbed_run>,
          task_cls = 0x0, v4_denied = 0x0, v6_denied = 0x0, v4_allowed = 0x2261dd0,
          v6_allowed = 0x225e470, my_handlers = 0x2255740, addrlens = 0x226ef50,
          lsocks = 0x0, shutdown_task = 0x225fef0, timeout = {
            rel_value_us = 18446744073709551615}, ret = 1, ready_confirm_fd = -1,
          require_found = 1, match_uid = 1, match_gid = 1,
          options = GNUNET_SERVICE_OPTION_NONE}
        cfg = 0x2255720
        xdg = <optimized out>
        service_options = {{shortName = 99 'c', name = 0x7fd91475f723 "config",
            argumentHelp = 0x7fd91475f72a "FILENAME",
            description = 0x7fd91475f7f0 "use configuration file FILENAME",
            require_argument = 1,
            processor = 0x7fd914738ca0 <GNUNET_GETOPT_set_string>,
            scls = 0x7fff17780160}, {shortName = 100 'd',
            name = 0x7fd914760fbb "daemonize", argumentHelp = 0x0,
            description = 0x7fd914761318 "do daemonize (detach from terminal)",
            require_argument = 0,
            processor = 0x7fd914738c90 <GNUNET_GETOPT_set_one>,
            scls = 0x7fff17780154}, {shortName = 104 'h',
            name = 0x7fd91475f73e "help", argumentHelp = 0x0,
            description = 0x7fd91475f733 "print this help", require_argument = 0,
            processor = 0x7fd914738930 <GNUNET_GETOPT_format_help_>, scls = 0x0}, {
            shortName = 76 'L', name = 0x7fd91475f743 "log",
            argumentHelp = 0x7fd91475f747 "LOGLEVEL",
            description = 0x7fd91475f810 "configure logging to use LOGLEVEL",
            require_argument = 1,
            processor = 0x7fd914738ca0 <GNUNET_GETOPT_set_string>,
            scls = 0x7fff17780168}, {shortName = 108 'l',
            name = 0x7fd91475f750 "logfile",
            argumentHelp = 0x7fd91475b484 "LOGFILE",
            description = 0x7fd91475f838 "configure logging to write logs to LOGFILE", require_argument = 1, processor = 0x7fd914738ca0 <GNUNET_GETOPT_set_string>,
            scls = 0x7fff17780170}, {shortName = 118 'v',
            name = 0x7fd91475f758 "version", argumentHelp = 0x0,
            description = 0x7fd91475f760 "print the version number",
            require_argument = 0,
            processor = 0x7fd914738910 <GNUNET_GETOPT_print_version_>,
            scls = 0x7fd91475f779}, {shortName = 0 '\000', name = 0x0,
            argumentHelp = 0x0, description = 0x0, require_argument = 0,
            processor = 0x0, scls = 0x0}}
        __FUNCTION__ = "GNUNET_SERVICE_run"
#7 0x0000000000404119 in main (argc=<optimized out>, argv=<optimized out>)
    at gnunet-service-testbed.c:963
---Type <return> to continue, or q <return> to quit---
No locals.
(gdb)
TagsNo tags attached.

Relationships

related to 0004034 confirmedSree Harsha Totakura Crash while running multi-host experiments 

Activities

Christian Grothoff

2015-10-29 11:12

manager   ~0009831

occ_cache_get_handle_ats_occ_cb is scheduled with the pool after assertiong that occ->type is LOCAL. Inside of it, we check again and then the assertion fails (!?). As with 0004034, my best guess would be memory corruption.

What makes this more odd is that this code path obviously shouldn't run in a multi-host scenario, as it is the code for the 1-host scenario.

Christian Grothoff

2015-10-29 11:31

manager   ~0009832

I've tried to reproduce this using 'localhost' twice in 'hosts.txt' given to the profiler with -H. The bug failed to appear, and it was also valgrind-clean. Strange.

Christian Grothoff

2015-10-29 11:32

manager   ~0009835

Oh, to clarify: I did valgrind gnunet-service-testbed by modifying the helper to spawn valgrind and then start testbed 'indirectly'.

Sree Harsha Totakura

2015-10-29 11:47

developer   ~0009839

I just observed this with a single host experiment

Christian Grothoff

2015-10-29 11:56

manager   ~0009843

Hmm. Here single-host experiments work fine, otherwise I'd not have committed my changes. Could you give me details on the experiment (which config, which command line) so I can hopefully reproduce?

Sree Harsha Totakura

2015-10-29 12:41

developer  

test.conf (2,281 bytes)

Sree Harsha Totakura

2015-10-29 12:42

developer   ~0009844

try running with the attached configuration:
./gnunet-testbed-profiler -c test.conf -p 20

Sree Harsha Totakura

2015-10-29 12:49

developer   ~0009845

For completeness: I tried with version #36615.

Christian Grothoff

2015-10-29 13:59

manager   ~0009849

Reproduced:

==28987== Process terminating with default action of signal 6 (SIGABRT): dumping core
==28987== at 0x6472107: raise (raise.c:56)
==28987== by 0x64734E7: abort (abort.c:89)
==28987== by 0x4E4FA30: GNUNET_abort_ (common_logging.c:290)
==28987== by 0x414F6E: occ_cache_get_handle_ats_occ_cb (gnunet-service-testbed_oc.c:681)
==28987== by 0x41F5C1: connection_ready (gnunet-service-testbed_connectionpool.c:465)
==28987== by 0x4E8C9FC: run_ready (scheduler.c:587)
==28987== by 0x4E8D30E: GNUNET_SCHEDULER_run (scheduler.c:868)
==28987== by 0x4E9BB4A: GNUNET_SERVICE_run (service.c:1503)
==28987== by 0x408181: main (gnunet-service-testbed.c:963)

Christian Grothoff

2015-10-29 14:09

manager   ~0009850

Fixed in SVN 36621.

Issue History

Date Modified Username Field Change
2015-10-28 18:01 Sree Harsha Totakura New Issue
2015-10-28 18:01 Sree Harsha Totakura Status new => assigned
2015-10-28 18:01 Sree Harsha Totakura Assigned To => Sree Harsha Totakura
2015-10-28 18:01 Sree Harsha Totakura Description Updated View Revisions
2015-10-28 18:02 Sree Harsha Totakura Relationship added related to 0004034
2015-10-29 11:12 Christian Grothoff Note Added: 0009831
2015-10-29 11:15 Christian Grothoff Severity minor => crash
2015-10-29 11:31 Christian Grothoff Note Added: 0009832
2015-10-29 11:32 Christian Grothoff Note Added: 0009835
2015-10-29 11:47 Sree Harsha Totakura Note Added: 0009839
2015-10-29 11:47 Sree Harsha Totakura Summary Testbed crashes in multi-host experiments => Crash in Testbed
2015-10-29 11:56 Christian Grothoff Note Added: 0009843
2015-10-29 12:41 Sree Harsha Totakura File Added: test.conf
2015-10-29 12:42 Sree Harsha Totakura Note Added: 0009844
2015-10-29 12:49 Sree Harsha Totakura Note Added: 0009845
2015-10-29 13:59 Christian Grothoff Note Added: 0009849
2015-10-29 14:09 Christian Grothoff Note Added: 0009850
2015-10-29 14:09 Christian Grothoff Status assigned => resolved
2015-10-29 14:09 Christian Grothoff Fixed in Version => 0.11.0pre66
2015-10-29 14:09 Christian Grothoff Resolution open => fixed
2015-10-29 14:09 Christian Grothoff Assigned To Sree Harsha Totakura => Christian Grothoff
2015-10-29 14:09 Christian Grothoff Target Version => 0.11.0pre66
2018-06-07 00:24 Christian Grothoff Status resolved => closed