View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0004035GNUnettestbed servicepublic2015-10-28 18:012018-06-07 00:24
ReporterSree Harsha Totakura Assigned ToChristian Grothoff  
PrioritynormalSeveritycrashReproducibilitysometimes
Status closedResolutionfixed 
Product VersionGit master 
Target Version0.11.0pre66Fixed in Version0.11.0pre66 
Summary0004035: Crash in Testbed
DescriptionAnother Testbed crash. Here, it is configured as a slave controller. May be related to 0004034.
Additional InformationCore was generated by `gnunet-service-testbed -c /tmp/totakura/testbed-helper7FdPnG/0/config'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd9123ac5f8 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0 0x00007fd9123ac5f8 in raise () from /usr/lib/libc.so.6
#1 0x00007fd9123ada7a in abort () from /usr/lib/libc.so.6
#2 0x00007fd9147179c9 in GNUNET_abort_ () at common_logging.c:290
#3 0x000000000041171a in occ_cache_get_handle_ats_occ_cb (cls=<optimized out>,
    ch=<optimized out>, th=<optimized out>, ac=<optimized out>,
    my_identity=<optimized out>) at gnunet-service-testbed_oc.c:680
#4 0x00007fd9147489a6 in run_ready (ws=0x225f6c0, rs=0x225f630) at scheduler.c:587
#5 GNUNET_SCHEDULER_run (task=task@entry=0x7fd91474fb60 <service_task>,
    task_cls=task_cls@entry=0x7fff17780230) at scheduler.c:868
#6 0x00007fd914752f6c in GNUNET_SERVICE_run (argc=<optimized out>,
    argv=<optimized out>, service_name=service_name@entry=0x41a892 "testbed",
    options=options@entry=GNUNET_SERVICE_OPTION_NONE,
    task=task@entry=0x4049e0 <testbed_run>, task_cls=task_cls@entry=0x0)
    at service.c:1503
#7 0x0000000000404119 in main (argc=<optimized out>, argv=<optimized out>)
    at gnunet-service-testbed.c:963
(gdb) bt full
#0 0x00007fd9123ac5f8 in raise () from /usr/lib/libc.so.6
No symbol table info available.
#1 0x00007fd9123ada7a in abort () from /usr/lib/libc.so.6
No symbol table info available.
#2 0x00007fd9147179c9 in GNUNET_abort_ () at common_logging.c:290
No locals.
#3 0x000000000041171a in occ_cache_get_handle_ats_occ_cb (cls=<optimized out>,
    ch=<optimized out>, th=<optimized out>, ac=<optimized out>,
    my_identity=<optimized out>) at gnunet-service-testbed_oc.c:680
        occ = <optimized out>
        lp2c = <optimized out>
        __FUNCTION__ = "occ_cache_get_handle_ats_occ_cb"
#4 0x00007fd9147489a6 in run_ready (ws=0x225f6c0, rs=0x225f630) at scheduler.c:587
        p = GNUNET_SCHEDULER_PRIORITY_DEFAULT
        pos = 0x23ebef0
        tc = {reason = GNUNET_SCHEDULER_REASON_TIMEOUT, read_ready = 0x225f630,
          write_ready = 0x225f6c0}
#5 GNUNET_SCHEDULER_run (task=task@entry=0x7fd91474fb60 <service_task>,
    task_cls=task_cls@entry=0x7fff17780230) at scheduler.c:868
        rs = 0x225f630
        ws = 0x225f6c0
        timeout = <optimized out>
        ret = <optimized out>
        shc_int = 0x225f770
        shc_term = 0x225f830
        shc_quit = 0x225f9b0
        shc_hup = 0x225fa70
        shc_pipe = 0x225f8f0
        last_tr = 5280
        busy_wait_warning = 0
        pr = 0x225f610
        c = 0 '\000'
        __FUNCTION__ = "GNUNET_SCHEDULER_run"
#6 0x00007fd914752f6c in GNUNET_SERVICE_run (argc=<optimized out>,
    argv=<optimized out>, service_name=service_name@entry=0x41a892 "testbed",
    options=options@entry=GNUNET_SERVICE_OPTION_NONE,
    task=task@entry=0x4049e0 <testbed_run>, task_cls=task_cls@entry=0x0)
    at service.c:1503
        err = 0
        ret = <optimized out>
        cfg_fn = 0x2255700 "~/.config/gnunet.conf"
        opt_cfg_fn = 0x2255850 "/tmp/totakura/testbed-helper7FdPnG/0/config"
        loglev = 0x0
        logfile = 0x0
        do_daemonize = 0
        i = <optimized out>
        skew_offset = 140570327512632
---Type <return> to continue, or q <return> to quit---
        skew_variance = 140570327534184
        clock_offset = <optimized out>
        sctx = {cfg = 0x2255720, server = 0x225fe60, addrs = 0x225f510,
          service_name = 0x41a892 "testbed", task = 0x4049e0 <testbed_run>,
          task_cls = 0x0, v4_denied = 0x0, v6_denied = 0x0, v4_allowed = 0x2261dd0,
          v6_allowed = 0x225e470, my_handlers = 0x2255740, addrlens = 0x226ef50,
          lsocks = 0x0, shutdown_task = 0x225fef0, timeout = {
            rel_value_us = 18446744073709551615}, ret = 1, ready_confirm_fd = -1,
          require_found = 1, match_uid = 1, match_gid = 1,
          options = GNUNET_SERVICE_OPTION_NONE}
        cfg = 0x2255720
        xdg = <optimized out>
        service_options = {{shortName = 99 'c', name = 0x7fd91475f723 "config",
            argumentHelp = 0x7fd91475f72a "FILENAME",
            description = 0x7fd91475f7f0 "use configuration file FILENAME",
            require_argument = 1,
            processor = 0x7fd914738ca0 <GNUNET_GETOPT_set_string>,
            scls = 0x7fff17780160}, {shortName = 100 'd',
            name = 0x7fd914760fbb "daemonize", argumentHelp = 0x0,
            description = 0x7fd914761318 "do daemonize (detach from terminal)",
            require_argument = 0,
            processor = 0x7fd914738c90 <GNUNET_GETOPT_set_one>,
            scls = 0x7fff17780154}, {shortName = 104 'h',
            name = 0x7fd91475f73e "help", argumentHelp = 0x0,
            description = 0x7fd91475f733 "print this help", require_argument = 0,
            processor = 0x7fd914738930 <GNUNET_GETOPT_format_help_>, scls = 0x0}, {
            shortName = 76 'L', name = 0x7fd91475f743 "log",
            argumentHelp = 0x7fd91475f747 "LOGLEVEL",
            description = 0x7fd91475f810 "configure logging to use LOGLEVEL",
            require_argument = 1,
            processor = 0x7fd914738ca0 <GNUNET_GETOPT_set_string>,
            scls = 0x7fff17780168}, {shortName = 108 'l',
            name = 0x7fd91475f750 "logfile",
            argumentHelp = 0x7fd91475b484 "LOGFILE",
            description = 0x7fd91475f838 "configure logging to write logs to LOGFILE", require_argument = 1, processor = 0x7fd914738ca0 <GNUNET_GETOPT_set_string>,
            scls = 0x7fff17780170}, {shortName = 118 'v',
            name = 0x7fd91475f758 "version", argumentHelp = 0x0,
            description = 0x7fd91475f760 "print the version number",
            require_argument = 0,
            processor = 0x7fd914738910 <GNUNET_GETOPT_print_version_>,
            scls = 0x7fd91475f779}, {shortName = 0 '\000', name = 0x0,
            argumentHelp = 0x0, description = 0x0, require_argument = 0,
            processor = 0x0, scls = 0x0}}
        __FUNCTION__ = "GNUNET_SERVICE_run"
#7 0x0000000000404119 in main (argc=<optimized out>, argv=<optimized out>)
    at gnunet-service-testbed.c:963
---Type <return> to continue, or q <return> to quit---
No locals.
(gdb)
TagsNo tags attached.
Attached Files
test.conf (2,281 bytes)    
[PATHS]
SERVICEHOME = /tmp/testbed/

[testbed]
AUTOSTART = NO
PORT = 2086
ACCEPT_FROM = 127.0.0.1; 131.159.20.0/24;
HOSTNAME = localhost
MAX_PARALLEL_TOPOLOGY_CONFIG_OPERATIONS = 5
OVERLAY_TOPOLOGY = CLIQUE
OVERLAY_RANDOM_LINKS = 1000
OPERATION_TIMEOUT = 300 s

[nse]
FORCESTART = NO
PORT = 12114
UNIXPATH = /tmp/test-nse-service-nse.unix
BINARY = gnunet-service-nse
#BINARY = /home/mrwiggles/documents/research/gnunet/gnunet-ng/src/nse/.libs/gnunet-service-nse
#PREFIX = valgrind --leak-check=full --log-file=valgrind_nse.%p
AUTOSTART = NO
# Overriding network settings for faster testing (do NOT use
# these values in production just because they are here)
WORKDELAY = 60 s
INTERVAL = 10 s
WORKBITS = 0
PROOFFILE = $SERVICEHOME/nse.proof

[arm]
UNIXPATH = /tmp/test-nse-service-arm.unix

[statistics]
AUTOSTART = NO
PORT = 12115

[fs]
AUTOSTART = NO
FORCESTART = NO

[datastore]
AUTOSTART = NO

[dht]
AUTOSTART = NO
FORCESTART = NO

[nat]
FORCESTART = NO
AUTOSTART = NO
DISABLEV6 = YES
BINDTO = 127.0.0.1
ENABLE_UPNP = NO
BEHIND_NAT = NO
ALLOW_NAT = NO
INTERNAL_ADDRESS = 127.0.0.1
EXTERNAL_ADDRESS = 127.0.0.1

[transport]
plugins = udp
FORCESTART = YES

[transport-udp]
PORT = 12116

[core]
AUTOSTART = YES
FORCESTART = YES

[peerinfo]
AUTOSTART = NO
FORCESTART = NO

[dns]
AUTOSTART = NO

[topology]
AUTOSTART = NO
FORCESTART = NO

[dv]
AUTOSTART = YES

[resolver]
AUTOSTART = NO

[cadet]
AUTOSTART = NO
FORCESTART = NO

[chat]
AUTOSTART = NO

[gns]
AUTOSTART = NO
FORCESTART = NO

[vpn]
AUTOSTART = NO
FORCESTART = NO

[set]
AUTOSTART = NO
FORCESTART = NO

[revocation]
AUTOSTART = NO
FORCESTART = NO

[rps]
AUTOSTART = NO
FORCESTART = NO

[nse-profiler]
OUTPUT_FILE = nse_output_2000_peers.dat
TOPOLOGY_OUTPUT_FILE = nse_topo_2000_peers
DATA_OUTPUT_FILE = nse_stats_2000_peers
ROUND0 = 1000
#ROUND1 = 2000
ROUND2 = 2000
ROUND3 = 2000
ROUND4 = 2000
ROUND5 = 2000
ROUND6 = 2000
ROUND7 = 2000
ROUND8 = 2000
ROUND9 = 2000
ROUND10 = 2000
ROUND11 = 1000
ROUND12 = 1000
ROUND13 = 1000
ROUND14 = 1000
ROUND15 = 1000
ROUND16 = 1000
ROUND17 = 1000
ROUND18 = 1000
ROUND19 = 1000
ROUND20 = 1000
ROUND21 = 2000
ROUND22 = 2000
ROUND23 = 2000
ROUND24 = 2000
ROUND25 = 2000
ROUND26 = 2000
ROUND27 = 2000
ROUND28 = 2000
ROUND29 = 2000
ROUND30 = 2000
WAIT_TIME = 1920 s
CONNECTION_LIMIT = 10
test.conf (2,281 bytes)   

Relationships

related to 0004034 confirmedSree Harsha Totakura Crash while running multi-host experiments 

Activities

Christian Grothoff

2015-10-29 11:12

manager   ~0009831

occ_cache_get_handle_ats_occ_cb is scheduled with the pool after assertiong that occ->type is LOCAL. Inside of it, we check again and then the assertion fails (!?). As with 0004034, my best guess would be memory corruption.

What makes this more odd is that this code path obviously shouldn't run in a multi-host scenario, as it is the code for the 1-host scenario.

Christian Grothoff

2015-10-29 11:31

manager   ~0009832

I've tried to reproduce this using 'localhost' twice in 'hosts.txt' given to the profiler with -H. The bug failed to appear, and it was also valgrind-clean. Strange.

Christian Grothoff

2015-10-29 11:32

manager   ~0009835

Oh, to clarify: I did valgrind gnunet-service-testbed by modifying the helper to spawn valgrind and then start testbed 'indirectly'.

Sree Harsha Totakura

2015-10-29 11:47

developer   ~0009839

I just observed this with a single host experiment

Christian Grothoff

2015-10-29 11:56

manager   ~0009843

Hmm. Here single-host experiments work fine, otherwise I'd not have committed my changes. Could you give me details on the experiment (which config, which command line) so I can hopefully reproduce?

Sree Harsha Totakura

2015-10-29 12:42

developer   ~0009844

try running with the attached configuration:
./gnunet-testbed-profiler -c test.conf -p 20

Sree Harsha Totakura

2015-10-29 12:49

developer   ~0009845

For completeness: I tried with version #36615.

Christian Grothoff

2015-10-29 13:59

manager   ~0009849

Reproduced:

==28987== Process terminating with default action of signal 6 (SIGABRT): dumping core
==28987== at 0x6472107: raise (raise.c:56)
==28987== by 0x64734E7: abort (abort.c:89)
==28987== by 0x4E4FA30: GNUNET_abort_ (common_logging.c:290)
==28987== by 0x414F6E: occ_cache_get_handle_ats_occ_cb (gnunet-service-testbed_oc.c:681)
==28987== by 0x41F5C1: connection_ready (gnunet-service-testbed_connectionpool.c:465)
==28987== by 0x4E8C9FC: run_ready (scheduler.c:587)
==28987== by 0x4E8D30E: GNUNET_SCHEDULER_run (scheduler.c:868)
==28987== by 0x4E9BB4A: GNUNET_SERVICE_run (service.c:1503)
==28987== by 0x408181: main (gnunet-service-testbed.c:963)

Christian Grothoff

2015-10-29 14:09

manager   ~0009850

Fixed in SVN 36621.

Issue History

Date Modified Username Field Change
2015-10-28 18:01 Sree Harsha Totakura New Issue
2015-10-28 18:01 Sree Harsha Totakura Status new => assigned
2015-10-28 18:01 Sree Harsha Totakura Assigned To => Sree Harsha Totakura
2015-10-28 18:01 Sree Harsha Totakura Description Updated
2015-10-28 18:02 Sree Harsha Totakura Relationship added related to 0004034
2015-10-29 11:12 Christian Grothoff Note Added: 0009831
2015-10-29 11:15 Christian Grothoff Severity minor => crash
2015-10-29 11:31 Christian Grothoff Note Added: 0009832
2015-10-29 11:32 Christian Grothoff Note Added: 0009835
2015-10-29 11:47 Sree Harsha Totakura Note Added: 0009839
2015-10-29 11:47 Sree Harsha Totakura Summary Testbed crashes in multi-host experiments => Crash in Testbed
2015-10-29 11:56 Christian Grothoff Note Added: 0009843
2015-10-29 12:41 Sree Harsha Totakura File Added: test.conf
2015-10-29 12:42 Sree Harsha Totakura Note Added: 0009844
2015-10-29 12:49 Sree Harsha Totakura Note Added: 0009845
2015-10-29 13:59 Christian Grothoff Note Added: 0009849
2015-10-29 14:09 Christian Grothoff Note Added: 0009850
2015-10-29 14:09 Christian Grothoff Status assigned => resolved
2015-10-29 14:09 Christian Grothoff Fixed in Version => 0.11.0pre66
2015-10-29 14:09 Christian Grothoff Resolution open => fixed
2015-10-29 14:09 Christian Grothoff Assigned To Sree Harsha Totakura => Christian Grothoff
2015-10-29 14:09 Christian Grothoff Target Version => 0.11.0pre66
2018-06-07 00:24 Christian Grothoff Status resolved => closed