View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0003690GNUnettransport servicepublic2015-02-27 16:242018-06-07 00:25
Reporteramatus Assigned ToChristian Grothoff  
PriorityhighSeveritycrashReproducibilityhave not tried
Status closedResolutionfixed 
Product VersionGit master 
Target Version0.11.0pre66Fixed in Version0.11.0pre66 
Summary0003690: Assertion failed at gnunet-service-transport_ats.c:306. NULL == ai
DescriptionTransport crashed on my node running svn rev 35298 and generated a core file. See the additional information section for the backtrace. This happened not long after I started gnunet and I also noticed that ATS was eating a lot of CPU. This might be caused by issue 3689 "ATS running with 100% CPU load".
Additional InformationProgram terminated with signal SIGABRT, Aborted.
#0 0xb74a9ddc in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt full
#0 0xb74a9ddc in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        resultvar = <optimized out>
        resultvar = <optimized out>
        pid = -1218535424
        selftid = 18357
#1 0xb74ab463 in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0},
          sa_mask = {__val = {0, 0, 543319366, 3077672960, 3077409632, 1,
              3216776064, 3077598367, 3077410072, 3074038808, 1, 1, 0, 0, 1,
              1, 3216776036, 3077262124, 3076790636, 3076780588, 3216776052,
              134674160, 134515516, 3074957588, 159203088, 3076852602,
              134616291, 3077260276, 3216776232, 176646872, 159203088,
              3077623840}}, sa_flags = 0,
          sa_restorer = 0xb74ab320 <__GI_abort>}
        sigs = {__val = {32, 0 <repeats 31 times>}}
#2 0xb76501b4 in GNUNET_abort () at common_logging.c:289
No locals.
#3 0x0804cdcf in GST_ats_block_address (address=0x97d3f10, session=0xa876ad8)
    at gnunet-service-transport_ats.c:306
        ai = 0x0
        __FUNCTION__ = "GST_ats_block_address"
#4 0x0805342f in free_address (na=0xa1b53ac)
    at gnunet-service-transport_neighbours.c:681
No locals.
#5 0x08058cb8 in GST_neighbours_session_terminated (peer=0x98450d8,
    session=0xa876ad8) at gnunet-service-transport_neighbours.c:3351
        __FUNCTION__ = "GST_neighbours_session_terminated"
#6 0x0804b303 in plugin_env_session_end (cls=0x9786678, address=0x98450d8,
    session=0xa876ad8) at gnunet-service-transport.c:572
        __FUNCTION__ = "plugin_env_session_end"
#7 0xb6c6d2b5 in http_client_plugin_session_disconnect (cls=0x97bf390,
    s=0xa876ad8) at plugin_transport_http_client.c:848
        plugin = 0x97bf390
#8 0xb6c6e509 in client_run (cls=0x97bf390, tc=0xbfbc1d34)
    at plugin_transport_http_client.c:1428
        easy_h = <optimized out>
        s = 0xa876ad8
        d = 0xa876ad8 "\300\363`\n\330P\204\t\220\363{\t\001"
        running = 2
        mret = CURLM_OK
        msg = <optimized out>
        put_request = 0
        http_statuscode = 0
        msgs_left = 1
        plugin = 0x97bf390
#9 0xb767d49e in run_ready (ws=<optimized out>, rs=<optimized out>)
    at scheduler.c:587
        p = GNUNET_SCHEDULER_PRIORITY_DEFAULT
        pos = 0x98e6410
        tc = {
          reason = (GNUNET_SCHEDULER_REASON_TIMEOUT | GNUNET_SCHEDULER_REASON_PREREQ_DONE), read_ready = 0xa196668, write_ready = 0x98bcd58}
#10 GNUNET_SCHEDULER_run (task=0xb7683460 <service_task>, task_cls=0xbfbc1e34)
    at scheduler.c:816
        timeout = {rel_value_us = 25769803776}
        ret = 0
        c = -73 '\267'
        __FUNCTION__ = "GNUNET_SCHEDULER_run"
#11 0xb76866c9 in GNUNET_SERVICE_run (argc=5, argv=0xbfbc2044,
    service_name=0x8061523 "transport", options=GNUNET_SERVICE_OPTION_NONE,
    task=0x0, task_cls=0x0) at service.c:1503
        err = -1078190452
        ret = 0
        cfg_fn = 0x9785658 "~/.config/gnunet.conf"
        opt_cfg_fn = 0x9785718 "/var/lib/gnunet/.config/gnunet.conf"
        loglev = 0x9785740 "ERROR"
        logfile = 0x0
        do_daemonize = 0
        skew_offset = 13218100385836308896
        skew_variance = 13815950673495195665
        sctx = {cfg = 0x9785678, server = 0x9794878, addrs = 0x0,
          service_name = 0x8061523 "transport", task = 0x804ad50 <run>,
          task_cls = 0x0, v4_denied = 0x0, v6_denied = 0x0,
          v4_allowed = 0x9794570, v6_allowed = 0x97949d8,
          my_handlers = 0x97856e0, addrlens = 0x0, lsocks = 0x978fda0,
          shutdown_task = 0x9785a88, timeout = {
            rel_value_us = 18446744073709551615}, ret = 1,
          ready_confirm_fd = -1, require_found = 1, match_uid = 0,
          match_gid = 1, options = GNUNET_SERVICE_OPTION_NONE}
        cfg = 0x9785678
        xdg = 0x0
        service_options = {{shortName = 99 'c', name = 0xb7690803 "config",
            argumentHelp = 0xb769080a "FILENAME",
            description = 0xb76908cc "use configuration file FILENAME",
            require_argument = 1,
            processor = 0xb766e670 <GNUNET_GETOPT_set_string>,
            scls = 0xbfbc1d9c}, {shortName = 100 'd',
            name = 0xb76914fb "daemonize", argumentHelp = 0x0,
            description = 0xb76917dc "do daemonize (detach from terminal)",
            require_argument = 0,
            processor = 0xb766e660 <GNUNET_GETOPT_set_one>,
            scls = 0xbfbc1da8}, {shortName = 104 'h',
            name = 0xb769081e "help", argumentHelp = 0x0,
            description = 0xb7690813 "print this help", require_argument = 0,
            processor = 0xb766e2d0 <GNUNET_GETOPT_format_help_>, scls = 0x0}, {
            shortName = 76 'L', name = 0xb7690823 "log",
            argumentHelp = 0xb7690827 "LOGLEVEL",
            description = 0xb76908ec "configure logging to use LOGLEVEL",
            require_argument = 1,
            processor = 0xb766e670 <GNUNET_GETOPT_set_string>,
            scls = 0xbfbc1da0}, {shortName = 108 'l',
            name = 0xb7690830 "logfile", argumentHelp = 0xb768d943 "LOGFILE",
            description = 0xb7690910 "configure logging to write logs to LOGFILE", require_argument = 1, processor = 0xb766e670 <GNUNET_GETOPT_set_string>,
            scls = 0xbfbc1da4}, {shortName = 118 'v',
            name = 0xb7690838 "version", argumentHelp = 0x0,
            description = 0xb7690840 "print the version number",
            require_argument = 0,
            processor = 0xb766e2a0 <GNUNET_GETOPT_print_version_>,
            scls = 0xb7690859}, {shortName = 0 '\000', name = 0x0,
            argumentHelp = 0x0, description = 0x0, require_argument = 0,
            processor = 0x0, scls = 0x0}}
        __FUNCTION__ = "GNUNET_SERVICE_run"
#12 0x0804aad9 in main (argc=5, argv=0xbfbc2044)
    at gnunet-service-transport.c:925
No locals.
TagsNo tags attached.

Relationships

related to 0003689 closedChristian Grothoff ATS running with 100% CPU load 
related to 0003719 closedChristian Grothoff GNUNET_assert (NULL == ai); failed in rev 35365 

Activities

Christian Grothoff

2015-02-28 14:21

manager   ~0008929

Added additional assertions to help narrow this down in SVN 35302.

Christian Grothoff

2015-03-02 21:39

manager   ~0008969

Do you still see this?

amatus

2015-03-03 04:57

developer   ~0008972

I hit it again at rev 35316. The cadet service crashed at the same time due to 0003696. The backtrace is different this time:

#0 0x00007fbb69acf107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fbb69ad04e8 in __GI_abort () at abort.c:89
#2 0x00007fbb6a15b996 in GNUNET_abort_ () at common_logging.c:289
#3 0x0000000000406051 in GST_ats_block_address (address=0x27eea00, session=0x1524310)
    at gnunet-service-transport_ats.c:306
#4 0x000000000040e9f2 in free_address (na=0x155f108) at gnunet-service-transport_neighbours.c:681
#5 0x000000000040f483 in unset_primary_address (n=0x155f0f0)
    at gnunet-service-transport_neighbours.c:940
#6 0x000000000040f701 in free_neighbour (n=0x155f0f0) at gnunet-service-transport_neighbours.c:977
#7 0x0000000000415934 in master_task (cls=0x155f0f0, tc=0x7ffc8a311a50)
    at gnunet-service-transport_neighbours.c:3085
#8 0x00007fbb6a1925e4 in run_ready (rs=0x142ba30, ws=0x1416960) at scheduler.c:587
#9 0x00007fbb6a192e87 in GNUNET_SCHEDULER_run (task=0x7fbb6a19e3ca <service_task>,
    task_cls=0x7ffc8a311de0) at scheduler.c:816
#10 0x00007fbb6a19fff2 in GNUNET_SERVICE_run (argc=7, argv=0x7ffc8a312068,
    service_name=0x41e92c "transport", options=GNUNET_SERVICE_OPTION_NONE, task=0x40560d <run>,
    task_cls=0x0) at service.c:1503
#11 0x0000000000405bd4 in main (argc=7, argv=0x7ffc8a312068) at gnunet-service-transport.c:925

The transport log shows:
Mar 02 20:36:46-473405 transport-4773 ERROR Assertion failed at gnunet-service-transport_ats.c:696.
Mar 02 20:36:47-019925 transport-4773 ERROR Assertion failed at gnunet-service-transport_ats.c:306.

amatus

2015-03-03 19:43

developer   ~0008978

Hit this again at rev 35339, it happened twice 6 min. apart, no accompanying cadet crash.

Mar 03 09:26:26-391276 transport-23689 ERROR Assertion failed at gnunet-service-transport_ats.c:696.
Mar 03 11:06:26-798724 transport-23689 ERROR Assertion failed at gnunet-service-transport_ats.c:696.
Mar 03 11:06:26-832257 transport-23689 ERROR Assertion failed at gnunet-service-transport_ats.c:306.
Mar 03 11:12:26-939251 transport-23971 ERROR Assertion failed at gnunet-service-transport_ats.c:696.
Mar 03 11:12:26-969665 transport-23971 ERROR Assertion failed at gnunet-service-transport_ats.c:696.
Mar 03 11:12:26-969783 transport-23971 ERROR Assertion failed at gnunet-service-transport_ats.c:696.
Mar 03 11:12:26-970572 transport-23971 ERROR Assertion failed at gnunet-service-transport_ats.c:306.

Christian Grothoff

2015-03-03 22:50

manager   ~0008979

How long is your peer running at this time? About 12h by chance?

Christian Grothoff

2015-03-03 22:59

manager   ~0008980

Assuming my speculation is correct, SVN 35340 should fix this one. Please let me know.

Christian Grothoff

2015-03-08 12:56

manager   ~0009000

ATS runs without problems for a while now on my peers.

Issue History

Date Modified Username Field Change
2015-02-27 16:24 amatus New Issue
2015-02-27 16:24 amatus Status new => assigned
2015-02-27 16:24 amatus Assigned To => Matthias Wachs
2015-02-28 14:07 Christian Grothoff Relationship added related to 0003689
2015-02-28 14:21 Christian Grothoff Note Added: 0008929
2015-02-28 14:21 Christian Grothoff Assigned To Matthias Wachs => Christian Grothoff
2015-02-28 14:55 Christian Grothoff Priority normal => urgent
2015-02-28 18:34 Christian Grothoff Status assigned => feedback
2015-03-02 21:39 Christian Grothoff Note Added: 0008969
2015-03-02 21:42 Christian Grothoff Assigned To Christian Grothoff =>
2015-03-02 21:42 Christian Grothoff Priority urgent => high
2015-03-03 04:57 amatus Note Added: 0008972
2015-03-03 04:57 amatus Status feedback => new
2015-03-03 19:43 amatus Note Added: 0008978
2015-03-03 22:50 Christian Grothoff Note Added: 0008979
2015-03-03 22:59 Christian Grothoff Note Added: 0008980
2015-03-03 22:59 Christian Grothoff Assigned To => Christian Grothoff
2015-03-03 22:59 Christian Grothoff Status new => feedback
2015-03-03 22:59 Christian Grothoff Target Version => 0.11.0pre66
2015-03-08 12:56 Christian Grothoff Note Added: 0009000
2015-03-08 12:56 Christian Grothoff Status feedback => resolved
2015-03-08 12:56 Christian Grothoff Fixed in Version => 0.11.0pre66
2015-03-08 12:56 Christian Grothoff Resolution open => fixed
2015-03-17 16:25 amatus Relationship added related to 0003719
2018-06-07 00:25 Christian Grothoff Status resolved => closed