View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0003718GNUnetcadet servicepublic2015-03-17 16:042018-06-07 00:24
Reporteramatus Assigned ToBart Polot  
PriorityurgentSeveritycrashReproducibilitysometimes
Status closedResolutionduplicate 
Product VersionGit master 
Target Version0.11.0pre66Fixed in Version0.11.0pre66 
Summary0003718: GNUNET_assert (NULL != fc->poll_msg); failed in rev 35365
DescriptionMar 17 12:42:40-894944 cadet-20769 ERROR Assertion failed at gnunet-service-cadet_connection.c:1305.
Steps To Reproducerun the gnunet-rps-profiler in `src/rps' and check for generated coredumps (you have to enable coredumping).
Additional InformationProgram terminated with signal SIGABRT, Aborted.
#0 0xb754cddc in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0xb754cddc in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0xb754e463 in __GI_abort () at abort.c:89
#2 0xb771eafb in GNUNET_abort_ () at common_logging.c:289
#3 0x08053189 in connection_poll (cls=0x9f392b8, tc=0xbf998284)
    at gnunet-service-cadet_connection.c:1305
#4 0xb7752535 in run_ready (rs=0x9f189c0, ws=0x9f18a48) at scheduler.c:587
#5 0xb7752dc3 in GNUNET_SCHEDULER_run (task=0xb775d951 <service_task>,
    task_cls=0xbf9984b0) at scheduler.c:867
#6 0xb775f4ef in GNUNET_SERVICE_run (argc=7, argv=0xbf9986a4,
    service_name=0x8069158 "cadet", options=GNUNET_SERVICE_OPTION_NONE,
    task=0x8065d67 <run>, task_cls=0x0) at service.c:1503
#7 0x08065ff9 in main (argc=7, argv=0xbf9986a4) at gnunet-service-cadet.c:174
(gdb) f 3
#3 0x08053189 in connection_poll (cls=0x9f392b8, tc=0xbf998284)
    at gnunet-service-cadet_connection.c:1305
1305 gnunet-service-cadet_connection.c: No such file or directory.
(gdb) p msg
$1 = {header = {size = 10240, type = 3329}, pid = 0, cid = {
    bits = "\252\027u\267@\202\231\277Ljw\267h\202\231\277\000\000\000\000\003\000\000\000_^\r\000_.hM"}}
(gdb) p fc
$2 = (struct CadetFlowControl *) 0x9f392b8
(gdb) p *fc
$3 = {c = 0x9f39280, queue_n = 0, queue_max = 11, last_pid_sent = 0,
  last_pid_recv = 0, recv_bitmap = 1, last_ack_sent = 11, last_ack_recv = 0,
  poll_task = 0x0, poll_time = {rel_value_us = 4000000}, poll_msg = 0x0,
  ack_msg = 0x0}
(gdb) p *c
$4 = {t = 0x0, fwd_fc = {c = 0x9f39280, queue_n = 0, queue_max = 11,
    last_pid_sent = 0, last_pid_recv = 2, recv_bitmap = 7, last_ack_sent = 10,
    last_ack_recv = 64, poll_task = 0x0, poll_time = {rel_value_us = 1000000},
    poll_msg = 0x0, ack_msg = 0x0}, bck_fc = {c = 0x9f39280, queue_n = 0,
    queue_max = 11, last_pid_sent = 0, last_pid_recv = 0, recv_bitmap = 1,
    last_ack_sent = 11, last_ack_recv = 0, poll_task = 0x0, poll_time = {
      rel_value_us = 4000000}, poll_msg = 0x0, ack_msg = 0x0}, perf = 0x0,
  id = {
    bits = "\375V\317\327E\237Dz'\323d0\243\373\200\244\060\262f\215\323\001w#}\257\203I\362\335j\344"}, state = CADET_CONNECTION_DESTROYED, path = 0x9f3ac08,
  own_pos = 3, fwd_maintenance_task = 0x9f51210,
  bck_maintenance_task = 0x9f6aeb8, maintenance_q = 0x0, create_retry = 1,
  pending_messages = 2, destroy = 1}
TagsNo tags attached.
Attached Files
cadet.log.xz (5,662,848 bytes)

Relationships

has duplicate 0003929 closedBart Polot cadet abort() in assertion failure 

Activities

amatus

2015-07-28 01:06

developer   ~0009494

Hit this one again at rev 36117:

Program terminated with signal SIGABRT, Aborted.
#0 0x00007f9ca9aef107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007f9ca9aef107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f9ca9af04e8 in __GI_abort () at abort.c:89
#2 0x00007f9caafb4296 in GNUNET_abort_ () at common_logging.c:290
#3 0x00000000004151fb in connection_poll (cls=0x1ed5d88, tc=0x7ffe3441fc40)
    at gnunet-service-cadet_connection.c:1438
#4 0x00007f9caaff0909 in run_ready (rs=0x1e61f00, ws=0x1e4ca20) at scheduler.c:587
#5 0x00007f9caaff1214 in GNUNET_SCHEDULER_run (task=0x7f9caaffdcef <service_task>,
    task_cls=0x7ffe3441ffd0) at scheduler.c:867
#6 0x00007f9caafffa28 in GNUNET_SERVICE_run (argc=7, argv=0x7ffe34420268,
    service_name=0x43aefa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433bd9 <run>, task_cls=0x0)
    at service.c:1503
#7 0x0000000000433f48 in main (argc=7, argv=0x7ffe34420268) at gnunet-service-cadet.c:174
(gdb) f 3
#3 0x00000000004151fb in connection_poll (cls=0x1ed5d88, tc=0x7ffe3441fc40)
    at gnunet-service-cadet_connection.c:1438
1438 gnunet-service-cadet_connection.c: No such file or directory.
(gdb) p *fc
$1 = {c = 0x1ed5d80, queue_n = 0, queue_max = 11, last_pid_sent = 0, last_pid_recv = 4294967295,
  recv_bitmap = 0, last_ack_sent = 0, last_ack_recv = 0, poll_task = 0x0, poll_time = {
    rel_value_us = 2000000}, poll_msg = 0x0, ack_msg = 0x0}
(gdb) p *c
$2 = {t = 0x0, fwd_fc = {c = 0x1ed5d80, queue_n = 0, queue_max = 11, last_pid_sent = 0,
    last_pid_recv = 4294967295, recv_bitmap = 0, last_ack_sent = 0, last_ack_recv = 0, poll_task = 0x0,
    poll_time = {rel_value_us = 2000000}, poll_msg = 0x0, ack_msg = 0x0}, bck_fc = {c = 0x1ed5d80,
    queue_n = 0, queue_max = 11, last_pid_sent = 4294967295, last_pid_recv = 0, recv_bitmap = 1,
    last_ack_sent = 10, last_ack_recv = 0, poll_task = 0x0, poll_time = {rel_value_us = 1000000},
    poll_msg = 0x0, ack_msg = 0x1ec0f50}, perf = 0x0, id = {
    bits = "\n\257\031%\177 \375ROdj\314u\230\316TA\373\272u\317\336K\002\335\001\033Z\235z\261b"},
  path = 0x1e6c1e0, fwd_maintenance_task = 0x1ea6070, bck_maintenance_task = 0x1e98270,
  maintenance_q = 0x0, next_peer = 0x0, prev_peer = 0x0, state = CADET_CONNECTION_DESTROYED,
  own_pos = 1, pending_messages = 3, destroy = 1, was_removed = 1, create_retry = 1}

Sree Harsha Totakura

2015-07-28 14:41

developer   ~0009496

This has to be fixed for Julius's thesis evaluations. Hitting this a lot in the in RPS profiler as of now.

amatus

2015-07-28 14:48

developer   ~0009497

I'm sure it would help if you could describe how you are reproducing it. For me it's rare.

Sree Harsha Totakura

2015-07-29 10:06

developer   ~0009501

run the gnunet-rps-profiler in `src/rps' (you may have to issue a `make') and check for the generated coredumps (you have to enable coredumping).

Sree Harsha Totakura

2015-07-29 10:08

developer   ~0009502

latest stack trace:
Core was generated by `/usr/local/lib//gnunet/libexec/gnunet-service-cadet -c /tmp/testbedDPKpRv/23/co'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f5f76747528 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0 0x00007f5f76747528 in raise () from /usr/lib/libc.so.6
#1 0x00007f5f7674893a in abort () from /usr/lib/libc.so.6
#2 0x00007f5f78a922bd in GNUNET_abort_ () at common_logging.c:290
#3 0x000000000041533a in connection_poll (cls=0x13e5010, tc=0x7fff7e8549d0)
    at gnunet-service-cadet_connection.c:1438
#4 0x00007f5f78acea0f in run_ready (rs=0x13c7310, ws=0x13c73a0) at scheduler.c:587
#5 0x00007f5f78acf31f in GNUNET_SCHEDULER_run (task=0x7f5f78adbe3d <service_task>,
    task_cls=0x7fff7e854d60) at scheduler.c:867
#6 0x00007f5f78addb89 in GNUNET_SERVICE_run (argc=3, argv=0x7fff7e854ff8,
    service_name=0x43b342 "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e84 <run>,
    task_cls=0x0) at service.c:1503
#7 0x00000000004341f4 in main (argc=3, argv=0x7fff7e854ff8) at gnunet-service-cadet.c:174

amatus

2015-08-05 17:50

developer   ~0009539

Last edited: 2015-08-05 19:34

 Hit this again in rev 36159:

Program terminated with signal SIGABRT, Aborted.
#0 0x00007f60934d2107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007f60934d2107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f60934d34e8 in __GI_abort () at abort.c:89
#2 0x00007f6094997856 in GNUNET_abort_ () at common_logging.c:290
#3 0x000000000041533f in connection_poll (cls=0xe35788, tc=0x7ffe6d194e00)
    at gnunet-service-cadet_connection.c:1439
#4 0x00007f60949d4263 in run_ready (rs=0xdce7e0, ws=0xdce870) at scheduler.c:587
#5 0x00007f60949d4b6e in GNUNET_SCHEDULER_run (task=0x7f60949e1649 <service_task>,
    task_cls=0x7ffe6d195190) at scheduler.c:867
#6 0x00007f60949e3382 in GNUNET_SERVICE_run (argc=7, argv=0x7ffe6d195428,
    service_name=0x43b1aa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e57 <run>, task_cls=0x0)
    at service.c:1503
#7 0x00000000004341c6 in main (argc=7, argv=0x7ffe6d195428) at gnunet-service-cadet.c:174
(gdb) f 3
#3 0x000000000041533f in connection_poll (cls=0xe35788, tc=0x7ffe6d194e00)
    at gnunet-service-cadet_connection.c:1439
1439 gnunet-service-cadet_connection.c: No such file or directory.
(gdb) p *fc
$1 = {c = 0xe35780, queue_n = 0, queue_max = 11, last_pid_sent = 0, last_pid_recv = 4294967295,
  recv_bitmap = 0, last_ack_sent = 0, last_ack_recv = 0, poll_task = 0x0, poll_time = {
    rel_value_us = 4000000}, poll_msg = 0x0, ack_msg = 0x0}
(gdb) p *c
$2 = {t = 0x0, fwd_fc = {c = 0xe35780, queue_n = 0, queue_max = 11, last_pid_sent = 0,
    last_pid_recv = 4294967295, recv_bitmap = 0, last_ack_sent = 0, last_ack_recv = 0, poll_task = 0x0,
    poll_time = {rel_value_us = 4000000}, poll_msg = 0x0, ack_msg = 0x0}, bck_fc = {c = 0xe35780,
    queue_n = 0, queue_max = 11, last_pid_sent = 4294967295, last_pid_recv = 0, recv_bitmap = 1,
    last_ack_sent = 11, last_ack_recv = 0, poll_task = 0x0, poll_time = {rel_value_us = 1000000},
    poll_msg = 0x0, ack_msg = 0x0}, perf = 0x0, id = {
    bits = ")\016{[\303[qEQ\322\326\345\025\342\213b\317\002\302\345F\352\201T\265D{\327d~\033\017"},
  path = 0xe32490, fwd_maintenance_task = 0xdfd220, bck_maintenance_task = 0xdeb350,
  maintenance_q = 0x0, next_peer = 0x0, prev_peer = 0x0, state = CADET_CONNECTION_DESTROYED,
  own_pos = 2, pending_messages = 2, destroy = 1, was_removed = 1, create_retry = 1}

cadet ERROR log:
Aug 03 18:51:03-259262 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:03-294264 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:03-947910 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:04-566360 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:05-734100 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:05-929670 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:06-283380 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:06-753440 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:09-446600 cadet-p2p-17515 ERROR core wait time 2444521 µs > 1 second
Aug 03 18:51:13-774981 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:32-857192 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:33-469782 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:51:43-498625 cadet-p2p-17515 ERROR core wait time 8150138 µs > 1 second
Aug 03 18:51:55-559977 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:52:05-796135 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_tunnel.c:2082.
Aug 03 18:52:46-613441 cadet-p2p-17515 ERROR core wait time 2453322 µs > 1 second
Aug 03 18:52:47-164922 cadet-17515 ERROR Assertion failed at gnunet-service-cadet_connection.c:1439.

And I just found another instance in the logs:
Aug 04 19:18:05-715180 cadet-p2p-30540 ERROR core wait time 1466055 µs > 1 second
Aug 04 19:18:07-863080 cadet-p2p-30540 ERROR core wait time 1147297 µs > 1 second
Aug 04 19:18:23-377381 cadet-p2p-30540 ERROR core wait time 1204973 µs > 1 second
Aug 04 19:18:25-740518 cadet-p2p-30540 ERROR core wait time 1563106 µs > 1 second
Aug 04 19:18:30-083713 cadet-p2p-30540 ERROR core wait time 4864903 µs > 1 second
Aug 04 19:19:20-737590 cadet-p2p-30540 ERROR core wait time 1261777 µs > 1 second
Aug 04 19:19:46-587324 cadet-p2p-30540 ERROR core wait time 1192873 µs > 1 second
Aug 04 19:20:10-585977 cadet-p2p-30540 ERROR core wait time 2932201 µs > 1 second
Aug 04 19:20:12-738817 cadet-p2p-30540 ERROR core wait time 1085819 µs > 1 second
Aug 04 19:20:19-492288 cadet-p2p-30540 ERROR core wait time 4699865 µs > 1 second
Aug 04 19:20:43-375934 cadet-p2p-30540 ERROR core wait time 1796169 µs > 1 second
Aug 04 19:21:25-544845 cadet-p2p-30540 ERROR core wait time 1141175 µs > 1 second
Aug 04 19:21:30-422918 cadet-p2p-30540 ERROR core wait time 1380466 µs > 1 second
Aug 04 19:21:58-366523 cadet-p2p-30540 ERROR core wait time 1698804 µs > 1 second
Aug 04 19:22:03-073632 cadet-p2p-30540 ERROR core wait time 1501961 µs > 1 second
Aug 04 19:22:17-837186 cadet-p2p-30540 ERROR core wait time 2518684 µs > 1 second
Aug 04 19:22:24-106435 cadet-p2p-30540 ERROR core wait time 2269005 µs > 1 second
Aug 04 19:22:33-142449 cadet-30540 ERROR Assertion failed at gnunet-service-cadet_connection.c:1439.

amatus

2015-08-05 21:06

developer   ~0009541

I'm attaching an INFO level log (last 1,000,000 lines) for cadet at rev 36159.

Issue History

Date Modified Username Field Change
2015-03-17 16:04 amatus New Issue
2015-03-17 16:04 amatus Status new => assigned
2015-03-17 16:04 amatus Assigned To => Bart Polot
2015-03-21 00:38 Christian Grothoff Target Version => 0.11.0
2015-07-28 01:06 amatus Note Added: 0009494
2015-07-28 14:40 Sree Harsha Totakura Priority normal => urgent
2015-07-28 14:41 Sree Harsha Totakura Note Added: 0009496
2015-07-28 14:48 amatus Note Added: 0009497
2015-07-29 10:06 Sree Harsha Totakura Note Added: 0009501
2015-07-29 10:06 Sree Harsha Totakura Reproducibility have not tried => sometimes
2015-07-29 10:06 Sree Harsha Totakura Steps to Reproduce Updated
2015-07-29 10:08 Sree Harsha Totakura Note Added: 0009502
2015-08-05 17:48 amatus Relationship added has duplicate 0003929
2015-08-05 17:50 amatus Note Added: 0009539
2015-08-05 19:32 amatus Note Edited: 0009539
2015-08-05 19:34 amatus Note Edited: 0009539
2015-08-05 21:06 amatus Note Added: 0009541
2015-08-05 21:07 amatus File Added: cadet.log.xz
2015-10-02 05:21 Bart Polot Status assigned => resolved
2015-10-02 05:21 Bart Polot Fixed in Version => Git master
2015-10-02 05:21 Bart Polot Resolution open => duplicate
2015-10-05 18:42 Christian Grothoff Fixed in Version Git master => 0.11.0pre66
2015-10-05 18:42 Christian Grothoff Target Version 0.11.0 => 0.11.0pre66
2018-06-07 00:24 Christian Grothoff Status resolved => closed