View Issue Details

IDProjectCategoryView StatusLast Update
0003930GNUnetcadet servicepublic2018-06-07 00:24
ReporterChristian GrothoffAssigned ToBart Polot 
PriorityurgentSeveritycrashReproducibilityrandom
Status closedResolutionfixed 
Platformi7OSDebian GNU/LinuxOS Versionsqueeze
Product VersionSVN HEAD 
Target Version0.11.0pre66Fixed in Version0.11.0pre66 
Summary0003930: cadet crash (looks like use after free) in pop_direct_path
DescriptionProgram terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000000042a3d6 in pop_direct_path (peer=0x762da0) at gnunet-service-cadet_peer.c:383
383 if (2 >= iter->length)
(gdb) ba
#0 0x000000000042a3d6 in pop_direct_path (peer=0x762da0) at gnunet-service-cadet_peer.c:383
#1 0x000000000042ab11 in core_disconnect (cls=0x0, peer=0x771650) at gnunet-service-cadet_peer.c:482
#2 0x00007fbda2f1bced in disconnect_and_free_peer_entry (cls=0x74c7d0, key=0x7ffff1f55488, value=0x771600) at core_api.c:389
#3 0x00007fbda2f1f364 in main_notify_handler (cls=0x74c7d0, msg=0x7ffff1f55480) at core_api.c:913
#4 0x00007fbda33490be in receive_task (cls=0x74b910, tc=0x7ffff1f55570) at client.c:623
#5 0x00007fbda338afa7 in run_ready (rs=0x74ac00, ws=0x74ac90) at scheduler.c:587
#6 0x00007fbda338b89b in GNUNET_SCHEDULER_run (task=0x7fbda33982a5 <service_task>, task_cls=0x7ffff1f55910) at scheduler.c:868
#7 0x00007fbda3399fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7ffff1f55ba8, service_name=0x43b1aa "cadet",
    options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503
#8 0x00000000004341c4 in main (argc=3, argv=0x7ffff1f55ba8) at gnunet-service-cadet.c:174
(gdb) print iter
$2 = (struct CadetPeerPath *) 0xf84db057004cc689
(gdb) print *peer
$3 = {id = 16, last_contact = {abs_value_us = 1438687807924839}, path_head = 0x76adc0, path_tail = 0x752440, search_h = 0x76dab0,
  search_delayed = 0x0, tunnel = 0x7725a0, connections = 0x76f510, core_transmit = 0x0, tmt_time = {abs_value_us = 0}, queue_head = 0x0,
  queue_tail = 0x0, queue_n = 0, hello = 0x770870}
TagsNo tags attached.

Relationships

related to 0003906 closedBart Polot segfault in GCP_remove_path 

Activities

Christian Grothoff

2015-08-04 14:35

manager   ~0009530

Another one, same place:

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000430b7e in GCP_remove_path (peer=0x18511c0, path=0x1877d80) at gnunet-service-cadet_peer.c:2241
2241 next = iter->next;
(gdb) ba
#0 0x0000000000430b7e in GCP_remove_path (peer=0x18511c0, path=0x1877d80) at gnunet-service-cadet_peer.c:2241
#1 0x0000000000432d2b in path_destroy_delayed (cls=0x1877d80, tc=0x7fff9026d040) at cadet_path.c:58
#2 0x00007f621f3e2fa7 in run_ready (rs=0x182cc00, ws=0x182cc90) at scheduler.c:587
#3 0x00007f621f3e389b in GNUNET_SCHEDULER_run (task=0x7f621f3f02a5 <service_task>, task_cls=0x7fff9026d3e0) at scheduler.c:868
#4 0x00007f621f3f1fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fff9026d678, service_name=0x43b1aa "cadet",
    options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503
#5 0x00000000004341c4 in main (argc=3, argv=0x7fff9026d678) at gnunet-service-cadet.c:174
(gdb) print iter
$1 = (struct CadetPeerPath *) 0x40
(gdb) print *iter
Cannot access memory at address 0x40
(gdb) ba
#0 0x0000000000430b7e in GCP_remove_path (peer=0x18511c0, path=0x1877d80) at gnunet-service-cadet_peer.c:2241
#1 0x0000000000432d2b in path_destroy_delayed (cls=0x1877d80, tc=0x7fff9026d040) at cadet_path.c:58
#2 0x00007f621f3e2fa7 in run_ready (rs=0x182cc00, ws=0x182cc90) at scheduler.c:587
#3 0x00007f621f3e389b in GNUNET_SCHEDULER_run (task=0x7f621f3f02a5 <service_task>, task_cls=0x7fff9026d3e0) at scheduler.c:868
#4 0x00007f621f3f1fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fff9026d678, service_name=0x43b1aa "cadet",
    options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503
#5 0x00000000004341c4 in main (argc=3, argv=0x7fff9026d678) at gnunet-service-cadet.c:174
(gdb) print *peer
$2 = {id = 42, last_contact = {abs_value_us = 1438689668058425}, path_head = 0x1878710, path_tail = 0x1877d80, search_h = 0x186b780,
  search_delayed = 0x0, tunnel = 0x186d7b0, connections = 0x185d080, core_transmit = 0x0, tmt_time = {abs_value_us = 0},
  queue_head = 0x0, queue_tail = 0x0, queue_n = 0, hello = 0x189de50}

Christian Grothoff

2015-08-04 14:37

manager   ~0009531

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000430b7e in GCP_remove_path (peer=0x13c9d30, path=0x13e10f0) at gnunet-service-cadet_peer.c:2241
2241 next = iter->next;
(gdb) ba
#0 0x0000000000430b7e in GCP_remove_path (peer=0x13c9d30, path=0x13e10f0) at gnunet-service-cadet_peer.c:2241
#1 0x0000000000432d2b in path_destroy_delayed (cls=0x13e10f0, tc=0x7fff6cf87b70) at cadet_path.c:58
#2 0x00007f4995615fa7 in run_ready (rs=0x13b2c00, ws=0x13b2c90) at scheduler.c:587
#3 0x00007f499561689b in GNUNET_SCHEDULER_run (task=0x7f49956232a5 <service_task>, task_cls=0x7fff6cf87f10) at scheduler.c:868
#4 0x00007f4995624fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fff6cf881a8, service_name=0x43b1aa "cadet",
    options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503
#5 0x00000000004341c4 in main (argc=3, argv=0x7fff6cf881a8) at gnunet-service-cadet.c:174
(gdb) print iter
$1 = (struct CadetPeerPath *) 0xdf0adba0df0adba
(gdb) ba
#0 0x0000000000430b7e in GCP_remove_path (peer=0x13c9d30, path=0x13e10f0) at gnunet-service-cadet_peer.c:2241
#1 0x0000000000432d2b in path_destroy_delayed (cls=0x13e10f0, tc=0x7fff6cf87b70) at cadet_path.c:58
#2 0x00007f4995615fa7 in run_ready (rs=0x13b2c00, ws=0x13b2c90) at scheduler.c:587
#3 0x00007f499561689b in GNUNET_SCHEDULER_run (task=0x7f49956232a5 <service_task>, task_cls=0x7fff6cf87f10) at scheduler.c:868
#4 0x00007f4995624fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fff6cf881a8, service_name=0x43b1aa "cadet",
    options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503
#5 0x00000000004341c4 in main (argc=3, argv=0x7fff6cf881a8) at gnunet-service-cadet.c:174

Christian Grothoff

2015-08-04 14:38

manager   ~0009532

Another, possibly related case of use-after-free:

Core was generated by `/home/gnunet9/lib//gnunet/libexec/gnunet-service-cadet -c /home/gnunet9/.config'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000000043182f in GCP_notify_broken_link (peer=0x17146a0, peer1=0x7fffadd9c578, peer2=0x7fffadd9c598)
    at gnunet-service-cadet_peer.c:2556
2556 if ((iter->peers[i] == p1 && iter->peers[i + 1] == p2)
(gdb) ba
#0 0x000000000043182f in GCP_notify_broken_link (peer=0x17146a0, peer1=0x7fffadd9c578, peer2=0x7fffadd9c598)
    at gnunet-service-cadet_peer.c:2556
#1 0x00000000004181a3 in GCC_handle_broken (cls=0x0, id=0x7fffadd9c534, message=0x7fffadd9c554)
    at gnunet-service-cadet_connection.c:2203
#2 0x00007fa6ba3509c5 in main_notify_handler (cls=0x16fe7d0, msg=0x7fffadd9c530) at core_api.c:967
#3 0x00007fa6ba77a0be in receive_task (cls=0x16fd910, tc=0x7fffadd9c680) at client.c:623
#4 0x00007fa6ba7bbfa7 in run_ready (rs=0x16fcc00, ws=0x16fcc90) at scheduler.c:587
#5 0x00007fa6ba7bc89b in GNUNET_SCHEDULER_run (task=0x7fa6ba7c92a5 <service_task>, task_cls=0x7fffadd9ca20) at scheduler.c:868
#6 0x00007fa6ba7cafb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fffadd9ccb8, service_name=0x43b1aa "cadet",
    options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503
#7 0x00000000004341c4 in main (argc=3, argv=0x7fffadd9ccb8) at gnunet-service-cadet.c:174
(gdb) ba
#0 0x000000000043182f in GCP_notify_broken_link (peer=0x17146a0, peer1=0x7fffadd9c578, peer2=0x7fffadd9c598)
    at gnunet-service-cadet_peer.c:2556
#1 0x00000000004181a3 in GCC_handle_broken (cls=0x0, id=0x7fffadd9c534, message=0x7fffadd9c554)
    at gnunet-service-cadet_connection.c:2203
#2 0x00007fa6ba3509c5 in main_notify_handler (cls=0x16fe7d0, msg=0x7fffadd9c530) at core_api.c:967
#3 0x00007fa6ba77a0be in receive_task (cls=0x16fd910, tc=0x7fffadd9c680) at client.c:623
#4 0x00007fa6ba7bbfa7 in run_ready (rs=0x16fcc00, ws=0x16fcc90) at scheduler.c:587
#5 0x00007fa6ba7bc89b in GNUNET_SCHEDULER_run (task=0x7fa6ba7c92a5 <service_task>, task_cls=0x7fffadd9ca20) at scheduler.c:868
#6 0x00007fa6ba7cafb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fffadd9ccb8, service_name=0x43b1aa "cadet",
    options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503
#7 0x00000000004341c4 in main (argc=3, argv=0x7fffadd9ccb8) at gnunet-service-cadet.c:174

amatus

2015-08-25 20:52

developer   ~0009585

I got a similar stacktrace to Grothoff's last one. You'll notice that path 0xa28b7ac0 was allocated then freed 1 minute later, then 4.45 minutes later we try to access it again. In the mean time that same address was allocated and freed by libasan.

Aug 22 08:38:45-237130 cadet-p2p-18275 INFO CONNECTED GN10 <= 29JX
Aug 22 08:38:45-237217 cadet-pth-18275 INFO New path 0xa28b78b0 (2)
Aug 22 08:38:45-237349 cadet-pth-18275 INFO New path 0xa28b7ac0 (2)
Aug 22 08:38:45-237447 cadet-pth-18275 INFO Invalidating path 0xa28b78b0 (2)
Aug 22 08:38:45-237485 cadet-pth-18275 INFO destroying path 0xa28b7ac0 (2)
...
Aug 22 08:39:45-238364 cadet-pth-18275 INFO Destroy delayed 0xa28b78b0 (2)
Aug 22 08:39:45-238462 cadet-pth-18275 INFO destroying path 0xa28b78b0 (2)
...
Aug 22 08:44:12-773526 cadet-pth-18275 INFO Invalidating path 0xabeafcf0 (4)
==18275==ERROR: AddressSanitizer: heap-use-after-free on address 0xa28b78b0 at pc 0x808d41a bp 0xbfcebaf8 sp 0xbfcebaec
READ of size 4 at 0xa28b78b0 thread T0
    #0 0x808d419 in GCP_notify_broken_link /root/gnunet/src/cadet/gnunet-service-cadet_peer.c:2553
    #1 0x8069b57 in GCC_handle_broken /root/gnunet/src/cadet/gnunet-service-cadet_connection.c:2237
    #2 0xb70bb4bc in main_notify_handler /root/gnunet/src/core/core_api.c:967
    #3 0xb710ba06 in receive_task /root/gnunet/src/util/client.c:623
    #4 0xb7194a97 in run_ready /root/gnunet/src/util/scheduler.c:587
    #5 0xb71957b5 in GNUNET_SCHEDULER_run /root/gnunet/src/util/scheduler.c:868
    #6 0xb71b2814 in GNUNET_SERVICE_run /root/gnunet/src/util/service.c:1503
    #7 0x8091246 in main /root/gnunet/src/cadet/gnunet-service-cadet.c:174
    #8 0xb6ed6722 in __libc_start_main (/lib/i386-linux-gnu/libc.so.6+0x19722)
    #9 0x804acf0 (/opt/gnunet/lib/gnunet/libexec/gnunet-service-cadet+0x804acf0)

0xa28b78b0 is located 0 bytes inside of 20-byte region [0xa28b78b0,0xa28b78c4)
freed by thread T0 here:
    #0 0xb72654c4 in free (/usr/lib/i386-linux-gnu/libasan.so.1+0x4e4c4)
    #1 0xb6ce9940 (/lib/i386-linux-gnu/libgcrypt.so.20+0xd940)

previously allocated by thread T0 here:
    #0 0xb72656e4 in malloc (/usr/lib/i386-linux-gnu/libasan.so.1+0x4e6e4)
    #1 0xb6ce97d8 (/lib/i386-linux-gnu/libgcrypt.so.20+0xd7d8)

SUMMARY: AddressSanitizer: heap-use-after-free /root/gnunet/src/cadet/gnunet-service-cadet_peer.c:2553 GCP_notify_broken_link
Shadow bytes around the buggy address:
  0x34516ec0: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x34516ed0: fd fa fa fa fd fd fd fd fa fa fd fd fd fd fa fa
  0x34516ee0: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x34516ef0: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x34516f00: fd fa fa fa fd fd fd fa fa fa fd fd fd fd fa fa
=>0x34516f10: fd fd fd fd fa fa[fd]fd fd fa fa fa fd fd fd fa
  0x34516f20: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x34516f30: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa
  0x34516f40: fd fd fd fa fa fa fd fd fd fd fa fa 00 00 04 fa
  0x34516f50: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x34516f60: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable: 00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone: fa
  Heap right redzone: fb
  Freed heap region: fd
  Stack left redzone: f1
  Stack mid redzone: f2
  Stack right redzone: f3
  Stack partial redzone: f4
  Stack after return: f5
  Stack use after scope: f8
  Global redzone: f9
  Global init order: f6
  Poisoned by user: f7
  Contiguous container OOB:fc
  ASan internal: fe
==18275==ABORTING

Bart Polot

2015-10-02 05:40

manager   ~0009672

Resolved at r36429.

Issue History

Date Modified Username Field Change
2015-08-04 14:34 Christian Grothoff New Issue
2015-08-04 14:34 Christian Grothoff Status new => assigned
2015-08-04 14:34 Christian Grothoff Assigned To => Bart Polot
2015-08-04 14:35 Christian Grothoff Note Added: 0009530
2015-08-04 14:37 Christian Grothoff Note Added: 0009531
2015-08-04 14:38 Christian Grothoff Note Added: 0009532
2015-08-04 14:38 Christian Grothoff Priority normal => urgent
2015-08-04 14:38 Christian Grothoff Reproducibility have not tried => random
2015-08-25 20:52 amatus Note Added: 0009585
2015-10-02 05:40 Bart Polot Relationship added related to 0003906
2015-10-02 05:40 Bart Polot Note Added: 0009672
2015-10-02 05:40 Bart Polot Status assigned => resolved
2015-10-02 05:40 Bart Polot Fixed in Version => SVN HEAD
2015-10-02 05:40 Bart Polot Resolution open => fixed
2015-10-02 14:33 Christian Grothoff Fixed in Version SVN HEAD => 0.11.0pre66
2018-06-07 00:24 Christian Grothoff Status resolved => closed