View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0003930 | GNUnet | cadet service | public | 2015-08-04 14:34 | 2018-06-07 00:24 |
| Reporter | Christian Grothoff | Assigned To | Bart Polot | ||
| Priority | urgent | Severity | crash | Reproducibility | random |
| Status | closed | Resolution | fixed | ||
| Platform | i7 | OS | Debian GNU/Linux | OS Version | squeeze |
| Product Version | Git master | ||||
| Target Version | 0.11.0pre66 | Fixed in Version | 0.11.0pre66 | ||
| Summary | 0003930: cadet crash (looks like use after free) in pop_direct_path | ||||
| Description | Program terminated with signal SIGSEGV, Segmentation fault. #0 0x000000000042a3d6 in pop_direct_path (peer=0x762da0) at gnunet-service-cadet_peer.c:383 383 if (2 >= iter->length) (gdb) ba #0 0x000000000042a3d6 in pop_direct_path (peer=0x762da0) at gnunet-service-cadet_peer.c:383 #1 0x000000000042ab11 in core_disconnect (cls=0x0, peer=0x771650) at gnunet-service-cadet_peer.c:482 #2 0x00007fbda2f1bced in disconnect_and_free_peer_entry (cls=0x74c7d0, key=0x7ffff1f55488, value=0x771600) at core_api.c:389 #3 0x00007fbda2f1f364 in main_notify_handler (cls=0x74c7d0, msg=0x7ffff1f55480) at core_api.c:913 #4 0x00007fbda33490be in receive_task (cls=0x74b910, tc=0x7ffff1f55570) at client.c:623 #5 0x00007fbda338afa7 in run_ready (rs=0x74ac00, ws=0x74ac90) at scheduler.c:587 #6 0x00007fbda338b89b in GNUNET_SCHEDULER_run (task=0x7fbda33982a5 <service_task>, task_cls=0x7ffff1f55910) at scheduler.c:868 #7 0x00007fbda3399fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7ffff1f55ba8, service_name=0x43b1aa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503 #8 0x00000000004341c4 in main (argc=3, argv=0x7ffff1f55ba8) at gnunet-service-cadet.c:174 (gdb) print iter $2 = (struct CadetPeerPath *) 0xf84db057004cc689 (gdb) print *peer $3 = {id = 16, last_contact = {abs_value_us = 1438687807924839}, path_head = 0x76adc0, path_tail = 0x752440, search_h = 0x76dab0, search_delayed = 0x0, tunnel = 0x7725a0, connections = 0x76f510, core_transmit = 0x0, tmt_time = {abs_value_us = 0}, queue_head = 0x0, queue_tail = 0x0, queue_n = 0, hello = 0x770870} | ||||
| Tags | No tags attached. | ||||
| related to | 0003906 | closed | Bart Polot | segfault in GCP_remove_path |
|
|
Another one, same place: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000430b7e in GCP_remove_path (peer=0x18511c0, path=0x1877d80) at gnunet-service-cadet_peer.c:2241 2241 next = iter->next; (gdb) ba #0 0x0000000000430b7e in GCP_remove_path (peer=0x18511c0, path=0x1877d80) at gnunet-service-cadet_peer.c:2241 #1 0x0000000000432d2b in path_destroy_delayed (cls=0x1877d80, tc=0x7fff9026d040) at cadet_path.c:58 #2 0x00007f621f3e2fa7 in run_ready (rs=0x182cc00, ws=0x182cc90) at scheduler.c:587 #3 0x00007f621f3e389b in GNUNET_SCHEDULER_run (task=0x7f621f3f02a5 <service_task>, task_cls=0x7fff9026d3e0) at scheduler.c:868 #4 0x00007f621f3f1fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fff9026d678, service_name=0x43b1aa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503 #5 0x00000000004341c4 in main (argc=3, argv=0x7fff9026d678) at gnunet-service-cadet.c:174 (gdb) print iter $1 = (struct CadetPeerPath *) 0x40 (gdb) print *iter Cannot access memory at address 0x40 (gdb) ba #0 0x0000000000430b7e in GCP_remove_path (peer=0x18511c0, path=0x1877d80) at gnunet-service-cadet_peer.c:2241 #1 0x0000000000432d2b in path_destroy_delayed (cls=0x1877d80, tc=0x7fff9026d040) at cadet_path.c:58 #2 0x00007f621f3e2fa7 in run_ready (rs=0x182cc00, ws=0x182cc90) at scheduler.c:587 #3 0x00007f621f3e389b in GNUNET_SCHEDULER_run (task=0x7f621f3f02a5 <service_task>, task_cls=0x7fff9026d3e0) at scheduler.c:868 #4 0x00007f621f3f1fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fff9026d678, service_name=0x43b1aa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503 #5 0x00000000004341c4 in main (argc=3, argv=0x7fff9026d678) at gnunet-service-cadet.c:174 (gdb) print *peer $2 = {id = 42, last_contact = {abs_value_us = 1438689668058425}, path_head = 0x1878710, path_tail = 0x1877d80, search_h = 0x186b780, search_delayed = 0x0, tunnel = 0x186d7b0, connections = 0x185d080, core_transmit = 0x0, tmt_time = {abs_value_us = 0}, queue_head = 0x0, queue_tail = 0x0, queue_n = 0, hello = 0x189de50} |
|
|
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000430b7e in GCP_remove_path (peer=0x13c9d30, path=0x13e10f0) at gnunet-service-cadet_peer.c:2241 2241 next = iter->next; (gdb) ba #0 0x0000000000430b7e in GCP_remove_path (peer=0x13c9d30, path=0x13e10f0) at gnunet-service-cadet_peer.c:2241 #1 0x0000000000432d2b in path_destroy_delayed (cls=0x13e10f0, tc=0x7fff6cf87b70) at cadet_path.c:58 #2 0x00007f4995615fa7 in run_ready (rs=0x13b2c00, ws=0x13b2c90) at scheduler.c:587 #3 0x00007f499561689b in GNUNET_SCHEDULER_run (task=0x7f49956232a5 <service_task>, task_cls=0x7fff6cf87f10) at scheduler.c:868 #4 0x00007f4995624fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fff6cf881a8, service_name=0x43b1aa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503 #5 0x00000000004341c4 in main (argc=3, argv=0x7fff6cf881a8) at gnunet-service-cadet.c:174 (gdb) print iter $1 = (struct CadetPeerPath *) 0xdf0adba0df0adba (gdb) ba #0 0x0000000000430b7e in GCP_remove_path (peer=0x13c9d30, path=0x13e10f0) at gnunet-service-cadet_peer.c:2241 #1 0x0000000000432d2b in path_destroy_delayed (cls=0x13e10f0, tc=0x7fff6cf87b70) at cadet_path.c:58 #2 0x00007f4995615fa7 in run_ready (rs=0x13b2c00, ws=0x13b2c90) at scheduler.c:587 #3 0x00007f499561689b in GNUNET_SCHEDULER_run (task=0x7f49956232a5 <service_task>, task_cls=0x7fff6cf87f10) at scheduler.c:868 #4 0x00007f4995624fb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fff6cf881a8, service_name=0x43b1aa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503 #5 0x00000000004341c4 in main (argc=3, argv=0x7fff6cf881a8) at gnunet-service-cadet.c:174 |
|
|
Another, possibly related case of use-after-free: Core was generated by `/home/gnunet9/lib//gnunet/libexec/gnunet-service-cadet -c /home/gnunet9/.config'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x000000000043182f in GCP_notify_broken_link (peer=0x17146a0, peer1=0x7fffadd9c578, peer2=0x7fffadd9c598) at gnunet-service-cadet_peer.c:2556 2556 if ((iter->peers[i] == p1 && iter->peers[i + 1] == p2) (gdb) ba #0 0x000000000043182f in GCP_notify_broken_link (peer=0x17146a0, peer1=0x7fffadd9c578, peer2=0x7fffadd9c598) at gnunet-service-cadet_peer.c:2556 #1 0x00000000004181a3 in GCC_handle_broken (cls=0x0, id=0x7fffadd9c534, message=0x7fffadd9c554) at gnunet-service-cadet_connection.c:2203 #2 0x00007fa6ba3509c5 in main_notify_handler (cls=0x16fe7d0, msg=0x7fffadd9c530) at core_api.c:967 #3 0x00007fa6ba77a0be in receive_task (cls=0x16fd910, tc=0x7fffadd9c680) at client.c:623 #4 0x00007fa6ba7bbfa7 in run_ready (rs=0x16fcc00, ws=0x16fcc90) at scheduler.c:587 #5 0x00007fa6ba7bc89b in GNUNET_SCHEDULER_run (task=0x7fa6ba7c92a5 <service_task>, task_cls=0x7fffadd9ca20) at scheduler.c:868 #6 0x00007fa6ba7cafb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fffadd9ccb8, service_name=0x43b1aa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503 #7 0x00000000004341c4 in main (argc=3, argv=0x7fffadd9ccb8) at gnunet-service-cadet.c:174 (gdb) ba #0 0x000000000043182f in GCP_notify_broken_link (peer=0x17146a0, peer1=0x7fffadd9c578, peer2=0x7fffadd9c598) at gnunet-service-cadet_peer.c:2556 #1 0x00000000004181a3 in GCC_handle_broken (cls=0x0, id=0x7fffadd9c534, message=0x7fffadd9c554) at gnunet-service-cadet_connection.c:2203 #2 0x00007fa6ba3509c5 in main_notify_handler (cls=0x16fe7d0, msg=0x7fffadd9c530) at core_api.c:967 #3 0x00007fa6ba77a0be in receive_task (cls=0x16fd910, tc=0x7fffadd9c680) at client.c:623 #4 0x00007fa6ba7bbfa7 in run_ready (rs=0x16fcc00, ws=0x16fcc90) at scheduler.c:587 #5 0x00007fa6ba7bc89b in GNUNET_SCHEDULER_run (task=0x7fa6ba7c92a5 <service_task>, task_cls=0x7fffadd9ca20) at scheduler.c:868 #6 0x00007fa6ba7cafb7 in GNUNET_SERVICE_run (argc=3, argv=0x7fffadd9ccb8, service_name=0x43b1aa "cadet", options=GNUNET_SERVICE_OPTION_NONE, task=0x433e55 <run>, task_cls=0x0) at service.c:1503 #7 0x00000000004341c4 in main (argc=3, argv=0x7fffadd9ccb8) at gnunet-service-cadet.c:174 |
|
|
I got a similar stacktrace to Grothoff's last one. You'll notice that path 0xa28b7ac0 was allocated then freed 1 minute later, then 4.45 minutes later we try to access it again. In the mean time that same address was allocated and freed by libasan. Aug 22 08:38:45-237130 cadet-p2p-18275 INFO CONNECTED GN10 <= 29JX Aug 22 08:38:45-237217 cadet-pth-18275 INFO New path 0xa28b78b0 (2) Aug 22 08:38:45-237349 cadet-pth-18275 INFO New path 0xa28b7ac0 (2) Aug 22 08:38:45-237447 cadet-pth-18275 INFO Invalidating path 0xa28b78b0 (2) Aug 22 08:38:45-237485 cadet-pth-18275 INFO destroying path 0xa28b7ac0 (2) ... Aug 22 08:39:45-238364 cadet-pth-18275 INFO Destroy delayed 0xa28b78b0 (2) Aug 22 08:39:45-238462 cadet-pth-18275 INFO destroying path 0xa28b78b0 (2) ... Aug 22 08:44:12-773526 cadet-pth-18275 INFO Invalidating path 0xabeafcf0 (4) ==18275==ERROR: AddressSanitizer: heap-use-after-free on address 0xa28b78b0 at pc 0x808d41a bp 0xbfcebaf8 sp 0xbfcebaec READ of size 4 at 0xa28b78b0 thread T0 #0 0x808d419 in GCP_notify_broken_link /root/gnunet/src/cadet/gnunet-service-cadet_peer.c:2553 #1 0x8069b57 in GCC_handle_broken /root/gnunet/src/cadet/gnunet-service-cadet_connection.c:2237 #2 0xb70bb4bc in main_notify_handler /root/gnunet/src/core/core_api.c:967 #3 0xb710ba06 in receive_task /root/gnunet/src/util/client.c:623 #4 0xb7194a97 in run_ready /root/gnunet/src/util/scheduler.c:587 #5 0xb71957b5 in GNUNET_SCHEDULER_run /root/gnunet/src/util/scheduler.c:868 #6 0xb71b2814 in GNUNET_SERVICE_run /root/gnunet/src/util/service.c:1503 #7 0x8091246 in main /root/gnunet/src/cadet/gnunet-service-cadet.c:174 #8 0xb6ed6722 in __libc_start_main (/lib/i386-linux-gnu/libc.so.6+0x19722) #9 0x804acf0 (/opt/gnunet/lib/gnunet/libexec/gnunet-service-cadet+0x804acf0) 0xa28b78b0 is located 0 bytes inside of 20-byte region [0xa28b78b0,0xa28b78c4) freed by thread T0 here: #0 0xb72654c4 in free (/usr/lib/i386-linux-gnu/libasan.so.1+0x4e4c4) #1 0xb6ce9940 (/lib/i386-linux-gnu/libgcrypt.so.20+0xd940) previously allocated by thread T0 here: #0 0xb72656e4 in malloc (/usr/lib/i386-linux-gnu/libasan.so.1+0x4e6e4) #1 0xb6ce97d8 (/lib/i386-linux-gnu/libgcrypt.so.20+0xd7d8) SUMMARY: AddressSanitizer: heap-use-after-free /root/gnunet/src/cadet/gnunet-service-cadet_peer.c:2553 GCP_notify_broken_link Shadow bytes around the buggy address: 0x34516ec0: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x34516ed0: fd fa fa fa fd fd fd fd fa fa fd fd fd fd fa fa 0x34516ee0: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x34516ef0: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x34516f00: fd fa fa fa fd fd fd fa fa fa fd fd fd fd fa fa =>0x34516f10: fd fd fd fd fa fa[fd]fd fd fa fa fa fd fd fd fa 0x34516f20: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x34516f30: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa 0x34516f40: fd fd fd fa fa fa fd fd fd fd fa fa 00 00 04 fa 0x34516f50: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x34516f60: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Contiguous container OOB:fc ASan internal: fe ==18275==ABORTING |
|
|
Resolved at r36429. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2015-08-04 14:34 | Christian Grothoff | New Issue | |
| 2015-08-04 14:34 | Christian Grothoff | Status | new => assigned |
| 2015-08-04 14:34 | Christian Grothoff | Assigned To | => Bart Polot |
| 2015-08-04 14:35 | Christian Grothoff | Note Added: 0009530 | |
| 2015-08-04 14:37 | Christian Grothoff | Note Added: 0009531 | |
| 2015-08-04 14:38 | Christian Grothoff | Note Added: 0009532 | |
| 2015-08-04 14:38 | Christian Grothoff | Priority | normal => urgent |
| 2015-08-04 14:38 | Christian Grothoff | Reproducibility | have not tried => random |
| 2015-08-25 20:52 | amatus | Note Added: 0009585 | |
| 2015-10-02 05:40 | Bart Polot | Relationship added | related to 0003906 |
| 2015-10-02 05:40 | Bart Polot | Note Added: 0009672 | |
| 2015-10-02 05:40 | Bart Polot | Status | assigned => resolved |
| 2015-10-02 05:40 | Bart Polot | Fixed in Version | => Git master |
| 2015-10-02 05:40 | Bart Polot | Resolution | open => fixed |
| 2015-10-02 14:33 | Christian Grothoff | Fixed in Version | Git master => 0.11.0pre66 |
| 2018-06-07 00:24 | Christian Grothoff | Status | resolved => closed |
