View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0004817libmicrohttpdthread poolpublic2016-11-26 11:402017-02-14 19:02
Reporterbeberking Assigned ToChristian Grothoff  
PrioritynormalSeveritymajorReproducibilityhave not tried
Status closedResolutionopen 
PlatformAmazon EC2 with TSX extensionsOSDebian GNU/Linux 
Product Version0.9.51 
Target Version0.9.53Fixed in Version0.9.52 
Summary0004817: Test failure with TSX extension
DescriptionHi,

MHD test_timeout fail on a CPU with TSX extension. This could be caused by a double unlocking somewhere, which is not handled differently with TSX extensions.

It has been reported that building with -fsanitize=thread -no-pie may help to identify the issue (this currently reports a data race in test_daemon).

Please see the Debian bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844146

Cheers,
Bertrand
TagsNo tags attached.

Activities

Christian Grothoff

2016-11-29 14:37

manager   ~0011554

I don't have access to a CPU with TSX, which would obviously be helpful here. Anyone who could help with that?

beberking

2016-12-04 21:56

developer   ~0011566

I don't either, unfortunately. IMHO, fixing the data race condition could be a good start (I wouldn't know how).

Once this is done, I could ask to the Debian community to test a patched version of libmicrohttpd against a CPU with TSX.

Cheers,
Bertrand

Christian Grothoff

2016-12-10 10:56

manager   ~0011585

-fsanitize=thread requires a 64-bit system, and I'll be on 32 bit until 2017...

Christian Grothoff

2017-02-13 22:12

manager   ~0011741

Got my hands on a Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz. flags from /proc/cpuinfo are: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap, so I don't see "tsx" in there, even though AFAIK the model should support TSX!?? (Should I see the flag there?). Still investigating.

Christian Grothoff

2017-02-13 22:38

manager   ~0011742

http://ark.intel.com/products/91772/Intel-Xeon-Processor-E5-2660-v4-35M-Cache-2_00-GHz says this one supports TSX -- barely, E5-2660-v3 does not!

Christian Grothoff

2017-02-13 23:00

manager   ~0011743

Ok, I'm now getting perf_get_concurrent testcase failures on the TSX system. However, I have an ancient gcc where -fsanitize=thread doesn't work. Will try to get that fixed next.

Christian Grothoff

2017-02-14 02:12

manager   ~0011744

Ok, finally getting "something":

christian@298dbad95bcc:~/libmicrohttpd/src/microhttpd$ more test_daemon.log
==================
WARNING: ThreadSanitizer: data race (pid=8171)
  Read of size 4 at 0x7d500000ff74 by thread T1:
    #0 MHD_select_thread /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4053 (libmicrohttpd.so.12+0x000000015980)
    #1 named_thread_starter /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:281 (libmicrohttpd.so.12+0x00000001d55d)
    #2 <null> <null> (libtsan.so.0+0x000000024459)

  Previous write of size 4 at 0x7d500000ff74 by main thread:
    #0 MHD_stop_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5744 (libmicrohttpd.so.12+0x00000001a387)
    #1 testStartStop /home/christian/libmicrohttpd/src/microhttpd/test_daemon.c:84 (test_daemon+0x000000000bfc)
    #2 main /home/christian/libmicrohttpd/src/microhttpd/test_daemon.c:160 (test_daemon+0x000000000eb1)

  Location is heap block of size 488 at 0x7d500000fe00 allocated by main thread:
    #0 calloc <null> (libtsan.so.0+0x000000026b41)
    #1 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4862 (libmicrohttpd.so.12+0x000000018013)
    #2 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #3 testStartStop /home/christian/libmicrohttpd/src/microhttpd/test_daemon.c:78 (test_daemon+0x000000000bda)
    #4 main /home/christian/libmicrohttpd/src/microhttpd/test_daemon.c:160 (test_daemon+0x000000000eb1)

  Thread T1 'MHD-single' (tid=8186, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 MHD_create_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:211 (libmicrohttpd.so.12+0x00000001d484)
    #2 MHD_create_named_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:356 (libmicrohttpd.so.12+0x00000001d
64b)
    #3 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5368 (libmicrohttpd.so.12+0x000000019254)
    #4 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #5 testStartStop /home/christian/libmicrohttpd/src/microhttpd/test_daemon.c:78 (test_daemon+0x000000000bda)
    #6 main /home/christian/libmicrohttpd/src/microhttpd/test_daemon.c:160 (test_daemon+0x000000000eb1)

Christian Grothoff

2017-02-14 02:12

manager   ~0011745

$ more test_upgrade.log
==================
WARNING: ThreadSanitizer: data race (pid=8194)
  Write of size 4 at 0x7d4c0000df68 by thread T2:
    #0 MHD_upgrade_action /home/christian/libmicrohttpd/src/microhttpd/response.c:670 (libmicrohttpd.so.12+0x00000001ea21)
    #1 run_usock /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:704 (test_upgrade+0x00000000223b)
    #2 <null> <null> (libtsan.so.0+0x000000024459)

  Previous write of size 4 at 0x7d4c0000df68 by main thread:
    #0 MHD_connection_handle_idle /home/christian/libmicrohttpd/src/microhttpd/connection.c:3019 (libmicrohttpd.so.12+0x0000000
0a884)
    #1 call_handlers /home/christian/libmicrohttpd/src/microhttpd/daemon.c:925 (libmicrohttpd.so.12+0x00000000d702)
    #2 MHD_run_from_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3049 (libmicrohttpd.so.12+0x000000012cf3)
    #3 run_mhd_select_loop /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:911 (test_upgrade+0x000000002730)
    #4 run_mhd_loop /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:986 (test_upgrade+0x000000002a3b)
    #5 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1072 (test_upgrade+0x000000002c59)
    #6 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1146 (test_upgrade+0x000000002e24)

  Location is heap block of size 392 at 0x7d4c0000de40 allocated by main thread:
    #0 calloc <null> (libtsan.so.0+0x000000026b41)
    #1 internal_add_connection /home/christian/libmicrohttpd/src/microhttpd/daemon.c:2109 (libmicrohttpd.so.12+0x00000000f21a)
    #2 MHD_accept_connection /home/christian/libmicrohttpd/src/microhttpd/daemon.c:2758 (libmicrohttpd.so.12+0x000000011c2c)
    #3 MHD_run_from_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3037 (libmicrohttpd.so.12+0x000000012b99)
    #4 run_mhd_select_loop /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:911 (test_upgrade+0x000000002730)
    #5 run_mhd_loop /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:986 (test_upgrade+0x000000002a3b)
    #6 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1072 (test_upgrade+0x000000002c59)
    #7 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1146 (test_upgrade+0x000000002e24)

  Thread T2 (tid=8210, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 upgrade_cb /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:792 (test_upgrade+0x0000000023cd)
    #2 MHD_response_execute_upgrade_ /home/christian/libmicrohttpd/src/microhttpd/response.c:925 (libmicrohttpd.so.12+0x0000000
1ec6b)
    #3 MHD_connection_handle_idle /home/christian/libmicrohttpd/src/microhttpd/connection.c:3011 (libmicrohttpd.so.12+0x0000000
0a857)
    #4 call_handlers /home/christian/libmicrohttpd/src/microhttpd/daemon.c:925 (libmicrohttpd.so.12+0x00000000d702)
    #5 MHD_run_from_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3049 (libmicrohttpd.so.12+0x000000012cf3)
    #6 run_mhd_select_loop /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:911 (test_upgrade+0x000000002730)
    #7 run_mhd_loop /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:986 (test_upgrade+0x000000002a3b)
    #8 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1072 (test_upgrade+0x000000002c59)
    #9 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1146 (test_upgrade+0x000000002e24)

SUMMARY: ThreadSanitizer: data race /home/christian/libmicrohttpd/src/microhttpd/response.c:670 in MHD_upgrade_action
==================
==================
WARNING: ThreadSanitizer: data race (pid=8194)
  Write of size 4 at 0x7f10eb61d1f0 by thread T1:
    #0 run_usock_client /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:731 (test_upgrade+0x00000000233a)
    #1 <null> <null> (libtsan.so.0+0x000000024459)

  Previous read of size 4 at 0x7f10eb61d1f0 by main thread:
    #0 run_mhd_select_loop /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:884 (test_upgrade+0x00000000273c)
    #1 run_mhd_loop /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:986 (test_upgrade+0x000000002a3b)
    #2 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1072 (test_upgrade+0x000000002c59)
    #3 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1146 (test_upgrade+0x000000002e24)

  Location is global 'done' of size 4 at 0x7f10eb61d1f0 (test_upgrade+0x0000002051f0)

  Thread T1 (tid=8209, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1066 (test_upgrade+0x000000002c35)
    #2 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1146 (test_upgrade+0x000000002e24)

SUMMARY: ThreadSanitizer: data race /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:731 in run_usock_client
==================
==================
WARNING: ThreadSanitizer: data race (pid=8194)
  Write of size 4 at 0x7d4c0001bf68 by thread T10:
    #0 MHD_upgrade_action /home/christian/libmicrohttpd/src/microhttpd/response.c:670 (libmicrohttpd.so.12+0x00000001ea21)
    #1 run_usock /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:704 (test_upgrade+0x00000000223b)
    #2 <null> <null> (libtsan.so.0+0x000000024459)

  Previous write of size 4 at 0x7d4c0001bf68 by thread T9:
    #0 MHD_connection_handle_idle /home/christian/libmicrohttpd/src/microhttpd/connection.c:3019 (libmicrohttpd.so.12+0x0000000
0a884)
    #1 call_handlers /home/christian/libmicrohttpd/src/microhttpd/daemon.c:925 (libmicrohttpd.so.12+0x00000000d702)
    #2 thread_main_handle_connection /home/christian/libmicrohttpd/src/microhttpd/daemon.c:1653 (libmicrohttpd.so.12+0x00000000
e174)
    #3 named_thread_starter /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:281 (libmicrohttpd.so.12+0x00000001d55d)
    #4 <null> <null> (libtsan.so.0+0x000000024459)

  Location is heap block of size 392 at 0x7d4c0001be40 allocated by thread T7:
    #0 calloc <null> (libtsan.so.0+0x000000026b41)
    #1 internal_add_connection /home/christian/libmicrohttpd/src/microhttpd/daemon.c:2109 (libmicrohttpd.so.12+0x00000000f21a)
    #2 MHD_accept_connection /home/christian/libmicrohttpd/src/microhttpd/daemon.c:2758 (libmicrohttpd.so.12+0x000000011c2c)
    #3 MHD_run_from_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3037 (libmicrohttpd.so.12+0x000000012b99)
    #4 MHD_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3240 (libmicrohttpd.so.12+0x00000001348e)
    #5 MHD_select_thread /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4062 (libmicrohttpd.so.12+0x000000015962)
    #6 named_thread_starter /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:281 (libmicrohttpd.so.12+0x00000001d55d)
    #7 <null> <null> (libtsan.so.0+0x000000024459)

  Thread T10 (tid=8218, running) created by thread T9 at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 upgrade_cb /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:792 (test_upgrade+0x0000000023cd)
    #2 MHD_response_execute_upgrade_ /home/christian/libmicrohttpd/src/microhttpd/response.c:925 (libmicrohttpd.so.12+0x0000000
1ec6b)
    #3 MHD_connection_handle_idle /home/christian/libmicrohttpd/src/microhttpd/connection.c:3011 (libmicrohttpd.so.12+0x0000000
0a857)
    #4 call_handlers /home/christian/libmicrohttpd/src/microhttpd/daemon.c:925 (libmicrohttpd.so.12+0x00000000d702)
    #5 thread_main_handle_connection /home/christian/libmicrohttpd/src/microhttpd/daemon.c:1653 (libmicrohttpd.so.12+0x00000000
e174)
    #6 named_thread_starter /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:281 (libmicrohttpd.so.12+0x00000001d55d)
    #7 <null> <null> (libtsan.so.0+0x000000024459)

  Thread T9 'MHD-connection' (tid=8217, finished) created by thread T7 at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 MHD_create_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:211 (libmicrohttpd.so.12+0x00000001d484)
    #2 MHD_create_named_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:356 (libmicrohttpd.so.12+0x00000001d
64b)
    #3 internal_add_connection /home/christian/libmicrohttpd/src/microhttpd/daemon.c:2251 (libmicrohttpd.so.12+0x00000000fa74)
    #4 MHD_accept_connection /home/christian/libmicrohttpd/src/microhttpd/daemon.c:2758 (libmicrohttpd.so.12+0x000000011c2c)
    #5 MHD_run_from_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3037 (libmicrohttpd.so.12+0x000000012b99)
    #6 MHD_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3240 (libmicrohttpd.so.12+0x00000001348e)
    #7 MHD_select_thread /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4062 (libmicrohttpd.so.12+0x000000015962)
    #8 named_thread_starter /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:281 (libmicrohttpd.so.12+0x00000001d55d)
    #9 <null> <null> (libtsan.so.0+0x000000024459)

  Thread T7 'MHD-listen' (tid=8215, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 MHD_create_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:211 (libmicrohttpd.so.12+0x00000001d484)
    #2 MHD_create_named_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:356 (libmicrohttpd.so.12+0x00000001d
64b)
    #3 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5368 (libmicrohttpd.so.12+0x000000019254)
    #4 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #5 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1011 (test_upgrade+0x000000002af2)
    #6 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1174 (test_upgrade+0x000000002f68)

SUMMARY: ThreadSanitizer: data race /home/christian/libmicrohttpd/src/microhttpd/response.c:670 in MHD_upgrade_action
==================
==================
WARNING: ThreadSanitizer: data race (pid=8194)
  Write of size 4 at 0x7d500000ff74 by main thread:
    #0 MHD_stop_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5744 (libmicrohttpd.so.12+0x00000001a387)
    #1 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1081 (test_upgrade+0x000000002ca5)
    #2 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1174 (test_upgrade+0x000000002f68)

  Previous read of size 4 at 0x7d500000ff74 by thread T7:
    #0 MHD_select_thread /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4053 (libmicrohttpd.so.12+0x000000015980)
    #1 named_thread_starter /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:281 (libmicrohttpd.so.12+0x00000001d55d)
    #2 <null> <null> (libtsan.so.0+0x000000024459)

  Location is heap block of size 488 at 0x7d500000fe00 allocated by main thread:
    #0 calloc <null> (libtsan.so.0+0x000000026b41)
    #1 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4862 (libmicrohttpd.so.12+0x000000018013)
    #2 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #3 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1011 (test_upgrade+0x000000002af2)
    #4 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1174 (test_upgrade+0x000000002f68)

  Thread T7 'MHD-listen' (tid=8215, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 MHD_create_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:211 (libmicrohttpd.so.12+0x00000001d484)
    #2 MHD_create_named_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:356 (libmicrohttpd.so.12+0x00000001d
64b)
    #3 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5368 (libmicrohttpd.so.12+0x000000019254)
    #4 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #5 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1011 (test_upgrade+0x000000002af2)
    #6 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1174 (test_upgrade+0x000000002f68)

SUMMARY: ThreadSanitizer: data race /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5744 in MHD_stop_daemon
==================
==================
WARNING: ThreadSanitizer: data race (pid=8194)
  Write of size 4 at 0x7d600000fd74 by main thread:
    #0 MHD_stop_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5757 (libmicrohttpd.so.12+0x00000001a467)
    #1 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1081 (test_upgrade+0x000000002ca5)
    #2 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1207 (test_upgrade+0x000000003118)

  Previous read of size 4 at 0x7d600000fd74 by thread T3:
    #0 MHD_get_fdset2 /home/christian/libmicrohttpd/src/microhttpd/daemon.c:782 (libmicrohttpd.so.12+0x00000000d226)
    #1 MHD_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3126 (libmicrohttpd.so.12+0x000000012eea)
    #2 MHD_select_thread /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4062 (libmicrohttpd.so.12+0x000000015962)
    #3 named_thread_starter /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:281 (libmicrohttpd.so.12+0x00000001d55d)
    #4 <null> <null> (libtsan.so.0+0x000000024459)

  Location is heap block of size 976 at 0x7d600000fc00 allocated by main thread:
    #0 malloc <null> (libtsan.so.0+0x0000000269dc)
    #1 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5400 (libmicrohttpd.so.12+0x0000000194b6)
    #2 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #3 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1011 (test_upgrade+0x000000002af2)
    #4 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1207 (test_upgrade+0x000000003118)

  Thread T3 'MHD-worker' (tid=8230, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 MHD_create_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:211 (libmicrohttpd.so.12+0x00000001d484)
    #2 MHD_create_named_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:356 (libmicrohttpd.so.12+0x00000001d
64b)
    #3 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5466 (libmicrohttpd.so.12+0x000000019826)
    #4 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #5 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1011 (test_upgrade+0x000000002af2)
    #6 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1207 (test_upgrade+0x000000003118)

SUMMARY: ThreadSanitizer: data race /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5757 in MHD_stop_daemon
==================
==================
WARNING: ThreadSanitizer: data race (pid=8194)
  Write of size 4 at 0x7d600000fd60 by main thread:
    #0 MHD_stop_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5758 (libmicrohttpd.so.12+0x00000001a4ab)
    #1 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1081 (test_upgrade+0x000000002ca5)
    #2 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1207 (test_upgrade+0x000000003118)

  Previous read of size 4 at 0x7d600000fd60 by thread T3:
    #0 MHD_get_fdset2 /home/christian/libmicrohttpd/src/microhttpd/daemon.c:798 (libmicrohttpd.so.12+0x00000000d2f7)
    #1 MHD_select /home/christian/libmicrohttpd/src/microhttpd/daemon.c:3126 (libmicrohttpd.so.12+0x000000012eea)
    #2 MHD_select_thread /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4062 (libmicrohttpd.so.12+0x000000015962)
    #3 named_thread_starter /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:281 (libmicrohttpd.so.12+0x00000001d55d)
    #4 <null> <null> (libtsan.so.0+0x000000024459)

  Location is heap block of size 976 at 0x7d600000fc00 allocated by main thread:
    #0 malloc <null> (libtsan.so.0+0x0000000269dc)
    #1 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5400 (libmicrohttpd.so.12+0x0000000194b6)
    #2 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #3 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1011 (test_upgrade+0x000000002af2)
    #4 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1207 (test_upgrade+0x000000003118)

  Thread T3 'MHD-worker' (tid=8230, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000027e9d)
    #1 MHD_create_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:211 (libmicrohttpd.so.12+0x00000001d484)
    #2 MHD_create_named_thread_ /home/christian/libmicrohttpd/src/microhttpd/mhd_threads.c:356 (libmicrohttpd.so.12+0x00000001d
64b)
    #3 MHD_start_daemon_va /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5466 (libmicrohttpd.so.12+0x000000019826)
    #4 MHD_start_daemon /home/christian/libmicrohttpd/src/microhttpd/daemon.c:4125 (libmicrohttpd.so.12+0x000000015ac3)
    #5 test_upgrade /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1011 (test_upgrade+0x000000002af2)
    #6 main /home/christian/libmicrohttpd/src/microhttpd/test_upgrade.c:1207 (test_upgrade+0x000000003118)

SUMMARY: ThreadSanitizer: data race /home/christian/libmicrohttpd/src/microhttpd/daemon.c:5758 in MHD_stop_daemon
==================
ThreadSanitizer: reported 6 warnings

Christian Grothoff

2017-02-14 18:51

manager   ~0011754

I've found a few (AFAIK) unrelated concurrency issues with -fsanitize=thread, but also a bunch are false positives, as it IS safe to raise a (now additionally volatile) flag to signal termination without explicit locking (as long as one is very careful...).

Anyway, there is one issue left, but it is only related to MHD_quiesce_daemon(), which does not appear in this report. The other issues were mostly related to the 0.9.52 introduction of 'upgrade', and there was one (rare) race on shutdown in thread-per-connection mode that could cause a panic or other semi-arbitrary havoc.

I have NOT found any 0.9.51-issue related to double-unlocking or test_timeout so far, despite trying on a TSX-enabled machine with -fsanitize=threads.

Christian Grothoff

2017-02-14 18:53

manager   ~0011755

Last edited: 2017-02-14 18:57

 Actually, looking at the Debian logs more closely, it was "test_callback" that failed, not "test_timeout".

Christian Grothoff

2017-02-14 18:57

manager   ~0011756

Gotcha:

$ ./test_callback
==================
WARNING: ThreadSanitizer: unlock of an unlocked mutex (or by a wrong thread) (pid=11165)
    #0 pthread_mutex_unlock <null> (libtsan.so.0+0x0000000365f4)
    #1 MHD_connection_handle_write <null> (libmicrohttpd.so.12+0x0000000063d8)
    #2 MHD_run_from_select <null> (libmicrohttpd.so.12+0x000000012ce7)
    #3 MHD_select <null> (libmicrohttpd.so.12+0x000000012fd3)
    #4 MHD_run <null> (libmicrohttpd.so.12+0x0000000127c8)
    #5 main <null> (test_callback+0x0000000012fc)

  Location is heap block of size 1160 at 0x7d640001c700 allocated by main thread:
    #0 malloc <null> (libtsan.so.0+0x0000000269dc)
    #1 MHD_create_response_from_callback <null> (libmicrohttpd.so.12+0x000000018024)
    #2 callback <null> (test_callback+0x0000000016a3)
    #3 MHD_connection_handle_idle <null> (libmicrohttpd.so.12+0x00000000a022)
    #4 MHD_run_from_select <null> (libmicrohttpd.so.12+0x000000012bc0)
    #5 MHD_select <null> (libmicrohttpd.so.12+0x000000012fd3)
    #6 MHD_run <null> (libmicrohttpd.so.12+0x0000000127c8)
    #7 main <null> (test_callback+0x0000000012fc)

  Mutex M11 (0x7d640001c728) created at:
    #0 pthread_mutex_init <null> (libtsan.so.0+0x0000000285ee)
    #1 MHD_create_response_from_callback <null> (libmicrohttpd.so.12+0x000000018079)
    #2 callback <null> (test_callback+0x0000000016a3)
    #3 MHD_connection_handle_idle <null> (libmicrohttpd.so.12+0x00000000a022)
    #4 MHD_run_from_select <null> (libmicrohttpd.so.12+0x000000012bc0)
    #5 MHD_select <null> (libmicrohttpd.so.12+0x000000012fd3)
    #6 MHD_run <null> (libmicrohttpd.so.12+0x0000000127c8)
    #7 main <null> (test_callback+0x0000000012fc)

SUMMARY: ThreadSanitizer: unlock of an unlocked mutex (or by a wrong thread) (/lib/x86_64-linux-gnu/libtsan.so.0+0x365f4) in pthread_mutex_unlock
==================

Christian Grothoff

2017-02-14 19:01

manager   ~0011757

This was already fixed in Git e0c14df5 on 2016-10-09 18:43:27
(comment was: /* mutex was already unlocked by try_ready_normal_body */).

But, good to confirm. So 0.9.52 fixes this, or you can cherry pick this patch:

https://gnunet.org/git/libmicrohttpd.git/commit/?id=e0c14df575785d06c22b28ecc9433e2b5d7a69d7

Issue History

Date Modified Username Field Change
2016-11-26 11:40 beberking New Issue
2016-11-29 14:37 Christian Grothoff Note Added: 0011554
2016-12-04 21:56 beberking Note Added: 0011566
2016-12-10 10:56 Christian Grothoff Note Added: 0011585
2017-01-29 01:05 Christian Grothoff Status new => acknowledged
2017-02-13 22:12 Christian Grothoff Note Added: 0011741
2017-02-13 22:38 Christian Grothoff Note Added: 0011742
2017-02-13 23:00 Christian Grothoff Note Added: 0011743
2017-02-13 23:00 Christian Grothoff Assigned To => Christian Grothoff
2017-02-13 23:00 Christian Grothoff Status acknowledged => assigned
2017-02-14 02:12 Christian Grothoff Note Added: 0011744
2017-02-14 02:12 Christian Grothoff Note Added: 0011745
2017-02-14 18:51 Christian Grothoff Note Added: 0011754
2017-02-14 18:53 Christian Grothoff Note Added: 0011755
2017-02-14 18:57 Christian Grothoff Note Edited: 0011755
2017-02-14 18:57 Christian Grothoff Note Added: 0011756
2017-02-14 19:01 Christian Grothoff Note Added: 0011757
2017-02-14 19:02 Christian Grothoff Status assigned => closed
2017-02-14 19:02 Christian Grothoff Fixed in Version => 0.9.52
2017-02-14 19:02 Christian Grothoff Target Version => 0.9.53
2024-01-21 13:25 Christian Grothoff Category libmicrohttpd multi-threaded operation => thread pool