View Issue Details

IDProjectCategoryView StatusLast Update
0001434libmicrohttpdperformancepublic2008-11-16 17:00
ReporterMatthew Moore Assigned ToChristian Grothoff  
PrioritynormalSeverityfeatureReproducibilityalways
Status closedResolutionfixed 
Product Version0.4.0pre0 
Summary0001434: Absence of TCP_NDELAY or TCP_CORK can result in performance-degrading interaction with TCP delayed ack
DescriptionObserved this while performance testing with a Windows/Java load-testing client ("The Grinder").

Not setting TCP_NDELAY or TCP_CORK means the Nagle algorithm is engaged in libmicrohttpd. This causes incomplete packets (which can occur because of either a short response body or at the tail end of a larger response) to be held up waiting for an ACK from the client system. Unfortunately under some circumstances the client will wait up to 200ms to send an ACK, hoping it can piggyback the ACK on an outgoing packet (of which there is none with HTTP). As a result I was seeing an extra 200ms of latency on each request.

This article describes the issue well:

http://www.stuartcheshire.org/papers/NagleDelayedAck/
Additional InformationThe TCP_CORK socket option is set while generating an outbound message and unset when the message is complete. It instructs the TCP stack to not send less-than-full packets until TCP_CORK is unset, resulting in optimal network utilization. Unfortunately it is not as portable as TCP_NDELAY which merely disables the Nagle algorithm. Apache sets TCP_NDELAY I believe.

I have attached a patch against SVN revision 7820 that sets TCP_CORK while sending a response and unsets it when finished. I added the autoconf stuff to make the use of TCP_CORK conditional. It fixed my performance problem.
TagsNo tags attached.
Attached Files
tcpcork.patch (2,035 bytes)   
Index: configure.ac
===================================================================
--- configure.ac	(revision 7846)
+++ configure.ac	(working copy)
@@ -136,7 +136,7 @@
 AC_CHECK_HEADERS([fcntl.h math.h errno.h limits.h stdio.h locale.h sys/stat.h sys/types.h pthread.h],,AC_MSG_ERROR([Compiling libmicrohttpd requires standard UNIX headers files]))
 
 # Check for optional headers
-AC_CHECK_HEADERS([sys/select.h sys/types.h sys/time.h sys/msg.h netdb.h netinet/in.h time.h sys/socket.h sys/mman.h arpa/inet.h])
+AC_CHECK_HEADERS([sys/select.h sys/types.h sys/time.h sys/msg.h netdb.h netinet/in.h netinet/tcp.h time.h sys/socket.h sys/mman.h arpa/inet.h])
 
 # IPv6
 AC_MSG_CHECKING(for IPv6)
@@ -161,6 +161,8 @@
 )
 AC_MSG_RESULT($have_inet6)
 
+# TCP_CORK
+AC_CHECK_DECL([TCP_CORK], [], [], [[#include <netinet/tcp.h>]])
 
 # libcurl (required for testing)
 SAVE_LIBS=$LIBS
Index: src/daemon/connection.c
===================================================================
--- src/daemon/connection.c	(revision 7846)
+++ src/daemon/connection.c	(working copy)
@@ -1921,6 +1921,14 @@
               continue;
             }
           connection->state = MHD_CONNECTION_HEADERS_SENDING;
+
+#if HAVE_TCP_CORK
+          /* starting header send, set TCP cork */
+          {
+            const int val = 1;
+            setsockopt(connection->socket_fd, IPPROTO_TCP, TCP_CORK, &val, sizeof(val));
+          }
+#endif
           break;
         case MHD_CONNECTION_HEADERS_SENDING:
           /* no default action */
@@ -1976,6 +1984,13 @@
           /* no default action */
           break;
         case MHD_CONNECTION_FOOTERS_SENT:
+#if HAVE_TCP_CORK
+          /* done sending, uncork */
+          {
+            const int val = 0;
+            setsockopt(connection->socket_fd, IPPROTO_TCP, TCP_CORK, &val, sizeof(val));
+          }
+#endif
           MHD_destroy_response (connection->response);
           if (connection->daemon->notify_completed != NULL)
             connection->daemon->notify_completed (connection->
tcpcork.patch (2,035 bytes)   

Activities

Christian Grothoff

2008-11-08 02:21

manager   ~0003729

Looks very reasonable. Applied in SVN 7864.

Issue History

Date Modified Username Field Change
2008-11-03 17:16 Matthew Moore New Issue
2008-11-03 17:16 Matthew Moore File Added: tcpcork.patch
2008-11-08 02:20 Christian Grothoff Status new => assigned
2008-11-08 02:20 Christian Grothoff Assigned To => Christian Grothoff
2008-11-08 02:21 Christian Grothoff Status assigned => resolved
2008-11-08 02:21 Christian Grothoff Resolution open => fixed
2008-11-08 02:21 Christian Grothoff Note Added: 0003729
2008-11-16 17:00 Christian Grothoff Status resolved => closed