View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001716 | GNUnet | util library | public | 2011-06-18 18:53 | 2011-07-02 09:26 |
| Reporter | LRN | Assigned To | LRN | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | closed | Resolution | open | ||
| Product Version | Git master | ||||
| Summary | 0001716: Lack of graceful socket shutdown on process termination in W32 | ||||
| Description | Apparently, when a W32 process terminates, its socket doesn't get flushed or gracefully shut down. This prevents a persistent socket that connects gnunet-service-arm with gnunet-arm from delivering the last 4 bytes (shutdown ACK), which causes test-arm-api to fail. This patch shuts down persistent connections by first shutting down writing on them, and then reading them until they are shut down (which might block until the party on the other side also shuts down its socket - which usually happens pretty quickly). If there was an intention for shutdown ACK to arrive ONLY after the process that sent it is dead, well, tough luck, that doesn't work on Windows. | ||||
| Tags | No tags attached. | ||||
| Attached Files | 0001-Reduce-server-client-socket-buffer-size-to-0.patch (6,099 bytes)
From 55b8446528ad3915b3d28bdedd88c68f7334920f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=D0=A0=D1=83=D1=81=D0=BB=D0=B0=D0=BD=20=D0=98=D0=B6=D0=B1=D1?=
=?UTF-8?q?=83=D0=BB=D0=B0=D1=82=D0=BE=D0=B2?= <lrn1986@gmail.com>
Date: Sun, 19 Jun 2011 21:44:43 +0400
Subject: [PATCH] Reduce server-client socket buffer size to 0
to make sure that SHUTDOWN_ACK reaches the other end before the process
terminates.
---
src/arm/gnunet-service-arm.c | 7 +++++++
src/include/gnunet_connection_lib.h | 11 +++++++++++
src/include/gnunet_network_lib.h | 7 +++++++
src/include/gnunet_server_lib.h | 9 +++++++++
src/util/connection.c | 12 ++++++++++++
src/util/network.c | 20 ++++++++++++++++++++
src/util/server.c | 12 ++++++++++++
7 files changed, 78 insertions(+), 0 deletions(-)
diff --git a/src/arm/gnunet-service-arm.c b/src/arm/gnunet-service-arm.c
index e357668..fa3caa3 100644
--- a/src/arm/gnunet-service-arm.c
+++ b/src/arm/gnunet-service-arm.c
@@ -958,6 +958,13 @@ transmit_shutdown_ack (void *cls, size_t size, void *buf)
GNUNET_log (GNUNET_ERROR_TYPE_INFO,
_("Transmitting shutdown ACK.\n"));
+#if defined (WINDOWS)
+ /* Make the connection flushing for the purpose of ACK transmitting */
+ if (GNUNET_OK != GNUNET_SERVER_client_set_flushing (client))
+ GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
+ _("Failed to make the client connection flushing.\n"));
+#endif
+
msg = (struct GNUNET_MessageHeader *) buf;
msg->type = htons (GNUNET_MESSAGE_TYPE_ARM_SHUTDOWN_ACK);
msg->size = htons (sizeof (struct GNUNET_MessageHeader));
diff --git a/src/include/gnunet_connection_lib.h b/src/include/gnunet_connection_lib.h
index 8d2dbb6..d080852 100644
--- a/src/include/gnunet_connection_lib.h
+++ b/src/include/gnunet_connection_lib.h
@@ -114,6 +114,17 @@ void
GNUNET_CONNECTION_persist_(struct GNUNET_CONNECTION_Handle *sock);
/**
+ * Make the connection flush itself and be blocking.
+ * Used to make sure that the last messages sent through the connection
+ * reach the other side before the process is terminated.
+ *
+ * @param sock the connection to make flushing and blocking
+ */
+int
+GNUNET_CONNECTION_set_flushing (struct GNUNET_CONNECTION_Handle *sock);
+
+
+/**
* Create a socket handle by boxing an existing OS socket. The OS
* socket should henceforth be no longer used directly.
* GNUNET_socket_destroy will close it.
diff --git a/src/include/gnunet_network_lib.h b/src/include/gnunet_network_lib.h
index 34cb7bc..3afb403 100644
--- a/src/include/gnunet_network_lib.h
+++ b/src/include/gnunet_network_lib.h
@@ -240,6 +240,13 @@ int GNUNET_NETWORK_socket_setsockopt (struct GNUNET_NETWORK_Handle *fd,
int GNUNET_NETWORK_socket_shutdown (struct GNUNET_NETWORK_Handle *desc,
int how);
+/**
+ * Reduce send and recv buffers to zero, flushing the socket.
+ * @param desc socket
+ * @return GNUNET_OK on success, GNUNET_SYSERR otherwise
+ */
+int GNUNET_NETWORK_socket_empty (struct GNUNET_NETWORK_Handle *desc);
+
/**
* Create a new socket. Configure it for non-blocking IO and
diff --git a/src/include/gnunet_server_lib.h b/src/include/gnunet_server_lib.h
index ca20419..281a868 100644
--- a/src/include/gnunet_server_lib.h
+++ b/src/include/gnunet_server_lib.h
@@ -384,6 +384,15 @@ GNUNET_SERVER_ignore_shutdown (struct GNUNET_SERVER_Handle *h,
/**
+ * FIXME
+ *
+ * @param sock handle to the service connection
+ */
+int
+GNUNET_SERVER_client_set_flushing (struct GNUNET_SERVER_Client *client);
+
+
+/**
* The tansmit context is the key datastructure for a conveniance API
* used for transmission of complex results to the client followed
* ONLY by signaling receive_done with success or error
diff --git a/src/util/connection.c b/src/util/connection.c
index 6cb3495..c7bf325 100644
--- a/src/util/connection.c
+++ b/src/util/connection.c
@@ -302,6 +302,18 @@ void GNUNET_CONNECTION_persist_(struct GNUNET_CONNECTION_Handle *sock)
}
/**
+ * Make the connection flush itself and be blocking.
+ * Used to make sure that the last messages sent through the connection
+ * reach the other side before the process is terminated.
+ *
+ * @param sock the connection to make flushing and blocking
+ */
+int GNUNET_CONNECTION_set_flushing (struct GNUNET_CONNECTION_Handle *sock)
+{
+ return GNUNET_NETWORK_socket_empty (sock->sock);
+}
+
+/**
* Create a socket handle by boxing an existing OS socket. The OS
* socket should henceforth be no longer used directly.
* GNUNET_socket_destroy will close it.
diff --git a/src/util/network.c b/src/util/network.c
index 5ff49b1..c5cef24 100644
--- a/src/util/network.c
+++ b/src/util/network.c
@@ -721,6 +721,26 @@ GNUNET_NETWORK_socket_shutdown (struct GNUNET_NETWORK_Handle *desc, int how)
/**
+ * Reduce send and recv buffers to zero, flushing the socket.
+ * @param desc socket
+ * @return GNUNET_OK on success, GNUNET_SYSERR otherwise
+ */
+int
+GNUNET_NETWORK_socket_empty (struct GNUNET_NETWORK_Handle *desc)
+{
+ int value = 0;
+ int ret = 0;
+
+ if (0 != (ret = setsockopt (desc->fd, SOL_SOCKET, SO_SNDBUF, &value, sizeof (value))))
+ GNUNET_log_strerror (GNUNET_ERROR_TYPE_WARNING, "setsockopt");
+ if (0 != (ret = setsockopt (desc->fd, SOL_SOCKET, SO_RCVBUF, &value, sizeof (value))))
+ GNUNET_log_strerror (GNUNET_ERROR_TYPE_WARNING, "setsockopt");
+
+ return ret == 0 ? GNUNET_OK : GNUNET_SYSERR;
+}
+
+
+/**
* Reset FD set
* @param fds fd set
*/
diff --git a/src/util/server.c b/src/util/server.c
index 03cf48f..1282864 100644
--- a/src/util/server.c
+++ b/src/util/server.c
@@ -1154,6 +1154,18 @@ GNUNET_SERVER_client_disconnect (struct GNUNET_SERVER_Client *client)
/**
+ * FIXME
+ *
+ * @param sock handle to the service connection
+ */
+int
+GNUNET_SERVER_client_set_flushing (struct GNUNET_SERVER_Client *client)
+{
+ return GNUNET_CONNECTION_set_flushing (client->connection);
+}
+
+
+/**
* Notify us when the server has enough space to transmit
* a message of the given size to the given client.
*
--
1.7.4
| ||||
|
|
Did you try no-delay or setting the winsock buffer size to 0? http://support.microsoft.com/kb/214397 |
|
|
http://msdn.microsoft.com/en-us/library/ms740476%28VS.85%29.aspx |
|
|
TCP_NODELAY is UNCONDITIONALLY applied to ALL sockets in GNUnet (except for UNIX sockets). I did try setting winsock buffer size to 0, but only in conjunction with shutdown-send-and-then-recv-until-zero. Didn't try it on its own. You think that setting the buffer to zero will force winsock to flush it? |
|
|
Tried set-buffers-to-zero-only. Also tried to set buffers to zero and then send 0 bytes. Also tried to do all that after setting the socket mode to blocking. Doesn't help. |
|
|
Let me get this straight: even with nodelay and a buffer size of zero, the SHUTDOWN_ACK is not delivered to the client if the service does not perform "shutdown" or "close" on the TCP socket as well? Or is this about no FIN being generated upon process termination? Just trying to understand what is going on... |
|
|
Well, not exactly. I've tried to set buffer size to zero sometime _after_ SHUTDOWN_ACK was sent (because it's kinda difficult to change the mode of a socket _just_ before SHUTDOWN_ACK is sent, since the code that queues it up is in one place, and the code that does send it is in another and is generic). As for the topic, it is about SHUTDOWN_ACK not being delivered. AFAIU, FIN is not critical (the other party will fail with 'connection reset by peer', but as long as it gets SHUTDOWN_ACK that is not considered an error condition) and does not concern me. |
|
|
I don't understand why you cannot set the mode before --- just add some API that, given your client handle (which you will use to send the shutdown ack), will set whatever options you want. Maybe ugly, but we first need to find out what exactly the problem is. I cannot imagine that there is no way to set flags to force W32 to immediately send the buffer. |
|
|
OK, setting the buffer size to zero BEFORE sending SHUTDOWN_ACK does help. Here's the proof: Without setting the buffer size to 0: ??? 19 21:28:05 00000000000720529979 arm-7636 INFO Transmitting shutdown ACK. ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG Client is being disconnected from the server. ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG RC still positive, not destroying everything. ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG Client is being disconnected from the server. ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG Adding task: 34 / 0x1f7c100 ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG transmit_ready transmitted 4/4 bytes to `[::1]:62441' (0x1f7c100) ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG Checking readiness of task: 34 / 0x1f7c100 ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG Running task: 34 / 0x1f7c100 ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG Shutting down socket (0x1f7c100) ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG Destroy actually runs (0x1f7c100)! ??? 19 21:28:05 00000000000720529979 arm-7636 DEBUG Freeing memory of connection 0x1f7c100. With setting the buffer size to 0: ??? 19 21:27:10 00000000000720474610 arm-6848 INFO Transmitting shutdown ACK. ??? 19 21:27:10 00000000000720474610 arm-6848 DEBUG Client is being disconnected from the server. ??? 19 21:27:10 00000000000720474610 arm-6848 DEBUG RC still positive, not destroying everything. ??? 19 21:27:10 00000000000720474610 arm-6848 DEBUG Client is being disconnected from the server. ??? 19 21:27:10 00000000000720474610 arm-6848 DEBUG Adding task: 34 / 0x1d5c100 ??? 19 21:27:10 00000000000720474626 arm-6848 DEBUG transmit_ready transmitted 4/4 bytes to `[::1]:62401' (0x1d5c100) ??? 19 21:27:10 00000000000720474626 arm-6848 DEBUG Checking readiness of task: 34 / 0x1d5c100 ??? 19 21:27:10 00000000000720474627 arm-6848 DEBUG Running task: 34 / 0x1d5c100 ??? 19 21:27:10 00000000000720474627 arm-6848 DEBUG Shutting down socket (0x1d5c100) ??? 19 21:27:10 00000000000720474627 arm-6848 DEBUG Destroy actually runs (0x1d5c100)! ??? 19 21:27:10 00000000000720474627 arm-6848 DEBUG Freeing memory of connection 0x1d5c100. Notice the millisecond counter - it skips 16 ms. That is, presumably, the socket blocking on a send operation. In unmodified version the counter doesn't change - everything happens within 1ms. |
|
|
Fixed in SVN 15722 (with minor renaming and some more comments). |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2011-06-18 18:53 | LRN | New Issue | |
| 2011-06-18 18:53 | LRN | File Added: 0001-Fix-the-lack-of-graceful-socket-closure.patch | |
| 2011-06-18 18:53 | LRN | File Added: 0003-Only-shutdown-non-persistent-sockets.patch | |
| 2011-06-18 20:30 | Christian Grothoff | Note Added: 0004432 | |
| 2011-06-18 20:43 | Christian Grothoff | Note Added: 0004433 | |
| 2011-06-18 20:54 | LRN | Note Added: 0004434 | |
| 2011-06-18 23:19 | LRN | Note Added: 0004435 | |
| 2011-06-19 12:20 | Christian Grothoff | Note Added: 0004436 | |
| 2011-06-19 12:54 | LRN | Note Added: 0004437 | |
| 2011-06-19 14:48 | Christian Grothoff | Note Added: 0004438 | |
| 2011-06-19 19:31 | LRN | Note Added: 0004439 | |
| 2011-06-19 19:31 | LRN | File Deleted: 0001-Fix-the-lack-of-graceful-socket-closure.patch | |
| 2011-06-19 19:31 | LRN | File Deleted: 0003-Only-shutdown-non-persistent-sockets.patch | |
| 2011-06-19 19:46 | LRN | File Added: 0001-Reduce-server-client-socket-buffer-size-to-0.patch | |
| 2011-06-20 11:24 | Christian Grothoff | Note Added: 0004441 | |
| 2011-06-20 11:24 | Christian Grothoff | Assigned To | => LRN |
| 2011-06-20 11:24 | Christian Grothoff | Status | new => resolved |
| 2011-07-02 09:26 | Christian Grothoff | Status | resolved => closed |