View Issue Details

IDProjectCategoryView StatusLast Update
0012184Talerexchangepublic2026-11-18 14:42
Reporterbot-cg [agent]  
PrioritynormalSeverityminor 
Status new 
Summary0012184: taler-api: answer every accepted connection on a graceful shutdown
DescriptionOn a graceful shutdown, axum::serve tells every open connection to shut down. A connection whose first request has not been read yet then closes at once: hyper-util's auto builder cancels the detection of its HTTP version, and hyper closes an HTTP/1 connection that has no request in progress. The server had accepted the connection, but the client's request is lost, with a reset or a close without an answer. Under systemd socket activation with back-to-back restarts, about one request in 6,000 failed this way. serve() now runs its own accept loop. At shutdown it stops accepting, gives each connection that has not started a request up to ten seconds to send its first one, and only then shuts the connection down gracefully, so that the request in progress is answered. It returns when every connection has closed. A test sends a request on a connection accepted before the shutdown; with axum::serve, it got no answer.

Files: common/taler-api/Cargo.toml, common/taler-api/src/api.rs (paths in the taler-rust tree). The patch is attached; it is number 9 of the series in the parent issue, against taler-rust 2c678873, and applies on top of the patches before it.
TagsNo tags attached.
Attached Files
0009-api-answer-every-accepted-connection-on-a-graceful-s.patch (10,887 bytes)   
From fe0f44d710fff4a6a47717126a2bcc2cf2cf7fe8 Mon Sep 17 00:00:00 2001
From: Christian Grothoff <christian@grothoff.org>
Date: Wed, 18 Nov 2026 15:13:01 +0100
Subject: [PATCH 9/9] api: answer every accepted connection on a graceful
 shutdown

On a graceful shutdown, axum::serve tells every open connection to shut
down.  A connection whose first request has not been read yet then closes at
once: hyper-util's auto builder cancels the detection of its HTTP version,
and hyper closes an HTTP/1 connection that has no request in progress.  The
server had accepted the connection, but the client's request is lost, with a
reset or a close without an answer.  Under systemd socket activation with
back-to-back restarts, about one request in 6,000 failed this way.

serve() now runs its own accept loop.  At shutdown it stops accepting, gives
each connection that has not started a request up to ten seconds to send
its first one, and only then shuts the connection down gracefully, so that
the request in progress is answered.  It returns when every connection has
closed.  A test sends a request on a connection accepted before the
shutdown; with axum::serve, it got no answer.

Assisted by LLM
---
 common/taler-api/Cargo.toml |   2 +
 common/taler-api/src/api.rs | 194 ++++++++++++++++++++++++++++++++++--
 2 files changed, 187 insertions(+), 9 deletions(-)

diff --git a/common/taler-api/Cargo.toml b/common/taler-api/Cargo.toml
index 6bed8d6..1debe1f 100644
--- a/common/taler-api/Cargo.toml
+++ b/common/taler-api/Cargo.toml
@@ -15,6 +15,8 @@ doctest = false
 listenfd = "1.0.0"
 dashmap = "6.1"
 http-body-util.workspace = true
+hyper.workspace = true
+hyper-util = { workspace = true, features = ["server", "service", "tokio"] }
 zlib-rs = "0.6.3"
 tokio = { workspace = true, features = ["signal"] }
 rand.workspace = true
diff --git a/common/taler-api/src/api.rs b/common/taler-api/src/api.rs
index e7ba351..52f34b5 100644
--- a/common/taler-api/src/api.rs
+++ b/common/taler-api/src/api.rs
@@ -19,7 +19,7 @@ use std::{
         Arc,
         atomic::{AtomicU32, Ordering},
     },
-    time::Instant,
+    time::{Duration, Instant},
 };
 
 use axum::{
@@ -29,6 +29,12 @@ use axum::{
     response::Response,
 };
 use compact_str::CompactString;
+use hyper::{body::Incoming, service::Service as _};
+use hyper_util::{
+    rt::{TokioExecutor, TokioIo},
+    server::conn::auto,
+    service::TowerToHyperService,
+};
 use rand::RngExt as _;
 use revenue::Revenue;
 use taler_common::{
@@ -36,8 +42,12 @@ use taler_common::{
     log::LOG_TASK_ID,
     types::amount::{Amount, Currency},
 };
-use tokio::signal;
-use tracing::{Level, debug, info};
+use tokio::{
+    io::{AsyncRead, AsyncWrite},
+    signal,
+    sync::watch,
+};
+use tracing::{Level, debug, info, trace};
 use wire::WireGateway;
 
 use crate::{
@@ -144,14 +154,10 @@ impl TalerRouter for Router {
         let signal = shutdown_signal(notify);
         match listener {
             Listener::Tcp(tcp_listener) => {
-                axum::serve(tcp_listener, router)
-                    .with_graceful_shutdown(signal)
-                    .await?;
+                serve_connections(tcp_listener, router, signal, FIRST_REQUEST_TIMEOUT).await
             }
             Listener::Unix(unix_listener) => {
-                axum::serve(unix_listener, router)
-                    .with_graceful_shutdown(signal)
-                    .await?;
+                serve_connections(unix_listener, router, signal, FIRST_REQUEST_TIMEOUT).await
             }
         }
 
@@ -160,6 +166,99 @@ impl TalerRouter for Router {
     }
 }
 
+/// How long a connection that was accepted before a graceful shutdown may
+/// take to send its first request.
+const FIRST_REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// Serve `router` on `listener` until `signal`, then finish every connection
+/// that was accepted, and return when the last one has closed.
+///
+/// This is `axum::serve` with a graceful shutdown, except for connections
+/// whose first request has not been read yet.  axum tells every connection
+/// to shut down, and such a connection closes at once: hyper-util cancels the
+/// detection of its protocol, and hyper closes an HTTP/1 connection that has
+/// no request in progress.  The client, whose connection the server had
+/// accepted, loses its request (a reset, or a close without an answer).
+/// Under systemd socket activation with back-to-back restarts, about one
+/// request in 6,000 was lost this way.
+///
+/// Here a connection that has not started a request yet gets up to
+/// `first_request` to send one.  Once it has, or once that time is up, it is
+/// told to shut down: the request in progress is answered, and the
+/// connection closes.  The listener is dropped first, so that the server
+/// accepts nothing new; connections that it has not accepted stay in the
+/// queue of an inherited socket for the next process.
+async fn serve_connections<L: axum::serve::Listener>(
+    mut listener: L,
+    router: Router,
+    signal: impl Future<Output = ()>,
+    first_request: Duration,
+) {
+    // Dropped to tell the connections that the server shuts down.
+    let (shutdown_tx, shutdown_rx) = watch::channel(());
+    // Each connection holds a receiver: when none is left, all are closed.
+    let (closed_tx, closed_rx) = watch::channel(());
+    let mut signal = std::pin::pin!(signal);
+    loop {
+        let (io, _) = tokio::select! {
+            accepted = listener.accept() => accepted,
+            () = &mut signal => break,
+        };
+        tokio::spawn(serve_connection(
+            io,
+            router.clone(),
+            shutdown_rx.clone(),
+            closed_rx.clone(),
+            first_request,
+        ));
+    }
+    drop(listener);
+    drop(shutdown_tx);
+    drop(closed_rx);
+    closed_tx.closed().await;
+}
+
+/// Serve one connection; see [`serve_connections`].
+async fn serve_connection<I: AsyncRead + AsyncWrite + Unpin + Send + 'static>(
+    io: I,
+    router: Router,
+    mut shutdown: watch::Receiver<()>,
+    _closed: watch::Receiver<()>,
+    first_request: Duration,
+) {
+    // Becomes true when the connection's first request reaches the router.
+    let (requested_tx, mut requested) = watch::channel(false);
+    let router = TowerToHyperService::new(router);
+    let service = hyper::service::service_fn(move |request: hyper::Request<Incoming>| {
+        requested_tx.send_replace(true);
+        router.call(request)
+    });
+    let builder = auto::Builder::new(TokioExecutor::new());
+    let mut connection =
+        std::pin::pin!(builder.serve_connection_with_upgrades(TokioIo::new(io), service));
+    let finished = |result: Result<(), Box<dyn std::error::Error + Send + Sync>>| {
+        if let Err(e) = result {
+            trace!(target: "api", "connection: {e}");
+        }
+    };
+    tokio::select! {
+        result = connection.as_mut() => return finished(result),
+        // The sender is dropped, which is the signal.
+        _ = shutdown.changed() => {}
+    }
+    if !*requested.borrow() {
+        let first = async {
+            let _ = requested.wait_for(|r| *r).await;
+        };
+        tokio::select! {
+            result = connection.as_mut() => return finished(result),
+            _ = tokio::time::timeout(first_request, first) => {}
+        }
+    }
+    connection.as_mut().graceful_shutdown();
+    finished(connection.await);
+}
+
 /// Allow cross-origin requests from any origin, with any method and headers.
 ///
 /// Only a CORS preflight, an OPTIONS request with Access-Control-Request-Method
@@ -594,4 +693,81 @@ mod tests {
             warnings[1]
         );
     }
+
+    /// A listener that reports each connection that it has accepted.
+    struct Reporting(
+        tokio::net::TcpListener,
+        tokio::sync::mpsc::UnboundedSender<()>,
+    );
+
+    impl axum::serve::Listener for Reporting {
+        type Io = tokio::net::TcpStream;
+        type Addr = std::net::SocketAddr;
+
+        async fn accept(&mut self) -> (Self::Io, Self::Addr) {
+            let accepted = self.0.accept().await.unwrap();
+            self.1.send(()).unwrap();
+            accepted
+        }
+
+        fn local_addr(&self) -> std::io::Result<Self::Addr> {
+            self.0.local_addr()
+        }
+    }
+
+    /// A graceful shutdown answers a connection that the server accepted
+    /// before it, even if the request arrives only afterwards; with
+    /// axum::serve, such a connection closed without an answer.  A
+    /// connection that sends nothing is closed after the time for its first
+    /// request, and then the server returns.
+    #[tokio::test]
+    async fn shutdown_answers_accepted_connections() {
+        use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _};
+
+        let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+        let addr = listener.local_addr().unwrap();
+        let (accepted_tx, mut accepted) = tokio::sync::mpsc::unbounded_channel();
+        let (stop, stopped) = tokio::sync::oneshot::channel::<()>();
+        let router = Router::new().route("/", get(async || "ok"));
+        let server = tokio::spawn(serve_connections(
+            Reporting(listener, accepted_tx),
+            router,
+            async {
+                let _ = stopped.await;
+            },
+            Duration::from_millis(300),
+        ));
+
+        let mut late = tokio::net::TcpStream::connect(addr).await.unwrap();
+        accepted.recv().await.unwrap();
+        let mut silent = tokio::net::TcpStream::connect(addr).await.unwrap();
+        accepted.recv().await.unwrap();
+        stop.send(()).unwrap();
+        // The server accepts nothing more once it shuts down.
+        for _ in 0..100 {
+            if tokio::net::TcpStream::connect(addr).await.is_err() {
+                break;
+            }
+            tokio::time::sleep(Duration::from_millis(10)).await;
+        }
+        assert!(tokio::net::TcpStream::connect(addr).await.is_err());
+        // Let the connections see the shutdown before the request comes.
+        tokio::time::sleep(Duration::from_millis(100)).await;
+
+        late.write_all(b"GET / HTTP/1.1\r\nhost: example\r\n\r\n")
+            .await
+            .unwrap();
+        let mut answer = String::new();
+        late.read_to_string(&mut answer).await.unwrap();
+        assert!(answer.starts_with("HTTP/1.1 200 OK\r\n"), "{answer}");
+        assert!(answer.ends_with("\r\n\r\nok"), "{answer}");
+
+        let mut nothing = Vec::new();
+        silent.read_to_end(&mut nothing).await.unwrap();
+        assert!(nothing.is_empty());
+        tokio::time::timeout(Duration::from_secs(5), server)
+            .await
+            .expect("the server returns once all connections are closed")
+            .unwrap();
+    }
 }
-- 
2.47.3

Required notifications

Address or linkRemarkAdded byNotifiedActions

Relationships

child of 0012175 new taler-api: nine patches that Beekeeper carries on its vendored copy (patch series attached) 

Activities

bot-cg [agent]

2026-11-18 14:42

updater   ~0030583

Correction: this issue was filed in the category "exchange" by mistake. It concerns taler-rust (common/taler-api or common/taler-common), like its parent 0012175, and belongs in the category "taler-rust". This reporter account cannot change the category; could someone with updater rights move it?

Issue History

Date ModifiedUsernameFieldChange
2026-11-18 14:41bot-cgNew Issue
2026-11-18 14:41bot-cgRelationship addedchild of 0012175
2026-11-18 14:41bot-cgImplementer => oec
2026-11-18 14:41bot-cgReviewer => Christian Grothoff
2026-11-18 14:41bot-cgRelease signed by => Christian Grothoff
2026-11-18 14:41bot-cgDeployed by => Christian Grothoff
2026-11-18 14:41bot-cgVerifier => oec
2026-11-18 14:41bot-cgCommunicator => CM_7
2026-11-18 14:42bot-cgFile Added: 0009-api-answer-every-accepted-connection-on-a-graceful-s.patch
2026-11-18 14:42bot-cgNote Added: 0030583