View Issue Details

IDProjectCategoryView StatusLast Update
0007589GNUnetDHT servicepublic2024-10-13 18:53
Reporterulfvonbelow Assigned ToChristian Grothoff  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Platformx86-64OSGuix SystemOS VersionJan 5 2023
Product Version0.19.2 
Target Version0.22.2Fixed in Version0.22.1 
Summary0007589: gnunet-service-dht is constantly at 100% CPU utilization
DescriptionWhile gnunet is running, 'top' reports 100% CPU utilization for gnunet-service-dht. This persists even after I leave it running for a day, so it doesn't seem to be something that only happens when the peer is first created. I don't know how related it is, but 'top' also reports '20t' as the virtual memory usage for every gnunet service, which seems rather strange.

Quote shamelessly taken out of context from the testbed documentation: "On a single affordable computer, it should be possible to run around tens of peers without drastically increasing the load on the system."
Steps To Reproduce1. Get a guix checkout from around Jan 5 on a running guix system
2. Apply the attached patches
3. Reconfigure system with a gnunet service with the attached gnunet.conf (adjust to indicate your own external IP)
4. Run 'top'
5. Observe that gnunet-service-dht is hogging the CPU like mad
Additional InformationMy /etc/gnunet.conf seems relatively normal, I've attached it with some minor redactions. Basically just setting up some paths, and logging, selecting tcp, http_client, and https_client as the transport plugins to use, specifying HOLE_EXTERNAL, and disabling IPv6.

My guix repository includes some extra commits on top of guix master that I rebase every once in awhile - I last rebased on January 5. I modified the gnunet package definition and added a gnunet service to try to enable full functionality. I'll include the relevant patches on top of guix to get that package and service definition. Noteworthy is that I didn't enable the test suite because 0.19.2 and the past 10 versions or so I've tried all have tests that make assumptions that fail in guix's build environment in relation to network namespaces. It's only usually one test, though. I'm currently using Linux 6.1.2 from the nonguix channel, though I don't think that makes much difference compared to linux-libre in this case.

My CPU is an AMD A10-5800k from 2012 or so. I think this fits within the realm of devices gnunet should be able to run decently on.
Tagsngi-entrust-2022-08-104, performance
Attached Files
0001-gnu-setuid-add-setuid-program-base-perms-field-to-se.patch (4,432 bytes)   
gnu: setuid: add setuid-program-base-perms field to <setuid-program>

* gnu/system/setuid.scm (setuid-program-base-perms): new field of
  <setuid-program>.  Export it.
* gnu/services.scm (setuid-program->activation-gexp): include base-perms in
  gexp.
* gnu/build/activation.scm (activate-setuid-programs): honor
  setuid-program-base-perms.
---
 gnu/build/activation.scm | 10 ++++++----
 gnu/services.scm         |  6 ++++--
 gnu/system/setuid.scm    |  7 ++++++-
 3 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/gnu/build/activation.scm b/gnu/build/activation.scm
index eea2233563..a76e8af700 100644
--- a/gnu/build/activation.scm
+++ b/gnu/build/activation.scm
@@ -284,10 +284,10 @@ (define %setuid-directory
 (define (activate-setuid-programs programs)
   "Turn PROGRAMS, a list of file setuid-programs record, into setuid programs
 stored under %SETUID-DIRECTORY."
-  (define (make-setuid-program program setuid? setgid? uid gid)
+  (define (make-setuid-program program setuid? setgid? uid gid base-perms)
     (let ((target (string-append %setuid-directory
                                  "/" (basename program)))
-          (mode (+ #o0555                   ; base permissions
+          (mode (+ (logand #o777 base-perms)
                    (if setuid? #o4000 0)    ; setuid bit
                    (if setgid? #o2000 0)))) ; setgid bit
       (copy-file program target)
@@ -318,8 +318,10 @@ (define (make-setuid-program program setuid? setgid? uid gid)
                                 ((? integer?) user)))
                          (gid (match group
                                 ((? string?) (group:gid (getgrnam group)))
-                                ((? integer?) group))))
-                    (make-setuid-program program-name setuid? setgid? uid gid)))
+                                ((? integer?) group)))
+                         (base-perms (setuid-program-base-perms program)))
+                    (make-setuid-program program-name setuid? setgid? uid gid
+                                         base-perms)))
                 (lambda args
                   ;; If we fail to create a setuid program, better keep going
                   ;; so that we don't leave %SETUID-DIRECTORY empty or
diff --git a/gnu/services.scm b/gnu/services.scm
index 2abef557d4..c5a4173d1c 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -818,13 +818,15 @@ (define (setuid-program->activation-gexp programs)
                                (setuid?      (setuid-program-setuid? program))
                                (setgid?      (setuid-program-setgid? program))
                                (user         (setuid-program-user program))
-                               (group        (setuid-program-group program)) )
+                               (group        (setuid-program-group program))
+                               (base-perms   (setuid-program-base-perms program)))
                            #~(setuid-program
                               (setuid? #$setuid?)
                               (setgid? #$setgid?)
                               (user    #$user)
                               (group   #$group)
-                              (program #$program-name))))
+                              (program #$program-name)
+                              (base-perms #$base-perms))))
                        programs)))
     (with-imported-modules (source-module-closure
                             '((gnu system setuid)))
diff --git a/gnu/system/setuid.scm b/gnu/system/setuid.scm
index 83111d932c..2cc0d22b65 100644
--- a/gnu/system/setuid.scm
+++ b/gnu/system/setuid.scm
@@ -25,6 +25,7 @@ (define-module (gnu system setuid)
             setuid-program-setgid?
             setuid-program-user
             setuid-program-group
+            setuid-program-base-perms
 
             file-like->setuid-program))
 
@@ -51,7 +52,11 @@ (define-record-type* <setuid-program>
                  (default 0))
   ;; Group we want to set this to (defaults to root)
   (group         setuid-program-group   ;integer or string
-                 (default 0)))
+                 (default 0))
+  ;; Base permissions (as passed to chmod) for the program.  Bits above #o777
+  ;; will be ignored.
+  (base-perms    setuid-program-base-perms  ;integer
+                 (default #o555)))
 
 (define (file-like->setuid-program program)
   (setuid-program (program program)))
-- 
2.38.1

Relationships

related to 0007602 closedChristian Grothoff Implement probabilistic block verification in the DHT, as the signature check at this rate kills less powerful peers. 
related to 0007603 new Increase DHT block lifetime or whatever else helps for performance 

Activities

ulfvonbelow

2023-01-15 06:21

reporter   ~0019666

Apparently I can only upload one file per post, so here's 2/3
update-gnunet-and-add-service.patch (14,913 bytes)   
diff --git a/gnu/packages/gnunet.scm b/gnu/packages/gnunet.scm
index c98ceefad7..9012b25158 100644
--- a/gnu/packages/gnunet.scm
+++ b/gnu/packages/gnunet.scm
@@ -46,6 +46,7 @@ (define-module (gnu packages gnunet)
   #:use-module (gnu packages guile)
   #:use-module (gnu packages guile-xyz)
   #:use-module (gnu packages gstreamer)
+  #:use-module (gnu packages jose)
   #:use-module (gnu packages libidn)
   #:use-module (gnu packages linux)
   #:use-module (gnu packages image)
@@ -257,69 +258,94 @@ (define-public gnurl
 
 (define-public gnunet
   (package
-   (name "gnunet")
-   (version "0.16.3")
-   (source
-    (origin
-      (method url-fetch)
-      (uri (string-append "mirror://gnu/gnunet/gnunet-" version
-                          ".tar.gz"))
-      (sha256
-       (base32
-        "12n33r9nnkl5xwx8pwf571l2zvnvfllc8vm6mamrlyjk2cphaf9j"))))
-   (build-system gnu-build-system)
-   (inputs
-    (list bluez
-          glpk
-          gnurl
-          gnutls/dane
-          gstreamer
-          jansson
-          libextractor
-          libidn2
-          libgcrypt
-          libjpeg-turbo
-          libltdl
-          libmicrohttpd
-          libogg
-          libsodium
-          libunistring
-          miniupnpc
-          opus
-          pulseaudio
-          sqlite
-          zbar
-          zlib))
-   (native-inputs
-    (list curl openssl pkg-config python xxd
-          (@ (gnu packages base) which)))
-   (arguments
-    '(#:parallel-tests? #f ; Parallel tests aren't supported.
+    (name "gnunet")
+    (version "0.19.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "mirror://gnu/gnunet/gnunet-" version
+                           ".tar.gz"))
+       (sha256
+        (base32
+         "1zr25jn2jwnlx79wkykgp5jf0rhlxlfh7wcmmlyn5xpqxf94s0w6"))))
+    (build-system gnu-build-system)
+    (inputs
+     (list bluez
+           glpk
+           gnurl
+           gnutls/dane
+           ;;; pulse backend is going to be used anyway... also the configure
+           ;;; script will always say "gstreamer: no" even if it's there
+           ;;; unless it's chosen as the conversations backend.
+           ;;gstreamer
+           ;;gst-plugins-base
+           iproute
+           iptables
+           jansson
+           jose
+           libextractor
+           libidn2
+           libgcrypt
+           libjpeg-turbo
+           libltdl
+           libmicrohttpd
+           libogg
+           libsodium
+           libunistring
+           miniupnpc
+           net-tools
+           opus
+           procps
+           pulseaudio
+           sqlite
+           zbar
+           zlib))
+    (native-inputs
+     (list curl openssl pkg-config python xxd
+           (@ (gnu packages base) which)))
+    (outputs (list "out" "debug"))
+    (arguments
+     (list
+      #:tests? #f ;; tests currently fail, most likely guix-specific?
+      #:parallel-tests? #f              ; Parallel tests aren't supported.
+      #:configure-flags #~(list "--enable-gcc-hardening"
+                                "--enable-linker-hardening"
+                                "--enable-sanitizer"
+                                ;; have to do this to get udp working,
+                                ;; contrary to what configure outputs...
+                                ;;"--enable-experimental"
+                                "CFLAGS=-ggdb3"
+                                "--enable-logging=veryverbose")
       #:phases
-      (modify-phases %standard-phases
-        (add-before 'check 'set-env-var-for-tests
-          (lambda _
-            (setenv "LANG" "en_US.UTF-8")))
-        ;; Swap 'check and 'install phases and add installed binaries to $PATH.
-        (add-before 'check 'set-path-for-check
-          (lambda* (#:key outputs #:allow-other-keys)
-           (let ((out (assoc-ref outputs "out")))
-             (setenv "GNUNET_PREFIX" (string-append out "/lib"))
-             (setenv "PATH" (string-append (getenv "PATH") ":" out "/bin")))
-           #t))
-        (delete 'check)
-        (add-after 'install 'check
-          (assoc-ref %standard-phases 'check)))))
-   (synopsis "Secure, decentralized, peer-to-peer networking framework")
-   (description
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'fix-typo
+            (lambda _
+              (substitute* "./configure"
+                (("-Wstacl-protector")
+                 "-Wstack-protector"))))
+          (add-before 'check 'set-env-var-for-tests
+            (lambda _
+              (setenv "LANG" "en_US.UTF-8")))
+          ;; Swap 'check and 'install phases and add installed binaries to $PATH.
+          (add-before 'check 'set-path-for-check
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let ((out (assoc-ref outputs "out")))
+                (setenv "GNUNET_PREFIX" (string-append out "/lib"))
+                (setenv "PATH" (string-append (getenv "PATH") ":" out "/bin")))
+              #t))
+          (delete 'check)
+          (add-after 'install 'check
+            (assoc-ref %standard-phases 'check)))))
+    (synopsis "Secure, decentralized, peer-to-peer networking framework")
+    (description
      "GNUnet is a framework for secure peer-to-peer networking.  The
 high-level goal is to provide a strong foundation of free software for a
 global, distributed network that provides security and privacy.  GNUnet in
 that sense aims to replace the current internet protocol stack.  Along with
 an application for secure publication of files, it has grown to include all
 kinds of basic applications for the foundation of a GNU internet.")
-   (license license:agpl3+)
-   (home-page "https://gnunet.org/en/")))
+    (license license:agpl3+)
+    (home-page "https://gnunet.org/en/")))
 
 (define-public guile-gnunet                       ;GSoC 2015!
   (let ((commit "d12167ab3c8d7d6caffd9c606e389ef043760602")
@@ -422,14 +448,15 @@ (define-public gnunet-scheme
 (define-public gnunet-gtk
   (package (inherit gnunet)
     (name "gnunet-gtk")
-    (version "0.13.1")
+    (version "0.19.0")
     (source (origin
               (method url-fetch)
               (uri (string-append "mirror://gnu/gnunet/gnunet-gtk-"
                                   version ".tar.gz"))
               (sha256
                (base32
-                "1zdzgq16h77w6ybwg3lqjsjr965np6iqvncqvkbj07glqd4wss0j"))))
+                ;;"1zdzgq16h77w6ybwg3lqjsjr965np6iqvncqvkbj07glqd4wss0j"
+                "0z2731l69vnfsa0cdsw8wh8g1d08wz15y5n0a58qjpf7baric01k"))))
     (arguments
      `(#:configure-flags
        (list "--with-libunique"
@@ -444,7 +471,9 @@ (define-public gnunet-gtk
        ("libextractor" ,libextractor)
        ("libgcrypt" ,libgcrypt)
        ("libunique" ,libunique)
-       ("qrencode" ,qrencode)))
+       ("qrencode" ,qrencode)
+       ;; XXX SHOULD BE FIXED IN GNUNET
+       ("libsodium" ,libsodium)))
     (native-inputs
      (list pkg-config libglade))
     (synopsis "Graphical front-end tools for GNUnet")
diff --git a/gnu/services/networking.scm b/gnu/services/networking.scm
index 702404bc6c..7f76ec8b2e 100644
--- a/gnu/services/networking.scm
+++ b/gnu/services/networking.scm
@@ -45,6 +45,7 @@ (define-module (gnu services networking)
   #:use-module (gnu services admin)
   #:use-module (gnu system shadow)
   #:use-module (gnu system pam)
+  #:use-module (gnu system setuid)
   #:use-module ((gnu system file-systems) #:select (file-system-mapping))
   #:use-module (gnu packages admin)
   #:use-module (gnu packages base)
@@ -60,6 +61,7 @@ (define-module (gnu services networking)
   #:use-module (gnu packages ntp)
   #:use-module (gnu packages gnome)
   #:use-module (gnu packages ipfs)
+  #:use-module (gnu packages gnunet)
   #:use-module (gnu build linux-container)
   #:autoload   (guix least-authority) (least-authority-wrapper)
   #:use-module (guix gexp)
@@ -226,7 +228,15 @@ (define-module (gnu services networking)
 
             keepalived-configuration
             keepalived-configuration?
-            keepalived-service-type))
+            keepalived-service-type
+
+            %default-gnunet-config-file-contents
+            <gnunet-configuration>
+            gnunet-configuration
+            make-gnunet-configuration
+            gnunet-configuration-gnunet
+            gnunet-configuration-config-file
+            gnunet-service-type))
 
 ;;; Commentary:
 ;;;
@@ -2180,4 +2190,171 @@ (define keepalived-service-type
                  "Run @uref{https://www.keepalived.org/, Keepalived}
 routing software.")))
 
+
+;;;
+;;; Gnunet
+;;;
+
+(define %default-gnunet-config-file-contents
+  (string-join
+   '("[PATHS]"
+     "GNUNET_DATA_HOME = /var/lib/gnunet/data/"
+     "GNUNET_RUNTIME_DIR = /var/run/gnunet/"
+     "GNUNET_HOME = /var/lib/gnunet/"
+     "SERVICEHOME = /var/lib/gnunet/"
+     ;; Thank you, this is very useful :-)
+     "SUID_BINARY_PATH = /run/setuid-programs/"
+     ""
+     "[arm]"
+     "SYSTEM_ONLY = YES"
+     "USER_ONLY = NO"
+     "")
+   "\n"
+   'suffix))
+
+(define-record-type* <gnunet-configuration>
+  gnunet-configuration make-gnunet-configuration
+  gnunet-configuration?
+  (gnunet gnunet-configuration-gnunet   ;file-like
+          (default gnunet))
+  (config-file gnunet-configuration-config-file ;file-like
+               (default (plain-file "gnunet.conf"
+	                            %default-gnunet-config-file-contents))))
+
+(define gnunet-setuid-programs
+  (match-lambda
+    (($ <gnunet-configuration> gnunet)
+     (append
+      (list
+       (setuid-program
+        ;; gnunetdns group --> root user
+        (program
+         (file-append gnunet 
+                      "/lib/gnunet/libexec/gnunet-helper-dns"))
+        (setuid? #t)
+        (setgid? #f)
+        (user 0)
+        (group "gnunetdns")
+        (base-perms #o750))
+       (setuid-program
+        ;; gnunet user --> gnunetdns group
+        (program
+         (file-append gnunet
+                      "/lib/gnunet/libexec/gnunet-service-dns"))
+        (setuid? #f)
+        (setgid? #t)
+        (user "gnunet")
+        (group "gnunetdns")
+        (base-perms #o700)))
+      (map (lambda (name)
+             (setuid-program
+              (program (file-append gnunet
+                                    (string-append
+                                     "/lib/gnunet/libexec/gnunet-helper-"
+                                     name)))
+              (setuid? #t)
+              (setgid? #f)
+              (user 0)
+              (group "gnunet")
+              (base-perms #o750)))
+           '("vpn" "wlan" "bluetooth" "exit" "nat-server" "nat-client"))))))
+
+(define gnunet-shepherd-service
+  (match-lambda
+    (($ <gnunet-configuration> gnunet)
+     (define gnunet-arm
+       (file-append gnunet "/bin/gnunet-arm"))
+     (define gnunet-arm-cmd
+       (list gnunet-arm "-c" "/etc/gnunet.conf"))
+     (define (gnunet-arm-with-args . args)
+       #~(`(#$@gnunet-arm-cmd #$@args)
+          #:user "gnunet"
+          #:group "gnunet"
+          #:directory "/var/lib/gnunet"
+          #:environment-variables '(;; necessary for expanding in config files
+                                    "HOME=/var/lib/gnunet")))
+     (define (gnunet-arm-on-services flag services)
+       (gnunet-arm-with-args
+        #~,@(apply append
+                   (map (lambda (service)
+                          (list #$flag service))
+                        #$services))))
+     (define (wait/success? g-exp)
+       #~(zero? (status:exit-val (cdr (waitpid #$g-exp)))))
+
+     (list
+      (shepherd-service
+       (provision '(gnunet))
+       (documentation "Run a gnunet peer as a system user.")
+       (requirement '(networking))
+       (start (with-imported-modules (source-module-closure '((guix build utils)))
+                #~(lambda ()
+                    (use-modules (guix build utils))
+                    (mkdir-p "/var/run/gnunet")
+                    (chmod "/var/run/gnunet" #o755)
+                    (let* ((passwd (getpwnam "gnunet"))
+                           (uid (passwd:uid passwd))
+                           (gid (passwd:gid passwd)))
+                      (chown "/var/run/gnunet" uid gid))
+                    (fork+exec-command #$@(gnunet-arm-with-args "-s" "-m")))))
+       (stop #~(lambda (pid)
+                 (and #$(wait/success?
+                         #~(fork+exec-command
+                            #$@(gnunet-arm-with-args "-e")))
+                      ;; We started it with "-m", so it won't stop till ^C...
+                      (begin (kill pid SIGINT)
+                             (waitpid pid)))))
+       (actions
+        (list
+         (shepherd-action
+          (name 'start-subservices)
+          (documentation "Start the gnunet services provided, a sequence of
+service names as passed to 'gnunet-arm -i'.")
+          (procedure #~(lambda* (_ #:rest services)
+                         (or (null? services)
+                             #$(wait/success?
+                                #~(fork+exec-command
+                                   #$@(gnunet-arm-on-services "-i" 'services)))))))
+         (shepherd-action
+          (name 'stop-subservices)
+          (documentation "Stop the gnunet services provided, a sequence of
+service names as passed to 'gnunet-arm -k'.")
+          (procedure
+           #~(lambda* (_ #:rest services)
+               (or (null? services)
+                   #$(wait/success?
+                      #~(fork+exec-command
+                         #$@(gnunet-arm-on-services "-k" 'services))))))))))))))
+
+
+
+(define gnunet-service-type
+  (service-type
+   (name 'gnunet)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        gnunet-shepherd-service)
+     (service-extension account-service-type
+                        (lambda _
+                          (list (user-group
+                                 (name "gnunet")
+                                 (system? #t))
+                                (user-group
+                                 (name "gnunetdns")
+                                 (system? #t))
+                                (user-account
+                                 (name "gnunet")
+                                 (group "gnunet")
+                                 (system? #t)
+                                 (home-directory "/var/lib/gnunet")))))
+     (service-extension setuid-program-service-type
+                        gnunet-setuid-programs)
+     (service-extension etc-service-type
+                        (lambda (config)
+                          `(("gnunet.conf"
+                             ,(gnunet-configuration-config-file config)))))))
+   (description "Run a gnunet peer as a system user.")
+   (default-value (gnunet-configuration))))
+
 ;;; networking.scm ends here

ulfvonbelow

2023-01-15 06:22

reporter   ~0019667

3/3
gnunet.conf (609 bytes)   
[PATHS]
GNUNET_DATA_HOME = /var/lib/gnunet/data/
GNUNET_RUNTIME_DIR = /var/run/gnunet/
GNUNET_HOME = /var/lib/gnunet/
SERVICEHOME = /var/lib/gnunet/
SUID_BINARY_PATH = /run/setuid-programs/

[arm]
SYSTEM_ONLY = YES
USER_ONLY = NO


GLOBAL_POSTFIX = -l $SERVICEHOME/{}-logs -L INFO

[transport]
PLUGINS = tcp http_client https_client
DISABLEV6 = YES
OPTIONS = -l $SERVICEHOME/transport-logs -L WARNING

[ats]
OPTIONS = -l $SERVICEHOME/ats-logs -L WARNING

[dht]
OPTIONS = -l $SERVICEHOME/dht-logs -L INFO
[transport-tcp]
HOLE_EXTERNAL = <MYDYNAMICDNS>:2086

[transport-udp]
HOLE_EXTERNAL = <MYDYNAMICDNS>:2086
gnunet.conf (609 bytes)   

Christian Grothoff

2023-01-16 09:41

manager   ~0019678

What does
$ gnunet-statistics -s dht
say?
Ideally report twice, with a few minutes wait in between. Maybe the DHT is indeed processing tons of requests? I'm not seeing excessive DHT load on my end, but if someone is currently importing millions of records into GNS (say all of ".fr") and few peers are running, some might be hit with a bit much signature verification work. At least that's one possible theory, but would be good to first identify the source of the high CPU load on your system.

Attaching gdb and getting a few stack traces (to the process with the high CPU load) or generating a flamegraph could all also help to find out where the load is coming from. On my systems, load right now is not very high at all.

Christian Grothoff

2023-01-16 10:42

manager   ~0019679

I just checked, and it looks like we're currently importing 250k records from ".se" into GNS/DHT. This alone results (for some reason, likely the frequency is excessively high due to configuration failures or bugs) in 38 million (!) DHT PUT operations per day. I suspect your older CPU is not quite able to validate it's share of cryptographic signatures, hence the 100% CPU load. The real bug is probably related to the logic or configuration for importing ".se".

schanzen

2023-01-16 13:46

administrator   ~0019680

Isn't the real bug that our server is DoSing this client??

schanzen

2023-01-16 13:48

administrator   ~0019681

Oh and can you open another issue for the packagin packages please with detailed descriptions on what they do? We are happy to include packaging stuff.

ulfvonbelow

2023-01-17 03:28

reporter   ~0019686

$ gnunet-statistics -s dht && sleep 60 && echo && gnunet-statistics -s dht
          dht # P2P GET bytes received: 1912
          dht # PUT requests received from clients: 5
          dht # P2P PUT requests received: 4436
          dht # Local PUT requests not routed: 5
          dht # GET requests given to datacache: 2
          dht # Entries added to routing table: 7
          dht # P2P HELLO lookup requests processed: 3
          dht # P2P GET requests received: 7
          dht # peers connected: 5
          dht # FIND PEER messages initiated: 1
          dht # P2P PUT bytes received: 2250912
          dht # requests TTL-dropped: 24
          dht # ITEMS stored in datacache: 4441
          dht # PUT messages queued for transmission: 2484
          dht # P2P GET requests ONLY routed: 2
          dht # REPLIES ignored for CLIENTS (no match): 4441
          dht # messages dropped (underlays busy): 1927
          dht # GET messages queued for transmission: 8
          dht # PUT requests routed: 4441
          dht # Peer selection failed: 2311
          dht # GET requests routed: 8
          dht # RESULT messages queued for transmission: 3
          dht # DHT requests combined: 1
          dht # Network size estimates received: 3

          dht # P2P GET bytes received: 2148
          dht # PUT requests received from clients: 5
          dht # P2P PUT requests received: 10400
          dht # Local PUT requests not routed: 5
          dht # GET requests given to datacache: 2
          dht # Entries added to routing table: 8
          dht # P2P HELLO lookup requests processed: 3
          dht # P2P GET requests received: 8
          dht # peers connected: 5
          dht # FIND PEER messages initiated: 1
          dht # P2P PUT bytes received: 5302720
          dht # requests TTL-dropped: 24
          dht # ITEMS stored in datacache: 10405
          dht # PUT messages queued for transmission: 7068
          dht # P2P GET requests ONLY routed: 3
          dht # REPLIES ignored for CLIENTS (no match): 10405
          dht # messages dropped (underlays busy): 6362
          dht # GET messages queued for transmission: 9
          dht # PUT requests routed: 10405
          dht # Peer selection failed: 4354
          dht # GET requests routed: 9
          dht # RESULT messages queued for transmission: 3
          dht # DHT requests combined: 1
          dht # Network size estimates received: 3

It looks like that comes out to around 99.4 PUT requests received per second. If you're putting out 38 million per day, that's 439 per second, so a little under 1/4 of them (not taking other PUTers into account) are ending up at my peer. Small world so far, I guess.

I don't know how to generate that flame graph you mentioned, so I've attached a random sampling of a few gdb backtraces.
(gdb) bt
#0 0x00007fb1da22b034 in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#1 0x00007fb1da2acb04 in free ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#2 0x00007fb1d98ef198 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#3 0x00007fb1d99ad12a in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#4 0x00007fb1d99ad4d3 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#5 0x00007fb1d99b4437 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#6 0x00007fb1d99b7156 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#7 0x00007fb1d998054d in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#8 0x00007fb1d9979aa9 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#9 0x00007fb1d9900832 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#10 0x00007fb1d98ec9ae in gcry_pk_verify ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#11 0x00007fb1d9c4ee01 in GNUNET_CRYPTO_ecdsa_verify_ (purpose=15,
    validate=0x61200005c2c0, sig=0x62f000006d28, pub=0x62f000006d08)
    at crypto_ecc.c:682
#12 0x00007fb1d577a3b9 in GNUNET_GNSRECORD_block_verify (block=0x62f000006d00)
    at gnsrecord_crypto.c:708
#13 0x00007fb1d57b9dde in block_plugin_gns_check_block (cls=0x0,
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000006d00, block_size=368)
    at plugin_block_gns.c:197
#14 0x00007fb1da0351da in GNUNET_BLOCK_check_block (ctx=0x60300000b1d0,
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000006d00, block_size=368)
    at block.c:329
#15 0x0000559ffd8991f1 in handle_dht_p2p_put (cls=0x606000232b80,
    put=0x62f000006c28) at gnunet-service-dht_neighbours.c:2000
#16 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x7ffdd3e4a890,
    mh=0x62f000006c28) at mq.c:242
#17 0x0000559ffd8a10eb in GDS_u_receive (cls=0x60b000004a80, tctx=0x60800000ed20,
    sctx=0x608000009020, message=0x62f000006c28, message_size=584)
    at gnunet-service-dht_neighbours.c:2836
#18 0x00007fb1d571baab in handle_core_message (cls=0x60800000ed20,
    msg=0x62f000006c24) at plugin_dhtu_gnunet.c:515
#19 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x6060002b2d40,
    mh=0x62f000006c24) at mq.c:242
#20 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000042880,
    mh=0x62f000006c24) at mq.c:192
#21 0x00007fb1da198b0d in handle_notify_inbound (cls=0x60c000000a00,
    ntm=0x62f000006c00) at core_api.c:574
#22 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x610000001540,
    mh=0x62f000006c00) at mq.c:242
#23 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000000b80,
    mh=0x62f000006c00) at mq.c:192
#24 0x00007fb1d9be6f06 in recv_message (cls=0x60d0000006c0, msg=0x62f000006c00)
    at client.c:347
#25 0x00007fb1d9ca44d4 in GNUNET_MST_from_buffer (mst=0x60400000f990, buf=0x0,
    size=0, purge=0, one_shot=0) at mst.c:222
#26 0x00007fb1d9ca5e84 in GNUNET_MST_read (mst=0x60400000f990,
    sock=0x60300000d9f0, purge=0, one_shot=0) at mst.c:367
#27 0x00007fb1d9be8784 in receive_ready (cls=0x60d0000006c0) at client.c:447
#28 0x00007fb1d9cf7663 in GNUNET_SCHEDULER_do_work (sh=0x606000007be0)
    at scheduler.c:2107
#29 0x00007fb1d9cf9fa0 in select_loop (sh=0x606000007be0, context=0x7ffdd3e4af90)
    at scheduler.c:2412
#30 0x00007fb1d9ce8649 in GNUNET_SCHEDULER_run (
    task=0x7fb1d9d01f94 <service_main>, task_cls=0x7ffdd3e4b240) at scheduler.c:710
#31 0x00007fb1d9d105c1 in GNUNET_SERVICE_run_ (argc=11, argv=0x7ffdd3e4bab8,
    service_name=0x559ffd8a6180 "dht", options=GNUNET_SERVICE_OPTION_NONE,
    service_init_cb=0x559ffd885b78 <run>,
    connect_cb=0x559ffd8739b0 <client_connect_cb>,
    disconnect_cb=0x559ffd873af5 <client_disconnect_cb>, cls=0x0,
    handlers=0x7ffdd3e4b820) at service.c:2136
#32 0x0000559ffd88694f in main (argc=11, argv=0x7ffdd3e4bab8)
    at gnunet-service-dht.c:540
(gdb) continue
Continuing.
  C-c C-c
Program received signal SIGINT, Interrupt.
0x00007fb1da22ef3c in ?? () from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
(gdb) bt
#0 0x00007fb1da22ef3c in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#1 0x00007fb1da22f3e6 in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#2 0x00007fb1da22f59f in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#3 0x00007fb1da22b363 in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#4 0x00007fb1da2acb04 in free ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#5 0x00007fb1d98ef198 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#6 0x00007fb1d99ad12a in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#7 0x00007fb1d99ad4d3 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#8 0x00007fb1d99b57b7 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#9 0x00007fb1d99b7137 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#10 0x00007fb1d998057c in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#11 0x00007fb1d9979aa9 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#12 0x00007fb1d9900832 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#13 0x00007fb1d98ec9ae in gcry_pk_verify ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#14 0x00007fb1d9c4ee01 in GNUNET_CRYPTO_ecdsa_verify_ (purpose=15,
    validate=0x60d00001a1e0, sig=0x62f000006968, pub=0x62f000006948)
    at crypto_ecc.c:682
#15 0x00007fb1d577a3b9 in GNUNET_GNSRECORD_block_verify (block=0x62f000006940)
    at gnsrecord_crypto.c:708
#16 0x00007fb1d57b9dde in block_plugin_gns_check_block (cls=0x0,
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000006940, block_size=240)
    at plugin_block_gns.c:197
#17 0x00007fb1da0351da in GNUNET_BLOCK_check_block (ctx=0x60300000b1d0,
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000006940, block_size=240)
    at block.c:329
#18 0x0000559ffd8991f1 in handle_dht_p2p_put (cls=0x6060000097a0,
    put=0x62f000006868) at gnunet-service-dht_neighbours.c:2000
#19 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x7ffdd3e4a890,
    mh=0x62f000006868) at mq.c:242
#20 0x0000559ffd8a10eb in GDS_u_receive (cls=0x60b000004a80, tctx=0x60800000aea0,
    sctx=0x608000009020, message=0x62f000006868, message_size=456)
    at gnunet-service-dht_neighbours.c:2836
#21 0x00007fb1d571baab in handle_core_message (cls=0x60800000aea0,
    msg=0x62f000006864) at plugin_dhtu_gnunet.c:515
#22 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x606000009740,
    mh=0x62f000006864) at mq.c:242
#23 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000009340,
    mh=0x62f000006864) at mq.c:192
#24 0x00007fb1da198b0d in handle_notify_inbound (cls=0x60c000000a00,
    ntm=0x62f000006840) at core_api.c:574
#25 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x610000001540,
    mh=0x62f000006840) at mq.c:242
#26 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000000b80,
    mh=0x62f000006840) at mq.c:192
#27 0x00007fb1d9be6f06 in recv_message (cls=0x60d0000006c0, msg=0x62f000006840)
    at client.c:347
#28 0x00007fb1d9ca44d4 in GNUNET_MST_from_buffer (mst=0x60400000f990, buf=0x0,
    size=0, purge=0, one_shot=0) at mst.c:222
#29 0x00007fb1d9ca5e84 in GNUNET_MST_read (mst=0x60400000f990,
    sock=0x60300000d9f0, purge=0, one_shot=0) at mst.c:367
#30 0x00007fb1d9be8784 in receive_ready (cls=0x60d0000006c0) at client.c:447
#31 0x00007fb1d9cf7663 in GNUNET_SCHEDULER_do_work (sh=0x606000007be0)
    at scheduler.c:2107
#32 0x00007fb1d9cf9fa0 in select_loop (sh=0x606000007be0, context=0x7ffdd3e4af90)
    at scheduler.c:2412
#33 0x00007fb1d9ce8649 in GNUNET_SCHEDULER_run (
    task=0x7fb1d9d01f94 <service_main>, task_cls=0x7ffdd3e4b240) at scheduler.c:710
#34 0x00007fb1d9d105c1 in GNUNET_SERVICE_run_ (argc=11, argv=0x7ffdd3e4bab8,
    service_name=0x559ffd8a6180 "dht", options=GNUNET_SERVICE_OPTION_NONE,
    service_init_cb=0x559ffd885b78 <run>,
    connect_cb=0x559ffd8739b0 <client_connect_cb>,
    disconnect_cb=0x559ffd873af5 <client_disconnect_cb>, cls=0x0,
    handlers=0x7ffdd3e4b820) at service.c:2136
#35 0x0000559ffd88694f in main (argc=11, argv=0x7ffdd3e4bab8)
    at gnunet-service-dht.c:540
(gdb) continue
Continuing.
  C-c C-c
Program received signal SIGINT, Interrupt.
0x00007fb1d99b1632 in ?? () from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
(gdb) bt
#0 0x00007fb1d99b1632 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#1 0x00007fb1d99ad096 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#2 0x00007fb1d99ad4d3 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#3 0x00007fb1d99b42f7 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#4 0x00007fb1d99b7156 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#5 0x00007fb1d998054d in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#6 0x00007fb1d9979aa9 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#7 0x00007fb1d9900832 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#8 0x00007fb1d98ec9ae in gcry_pk_verify ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#9 0x00007fb1d9c4ee01 in GNUNET_CRYPTO_ecdsa_verify_ (purpose=15,
    validate=0x61200005f140, sig=0x62f000007858, pub=0x62f000007838)
    at crypto_ecc.c:682
#10 0x00007fb1d577a3b9 in GNUNET_GNSRECORD_block_verify (block=0x62f000007830)
    at gnsrecord_crypto.c:708
#11 0x00007fb1d57b9dde in block_plugin_gns_check_block (cls=0x0,
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000007830, block_size=368)
    at plugin_block_gns.c:197
#12 0x00007fb1da0351da in GNUNET_BLOCK_check_block (ctx=0x60300000b1d0,
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000007830, block_size=368)
    at block.c:329
#13 0x0000559ffd8991f1 in handle_dht_p2p_put (cls=0x6060000097a0,
    put=0x62f000007758) at gnunet-service-dht_neighbours.c:2000
#14 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x7ffdd3e4a890,
    mh=0x62f000007758) at mq.c:242
#15 0x0000559ffd8a10eb in GDS_u_receive (cls=0x60b000004a80, tctx=0x60800000aea0,
    sctx=0x608000009020, message=0x62f000007758, message_size=584)
    at gnunet-service-dht_neighbours.c:2836
#16 0x00007fb1d571baab in handle_core_message (cls=0x60800000aea0,
    msg=0x62f000007754) at plugin_dhtu_gnunet.c:515
#17 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x606000009740,
    mh=0x62f000007754) at mq.c:242
#18 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000009340,
    mh=0x62f000007754) at mq.c:192
#19 0x00007fb1da198b0d in handle_notify_inbound (cls=0x60c000000a00,
    ntm=0x62f000007730) at core_api.c:574
#20 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x610000001540,
    mh=0x62f000007730) at mq.c:242
#21 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000000b80,
    mh=0x62f000007730) at mq.c:192
#22 0x00007fb1d9be6f06 in recv_message (cls=0x60d0000006c0, msg=0x62f000007730)
    at client.c:347
#23 0x00007fb1d9ca44d4 in GNUNET_MST_from_buffer (mst=0x60400000f990, buf=0x0,
    size=0, purge=0, one_shot=0) at mst.c:222
#24 0x00007fb1d9ca5e84 in GNUNET_MST_read (mst=0x60400000f990,
    sock=0x60300000d9f0, purge=0, one_shot=0) at mst.c:367
#25 0x00007fb1d9be8784 in receive_ready (cls=0x60d0000006c0) at client.c:447
#26 0x00007fb1d9cf7663 in GNUNET_SCHEDULER_do_work (sh=0x606000007be0)
    at scheduler.c:2107
#27 0x00007fb1d9cf9fa0 in select_loop (sh=0x606000007be0, context=0x7ffdd3e4af90)
    at scheduler.c:2412
#28 0x00007fb1d9ce8649 in GNUNET_SCHEDULER_run (
    task=0x7fb1d9d01f94 <service_main>, task_cls=0x7ffdd3e4b240) at scheduler.c:710
#29 0x00007fb1d9d105c1 in GNUNET_SERVICE_run_ (argc=11, argv=0x7ffdd3e4bab8,
    service_name=0x559ffd8a6180 "dht", options=GNUNET_SERVICE_OPTION_NONE,
    service_init_cb=0x559ffd885b78 <run>,
    connect_cb=0x559ffd8739b0 <client_connect_cb>,
    disconnect_cb=0x559ffd873af5 <client_disconnect_cb>, cls=0x0,
    handlers=0x7ffdd3e4b820) at service.c:2136
#30 0x0000559ffd88694f in main (argc=11, argv=0x7ffdd3e4bab8)
    at gnunet-service-dht.c:540
(gdb) continue
Continuing.
  C-c C-c
Program received signal SIGINT, Interrupt.
0x00007fb1da22db5f in ?? () from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
(gdb) bt
#0 0x00007fb1da22db5f in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#1 0x00007fb1da22a70c in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#2 0x00007fb1da2acfdd in calloc ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#3 0x00007fb1d9c63347 in w_malloc (n=32) at crypto_random.c:344
#4 0x00007fb1d98ee78d in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#5 0x00007fb1d98efe7d in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#6 0x00007fb1d99b2d55 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#7 0x00007fb1d99ad02f in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#8 0x00007fb1d99ad4d3 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#9 0x00007fb1d99b42f7 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#10 0x00007fb1d99b7156 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#11 0x00007fb1d998057c in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#12 0x00007fb1d9979aa9 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#13 0x00007fb1d9900832 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#14 0x00007fb1d98ec9ae in gcry_pk_verify ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#15 0x00007fb1d9c4ee01 in GNUNET_CRYPTO_ecdsa_verify_ (purpose=15,
    validate=0x60d0000059a0, sig=0x62f00000a2a8, pub=0x62f00000a288)
    at crypto_ecc.c:682
#16 0x00007fb1d577a3b9 in GNUNET_GNSRECORD_block_verify (block=0x62f00000a280)
    at gnsrecord_crypto.c:708
#17 0x00007fb1d57b9dde in block_plugin_gns_check_block (cls=0x0,
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f00000a280, block_size=240)
    at plugin_block_gns.c:197
#18 0x00007fb1da0351da in GNUNET_BLOCK_check_block (ctx=0x60300000b1d0,
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f00000a280, block_size=240)
    at block.c:329
#19 0x0000559ffd8991f1 in handle_dht_p2p_put (cls=0x60600007d4e0,
    put=0x62f00000a1a8) at gnunet-service-dht_neighbours.c:2000
#20 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x7ffdd3e4a890,
    mh=0x62f00000a1a8) at mq.c:242
#21 0x0000559ffd8a10eb in GDS_u_receive (cls=0x60b000004a80, tctx=0x60800001ffa0,
    sctx=0x608000009020, message=0x62f00000a1a8, message_size=456)
    at gnunet-service-dht_neighbours.c:2836
#22 0x00007fb1d571baab in handle_core_message (cls=0x60800001ffa0,
    msg=0x62f00000a1a4) at plugin_dhtu_gnunet.c:515
#23 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x60600007d540,
    mh=0x62f00000a1a4) at mq.c:242
#24 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000033f40,
    mh=0x62f00000a1a4) at mq.c:192
#25 0x00007fb1da198b0d in handle_notify_inbound (cls=0x60c000000a00,
    ntm=0x62f00000a180) at core_api.c:574
#26 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x610000001540,
    mh=0x62f00000a180) at mq.c:242
#27 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000000b80,
    mh=0x62f00000a180) at mq.c:192
#28 0x00007fb1d9be6f06 in recv_message (cls=0x60d0000006c0, msg=0x62f00000a180)
    at client.c:347
#29 0x00007fb1d9ca44d4 in GNUNET_MST_from_buffer (mst=0x60400000f990, buf=0x0,
    size=0, purge=0, one_shot=0) at mst.c:222
#30 0x00007fb1d9ca5e84 in GNUNET_MST_read (mst=0x60400000f990,
    sock=0x60300000d9f0, purge=0, one_shot=0) at mst.c:367
#31 0x00007fb1d9be8784 in receive_ready (cls=0x60d0000006c0) at client.c:447
#32 0x00007fb1d9cf7663 in GNUNET_SCHEDULER_do_work (sh=0x606000007be0)
    at scheduler.c:2107
#33 0x00007fb1d9cf9fa0 in select_loop (sh=0x606000007be0, context=0x7ffdd3e4af90)
    at scheduler.c:2412
#34 0x00007fb1d9ce8649 in GNUNET_SCHEDULER_run (
    task=0x7fb1d9d01f94 <service_main>, task_cls=0x7ffdd3e4b240) at scheduler.c:710
#35 0x00007fb1d9d105c1 in GNUNET_SERVICE_run_ (argc=11, argv=0x7ffdd3e4bab8,
    service_name=0x559ffd8a6180 "dht", options=GNUNET_SERVICE_OPTION_NONE,
    service_init_cb=0x559ffd885b78 <run>,
    connect_cb=0x559ffd8739b0 <client_connect_cb>,
    disconnect_cb=0x559ffd873af5 <client_disconnect_cb>, cls=0x0,
    handlers=0x7ffdd3e4b820) at service.c:2136
#36 0x0000559ffd88694f in main (argc=11, argv=0x7ffdd3e4bab8)
    at gnunet-service-dht.c:540
gdb-backtraces (18,246 bytes)   
(gdb) bt
#0  0x00007fb1da22b034 in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#1  0x00007fb1da2acb04 in free ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#2  0x00007fb1d98ef198 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#3  0x00007fb1d99ad12a in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#4  0x00007fb1d99ad4d3 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#5  0x00007fb1d99b4437 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#6  0x00007fb1d99b7156 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#7  0x00007fb1d998054d in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#8  0x00007fb1d9979aa9 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#9  0x00007fb1d9900832 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#10 0x00007fb1d98ec9ae in gcry_pk_verify ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#11 0x00007fb1d9c4ee01 in GNUNET_CRYPTO_ecdsa_verify_ (purpose=15, 
    validate=0x61200005c2c0, sig=0x62f000006d28, pub=0x62f000006d08)
    at crypto_ecc.c:682
#12 0x00007fb1d577a3b9 in GNUNET_GNSRECORD_block_verify (block=0x62f000006d00)
    at gnsrecord_crypto.c:708
#13 0x00007fb1d57b9dde in block_plugin_gns_check_block (cls=0x0, 
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000006d00, block_size=368)
    at plugin_block_gns.c:197
#14 0x00007fb1da0351da in GNUNET_BLOCK_check_block (ctx=0x60300000b1d0, 
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000006d00, block_size=368)
    at block.c:329
#15 0x0000559ffd8991f1 in handle_dht_p2p_put (cls=0x606000232b80, 
    put=0x62f000006c28) at gnunet-service-dht_neighbours.c:2000
#16 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x7ffdd3e4a890, 
    mh=0x62f000006c28) at mq.c:242
#17 0x0000559ffd8a10eb in GDS_u_receive (cls=0x60b000004a80, tctx=0x60800000ed20, 
    sctx=0x608000009020, message=0x62f000006c28, message_size=584)
    at gnunet-service-dht_neighbours.c:2836
#18 0x00007fb1d571baab in handle_core_message (cls=0x60800000ed20, 
    msg=0x62f000006c24) at plugin_dhtu_gnunet.c:515
#19 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x6060002b2d40, 
    mh=0x62f000006c24) at mq.c:242
#20 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000042880, 
    mh=0x62f000006c24) at mq.c:192
#21 0x00007fb1da198b0d in handle_notify_inbound (cls=0x60c000000a00, 
    ntm=0x62f000006c00) at core_api.c:574
#22 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x610000001540, 
    mh=0x62f000006c00) at mq.c:242
#23 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000000b80, 
    mh=0x62f000006c00) at mq.c:192
#24 0x00007fb1d9be6f06 in recv_message (cls=0x60d0000006c0, msg=0x62f000006c00)
    at client.c:347
#25 0x00007fb1d9ca44d4 in GNUNET_MST_from_buffer (mst=0x60400000f990, buf=0x0, 
    size=0, purge=0, one_shot=0) at mst.c:222
#26 0x00007fb1d9ca5e84 in GNUNET_MST_read (mst=0x60400000f990, 
    sock=0x60300000d9f0, purge=0, one_shot=0) at mst.c:367
#27 0x00007fb1d9be8784 in receive_ready (cls=0x60d0000006c0) at client.c:447
#28 0x00007fb1d9cf7663 in GNUNET_SCHEDULER_do_work (sh=0x606000007be0)
    at scheduler.c:2107
#29 0x00007fb1d9cf9fa0 in select_loop (sh=0x606000007be0, context=0x7ffdd3e4af90)
    at scheduler.c:2412
#30 0x00007fb1d9ce8649 in GNUNET_SCHEDULER_run (
    task=0x7fb1d9d01f94 <service_main>, task_cls=0x7ffdd3e4b240) at scheduler.c:710
#31 0x00007fb1d9d105c1 in GNUNET_SERVICE_run_ (argc=11, argv=0x7ffdd3e4bab8, 
    service_name=0x559ffd8a6180 "dht", options=GNUNET_SERVICE_OPTION_NONE, 
    service_init_cb=0x559ffd885b78 <run>, 
    connect_cb=0x559ffd8739b0 <client_connect_cb>, 
    disconnect_cb=0x559ffd873af5 <client_disconnect_cb>, cls=0x0, 
    handlers=0x7ffdd3e4b820) at service.c:2136
#32 0x0000559ffd88694f in main (argc=11, argv=0x7ffdd3e4bab8)
    at gnunet-service-dht.c:540
(gdb) continue
Continuing.
  C-c C-c
Program received signal SIGINT, Interrupt.
0x00007fb1da22ef3c in ?? () from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
(gdb) bt
#0  0x00007fb1da22ef3c in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#1  0x00007fb1da22f3e6 in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#2  0x00007fb1da22f59f in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#3  0x00007fb1da22b363 in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#4  0x00007fb1da2acb04 in free ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#5  0x00007fb1d98ef198 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#6  0x00007fb1d99ad12a in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#7  0x00007fb1d99ad4d3 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#8  0x00007fb1d99b57b7 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#9  0x00007fb1d99b7137 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#10 0x00007fb1d998057c in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#11 0x00007fb1d9979aa9 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#12 0x00007fb1d9900832 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#13 0x00007fb1d98ec9ae in gcry_pk_verify ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#14 0x00007fb1d9c4ee01 in GNUNET_CRYPTO_ecdsa_verify_ (purpose=15, 
    validate=0x60d00001a1e0, sig=0x62f000006968, pub=0x62f000006948)
    at crypto_ecc.c:682
#15 0x00007fb1d577a3b9 in GNUNET_GNSRECORD_block_verify (block=0x62f000006940)
    at gnsrecord_crypto.c:708
#16 0x00007fb1d57b9dde in block_plugin_gns_check_block (cls=0x0, 
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000006940, block_size=240)
    at plugin_block_gns.c:197
#17 0x00007fb1da0351da in GNUNET_BLOCK_check_block (ctx=0x60300000b1d0, 
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000006940, block_size=240)
    at block.c:329
#18 0x0000559ffd8991f1 in handle_dht_p2p_put (cls=0x6060000097a0, 
    put=0x62f000006868) at gnunet-service-dht_neighbours.c:2000
#19 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x7ffdd3e4a890, 
    mh=0x62f000006868) at mq.c:242
#20 0x0000559ffd8a10eb in GDS_u_receive (cls=0x60b000004a80, tctx=0x60800000aea0, 
    sctx=0x608000009020, message=0x62f000006868, message_size=456)
    at gnunet-service-dht_neighbours.c:2836
#21 0x00007fb1d571baab in handle_core_message (cls=0x60800000aea0, 
    msg=0x62f000006864) at plugin_dhtu_gnunet.c:515
#22 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x606000009740, 
    mh=0x62f000006864) at mq.c:242
#23 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000009340, 
    mh=0x62f000006864) at mq.c:192
#24 0x00007fb1da198b0d in handle_notify_inbound (cls=0x60c000000a00, 
    ntm=0x62f000006840) at core_api.c:574
#25 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x610000001540, 
    mh=0x62f000006840) at mq.c:242
#26 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000000b80, 
    mh=0x62f000006840) at mq.c:192
#27 0x00007fb1d9be6f06 in recv_message (cls=0x60d0000006c0, msg=0x62f000006840)
    at client.c:347
#28 0x00007fb1d9ca44d4 in GNUNET_MST_from_buffer (mst=0x60400000f990, buf=0x0, 
    size=0, purge=0, one_shot=0) at mst.c:222
#29 0x00007fb1d9ca5e84 in GNUNET_MST_read (mst=0x60400000f990, 
    sock=0x60300000d9f0, purge=0, one_shot=0) at mst.c:367
#30 0x00007fb1d9be8784 in receive_ready (cls=0x60d0000006c0) at client.c:447
#31 0x00007fb1d9cf7663 in GNUNET_SCHEDULER_do_work (sh=0x606000007be0)
    at scheduler.c:2107
#32 0x00007fb1d9cf9fa0 in select_loop (sh=0x606000007be0, context=0x7ffdd3e4af90)
    at scheduler.c:2412
#33 0x00007fb1d9ce8649 in GNUNET_SCHEDULER_run (
    task=0x7fb1d9d01f94 <service_main>, task_cls=0x7ffdd3e4b240) at scheduler.c:710
#34 0x00007fb1d9d105c1 in GNUNET_SERVICE_run_ (argc=11, argv=0x7ffdd3e4bab8, 
    service_name=0x559ffd8a6180 "dht", options=GNUNET_SERVICE_OPTION_NONE, 
    service_init_cb=0x559ffd885b78 <run>, 
    connect_cb=0x559ffd8739b0 <client_connect_cb>, 
    disconnect_cb=0x559ffd873af5 <client_disconnect_cb>, cls=0x0, 
    handlers=0x7ffdd3e4b820) at service.c:2136
#35 0x0000559ffd88694f in main (argc=11, argv=0x7ffdd3e4bab8)
    at gnunet-service-dht.c:540
(gdb) continue
Continuing.
  C-c C-c
Program received signal SIGINT, Interrupt.
0x00007fb1d99b1632 in ?? () from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
(gdb) bt
#0  0x00007fb1d99b1632 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#1  0x00007fb1d99ad096 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#2  0x00007fb1d99ad4d3 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#3  0x00007fb1d99b42f7 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#4  0x00007fb1d99b7156 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#5  0x00007fb1d998054d in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#6  0x00007fb1d9979aa9 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#7  0x00007fb1d9900832 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#8  0x00007fb1d98ec9ae in gcry_pk_verify ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#9  0x00007fb1d9c4ee01 in GNUNET_CRYPTO_ecdsa_verify_ (purpose=15, 
    validate=0x61200005f140, sig=0x62f000007858, pub=0x62f000007838)
    at crypto_ecc.c:682
#10 0x00007fb1d577a3b9 in GNUNET_GNSRECORD_block_verify (block=0x62f000007830)
    at gnsrecord_crypto.c:708
#11 0x00007fb1d57b9dde in block_plugin_gns_check_block (cls=0x0, 
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000007830, block_size=368)
    at plugin_block_gns.c:197
#12 0x00007fb1da0351da in GNUNET_BLOCK_check_block (ctx=0x60300000b1d0, 
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f000007830, block_size=368)
    at block.c:329
#13 0x0000559ffd8991f1 in handle_dht_p2p_put (cls=0x6060000097a0, 
    put=0x62f000007758) at gnunet-service-dht_neighbours.c:2000
#14 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x7ffdd3e4a890, 
    mh=0x62f000007758) at mq.c:242
#15 0x0000559ffd8a10eb in GDS_u_receive (cls=0x60b000004a80, tctx=0x60800000aea0, 
    sctx=0x608000009020, message=0x62f000007758, message_size=584)
    at gnunet-service-dht_neighbours.c:2836
#16 0x00007fb1d571baab in handle_core_message (cls=0x60800000aea0, 
    msg=0x62f000007754) at plugin_dhtu_gnunet.c:515
#17 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x606000009740, 
    mh=0x62f000007754) at mq.c:242
#18 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000009340, 
    mh=0x62f000007754) at mq.c:192
#19 0x00007fb1da198b0d in handle_notify_inbound (cls=0x60c000000a00, 
    ntm=0x62f000007730) at core_api.c:574
#20 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x610000001540, 
    mh=0x62f000007730) at mq.c:242
#21 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000000b80, 
    mh=0x62f000007730) at mq.c:192
#22 0x00007fb1d9be6f06 in recv_message (cls=0x60d0000006c0, msg=0x62f000007730)
    at client.c:347
#23 0x00007fb1d9ca44d4 in GNUNET_MST_from_buffer (mst=0x60400000f990, buf=0x0, 
    size=0, purge=0, one_shot=0) at mst.c:222
#24 0x00007fb1d9ca5e84 in GNUNET_MST_read (mst=0x60400000f990, 
    sock=0x60300000d9f0, purge=0, one_shot=0) at mst.c:367
#25 0x00007fb1d9be8784 in receive_ready (cls=0x60d0000006c0) at client.c:447
#26 0x00007fb1d9cf7663 in GNUNET_SCHEDULER_do_work (sh=0x606000007be0)
    at scheduler.c:2107
#27 0x00007fb1d9cf9fa0 in select_loop (sh=0x606000007be0, context=0x7ffdd3e4af90)
    at scheduler.c:2412
#28 0x00007fb1d9ce8649 in GNUNET_SCHEDULER_run (
    task=0x7fb1d9d01f94 <service_main>, task_cls=0x7ffdd3e4b240) at scheduler.c:710
#29 0x00007fb1d9d105c1 in GNUNET_SERVICE_run_ (argc=11, argv=0x7ffdd3e4bab8, 
    service_name=0x559ffd8a6180 "dht", options=GNUNET_SERVICE_OPTION_NONE, 
    service_init_cb=0x559ffd885b78 <run>, 
    connect_cb=0x559ffd8739b0 <client_connect_cb>, 
    disconnect_cb=0x559ffd873af5 <client_disconnect_cb>, cls=0x0, 
    handlers=0x7ffdd3e4b820) at service.c:2136
#30 0x0000559ffd88694f in main (argc=11, argv=0x7ffdd3e4bab8)
    at gnunet-service-dht.c:540
(gdb) continue
Continuing.
  C-c C-c
Program received signal SIGINT, Interrupt.
0x00007fb1da22db5f in ?? () from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
(gdb) bt
#0  0x00007fb1da22db5f in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#1  0x00007fb1da22a70c in ?? ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#2  0x00007fb1da2acfdd in calloc ()
   from /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libasan.so.6
#3  0x00007fb1d9c63347 in w_malloc (n=32) at crypto_random.c:344
#4  0x00007fb1d98ee78d in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#5  0x00007fb1d98efe7d in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#6  0x00007fb1d99b2d55 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#7  0x00007fb1d99ad02f in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#8  0x00007fb1d99ad4d3 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#9  0x00007fb1d99b42f7 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#10 0x00007fb1d99b7156 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#11 0x00007fb1d998057c in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#12 0x00007fb1d9979aa9 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#13 0x00007fb1d9900832 in ?? ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#14 0x00007fb1d98ec9ae in gcry_pk_verify ()
   from /gnu/store/3kl94m3ksm45a880b6lnn3kagk857lj9-libgcrypt-1.8.8/lib/libgcrypt.so.20
#15 0x00007fb1d9c4ee01 in GNUNET_CRYPTO_ecdsa_verify_ (purpose=15, 
    validate=0x60d0000059a0, sig=0x62f00000a2a8, pub=0x62f00000a288)
    at crypto_ecc.c:682
#16 0x00007fb1d577a3b9 in GNUNET_GNSRECORD_block_verify (block=0x62f00000a280)
    at gnsrecord_crypto.c:708
#17 0x00007fb1d57b9dde in block_plugin_gns_check_block (cls=0x0, 
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f00000a280, block_size=240)
    at plugin_block_gns.c:197
#18 0x00007fb1da0351da in GNUNET_BLOCK_check_block (ctx=0x60300000b1d0, 
    type=GNUNET_BLOCK_TYPE_GNS_NAMERECORD, block=0x62f00000a280, block_size=240)
    at block.c:329
#19 0x0000559ffd8991f1 in handle_dht_p2p_put (cls=0x60600007d4e0, 
    put=0x62f00000a1a8) at gnunet-service-dht_neighbours.c:2000
#20 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x7ffdd3e4a890, 
    mh=0x62f00000a1a8) at mq.c:242
#21 0x0000559ffd8a10eb in GDS_u_receive (cls=0x60b000004a80, tctx=0x60800001ffa0, 
    sctx=0x608000009020, message=0x62f00000a1a8, message_size=456)
    at gnunet-service-dht_neighbours.c:2836
#22 0x00007fb1d571baab in handle_core_message (cls=0x60800001ffa0, 
    msg=0x62f00000a1a4) at plugin_dhtu_gnunet.c:515
#23 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x60600007d540, 
    mh=0x62f00000a1a4) at mq.c:242
#24 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000033f40, 
    mh=0x62f00000a1a4) at mq.c:192
#25 0x00007fb1da198b0d in handle_notify_inbound (cls=0x60c000000a00, 
    ntm=0x62f00000a180) at core_api.c:574
#26 0x00007fb1d9ca68f2 in GNUNET_MQ_handle_message (handlers=0x610000001540, 
    mh=0x62f00000a180) at mq.c:242
#27 0x00007fb1d9ca6144 in GNUNET_MQ_inject_message (mq=0x60c000000b80, 
    mh=0x62f00000a180) at mq.c:192
#28 0x00007fb1d9be6f06 in recv_message (cls=0x60d0000006c0, msg=0x62f00000a180)
    at client.c:347
#29 0x00007fb1d9ca44d4 in GNUNET_MST_from_buffer (mst=0x60400000f990, buf=0x0, 
    size=0, purge=0, one_shot=0) at mst.c:222
#30 0x00007fb1d9ca5e84 in GNUNET_MST_read (mst=0x60400000f990, 
    sock=0x60300000d9f0, purge=0, one_shot=0) at mst.c:367
#31 0x00007fb1d9be8784 in receive_ready (cls=0x60d0000006c0) at client.c:447
#32 0x00007fb1d9cf7663 in GNUNET_SCHEDULER_do_work (sh=0x606000007be0)
    at scheduler.c:2107
#33 0x00007fb1d9cf9fa0 in select_loop (sh=0x606000007be0, context=0x7ffdd3e4af90)
    at scheduler.c:2412
#34 0x00007fb1d9ce8649 in GNUNET_SCHEDULER_run (
    task=0x7fb1d9d01f94 <service_main>, task_cls=0x7ffdd3e4b240) at scheduler.c:710
#35 0x00007fb1d9d105c1 in GNUNET_SERVICE_run_ (argc=11, argv=0x7ffdd3e4bab8, 
    service_name=0x559ffd8a6180 "dht", options=GNUNET_SERVICE_OPTION_NONE, 
    service_init_cb=0x559ffd885b78 <run>, 
    connect_cb=0x559ffd8739b0 <client_connect_cb>, 
    disconnect_cb=0x559ffd873af5 <client_disconnect_cb>, cls=0x0, 
    handlers=0x7ffdd3e4b820) at service.c:2136
#36 0x0000559ffd88694f in main (argc=11, argv=0x7ffdd3e4bab8)
    at gnunet-service-dht.c:540
(gdb) quit
gdb-backtraces (18,246 bytes)   

ulfvonbelow

2023-01-17 03:36

reporter   ~0019687

just went back and saw multiple minutes were requested, so here's another statistics sampling with 5 minutes between:
$ gnunet-statistics -s dht && sleep 5m && echo && gnunet-statistics -s dht
          dht # P2P PUT requests received: 84792
          dht # Local PUT requests not routed: 5
          dht # P2P GET requests received: 59
          dht # peers connected: 9
          dht # requests TTL-dropped: 24
          dht # ITEMS stored in datacache: 84807
          dht # P2P RESULT bytes received: 198873
          dht # P2P GET requests ONLY routed: 13
          dht # GET messages queued for transmission: 62
          dht # messages dropped (underlays busy): 68898
          dht # PUT requests routed: 84797
          dht # Peer selection failed: 23037
          dht # GET requests routed: 61
          dht # DHT requests combined: 7
          dht # Network size estimates received: 3
          dht # P2P GET bytes received: 15184
          dht # PUT requests received from clients: 5
          dht # GET requests given to datacache: 33
          dht # Entries added to routing table: 59
          dht # P2P HELLO lookup requests processed: 13
          dht # P2P PUT bytes received: 43659588
          dht # FIND PEER messages initiated: 2
          dht # PUT messages queued for transmission: 70407
          dht # P2P RESULTS received: 10
          dht # REPLIES ignored for CLIENTS (no match): 84807
          dht # Good REPLIES matched against routing table: 23
          dht # Duplicate REPLIES matched against routing table: 3
          dht # RESULT messages queued for transmission: 36

          dht # P2P PUT requests received: 111473
          dht # Local PUT requests not routed: 5
          dht # P2P GET requests received: 72
          dht # peers connected: 9
          dht # requests TTL-dropped: 24
          dht # ITEMS stored in datacache: 110191
          dht # P2P RESULT bytes received: 198873
          dht # P2P GET requests ONLY routed: 14
          dht # GET messages queued for transmission: 82
          dht # messages dropped (underlays busy): 97847
          dht # PUT requests routed: 110181
          dht # Peer selection failed: 24546
          dht # GET requests routed: 75
          dht # DHT requests combined: 9
          dht # Network size estimates received: 3
          dht # P2P GET bytes received: 19044
          dht # PUT requests received from clients: 5
          dht # Entries added to routing table: 72
          dht # GET requests given to datacache: 39
          dht # P2P HELLO lookup requests processed: 19
          dht # P2P PUT bytes received: 57353992
          dht # FIND PEER messages initiated: 3
          dht # PUT messages queued for transmission: 99476
          dht # P2P RESULTS received: 10
          dht # REPLIES ignored for CLIENTS (no match): 110191
          dht # Good REPLIES matched against routing table: 23
          dht # Duplicate REPLIES matched against routing table: 3
          dht # RESULT messages queued for transmission: 42

Christian Grothoff

2023-01-17 14:15

manager   ~0019691

Well, either way the answer is clear on this one:
1) We shouldn't have Ascension create this many DHT PUTs at such a high frequency, either by increasing DHT block lifetime or whatever else helps.
2) We should implement probabilistic block verification in the DHT, as the signature check at this rate kills less powerful peers.

schanzen

2023-01-23 07:26

administrator   ~0019698

I just now notices that the above patches are against the guix package repository. That is not under our control. Maybe it is managed by a member of the community. In any case, nothing to apply to gnunet git so no need to open another report.

thejackimonster

2024-10-03 00:57

developer   ~0023432

Could this issue be related to breaking assertions in either handling PUTs or GETs (potentially causing a restart of the service or repeated requests)? Because I have both easily triggered currently:
ERROR Assertion failed at gnunet-service-dht_neighbours.c:1292
ERROR Assertion failed at gnunet-service-dht_neighbours.c:1395

Both are caused by a bloomfilter collision, I assume. If this is unrelated I open another issue.

Christian Grothoff

2024-10-07 12:57

manager   ~0023464

Last edited: 2024-10-07 12:57

Assertions fixed in e800645b8..151674fc6

schanzen

2024-10-11 11:09

administrator   ~0023511

Closing this bug. The comments are already diverging. Please open new issues for specific bugs.

Issue History

Date Modified Username Field Change
2023-01-15 06:20 ulfvonbelow New Issue
2023-01-15 06:20 ulfvonbelow Tag Attached: bug
2023-01-15 06:20 ulfvonbelow Tag Attached: performance
2023-01-15 06:20 ulfvonbelow File Added: 0001-gnu-setuid-add-setuid-program-base-perms-field-to-se.patch
2023-01-15 06:21 ulfvonbelow Note Added: 0019666
2023-01-15 06:21 ulfvonbelow File Added: update-gnunet-and-add-service.patch
2023-01-15 06:22 ulfvonbelow Note Added: 0019667
2023-01-15 06:22 ulfvonbelow File Added: gnunet.conf
2023-01-16 09:41 Christian Grothoff Note Added: 0019678
2023-01-16 10:42 Christian Grothoff Note Added: 0019679
2023-01-16 10:42 Christian Grothoff Assigned To => schanzen
2023-01-16 10:42 Christian Grothoff Status new => assigned
2023-01-16 13:46 schanzen Note Added: 0019680
2023-01-16 13:48 schanzen Note Added: 0019681
2023-01-17 03:28 ulfvonbelow Note Added: 0019686
2023-01-17 03:28 ulfvonbelow File Added: gdb-backtraces
2023-01-17 03:36 ulfvonbelow Note Added: 0019687
2023-01-17 14:15 Christian Grothoff Note Added: 0019691
2023-01-20 07:48 schanzen Relationship added related to 0007602
2023-01-20 07:49 schanzen Relationship added related to 0007603
2023-01-20 07:50 schanzen Target Version => 0.20.0
2023-01-23 07:26 schanzen Note Added: 0019698
2023-08-06 19:02 schanzen Tag Attached: ngi-entrust-2022-08-104
2024-02-29 22:46 Christian Grothoff Tag Detached: bug
2024-10-03 00:57 thejackimonster Note Added: 0023432
2024-10-07 12:57 Christian Grothoff Note Added: 0023464
2024-10-07 12:57 Christian Grothoff Note Edited: 0023464
2024-10-10 10:23 schanzen Target Version 0.20.0 => 0.22.2
2024-10-10 10:23 schanzen Assigned To schanzen => Christian Grothoff
2024-10-11 11:09 schanzen Status assigned => resolved
2024-10-11 11:09 schanzen Resolution open => fixed
2024-10-11 11:09 schanzen Fixed in Version => 0.22.1
2024-10-11 11:09 schanzen Note Added: 0023511
2024-10-13 18:52 schanzen Fixed in Version 0.22.1 => 0.22.2
2024-10-13 18:53 schanzen Fixed in Version 0.22.2 => 0.22.1
2024-10-13 18:53 schanzen Status resolved => closed