View Issue Details

IDProjectCategoryView StatusLast Update
0003709GNUnetcore servicepublic2018-06-07 00:25
ReporterBart PolotAssigned ToChristian Grothoff 
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product VersionSVN HEAD 
Target Version0.11.0pre66Fixed in Version0.11.0pre66 
Summary0003709: Core leaks memory
DescriptionCore uses 124M - 400M of VIRT memory. Happens on arm as well as x86.
Steps To ReproduceRun a peer for a few hours.
TagsNo tags attached.

Activities

Christian Grothoff

2015-03-09 22:55

manager   ~0009013

I've not seen this, what are your bandwidth settings? Did you get anything with valgrind? My first guess might be that we queue quite a bit of messages.

Bart Polot

2015-03-09 22:59

manager   ~0009014

I have unlimited bandwidth on one peer (187M), default settings on other peers.

Here are the statistics on one peer that uses 323M:

   revocation # peers connected: 8
   revocation # revocation set unions failed: 2
   revocation # revocation set unions completed: 12
          gns Current zone iteration interval (ms): 4800000
          gns Number of public records in DHT: 3
          gns Number of zone iterations: 4
          gns Current zone iteration interval (in ms): 4800000
    datacache # requests received: 125535
    datacache # items stored: 5894
    datacache # bytes stored: 1048094
    datacache # requests filtered by bloom filter: 1673
        cadet # messages forwarded: 66179
        cadet # messages received: 292436
        cadet # connections: 0
        cadet # bad paths: 2798
        cadet # DHT search: 2412
        cadet # channels: 1
        cadet # peers: 9
        cadet # keepalives sent: 84
        cadet # keepalives received: 38
        cadet # control on unknown connection: 66
        cadet # unknown connection: 413
        cadet # tunnels: 0
        cadet # data retransmitted: 58
        cadet # DHT announce: 13
        cadet # data received: 78
        cadet # local bad paths: 2387
        cadet # duplicate PONG messages: 18
        cadet # clients: 3
        cadet # data on unknown channel: 6
        cadet # data ack on unknown channel: 6
        cadet # messages dropped due to wrong hop: 31
        cadet # ack on unknown connection: 2
        cadet # DHT announce skipped (no hello): 1
     topology # HELLO messages received: 1133
     topology # HELLO messages gossipped: 1182
     topology # peers connected: 9
     topology # connect requests issued to transport: 474
         core # bytes of messages of type 262 received: 26933616
         core # bytes encrypted: 5236777598
         core # avg payload per encrypted message: 6858
         core # encrypted bytes given to transport: 4807371066
         core # bytes of messages of type 147 received: 37977248
         core # bytes of payload decrypted: 5734552735
         core # bytes decrypted: 5747308143
         core # bytes of messages of type 146 received: 42148015
         core # bytes of messages of type 257 received: 119160
         core # bytes of messages of type 139 received: 1409104
         core # bytes of messages of type 148 received: 28357740
         core # bytes of messages of type 137 received: 237659808
         core # bytes of messages of type 138 received: 4847692581
         core # bytes of messages of type 268 received: 16400
         core # bytes of messages of type 280 received: 556772
         core # bytes of messages of type 256 received: 1486004
         core # bytes of messages of type 258 received: 234500
         core # bytes of messages of type 17 received: 95302
         core # PING messages transmitted: 440
         core # valid typemap confirmations received: 100
         core # bytes of messages of type 322 received: 28644
         core # type maps received: 292
         core # type map refreshes sent: 601
         core # peers connected: 9
         core # session keys confirmed via PONG: 116
         core # PONG messages decrypted: 242
         core # PONG messages received: 242
         core # PONG messages created: 187
         core # PING messages received: 207
         core # EPHEMERAL_KEY messages received: 233
         core # ephemeral keys received: 233
         core # key exchanges initiated: 137
         core # neighbour entries allocated: 11
         core # messages discarded (expired prior to transmission): 2304
         core # key exchanges stopped: 126
         core # sessions terminated by transport disconnect: 126
         core # bytes of messages of type 266 received: 2124
         core # DATA message dropped (out of order): 4106
         core # sessions terminated by timeout: 13
         core # keepalive messages sent: 157
         core # rekey operations confirmed via PONG: 28
         core # bytes of messages of type 269 received: 1240
         core # timeouts prevented via PONG: 98
         core # PING messages dropped (out of order): 15
         core # send requests dropped (disconnected): 1
         core # messages discarded (session disconnected): 2
         core # old ephemeral keys ignored: 3
         core # updates to my type map: 6
           fs # P2P searches active: 162
           fs # query plan entries: 1931
           fs # Datastore lookups concluded (found last result): 240568
           fs # P2P searches destroyed due to ultimate reply: 246776
           fs # replies received for other peers: 260304
           fs # replies received and matched: 520220
           fs # P2P query messages received and processed: 626170
           fs # Pending requests active: 65536
           fs # GET requests received (from other peers): 1183104
           fs # migration stop messages received: 63125
           fs # running average P2P latency (ms): 4
           fs # migration stop messages sent: 88069
           fs # Datastore `PUT' failures: 189937
           fs # GAP PUT messages received: 199846
           fs # delay heap timeout (ms): 2915
           fs # query messages sent to other peers: 2421782
           fs # average retransmission delay (ms): 1238763
           fs # requests dropped due to full reply queue: 550573
           fs # replies transmitted to other peers: 179523
           fs # Datastore lookups concluded (seen all): 7847
           fs # Datastore lookups concluded (load too high): 1152
           fs # results found locally: 7670
           fs # replies dropped: 80152
           fs # requests dropped due to missing reverse route: 6361
           fs # Loopback routes suppressed: 977310
           fs # peers connected: 9
          nse # flood messages received: 176
          nse # flood messages transmitted: 217
          nse # peers connected: 10
          nse # nodes in the network (estimate): 21
          nse # estimated network diameter: 5
          nse # flood messages ignored (had closer already): 3
          nse # flood messages started: 3
          nse # flood messages not generated (no proof yet): 9
    transport # bytes transmitted via TCP: 4629743751
    transport # bytes currently in TCP buffers: 5006
    transport # messages transmitted to other peers: 732642
    transport # bytes in message queue for other peers: 334833
    transport # UDP, total bytes in send buffers: 0
    transport # UDP, fragmented msgs, fragments bytes, sent, success: 203453746
    transport # UDP, fragmented msgs, fragments, sent, success: 149646
    transport # UDP, total, messages, sent, success: 869770
    transport # UDP, total, bytes, sent, success: 272827437
    transport # UDP, total messages in send buffers: 0
    transport # fragments transmitted: 149663
    transport # QUOTA messages received: 359577
    transport # bytes total received: 5870090466
    transport # bytes received via TCP: 5838676033
    transport # bytes payload received: 5861061313
    transport # REQUEST CONNECT messages received: 47966
    transport # UDP, unfragmented bytes payload queued total: 40545063
    transport # UDP, unfragmented messages queued total: 717601
    transport # ATS suggestions received: 1456975
    transport # UDP, total bytes received: 48715026
    transport # UDP, total, bytes payload, sent: 195603152
    transport # UDP, total, bytes overhead, sent: 62391344
    transport # UDP, unfragmented msgs, bytes overhead, sent, success: 28701320
    transport # UDP, unfragmented msgs, bytes payload, sent, success: 40527275
    transport # UDP, unfragmented msgs, messages, sent, success: 717533
    transport # KEEPALIVES received in good order: 10221
    transport # UDP, fragmented bytes (payload): 156456036
    transport # UDP, fragmented messages, total: 7974
    transport # UDP, fragmented messages active: 4
    transport # total size of fragmented messages: 156774996
    transport # messages fragmented: 7974
    transport # fragments retransmitted: 32803
    transport # UDP sessions active: 26
    transport # KEEPALIVE_RESPONSEs received (OK): 13510
    transport # UDP, fragmented msgs, bytes overhead, sent, success: 33690024
    transport # UDP, fragmented msgs, bytes payload, sent, success: 155075877
    transport # UDP, fragmented msgs, messages, sent, success: 7935
    transport # fragmentation transmissions completed: 7935
    transport # fragment acknowledgements received: 8712
    transport # fragments wrap arounds: 13224
    transport # validations delayed by global throttle: 831326
    transport # validations running: 6439
    transport # PINGs for address validation sent: 20929
    transport # TCP sessions active: 43
    transport # TCP server connect events: 9245
    transport # TCP server connections active: 43
    transport # address revalidations started: 20977
    transport # Multicast HELLO beacons received via UDP: 3430
    transport # KEEPALIVES sent: 13691
    transport # blocked addresses: 31
    transport # Addresses given to ATS: 73
    transport # validations succeeded: 8480
    transport # PONG messages received: 8482
    transport # bytes discarded by TCP (disconnect): 2396256
    transport # network-level TCP disconnect events: 7808
    transport # Addresses in validation map: 190
    transport # address records discarded (timeout): 7201
    transport # PONGs unicast via reliable transport: 3449
    transport # successful address checks during validation: 3449
    transport # PING messages received: 3914
    transport # SYN messages sent: 13037
    transport # UDP, ACK messages sent: 2591
    transport # acknowledgements sent for fragment: 2631
    transport # duplicate fragments received: 6383
    transport # fragments received: 20466
    transport # messages defragmented: 785
    transport # failed address checks during validation: 465
    transport # ACK messages received: 1050
    transport # SYN_ACK messages sent: 1064
    transport # SYN messages received: 1342
    transport # TCP WELCOME messages received: 605
    transport # UDP, fragments discarded without ACK: 40
    transport # bytes discarded by TCP (timeout): 6368
    transport # peers connected: 12
    transport # SYN_ACK messages received: 1361
    transport # messages discarded due to timeout: 63
    transport # UDP, unfragmented msgs, bytes overhead, sent, failure: 2720
    transport # UDP, unfragmented msgs, bytes payload, sent, failure: 17788
    transport # UDP, unfragmented msgs, messages, sent, failure: 68
    transport # DISCONNECT messages sent: 51
    transport # Successful attempts to switch addresses: 209
    transport # Attempts to switch addresses: 266
    transport # KEEPALIVE_RESPONSEs discarded (not expected): 49
    transport # refreshed my HELLO: 70
    transport # transport addresses: 10
    transport # PONGs dropped, no matching pending validation: 2
    transport # UDP, fragmented msgs, bytes payload, sent, failure: 27700502
    transport # UDP, fragmented msgs, messages, sent, failure: 35
    transport # transmission failures for messages to other peers: 39
    transport # PING message for different peer received: 1217
    transport # requests to create session with invalid address: 48
    transport # bytes payload discarded due to not connected peer: 11042863
    transport # KEEPALIVE_RESPONSEs discarded (bad nonce): 4
    transport # disconnected from peer upon explicit request: 74
    transport # Failed attempts to switch addresses (failed to send SYN CONT): 1
    transport # messages timed out while in transport queue: 2
    transport # unexpected SYN_ACK messages (not ready): 2
    transport # bytes payload dropped (other peer was not connected): 33164
    transport # DISCONNECT messages received: 13
    transport # KEEPALIVE messages discarded (peer unknown): 29
    transport # bits removed from fragmentation ACKs: 1
    transport # unexpected SYN_ACK messages (no peer): 1
    transport # KEEPALIVE messages discarded (no session): 4
    transport # KEEPALIVE_RESPONSEs discarded (no nonce): 2
    transport # unexpected ACK messages: 2
          dht # Bytes transmitted to other peers: 108483003
          dht # DHT requests combined: 132228
          dht # Peer selection failed: 24183
          dht # Entries added to routing table: 133387
          dht # P2P GET bytes received: 36604092
          dht # P2P GET requests received: 133387
          dht # Bytes of bandwidth requested from core: 48281786
          dht # GET messages queued for transmission: 153492
          dht # Peers excluded from routing due to Bloomfilter: 708718
          dht # GET requests routed: 136192
          dht # Duplicate RESULTS found in datacache: 505282
          dht # GET requests given to datacache: 125535
          dht # PUT messages queued for transmission: 103918
          dht # PUT requests routed: 83620
          dht # P2P PUT bytes received: 74131271
          dht # P2P PUT requests received: 83594
          dht # Duplicate REPLIES matched against routing table: 911544
          dht # ITEMS stored in datacache: 142474
          dht # REPLIES ignored for CLIENTS (no match): 183938
          dht # P2P RESULT bytes received: 14715990
          dht # P2P RESULTS received: 58854
          dht # RESULT messages queued for transmission: 92615
          dht # P2P messages dropped due to full queue: 77410
          dht # Good REPLIES matched against routing table: 98491
          dht # P2P GET requests ONLY routed: 7566
          dht # Preference updates given to core: 964
          dht # Good RESULTS found in datacache: 49413
          dht # HELLOs obtained from peerinfo: 8320
          dht # FIND PEER requests ignored due to Bloomfilter: 399
          dht # P2P FIND PEER requests processed: 2698
          dht # FIND PEER messages initiated: 170
          dht # Duplicate REPLIES to CLIENT request dropped: 17118
          dht # GET requests from clients injected: 2635
          dht # RESULTS queued for clients: 7949
          dht # GET requests received from clients: 2412
          dht # peers connected: 9
          dht # Messages dropped (CORE timeout): 5559
          dht # Queued messages discarded (peer disconnected): 325
          dht # PUT requests received from clients: 26
          dht # Network size estimates received: 17
          dht # GET STOP requests received from clients: 2406
          dht # requests TTL-dropped: 9
 datastore-api # status messages received: 199817
 datastore-api # PUT requests executed: 199817
 datastore-api # GET ZERO ANONYMITY requests executed: 149
 datastore-api # queue overflows: 29
 datastore-api # GET REPLICATION requests executed: 9
          ats # address suggestions made: 1458769
          ats # performance updates given to clients: 2022089
          ats # preference change requests processed: 144792
          ats # reservation requests processed: 2447565
          ats # address updates received: 506541
          ats # ATS addresses total: 134
          ats # ATS active addresses WAN total: 8
          ats # ATS addresses WAN total: 23
          ats # addresses: 72
          ats # addresses destroyed: 3375
          ats # ATS active addresses total: 3325
          ats # addresses created: 3447
          ats # ATS addresses LAN total: 46
          ats # ATS active addresses UNSPECIFIED total: 1
          ats # ATS addresses UNSPECIFIED total: 3
          ats # ATS active addresses LAN total: 3
     peerinfo # peers known: 24
     hostlist # active connections: 9
! hostlist # milliseconds between hostlist downloads: 39600000
     hostlist # hostlist downloads initiated: 2
    datastore # results found: 503098
    datastore # GET requests received: 736493
    datastore # requests filtered by bloomfilter: 233404
! datastore # bytes stored: 415740329
! datastore # bytes expired: 63013472
    datastore # GET ZERO ANONYMITY requests received: 149
    datastore # utilization by current datastore: 355288891
! datastore # bytes used in file-sharing datastore `sqlite': 355288891
    datastore # GET REPLICATION requests received: 9
    datastore # cache size: 625000000
    datastore # quota: 5000000000

Bart Polot

2015-03-10 01:58

manager   ~0009015

Seems to be bursty. I killed core on one computer (SIGTERM, arm restarted) and after more than 1 hour it has gone from 38k to 51k.

Might be of interest, fs is taking 152M on one machine as well.

Bart Polot

2015-03-10 02:05

manager   ~0009016

More info: after a whole peer restart, fs takes up 106M immediately.

Bart Polot

2015-03-10 02:21

manager   ~0009017

10 minutes later, fs is at 110M, core at 97M VIRT, 80k RES. VIRT reverts to 98k after a minute (big pre-allocation? mmap?).

On other machines both values are high, saturn uses 187M VIRT and 158M RES right now.

Christian Grothoff

2015-03-10 07:55

manager   ~0009018

Last edited: 2015-03-10 07:55

View 2 revisions

I know about FS, know where it is and why (it's queuing replies, not leaking). The growth of the main buffer there is bounded to like 32k * 128 * #connections, but that can still be quite a bit. So core is worrisome, FS is something I can easily 'tune'.

Christian Grothoff

2015-03-21 00:11

manager   ~0009029

heap_tree=peak
n3: 7054165 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 6366986 0x525B232: GNUNET_xmalloc_unchecked_ (common_allocation.c:144)
  n4: 6366986 0x525ADC0: GNUNET_xmalloc_ (common_allocation.c:71)
   n2: 6065759 0x4068BC: GSC_NEIGHBOURS_transmit (gnunet-service-core_neighbours.c:457)
    n2: 6065331 0x40A0E4: GSC_KX_encrypt_and_transmit (gnunet-service-core_kx.c:1410)
     n2: 6064615 0x40D25B: try_transmission (gnunet-service-core_sessions.c:771)
      n1: 6040929 0x40D9BB: GSC_SESSIONS_transmit (gnunet-service-core_sessions.c:892)
       n1: 6040929 0x40423D: client_tokenizer_callback (gnunet-service-core_clients.c:571)
        n1: 6040929 0x529D43B: GNUNET_SERVER_mst_receive (server_mst.c:262)
         n1: 6040929 0x403EC9: handle_client_send (gnunet-service-core_clients.c:505)
          n1: 6040929 0x5299FEF: GNUNET_SERVER_inject (server.c:997)
           n2: 6040929 0x529ACC0: client_message_tokenizer_callback (server.c:1256)
            n1: 6040929 0x529D43B: GNUNET_SERVER_mst_receive (server_mst.c:262)
             n1: 6040929 0x529A8D6: process_incoming (server.c:1178)
              n1: 6040929 0x5265783: receive_ready (connection.c:1139)
               n1: 6040929 0x52955C1: run_ready (scheduler.c:587)
                n1: 6040929 0x5295ECC: GNUNET_SCHEDULER_run (scheduler.c:867)
                 n1: 6040929 0x52A46E0: GNUNET_SERVICE_run (service.c:1503)
                  n0: 6040929 0x402C04: main (gnunet-service-core.c:144)

Christian Grothoff

2015-03-21 00:22

manager   ~0009030

Fixed in SVN 35414.

Issue History

Date Modified Username Field Change
2015-03-09 21:56 Bart Polot New Issue
2015-03-09 21:56 Bart Polot Status new => assigned
2015-03-09 21:56 Bart Polot Assigned To => Christian Grothoff
2015-03-09 22:55 Christian Grothoff Note Added: 0009013
2015-03-09 22:59 Bart Polot Note Added: 0009014
2015-03-10 01:58 Bart Polot Note Added: 0009015
2015-03-10 02:05 Bart Polot Note Added: 0009016
2015-03-10 02:21 Bart Polot Note Added: 0009017
2015-03-10 07:55 Christian Grothoff Note Added: 0009018
2015-03-10 07:55 Christian Grothoff Note Edited: 0009018 View Revisions
2015-03-20 21:34 Christian Grothoff Target Version 0.11.0pre66 =>
2015-03-20 21:36 Christian Grothoff Target Version => 0.11.0pre66
2015-03-21 00:11 Christian Grothoff Note Added: 0009029
2015-03-21 00:22 Christian Grothoff Note Added: 0009030
2015-03-21 00:22 Christian Grothoff Status assigned => resolved
2015-03-21 00:22 Christian Grothoff Fixed in Version => 0.11.0pre66
2015-03-21 00:22 Christian Grothoff Resolution open => fixed
2018-06-07 00:25 Christian Grothoff Status resolved => closed