View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0003390 | GNUnet | transport service | public | 2014-04-25 15:58 | 2018-06-07 00:25 |
| Reporter | Bart Polot | Assigned To | Matthias Wachs | ||
| Priority | urgent | Severity | major | Reproducibility | have not tried |
| Status | closed | Resolution | fixed | ||
| Product Version | Git master | ||||
| Target Version | 0.11.0pre66 | ||||
| Summary | 0003390: Memory leak in transport | ||||
| Description | Usage 25.9%out of 24GB VIRT 6288M RES 6248M CPU: 1.3% or 0% [bart@saturn ~/g/src/mesh] (master *% u=)$ gnunet-statistics transport # REQUEST CONNECT messages received: 6031313 topology # connect requests issued to transport: 5782610 ats # preference change requests processed: 49044 dht # Preference updates given to core: 49044 transport # UDP, total, bytes payload, sent: 432241262 transport # UDP, total, bytes overhead, sent: 172350257 transport # UDP, unfragmented msgs, bytes overhead, sent, success: 152285240 transport # UDP, unfragmented msgs, bytes payload, sent, success: 399815246 transport # UDP, unfragmented msgs, messages, sent, success: 3807131 transport # UDP, total, messages, sent, success: 3856889 transport # UDP, total, bytes, sent, success: 604591519 transport # UDP, unfragmented msgs, bytes payload, attempt: 399819452 transport # UDP, unfragmented msgs, messages, attempt: 3807178 transport # UDP, total, msgs in buffers: 0 transport # UDP, total, bytes in buffers: 0 transport # bytes total received: 550787085 transport # UDP, total, bytes, received: 542907351 transport # messages transmitted to other peers: 1234359 transport # bytes in message queue for other peers: 38214 core # encrypted bytes given to transport: 376516574 core # bytes of messages of type 148 received: 170851898 core # bytes encrypted: 297934290 core # avg payload per encrypted message: 277 dht # Bytes transmitted to other peers: 270567046 dht # Duplicate REPLIES matched against routing table: 12790262 dht # Bytes of bandwidth requested from core: 261944471 dht # RESULT messages queued for transmission: 797685 dht # Good REPLIES matched against routing table: 773020 dht # ITEMS stored in datacache: 711526 dht # REPLIES ignored for CLIENTS (no match): 711555 dht # P2P RESULT bytes received: 148258249 dht # P2P RESULTS received: 702742 datacache # bytes stored: 25431606 datacache # items stored: 105605 core # bytes of payload decrypted: 324109920 core # bytes decrypted: 351183904 transport # bytes payload received: 460188625 dht # Peer selection failed: 48125 dht # Peers excluded from routing due to Bloomfilter: 519093 dht # GET requests routed: 175063 dht # P2P GET requests ONLY routed: 131289 dht # DHT requests combined: 168500 dht # Entries added to routing table: 170592 dht # P2P GET bytes received: 59599448 dht # P2P GET requests received: 170592 transport # keepalives sent: 1342332 transport # bytes received via TCP: 173958401 core # bytes of messages of type 322 received: 1031448 nse # flood messages transmitted: 7814 nse # flood messages started: 5875 transport # UDP, sessions active: 8 dht # HELLOs obtained from peerinfo: 170719 transport # PONG messages received: 167929 transport # PING without HELLO messages sent: 234578 transport # address revalidations started: 410631 transport # IPv4 broadcast HELLO beacons received via udp: 49077 core # bytes of messages of type 147 received: 57659548 dht # GET messages queued for transmission: 165385 core # type map refreshes sent: 102474 transport # address records discarded: 223449 dht # P2P FIND PEER requests processed: 33233 core # type maps received: 102036 transport # PONGs unicast via reliable transport: 200096 transport # successful address checks during validation: 200096 transport # PING messages received: 200096 nse # flood messages discarded (clock skew too large): 497126 nse # flood messages received: 505992 dht # FIND PEER requests ignored due to Bloomfilter: 13490 transport # bytes transmitted via TCP: 84155046 transport # bytes currently in TCP buffers: 0 dht # PUT requests routed: 8784 dht # P2P PUT bytes received: 45124609 dht # P2P PUT requests received: 8375 dht # FIND PEER messages initiated: 4471 dht # Network size estimates received: 445 nse # nodes in the network (estimate): 3 core # PONG messages created: 89284 core # PING messages received: 89466 transport # refreshed my HELLO: 2807 core # bytes of messages of type 146 received: 42055600 dht # PUT messages queued for transmission: 8094 transport # CONNECT messages received: 55984 core # rekey operations confirmed via PONG: 1368 core # PONG messages decrypted: 2046 core # PONG messages received: 2053 core # keepalive messages sent: 1431 core # DATA message dropped (out of order): 3425 dht # PUT requests received from clients: 409 transport # DISCONNECT messages received: 335 dht # Duplicate RESULTS found in datacache: 20187 dht # GET requests given to datacache: 6070 datacache # requests received: 6070 peerinfo # peers known: 161 transport # requests to create session with invalid address: 445 core # EPHEMERAL_KEY messages received: 792 core # ephemeral keys received: 812 nse # estimated network diameter: 3 datacache # requests filtered by bloom filter: 2650 transport # UDP, ACK msgs, bytes overhead, sent, success: 674520 transport # UDP, ACK msgs, messages, sent, success: 12045 transport # acknowledgements sent for fragment: 12045 transport # duplicate fragments received: 11401 transport # fragments received: 33360 transport # messages defragmented: 906 dht # peers connected: 5 nse # peers connected: 6 core # neighbour entries allocated: 5 core # key exchanges stopped: 88 core # peers connected: 5 core # sessions terminated by transport disconnect: 88 transport # peers connected: 5 transport # UDP, fragmented msgs, messages, pending: 0 transport # UDP, fragmented msgs, bytes overhead, sent, success: 19390497 transport # UDP, fragmented msgs, bytes payload, sent, success: 32426016 transport # UDP, fragmented msgs, messages, sent, success: 990 transport # fragmentation transmissions completed: 990 transport # fragment acknowledgements received: 1099 transport # UDP, fragmented msgs, fragments bytes, sent, success: 51816513 transport # UDP, fragmented msgs, fragments, sent, success: 37713 transport # fragments wrap arounds: 1441 transport # fragments retransmitted: 13829 transport # fragments transmitted: 37736 transport # UDP, fragmented msgs, bytes payload, attempt: 32426016 transport # UDP, fragmented msgs, messages, attempt: 990 transport # total size of fragmented messages: 32465616 transport # messages fragmented: 990 transport # TCP sessions active: 1 transport # transport-service disconnect requests for TCP: 5 transport # DISCONNECT messages sent: 46 transport # bytes discarded by TCP (disconnect): 332196 transport # TCP WELCOME messages received: 640 transport # network-level TCP disconnect events: 2350 transport # KEEPALIVE messages discarded (peer unknown): 19 transport # bytes payload discarded due to not connected peer: 12076 transport # KEEPALIVE_RESPONSE messages discarded (wrong nonce): 35 transport # KEEPALIVE messages discarded (no session): 7 transport # KEEPALIVE_RESPONSE messages discarded (not expected): 26 transport # KEEPALIVE_RESPONSE messages discarded (not connected): 1 core # session keys confirmed via PONG: 111 core # sessions terminated by timeout: 22 nse # flood messages ignored (had closer already): 292 core # old ephemeral keys ignored: 284 dht # Good RESULTS found in datacache: 29 core # timeouts prevented via PONG: 567 core # sessions terminated by key expiration: 2 transport # bytes discarded by TCP (timeout): 9232 transport # messages dismissed due to timeout: 44 transport # UDP, unfragmented msgs, bytes overhead, sent, failure: 1880 transport # UDP, unfragmented msgs, bytes payload, sent, failure: 4206 transport # UDP, unfragmented msgs, messages, sent, failure: 47 transport # UDP, unfragmented msgs, bytes, sent, timeout: 4182 transport # UDP, unfragmented msgs, messages, sent, timeout: 44 transport # UDP, total, messages, sent, timeout: 44 transport # UDP, total, bytes, sent, timeout: 5942 transport # UDP, fragmented msgs, fragments bytes, sent, failure: 30879 transport # UDP, fragmented msgs, fragments, sent, failure: 23 core # PING messages dropped (out of order): 22 core # PONG messages dropped (out of order): 7 transport # PONGs dropped, signature expired: 72 transport # PONGs dropped, no matching pending validation: 1 transport # CONNECT_ACK messages sent: 37 mesh # clients: 0 mesh # peers: 0 topology # peers connected: 0 core # bytes of messages of type 17 received: 8438 topology # HELLO messages gossipped: 109 topology # HELLO messages received: 107 transport # SESSION_ACK messages received: 19 transport # CONNECT_ACK messages received: 103 ats # address suggestions made: 1402877 ats # performance updates given to clients: 1520783 ats # ATS active addresses WAN total: 3 ats # ATS addresses total: 0 ats # ATS addresses WAN total: 0 transport # Successful attempts to switch addresses: 12 ats # address updates received: 644210 transport # SESSION_CONNECT messages sent: 7495 transport # Attempts to switch addresses: 12 ats # ATS active addresses total: 571 ats # addresses destroyed: 1132 ats # addresses: 3 core # key exchanges initiated: 93 ats # ATS addresses LAN total: 0 ats # ATS active addresses LAN total: 4 ats # ATS active addresses UNSPECIFIED total: 1 ats # ATS addresses UNSPECIFIED total: 0 core # bytes of messages of type 258 received: 100 mesh # connections: 0 mesh # bad paths: 1 transport # unexpected CONNECT_ACK messages (not ready): 1 dht # requests TTL-dropped: 55 ats # active performance clients: 1 core # updates to my type map: 4 ! datastore # bytes stored: 525200796 ! datastore # bytes used in file-sharing datastore `sqlite': 529659804 | ||||
| Tags | No tags attached. | ||||
| Attached Files | |||||
|
|
[bart@saturn ~]$ cat .local/share/gnunet/gnunet.conf [transport-tcp] PORT = 10003 [transport-udp] PORT = 10003 [bart@saturn ~]$ cat .config/gnunet.conf [arm] DEFAULTSERVICES = topology hostlist dht nse mesh [transport-tcp] PORT = 10003 [transport-udp] PORT = 10003 [fs] AUTOSTART = NO CONTENT_CACHING = YES CONTENT_PUSHING = YES [ats] UNSPECIFIED_QUOTA_IN = unlimited UNSPECIFIED_QUOTA_OUT = unlimited LOOPBACK_QUOTA_IN = unlimited LOOPBACK_QUOTA_OUT = unlimited LAN_QUOTA_IN = unlimited LAN_QUOTA_OUT = unlimited WAN_QUOTA_IN = unlimited WAN_QUOTA_OUT = unlimited WLAN_QUOTA_IN = unlimited WLAN_QUOTA_OUT = unlimited BLUETOOTH_QUOTA_IN = unlimited BLUETOOTH_QUOTA_OUT = unlimited |
|
|
[bart@saturn ~]$ gnunet-arm -I Running services: mesh (gnunet-service-mesh) topology (gnunet-daemon-topology) dht (gnunet-service-dht) nse (gnunet-service-nse) core (gnunet-service-core) transport (gnunet-service-transport) ats (gnunet-service-ats) peerinfo (gnunet-service-peerinfo) statistics (gnunet-service-statistics) resolver (gnunet-service-resolver) |
|
|
I assume that it is one of the plugins ... most probably UDP, but this is an educated guess ... Attached an analysis about memory consumption for a peer... After running a first test for 20 hours I saw that comparing data without paying respect to current connections does not help... |
|
|
Using TCP results in a stable consumption over 4 1/2 days ... see attached plot memory_tcp.pdf |
|
|
No obvious issues with HTTP and HTTPS ... see attached plot memory_http.pdf |
|
|
No obvious issues with UDO ... see attached plot memory_udp.pdf |
|
|
Still an issue: 2095 gnunet 1 20 0 1074M 1030M select 2 21:15 0.00% gnunet-service-tran 2099 gnunet 1 20 0 46520K 3956K select 3 13:38 0.00% gnunet-service-mesh 2096 gnunet 1 20 0 38368K 5216K select 3 4:06 0.00% gnunet-service-core 2097 gnunet 1 20 0 34084K 3444K select 1 3:26 0.00% gnunet-service-stat 2087 gnunet 1 20 0 126M 48640K select 6 2:53 0.00% gnunet-service-dht 2086 gnunet 1 20 0 44444K 3608K select 7 2:30 0.00% gnunet-daemon-topol |
|
|
Fixed one memory leak in transport_clients today causing ~ 100 bytes to be leaked ... maybe it was a hit? Was caused by address to string used by gnunet-ats and other CLIs ... and since this was not covered by my previous tests there is chance that it is resolved by > 33870. |
|
|
So far, after 4 days it seems to be using 44k of memory, seems to be solved. I will reopen the bug if I see it again. |
|
|
ACK: 96 Kb on 4YCN running for several days |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2014-04-25 15:58 | Bart Polot | New Issue | |
| 2014-04-25 15:58 | Bart Polot | Status | new => assigned |
| 2014-04-25 15:58 | Bart Polot | Assigned To | => Matthias Wachs |
| 2014-04-25 15:59 | Bart Polot | Note Added: 0008272 | |
| 2014-04-25 16:00 | Bart Polot | Note Added: 0008273 | |
| 2014-04-25 23:36 | Christian Grothoff | Priority | normal => urgent |
| 2014-04-30 10:23 | Matthias Wachs | Note Added: 0008274 | |
| 2014-04-30 10:24 | Matthias Wachs | File Added: memory.pdf | |
| 2014-05-05 10:58 | Matthias Wachs | Note Added: 0008278 | |
| 2014-05-05 10:58 | Matthias Wachs | File Added: memory_tcp.pdf | |
| 2014-05-06 09:39 | Matthias Wachs | Note Added: 0008281 | |
| 2014-05-06 09:39 | Matthias Wachs | File Added: memory_http.pdf | |
| 2014-05-06 09:44 | Matthias Wachs | Note Added: 0008282 | |
| 2014-05-06 09:44 | Matthias Wachs | File Added: memory_udp.pdf | |
| 2014-05-09 09:47 | Matthias Wachs | Note Added: 0008309 | |
| 2014-06-26 16:46 | Matthias Wachs | Note Added: 0008472 | |
| 2014-06-26 16:46 | Matthias Wachs | Status | assigned => feedback |
| 2014-06-26 16:48 | Matthias Wachs | Note Edited: 0008472 | |
| 2014-06-30 16:26 | Bart Polot | Note Added: 0008487 | |
| 2014-06-30 16:26 | Bart Polot | Status | feedback => assigned |
| 2014-07-01 09:05 | Matthias Wachs | Note Added: 0008490 | |
| 2014-07-01 09:06 | Matthias Wachs | Status | assigned => resolved |
| 2014-07-01 09:06 | Matthias Wachs | Resolution | open => fixed |
| 2018-06-07 00:25 | Christian Grothoff | Status | resolved => closed |
