View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0003709 | GNUnet | core service | public | 2015-03-09 21:56 | 2018-06-07 00:25 |
| Reporter | Bart Polot | Assigned To | Christian Grothoff | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | closed | Resolution | fixed | ||
| Product Version | Git master | ||||
| Target Version | 0.11.0pre66 | Fixed in Version | 0.11.0pre66 | ||
| Summary | 0003709: Core leaks memory | ||||
| Description | Core uses 124M - 400M of VIRT memory. Happens on arm as well as x86. | ||||
| Steps To Reproduce | Run a peer for a few hours. | ||||
| Tags | No tags attached. | ||||
|
|
I've not seen this, what are your bandwidth settings? Did you get anything with valgrind? My first guess might be that we queue quite a bit of messages. |
|
|
I have unlimited bandwidth on one peer (187M), default settings on other peers. Here are the statistics on one peer that uses 323M: revocation # peers connected: 8 revocation # revocation set unions failed: 2 revocation # revocation set unions completed: 12 gns Current zone iteration interval (ms): 4800000 gns Number of public records in DHT: 3 gns Number of zone iterations: 4 gns Current zone iteration interval (in ms): 4800000 datacache # requests received: 125535 datacache # items stored: 5894 datacache # bytes stored: 1048094 datacache # requests filtered by bloom filter: 1673 cadet # messages forwarded: 66179 cadet # messages received: 292436 cadet # connections: 0 cadet # bad paths: 2798 cadet # DHT search: 2412 cadet # channels: 1 cadet # peers: 9 cadet # keepalives sent: 84 cadet # keepalives received: 38 cadet # control on unknown connection: 66 cadet # unknown connection: 413 cadet # tunnels: 0 cadet # data retransmitted: 58 cadet # DHT announce: 13 cadet # data received: 78 cadet # local bad paths: 2387 cadet # duplicate PONG messages: 18 cadet # clients: 3 cadet # data on unknown channel: 6 cadet # data ack on unknown channel: 6 cadet # messages dropped due to wrong hop: 31 cadet # ack on unknown connection: 2 cadet # DHT announce skipped (no hello): 1 topology # HELLO messages received: 1133 topology # HELLO messages gossipped: 1182 topology # peers connected: 9 topology # connect requests issued to transport: 474 core # bytes of messages of type 262 received: 26933616 core # bytes encrypted: 5236777598 core # avg payload per encrypted message: 6858 core # encrypted bytes given to transport: 4807371066 core # bytes of messages of type 147 received: 37977248 core # bytes of payload decrypted: 5734552735 core # bytes decrypted: 5747308143 core # bytes of messages of type 146 received: 42148015 core # bytes of messages of type 257 received: 119160 core # bytes of messages of type 139 received: 1409104 core # bytes of messages of type 148 received: 28357740 core # bytes of messages of type 137 received: 237659808 core # bytes of messages of type 138 received: 4847692581 core # bytes of messages of type 268 received: 16400 core # bytes of messages of type 280 received: 556772 core # bytes of messages of type 256 received: 1486004 core # bytes of messages of type 258 received: 234500 core # bytes of messages of type 17 received: 95302 core # PING messages transmitted: 440 core # valid typemap confirmations received: 100 core # bytes of messages of type 322 received: 28644 core # type maps received: 292 core # type map refreshes sent: 601 core # peers connected: 9 core # session keys confirmed via PONG: 116 core # PONG messages decrypted: 242 core # PONG messages received: 242 core # PONG messages created: 187 core # PING messages received: 207 core # EPHEMERAL_KEY messages received: 233 core # ephemeral keys received: 233 core # key exchanges initiated: 137 core # neighbour entries allocated: 11 core # messages discarded (expired prior to transmission): 2304 core # key exchanges stopped: 126 core # sessions terminated by transport disconnect: 126 core # bytes of messages of type 266 received: 2124 core # DATA message dropped (out of order): 4106 core # sessions terminated by timeout: 13 core # keepalive messages sent: 157 core # rekey operations confirmed via PONG: 28 core # bytes of messages of type 269 received: 1240 core # timeouts prevented via PONG: 98 core # PING messages dropped (out of order): 15 core # send requests dropped (disconnected): 1 core # messages discarded (session disconnected): 2 core # old ephemeral keys ignored: 3 core # updates to my type map: 6 fs # P2P searches active: 162 fs # query plan entries: 1931 fs # Datastore lookups concluded (found last result): 240568 fs # P2P searches destroyed due to ultimate reply: 246776 fs # replies received for other peers: 260304 fs # replies received and matched: 520220 fs # P2P query messages received and processed: 626170 fs # Pending requests active: 65536 fs # GET requests received (from other peers): 1183104 fs # migration stop messages received: 63125 fs # running average P2P latency (ms): 4 fs # migration stop messages sent: 88069 fs # Datastore `PUT' failures: 189937 fs # GAP PUT messages received: 199846 fs # delay heap timeout (ms): 2915 fs # query messages sent to other peers: 2421782 fs # average retransmission delay (ms): 1238763 fs # requests dropped due to full reply queue: 550573 fs # replies transmitted to other peers: 179523 fs # Datastore lookups concluded (seen all): 7847 fs # Datastore lookups concluded (load too high): 1152 fs # results found locally: 7670 fs # replies dropped: 80152 fs # requests dropped due to missing reverse route: 6361 fs # Loopback routes suppressed: 977310 fs # peers connected: 9 nse # flood messages received: 176 nse # flood messages transmitted: 217 nse # peers connected: 10 nse # nodes in the network (estimate): 21 nse # estimated network diameter: 5 nse # flood messages ignored (had closer already): 3 nse # flood messages started: 3 nse # flood messages not generated (no proof yet): 9 transport # bytes transmitted via TCP: 4629743751 transport # bytes currently in TCP buffers: 5006 transport # messages transmitted to other peers: 732642 transport # bytes in message queue for other peers: 334833 transport # UDP, total bytes in send buffers: 0 transport # UDP, fragmented msgs, fragments bytes, sent, success: 203453746 transport # UDP, fragmented msgs, fragments, sent, success: 149646 transport # UDP, total, messages, sent, success: 869770 transport # UDP, total, bytes, sent, success: 272827437 transport # UDP, total messages in send buffers: 0 transport # fragments transmitted: 149663 transport # QUOTA messages received: 359577 transport # bytes total received: 5870090466 transport # bytes received via TCP: 5838676033 transport # bytes payload received: 5861061313 transport # REQUEST CONNECT messages received: 47966 transport # UDP, unfragmented bytes payload queued total: 40545063 transport # UDP, unfragmented messages queued total: 717601 transport # ATS suggestions received: 1456975 transport # UDP, total bytes received: 48715026 transport # UDP, total, bytes payload, sent: 195603152 transport # UDP, total, bytes overhead, sent: 62391344 transport # UDP, unfragmented msgs, bytes overhead, sent, success: 28701320 transport # UDP, unfragmented msgs, bytes payload, sent, success: 40527275 transport # UDP, unfragmented msgs, messages, sent, success: 717533 transport # KEEPALIVES received in good order: 10221 transport # UDP, fragmented bytes (payload): 156456036 transport # UDP, fragmented messages, total: 7974 transport # UDP, fragmented messages active: 4 transport # total size of fragmented messages: 156774996 transport # messages fragmented: 7974 transport # fragments retransmitted: 32803 transport # UDP sessions active: 26 transport # KEEPALIVE_RESPONSEs received (OK): 13510 transport # UDP, fragmented msgs, bytes overhead, sent, success: 33690024 transport # UDP, fragmented msgs, bytes payload, sent, success: 155075877 transport # UDP, fragmented msgs, messages, sent, success: 7935 transport # fragmentation transmissions completed: 7935 transport # fragment acknowledgements received: 8712 transport # fragments wrap arounds: 13224 transport # validations delayed by global throttle: 831326 transport # validations running: 6439 transport # PINGs for address validation sent: 20929 transport # TCP sessions active: 43 transport # TCP server connect events: 9245 transport # TCP server connections active: 43 transport # address revalidations started: 20977 transport # Multicast HELLO beacons received via UDP: 3430 transport # KEEPALIVES sent: 13691 transport # blocked addresses: 31 transport # Addresses given to ATS: 73 transport # validations succeeded: 8480 transport # PONG messages received: 8482 transport # bytes discarded by TCP (disconnect): 2396256 transport # network-level TCP disconnect events: 7808 transport # Addresses in validation map: 190 transport # address records discarded (timeout): 7201 transport # PONGs unicast via reliable transport: 3449 transport # successful address checks during validation: 3449 transport # PING messages received: 3914 transport # SYN messages sent: 13037 transport # UDP, ACK messages sent: 2591 transport # acknowledgements sent for fragment: 2631 transport # duplicate fragments received: 6383 transport # fragments received: 20466 transport # messages defragmented: 785 transport # failed address checks during validation: 465 transport # ACK messages received: 1050 transport # SYN_ACK messages sent: 1064 transport # SYN messages received: 1342 transport # TCP WELCOME messages received: 605 transport # UDP, fragments discarded without ACK: 40 transport # bytes discarded by TCP (timeout): 6368 transport # peers connected: 12 transport # SYN_ACK messages received: 1361 transport # messages discarded due to timeout: 63 transport # UDP, unfragmented msgs, bytes overhead, sent, failure: 2720 transport # UDP, unfragmented msgs, bytes payload, sent, failure: 17788 transport # UDP, unfragmented msgs, messages, sent, failure: 68 transport # DISCONNECT messages sent: 51 transport # Successful attempts to switch addresses: 209 transport # Attempts to switch addresses: 266 transport # KEEPALIVE_RESPONSEs discarded (not expected): 49 transport # refreshed my HELLO: 70 transport # transport addresses: 10 transport # PONGs dropped, no matching pending validation: 2 transport # UDP, fragmented msgs, bytes payload, sent, failure: 27700502 transport # UDP, fragmented msgs, messages, sent, failure: 35 transport # transmission failures for messages to other peers: 39 transport # PING message for different peer received: 1217 transport # requests to create session with invalid address: 48 transport # bytes payload discarded due to not connected peer: 11042863 transport # KEEPALIVE_RESPONSEs discarded (bad nonce): 4 transport # disconnected from peer upon explicit request: 74 transport # Failed attempts to switch addresses (failed to send SYN CONT): 1 transport # messages timed out while in transport queue: 2 transport # unexpected SYN_ACK messages (not ready): 2 transport # bytes payload dropped (other peer was not connected): 33164 transport # DISCONNECT messages received: 13 transport # KEEPALIVE messages discarded (peer unknown): 29 transport # bits removed from fragmentation ACKs: 1 transport # unexpected SYN_ACK messages (no peer): 1 transport # KEEPALIVE messages discarded (no session): 4 transport # KEEPALIVE_RESPONSEs discarded (no nonce): 2 transport # unexpected ACK messages: 2 dht # Bytes transmitted to other peers: 108483003 dht # DHT requests combined: 132228 dht # Peer selection failed: 24183 dht # Entries added to routing table: 133387 dht # P2P GET bytes received: 36604092 dht # P2P GET requests received: 133387 dht # Bytes of bandwidth requested from core: 48281786 dht # GET messages queued for transmission: 153492 dht # Peers excluded from routing due to Bloomfilter: 708718 dht # GET requests routed: 136192 dht # Duplicate RESULTS found in datacache: 505282 dht # GET requests given to datacache: 125535 dht # PUT messages queued for transmission: 103918 dht # PUT requests routed: 83620 dht # P2P PUT bytes received: 74131271 dht # P2P PUT requests received: 83594 dht # Duplicate REPLIES matched against routing table: 911544 dht # ITEMS stored in datacache: 142474 dht # REPLIES ignored for CLIENTS (no match): 183938 dht # P2P RESULT bytes received: 14715990 dht # P2P RESULTS received: 58854 dht # RESULT messages queued for transmission: 92615 dht # P2P messages dropped due to full queue: 77410 dht # Good REPLIES matched against routing table: 98491 dht # P2P GET requests ONLY routed: 7566 dht # Preference updates given to core: 964 dht # Good RESULTS found in datacache: 49413 dht # HELLOs obtained from peerinfo: 8320 dht # FIND PEER requests ignored due to Bloomfilter: 399 dht # P2P FIND PEER requests processed: 2698 dht # FIND PEER messages initiated: 170 dht # Duplicate REPLIES to CLIENT request dropped: 17118 dht # GET requests from clients injected: 2635 dht # RESULTS queued for clients: 7949 dht # GET requests received from clients: 2412 dht # peers connected: 9 dht # Messages dropped (CORE timeout): 5559 dht # Queued messages discarded (peer disconnected): 325 dht # PUT requests received from clients: 26 dht # Network size estimates received: 17 dht # GET STOP requests received from clients: 2406 dht # requests TTL-dropped: 9 datastore-api # status messages received: 199817 datastore-api # PUT requests executed: 199817 datastore-api # GET ZERO ANONYMITY requests executed: 149 datastore-api # queue overflows: 29 datastore-api # GET REPLICATION requests executed: 9 ats # address suggestions made: 1458769 ats # performance updates given to clients: 2022089 ats # preference change requests processed: 144792 ats # reservation requests processed: 2447565 ats # address updates received: 506541 ats # ATS addresses total: 134 ats # ATS active addresses WAN total: 8 ats # ATS addresses WAN total: 23 ats # addresses: 72 ats # addresses destroyed: 3375 ats # ATS active addresses total: 3325 ats # addresses created: 3447 ats # ATS addresses LAN total: 46 ats # ATS active addresses UNSPECIFIED total: 1 ats # ATS addresses UNSPECIFIED total: 3 ats # ATS active addresses LAN total: 3 peerinfo # peers known: 24 hostlist # active connections: 9 ! hostlist # milliseconds between hostlist downloads: 39600000 hostlist # hostlist downloads initiated: 2 datastore # results found: 503098 datastore # GET requests received: 736493 datastore # requests filtered by bloomfilter: 233404 ! datastore # bytes stored: 415740329 ! datastore # bytes expired: 63013472 datastore # GET ZERO ANONYMITY requests received: 149 datastore # utilization by current datastore: 355288891 ! datastore # bytes used in file-sharing datastore `sqlite': 355288891 datastore # GET REPLICATION requests received: 9 datastore # cache size: 625000000 datastore # quota: 5000000000 |
|
|
Seems to be bursty. I killed core on one computer (SIGTERM, arm restarted) and after more than 1 hour it has gone from 38k to 51k. Might be of interest, fs is taking 152M on one machine as well. |
|
|
More info: after a whole peer restart, fs takes up 106M immediately. |
|
|
10 minutes later, fs is at 110M, core at 97M VIRT, 80k RES. VIRT reverts to 98k after a minute (big pre-allocation? mmap?). On other machines both values are high, saturn uses 187M VIRT and 158M RES right now. |
|
|
I know about FS, know where it is and why (it's queuing replies, not leaking). The growth of the main buffer there is bounded to like 32k * 128 * #connections, but that can still be quite a bit. So core is worrisome, FS is something I can easily 'tune'. |
|
|
heap_tree=peak n3: 7054165 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n1: 6366986 0x525B232: GNUNET_xmalloc_unchecked_ (common_allocation.c:144) n4: 6366986 0x525ADC0: GNUNET_xmalloc_ (common_allocation.c:71) n2: 6065759 0x4068BC: GSC_NEIGHBOURS_transmit (gnunet-service-core_neighbours.c:457) n2: 6065331 0x40A0E4: GSC_KX_encrypt_and_transmit (gnunet-service-core_kx.c:1410) n2: 6064615 0x40D25B: try_transmission (gnunet-service-core_sessions.c:771) n1: 6040929 0x40D9BB: GSC_SESSIONS_transmit (gnunet-service-core_sessions.c:892) n1: 6040929 0x40423D: client_tokenizer_callback (gnunet-service-core_clients.c:571) n1: 6040929 0x529D43B: GNUNET_SERVER_mst_receive (server_mst.c:262) n1: 6040929 0x403EC9: handle_client_send (gnunet-service-core_clients.c:505) n1: 6040929 0x5299FEF: GNUNET_SERVER_inject (server.c:997) n2: 6040929 0x529ACC0: client_message_tokenizer_callback (server.c:1256) n1: 6040929 0x529D43B: GNUNET_SERVER_mst_receive (server_mst.c:262) n1: 6040929 0x529A8D6: process_incoming (server.c:1178) n1: 6040929 0x5265783: receive_ready (connection.c:1139) n1: 6040929 0x52955C1: run_ready (scheduler.c:587) n1: 6040929 0x5295ECC: GNUNET_SCHEDULER_run (scheduler.c:867) n1: 6040929 0x52A46E0: GNUNET_SERVICE_run (service.c:1503) n0: 6040929 0x402C04: main (gnunet-service-core.c:144) |
|
|
Fixed in SVN 35414. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2015-03-09 21:56 | Bart Polot | New Issue | |
| 2015-03-09 21:56 | Bart Polot | Status | new => assigned |
| 2015-03-09 21:56 | Bart Polot | Assigned To | => Christian Grothoff |
| 2015-03-09 22:55 | Christian Grothoff | Note Added: 0009013 | |
| 2015-03-09 22:59 | Bart Polot | Note Added: 0009014 | |
| 2015-03-10 01:58 | Bart Polot | Note Added: 0009015 | |
| 2015-03-10 02:05 | Bart Polot | Note Added: 0009016 | |
| 2015-03-10 02:21 | Bart Polot | Note Added: 0009017 | |
| 2015-03-10 07:55 | Christian Grothoff | Note Added: 0009018 | |
| 2015-03-10 07:55 | Christian Grothoff | Note Edited: 0009018 | |
| 2015-03-20 21:34 | Christian Grothoff | Target Version | 0.11.0pre66 => |
| 2015-03-20 21:36 | Christian Grothoff | Target Version | => 0.11.0pre66 |
| 2015-03-21 00:11 | Christian Grothoff | Note Added: 0009029 | |
| 2015-03-21 00:22 | Christian Grothoff | Note Added: 0009030 | |
| 2015-03-21 00:22 | Christian Grothoff | Status | assigned => resolved |
| 2015-03-21 00:22 | Christian Grothoff | Fixed in Version | => 0.11.0pre66 |
| 2015-03-21 00:22 | Christian Grothoff | Resolution | open => fixed |
| 2018-06-07 00:25 | Christian Grothoff | Status | resolved => closed |
