View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0004007libmicrohttpdHTTPS (TLS)public2015-10-15 17:342015-10-20 18:20
Reporterkay Assigned ToChristian Grothoff  
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionno change required 
Platformamd64OSCoreOSOS Version833.0.0
Product Version0.9.44 
Summary0004007: "Not enough memory for write" when using HTTPS
Descriptionsystemd-journal-remote combined with libmicrohttpd + HTTPS fails.

<code>
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: Fixed max_use=605.6M max_size=75.7M min_size=4.0M keep_free=908.5M
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: Opened output file /var/log/journal/remote/remote-coreos1.journal
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: Creating source for passive fd:9 (coreos1)
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: Added RemoteSource as connection metadata 0x56278f394b20
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: Queing response 202: OK.
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: microhttpd: Not enough memory for write!
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: microhttpd: Closing connection (failed to create response header)
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: Cleaning up connection metadata 0x56278f394b20
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: Writer ref count 1
Oct 15 15:25:24 coreos1 systemd-journal-remote[13614]: Closing journal file /var/log/journal/remote/remote-coreos1.journal.
</code>

tested with "while true ;do logger 235235 ;done" and *systemd-journal-upload*
Steps To ReproduceHTTPS doesn't work at all on default coreos, so I had to fix that, I've compiled libmicrohttpd with HTTPS support. And ran "while true ;do logger 235235 ;done" for test.

When I configure libmicrohttpd to listen HTTP - it works fine. When I switch to HTTPS - I receive above error message.
Additional Information./configure --disable-dependency-tracking --disable-silent-rules --enable-bauth --enable-dauth --disable-examples --disable-spdy --disable-postprocessor --disable-curl --with-gnutls --disable-static

systemd-journal-upload returns:

<code>
No more entries, waiting for journal.
No more entries, waiting for journal.
< HTTP/1.1 100 Continue
* Empty reply from server
* Connection #0 to host 192.168.122.164 left intact
Upload to https://192.168.122.164:19532/upload failed: Empty reply from server
Closing journal input.
Root directory /var/log/journal removed.
Directory /var/log/journal/5e54b5c6f91a9ee3c158811537baa8ac removed.
mmap cache statistics: 358368 hit, 16 miss
<code>

Let me know how can I debug this for you.
TagsNo tags attached.

Activities

Christian Grothoff

2015-10-17 18:49

manager   ~0009734

By default, MHD tries to allocate ~32kb RAM per connection. The message you are getting is to indicate that this failed. systemd may be overriding the limit (MHD_OPTION_CONNECTION_MEMORY_LIMIT). I cannot tell why this fails with HTTPS, it could be that you have less memory in the system, or that systemd sets the limit differently. Regardless, this doesn't look like a bug in MHD, as MHD simply handles malloc() returning "out of memory".

kay

2015-10-17 18:52

reporter   ~0009735

Hi Christian,

I'll check that next week and let you know more info. Thank you for looking into this issue.

kay

2015-10-18 18:38

reporter   ~0009742

looks like systemd doesn't override the memory limit... https://github.com/systemd/systemd/blob/master/src/journal-remote/journal-remote.c#L639

will try to debug this with additional debug messages.

kay

2015-10-19 18:56

reporter   ~0009748

I've added some debug info:

./src/microhttpd/connection.c:
+ fprintf (stderr, "MHD_pool_allocate = %d\n", size+1);
  data = MHD_pool_allocate (connection->pool, size + 1, MHD_NO);

and
./src/microhttpd/memorypool.c:
MHD_pool_allocate
...
  asize = ROUND_TO_ALIGN (size);

+ fprintf (stderr, "(pool->pos (%d) + asize (%d) > pool->end (%d)) || (pool->pos (%d) + asize (%d) < pool->pos (%d)) \n", pool->pos, asize, pool->end, pool->pos, asize, pool->pos);


And here are results I've got with default systemd-journal-remote:
https://gist.github.com/kayrus/4a934f799b11a16f6419

Here are results with "MHD_OPTION_CONNECTION_MEMORY_LIMIT, (size_t) (256 * 1024)":
https://gist.github.com/kayrus/37828e94aecf6752a2d1

in last example you can see that initial pos was "131072" and after 1:30 it was decreased down to: "16048" with sudden jumps up to "48400" or "33456".

All debug was made with one "systemd-journal-upload"+"infitine logger dummy_info" and "systemd-journal-remote"+"HTTPS".

Christian Grothoff

2015-10-20 00:15

manager   ~0009751

Well, looks to me like with whatever you are doing (client headers, response headers, etc.), the default 32k/connection are not enough (sometimes), and if you give it more (like 256k), then things work fine. So you should just pass the option to give it more memory per connection and be done.

Note that yes, it can jump because MHD can sometimes 'release' memory if it has been processed. This depends on the request (POST vs. GET, etc.).

kay

2015-10-20 00:19

reporter   ~0009752

Last edited: 2015-10-20 00:21

 Do you mean that the issue could be in headers? Does it worth to debug whole request body? I mean I can figure out that systemd-journal-upload sends much more data rather it sends within insecure connection.

Christian Grothoff

2015-10-20 00:32

manager   ~0009753

The memory pool is for "everything": headers received, POST data, response headers generated, and finally possibly a buffer for response data. All has to fit within the limit, and 32k is deliberately chosen small as MHD is for small, embedded systems. If you can afford 128k or 256k, it is generally simply a good idea to allow MHD to use more memory per connection and it seems then your issue goes away.

kay

2015-10-20 09:11

reporter   ~0009756

Last edited: 2015-10-20 09:11

 Thank you very much for your explanations, Christian. I suppose you can close the ticket.

Issue History

Date Modified Username Field Change
2015-10-15 17:34 kay New Issue
2015-10-17 18:49 Christian Grothoff Note Added: 0009734
2015-10-17 18:49 Christian Grothoff Assigned To => Christian Grothoff
2015-10-17 18:49 Christian Grothoff Status new => feedback
2015-10-17 18:52 kay Note Added: 0009735
2015-10-17 18:52 kay Status feedback => assigned
2015-10-18 18:38 kay Note Added: 0009742
2015-10-19 18:56 kay Note Added: 0009748
2015-10-20 00:15 Christian Grothoff Note Added: 0009751
2015-10-20 00:19 kay Note Added: 0009752
2015-10-20 00:21 kay Note Edited: 0009752
2015-10-20 00:32 Christian Grothoff Note Added: 0009753
2015-10-20 09:11 kay Note Added: 0009756
2015-10-20 09:11 kay Note Edited: 0009756
2015-10-20 18:20 Christian Grothoff Status assigned => closed
2015-10-20 18:20 Christian Grothoff Resolution open => no change required
2024-01-21 13:25 Christian Grothoff Category HTTPS (SSL) => HTTPS (TLS)