Large memory consumption (>35G!!!) while downloading a http request

Minimal example

Here you can find minimal reproducible example. It tries to download just one file - http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-3gram-20120701-an.gz.

I measure memory consumption via two tools: standard KDE's System Monitor, and google's tcmalloc heapprofile. More precisely, I link against tcmalloc, and then run the binary from the example as follows: $ HEAPPROFILE=mybin.hprof ./google_ngram_qt_bug.

I also used strace to figure out what is going on.

The problem

So, heapprofile reports two things:

• Lines in logs like "Dumping heap profile to mybin.hprof.0361.heap (36215 MB currently in use)".
• and, more importantly, mybin.hprof.0361.heap files that can later be visualised (see the attached screenshot). In particular, on the screenshot you can see how some mysterious (non-main) thread (QTcpServer -> QAbstractSocket -> <signal-slot> -> QHttpPart) in the background downloaded 36GB.

Apart from the tool, I have a very useful cout in my app itself:

      std::cout << Reply->url().toString().toStdString()
                << " Reply->bytesAvailable(): " << Reply->bytesAvailable()
                << ", Reply->readBufferSize(): " << Reply->readBufferSize()
                << ", bytesAvailable Before Polling: " << BeforePolling
                << " => diff = " << Reply->bytesAvailable() - BeforePolling << ", Was Requested: " << MaxSize
                << std::endl;

which gives the following output (slightly redacted):

...an.gz bytesAvailable: 988118, bytesAvailable Before Polling: 988118, Was Requested: 16384
an.gz Reply->bytesAvailable(): 971734 <--- NOTE bytesAvailable is decreasing
an.gz Reply->bytesAvailable(): 955350
an.gz Reply->bytesAvailable(): 938966
an.gz Reply->bytesAvailable(): 21462  <--- Finally it becomes 21K

!!! All of a sudden, from thin air, bytesAvailable() become 500MB(!):

an.gz Reply->bytesAvailable(): 540618996, Reply->readBufferSize(): 1048576, bytesAvailable Before Polling: 5078 => diff = 540613918, Was Requested: 16384
...

So, code in the main

• request 16K,
• but in response to get 16K, Qt got 500MB.

strace shows huge amount of read traffic from socket 8 in thread id 4418:

[pid  4418] ioctl(8, FIONREAD, [7090])  = 0
[pid  4418] ioctl(8, FIONREAD, [12762]) = 0
[pid  4418] ioctl(8, FIONREAD, [25524]) = 0
[pid  4418] read(8, "%i\200}\31\370\210\1\233\\\361o`\251/\325\235\6\300\254Va\33`\331\26\34\17\274\261\252\346"..., 25524) = 25524
[pid  4418] write(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid  4418] ioctl(8, FIONREAD, [26942]) = 0
[pid  4418] read(8, "\363\206\233\316\300\r/A\371*\361\246\21\336\\\v\322\343o\311\373>\260\220=\366\346\226\277\6\355W"..., 26942) = 26942

Now if we grep for the thread id we will get:
$ ps -e -T | grep 4418
4415 4418 pts/1 00:00:20 QNetworkAccessM
 
strace reported that file descriptor #8 was used for reading. let's have a look at what it is associated with:

lsof -i -a -p 4443
COMMAND    PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
google_ng 4443 dimanne    8u  IPv4 5585565      0t0  TCP fb4c8169a8dd:38282->lhr35s07-in-f16.1e100.net:http (ESTABLISHED)

8u IPv4 fb4c8169a8dd:38282->lhr35s07-in-f16.1e100.net:http

My surmise

I presume that someone decided that it would be perfectly fine to implicitly download a page in the background in a different thread. It also means that:

• QIODevice (from the point of view of end-user) does not behave like a stream - it eagerly reads everything in memory. (while I understand that it is not QIODevice that does it, but rather a different component in a different thread).
• setReadBufferSize() is completely broken.

So, since QIOStream/setReadBufferSize are not working, is there a way to disable eager and implicit downloading in the background?