Details
P2: Important Description
This report describes a problem with the signal QProcess::bytesWritten on Windows.
The context
QProcess is an asynchronous device. When you write data to the "write channel" of the QProcess (the stdin of the created process), the QProcess::write operation completes immediately. Internally, the data are buffered in a QRingBuffer and later written to the system device when space is available.
A classical problem of the producer / consumer pattern is the flow regulation when the producer (the parent application writing to the QProcess instance) produces data much faster than the consumer (the child process reading its stdin) can consume them.
Having a reliable regulation mechanism is mandatory. Otherwise, the data will accumulate endlessly in the QRingBuffer, the virtual memory of the process increases up to the limit and the producer application crashes from a memory exhaustion.
The regulation mechanism in Qt is the signal QProcess::bytesWritten(qint64), inherited from QIODevice. The documentation says that "this signal is emitted every time a payload of data has been written to the device. The bytes argument is set to the number of bytes that were written in this payload."
Thus, an application can evaluate the amount of buffered data, make sure it never runs out of control, stop generating data when too much data are not yet written to the device, and restart generating data when receiving a QProcess::bytesWritten indicating that suficient data have left the application's memory. So far, so good.
The problem
This works as expected on Linux (see example below). However, it fails on Windows. On Windows, QProcess::bytesWritten is emitted immediately after QProcess::write, when the data are still in the internal QRingBuffer. Thus, the application has no way to regulate the flow.
Demonstration
The attached source code illustrates the problem. The application creates a process which runs the same executable as the parent. The ChildProcess class, executed in the created process, reads its stdin very slowly: it reads 1 byte every 5 seconds. The ParentProcess class, executed in the main application, writes data very fast on the write channel of the QProcess: it writes 1 megabyte every second. The parent process also receives the QProcess::bytesWritten signal and displays the amount of data.
On Linux, the application works as expected. The message "Parent: periodic: wrote 1048576 bytes" is displayed every second, for each QProcess::write. But the message "Parent: notified of XX bytes written to process", reporting the signal QProcess::bytesWritten, is displayed only twice, the first time 64 kB are written and then a second time 16 kB have been written when the child process reads 1 byte from its input. Thus, the application is aware that data don't go out fast and can slow down.
On Windows, every second, the message "Parent: periodic: wrote 1048576 bytes" is immediately followed by "Parent: notified of 1048576 bytes written to process", endlessly until the application runs out of memory because the data are accumulated into the internal QRingBuffer of the QProcess.
When I first reported the issue in the Qt forums (https://forum.qt.io/topic/68203/qiodevice-byteswritten-but-bytes-are-not-written-on-asynchronous-devices/9), a discussion started about the appropriate time where QProcess::bytesWritten should be emitted, when the data leaves the application user's space or when the kernel fifo is emptied. This can be discussed. However, at the end of the day, the signal must be useful. On Windows, it is useless in the sense that it does not bring any useful information to the application. Worse, it prevents the application from the only flow regulation control mechanism we have.
So, this is clearly a bug since it makes it impossible for a Qt application running on Windows to regulate the flow control in the classical pattern of a fast producer / slow consumer.
Parent: starting Child: starting Parent: periodic: wrote 1048576 bytes Parent: notified of 65536 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Child: read 1 input bytes Parent: notified of 16384 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Child: read 1 input bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Child: read 1 input bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes Parent: periodic: wrote 1048576 bytes
Parent: starting Child: starting Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Child: read 1 input bytes Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Child: read 1 input bytes Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process Parent: periodic: wrote 1048576 bytes Parent: notified of 1048576 bytes written to process