VNC QPA ignores the client's requested colour order in some cases causing incorrect colours to be displayed

I tried using the VNC QPA platform plugin to view a Qt5 application running under debian 11 via a VNC viewer by running

application -platform vnc

I then tried a number of VNC clients to see if they could connect to this application.  I tried TightVNC, the Gnome Remote Desktop Viewer and https://novnc.com/.

All 3 connected fine, but under noVNC, the colours were wrong.  The Blue and Red channels were swapped.

I originally raised an issue on noVNC for this problem (https://github.com/novnc/noVNC/issues/1715) but after investigation it turned out that Qt was at fault.

noVNC, for its own performance, asks for the server to supply the pixel data in its preferred order (RGB) in a setPixelFormat message.  Qt accepts this message and processes it correctly.  This is done

https://github.com/qt/qtbase/blob/5.15.2/src/plugins/platforms/vnc/qvncclient.cpp#L461

https://github.com/qt/qtbase/blob/5.15.2/src/plugins/platforms/vnc/qvnc.cpp#L265

Qt appears to end up with the 'correct' red/green/blue shift values.

The problem appears to be when Qt then generates the pixel data in QVncClient::convertPixels.

If the client and server are the same endian-ness and color depth, then Qt ignores the redshift/greenshift/blueshift values.  In all other cases, it "appears" to honour these settings.

I assume the condition that says that memcpy can be used needs to be augmented with a check that the byte order of the colours is also the same.  Basically that

m_pixelFormat.redShift = 16

for the screen depth = 32 case.

From what I can tell, it is partly a fluke that TightVNC worked in that it happened to ask for the same colour order that Qt generates.

As a reasonably easy to reproduce test case I used qt5ct running on debian 11 (any qt application will do, I just picked qt5ct as it is generally available on lots of platforms).  Basic steps were

git clone https://github.com/novnc/noVNC.git
cd noVNC
QT_QPA_PLATFORMTHEME=qt5ct qt5ct -platform vnc&
./utils/novnc_proxy --vnc localhost:5900
Open listed URL in your browser

What you should see is something like

what you do see is something like the following where the blues are replaced with oranges/reds