An InnerShadow with fast set to true uses an FastInnerShadow. The FastInnerShadow doesn't apply verticalOffset correctly.  It is scaling it based on width instead of height. So the more you have a non 1:1 aspect ratio on your source, the more distortion you will see.

Note: This is not an issue for GaussianInnerShadow.

I've attached a screenshot of the issue and the source code to reproduce it.  The left side uses the GaussianInnerShadow and is correct; all four rectangles have the proper inner shadow.  The right side uses FastInnerShadow and boxes B and C are distorted; B is stretched while C is squished.  But A and D are correct because the Rectangle is a square.

From FastInnerShadow.qml, lines 72-83, in Qt 5.15.2:

72.    ShaderEffect {
73.        id: level0
74.        property variant source: sourceProxy.output
75.        property real horizontalOffset: rootItem.horizontalOffset / rootItem.width
76.        property real verticalOffset: rootItem.verticalOffset / rootItem.width
77.        property color color: rootItem.color
78.
79.        anchors.fill: parent
80.        visible: false
81.        smooth: true
82.        fragmentShader: "qrc:/qt-project.org/imports/QtGraphicalEffects/shaders/fastinnershadow_level0.frag"
83.    }

On line 76, verticalOffset is being divided by width instead of height.  Changing line 76 to...

property real verticalOffset: rootItem.verticalOffset / rootItem.height

...fixes the issue.