QML elements do not use font loaded in FontLoader

When loading fonts in QML from local file in resources and trying to use it in a Text{} the font used is loaded from the system based on the family name.

I have tried loading multiple versions of the same font (Avenir) on OSX by embedding multiple TTF or OTF fonts with my application. When I load the fonts with FontLoader{} I get a "generic" family name "Avenir" and then instead of using this font in my Text {} I get a rendering looking like "Avenir" on my OSX system.

I then modified the font I embed in my app to return a different family name: "AvenirTestQt" and then my Text {} was picking up the proper font loaded from FontLoader.

EDIT: I added a sample app which is loading 4 different versions of the Arial font in TTF retrieved from OSX 10.8.5. The Arial Black has a different family name so is properly rendered. The three other have the same family name and end up rendering the same way but they should be different.
If I edit the Arial Bold Italic and change its family to something like ArialBoldItalicQtTest then it renders as expected