Hello and thank you for your quick reply.
I tested this and unfortunalety this did not solve the problem. It is an interesting idea though.
Setting the stayAtBottom hint is something we actually tried already. This leads to the mainwindow staying even further in the back than the undocked widget. So the rule child-not-infront-of-parent stays active for that case.
So what i did try now was setting the stayOnTop hint whenever the mainwindow gets activated and removing it whenever it gets deactivated. this actually enables the child to be put behind its parent!

In order to make this (partly) work i had to install an event filter for all undocked widgets because the windowActivate event is relayed from the dockwidget to the mainwindow apparently. So without the filter, everytime a user clicks on the undocked widget, the mainwindow would pop up back again over the child that should be in the foreground.
So far, so good. All of this breaks however when trying to remove the stayOnTop flag from the mainwindow. The removal needs to happen everytime the focus of the mainwindow is lost. So i remove the WindowStayOnTopHint and afterwards have to call show() because windows disappear when changing window flags. The problem is that show() then again triggers the WindowActivate event, re-enabling the stayontop hint. This can be filtered but when i do this i end up with a situation where - for some reason - i need to click an undocked widget twice, once to put it to the foreground and a second time to actually activate the widget window.
I am sure that this is a problem in my code and/or can be worked around but this solution starts to be getting dirtier and more hacky. So before diving further down that rabbit hole i would like to double check if there is a more straight forward solution the the actual problem. I also the fear side effects of setting the mainwindow to stay on top, especially if there is a thrid party application window (file explorer or whatever) involved that the user would expect to be "in between" my application windows (a user clicks on myMainwindow, file explorer, myUndockedWidget in that order type-of-situation).

Thanks for reading all of this

@ambershark Thanks for the input. I was afraid the window manager changes might be the issue. We are using KDE as our window manager. I have limited control on what I am allowed to change unfortunately. For now, I will likely move forward with work around. Since the applications dialogues now appear in the task bar, there is little downside to minimizing them. The issue was they would simply get "lost" when they were minimized. I was able to remove the maximize button by setting fixed size restraints on the dialogues. I will update the post if I ever figure out how to get the KDE settings and the Qt Window Flags to play nicely together. I have been investigating the kwindecorations settings, but had no luck yet.