[Feature Request] Allow WindowModal to exclude transient top-level windows from modal blocking

Description

Qt::WindowModal currently treats transient parents as part of the modal window hierarchy.

This can be problematic when a transient/owned top-level window is used for Z-order and lifetime management, but is intended to remain interactive while a modal dialog is open for another top-level window.

Example

Consider two independent top-level windows:

A = main/document window
B = independent tool window

B is a top-level window and has a transient relationship with A.

The transient relationship is used for:

• keeping B above A;
• following A's minimize/restore behavior;
• following A's visibility/lifetime;
• maintaining the desired relationship between the two windows.

Now A opens a Qt::WindowModal dialog C:

        C
        │
        ▼
        A
        │
        B

The desired behavior is:

C blocks A
C does not block B

However, B is currently considered part of A's modal hierarchy because Qt's WindowModal implementation follows transient parents.

As a result:

C blocks A
C also blocks B

Relevant Qt API

Qt already has an explicit distinction between normal parent relationships and transient-parent relationships:

enum AncestorMode {
    ExcludeTransients,
    IncludeTransients
};

The documentation for QWindow::AncestorMode states that ExcludeTransients means that transient parents are not considered ancestors. QWindow::isAncestorOf() and QWindow::parent() support this distinction.

However, QGuiApplicationPrivate::isWindowBlocked() currently uses the transient relationship when calculating Qt::WindowModal blocking.

In Qt 5, see:

qtbase/src/gui/kernel/qguiapplication.cpp

QGuiApplicationPrivate::isWindowBlocked()

The Qt::WindowModal implementation walks both parent() and transientParent().

In Qt 6, the implementation explicitly uses:

QWindow::IncludeTransients

for the WindowModal ancestor checks:

if (current->isAncestorOf(modalWindow,
                          QWindow::IncludeTransients)) {
    ...
}

current = current->parent(QWindow::IncludeTransients);

Therefore, the current implementation effectively couples:

transient-parent relationship
        +
WindowModal blocking hierarchy

These relationships are not necessarily equivalent.

Why this matters

A transient relationship can be needed purely for window-management purposes:

A <---- transient ----> B

Purpose:
    Z-order
    minimize/restore
    lifetime

while the desired modal relationship may be:

C ---- modal ----> A

C blocks:
    A

C does not block:
    B

In other words:

Z-order/lifetime hierarchy
        !=
modal input-blocking hierarchy

Proposed behavior

I suggest providing a way for applications to opt out of transient windows participating in WindowModal blocking.

For example, conceptually:

dialog->setWindowModality(Qt::WindowModal);
dialog->setModalAncestorMode(QWindow::ExcludeTransients);

or another API with equivalent semantics.

The important point is that this should be opt-in, so existing applications retain the current behavior.

Minimal implementation experiment

The existing implementation can be demonstrated with a very small change.

In the Qt 6 implementation, the relevant code currently uses:

QWindow::IncludeTransients

for WindowModal.

As an experiment, replacing the relevant WindowModal ancestor traversal with:

QWindow::ExcludeTransients

makes transient top-level windows no longer participate in the modal hierarchy.

This is not necessarily proposed as the final API/patch, but it demonstrates that Qt already has the necessary ancestor semantics internally.

For Qt 5, the equivalent minimal experiment is to make the WindowModal traversal follow only:

QWindow::parent()

and not:

QWindow::transientParent()

Possible patch direction

A minimal API could be an opt-in property controlling the ancestor mode used by WindowModal.

For example:

class QWindow
{
public:
    enum ModalAncestorMode {
        IncludeTransientParents,
        ExcludeTransientParents
    };

    void setModalAncestorMode(ModalAncestorMode mode);
    ModalAncestorMode modalAncestorMode() const;
};

Then QGuiApplicationPrivate::isWindowBlocked() could use that mode when processing Qt::WindowModal.

Alternatively, an API at the application/dialog level could expose the same choice.

Compatibility consideration

I do not suggest changing the existing default behavior unconditionally.

There may be applications that intentionally rely on transient windows being blocked together with their parent.

The feature could therefore default to the current behavior:

WindowModal
    -> IncludeTransients

while allowing applications with independent top-level tool windows to request:

WindowModal
    -> ExcludeTransients

Minimal reproduction

A minimal reproduction would consist of:

A: top-level window
B: independent top-level window with transientParent(A)
C: WindowModal dialog for A

Expected:

C blocks A
B remains interactive

Actual:

C blocks A
B is also blocked

This issue is not about WindowStaysOnTopHint or global always-on-top behavior.

The requirement is specifically to use the transient relationship for window-management/Z-order purposes while keeping the modal input-blocking hierarchy independent from that relationship.

Would an API allowing WindowModal to use ExcludeTransients be considered reasonable?