[Feature Request] Allow WindowModal to exclude transient top-level windows from modal blocking

Aidener

Title

[Feature Request] Allow WindowModal to exclude transient top-level windows from modal blocking

Description

Qt::WindowModal currently treats transient parents as part of the modal window hierarchy.

This can be problematic when a transient/owned top-level window is used for Z-order and lifetime management, but is intended to remain interactive while a modal dialog is open for another top-level window.

Example

Consider two independent top-level windows:

A = main/document window
B = independent tool window

B is a top-level window and has a transient relationship with A.

The transient relationship is used for:

keeping B above A;
following A's minimize/restore behavior;
following A's visibility/lifetime;
maintaining the desired relationship between the two windows.

Now A opens a Qt::WindowModal dialog C:

The desired behavior is:

C blocks A
C does not block B

However, B is currently considered part of A's modal hierarchy because Qt's WindowModal implementation follows transient parents.

As a result:

C blocks A
C also blocks B

Relevant Qt API

Qt already has an explicit distinction between normal parent relationships and transient-parent relationships:

enum AncestorMode {
    ExcludeTransients,
    IncludeTransients
};

The documentation for QWindow::AncestorMode states that ExcludeTransients means that transient parents are not considered ancestors. QWindow::isAncestorOf() and QWindow::parent() support this distinction.

However, QGuiApplicationPrivate::isWindowBlocked() currently uses the transient relationship when calculating Qt::WindowModal blocking.

In Qt 5, see:

qtbase/src/gui/kernel/qguiapplication.cpp

QGuiApplicationPrivate::isWindowBlocked()

The Qt::WindowModal implementation walks both parent() and transientParent().

In Qt 6, the implementation explicitly uses:

QWindow::IncludeTransients

for the WindowModal ancestor checks:

if (current->isAncestorOf(modalWindow,
                          QWindow::IncludeTransients)) {
    ...
}

current = current->parent(QWindow::IncludeTransients);

Therefore, the current implementation effectively couples:

transient-parent relationship
        +
WindowModal blocking hierarchy

These relationships are not necessarily equivalent.

Why this matters

A transient relationship can be needed purely for window-management purposes:

A <---- transient ----> B

Purpose:
    Z-order
    minimize/restore
    lifetime

while the desired modal relationship may be:

C ---- modal ----> A

C blocks:
    A

C does not block:
    B

In other words:

Z-order/lifetime hierarchy
        !=
modal input-blocking hierarchy

Proposed behavior

I suggest providing a way for applications to opt out of transient windows participating in WindowModal blocking.

For example, conceptually:

dialog->setWindowModality(Qt::WindowModal);
dialog->setModalAncestorMode(QWindow::ExcludeTransients);

or another API with equivalent semantics.

The important point is that this should be opt-in, so existing applications retain the current behavior.

Minimal implementation experiment

The existing implementation can be demonstrated with a very small change.

In the Qt 6 implementation, the relevant code currently uses:

QWindow::IncludeTransients

for WindowModal.

As an experiment, replacing the relevant WindowModal ancestor traversal with:

QWindow::ExcludeTransients

makes transient top-level windows no longer participate in the modal hierarchy.

This is not necessarily proposed as the final API/patch, but it demonstrates that Qt already has the necessary ancestor semantics internally.

For Qt 5, the equivalent minimal experiment is to make the WindowModal traversal follow only:

QWindow::parent()

and not:

QWindow::transientParent()

Possible patch direction

A minimal API could be an opt-in property controlling the ancestor mode used by WindowModal.

For example:

class QWindow
{
public:
    enum ModalAncestorMode {
        IncludeTransientParents,
        ExcludeTransientParents
    };

    void setModalAncestorMode(ModalAncestorMode mode);
    ModalAncestorMode modalAncestorMode() const;
};

Then QGuiApplicationPrivate::isWindowBlocked() could use that mode when processing Qt::WindowModal.

Alternatively, an API at the application/dialog level could expose the same choice.

Compatibility consideration

I do not suggest changing the existing default behavior unconditionally.

There may be applications that intentionally rely on transient windows being blocked together with their parent.

The feature could therefore default to the current behavior:

WindowModal
    -> IncludeTransients

while allowing applications with independent top-level tool windows to request:

WindowModal
    -> ExcludeTransients

Minimal reproduction

A minimal reproduction would consist of:

A: top-level window
B: independent top-level window with transientParent(A)
C: WindowModal dialog for A

Expected:

C blocks A
B remains interactive

Actual:

C blocks A
B is also blocked

This issue is not about WindowStaysOnTopHint or global always-on-top behavior.

The requirement is specifically to use the transient relationship for window-management/Z-order purposes while keeping the modal input-blocking hierarchy independent from that relationship.

Would an API allowing WindowModal to use ExcludeTransients be considered reasonable?

SimonSchroeder

@Aidener said in [Feature Request] Allow WindowModal to exclude transient top-level windows from modal blocking:

B = independent tool window

Don't make A the parent of B if you really mean "independent" tool window. Just pass nullptr as parent to B. If you using WindowModal instead of AppModal this should work. But I see your point about visibility and lifetime.

BTW, I don't think that AI made your feature request any better. Longer text does not necessarily better provides the point of your feature request. The way you told AI what feature you expect would have been a lot more helpful for us (I guess). The AI just gave the general structure for a proposal. The "minimal implementation experiment" is not an experiment because it does not contain any real source code (you'd have to fill that section yourself). The "possible patch direction" is still incomplete (it is only half the patch). And the "minimal reproduction" is no reproduction at all because it does not contain any source code that could be tested. Again, you'd have to fill out that section yourself. I am quite annoyed by people just posting AI slop. At least turn on your brain and check the output of AI (and modify it to make it suitable for the target audience). (I just tried it: Even AI cannot summerize your post and get to the key points.)

Christian Ehrlicher

@SimonSchroeder said in [Feature Request] Allow WindowModal to exclude transient top-level windows from modal blocking:

BTW, I don't think that AI made your feature request any better.

tbh - I don't read such stuff at all anymore, it's to much with no real information. Just bloated AI slop.

Aidener

Re: [Feature Request] Allow WindowModal to exclude transient top-level windows from modal blocking

I am So sory for reply lately. Using Ai just for saving the time and hope to statement clearly but seem as not well. This is Minimal code to show my question. Somebody try and comment will be appreciateed!

#include <QApplication>
#include <QMainWindow>
#include <QDialog>
#include <QPushButton>
#include <QVBoxLayout>
#include <QWindow>

#define SECTION_USE_setTransientParent

int main(int argc, char* argv[])
{
    QApplication app(argc, argv);

    QMainWindow* mainWindow = new QMainWindow;
    mainWindow->setWindowTitle("A - MainWindow");
    mainWindow->resize(400, 300);

    auto* openButton = new QPushButton(
        "Open WindowModal Dialog",
        mainWindow
    );
    mainWindow->setCentralWidget(openButton);

    // B: A top level Dialog for message. example for license, global tip for document
    // Sometimes we need to show some unimportant message and just affirm when user is free
    QDialog* TipDialog = new QDialog(); //parent NULL for C with Qt::WindowModal can not block it.
    TipDialog->setWindowTitle("B - Non-modal Dialog");
    TipDialog->setModal(false);
    TipDialog->setWindowModality(Qt::NonModal);
    TipDialog->resize(250, 150);

    auto* toolButton = new QPushButton("Click B");
    auto* toolLayout = new QVBoxLayout(TipDialog);
    toolLayout->addWidget(toolButton);

#ifdef SECTION_USE_setTransientParent
    // set B -> A transient relationship
    TipDialog->winId(); //Just for get windwow hwd. Another way By Using show() instead.
    mainWindow->winId();
    //keep TipDialog always over mainWindow and 
    //Z-order : lower with mainWindow when other process activate
    //minimize / restore
    //attention that setWindowFlags(windowFlags() | Qt::WindowStaysOnTopHint) can not get the same result
    TipDialog->windowHandle()->setTransientParent(
        mainWindow->windowHandle()
    );
#else
    //nothing
    //TipDialog is independent but as a message dialog, it is unaccepted
    // when other process activate and show inside of TipDialog and mainWindow.
    TipDialog->setWindowFlags(TipDialog->windowFlags() | Qt::WindowStaysOnTopHint);
#endif 

    // C:  WindowModal Dialog of MainWindow
    QObject::connect(openButton, &QPushButton::clicked, [&] {
        //as child of MainWindow. Sometimes like the QMessageBox::Information but excec() was Qt::ApplicationModal
        auto* dialog = new QDialog(mainWindow);
        dialog->setWindowTitle("C - WindowModal");
        auto* closeButton = new QPushButton("Close", dialog);
        auto* layout = new QVBoxLayout(dialog);
        layout->addWidget(closeButton);

        QObject::connect(
            closeButton,
            &QPushButton::clicked,
            dialog,
            &QDialog::accept
        );

        dialog->setWindowModality(Qt::WindowModal); //only block itself and parent window
        dialog->open(); //this calls will block B out of expectation
        });

    mainWindow->show();

    TipDialog->move(
        mainWindow->x() + mainWindow->width() + 20,
        mainWindow->y()
    );
    TipDialog->show();

    return app.exec();
}

After compiling it will show like that.

SGaist

@Aidener I merged your answer with the original thread. Please keep things together.