A major theme I have in my code is keeping UI code and solver code completely separate, and I would like to continue that trend with this specific task.

My question is - how can I make these functions show a message box to the user when the function is executed? For example, when the user presses the button, and the function executes, if it exceeds the maximum number of iterations, the function should finish executing, and then a message box should show up warning the user that the function failed to converge. Furthermore, how can I implement such a thing without using UI code in the file with the solver source code?

Make your solver return information about whether it converged on an answer or not:

struct SolverResult {
    double finalValue;
    bool converged;
};

Then, do extra checks when processing the result.

If your result-processing code currently looks like this...

void MainWindow::onSolverFinished(double finalValue)
{
    this->doSomethingWith(finalValue);
}

...now you can do extra checks:

void MainWindow::onSolverFinished(const SolverResult &result)
{
    if (result.converged)
    {
        // Yay, we found a solution!
        this->doSomethingWith(result.finalValue);
    }
    else
    {
        QMessageBox::warning(this,
            "No solution found", 
            QStringLiteral("Failed to converge on a solution within %1 iterations").arg(m_maxIterations)
        );
    }
}

Alternatively, you could adopt a convention like "Return NaN if it fails to converge", and show the QMessageBox inside if (qIsNan(finalValue))

Never used invokeMethod(). Is it synchronous or asynchronous?

invokeMethod() basically lets you post calls to slots in the event queue. That's at least how I am using it. I makes it easier to not have a bunch of signals just so you connect them and call them once.

One problem with a simple callback will arise when you run the solver in a different thread. Then you have to add a context switch eg. with a QTimer single shot or a signal to switch to the main thread.

Beside the suggestion of @Christian-Ehrlicher, one can also make use of invokeMethod so everything GUI can properly stay in the place of concern rather than having unrelated code handling GUI related stuff.

This might look like the following:

typedef void (*solver_warning_callback_t)(const std::string &message, int current_iteration, int max_iterations, void *callback_data);
...
solver_warning_callback_t solver_warning_callback = nullptr;
void *solver_warning_callback_data = nullptr;

From your main window you would have to register the callback and set the callback data:

solver_warning_callback = [](const std::string &message, int current_iteration, int max_iterations, void *callback_data)
{
    MainWindow *mw = reinterpret_cast<MainWindow*>(callback_data);
    QMessageBox::warning(mw, tr("Solver warning"), QString::fromStdString(message));
};
solver_warning_callback_data = this; // setting callback data from a member function within MainWindow

Then you can call it from your solver function:

solver_warning_callback("Solver did not converge withing specified iterations", i, maxIterations, callback_data);

Notice that I included the iterations and max iterations in this example to show you would handle additional parameters (but they don't have to be used by the callback itself). Assuming you might want to have no Qt inside your solver I also used std::string instead of QString. Usually, the callback data is the last parameter to a callback function. It is always a void* pointer to allow for any data. This can safely be cast back and forth. If you want to use more than just a single parameter as the callback data you just put it into a struct. However, you have to make sure that the struct variable lives long enough! If you don't have any callback data, just leave it as nullptr. The callback data does not have to be used by the callback function, but usually sooner or later you'll need it. So, any callback function should have that callback data as a parameter. Using a lambda function that captures the data instead would not work. The capture list needs to be empty (if you are using a lambda; regular functions also work!).

Whether you use socket, signal or similar, your solver module may not need knowledge of UI code but will still need to include Qt headers to call it. If you want to avoid that, to keep your solver non-Qt-aware, you can use a function pointer or class with virtual method to do the socketing/signalling from the solver side. Since there are static methods of QMessageBox (https://doc.qt.io/qt-6/qmessagebox.html#static-public-members) which display a dialog and do not demand any knowledge of the UI if you go for application modal you can even use one of those in the function to display it directly without sockets/signals etc. if you want the simplest, dirty solution. Or even Qt's logging/qInstallMessageHandler(QtMessageHandler handler) where you could cause your qDebug() or similar calls to be shown to the user in a message box. So many ways! :)