How to stream audio to QAudioSink in a separate thread

[paulmasri]

@SGaist said in How to stream audio to QAudioSink in a separate thread:

Hi,

How did you "move them" ?

Hi!

void AudioThread::setAudioSink(QAudioSink *audioSink)
{
    if (_running)
        qFatal("%s called while thread already running", Q_FUNC_INFO);

    _audioSink = audioSink;
    _audioSink->moveToThread(this);
}

Then the same for the SynthDevice.
Followed by start(TimeCriticalPriority);

Then, within AudioThread::run():

    _synth->open(QIODevice::ReadOnly);
    _audioSink->start(_synth);

[SGaist]

Did you give a parent to your audioSink and synth objects ?

[paulmasri]

Ah! No I didn't. It needs to be AudioThread doesn't it?

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

Ah! No I didn't. It needs to be AudioThread doesn't it?

No, and that's the point. It shouldn't have a parent for you to be able to move it to another thread. Also this sort of thing:

_audioSink = audioSink;
_audioSink->moveToThread(this);

is quite sneaky, as this must be called from the thread _audioSink belongs to and that thread also must be the one that created the AudioThread instance, otherwise in the former case the moveToThread isn't correct, while in the latter the first line is a race condition. And to pile up, after moveToThread gets executed any consequent call to setAudioSink is already a race.

[paulmasri]

I've tried a few things and it's not making sense yet.

@kshegunov said in [How to stream audio to QAudioSink in a separate thread]

No, and that's the point. It shouldn't have a parent for you to be able to move it to another thread.

OK thanks. They are already instantiated without a parent.

@kshegunov said in [How to stream audio to QAudioSink in a separate thread]

Also this sort of thing:

_audioSink = audioSink;
_audioSink->moveToThread(this);

is quite sneaky, as this must be called from the thread _audioSink belongs to and that thread also must be the one that created the AudioThread instance, otherwise in the former case the moveToThread isn't correct, while in the latter the first line is a race condition. And to pile up, after moveToThread gets executed any consequent call to setAudioSink is already a race.

If I understand you correctly, I'm doing this as required:

• The first of those 2 lines assigns a pointer only, making it available within the AudioThread class.
• I've included fuller code excerpts below to avoid confusion, but see the logs beneath as these show what's happening in what thread.

QMainWindow (main thread):

_audioThread = new AudioThread(this);

auto synthDevice = new SynthDevice(args but parent = nullptr);
_audioThread->setSynthDevice(synthDevice);

auto audioSink = new QAudioSink(args but parent = nullptr);
_audioThread->setAudioSink(audioSink);
...

_audioThread->startAudio();

AudioThread (subclasses QThread):

class AudioThread : public QThread
{
...
public:
    void setAudioSink(QAudioSink* audioSink);
    void setSynthDevice(SynthDevice* synthDevice);

    void startAudio();
    void stopAudio();
    bool isRunning() { return _running; }
private:
        std::atomic<bool> _running;
        std::atomic<bool> _stopping;
    
        QAudioSink* _audioSink;
        SynthDevice* _synthDevice;
...
};

void AudioThread::startAudio()
{
    if (_running || _audioSink == nullptr || _synthDevice == nullptr)
        return;

    start(TimeCriticalPriority);

    while (_running)
        usleep(100);
}

void AudioThread::stopAudio()
{
    if (_running)
        _stopping = true;

    while (_running)
        msleep(5);

    _stopping = false;
}

void AudioThread::run()
{
    Q_ASSERT(_audioSink);
    Q_ASSERT(_synthDevice);

    _running = true;
    _synthDevice->open(QIODevice::ReadOnly);
    _audioSink->start(_synthDevice);

    while (!_stopping)
        msleep(50);

    _audioSink->stop();
    _synthDevice->close();
    _running = false;
}

The debug log shows...

• The above snippet of QMainWindow happens in thread 0x600000bc4330
• AudioThread::setSynthDevice() also happens in thread 0x600000bc4330
• AudioThread::setAudioSink() also happens in thread 0x600000bc4330
• AudioThread::startAudio() also happens in thread 0x600000bc4330
• AudioThread::run() happens in thread: 0x6000007f8f00
• SynthDevice::open() happens in thread: 0x6000007f8f00
• During the call to _audioSink->start(), I get the following:

QObject: Cannot create children for a parent that is in a different thread.
(Parent is QDarwinAudioSink(0x6000035c1700), parent's thread is QThread(0x600000bc4330), current thread is AudioThread(0x6000007f8f00)

followed by a single call to SynthDevice::readData() in thread 0x6000007f8f00

So the problem seems to be associated with the thread that QAudioSink is in?? Is QDarwinAudioSink trying to do its own thread thing that's conflicting?

I tried removing the statement _audioSink->moveToThread(this); but that didn't appear to change anything, suggesting the issue is with the thread the QIODevice is in. So is it even possible to move readData() into a separate high priority thread?

I truly appreciate the time you've put into helping me with this. Thank you!

[SGaist]

Since you reimplement the run method, try creating your audio sink in that method.

[paulmasri]

@SGaist said in How to stream audio to QAudioSink in a separate thread:

Since you reimplement the run method, try creating your audio sink in that method.

I've made the change you suggested. Now QMainWindow instantiates AudioThread passing in the QAudioDevice & QAudioFormat information and the QAudioSink is instantiated within AudioThread::run().

At first I passed this as parent for QAudioSink but got the same warning from QObject as before.

Then I tried leaving parent = nullptr. This solved the warning. So that's one step forward. Thanks!

However I still only get one occurrence of readData(). It's as though the audio sink thread detects an error somewhere and closes the stream silently after processing one block of data (which I can hear).

NB: If I set the frame buffer for SynthDevice very low (e.g. 256 frames), then readData() gets called multiple times in quick succession. Having looked at the QtMultimedia source, I believe this is QAudioSinkBuffer repeatedly requesting data until it has filled its buffer. But that's prior to it passing the data to the actual audio device.

Any ideas why this is happening or how to cure it?

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

The first of those 2 lines assigns a pointer only, making it available within the AudioThread class.

This is great, if you can guarantee that nothing touches the pointer from AudioThread::run, otherwise - QAtomicPointer and CAS or a blocking synchronization like QMutex. However this is a detail at this point that isn't so important.

Are you sure the sink can be put in another thread to begin with? Because the documentation doesn't mention the class being reentrant, which in turn most probably means, you can't do this to begin with.

[paulmasri]

Following the suggestion by @SGaist, the QAudioSink is now created within AudioThread::run() and started within it too.

In terms of thread safety, with respect to QAudioSink, is there any difference between creating and starting QAudioSink in QMainWindow (i.e. not multithreaded approach) and creating and starting QAudioSink in AudioThread::run()?

Yet the non-multithreaded version works but the multithreaded version doesn't (requesting and playing a single buffer's worth only).

The only difference I can see now is that SynthDevice is being created in QMainWindow and moved to the AudioThread. Then within AudioThread::run(), SynthDevice is opened.

So to mitigate against a possible issue here, I have tested instantiating and opening SynthDevice within AudioThread::run() too. It did not solve the problem.

Which made me wonder if the issue was in SynthDevice::readData(), so I have reduced this to:

qint64 SynthDevice::readData(char *data, qint64 maxSize)
{
    return maxSize;
}

This should leave the contents of data untouched and report that this is ready to be streamed. This method is both reentrant and thread-safe as you can see. Yet there is still a single call to readData() (which predictably yields a burst of noise).

Any ideas?

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

In terms of thread safety, with respect to QAudioSink, is there any difference between creating and starting QAudioSink in QMainWindow (i.e. not multithreaded approach) and creating and starting QAudioSink in AudioThread::run()?

The module isn't indexed in woboq's site, so I can't take a quick peek and I couldn't be bothered to check it out from git, as I have a build that I have modification on, but if the class isn't marked as reentrant in the documentation it's a good bet it isn't. Thus, I'd advise you not to use it from a thread different from main.

Any ideas?

Perhaps roll back a bit and tell us what is the problem with playing the audio from the main thread? Maybe there's a better approach ...

[paulmasri]

@kshegunov said in How to stream audio to QAudioSink in a separate thread:

Perhaps roll back a bit and tell us what is the problem with playing the audio from the main thread? Maybe there's a better approach ...

That's a good question.

My app is a music synthesizer. A key part of this is audio stability and low latency. This means no audio glitches. I've been achieving a latency of 3ms between UI control changes and changes within the synth engine since 2016. (For a good feeling of responsiveness, latency needs to be <10ms, but I prefer to get as close to 1ms as possible.)

To date I have achieved this by either making use of 3rd party libraries (e.g. RtAudio) or by re-engineering Qt or other audio streaming libraries to service the individual platforms. This works in concert with my synth class as a subclass of QThread with TimeCriticalPriority.

I took this approach because working with QtMultimedia (pre Qt 6.2) resulted in audio glitches. These would be almost continuous if I used a buffer length corresponding to less than 10ms. And even with a longer buffer length, they would be frequent in normal operation due to interruptions upon UI actions and other OS-generated interruptions (e.g. other processes, HDD I/O etc.).

My approach has worked reliably, but as platforms evolve, their audio API changes and I periodically have to re-engineer the audio streaming code. This is time-intensive and hence not very maintainable. With the release of the re-engineered QtMultimedia I have been hoping to use this to handle the 'audio sink' for all platforms, as I'm sure it will be updated as the platforms evolve.

Looking at the audiooutput example, this suffers glitches every time you interact with the UI. (NB: Although the screenshot they provide doesn't show it, the example actually includes a volume control slider.) I replaced the Generator class that came with this with my own synth class that makes smooth transitions upon any change requests (e.g. move the volume slider). Nonetheless, running this in the main thread results in an audio glitch for every mouse press or slider movement. Even a click on the slider without changing the value.

I suspect the only solution is to move the audio processes to a TimeCriticalPriority thread as before, but as this thread shows, I am struggling to see how this can work with QtMultimedia.

I am open to alternative approaches that would meet the same goals of reliable audio streaming (no glitches) and high responsiveness (latency around 1-3ms).

One approach I am starting to consider is to run my synth class in a time-critical thread using its own timer rather than a purely pull approach, and using a thread-safe buffer so that QAudioSink can run in the main thread. However I've already tested a situation where readData() does nothing, so it glitches even when there's zero overhead from my synth class. This leaves me skeptical that QAudioSink can run in the main thread without glitches.

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

My app is a music synthesizer. A key part of this is audio stability and low latency. This means no audio glitches. I've been achieving a latency of 3ms between UI control changes and changes within the synth engine since 2016. (For a good feeling of responsiveness, latency needs to be <10ms, but I prefer to get as close to 1ms as possible.)

Okay, fair enough. Soft realtime then.

To date I have achieved this by either making use of 3rd party libraries (e.g. RtAudio) or by re-engineering Qt or other audio streaming libraries to service the individual platforms. This works in concert with my synth class as a subclass of QThread with TimeCriticalPriority.

The priority change is so the scheduler doesn't stop you mid-way?

Looking at the audiooutput example, this suffers glitches every time you interact with the UI. (NB: Although the screenshot they provide doesn't show it, the example actually includes a volume control slider.) I replaced the Generator class that came with this with my own synth class that makes smooth transitions upon any change requests (e.g. move the volume slider). Nonetheless, running this in the main thread results in an audio glitch for every mouse press or slider movement. Even a click on the slider without changing the value.

Is it possible you get a burst of events that is pushing your important work to the side? I'd be rather suprised if a single event takes more than ms to be processed. Could you perhaps check this?

I suspect the only solution is to move the audio processes to a TimeCriticalPriority thread as before, but as this thread shows, I am struggling to see how this can work with QtMultimedia.

I have to look at the actual code, but it may be a couple of days before that happens. In the mean time, if there's no UI interaction you can hear the sound playing fine? If that is so, injecting a single synthetic UI event breaks it? Or does the sequence of UI events actually cause the glitch?

One approach I am starting to consider is to run my synth class in a time-critical thread using its own timer rather than a purely pull approach, and using a thread-safe buffer so that QAudioSink can run in the main thread.

Well, that's the thing. At least from your explanation it isn't clear to me if the problem is that the "pull a sample" or w/e it is doing is lagging due to other events interfering, or perhaps that the UI events are too slow to be processed. If it's the former, well we can think mitigation strategies, if it's the latter, that'd be worse.

However I've already tested a situation where readData() does nothing, so it glitches even when there's zero overhead from my synth class. This leaves me skeptical that QAudioSink can run in the main thread without glitches.

Possibly, I don't know. I could tell you more of an opinion after I take a look at how QtMultimedia is implemented. What platform(s) are we talking, btw?

[SGaist]

In between, wouldn't a project like PortAudio be more suitable for your application ?

[paulmasri]

@kshegunov said in How to stream audio to QAudioSink in a separate thread:

...This works in concert with my synth class as a subclass of QThread with TimeCriticalPriority.

The priority change is so the scheduler doesn't stop you mid-way?

Yes and also so that there isn't a delay in it getting called. i.e. minimise time into-through-out of readData().

Looking at the audiooutput example, this suffers glitches every time you interact with the UI. (NB: Although the screenshot they provide doesn't show it, the example actually includes a volume control slider.) I replaced the Generator class that came with this with my own synth class that makes smooth transitions upon any change requests (e.g. move the volume slider). Nonetheless, running this in the main thread results in an audio glitch for every mouse press or slider movement. Even a click on the slider without changing the value.

Is it possible you get a burst of events that is pushing your important work to the side? I'd be rather suprised if a single event takes more than ms to be processed. Could you perhaps check this?

I'm not sure how to check this.

I suspect the only solution is to move the audio processes to a TimeCriticalPriority thread as before, but as this thread shows, I am struggling to see how this can work with QtMultimedia.

I have to look at the actual code, but it may be a couple of days before that happens. In the mean time, if there's no UI interaction you can hear the sound playing fine? If that is so, injecting a single synthetic UI event breaks it? Or does the sequence of UI events actually cause the glitch?

Thanks for the offer of looking. I've been going through the source code myself. It feels more cleanly written than the old QtMultimedia. But I get the sense you're more of an expert in multithreading so your insights would be appreciated.

I'll have a think about injecting a synthetic UI event. I'm not sure how to achieve that but I'm sure I can work it out. However my suspicion is that the issue won't be the event (slot?) so much as the OS involvement in passing the event to Qt and Qt's steps to generate the signal. Worth a test though.

One approach I am starting to consider is to run my synth class in a time-critical thread using its own timer rather than a purely pull approach, and using a thread-safe buffer so that QAudioSink can run in the main thread.

Well, that's the thing. At least from your explanation it isn't clear to me if the problem is that the "pull a sample" or w/e it is doing is lagging due to other events interfering, or perhaps that the UI events are too slow to be processed. If it's the former, well we can think mitigation strategies, if it's the latter, that'd be worse.

However I've already tested a situation where readData() does nothing, so it glitches even when there's zero overhead from my synth class. This leaves me skeptical that QAudioSink can run in the main thread without glitches.

Possibly, I don't know. I could tell you more of an opinion after I take a look at how QtMultimedia is implemented. What platform(s) are we talking, btw?

At this point Windows 10 (UWP for Microsoft Store and also executable for direct download) and iPad. In future, I may be adding Android & WebGL.

@SGaist said in How to stream audio to QAudioSink in a separate thread:

In between, wouldn't a project like PortAudio be more suitable for your application ?

When I first created the app 5 years ago I looked at various libraries including PortAudio. In the end I went with RtAudio. I don't recall why. However neither library supports mobile platforms or WebGL. Hence my wish to make use of QtMultimedia as a regularly updated, fully cross-platform library.

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

Yes and also so that there isn't a delay in it getting called. i.e. minimise time into-through-out of readData().

This doesn't sound quite right. I'd speculate that the latency is due to event (re)ordering. If you take a simple imperatively run thread, given that there's nothing much happening on the system, the OS scheduler is going to happily allocate you 100% of the CPU time. I'd speculate that you observe a latency more as a consequence of the way the events are ordered/processed (I'm not claiming proof, just a feeling).

I'm not sure how to check this.

Well, firstly, if you run the application without any UI interaction(s), does it play smoothly? And secondly, if you for example post few regular events from the code (QCoreApplication::postEvent) does it break? What about sending them instead (QCoreApplication::sendEvent)?

I'll have a think about injecting a synthetic UI event. I'm not sure how to achieve that but I'm sure I can work it out. However my suspicion is that the issue won't be the event (slot?) so much as the OS involvement in passing the event to Qt and Qt's steps to generate the signal. Worth a test though.

You could take some inspiration from Qt's own testing library, the UI autotests do simulate clicks, resizes and such. If synthetic events don't produce the glitch, then I have some idea where you could dig up next. In a nutshell I'm wondering if the call into the backend is the problem, or how Qt processes the events, or the speed with which the events are processed. These are the obvious sources for what you observe, from where I'm standing.

But I get the sense you're more of an expert in multithreading so your insights would be appreciated.

Ha, I doubt it you could call me an expert. I'm a lowly physicist. ;)

@SGaist said in How to stream audio to QAudioSink in a separate thread:
In between, wouldn't a project like PortAudio be more suitable for your application ?

Also assuming this problem is indeed confirmed as described, I'd say QtMultimedia is of very limited utility ... so I'd say worth investigating, right?

[paulmasri]

@kshegunov said in How to stream audio to QAudioSink in a separate thread:

Well, firstly, if you run the application without any UI interaction(s), does it play smoothly? And secondly, if you for example post few regular events from the code (QCoreApplication::postEvent) does it break? What about sending them instead (QCoreApplication::sendEvent)?

I set up a timer and first of all tested whether the timer events caused any issues — they don't. Then I tried both postEvent & sendEvent with a synthetic mouse event (alternating between press and release) on a point along the slider. In both cases I get an audio glitch upon almost every MouseButtonPressed, just as when it was a real UI event.

Does this tell you anything useful?

[kshegunov]

Sorry for the delay, I was traveling.

Does this tell you anything useful?

That's some interesting findings. Maybe. I'd suspect in that case that the problem is the time it takes to do the paint/GUI events, not so much events in general. That's rather unfortunate as I'm unsure that this can be really fixed.

[paulmasri]

Yeah :-(
I included logging and included time elapsed. This showed that typically the time between calls to readData() for a long buffer (10ms) is typically 1-2ms, with occasional gaps 3-5ms. However every real/simulated GUI interaction results in a gap of 10-30ms.

Nevertheless I do like the QtMultimedia library for its cross-platform support, so unless someone has a great idea how to solve this situation, I'm going to fork the library and see if I can make QAudioSink and related classes thread-safe, so that I can put them in a high priority thread.

I've not tried building any part of Qt from source before, so I'm already seeking support in the forums. Hopefully I'll be able to get somewhere with it, and who knows, maybe even contribute something back.

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

Nevertheless I do like the QtMultimedia library for its cross-platform support, so unless someone has a great idea how to solve this situation, I'm going to fork the library and see if I can make QAudioSink and related classes thread-safe, so that I can put them in a high priority thread.

What I was thinking of initially (hence the million questions game) was to drive the event loop manually with some timeout that should be okay for your application, so you control how long events are processed. I know sounds like an abomination, but should approximate what you want. Although from what you'd observed I'm utterly unconvinced this is going to truly work. It's probably worth a shot still, but a long one.

[paulmasri]

@kshegunov said in How to stream audio to QAudioSink in a separate thread:

What I was thinking of initially (hence the million questions game) was to drive the event loop manually with some timeout that should be okay for your application, so you control how long events are processed. I know sounds like an abomination, but should approximate what you want. Although from what you'd observed I'm utterly unconvinced this is going to truly work. It's probably worth a shot still, but a long one.

I agree it does sound an abomination! Aside of feeling wrong — messing with something unrelated to audio streaming in order to solve audio streaming issues — I'm doubtful it would work. As it says in the documentation, any time I would call processEvents(), it will process all queued events "however long it takes." This seems guaranteed to perpetuate the current issues.

I'm currently working through the audio streaming classes to understand them and see if I can work with them in some way, ideally to improve QtMultimedia and submit a pull request, but otherwise to pull them out of QtMultimedia and make use of them somehow myself.