How to stream audio to QAudioSink in a separate thread

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

My app is a music synthesizer. A key part of this is audio stability and low latency. This means no audio glitches. I've been achieving a latency of 3ms between UI control changes and changes within the synth engine since 2016. (For a good feeling of responsiveness, latency needs to be <10ms, but I prefer to get as close to 1ms as possible.)

Okay, fair enough. Soft realtime then.

To date I have achieved this by either making use of 3rd party libraries (e.g. RtAudio) or by re-engineering Qt or other audio streaming libraries to service the individual platforms. This works in concert with my synth class as a subclass of QThread with TimeCriticalPriority.

The priority change is so the scheduler doesn't stop you mid-way?

Looking at the audiooutput example, this suffers glitches every time you interact with the UI. (NB: Although the screenshot they provide doesn't show it, the example actually includes a volume control slider.) I replaced the Generator class that came with this with my own synth class that makes smooth transitions upon any change requests (e.g. move the volume slider). Nonetheless, running this in the main thread results in an audio glitch for every mouse press or slider movement. Even a click on the slider without changing the value.

Is it possible you get a burst of events that is pushing your important work to the side? I'd be rather suprised if a single event takes more than ms to be processed. Could you perhaps check this?

I suspect the only solution is to move the audio processes to a TimeCriticalPriority thread as before, but as this thread shows, I am struggling to see how this can work with QtMultimedia.

I have to look at the actual code, but it may be a couple of days before that happens. In the mean time, if there's no UI interaction you can hear the sound playing fine? If that is so, injecting a single synthetic UI event breaks it? Or does the sequence of UI events actually cause the glitch?

One approach I am starting to consider is to run my synth class in a time-critical thread using its own timer rather than a purely pull approach, and using a thread-safe buffer so that QAudioSink can run in the main thread.

Well, that's the thing. At least from your explanation it isn't clear to me if the problem is that the "pull a sample" or w/e it is doing is lagging due to other events interfering, or perhaps that the UI events are too slow to be processed. If it's the former, well we can think mitigation strategies, if it's the latter, that'd be worse.

However I've already tested a situation where readData() does nothing, so it glitches even when there's zero overhead from my synth class. This leaves me skeptical that QAudioSink can run in the main thread without glitches.

Possibly, I don't know. I could tell you more of an opinion after I take a look at how QtMultimedia is implemented. What platform(s) are we talking, btw?

[SGaist]

In between, wouldn't a project like PortAudio be more suitable for your application ?

[paulmasri]

@kshegunov said in How to stream audio to QAudioSink in a separate thread:

...This works in concert with my synth class as a subclass of QThread with TimeCriticalPriority.

The priority change is so the scheduler doesn't stop you mid-way?

Yes and also so that there isn't a delay in it getting called. i.e. minimise time into-through-out of readData().

Looking at the audiooutput example, this suffers glitches every time you interact with the UI. (NB: Although the screenshot they provide doesn't show it, the example actually includes a volume control slider.) I replaced the Generator class that came with this with my own synth class that makes smooth transitions upon any change requests (e.g. move the volume slider). Nonetheless, running this in the main thread results in an audio glitch for every mouse press or slider movement. Even a click on the slider without changing the value.

Is it possible you get a burst of events that is pushing your important work to the side? I'd be rather suprised if a single event takes more than ms to be processed. Could you perhaps check this?

I'm not sure how to check this.

I suspect the only solution is to move the audio processes to a TimeCriticalPriority thread as before, but as this thread shows, I am struggling to see how this can work with QtMultimedia.

I have to look at the actual code, but it may be a couple of days before that happens. In the mean time, if there's no UI interaction you can hear the sound playing fine? If that is so, injecting a single synthetic UI event breaks it? Or does the sequence of UI events actually cause the glitch?

Thanks for the offer of looking. I've been going through the source code myself. It feels more cleanly written than the old QtMultimedia. But I get the sense you're more of an expert in multithreading so your insights would be appreciated.

I'll have a think about injecting a synthetic UI event. I'm not sure how to achieve that but I'm sure I can work it out. However my suspicion is that the issue won't be the event (slot?) so much as the OS involvement in passing the event to Qt and Qt's steps to generate the signal. Worth a test though.

One approach I am starting to consider is to run my synth class in a time-critical thread using its own timer rather than a purely pull approach, and using a thread-safe buffer so that QAudioSink can run in the main thread.

Well, that's the thing. At least from your explanation it isn't clear to me if the problem is that the "pull a sample" or w/e it is doing is lagging due to other events interfering, or perhaps that the UI events are too slow to be processed. If it's the former, well we can think mitigation strategies, if it's the latter, that'd be worse.

However I've already tested a situation where readData() does nothing, so it glitches even when there's zero overhead from my synth class. This leaves me skeptical that QAudioSink can run in the main thread without glitches.

Possibly, I don't know. I could tell you more of an opinion after I take a look at how QtMultimedia is implemented. What platform(s) are we talking, btw?

At this point Windows 10 (UWP for Microsoft Store and also executable for direct download) and iPad. In future, I may be adding Android & WebGL.

@SGaist said in How to stream audio to QAudioSink in a separate thread:

In between, wouldn't a project like PortAudio be more suitable for your application ?

When I first created the app 5 years ago I looked at various libraries including PortAudio. In the end I went with RtAudio. I don't recall why. However neither library supports mobile platforms or WebGL. Hence my wish to make use of QtMultimedia as a regularly updated, fully cross-platform library.

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

Yes and also so that there isn't a delay in it getting called. i.e. minimise time into-through-out of readData().

This doesn't sound quite right. I'd speculate that the latency is due to event (re)ordering. If you take a simple imperatively run thread, given that there's nothing much happening on the system, the OS scheduler is going to happily allocate you 100% of the CPU time. I'd speculate that you observe a latency more as a consequence of the way the events are ordered/processed (I'm not claiming proof, just a feeling).

I'm not sure how to check this.

Well, firstly, if you run the application without any UI interaction(s), does it play smoothly? And secondly, if you for example post few regular events from the code (QCoreApplication::postEvent) does it break? What about sending them instead (QCoreApplication::sendEvent)?

I'll have a think about injecting a synthetic UI event. I'm not sure how to achieve that but I'm sure I can work it out. However my suspicion is that the issue won't be the event (slot?) so much as the OS involvement in passing the event to Qt and Qt's steps to generate the signal. Worth a test though.

You could take some inspiration from Qt's own testing library, the UI autotests do simulate clicks, resizes and such. If synthetic events don't produce the glitch, then I have some idea where you could dig up next. In a nutshell I'm wondering if the call into the backend is the problem, or how Qt processes the events, or the speed with which the events are processed. These are the obvious sources for what you observe, from where I'm standing.

But I get the sense you're more of an expert in multithreading so your insights would be appreciated.

Ha, I doubt it you could call me an expert. I'm a lowly physicist. ;)

@SGaist said in How to stream audio to QAudioSink in a separate thread:
In between, wouldn't a project like PortAudio be more suitable for your application ?

Also assuming this problem is indeed confirmed as described, I'd say QtMultimedia is of very limited utility ... so I'd say worth investigating, right?

[paulmasri]

@kshegunov said in How to stream audio to QAudioSink in a separate thread:

Well, firstly, if you run the application without any UI interaction(s), does it play smoothly? And secondly, if you for example post few regular events from the code (QCoreApplication::postEvent) does it break? What about sending them instead (QCoreApplication::sendEvent)?

I set up a timer and first of all tested whether the timer events caused any issues — they don't. Then I tried both postEvent & sendEvent with a synthetic mouse event (alternating between press and release) on a point along the slider. In both cases I get an audio glitch upon almost every MouseButtonPressed, just as when it was a real UI event.

Does this tell you anything useful?

[kshegunov]

Sorry for the delay, I was traveling.

Does this tell you anything useful?

That's some interesting findings. Maybe. I'd suspect in that case that the problem is the time it takes to do the paint/GUI events, not so much events in general. That's rather unfortunate as I'm unsure that this can be really fixed.

[paulmasri]

Yeah :-(
I included logging and included time elapsed. This showed that typically the time between calls to readData() for a long buffer (10ms) is typically 1-2ms, with occasional gaps 3-5ms. However every real/simulated GUI interaction results in a gap of 10-30ms.

Nevertheless I do like the QtMultimedia library for its cross-platform support, so unless someone has a great idea how to solve this situation, I'm going to fork the library and see if I can make QAudioSink and related classes thread-safe, so that I can put them in a high priority thread.

I've not tried building any part of Qt from source before, so I'm already seeking support in the forums. Hopefully I'll be able to get somewhere with it, and who knows, maybe even contribute something back.

[kshegunov]

@paulmasri said in How to stream audio to QAudioSink in a separate thread:

Nevertheless I do like the QtMultimedia library for its cross-platform support, so unless someone has a great idea how to solve this situation, I'm going to fork the library and see if I can make QAudioSink and related classes thread-safe, so that I can put them in a high priority thread.

What I was thinking of initially (hence the million questions game) was to drive the event loop manually with some timeout that should be okay for your application, so you control how long events are processed. I know sounds like an abomination, but should approximate what you want. Although from what you'd observed I'm utterly unconvinced this is going to truly work. It's probably worth a shot still, but a long one.

[paulmasri]

@kshegunov said in How to stream audio to QAudioSink in a separate thread:

What I was thinking of initially (hence the million questions game) was to drive the event loop manually with some timeout that should be okay for your application, so you control how long events are processed. I know sounds like an abomination, but should approximate what you want. Although from what you'd observed I'm utterly unconvinced this is going to truly work. It's probably worth a shot still, but a long one.

I agree it does sound an abomination! Aside of feeling wrong — messing with something unrelated to audio streaming in order to solve audio streaming issues — I'm doubtful it would work. As it says in the documentation, any time I would call processEvents(), it will process all queued events "however long it takes." This seems guaranteed to perpetuate the current issues.

I'm currently working through the audio streaming classes to understand them and see if I can work with them in some way, ideally to improve QtMultimedia and submit a pull request, but otherwise to pull them out of QtMultimedia and make use of them somehow myself.

[tomtomtomtomtomtom]

@paulmasri
Hi, I'm encountering the exact same issue with Qt 6.5.1. Have you found a solution yet?