I figured out my issue.
I set the following environment variable:
QT_QPA_EGLFS_DISABLE_INPUT=1
Of course, the cursor dissapears, but when pressing around my Gui running on the Compositor I'm not noticing any buttons being pressed except when I press in them, so that's definitely preferred to a wonky cursor!