DSP library compatible with QT?
Not quite the answer you requested, but I think the sanest thing(OOP wise)  to do in most of the situations is to have the backend of the application separated from the frontend.

You could create a standalone library(either static or dynamic) for your dsp code and separate it from the QT code.

This way your DSP code would be independent of the frontend you are using.
You could switch from QT to anything else(wxWidgets/GTK+ etc) and your DSP functions inside the lib would remain the same.