I suspect that the serialiser might not be sending the correction signals in the right way.
/I and /Q should occur slightly ahead of I and Q.
The result is that an IQ plot shows a spread of points and the spectrum appears to have excessive spurs which I can't cancel.
IQ Plot showing the Output after the Digilite is turned on ( presumably a PRBS)

IQ Plot Showing USB Mode

As far as I can understand from the F4DAY original article - in which he shows a system clock of 8.192Mhz , and a symbol rate of 1.024MBaud - I think the correction signal should occur one quarter of the symbol time ahead of the main signal ( if that makes sense ).
I also note that the serialiser appears 'bursty' - i.e. if I set it to output USB only - it occasionally sends an unmodulated carrier. ( approximately every 3 or 4 seconds ).
Anyone dispute / agree ?