Use the Digilite configuration program to set TEST MODE to "IN PHASE"
Make sure the symbol rate matches the Nyquist filter frequency, (4000K typ)
Use an oscilloscope to set the I drive level at IC3 pin 7. Adjust Level pot R23 and Bias pot R29 for a 2 volt peak to peak sine wave signal swinging from 1.4 VDC to 3.4 VDC.
Set the Q drive level at IC3 pin1 using Level pot R16 and Bias pot R30 for the same waveform.
You will likely have a passable signal at this point. To finish the setup connect a spectrum analyzer and tweak the Bias pots to null out the carrier.