The plot below is 333k symbols per second, 200kHz / 10dB per division.
I thought that it would be possible to reduce the 'hips' (lower than shoulders ) on the signal by manipulating the digital filter, but I have not been able to do so. I think the problem is that the DAC is being pushed too hard and it can't settle fast enough to reproduce the output signal with the accuracy required. It's working more like an 8 bit rather than a 12 bit DAC. The hips tail off rapidly because the analogue filter after the DAC that removes harmonics of the sampling rate cuts in fairly early.
The problem is that with lower symbol rates, the hips will still start at 60dB below the reference point, at a narrower bandwidth, but they will extend out to where they are for 333kS. The intention was to be able to use different symbol rates without any analogue filter changes, which should have kept the shape the same as for 333kS. I can't see any way around this with the current hardware.
The DAC7801 was the only one that met all the requirements and was large enough for homebrew. It might be possible to retrofit another DAC on a daughterboard at some point, but fitting the DAC on the daughterboard itself won't be a homebrew option.
If you decide to only use it at only one lower symbol rate, then it's possible to modify the analogue filter to give the shape below, but it does defeat the object of being able to change symbol rates and keep the same waveform shape.
Another issue is local oscillator suppression. The unit was designed to have no adjustments. The modulator data sheet gives -35dB as the minimum LO suppression without adjustment, which would be fine, but I've had to reduce the IQ drive levels to reduce the level of modulator sidebands. This reduces the modulated RF output level, but not the LO level, so the LO suppression becomes relatively worse.
It's possible to adjust the LO feedthrough suppression by applying a resistor to one of the four IQ input lines on the modulator. You can see the MER figure on Tutioune change as this is done. This is the reason for the 2M2 pot in the kit. The pot can be then be measured and replaced by a fixed resistor. 30dB MER is a very good figure. My first unit gave 15dB MER until I added a resistor and it's now improved to 29dB. My second unit is showing 26dB without any adjustment, so it's going to be different for each unit. I'll add an LO suppression procedure to the instructions.