That sounds like only one of the phase outputs is working. I think the Arduino library still registers counts even if only one phase changes.
It may be the voltage is a bit too low. Although the spec for the encoder says 5-24V, internally it feeds into a 78M05 regulator. So really the lowest input voltage should be more like 8V to get a regulated 5V for the encoder. A low voltage could be causing one of the phases to not see the movement of the encoder disk.
Of course it might also be something simple like a shorted pin or faulty Arduino.
If you can measure the voltages on the two phase signals you should be able to see them changing between 0 and 5V.
PS I should say thanks to you for the Langstone - a great project.