FYI there were issues with the first batches of 002 rack and mixer. These manifested themselves in a variety of fun ways, and I would not be at all surprised if your audio quality issue was hardware related, even though it seems to be a software issue.
I have been using an LE/ 002 rack system recently with G5 and it sounds great. Bounce to disk in real time sounds perfect to me. 24 bit, no dither. I can't tell the difference between 32 bit float and 24 bit in PT. I certainly can in Cubase however!