Was your X800 genuine Apple one (AGP Pro connector with additional power connector, DVI and ADC ports present in the backplate, blue color, no Molex for additional power) or an Apple-reflashed PC one? That would explain some problems. If it's genuine then maybe test it under OSX? Maybe it's the card that's faulty?

I had once a properly reflashed X800 XT in my G5 and it gave me no problems whatsoever. Still waiting for my Apple ver X800 to try it for myself though.