The PC&Mac-Edition isn't the same card as the 9650 that shipped with 2.7GHz G5s which is known working.

I do have 256MB 9800PC&Mac which I haven't really used that much as it has mechanical issues, but when I had it in the the QS G4 I was also unable to use a 2560x1440 mode and some research suggested that it does not have a DualLink output. So it stands to reason that this might also be true for the 9600.

Why it goes down to 1280x800 on the 2nd port......

It's been a decade since I fiddled with PPC OSX (and even more for Tiger), but you might want to try this:

- select "about this Mac" from menu
- click system information
- check GFX/displays