I have an ATI Radeon 5450 HD graphics card of 256 MB of memory I think.The problem comes that when I try to run some games or emulators like MAME it requires a 320x240 screenmode it gets out of range (no signal or similar).
Since my MorphOS systems in regular all run on internal Apple displays (PowerBook/iMac) and these have their own caveats at non native resolution I can only do some guesswork.
Open Prefs->Monitors See if there is a 320x240 mode listed -> test it If not try to create one by cloning an existing mode and changing it's values -> test it
If it works something else is wrong in MAME.
If it doesn't fiddle with the setting till it does. Screen refresh should always be close to 60Hz and main frequency at 31kHz or higher. You may try to set "Doublescan" to achieve this.
Just for fun I tried it on the iMac and changing an existing 1680x1050 mode to 320x240 did work in "test". No idea if MAME would accept it.