Since my MorphOS systems in regular all run on internal Apple displays (PowerBook/iMac) and these have their own caveats at non native resolution I can only do some guesswork.

Open Prefs->Monitors
See if there is a 320x240 mode listed -> test it
If not try to create one by cloning an existing mode and changing it's values -> test it

If it works something else is wrong in MAME.

If it doesn't fiddle with the setting till it does.
Screen refresh should always be close to 60Hz and main frequency at 31kHz or higher.
You may try to set "Doublescan" to achieve this.

Just for fun I tried it on the iMac and changing an existing 1680x1050 mode to 320x240 did work in "test".
No idea if MAME would accept it.