Forget the 320x240 resolution for now (sorry that I mentioned in the first place), but try to get at least 800x600 to work with your monitor.

You could try to change your 800x600 mode to another preset by editing the mode and selecting different 800x600 entries from the list on the Edit Mode window. VESA modes are more likely to work, and 800x600 @ 60Hz would be the first I'd try.

If none of the preset modes do work, then refer your monitor specifications (google your monitor manufacturer + model + "specs") that which resolutions in which frequencies it should work, and adjust the sliders accordingly until you get it to work.

Or try with that another monitor if you like.