You can have the desktop (Ambient) screen in 1080p/24bit no problem, but just select 800x600x8bit in the DCTelnet screen settings (and don't use it on "Workbench").

Another option is to configure DCTelnet to open on a 1920x1080x8bit screen, but then you have to set the Width and Height manually to 800 and 600 instead of the default values. Then it opens another screen in the same resolution as your desktop screen, but limits drawing area which fixes the program's issue with bigger resolutions.