I think a good idea would be an option where the gfx system avoids VRAM to RAM (but not RAM to VRAM) copyings altogether. What this means is that basically all bitmaps (especially window bitmaps) will be in RAM (and all gfx operations in it software rendered). This bitmap buffer in RAM will never go away. At any possible point the gfx system may "attach" a real VRAM bitmap to the RAM bitmap. The two bitmaps are kept in sync (maybe in realtime by repeating gfx operations done to RAM bitmap also in VRAM bimap connected to it. Or maybe only at specific times using things like damage list to keep track of what areas of VRAM bitmap are not yet in sync). Whenever a window bitmap needs to be composited to the screen the gfx system makes sure that real VRAM bitmap is attached so it can do the compositing hw accelerated. If system is low on free VRAM it can kill any other VRAM bitmaps connected to RAM bitmaps but it does not need to do the VRAM to RAM swapping as the RAM bitmap is always there and up to date.

Doing it like this causes some speed loss but I think this is how other OSes avoid described problems on systems with low VRAM.