Hardware overlay is usually a feature where you can define a rectangle. e.g 1024x768 and define a screen size and position. Now you can stream a game or video feed into the 1024x768
buffer and the gpu will auto position and auto scale the source material into the destination buffer. There is more to it, but this is the main feature.

This is usually combined with a window. MPlayer is using this feature. The advantage is that you just need to decode the video stream natively into the buffer and the graphics card does the rest.

More modern cards lack this feature as they are so powerful, so they can decode and scale the streams on the fly anyway. And the overlay feature got removed.

Do not hold you breath on specific graphic card or 3D support.

For UAE/WHDLoad even the smallest Radeon will work, so for a G5 just get one that works right now and done.