Using boot menu can be really slow when Mac is trying to find netboot devices. Could that be your problem? Wait long enough until it stops trying it and then menu should work fast with kb and mouse.

And if you're not going to boot from network in near future, type "setenv skip-netboot? true" on OF prompt to get things quick on bootmenu always.