press menu plus select to reboot
Do this. If it doesn't work, flip the hold switch on and then off again, and do it again. Make sure that you hold the menu and select buttons without pressing any others. You may need to hold them for up to 30 seconds.
As soon as the player reboots, press Menu and Select simultaneously until the player enters disc mode. Then, connect the player to your computer and finish the installation. (Refer to section 2.2.3 of the Rockbox Manual.)