How long are you holding the select+menu buttons for? Are you doing it while it's connected to the PC?
I assume you have tried rebooting the PC (and using task manager to terminate every iTunes-related process before restarting iTunes)?
Will it go into disk mode (hold down select+play)? If it does, could do that then either have another go with installing rockbox or restoring with iTunes.