There is no Rockbox installer, so whatever software you used wasn't part of the Rockbox project, but written by a third-party. You should ask that person for help.
But the general advice is: hold MENU+SELECT to reboot, then immediately hold SELECT+PLAY to force your ipod into disk mode, then connect to your computer, and pray that itunes will recognise it and restore.