Is that Ipod actually an Ipod 5G? How does it behave when you install Rockbox, i.e. not use the "Quick Start" tab? Do you have an antivirus program or similar running? Does it help shutting that off during installation?
I'm suspecting some other software interfering here. Rockbox Utility needs to directly access the drive when installing the bootloader, which is something an antivirus program could try to block (resulting in strange behaviour in Rockbox Utility).
Also, please post the output of the About / System Info dialog.