Normally that should work if you just do what it says - reboot it manually with centre/menu, then it reboots and remounts and the installation continues. Not sure what you mean by "something about formatting it". Was that a windows message? Possibly windows got confused about the device and asked to reformat it. Should probably have just told it not to do so, and then let the rockboxutility get on with doing its thing. Sounds as if maybe you let windows do what it wanted to do and reformat the ipod, which would have interfered with what rockbox utility was doing. You might have to start the whole process from the beginning, or maybe you could just copy the rockbox directory onto the ipod manually.
Also, is this an unmodded 6g or an iflash mod? Latter could have all sorts of other complications going on.