By "iPod classic 3G (160GB)" I am assuming you mean the slim 160gb ipod classic.
Are you saying you restored it with iTunes and then in original firmware mode it works and will accept sync of music in OF mode? So it is working hardware-wise?
The rockbox utility will say there isn't a bootloader because the classic, as far as I understand it, is not yet supported by the rockbox utility. It never has been and, again, assuming I'm not out-of-date, the special iPod Classic version of it still hasn't been officially released.
You either have to use the old emcore method (not recommended) or go find the special version of rockbox utility from freemyipod (it's still there somewhere, been linked to a few times from this forum and the head-fi thread).
Alternatively, could be what cereal_killer said - if it doesn't work fully with the OF, then it's clearly a hardware issue.