Rockbox is a completely different software than what is currently on your iPod.
That said - Rockbox does not work on the Classic (6th generation iPod) - is that what you have? Many people (wrongly) call the 5th generation 80 GB an "80 gig classic".
Regardless, it sounds like a hardware problem - if you're lucky it is just the headphone detect switch which has gone bad. Rockbox has an option to pause on headphone removal / insertion so IF that is the problem Rockbox could ignore it.