As I said on IRC, my hypothesis is that the ipod wasn't put together entirely right after replacing the battery.
We've seen this exact problem on ipods mini, in those cases caused by the hard drive being replaced by a compactflash card that's thinner, with the effect that the button backplane didn't have enough support. It could in theory also be caused by the cable from the button assembly not being connected well enough.
The reason these things can behave differently in rockbox and the OF is that buttons aren't (as you might naively expect) read out as on or off. They're typically all connected together in a resistor network, so you measure *one* voltage and deduce the state of several buttons from it. This being an analog measurement, there are going to be variations, and the software reading this will have to use a range of allowable values when interpreting it. This range isn't necessarily the same in the OF or rockbox (nor can it necessarily be. One reason rockbox can often exceed OF battery life is by being more creative with the voltage going *into* various systems, which obviously implies that the voltages coming out will be different).
A slightly loose connection somewhere could easily change the measured values a bit.
What you call "reboot" is actually the hard reset (rebooting rockbox properly is done by holding the play button for a few seconds to cleanly shutdown the device and then powering it on again). The hard reset feature is implemented entirely in hardware (in the power management chip), so the fact that that works while other functions of the select button don't also isn't surprising.