Actually, most of the time spent loading the Apple firmware is the Apple loader that runs before the Rockbox bootloader.
The trick is that the iPod normally doesn't shut down, it just sleeps (it's running, but in a very low power mode). This is why it seems to load so quickly, but if you just have the Apple software on your iPod, and hold Menu+Select to reboot, you'll notice it still takes a decent bit of time to start up.
If you remove the Apple software entirely, on the other hand, you'll notice that Rockbox can be loaded noticeable more quickly.