This post doesn't give a lot of information to work with, but if I make a couple of assumptions I believe I can ascertain what's going on here.
You say the car stereo head unit works with {an|your} iPod (the assumption I'm making here is that you mean "works with the original firmware").
You also state that it does not work when said iPod is running Rockbox.
You also assert that the head unit is able to read from generic USB mass storage devices, but not the iPod using Rockbox (see above).
From this, I can gather that the head unit is attempting to communicate with the iPod using IAP (iPod Accessory Protocol - not In App Purchasing

).
This is not uncommon at all and perfectly understandable.
The head unit "sees" that the device is an iPod (by using USB vendor and product ID codes), and assumes it can communicate with it using the iPod Accessory Protocol (which is partially supported in Rockbox but very, very, very far from complete). This is a valid assumption, in their world, alternative firmware for this device doesn't exist.
There's potentially a couple of ways around this, depending on how the head unit behaves:
You may find that if you force Apple's Disk Mode, things will "Just Work(TM)".
Failing that, the alternative solution is quite a bit more in depth and would require you to compile your own modified Rockbox binary. The solution here would be to change the USB VID/PID codes that Rockbox uses (we recycle the VID/PID from the original hardware) so that the head unit can no longer see the device is an iPod.
Regarding USB vs. 3.5mm, it is incredibly likely that the sound system and/or source material will negate any potential improvement USB may offer.
[Saint]