There is no way to remove silence that is encoded into files, but if you don't encode silence, none will be added. The reason there is no way to remove it is that the files are streamed from disk and decoded in real-time, so there is no way to look ahead and remove silence.
FWIW, if you have CDs with silent transitions between tracks, lots of software (e.g. foobar) can remove them while encoding to MP3/etc.