As I understand it, this is not a hardware problem, and not a firmware or software problem with either Rockbox or Apple code...
In fact it is not a problem at all.
It is simply something you have failed to notice until now.
The border you describe around the screens edge is totally normal, it will be there whether or not you use Rockbox or Apple firmware. It is only possible to view this when the LCD is powered on, whilst off, the viewable area of the LCD appears larger than it actually is.
I respect that you believe it is a problem, and I imagine it worries you...however, there is nothing at all to worry about. It is simply an error in perception that has lead you to believe that something has changed when in fact it hasn't.
Installing Rockbox's firmware most likely opened your perceptions up to change, and you noticed something you hadn't noticed before...it was quite natural to assume that it was some fault of Rockbox's or something triggered by the installation. However this is not the case.
If you happen to have another iPod handy (I'm looking at all mine now: 6 Nano 1st Gen, 2 Nano 2nd Gen, and 2 iPod 4th Gen (Photo/Colour)) you can verify this.
All my iPods have the same black border (when the LCD is powered on, it is not visible when powered off) between the actual visible screen area and the edge of the clear section in the front panel from which the screen is viewed.
Sorry that this has turned into a bit of a novel, I just wanted to assure you that you needn't worry about this, and that what you're seeing is in fact perfectly normal.
[St.]