Apart from the fact that the 'MP Enhance' algorithms will more than likely be patented/protected/copyrighted etc., what exactly does it do?
If you can genuinely hear compression artifacts then no amount of tinkering will reduce them. This may be caused by Rockbox having a more accurate reproduction algorithms than the OF, but that seems unlikely. The only true solution to this problem is to use a better compression quality setting when encoding your music.
Alternatively, You may be hearing clipping instead, which could be caused by Rockbox if you have used too much treble/bass/EQ gain without reducing the overall signal level using the pre-cut.