The answer to the general question of whether it's possible to install a new firmware to a device that uses MTP is "yes". Actually producing a new firmware that works with your hardware is a separate matter that will involve a lot of work.
If, after reading the link that karashata posted, you feel you are up to the task of porting Rockbox then you can create a topic for this player in the New Ports forum if one doesn't exist already.