LambdaCalculus379 and b0hoon thanks for your reply!
I read all this past week:
http://www.rockbox.org/wiki/IriverH10PortDevInfohttp://www.rockbox.org/wiki/SamsungYH92xPorthttp://www.rockbox.org/wiki/PackardBellVibe500Porthttp://daniel.haxx.se/sansa/mi4.htmlhttp://www.rockbox.org/wiki/PortalPlayerhttp://www.rockbox.org/wiki/PortalPlayer502xAnd I think got the bootloader (sort of) working, let me explain you how I made it.
As saratoga pointed me in IRC I should be looking for a bootrom, but the system directory of my player only contained the update.exe.
Looked in the packard bell site for the last update but it seems to be unsupported, all I found was the link to the disassembly guide, I could browse the ftp just fine but with the client I was using I couldn't list directories recursively, all the directories were named "/pub/itemnr/7045080000/" and the like, with no idea of what you were heading for, Archive.org didn't help here as the old site redirected to the new site.
As I didn't want to completely mirror the ftp for just one file I took a look into the .ini file of the update program located in the system directory of my player here's it's content:
[UpdateVersion]
Firmware=01.00.06
[UpdateFilename]
Bootloader=BL_pp6005_5020_color.rom
Firmware=PP5020.mi4
Font=none
[PIDAndVIDOfPlayer]
VID=Portable
PID=Digital Player
[PID-Version Mapping]
Digital Player=01
Notice how the version reads: "Firmware=01.00.06" while the installed version is 01.00.16 ()
So I googled for "BL_pp6005_5020_color.rom"
And found this russian msi forum,
http://forum-ru.msi.com/viewtopic.php?f=20&t=6865&start=90&st=0&sk=t&sd=a My russian knowledge is next to none, but I could see they're talking about the msi mega 536, well it seems that the msi mega 536 and the vibe 300 are quite similar, similar case, similar if not same board (Same location of the sd slot, reset switch), same screen resolution,etc... (But different hd size).
So I decided to download the the firmware update of the msi mega 536, and it came with the BL_pp6005_5020_color.rom file, so all I did was keyscan with the msi bootrom and the msi mi4 file and got a key, then I tried the msi bootrom and the vibe 300 mi4... and got the same key.
I decrypted both mi4 files and both verified as ok.
After reading all the documentation and setting up the build enviroment I compiled the bootloader of the vibe500 but changed the corresponding keys in the tea keytable on the file bootloader\main.pp from the rockbox source.
I also built rockbox for the vibe 500 just to get the hang about building rockbox, tried decrypting the rockbox os with the vibe 500 bootrom key and the encrypting it with the msi mega 536 key, and adding the dummy signature later, I tried many things I don't remember right now but sometimes the player would identify itself as "New hardwared found - rockbox player" when plugged in under windows, I think that was with the vibe 500 rockbox os image signed with the vibe300/msi key and placed in /system directory as if it were the original firmware (without rockbox bootloader).
The rockbox bootloader I have is able to load the original firmware named OF.mi4 in the /system directory it seems to fail when loading the mi4 rockbox os from the .rockbox directory and default to of.
The vibe 300 loads the msi mega 536 firmware just fine, but it seems the touchbuttons controller is different as I can only use 4 of the 6 buttons, I can use up,down,left and right but back and accept are not detected (i can also use the record button and power-on/off).
b0hoon the packard bell ftp isn't working for me right now I guess they're working on maintenance duties i'll check the original firmware just to make sure it uses the same rom or a different rom with the same key, but i think I got it correct.
*EDIT* yes it's working, there's something wrong about this forum and ftp links, it adds the http:// prefix!, the thing is the bootrom of the firmware update you posted produces the same key as the msi bootrom but the mi4 file itself doesn't load in my player, could it be caused for having a different bootrom flashed? The firmware I have is labeled 1.00.06 in the update program and 1.00.16 in the menu of the firmare itself, the one you linked is labeled 01.00.03.
There's one think I don't understand from the buildsystem, at which point the mi4 rockbox os image is signed with the corresponding key, or if it's needed at all since you load from the bootloader and you have the control of the player at that point.
Well, I'm not sure the bootloader is fully working because if I place the OF.mi4 renamed as rockbox.mi4 in the .rockbox directory it fails and defaults to the OF in the /system directory, as I stated befor I don't understand fully how everything is signed don't know if they're interchangeable at all, all I can state is that it loads the OF.mi4 from the /system directory.
The Of.mi4 file has 01030100 at the corresponding offset, and loads just fine with the dummy signature (md5 changes after signing).
I guess the next step would be disassembling both the bootloader rom and the original firmware, but I have no idea what exactly I should be looking for plus I have no experience with ARM asm, I know x86 and mips assembly I hope arm is easy to pick up.
I'll post again when I find something useful or need some help.