Ok, finally I found a way how to create .ra/.rm atrac3 files.
The ffmpeg is not working. It can't encode to atrac3 but it can decode it or you can change container from native atrac extension .oma to .rm or .wav (ffmpeg -i test.oma -acodec copy test.rm) but after that you can't read those files with anything, so it's not working well.
The only way I found is to use my modified version of Helix DNA(TM) Producer : )http://rapidshare.com/files/455814907/real_producer.zip
This is a version 184.108.40.2063 with older (220.127.116.116) version of atrc.dll (the new version is not working correctly). I also removed unnecessary dll files.
With it you can make 66-352kbps Atrac3 .rm files compatible with Rockbox!
This time I am sure that I am encoding to atrac - the VLC said: "Codec: atrc".
In my zip file you can find "Convert2atrac.bat" that include:
producer -i input.wav -ad profile.rpad -dt -o output.rm
instead of input wav format you can use other format, like mp3, wma etc..
In "profile.rpad" you can change properties of the codec by changing line 10:
codecFlavor represent bitrate:
codecFlavor 0 = 66kbps (LP4)
codecFlavor 1 = 94kbps
codecFlavor 2 = 105kbps (LP2)
codecFlavor 3 = 132kbps (LP2)
codecFlavor 4 = 146kbps
codecFlavor 5 = 176kbps
codecFlavor 6 = 264kbps
codecFlavor 7 = 352kbps
I don't know why, but the encoder is crashing in Flavor 2-5.
I was able to play files in flavor 0, 1, 5, 6, 7 on my iPod (rbox r22903-091003).
I was not able to test flavor 2, 3, 4 because of encoder crashing.
But there is something wrong in sound quality ; /
While VLC is playing files clearly the Rockbox is making wierd noise in sound background.
I've recorded this from my iPod: http://rapidshare.com/files/288335532/ra_output.zip
in flavor0_output.ogg you can hear loud distorsion. (from Flavor 0 = 66kbps .rm)
in flavor7_output.ogg you can hear a quiet distortion in background like from high voltage device (from Flavor 7 = 352kbps .rm)
Also I think that CPU usage is very high, because everything seemed to work slow (like changing volume) while playing that files on my iPod.
Here you've got short .rm test files that I've encoded in flavor 0, 1, 5, 6, 7:http://rapidshare.com/files/288335045/atrac3ra.zip
Here you can find unmodified versions of Helix DNA Producer: https://helix-producer.helixcommunity.org/Downloads