The file should be streaming friendly, so that all file metadata is found before the audio data. In the mpeg4ip project there is a program, mp4creator, that can perform the operation (use the -optimize option). However, that wasn't enough for playback of the one file I've tested it on. Don't know why yet, as I haven't had the time to look closer at it.
Foobar2000 could possibly fix it too. At least version 0.9.4, with the proper option set.