If you have non-ASCII characters in your filenames you'll run into problems as Rockbox stores playlists in utf8 encoding. Windows (and AFAIK the OF) uses a local encoding (in europe, for example, it's latin9). So all those characters will be "messed up" and thus not play. Also, Rockbox uses a "/" as directory delimiter (which is standard in the unix world) rather than this stupid "\" windows uses. The iriver OF uses the windows' "\" and it thus incompatible. Please read the documentation for the plugin "iriverify".
Also, as Llorean said, experimental builds are unsupported. Try reproducing the problem with an official build. If it still persists please post one of those playlists and provide more detailed information.