I like this idea too. The configurations should be probably cached in the RAM either for the whole disk (like dircache) or just for the directories which contain the currently buffered files. The second option would not be much slower because the new configurations would get loaded when filling the buffer (no extra spin up).
I use even more types of files which require different settings: podcasts, music, recorded files, language lessons, audio books etc.