Okay, I'm searching in the wrong place firstly and secondly the problem is a different one. It has nothing to do with RAM allocation (well, not directly anyway)... The short answer is to bump up the "Max files in browser" setting, explanation is the following:
Normally if the database finds more matching entries than "max files in browse", it simply refuses to sort the list and seems to do nothing else (except setting dirfull = true, but I did not find anything that actually uses that var), so the error message "Dir buffer is full" is not displayed and things seem normal (this only applies to tagcache b/c as far as I can see, dir cache behaves differently). But when the tagnavi format specifies something like %sort and %limit, the database complains because those clauses is not possible without sorting the list first, so the error message is thrown and no entries is retrieved.
My C is rusty at the best, so correct me if I'm mistaken. I can't test my theory right now since I just mass renamed my music files and thus lost all my runtimedb data that originally caused the problem when I use them in tagnavi.
My complain is that the error message is very misleading and unhelpful
