Rockbox Technical Forums

Support and General Use => Audio Playback, Database and Playlists => Topic started by: ilmioalias on February 24, 2022, 03:26:59 PM

Title: Problem filtering by year
Post by: ilmioalias on February 24, 2022, 03:26:59 PM
Hello,
i'm trying to filter songs by year subsequently by album; i created this custom tagnavi (only a snippet):
Code: [Select]
%menu_start "custom_artist_initial" "Browse album artists"
"Non-alphabetic"   -> albumartist ? albumartist < "A" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"A"   -> albumartist ? albumartist ^ "A" | albumartist ^ "The A" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"B"   -> albumartist ? albumartist ^ "B" | albumartist ^ "The B" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"C"   -> albumartist ? albumartist ^ "C" | albumartist ^ "The C" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"D"   -> albumartist ? albumartist ^ "D" | albumartist ^ "The D" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"E"   -> albumartist ? albumartist ^ "E" | albumartist ^ "The E" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"F"   -> albumartist ? albumartist ^ "F" | albumartist ^ "The F" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"G"   -> albumartist ? albumartist ^ "G" | albumartist ^ "The G" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"H"   -> albumartist ? albumartist ^ "H" | albumartist ^ "The H" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"I"   -> albumartist ? albumartist ^ "I" | albumartist ^ "The I" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"J"   -> albumartist ? albumartist ^ "J" | albumartist ^ "The J" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"K"   -> albumartist ? albumartist ^ "K" | albumartist ^ "The K" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"L"   -> albumartist ? albumartist ^ "L" | albumartist ^ "The L" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"M"   -> albumartist ? albumartist ^ "M" | albumartist ^ "The M" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"N"   -> albumartist ? albumartist ^ "N" | albumartist ^ "The N" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"O"   -> albumartist ? albumartist ^ "O" | albumartist ^ "The O" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"P"   -> albumartist ? albumartist ^ "P" | albumartist ^ "The P" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"Q"   -> albumartist ? albumartist ^ "Q" | albumartist ^ "The Q" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"R"   -> albumartist ? albumartist ^ "R" | albumartist ^ "The R" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"S"   -> albumartist ? albumartist ^ "S" | albumartist ^ "The S" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"T"   -> albumartist ? albumartist ^ "T" & albumartist !^ "The A" & albumartist !^ "The B" & albumartist !^ "The C" & albumartist !^ "The D" & albumartist !^ "The E" & albumartist !^ "The F" & albumartist !^ "The G" & albumartist !^ "The H" & albumartist !^ "The I" & albumartist !^ "The J" & albumartist !^ "The K" & albumartist !^ "The L" & albumartist !^ "The M" & albumartist !^ "The N" & albumartist !^ "The O" & albumartist !^ "The P" & albumartist !^ "The Q" & albumartist !^ "The R" & albumartist !^ "The S" & albumartist !^ "The U" & albumartist !^ "The V" & albumartist !^ "The W" & albumartist !^ "The X" & albumartist !^ "The Y" & albumartist !^ "The Z" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"U"   -> albumartist ? albumartist ^ "U" | albumartist ^ "The U" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"V"   -> albumartist ? albumartist ^ "V" | albumartist ^ "The V" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"W"   -> albumartist ? albumartist ^ "W" | albumartist ^ "The W" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"X"   -> albumartist ? albumartist ^ "X" | albumartist ^ "The X" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"Y"   -> albumartist ? albumartist ^ "Y" | albumartist ^ "The Y" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
"Z"   -> albumartist ? albumartist ^ "Z" | albumartist ^ "The Z" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"

the problem is albums are not filtered by year; whichever year i select, in the subseguent directory there are all albums of the artist

for example:
big thief ->
    2022 ->
        U.F.O.F.
        Dragon New Warm Mountain I Believe in You

but only the last one is from 2022.

What i'm doing wrong?
Thanks in advance

EDIT:
maybe it's important: i use a FiiO M3K with last dev builds installation of rockbox.
Title: Re: Problem filtering by year
Post by: Bilgus on February 25, 2022, 10:06:18 AM
I can reproduce the bug but I haven't narrowed down why yet
Title: Re: Problem filtering by year
Post by: ilmioalias on February 25, 2022, 10:09:48 AM
OK, thanks.
Shall i open a bug?
Title: Re: Problem filtering by year
Post by: Bilgus on February 25, 2022, 05:11:04 PM
Well the issue appears to be with the logical OR
"L"   -> albumartist ? albumartist ^ "L" -> year -> album -> title = "fmt_title" Works

I'll try to figure out the issue but the tagcache is a big ole mess
Title: Re: Problem filtering by year
Post by: Bilgus on February 25, 2022, 10:06:28 PM
I have a patch that fixes this https://gerrit.rockbox.org/r/c/rockbox/+/4248 Tagcache Don't create filters when parsing a logical OR

https://www.mediafire.com/file/12kxvxrq2qdlacn/FioM3kBaremetal_tagcache_bugfix4248_rockbox-full.zip/file

Should be in latest dev builds tomorrow..
Title: Re: Problem filtering by year
Post by: ilmioalias on February 26, 2022, 07:21:49 AM
I have a patch that fixes this https://gerrit.rockbox.org/r/c/rockbox/+/4248 Tagcache Don't create filters when parsing a logical OR

https://www.mediafire.com/file/12kxvxrq2qdlacn/FioM3kBaremetal_tagcache_bugfix4248_rockbox-full.zip/file

Should be in latest dev builds tomorrow..

Thanks, it works!
Title: Re: Problem filtering by year
Post by: Bilgus on February 26, 2022, 08:03:00 AM
Good, thanks for testing
Title: Re: Problem filtering by year
Post by: ilmioalias on March 17, 2022, 05:39:39 PM
Good, thanks for testing

Unfortunately, the problem reappeared, but only for some combination of letters; for example some artists show the bug, but others don't
Title: Re: Problem filtering by year
Post by: Bilgus on March 17, 2022, 07:59:58 PM
could it possibly be a formatting error for some artists? could you try the ones that work and rename them to names that don't?
Title: Re: Problem filtering by year
Post by: ilmioalias on March 18, 2022, 01:47:15 PM
could it possibly be a formatting error for some artists? could you try the ones that work and rename them to names that don't?

You are right, there is a problem with artists names that start with "the"; in this case filtering doesn't work.
Title: Re: Problem filtering by year
Post by: Bilgus on March 18, 2022, 05:57:15 PM
looking at your original sample above I'd say likely the really long line is blowing the stack
but really IDK, the tagcache code is complicated
Title: Re: Problem filtering by year
Post by: amachronic on March 18, 2022, 08:12:04 PM
The problem here is how the tagtree constructs database queries. Due to the way the tagcache evaluates queries it can only deal with boolean expressions in "disjunctive normal form", which have this general form:
Code: [Select]
(a0 && ... && aN) || (b0 && .. && bN) || ...
or at least, "clauses" are dealt with in this fashion. This is further complicated by "filters" which (I think) become an additional AND at the outermost level but idk.

Each "level" of the menu in the tagtree corresponds to a new query (so albumartist -> year -> album -> title has four levels). When you are viewing the albums under a certain year you're at the 3rd level, and the tagtree will add clauses and filters corresponding to your selections at the 1st and 2nd levels:
Code: [Select]
FILTERS
  albumartist = "The A.."
CLAUSES
  year = 2005
(For some reason the tagcache cannot "filter" on numeric tags, so the year becomes a clause.)

Now to ensure it only finds stuff matching your actual query, the tagtree will append the entire conditional from the tagnavi file:
Code: [Select]
FILTERS
  albumartist = "The A..."
CLAUSES
  year = 2005 & albumartist ^ "A" | albumartist ^ "The A"
Now it is easy to see why the year gets ignored: the AND applies only to the left side of the disjunction and the right side is always true because we selected an albumartist that begins with "The A" already. If we selected an albumartist beginning with "A" then the query would work correctly because the right hand side would be false, forcing the year check to be taken into account. If you swap the order of the terms in the tagnavi file, like this:
Code: [Select]
"A"   -> albumartist ? albumartist ^ "The A" | albumartist ^ "A" -> year -> album  ->  title = "f_browse"
then the tagtree's final clause will end up being
Code: [Select]
  year = 2005 & albumartist ^ "The A" | albumartist ^ "A"
which make the menu work properly for albumartists beginning with "The A" but break all albumartists beginning with "A".

It's not too hard to fix the tagtree in theory but I won't be able to spend time on it anytime soon. Bilgus maybe you'd like to take a shot at it?
Title: Re: Problem filtering by year
Post by: Bilgus on March 19, 2022, 12:23:14 AM
Looked at it poked at it came away with:
Above my pay grade ???

Title: Re: Problem filtering by year
Post by: Bilgus on March 19, 2022, 01:07:40 AM
I think I've a solution, add a new clause operator
'@^' begins with one of

Code: [Select]
"L" -> albumartist ? albumartist @^ "L|The L" & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
Edit:
Its in https://git.rockbox.org/cgit/rockbox.git/commit?id=ddbca125a6
Title: Re: Problem filtering by year
Post by: Bilgus on March 19, 2022, 03:17:13 AM
@ilmioalias

when all is said and done could you post that script to the wiki, or even here to the forum so someone else might use
of all this time we've spent

Ps can 'T' be shortened to this?
Code: [Select]
"T"   -> albumartist ? albumartist ^ "T" & albumartist !^ "The " & filename !~ "_cmpl" -> year -> album  ->  title = "f_browse"
Title: Re: Problem filtering by year
Post by: amachronic on March 19, 2022, 07:02:58 AM
Looked at it poked at it came away with:
Above my pay grade ???
(math pays off sometimes I guess lol)

But seriously, instead of an ad-hoc operator to solve this problem (which won't address the general issue) you just need to move the "filter clauses" to a separate group.
I call them filter clauses because tagtree wants them to be filters but the tagcache can't support that.
https://github.com/Rockbox/rockbox/blob/master/apps/tagtree.c#L1486
instead of prepending them to the main clause group with tagcache_search_add_clause like tagtree is currently doing there, you can create an entirely separate clause group in the tagcache, add a new function tagcache_search_add_filter_clause to add stuff to it, and evaluate that group in addition to the main one.

I think that would mean adding another check_clauses() above the current one here
https://github.com/Rockbox/rockbox/blob/master/apps/tagcache.c#L1382
like so
Code: [Select]
            /* check filter clauses first */
            if (!check_clauses(tcs, idx, tcs->filter_clause, tcs->filter_clause_count))
                continue;
            if (!check_clauses(tcs, idx, tcs->clause, tcs->clause_count))
                continue;


That will solve the problem in this case and most simple cases. A more general solution will probably require an AST to represent the query expression and that's a lot of refactoring.
Title: Re: Problem filtering by year
Post by: Bilgus on March 19, 2022, 07:41:25 AM
Sounds to me that you have an idea, feel free to run with it.

Edit--
Filters can only be used with entries you can point at with a seek index into a db.tcd, virtual tags and number tags dont index into a tcd file AFAIU

Further I'm unsure how you decide the users intention without adding another operator or parens

Title: Re: Problem filtering by year
Post by: amachronic on March 19, 2022, 10:52:43 AM
OK, I'll put this stuff on the bugtracker then so it isn't forgotten.

re. the user's intention: right now the ANDs always bind more tightly than ORs so there is no ambiguity, it's just that tagnavi docs are very unclear about this, and it is too easy to get confused. Supporting parens would be better of course, and tie in nicely with AST-based queries.