Random track selection

[philden]

I periodically ponder the random play options in Rockbox. When I use an iPod in original firmware, I do like the option to easily shuffle all songs.

There are options in Rockbox to select 'Random' from the Database 'Artist', 'Album' menus, and others, but not from 'Track'. As far as I can see, the current random play options involve making a non-random playlist, and then shuffling it. That is why to 'shuffle all songs', you first need to make a playlist of all the songs, and then shuffle it. This becomes unwieldy with a large library, and is limited by the maximum playlist size.

I would be happy with the ability to make a random selection of <1,000 songs from the library. I doubt that I would ever get close to playing that number of songs before I next alter the library and would make a new list.

So, I'm wondering if it would be possible to add a random playlist generator of some sort?

Thanks!

[philden]

Adding some more information, as I've been devising a workaround to make the random playlist externally, using a Mac.

I currently organize my music for Rockbox in a dedicated iTunes library, then use Carbon Copy Cloner to copy the music to the iPod (after removing embedded art and replacing with suitably sized cover.bmp files). So I found a way of making a random playlist in iTunes.

1. Make a SmartPlaylist to select a suitable number of songs, less than the Rockbox limit. This is a random selection, but listed in a sorted order. I could put this is Rockbox and then shuffle, but instead:

2. Make a second SmartPlaylist by random selection from the first, with the same number of songs.

3. Export as m3u

4. Put in a text editor and globally replace the file paths to match those on the iPod.

5. Copy to iPod and load into Rockbox.

[philden]

Answering myself again, the best answer lies here - https://forums.rockbox.org/index.php/topic,53990.0.html

However, I would still like the option of a 'real' random play. By which I mean the next track is randomly selected from the whole library, which could even mean repeating the same song. This would go against the 'playlist' principle of Rockbox, unless it is a randomly created one-song playlist.

[Bilgus]

its pretty easy to throw something together..

Code: [Select]

--[[ Lua RB Random Playlist
/***************************************************************************
 *             __________               __   ___.
 *   Open      \______   \ ____   ____ |  | _\_ |__   _______  ___
 *   Source     |       _//  _ \_/ ___\|  |/ /| __ \ /  _ \  \/  /
 *   Jukebox    |    |   (  <_> )  \___|    < | \_\ (  <_> > <  <
 *   Firmware   |____|_  /\____/ \___  >__|_ \|___  /\____/__/\_ \
 *                     \/            \/     \/    \/            \/
 * $Id$
 *
 * Copyright (C) 2020 William Wilgus
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * as published by the Free Software Foundation; either version 2
 * of the License, or (at your option) any later version.
 *
 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
 * KIND, either express or implied.
 *
 ****************************************************************************/
]]
require ("actions")
--require("dbgettags")

local scrpath = rb.current_path()
local max_entries = 500;
local CANCEL_BUTTON = rb.actions.PLA_CANCEL

local sINVALIDDATABASE = "Invalid Database"
local sERROROPENING    = "Error opening"

-- tag cache header
local sTCVERSION = string.char(0x10)
local sTCHEADER  = string.reverse("TCH" .. sTCVERSION)
local DATASZ    = 4  -- int32_t
local TCHSIZE   = 3 * DATASZ -- 3 x int32_t

local function bytesLE_n(str)
    str = str or ""
    local tbyte={str:byte(1, -1)}
    local bpos = 1
    local num  = 0
    for k = 1,#tbyte do -- (k = #t, 1, -1 for BE)
        num = num + tbyte[k] * bpos
        bpos = bpos * 256
    end
    return num
end

-- uses database files to retrieve database tags
-- adds all unique tags into a lua table
-- ftable is optional
function get_tags(filename, hstr, ftable)

    if not filename then return end
    if not ftable then ftable = {} end
    hstr = hstr or filename

    local file = io.open('/' .. filename or "", "r") --read
    if not file then rb.splash(100, sERROROPENING .. " " ..  filename) return end

    local fsz = file:seek("end")

    local posln = 0
    local tag_len = TCHSIZE
    local idx

    local function readchrs(count)
        if posln >= fsz then return nil end
        file:seek("set", posln)
        posln = posln + count
        return file:read(count)
    end

    -- check the header and get size + #entries
    local tagcache_header = readchrs(DATASZ) or ""
    local tagcache_sz = readchrs(DATASZ) or ""
    local tagcache_entries = readchrs(DATASZ) or ""

    if tagcache_header ~= sTCHEADER or
        bytesLE_n(tagcache_sz) ~= (fsz - TCHSIZE) then
        rb.splash(100, sINVALIDDATABASE .. " " .. filename)
        return
    end
   
    -- local tag_entries = bytesLE_n(tagcache_entries)

    for k, v in pairs(ftable) do ftable[k] = nil end -- clear table
    ftable[1] = hstr

    local tline = #ftable + 1
    ftable[tline] = ""

    local str = ""

    while true do
        tag_len = bytesLE_n(readchrs(DATASZ))
        readchrs(DATASZ) -- idx = bytesLE_n(readchrs(DATASZ))
        str = readchrs(tag_len) or ""
        str = string.match(str, "(%Z+)%z") -- \0 terminated string

        if str and math.random(10) > 5 then
            if ftable[tline - 1] ~= str then -- Remove dupes
                ftable[tline] = str
                tline = tline + 1
                if tline >= max_entries then break end
            end
        elseif posln >= fsz then
            break
        end

        if rb.get_plugin_action(0) == CANCEL_BUTTON then
            break
        end
    end

    file:close()

    return ftable
end -- get_tags

local function create_playlist()
    math.randomseed(rb.current_tick()); -- some kind of randomness
    local files = get_tags(rb.ROCKBOX_DIR .. "/database_4.tcd", "",nil)
    local fcount = #files
    if fcount > 1 then
        rb.audio("stop")
        rb.playlist("create", scrpath .. "/", "playback.m3u8")
    end

    for i = 2, fcount - 1 do
        repeat
            next_item = math.random(2, fcount - 1)
        until (files[next_item] ~= "")
        rb.playlist("insert_track", string.match(files[next_item], "[^;]+") or "?")
        files[next_item] = ""
    end
    if fcount > 1 then
        rb.playlist("start", 0, 0, 0)
    end

    files = nil -- empty table
end -- create_playlist

local function main()
    if not rb.file_exists(rb.ROCKBOX_DIR .. "/database_4.tcd") then
        rb.splash(rb.HZ, "Initialize Database")
        os.exit(1);
    end
    rb.splash(10, "Searching for Files..")
    collectgarbage("collect")
    create_playlist();
    collectgarbage("collect")

    rb.splash(100, "Goodbye")
end

main() -- BILGUS


problem being that for most there isn't enough ram to load all the tracks in the database
something incremental would have to be made
currently it just (pseudo) randomly grabs tracks and shuffles them randomly and spits out a playlist

as you play around with it..
math.random(10) > 5 make this something less selective for really large databases math.random(1000) > 750

you could probably make it happen in place by looping through the database multiple times with a really selective function
it could even be expanded to take the names into consideration or other database fields (see tagnav lua script)
 

[Bilgus]

Well I went ahead and made probably exactly what you want

Code: [Select]

--[[ Lua RB Random Playlist
/***************************************************************************
 *             __________               __   ___.
 *   Open      \______   \ ____   ____ |  | _\_ |__   _______  ___
 *   Source     |       _//  _ \_/ ___\|  |/ /| __ \ /  _ \  \/  /
 *   Jukebox    |    |   (  <_> )  \___|    < | \_\ (  <_> > <  <
 *   Firmware   |____|_  /\____/ \___  >__|_ \|___  /\____/__/\_ \
 *                     \/            \/     \/    \/            \/
 * $Id$
 *
 * Copyright (C) 2020 William Wilgus
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * as published by the Free Software Foundation; either version 2
 * of the License, or (at your option) any later version.
 *
 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
 * KIND, either express or implied.
 *
 ****************************************************************************/
]]
require ("actions")
--require("dbgettags")

local scrpath = rb.current_path()
local max_entries = 500;
local CANCEL_BUTTON = rb.actions.PLA_CANCEL

local sINVALIDDATABASE = "Invalid Database"
local sERROROPENING    = "Error opening"

-- tag cache header
local sTCVERSION = string.char(0x10)
local sTCHEADER  = string.reverse("TCH" .. sTCVERSION)
local DATASZ    = 4  -- int32_t
local TCHSIZE   = 3 * DATASZ -- 3 x int32_t

local function bytesLE_n(str)
    str = str or ""
    local tbyte={str:byte(1, -1)}
    local bpos = 1
    local num  = 0
    for k = 1,#tbyte do -- (k = #t, 1, -1 for BE)
        num = num + tbyte[k] * bpos
        bpos = bpos * 256
    end
    return num
end

-- uses database files to retrieve track file names
-- adds songs at random to new playlist
-- play plays the playlist when done
function create_random_playlist(filename, play)

    if not filename then return end
    if not play then play = false end

    local file = io.open('/' .. filename or "", "r") --read
    if not file then rb.splash(100, sERROROPENING .. " " ..  filename) return end

    local fsz = file:seek("end")

    rb.audio("stop")
    rb.playlist("create", scrpath .. "/", "random_playback.m3u8")

    local posln = 0
    local tag_len = TCHSIZE
    local idx

    local function readchrs(count)
        if posln >= fsz then return nil end
        file:seek("set", posln)
        posln = posln + count
        return file:read(count)
    end

    -- check the header and get size + #entries
    local tagcache_header = readchrs(DATASZ) or ""
    local tagcache_sz = readchrs(DATASZ) or ""
    local tagcache_entries = readchrs(DATASZ) or ""

    if tagcache_header ~= sTCHEADER or
        bytesLE_n(tagcache_sz) ~= (fsz - TCHSIZE) then
        rb.splash(100, sINVALIDDATABASE .. " " .. filename)
        return
    end
   
    -- local tag_entries = bytesLE_n(tagcache_entries)
    local fbegin = file:seek("cur") --Mark the beginning for later loop
    local tracks = 0
    local str = ""
    local rand
    local spread = 10;
    while true do
        tag_len = bytesLE_n(readchrs(DATASZ))
        readchrs(DATASZ) -- idx = bytesLE_n(readchrs(DATASZ))
        str = readchrs(tag_len) or ""
        str = string.match(str, "(%Z+)%z") -- \0 terminated string
        rand = math.random(1000)
        if str and rand >= (500 - spread) and rand < (500 + spread) then
            rb.playlist("insert_track", str)
            tracks = tracks + 1
            if tracks >= max_entries then break end
        elseif posln >= fsz then
            if tracks < max_entries then
                spread = spread + 10
                posln = fbegin -- go through again
            else             
                break
            end
        end

        if rb.get_plugin_action(0) == CANCEL_BUTTON then
            break
        end
    end

    file:close()

    if tracks == max_entries and play == true then
        rb.playlist("start", 0, 0, 0)
    end
    return ftable
end -- create_playlist

local function main()
    if not rb.file_exists(rb.ROCKBOX_DIR .. "/database_4.tcd") then
        rb.splash(rb.HZ, "Initialize Database")
        os.exit(1);
    end
    math.randomseed(rb.current_tick()); -- some kind of randomness
    rb.splash(10, "Searching for Files..")
    collectgarbage("collect")
    create_random_playlist(rb.ROCKBOX_DIR .. "/database_4.tcd", true);
    collectgarbage("collect")

    rb.splash(100, "Goodbye")
end

main() -- BILGUS


Now:
 it will go through the database file multiple times if need be and it lowers the selectivity the more times it loops through
you can make max entries as large as you like but 500 songs takes quite a while with my clip zip

[philden]

Wow, thank you very much for doing this, Bilgus.

With a bit of experimentation I got it working. In case anyone else, like me, has no experience of lua files, this is what I did.
I copied the code to a file, named it 'random.lua' and placed it in .rockbox/rocks/demos/lau_scripts

I used the Mac GUI to copy the files and was initially misled by the icon files starting with '.'.

Running the file works perfectly, ending with a 'Goodbye' message, and resuming playback uses the new random playlist.

Thanks again! Phil.

[Bilgus]

Ive cleaned it up made it possible to set num tracks and selectivity mind testing this one out?

Code: [Select]

--[[ Lua RB Random Playlist --playlist_random.lua
/***************************************************************************
 *             __________               __   ___.
 *   Open      \______   \ ____   ____ |  | _\_ |__   _______  ___
 *   Source     |       _//  _ \_/ ___\|  |/ /| __ \ /  _ \  \/  /
 *   Jukebox    |    |   (  <_> )  \___|    < | \_\ (  <_> > <  <
 *   Firmware   |____|_  /\____/ \___  >__|_ \|___  /\____/__/\_ \
 *                     \/            \/     \/    \/            \/
 * $Id$
 *
 * Copyright (C) 2021 William Wilgus
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * as published by the Free Software Foundation; either version 2
 * of the License, or (at your option) any later version.
 *
 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
 * KIND, either express or implied.
 *
 ****************************************************************************/
]]
require ("actions")
--require("dbgettags")

local scrpath = rb.current_path()
local max_tracks = 500;
local min_repeat = 50;  --this many songs before a repeat 200k/50 careful with the mem!
local CANCEL_BUTTON = rb.actions.PLA_CANCEL
local OK_BUTTON = rb.actions.PLA_SELECT
local ADD_BUTTON = rb.actions.PLA_UP
local SUB_BUTTON = rb.actions.PLA_DOWN
local sINVALIDDATABASE = "Invalid Database"
local sERROROPENING    = "Error opening"

-- tag cache header
local sTCVERSION = string.char(0x10)
local sTCHEADER  = string.reverse("TCH" .. sTCVERSION)
local DATASZ    = 4  -- int32_t
local TCHSIZE   = 3 * DATASZ -- 3 x int32_t

local function bytesLE_n(str)
    str = str or ""
    local tbyte={str:byte(1, -1)}
    local bpos = 1
    local num  = 0
    for k = 1,#tbyte do -- (k = #t, 1, -1 for BE)
        num = num + tbyte[k] * bpos
        bpos = bpos * 256
    end
    return num
end

-- Gets size of text
local function text_extent(msg, font)
    font = font or rb.FONT_UI

    return rb.font_getstringsize(msg, font)
end

function create_random_playlist(filename, play)

    if not filename then return end
    if not play then play = false end

    local file = io.open('/' .. filename or "", "r") --read
    if not file then rb.splash(100, sERROROPENING .. " " ..  filename) return end

    local fsz = file:seek("end")

    rb.audio("stop")
    rb.playlist("create", scrpath .. "/", "random_playback.m3u8")

    local posln = 0
    local tag_len = TCHSIZE
    local idx

    local function readchrs(count)
        if posln >= fsz then return nil end
        file:seek("set", posln)
        posln = posln + count
        return file:read(count)
    end

    -- check the header and get size + #entries
    local tagcache_header = readchrs(DATASZ) or ""
    local tagcache_sz = readchrs(DATASZ) or ""
    local tagcache_entries = readchrs(DATASZ) or ""

    if tagcache_header ~= sTCHEADER or
        bytesLE_n(tagcache_sz) ~= (fsz - TCHSIZE) then
        rb.splash(100, sINVALIDDATABASE .. " " .. filename)
        return
    end

    local entries = bytesLE_n(tagcache_entries)
    local fbegin = file:seek("cur") --Mark the beginning for later loop
    local tracks = 0
    local str = ""
    local rand
    local rand_range = entries;
    local spread = (entries / max_tracks) / 2;
    if spread < 1 then
        spread = 1
        rand_range = max_tracks
    end

    local res, w, h = text_extent("I")
    local max_w = rb.LCD_WIDTH / w
    local max_h = rb.LCD_HEIGHT  - h
    local y = 0

    local action
    local ask
    local t_desc = {scroll = true}

    function do_progress_header()
        rb.lcd_put_line(1, 0, string.format("%d \\ %d tracks",tracks, max_tracks))
        rb.lcd_update()
        --rb.sleep(100)
    end

    function show_progress()
        local sdisp = str:match("([^/]+)$") or "?"
        sdisp = sdisp:sub(1, max_w)
        rb.lcd_put_line(1, y, sdisp);
        y = y + h
        if y >= max_h then
            do_progress_header()
            rb.lcd_clear_display()
            y = h
        end
        collectgarbage("collect")
    end

    function show_setup_header()
        rb.lcd_clear_display()
        rb.lcd_put_line(1, 0, "Random Playlist", {icon = 9, scroll = true}) -- 9 == ICON_PLUGIN
    end

    function get_play()
        rb.lcd_put_line(1, h, string.format("Play? [%s] (up/dn)", tostring(play)),t_desc)
        rb.lcd_put_line(1, h + h, "overwrites dynamic", tdesc)
        rb.lcd_update()
        action = rb.get_plugin_action(-1);
        if action == ADD_BUTTON then
            play = true
        elseif action == SUB_BUTTON then
            play = false
        end
    end

    function get_selectivity()
        if spread > max_tracks then spread = 0 end
        rb.lcd_put_line(1, h, string.format("selectivity? [%d] (up/dn)",
                        max_tracks - spread), t_desc)
        rb.lcd_put_line(1, h + h, "higher = more selective", tdesc)
        rb.lcd_update()
        action = rb.get_plugin_action(-1);
        if action == SUB_BUTTON then --LESS SELECTIVE
            spread = spread + 1
            if spread > max_tracks then spread = max_tracks end
        elseif action == ADD_BUTTON then
            spread = spread - 1
            if spread < 0 then spread = 0 end
        elseif action == OK_BUTTON then
            ask = get_play;
            get_selectivity = nil
            action = 0
        end

    end

    function get_playlist_size()
        rb.lcd_put_line(1, h, string.format("find [%d] tracks? (up/dn)", max_tracks), t_desc)
        rb.lcd_update()
        action = rb.get_plugin_action(-1);
        if action == ADD_BUTTON then
            if max_tracks == 1 then max_tracks = 0 end
            max_tracks = max_tracks + 50
        elseif action == SUB_BUTTON then
            max_tracks = max_tracks - 50
            if max_tracks < 1 then max_tracks = 1 end
        elseif action == OK_BUTTON then
            ask = get_selectivity;
            get_playlist_size = nil
            action = 0
        end
    end
    ask = get_playlist_size; -- \!FIRSTRUN!/

    repeat
        show_setup_header()
        ask()
        collectgarbage("collect")
        if action == CANCEL_BUTTON then return action end
    until (action == OK_BUTTON)
    rb.lcd_scroll_stop() -- I'm still weary of not doing this..

    rb.splash(10, "Searching for Files..")

    if min_repeat > entries then min_repeat = entries / 2 + 1 end
    local lastrpt = {}
    local irpt = 1
    local minspread
    local maxspread
    local tries
    while true do
        minspread = tracks - spread
        maxspread = tracks + spread
        tries = 0
        if minspread <= 0 then minspread = 1 end
        if maxspread > rand_range then maxspread = rand_range end

        rand = math.random(0, rand_range)
        while  not ((rand >= minspread and rand <= maxspread)) do
            posln = math.random(fbegin, fsz)
            rand = math.random(0, rand_range)
            tries = tries + 1
            if tries > fsz then break end
        end

        local rep = 0
        while posln > fbegin do -- need to get to end of a string
            str = readchrs(10) or ""
            str = string.match(str, "(%Z%Z%Z%Z%Z)%z%Z%Z?%z?%z%z") -- \0 terminated string and size
            if str == nil and rep < fsz then
                posln = math.random(fbegin + 1, fsz)
                rep = rep + 1;
            elseif str == nil then
                posln = fbegin
            else
                posln = posln - 4 -- start at next string size entry
                break;
            end
        end

        tag_len = bytesLE_n(readchrs(DATASZ))
        readchrs(DATASZ) -- idx = bytesLE_n(readchrs(DATASZ))
        str = readchrs(tag_len) or ""
        str = string.match(str, "(%Z+)%z") -- \0 terminated string

        if str ~= nil then -- check for recent repeats
            for _, pos in ipairs(lastrpt) do
                if pos == posln then
                    str = nil
                    break
                end
            end
        end

        if str ~= nil then
            local lastr = #lastrpt
            if lastr > min_repeat then
                lastrpt[irpt] = posln
                irpt = irpt + 1
                if irpt > min_repeat then irpt = 1 end
            else
                lastrpt[lastr + 1] = posln
            end
            rb.playlist("insert_track", str)
            tracks = tracks + 1
            show_progress()
        end

        if tracks >= max_tracks then
            do_progress_header()
            break
        end

        if rb.get_plugin_action(0) == CANCEL_BUTTON then
            break
        end
    end

    file:close()

    if tracks == max_tracks and play == true then
        rb.playlist("start", 0, 0, 0)
    end
end -- create_playlist

local function main()
    if not rb.file_exists(rb.ROCKBOX_DIR .. "/database_4.tcd") then
        rb.splash(rb.HZ, "Initialize Database")
        os.exit(1);
    end
    math.randomseed(rb.current_tick()); -- some kind of randomness
    rb.lcd_clear_display()
    rb.lcd_update()
    collectgarbage("collect")
    create_random_playlist(rb.ROCKBOX_DIR .. "/database_4.tcd", true);
    collectgarbage("collect")

    rb.splash(rb.HZ * 2, "Goodbye")
end

main() -- BILGUS


let me know anything you don't like or get hung up on and i'll push it to main in the next few days

--Edit
added a min repeat buffer I noticed even with all that randomizing songs still bunched upped more than I could stand
I didn't add a setting for it but be aware it eats ram fast keeping arrays full of last played items

[Duceboia]

Thanks Bilgus, very neat work.

[philden]

Thanks again!

This works much better than the first version. I tried more tests of that and found that selecting 500 random tracks from 59,000 never found anything from an artist beyond 'G' in the database, and the list was in alphabetical order.

The new version makes a nice random list and even selecting just 100 tracks goes through the database better. I do have two comments, though. Selecting 'yes' to play works fine. However, selecting 'no' doesn't offer the option to save the playlist. At least that is what I expected choosing 'no' to mean.

What does the value of 'selectivity' mean? My desire is for a totally random selection, should that have a high or low value?

A very minor point is that saving the playlist, while playing, via the context menu, offers the default path as the lua_scripts directory. It would be nice if this went to 'Playlists' instead.

Anyway, I'm very impressed, nice work!

[Bilgus]

@Duceboia, philden thanks I had fun with it

here is what should be the final
unfortunately the plugins get the dynamic playlist buffer and no function is exposed to save it
but, you can go to the playlist catalog long press and view it there

I added warnings made repeat history smaller and configurable
added better descriptions
create /Playlists if it doesn't exist, make default the Playlists/random_playback.m3u8
it also buffers the database file a bit better hopefully thrashing it less

Code: [Select]

--[[ Lua RB Random Playlist --playlist_random.lua V 0.4
/***************************************************************************
 *             __________               __   ___.
 *   Open      \______   \ ____   ____ |  | _\_ |__   _______  ___
 *   Source     |       _//  _ \_/ ___\|  |/ /| __ \ /  _ \  \/  /
 *   Jukebox    |    |   (  <_> )  \___|    < | \_\ (  <_> > <  <
 *   Firmware   |____|_  /\____/ \___  >__|_ \|___  /\____/__/\_ \
 *                     \/            \/     \/    \/            \/
 * $Id$
 *
 * Copyright (C) 2021 William Wilgus
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * as published by the Free Software Foundation; either version 2
 * of the License, or (at your option) any later version.
 *
 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
 * KIND, either express or implied.
 *
 ****************************************************************************/
]]
require ("actions")

local playlistpath = "/Playlists"
local max_tracks = 500;
local min_repeat = 500;  --this many songs before a repeat
local random = math.random;
local CANCEL_BUTTON = rb.actions.PLA_CANCEL
local OK_BUTTON = rb.actions.PLA_SELECT
local ADD_BUTTON = rb.actions.PLA_UP
local SUB_BUTTON = rb.actions.PLA_DOWN
local sINVALIDDATABASE = "Invalid Database"
local sERROROPENING    = "Error opening"

-- tag cache header
local sTCVERSION = string.char(0x10)
local sTCHEADER  = string.reverse("TCH" .. sTCVERSION)
local DATASZ    = 4  -- int32_t
local TCHSIZE   = 3 * DATASZ -- 3 x int32_t

local function bytesLE_n(str)
    str = str or ""
    local tbyte={str:byte(1, -1)}
    local bpos = 1
    local num  = 0
    for k = 1,#tbyte do -- (k = #t, 1, -1 for BE)
        num = num + tbyte[k] * bpos
        bpos = bpos * 256
    end
    return num
end

-- Gets size of text
local function text_extent(msg, font)
    font = font or rb.FONT_UI
    return rb.font_getstringsize(msg, font)
end

function create_random_playlist(database, playlist, play)

    if not database or not database then return end
    if not play then play = false end

    local file = io.open('/' .. database or "", "r") --read
    if not file then rb.splash(100, sERROROPENING .. " " ..  database) return end

    local fsz = file:seek("end")

    local posln = 0
    local tag_len = TCHSIZE
    local idx

    function readchrs(count)
        if posln >= fsz then return nil end
        file:seek("set", posln)
        posln = posln + count
        return file:read(count)
    end

    -- check the header and get size + #entries
    local tagcache_header = readchrs(DATASZ) or ""
    local tagcache_sz = readchrs(DATASZ) or ""
    local tagcache_entries = readchrs(DATASZ) or ""

    if tagcache_header ~= sTCHEADER or
        bytesLE_n(tagcache_sz) ~= (fsz - TCHSIZE) then
        rb.splash(100, sINVALIDDATABASE .. " " .. database)
        return
    end

    local tag_entries = bytesLE_n(tagcache_entries)
    local rand_range = tag_entries;
    local spread = (tag_entries / max_tracks) / 2;
    if spread < 1 then
        spread = 1
        rand_range = max_tracks
    end

    local res, w, h = text_extent("I")
    local max_w = rb.LCD_WIDTH / w
    local max_h = rb.LCD_HEIGHT  - h
    local y = 0

    local action
    local ask
    local t_desc = {scroll = true}

    function ask_user_action(desc, ln1, ln2, ln3)
        if ln1 then rb.lcd_put_line(1, h + 5, ln1, desc) end
        if ln2 then rb.lcd_put_line(1, h + h + 5, ln2, desc) end
        if ln3 then rb.lcd_put_line(1, h + h + h + 5, ln3, desc) end
        rb.lcd_hline(1,rb.LCD_WIDTH - 1, h);
        rb.lcd_update()
        return rb.get_plugin_action(-1); -- Waits for action
    end

    function show_setup_header()
        rb.lcd_clear_display()
        rb.lcd_put_line(1, 0, "Random Playlist", {icon = 2, show_icons = true, scroll = true}) -- 2 == Icon_Playlist
    end

    function get_play()
        action = ask_user_action(tdesc,
        string.format("Play? [%s] (up/dn)", tostring(play)));
        if action == ADD_BUTTON then
            play = true
        elseif action == SUB_BUTTON then
            play = false
        end
    end

    function get_repeat()
        if min_repeat > tag_entries then min_repeat = tag_entries end
        action = ask_user_action(t_desc,
        string.format("Repeat hist? [%d] (up/dn)",min_repeat),
        "higher = less repeated songs");
        if action == ADD_BUTTON then
            min_repeat = min_repeat + 50
        elseif action == SUB_BUTTON then -- MORE REPEATS LESS RAM
            min_repeat = min_repeat - 50
            if min_repeat < 0 then min_repeat = 0 end
        elseif action == OK_BUTTON then
            ask = get_play;
            get_repeat = nil
            action = 0
        end
    end

    function get_selectivity()
        if spread > max_tracks then spread = 0 end
        action = ask_user_action(tdesc,
        string.format("Selectivity? [%d] (up/dn)", max_tracks - spread),
        "higher = more random");
        if action == SUB_BUTTON then -- LESS SELECTIVE
            spread = spread + 1
            if spread > max_tracks then spread = max_tracks end
        elseif action == ADD_BUTTON then
            spread = spread - 1
            if spread < 0 then spread = 0 end
        elseif action == OK_BUTTON then
            ask = get_repeat;
            get_selectivity = nil
            action = 0
        end
    end

    function get_playlist_size()
        action = ask_user_action(t_desc,
        string.format("Find [%d] tracks? (up/dn)", max_tracks),
        "Warning overwrites dynamic playlist",
        "Press back to cancel");
        if action == ADD_BUTTON then
            if max_tracks == 1 then max_tracks = 0 end
            max_tracks = max_tracks + 50
        elseif action == SUB_BUTTON then
            max_tracks = max_tracks - 50
            if max_tracks < 1 then max_tracks = 1 end
        elseif action == OK_BUTTON then
            if max_tracks == 1 then max_tracks = 2 end
            ask = get_selectivity;
            get_playlist_size = nil
            action = 0
        end
    end
    ask = get_playlist_size; -- \!FIRSTRUN!/

    repeat
        show_setup_header()
        ask()
        collectgarbage("collect")
        if action == CANCEL_BUTTON then return action end
    until (action == OK_BUTTON)
    rb.lcd_scroll_stop() -- I'm still weary of not doing this..
    rb.lcd_clear_display()

    rb.audio("stop")
    rb.playlist("create", playlistpath .. "/", playlist)

    rb.splash(10, "Searching for Files..")

    function get_files()
        local minspread, maxspread, tries
        local fbegin = posln --Mark the beginning for later loop
        local tracks = 0
        local str = ""
        local rand
        local t_lru = {}
        local lru_widx = 1
        local lru_max = min_repeat
        if lru_max >= tag_entries then lru_max = tag_entries / 2 + 1 end

        function do_progress_header()
            rb.lcd_put_line(1, 0, string.format("%d \\ %d tracks",tracks, max_tracks))
            rb.lcd_update()
            --rb.sleep(300)
        end

        function show_progress()
            local sdisp = str:match("([^/]+)$") or "?"
            sdisp = sdisp:sub(1, max_w)
            rb.lcd_put_line(1, y, sdisp);
            y = y + h
            if y >= max_h then
                do_progress_header()
                rb.lcd_clear_display()
                y = h
                collectgarbage("collect")
            end
           
        end

        function check_lru(lru, val)
            if lru_max <= 0 or val == nil then return 0 end --user wants all repeats
            local lru_ridx = 1
            local rv
            repeat
                rv = lru[lru_ridx]
                if val == rv then
                    return lru_ridx
                end
                lru_ridx = lru_ridx + 1
            until (lru_ridx > lru_max or rv == nil)
            return 0
        end

        function push_lru(lru, val)
            lru[lru_widx] = val
            lru_widx = lru_widx + 1
            if lru_widx > lru_max then lru_widx = 1 end
        end

        function seek_next_string()
            local retry = 0
            local data, rem, start, rewind
            while posln > fbegin or posln > fsz do -- need to get to beginning of next string
                data = readchrs(64) or ""
                rem, start = data:match("(%Z%Z%Z%Z%Z)%z(.*)") -- \0 terminated string and next
                if (rem == nil or start == nil) and retry < fsz then
                    if posln > fsz then posln = random(fbegin, fsz) end
                    retry = retry + 1;
                elseif retry > fsz then
                    posln = fbegin
                else
                    rewind = string.len(start or "")
                    posln = posln - rewind -- start at next string size entry
                    break;
                end
            end
        end
        local tries
        while true do
            str = nil
            minspread = tracks - spread
            maxspread = tracks + spread
            tries = 0
            if minspread <= 0 then minspread = 1 end
            if maxspread > rand_range then maxspread = rand_range end

            rand = random(0, rand_range)
            while  not ((rand >= minspread and rand <= maxspread)) do
                rand = random(0, rand_range)
                posln = random(fbegin, fsz) + rand
                tries = tries + 1
                if tries > fsz then break end
            end
            seek_next_string()
            local pos = check_lru(t_lru, posln) -- check for recent repeats
            while pos > 0 and posln < fsz do
                tag_len = bytesLE_n(readchrs(DATASZ))
                -- idx = bytesLE_n(readchrs(DATASZ))
                posln = posln + DATASZ + tag_len
                pos = check_lru(t_lru, posln) -- check for recent repeats
            end

            local lpos = posln
            tag_len = bytesLE_n(readchrs(DATASZ))
            posln = posln + DATASZ -- idx = bytesLE_n(readchrs(DATASZ))
            str = readchrs(tag_len) or "\0"
            str = str:match("(%Z+)%z$") -- \0 terminated string


            if str ~= nil then
                push_lru(t_lru, lpos)
                rb.playlist("insert_track", str)
                tracks = tracks + 1
                show_progress()
            end

            if tracks >= max_tracks then
                do_progress_header()
                break
            end

            if rb.get_plugin_action(0) == CANCEL_BUTTON then
                break
            end
        end
    end

    get_files()
    file:close()

    if tracks == max_tracks and play == true then
        rb.playlist("start", 0, 0, 0)
    end
end -- create_playlist

local function main()
    if not rb.file_exists(rb.ROCKBOX_DIR .. "/database_4.tcd") then
        rb.splash(rb.HZ, "Initialize Database")
        os.exit(1);
    end
    if not rb.dir_exists(playlistpath) then
        luadir.mkdir(playlistpath)
    end
    math.randomseed(rb.current_tick()); -- some kind of randomness
    rb.lcd_clear_display()
    rb.lcd_update()
    collectgarbage("collect")
    create_random_playlist(rb.ROCKBOX_DIR .. "/database_4.tcd",
                            "random_playback.m3u8", true);
    rb.splash(rb.HZ * 2, "Goodbye")
end

main() -- BILGUS



[philden]

Works nicely!

I'm still bemused by the concept of more or less random, though!

Thanks very much, Bilgus.

[Bilgus]

sorry I updated it once more it stopped working with a single track in one of the updates

anyway the idea is that it has a large range to pick at random and you can lessen that range at the ends or in the middle
selectivity is that range in the middle
 2. the degree to which an electronic circuit or instrument can distinguish particular frequencies.

also consider the database file has a long range of positions that still end up with the same string
it is the whole file path of the track previous.
The code iterates to the next track which now has 5-260?
possible values that could match the random number

this is why it is or was prone to repeats and not so much the rand num generator being bad

[philden]

This is much more complicated than I expected, I'm impressed by your work, Bilgus. I had naively thought that if the database contains 60,000 songs, you'd just need to generate a random number between 1 and 60,000 and play that track.

My limited understanding of the database is that it doesn't involve such a list. Might the above approach be an option from a playlist file that did contain all of the tracks? I don't know if it would be any easier for the device to handle than just playing or shuffling the list, but it could mean indefinite random play including song repeats.

[Bilgus]

philden, thanks It really isn't much more complicated than that and the track file paths are indexed as well but we are kinda going around that whole
process and just reading the database

I wanted to parse the whole thing and index it that was my first try and SLOW

what I have currently works well but its still pretty slow and it's got a bug that garbles the first track from the database
I'll have to get that figured out before I can finish it

[Bilgus]

Now I can safely say that was more involved than I anticipated

I think what I have now checks all the boxes and it is much faster too

https://gerrit.rockbox.org/r/c/rockbox/+/4027

I went with building a sparse index with a lua coroutine that walks to the desired string saving everything on the way
these items may be deleted by the garbage collector under memory pressure and later rebuilt

finally this allows rand(0-ndatabaseentries) and should work with massive playlists