dbm - On The Go Last.fm-like Playlists from Rockbox

[Chronon]

It seems like this will really require an accessory application to interact with the network database and dump information specific to a user's Rockbox library.

[Llorean]

Are you sure the intelligent playlists aren't generated *before* you disconnect?

The traditional iTunes method of "smart" playlisting has been to let iTunes create the smart playlists, which then don't actually change until next time you sync.

Have you used this feature?

[audio-i]

The traditional iTunes method of "smart" playlisting has been to let iTunes create the smart playlists, which then don't actually change until next time you sync.

Have you used this feature?

Yes, those are the ones I meant when I mentioned

...desktop media players automatic playlists based on rules for genres, ratings, etc...

which from what I've seen, although different to the proposed playlists, work pretty well if your music is correctly and extensively tagged (they are of course no on-the-go playlists). The proposed playlist generation wouldn't really care if the music is tagged (besides probably title and artist), because it would use a "normalized external database". As an example, and I have my music almost "obsessively" tagged, a usual automatic itunes playlist wouldn't have put together for me U2's "One" and Led Zeppelin's "Starway to heaven"... but this new feature did, and they actually go pretty well together 

Are you sure the intelligent playlists aren't generated *before* you disconnect?

I'm not sure. From a user standpoint, what you see is that if you are listening to any song (through itunes OR the iphone), and you want to generate a limited playlist of similar songs, you press "genius" and the new playlist is generated. If you do it again, the playlist will be different (at least mostly).

Here's what I found:

- On the PC, there is a file called iTunes/iTunes Library Genius.itdb (about 5MB for a 9GB music files size).
- On the iPhone, I didn't find that file (which doesn't really discard that it's there - or a similar one- since the tools I have for exploring the iphone aren't really handy). What I found is something called GeniusPlaylist.plist; the size is quite small (726 bytes) so it could be just the last generated playlist on the iphone, I can't check the date the file was created or anything else.

But your question made me think that another possibility is, like you say, that a number of playlists are pre-generated and then transferred to the iphone, so that a specific song is in multiple playlists containing similar songs, and every song is at least in one playlist, and this new feature just generates a randomized sub-playlist, starting with that specific song.

[Myrmornis]

I've written a python program that achieves something similar to what's being dicussed.  For each artist in your library, it creates a playlist of (say) 500 tracks by 'similar' artists that are also in your library. You can use two methods to define 'similar' artists:
1. It connects to the last.fm server and retrieves a list of similar artists, and then restricts that list to artists present in your library
2. You maintain a csv-format spreadsheet file with a row for each artist, and an arbitrary number of columns. E.g. My columns correspond to "song", "metal", "electronic","ambient","hiphop". You enter a score between 0-9 for each artist on each attribute, thus positioning each artist in a (in my case) 5-dimensional space. Then tracks are chosen from artists with probability a decreasing function of the Euclidean distance between artists in this space.
Both methods work pretty well for me; obviously it depends on your music-space csv file. The playlists can then be saved into a directory on a rockboxed DAP with the .m3u extension.

My code is available at
http://www.stats.ox.ac.uk/~davison/software/dbm/
Please get in touch if you're interested in using it (davison@stats.ox.ac.uk).

Dan

[Chronon]

Are you interested in posting this to the UsefulTools page in the wiki?  You could link to a tutorial if you like.

[audio-i]

I think what Myrmornis have done is great news, it shows that it is possible to build the "similarities" database using last.fm, and I specially did like the possibility of building it according to one's personal preferences. Additionally, if what's proposed in this thread http://forums.rockbox.org/index.php?topic=20705.0 would be accomplished, it might be integrated to the whole database functionality. And what would make it really great, would be that it could be used for generating those playlists on-the-go, i.e a playlist generating option in Rockbox that would use the similarities database, using a song or an artist as a starting point.

In the meantime, I hope Chronon's suggestion is adopted 

Are you interested in posting this to the UsefulTools page in the wiki?  You could link to a tutorial if you like.

[Myrmornis]

Yes, no problem putting this on the useful tools page. I'll try to do that later today / tomorrow.

Edit: I've linked from the UsefulTools wiki page and put the beginnings of a tutorial here: http://www.stats.ox.ac.uk/~davison/software/dbm/dbm.html But it's *very* preliminary! If anyone would like to try it out, I'll be happy to help get it up and running.

In the meantime I should point out that music is mostly identified in my code using MusicBrainz unique identifiers. Sorry if this is obvious already to readers of this thread, but I'd suggest that any solution to the intelligent-playlist-generation issue should consider basing itself around musicbrainz identifiers. These are also used by last.fm, and solve problems caused by alternate spellings, non-ascii characters, etc. Each track, release and artist is assigned a unique code by musicbrainz. So this means that with my code you'll only get good results if a fair proportion of your music is tagged with the musicbrainz identifiers. I do that with the MusicBrainz tagger "picard"  http://musicbrainz.org/doc/PicardTagger which is excellent and is available for all platforms. When it tags your music, it automatically inserts all the unique identifiers, together with the normal tags. My code also includes options to report on albums that are missing their MBIDs, etc.

[audio-i]

Edit: I've linked from the UsefulTools wiki page and put the beginnings of a tutorial here: http://www.stats.ox.ac.uk/~davison/software/dbm/dbm.html But it's *very* preliminary! If anyone would like to try it out, I'll be happy to help get it up and running.

OK after spending some time making the Picard Tagger put the musicbrainz tags on my library (the ones that your program uses), I finally got to use what you developed, and I must say it is fantastic. It generates coherent, creative and fun playlists.
And thanks to your program results, it's now possible to make a "manual" approach to the functionality proposed in this thread.

Since one of the results of your program is a set of playlists, one for every artist in your library, containing for each artist a set of similar artists tracks, assume you are listening to a song (dynamic playlist started say from the album directory); if you want to generate an intelligent/similar music playlist for the current song, you just go to your program "Random" results stored in the portable device, find the current artist's playlist, and insert it. It's really great!

So from there, the "on-the-go" feature would mean making this process automatic. For a fancier functionality, trying to extend the results from "some random similar music" to "all the similar music in your library" and maybe also adding the option to limit the generated playlist by total playlist time, number of songs, size...  After that, Rockbox current playlist features would allow to save it, reshuffle it, modify it, merge it, etc.

Great work, thanks for sharing it.

[obo]

I thought I'd give this a go, but it bails out before producing any playlists with:

fatal error: no entry for releaseid c8cfc757-8bf2-4f19-801c-6533960d63d9 in releases database: shouldn't happen...

Is there any way for it to ignore this error and continue?

[Myrmornis]

I thought I'd give this a go, but it bails out before producing any playlists with:

fatal error: no entry for releaseid c8cfc757-8bf2-4f19-801c-6533960d63d9 in releases database: shouldn't happen...

Is there any way for it to ignore this error and continue?

Sorry about that -- as you suggest I have altered this so that it prints out a warning and continues (dbm version 0.14). [And if you run dbm without the -m flag, just using -i and -o, then it will not attempt the check for incomplete albums which generated this error.]

[Chronon]

I'll test drive this weekend if I find time.   

[obo]

Sorry about that -- as you suggest I have altered this so that it prints out a warning and continues (dbm version 0.14). [And if you run dbm without the -m flag, just using -i and -o, then it will not attempt the check for incomplete albums which generated this error.]

Thanks for the quick response.  I had a play around with it yesterday and I'm impressed.  I'm not sure about the usefulness of the Recommended section creating links to non-existent folders, but perhaps that's just because I look that info up for myself on my last.fm account page.  But a big thumbs up to the rest of the functionality.

[evilnick]

I'm having some real problems getting this to work. I've followed all the instructions and (on Windows) installed Python and the two packages within the dbm folder, the problem comes when scanning my music files.

The first time I ran it, it picked up approx 6 artists from 10GB of music (mpc mostly) and then I aimed it at my main flac library and it found 0.

This is a really stupid question, but is there anything obvious that I'm forgetting/doing wrong here?

[Myrmornis]

I'm having some real problems getting this to work. I've followed all the instructions and (on Windows) installed Python and the two packages within the dbm folder, the problem comes when scanning my music files.

The first time I ran it, it picked up approx 6 artists from 10GB of music (mpc mostly) and then I aimed it at my main flac library and it found 0.

This is a really stupid question, but is there anything obvious that I'm forgetting/doing wrong here?

Not a stupid question. So far I've implemented tag reading for mp3, ogg vorbis and 'm4a'. But I'll add flac and mpc. It won't be much work, as I'm using the mutagen python library for reading tags, and it supports all these formats -- I'll post again when it's done.

[evilnick]

Ah! That explains it nicely. I'm glad I listed which type of files I was using!

Thanks for such a quick response, I'm now transcoding all my flac to vorbis anyway, so will test when I have some time over the weekend.

EDIT: Having tested some ogg files, I've got one concern, that the .m3u generated for Artist name: "Weird Al" Yankovic caused an error, presumably as the quotation marks aren't allowed in Windows?