Rockbox Technical Forums
Third Party => Other Utilities => Topic started by: audio-i on February 13, 2009, 12:55:59 PM
-
Being Rockbox “playlist orientedâ€, I thought there is a playlist feature that could actually be very cool.
If one’s DAP music would have a “public tagging logic†associated (that is, according to what a large number of persons listen to, how they listen to it, etc) it would be possible to generate playlists with music that goes great together, starting with a specific song you choose, from the very digital music player itself.
OK, itunes did it first for it’s devices (it’s called genius, and it’s of course also oriented to motivate you to buy recommended music), but still, I’ve been using it on an iphone just for the new-way-to-listen-to-your-music feature, and it’s VERY cool. It generates very coherent, yet very “creative†playlist, putting there new and old music that go very good together; often you get to listen to “forgotten†music you have, or generates great music combinations that go far beyond a simple genre filter.
Yes, something like what Last.fm does; only in this case of course it would be from music that’s in your library. So, if besides transmitting your scrobbled music to Last.fm, Rockbox+Last.fm could get to associate to the library inside your DAP (locally), the large Last.fm public logic tagging, than such playlists could be generated.
I don’t even know if Last.fm would have any interest in allowing this. I wish I could do more for this idea, unfortunately I don’t have the programming knowledge; hopefully there will be more people interested, and at some point, someone that could technically do it
-
It sounds like a cool idea for playlist preparation. But it's also not really a Rockbox idea. This is functionality for some external application to generate a playlist based on fingerprinting a given set of songs. Here's one (http://www.musicip.com/) that appears to do what you want.
Edit: Unlocking this for further discussion since it seems I misunderstood something here.
-
The idea would be to add the functionality to Rockbox, so that such playlists could be generated on-the-go from the device. Of course generating those automatic playlists would require the appropiate logic, and Last.fm is mentioned as *probably/hopefuly* http://www.last.fm/api a feasible example for that logic to be used
Edit: I changed the thread's subject to try and make it clearer
-
There is this old topic: http://forums.rockbox.org/index.php?topic=1541.0
-
There is this old topic: http://forums.rockbox.org/index.php?topic=1541.0
I can't seem to be able to access it
"An Error Has Occurred!
The topic or board you are looking for appears to be either missing or off limits to you. "
-
Try now.
-
Oops.
Thanks, soap! ;)
-
I thought all users could read the archives? Can they not? What is the point of the archives if they can't?
I'll create a dummy account in a few hours and test if nobody answers.
-
I didn't even know there *were* archives. I don't see anything like an "Archives" link anywhere..
-
It looks like they are a subforum of "Removed Posts" to which most users do not have access.
-
There is this old topic: http://forums.rockbox.org/index.php?topic=1541.0
The idea discussed there is very similar, and also there the main problem for even start thinking about implementing such feature, seemed to be how to put the palylist generating logic in the device. That's why I mentioned Last.fm as a possibility. They have for sure one of the biggest and more complete music listeners database to make their playlist generating logic great, and the mentioned link here http://www.last.fm/api gives hope that it might be possible:
"The Last.fm API allows anyone to build their own programs using Last.fm data, whether they're on the web, the desktop or mobile devices"
-
"The Last.fm API allows anyone to build their own programs using Last.fm data, whether they're on the web, the desktop or mobile devices"
Sure, so long as you are connected to Last.FM's database. I rather doubt you would want to carry around the whole database just to make playlists (even if you had the capacity). Since no Rockbox target has WiFi at this point in time, I don't see how that API could be made functional.
-
Right. The API consists of calls to a server side application that queries the database. Just having the name of the command doesn't give us any details of how to replicate that functionality. Without a web connection we can't invoke any of those methods and the API is useless to us.
To make this tractable one needs to have access to data to be stored in a local, Rockbox database. Whoever works on this needs enough details about the information taken from last.fm or MusicIP or whichever source to actually devise a sensible metric for determining each track's "distance" from each other.
-
Not knowing really anything about the technical details, here's how *I've seen* it works for ipod touch / iphone (I don't know if also for other ipods). There is a desktop application that gathers the information from your local music files (in their case itunes) and transmits it to the public database (in our case would be last.fm), which returns information from your local music and it's correspondence to the public database - that includes, amongst other things, other users listening habits - to be stored locally.
That information is also stored in the device after syncing, and this feature (in our case the new Rockbox feature) allows to use that information to generate intelligent playlists on the go, by choosing a song to be the starting point for a set of songs that go really well together. Now, how interesting those playlists can be, is hard to explain, one would have to try it to see how good they are.. as I said, even much better than what can be done with desktop media players automatic playlists based on rules for genres, ratings, etc... and of course, the beauty of it is that it can be done wherever you are.
-
If it works off-line like that - the first step would be for someone to figure out what data is stored on the DAP and how it is stored.
-
It seems like this will really require an accessory application to interact with the network database and dump information specific to a user's Rockbox library.
-
Are you sure the intelligent playlists aren't generated *before* you disconnect?
The traditional iTunes method of "smart" playlisting has been to let iTunes create the smart playlists, which then don't actually change until next time you sync.
Have you used this feature?
-
The traditional iTunes method of "smart" playlisting has been to let iTunes create the smart playlists, which then don't actually change until next time you sync.
Have you used this feature?
Yes, those are the ones I meant when I mentioned
...desktop media players automatic playlists based on rules for genres, ratings, etc...
which from what I've seen, although different to the proposed playlists, work pretty well if your music is correctly and extensively tagged (they are of course no on-the-go playlists). The proposed playlist generation wouldn't really care if the music is tagged (besides probably title and artist), because it would use a "normalized external database". As an example, and I have my music almost "obsessively" tagged, a usual automatic itunes playlist wouldn't have put together for me U2's "One" and Led Zeppelin's "Starway to heaven"... but this new feature did, and they actually go pretty well together ;)
Are you sure the intelligent playlists aren't generated *before* you disconnect?
I'm not sure. From a user standpoint, what you see is that if you are listening to any song (through itunes OR the iphone), and you want to generate a limited playlist of similar songs, you press "genius" and the new playlist is generated. If you do it again, the playlist will be different (at least mostly).
Here's what I found:
- On the PC, there is a file called iTunes/iTunes Library Genius.itdb (about 5MB for a 9GB music files size).
- On the iPhone, I didn't find that file (which doesn't really discard that it's there - or a similar one- since the tools I have for exploring the iphone aren't really handy). What I found is something called GeniusPlaylist.plist; the size is quite small (726 bytes) so it could be just the last generated playlist on the iphone, I can't check the date the file was created or anything else.
But your question made me think that another possibility is, like you say, that a number of playlists are pre-generated and then transferred to the iphone, so that a specific song is in multiple playlists containing similar songs, and every song is at least in one playlist, and this new feature just generates a randomized sub-playlist, starting with that specific song.
-
I've written a python program that achieves something similar to what's being dicussed. For each artist in your library, it creates a playlist of (say) 500 tracks by 'similar' artists that are also in your library. You can use two methods to define 'similar' artists:
1. It connects to the last.fm server and retrieves a list of similar artists, and then restricts that list to artists present in your library
2. You maintain a csv-format spreadsheet file with a row for each artist, and an arbitrary number of columns. E.g. My columns correspond to "song", "metal", "electronic","ambient","hiphop". You enter a score between 0-9 for each artist on each attribute, thus positioning each artist in a (in my case) 5-dimensional space. Then tracks are chosen from artists with probability a decreasing function of the Euclidean distance between artists in this space.
Both methods work pretty well for me; obviously it depends on your music-space csv file. The playlists can then be saved into a directory on a rockboxed DAP with the .m3u extension.
My code is available at
http://www.stats.ox.ac.uk/~davison/software/dbm/
Please get in touch if you're interested in using it (davison@stats.ox.ac.uk).
Dan
-
Are you interested in posting this to the UsefulTools page in the wiki? You could link to a tutorial if you like.
-
I think what Myrmornis have done is great news, it shows that it is possible to build the "similarities" database using last.fm, and I specially did like the possibility of building it according to one's personal preferences. Additionally, if what's proposed in this thread http://forums.rockbox.org/index.php?topic=20705.0 would be accomplished, it might be integrated to the whole database functionality. And what would make it really great, would be that it could be used for generating those playlists on-the-go, i.e a playlist generating option in Rockbox that would use the similarities database, using a song or an artist as a starting point.
In the meantime, I hope Chronon's suggestion is adopted ;)
Are you interested in posting this to the UsefulTools page in the wiki? You could link to a tutorial if you like.
-
Yes, no problem putting this on the useful tools page. I'll try to do that later today / tomorrow.
Edit: I've linked from the UsefulTools wiki page and put the beginnings of a tutorial here: http://www.stats.ox.ac.uk/~davison/software/dbm/dbm.html But it's *very* preliminary! If anyone would like to try it out, I'll be happy to help get it up and running.
In the meantime I should point out that music is mostly identified in my code using MusicBrainz unique identifiers. Sorry if this is obvious already to readers of this thread, but I'd suggest that any solution to the intelligent-playlist-generation issue should consider basing itself around musicbrainz identifiers. These are also used by last.fm, and solve problems caused by alternate spellings, non-ascii characters, etc. Each track, release and artist is assigned a unique code by musicbrainz. So this means that with my code you'll only get good results if a fair proportion of your music is tagged with the musicbrainz identifiers. I do that with the MusicBrainz tagger "picard" http://musicbrainz.org/doc/PicardTagger which is excellent and is available for all platforms. When it tags your music, it automatically inserts all the unique identifiers, together with the normal tags. My code also includes options to report on albums that are missing their MBIDs, etc.
-
Edit: I've linked from the UsefulTools wiki page and put the beginnings of a tutorial here: http://www.stats.ox.ac.uk/~davison/software/dbm/dbm.html But it's *very* preliminary! If anyone would like to try it out, I'll be happy to help get it up and running.
OK after spending some time making the Picard Tagger put the musicbrainz tags on my library (the ones that your program uses), I finally got to use what you developed, and I must say it is fantastic. It generates coherent, creative and fun playlists.
And thanks to your program results, it's now possible to make a "manual" approach to the functionality proposed in this thread.
Since one of the results of your program is a set of playlists, one for every artist in your library, containing for each artist a set of similar artists tracks, assume you are listening to a song (dynamic playlist started say from the album directory); if you want to generate an intelligent/similar music playlist for the current song, you just go to your program "Random" results stored in the portable device, find the current artist's playlist, and insert it. It's really great!
So from there, the "on-the-go" feature would mean making this process automatic. For a fancier functionality, trying to extend the results from "some random similar music" to "all the similar music in your library" and maybe also adding the option to limit the generated playlist by total playlist time, number of songs, size... After that, Rockbox current playlist features would allow to save it, reshuffle it, modify it, merge it, etc.
Great work, thanks for sharing it.
-
I thought I'd give this a go, but it bails out before producing any playlists with:
fatal error: no entry for releaseid c8cfc757-8bf2-4f19-801c-6533960d63d9 in releases database: shouldn't happen...
Is there any way for it to ignore this error and continue?
-
I thought I'd give this a go, but it bails out before producing any playlists with:
fatal error: no entry for releaseid c8cfc757-8bf2-4f19-801c-6533960d63d9 in releases database: shouldn't happen...
Is there any way for it to ignore this error and continue?
Sorry about that -- as you suggest I have altered this so that it prints out a warning and continues (dbm version 0.14). [And if you run dbm without the -m flag, just using -i and -o, then it will not attempt the check for incomplete albums which generated this error.]
-
I'll test drive this weekend if I find time. :)
-
Sorry about that -- as you suggest I have altered this so that it prints out a warning and continues (dbm version 0.14). [And if you run dbm without the -m flag, just using -i and -o, then it will not attempt the check for incomplete albums which generated this error.]
Thanks for the quick response. I had a play around with it yesterday and I'm impressed. I'm not sure about the usefulness of the Recommended section creating links to non-existent folders, but perhaps that's just because I look that info up for myself on my last.fm account page. But a big thumbs up to the rest of the functionality.
-
I'm having some real problems getting this to work. I've followed all the instructions and (on Windows) installed Python and the two packages within the dbm folder, the problem comes when scanning my music files.
The first time I ran it, it picked up approx 6 artists from 10GB of music (mpc mostly) and then I aimed it at my main flac library and it found 0.
This is a really stupid question, but is there anything obvious that I'm forgetting/doing wrong here?
-
I'm having some real problems getting this to work. I've followed all the instructions and (on Windows) installed Python and the two packages within the dbm folder, the problem comes when scanning my music files.
The first time I ran it, it picked up approx 6 artists from 10GB of music (mpc mostly) and then I aimed it at my main flac library and it found 0.
This is a really stupid question, but is there anything obvious that I'm forgetting/doing wrong here?
Not a stupid question. So far I've implemented tag reading for mp3, ogg vorbis and 'm4a'. But I'll add flac and mpc. It won't be much work, as I'm using the mutagen python library for reading tags, and it supports all these formats -- I'll post again when it's done.
-
Ah! That explains it nicely. I'm glad I listed which type of files I was using!
Thanks for such a quick response, I'm now transcoding all my flac to vorbis anyway, so will test when I have some time over the weekend.
EDIT: Having tested some ogg files, I've got one concern, that the .m3u generated for Artist name: "Weird Al" Yankovic caused an error, presumably as the quotation marks aren't allowed in Windows?
-
You might find Mirage (http://hop.at/mirage/) interesting.
It requires computing a unique ID that can be used to map the songs by similarity. It's excruciatingly slow as a Banshee plugin but it's a good (free) start. Perhaps we could rewrite that in C/C++?
-
@Myrmornis
Hi, this program looks fantastic but I can't get it to work.
I think I'm doing everything right. I'm pointing it at my ipod but when I try to run it I get no output, just the prompt. I thought it may have been the size of my music collection so I set up a test folder with a small selection of artists (all tagged through picard) and the same thing happened.
I tried it against a folder on my hard drive, and while it told me it had output the playlists, when I go to check they are empty. It also completed the process in a matter of seconds.
Can you think of anything obvious I'm doing wrong? Or if you need more info let me know.
-
What format files are you using on your iPod?
-
My ipod is a mixture of mp3 and ogg. I have flac, mp3 and ogg on my pc. I transcode the flac files to ogg before transferring them to the ipod. When I created my test folder I also made sure that all files had been tagged through picard to give the program a fairly easy test.
-
@shayguitarra: Could you provide the exact command line you're using, and describe the contents of your test library? What operating system are you using? For testing, a single album by a single artist should be enough, i.e. a toy library like this:
library/sole_artist/sole_album/track1.ogg
library/sole_artist/sole_album/track2.ogg
...
Then
python dbm.py -i library -o output-directoryought to work.
I'm sorry that it's not producing any helpful error messages for you. I'm working on a new version at the moment that will be flexible about the way the library is organised, still try to work when musicbrainz tags are missing, deal with compilations / various artist albums better, support flac and mpc, have better error messages, ....
-
OK I posted a message here about 30 minutes ago and have deleted it. The problem was my own ineptitude in Ubuntu terminal.
I've run a test in a hard drive folder (my music library on my hard drive is broken down by artist initial so the selection wasn't great) and it's generated playlists in the random folder for each artist. Marvellous!
But when I ran it on my iPod I got the following:
sheamus@sheamus-desktop:~/dbm$ python dbm.py -i /media/"SHEAMUS'S I"/Media -o dbm-output-folder
dbm version 0.14
2009-03-19 21:21:33 reading contents of music library at /media/SHEAMUS'S I/Media
/AC_DC/High Voltage
/50FootWave/Golden Ocean
/The Uniques/Trojan Rocksteady Box Set (disc 1)
/L7/The Best of L7 The Slash Years
/MC5/Kick Out the Jams
Traceback (most recent call last):
File "dbm.py", line 669, in <module>
make_mdirs_and_tracks()
File "dbm.py", line 78, in make_mdirs_and_tracks
dirtracks = map(parse_mutagen_tags, map(readtags, dirpaths))
File "dbm.py", line 454, in parse_mutagen_tags
elif tags['format'] in 'flac':
TypeError: 'in <string>' requires string as left operand
And then it stops.
The selection of problem folders seems fairly random. There are over 1100 artist folders in total.
-
@shayguitarra: Sorry, that was a bug. Not your fault at all. Should be fixed now. Could you re-download it and try again? (Make sure you clear your firefox cache so you really do re-download it... the new version number is 0.15). I think I must have accidentally uploaded a change that I was half-way through making -- it doesn't support flac yet.
-
Has anyone seen this? It seems like it can create playlists from your own listening habits, tags etc.
http://www.last.fm/group/I+Use+Local.fm
How feasible would it be to create a version that acts like DBM does? That is, create these playlists from the music on your ipod or other media player?