Unpacking Boxes...

For the impatient people running Fedora 16 but who still want to get an aperçu of Boxes, today's your lucky day! I set up a preview repository with all the needed package to install Boxes on Fedora 16.
If you want to try it, download this file to /etc/yum.repos.d and then run  
yum install gnome-boxes && yum update

To go back to your previous setup, you can either use the convenient yum history, or remove /etc/yum.repos.d/gnome-boxes-preview.repo and use
yum remove libvirt-glib && yum distro-sync

Keep in mind that this is a new application still in heavy development, so you're likely to find bugs and missing features. But I'm sure you will enjoy it nonetheless :)

Feel free to join us in #boxes on irc.gnome.org if you have any issues, or if you just want to chat with us.

Transferring contacts to an iDevice

Recently, I wanted to transfer my addressbook from my good old but dying Sony Ericsson W910i phone to one of Apple iOS devices (aka iPhone 4).

Reading the contacts out of the old phone was a bit hackish but pretty straightforward using gammu/wammu and a bit of hacking. I didn't manage to import 100% of the data in GMail/evolution-data-server (had to retype a few addresses/phone numbers by hand), but I decided I could live with that.

Then I looked at the computer to iDevice writing, thinking it would be a pretty similar process, ie grab an existing tool, do a bit of fiddling and be done with it. Alas not :) I quickly realized that there was no tool on linux to send contacts to an iDevice. The alternatives involved booting into OS X, or sending a vcf file by mail and opening it on the iDevice. I didn't want to do the former, and the latter didn't work for some reason (and it's cheating anyway ;)

After talking with the nice people on #libimobiledevice, I realized the low-level building blocks were there in libimobiledevice, so I decided to go ahead and write the missing code (which involved parsing and writing XML plists from the device after figuring out their format). After a few days of hacking, eds-to-idevice was born! This C program can read contacts from evolution-data-server and writes them to an iDevice.
Be aware that this does not try to handle contact synchronization: when I use it, I tell it to erase all contacts from the iDevice and to unconditionnally write all contacts from evolution-data-server to the iDevice. If it's called multiple times, it will create duplicate contacts on the device.
However, I tried to make the code as generic and reusable as possible, with the hope that someone would pick up the ball and improve it to write a synchronization plugin for one of the linux synchronization framework. Volunteers :) ?

You can find the code on gitorious and I made a tarball. Enjoy!

Adventures in udev land

I've mentionned a while ago the work that has been done on libgpod hal callout. It's working nicely, but with HAL being deprecated, I thought now might be a good time at looking at how to do things in the future, and to check if udev already lets us do what we do in the HAL callout. The good news is that it's working now and is pretty straightforward. However, I got stuck on a few details, so I thought it might be useful to others if I documented my findings here.

Currently, libgpod installs a .fdi file with the iPod vendor ID/device ID and a binary name. When HAL detects that a device that matches these 2 IDs is plugged in, it runs the binary. The binary issues a SCSI inquiry command to the iPod to get various information, and set some HAL properties using libhal.

My goal was to do something similar with udev, ie get udev to run a binary when an iPod is inserted and then associate some information with the iPod device in udev database so that other applications can access it.

iPod detection

Running a binary when the iPod is inserted wasn't too hard, it's done with a udev rule file (the format is documented in udev manpage, don't forget to read it if you have to write such a file! ) which goes to /lib/udev/rules.d. My first version was simple enough:

ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_USAGE}=="filesystem",

ATTRS{idVendor}=="05ac", ATTRS{idProduct}=="1204",

RUN+="/tmp/udev-test.sh"

udev-test.sh is a simple shell script wrapping the actual udev callout. It makes it easier to dump various information to a log file (for example the callout environment which udev uses to pass useful information to the callout). And, lo and behold, after plugging an iPod, my shell script was run!

Adding information to the udev database

The next logical step was to add some values read from the iPod to udev database so that other apps can get this additionnal information. And this is one of the steps that gave me some troubles. I was a bit ashamed of finding a really informative post by Kay Sievers answering my question right after having sent this email...

In short, it's really easy to import new values in the udev database, all you have to do is to output key/value pairs on stdout. This is nice, since udev passes information through environment variables and adds information to its database by reading stdout, this means that your callout doesn't have to depend in anyway on libudev.

I quickly modified my test program to print a few key/value pairs to add to the environment, triggered an unplug/replug of my ipod with udevadm, and watch the 'block' subsystem devices with the devkit binary. But I was really disappointed not to find my values associated with the iPod :(

After double checking everything and fiddling a bit to try to figure out what was wrong, I read again Kay's email, and I saw there was another difference between his code and mine: he is using an IMPORT rule to run his binary while I was still using a RUN rule.

I changed my udev rule to an IMPORT rule and.... it still didn't work :;)After staring at udevadm monitor output, I noticed that when the iPod was plugged in, there was first an "add" udev event for the iPod device shortly followed by a "change" event. Since my rule was only catching the "add" event, I hypothetized maybe my changes to the udev database were first properly added, and then overwritten by the "change" event. So I changed my rule file to catch "change" events in addition to "add" events, and it finally worked!

ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_FS_USAGE}=="filesystem",

ATTRS{idVendor}=="05ac", ATTRS{idProduct}=="1204",

IMPORT{program}="/tmp/udev-test.sh"

I was very happy with my udev callout, however I shortly realized that when going from a RUN rule to an IMPORT rule, udev no longer passed me the device name (/dev/sd??) in the DEVNAME environment variable. I went to #udev on freenode to check if this was the expected behaviour, and Kay confirmed this is normal because when IMPORT rules are run, the final device doesn't exist yet.

However, the $tempnode variable can be used as an argument to the binary that is being run to give it access to a temporary device node which can be interacted with. And indeed, after adding this argument to my udev rule, I could do everything that I wanted to :)

Final polish

After this successful experiment, all that remained to be done was to make the udev callout as featureful as the HAL callout. This was pretty straightforward, I abstracted the information gathering part from the HAL callout. This generic code then uses some backend-specific code to set the values. The HAL backend does that by using libhal, the udev backend does that by just outputting values to stdout.

David Zeuthen was (as always) really helpful by pointing me at the udev/devicekit equivalent for info.desktop.icon and info.desktop.name: DKD_PRESENTATION_NAME and DKD_PRESENTATION_ICON_NAME. I also cooked up some variable names in a LIBGPOD namespace to have an udev equivalent to the stuff provided by podsleuth, let me know if it could be useful in your projects, it can be changed to fit your needs :)

End result

And here is the end result after plugging in my iPod:

# udevadm info --query=env --name=sdb2

DKD_PRESENTATION_ICON_NAME=multimedia-player-apple-ipod-color
LIBGPOD_VERSION=1
LIBGPOD_IS_UNKNOWN=0
LIBGPOD_FIREWIRE_ID=000A270002BAD546
LIBGPOD_SERIAL_NUMBER=JQ446FN4R5Q
LIBGPOD_FIRMWARE_VERSION=1.2.1
LIBGPOD_IMAGES_ALBUM_ART_SUPPORTED=1
LIBGPOD_IMAGES_PHOTOS_SUPPORTED=1
LIBGPOD_IMAGES_CHAPTER_IMAGES_SUPPORTED=1
LIBGPOD_DEVICE_CLASS=color
LIBGPOD_MODEL_GENERATION=4.000000
LIBGPOD_MODEL_SHELL_COLOR=white
LIBGPOD_PRODUCTION_FACTORY_ID=JQ
LIBGPOD_PRODUCTION_YEAR=2004
LIBGPOD_PRODUCTION_WEEK=46
LIBGPOD_PRODUCTION_INDEX=20272
LIBGPOD_MODEL_CONTROL_PATH=/iPod_Control


Et voilà! The code is available from the devicekit branch of my libgpod git repo
and will be committed to libgpod svn soon.

libgpod callout improvements

The other day, I mentioned that libgpod hal callout was setting volume.label to the iPod name to get a nicer name displayed for iPods in Nautilus (among other apps). What I didn't say is that I wasn't really sure that overriding that property with something that has nothing to do with the actual filesystem label was such a good idea.

And it wasn't. After asking David Zeuthen about that on IRC, he kindly told me about info.desktop.name which was added explicitly for that purpose. And I also learnt about info.desktop.icon which is to be preferred to info.icon_name. So I made these 2 changes and pushed them to the podsleuth branch of my git repository.

While I was at it, I worked on the few things that are still in the way to a libgpod 0.7 release, ie I cleaned up the exported symbols to make sure what we export make sense from an API point of view, and I added some missing API doc and made a few fixes in the existing one (some functions were renamed and the API doc wasn't properly updated).


On an unrelated note, I'm glad to see that some people find this blog worth some comments, thanks ;)

And my other project is...

After describing my latest work on Rhythmbox yesterday, here's what I did on libgpod in the last month. libgpod is a cross-platform library used by many different projects (amarok, gtkpod, rhythmbox, songbird to name a few) to access and modify your ipod content.

Extensive SysInfoExtended parsing

As has been known for a while now, the iPod can be queried about its capabilities using SCSI commands and returns XML data describing the iPod (serial number, firmware version, ...) and what it can do can do (podcast support, video formats supported, image formats that it knows how to display, ...). When we released libgpod 0.6, we introduced a hal callout to send the appropriate SCSI query to the iPod and to dump the returned XML data to a file that we named SysInfoExtended. Normal users aren't guaranteed to be able to send raw SCSI commands to a device, hence the use of a hal callout and the dumping of the information to a regular file. However, in libgpod 0.6, we only had a very basic parser for that file which only knew how to read the only SysInfoExtended field we needed. Most of the information about the iPod capabilities was hard-coded into per-ipod model tables, and libgpod had to be told the iPod model before being able to (for example) being able to write artwork to an iPod.

For the next release of libgpod, I decided that we had to be able to use the information from SysInfoExtended to its fullest. I started by writing a generic plist (which is the XML subset SysInfoExtended is in) to GValue parser using libxml instead of GMarkup. Then, I extracted the data I was interested in from the GValue collection the parser gave me to a nice C struct. To make the addition of new fields easy, most of the work is driven from a table indicating the field name in the plist file, the type we want to assign that field data and the offset we want to put that data in the resulting struct. Modifying that table and the struct definition are the only things that need to be done if we want to read additionnal fields from SysInfoExtended.

With that being done , I had everything I needed to have libgpod use the information provided by the device to write artwork to the iPod instead of relying on hard-coded tables. Some refactoring was needed to make it possible to use the artwork data from the iPod (there were some assumptions here and there that the formats supported by the iPod were known at compile time) but now that it's done, the code feels much more natural and maintainable than before.

Getting the iPod model from SysInfoExtended

With the aforementioned work, writing artwork to the iPod has been made much more flexible, but libgpod was still unable to automatically guess the iPod model/color/... from the device without asking the user. This is due to the fact that to do that, libgpod relied on the iPod "ModelNum" which used to be present in a file on the iPod filesystem but for quite some time now, the only way to get that model number is to read it on the iPod box, which is not really easy to do from software :)

But for all recent iPod models, there's another way to guess the iPod model, this is by parsing the iPod serial number. And this serial number is precisely one of the things that we can read from SysInfoExtended! So all we had to do to be able to automatically detect the model/color/.. of a plugged iPod was to properly parse the iPod serial number and to infer the iPod physical features from that serial number, just as what podsleuth does with that table.

Podsleuth

Given that libgpod had already installed a hal callout, and after all the work done to parse the SysInfoExtended data, I realized that libgpod had gathered all the pieces to build a podsleuth clone, so I decided to try to write one just for fun and to test the new API added to libgpod in real-world situations. This led to the work which can be found in the podsleuth branch of my libgpod git repository.

Even though I haven't tested it against banshee, I compared the properties exported by podsleuth and by this experimental stuff, so this code can probably be used as a drop-in replacement to podsleuth. Writing it also made me realize that podsleuth doesn't export enough information about artwork formats compared to what libgpod needs. I'm also not a big fan of how podsleuth exposes the artwork formats: it parses the XML to get the artwork data to immediatly serialize it again to a string. It's then up to the app using podsleuth to parse that string (again) to get the artwork formats supported by the ipod.

Anyway, since I now have hacked this nice tool, it's now up to me to experiment a bit with all of that and to make suggestions as to how things could be improved :) By the way, I already used that code to see how iPod integration with the desktop could be improved: it sets the volume.label HAL property to the name of the iPod as extracted from the iPod database which results in a nicer name for the iPod on your Nautilus desktop.

blogo ergo sum

After being kicked again and again (which hurts, the guy is a black belt in karate) by Dodji who wanted me to blog, here is a first post.

Album Artist support

These last days, I've been trying to get back to Rhythmbox development to scratch a few itches of my own.
First, I've looked at how Rhythmbox could handle compilations, ie albums containing tracks by different artists. Currently, if the album has 12 different artists, these 12 artists will appear separately in the artist list which can quickly create a big mess. I wrote a basic patch to make it possible to set an "album artist" for such albums and to use that instead of the multiple different artists in the artist list. I had to experiment a bit with various approaches, but in the end, the patch is surprisingly small.


Song UIDs

Then, I wanted Rhythmbox to be able to provide UIDs for the songs in its database. What I call an UID is some kind of identifier that is unique to a song and that can be generated by only looking at the song data. This can be useful for various things : iPod (or whatever your portable media player of choice is) synchronization, associating user data (rating, play count, ...) to a song which persists even if the user does a mv of the song from the shell, ... I learnt after doing that work that Charlotte had been looking for such a feature in Rhythmbox for her nice Rhythmbox SOC which was good news :)

To generate that UID, I chose to hash the song title, artist, album (read from the tags of the song) with the first 8kB of data of the file (actually, this hashing scheme was heavily inspired by what Amarok does). I'm not sure yet if this is the best way to uniquely identify a song, but we'll only know after people try to use it. Before you ask, I thought about using musicbrainz/musicDNS acoustic fingerprints but as far as I know, none of those fingerprints can be generated using free software end to end, there's always some closed source webservice that must be queried to get a fingerprint from a few parameters that were generated by analyzing the song audio data.

Once again, this feature was straightforward to implement.

The main issue I had was to debug the UID generation. Indeed, metadata reading (where I chose to add the UID generation) is done by a separate process which communicates with Rhythmbox through dbus. Reading metadata is basically equivalent to feeding random data to the tag reading library, so it's really hard to guarantee the library won't crash or hang in some corner cases. Using that external process allows Rhythmbox not to crash or hang if such an event should occur during metadata reading.

But this external process also makes debugging harder: it's short lived, spawned on-demand and run in the background (ie it's not possible to print stuff to stderr or stdout). So moch's help was really welcome since he explained me how to be able to run that metadata helper process by hand and to tell rhythmbox to use it. It's really simple, all you have to do is to (optionally) increase ATTENTION_SPAN in metadata/rb-metadata-dbus-service.c so that the helper stays alive longer (by default, it dies after 30 seconds of inactivity).
Then, you can run rhythmbox-metadata in nemiver (or in your favourite debugger), this will output a line like :

unix:abstract=/tmp/dbus-vXSVpHsnpL,guid=ba4e19b37904dba3bb1fc2214889d478

If you now set the RB_DBUS_METADATA_ADDRESS environment variable to that value before running Rhythmbox, then Rhythmbox will use the metadata helper you just launched in your debugger. Now all that is left to do is debugging!

The result of this work can be found in the uid branch of my Rhythmbox git tree. It still needs some polishing, but the basics should already be working (including automatically updating your database to add UIDs when you first run it).