One way to help free software projects

Free software projects are always looking for new volunteers to help, new coders, new translators, new documentation writers, ... However, sometimes you want to contribute but can't become any of those, either because you lack time, because you are already involved in other projects, or because you feel you don't have time. For some projects, another way to help is through donations, here are 2 examples for projects I'm involved with.

Rhythmbox

Rhythmbox has had a plugin for Magnatune for quite a while now. In addition to not being evil (their website says so, it must be true! Joke aside, they give 50% of what they sell to the artist), they give 10% back to the Rhythmbox project for every purchase through its plugin.

And since they are not evil, today they sent a $600 check to the GNOME Foundation (which we chose as the recipient for the money). So you can buy lots of cool music on Magnatune while giving money to the GNOME Foundation at the same time!

Thanks a lot to John Buckman from Magnatune who was a really nice guy to interact with and to Adam Zimmerman for all his hard work on Rhythmbox Magnatune plugin.

libgpod

libgpod is the iPod handling library used by gtkpod, rhythmbox, amarok, songbird, ... It recently gained support for most of the latest devices released by Apple (iPod Nano 5th generation, iPhone, iPod Touch) which were unsupported under Linux until now (thanks a lot to Marcan, Nikias and all the people who helped with that by the way :)

However, this development was made harder by the lack of devices to test the code on, forcing us to look for testers with the right devices and going back and forth with bug reports and bug fixes until things work as expected. So donations to the project (even small amounts) are really helpful so that we can buy these missing devices and move things forward.
This already let us fund an iPod Nano which was a tremendous help to polish support for it in libgpod, thanks to everyone that made that possible. Next on the list are an iPod Touch to make sure the iPhoneOS support is up to par, and a buttonless iPod Shuffle which is so far unsupported by libgpod.

I've already mentioned it here, but if you have old iPods you no longer use , please get in touch, they can also be helpful for testing (thanks Götz!)

Adventures in udev land

I've mentionned a while ago the work that has been done on libgpod hal callout. It's working nicely, but with HAL being deprecated, I thought now might be a good time at looking at how to do things in the future, and to check if udev already lets us do what we do in the HAL callout. The good news is that it's working now and is pretty straightforward. However, I got stuck on a few details, so I thought it might be useful to others if I documented my findings here.

Currently, libgpod installs a .fdi file with the iPod vendor ID/device ID and a binary name. When HAL detects that a device that matches these 2 IDs is plugged in, it runs the binary. The binary issues a SCSI inquiry command to the iPod to get various information, and set some HAL properties using libhal.

My goal was to do something similar with udev, ie get udev to run a binary when an iPod is inserted and then associate some information with the iPod device in udev database so that other applications can access it.

iPod detection

Running a binary when the iPod is inserted wasn't too hard, it's done with a udev rule file (the format is documented in udev manpage, don't forget to read it if you have to write such a file! ) which goes to /lib/udev/rules.d. My first version was simple enough:

ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_USAGE}=="filesystem",

ATTRS{idVendor}=="05ac", ATTRS{idProduct}=="1204",

RUN+="/tmp/udev-test.sh"

udev-test.sh is a simple shell script wrapping the actual udev callout. It makes it easier to dump various information to a log file (for example the callout environment which udev uses to pass useful information to the callout). And, lo and behold, after plugging an iPod, my shell script was run!

Adding information to the udev database

The next logical step was to add some values read from the iPod to udev database so that other apps can get this additionnal information. And this is one of the steps that gave me some troubles. I was a bit ashamed of finding a really informative post by Kay Sievers answering my question right after having sent this email...

In short, it's really easy to import new values in the udev database, all you have to do is to output key/value pairs on stdout. This is nice, since udev passes information through environment variables and adds information to its database by reading stdout, this means that your callout doesn't have to depend in anyway on libudev.

I quickly modified my test program to print a few key/value pairs to add to the environment, triggered an unplug/replug of my ipod with udevadm, and watch the 'block' subsystem devices with the devkit binary. But I was really disappointed not to find my values associated with the iPod :(

After double checking everything and fiddling a bit to try to figure out what was wrong, I read again Kay's email, and I saw there was another difference between his code and mine: he is using an IMPORT rule to run his binary while I was still using a RUN rule.

I changed my udev rule to an IMPORT rule and.... it still didn't work :;)After staring at udevadm monitor output, I noticed that when the iPod was plugged in, there was first an "add" udev event for the iPod device shortly followed by a "change" event. Since my rule was only catching the "add" event, I hypothetized maybe my changes to the udev database were first properly added, and then overwritten by the "change" event. So I changed my rule file to catch "change" events in addition to "add" events, and it finally worked!

ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_FS_USAGE}=="filesystem",

ATTRS{idVendor}=="05ac", ATTRS{idProduct}=="1204",

IMPORT{program}="/tmp/udev-test.sh"

I was very happy with my udev callout, however I shortly realized that when going from a RUN rule to an IMPORT rule, udev no longer passed me the device name (/dev/sd??) in the DEVNAME environment variable. I went to #udev on freenode to check if this was the expected behaviour, and Kay confirmed this is normal because when IMPORT rules are run, the final device doesn't exist yet.

However, the $tempnode variable can be used as an argument to the binary that is being run to give it access to a temporary device node which can be interacted with. And indeed, after adding this argument to my udev rule, I could do everything that I wanted to :)

Final polish

After this successful experiment, all that remained to be done was to make the udev callout as featureful as the HAL callout. This was pretty straightforward, I abstracted the information gathering part from the HAL callout. This generic code then uses some backend-specific code to set the values. The HAL backend does that by using libhal, the udev backend does that by just outputting values to stdout.

David Zeuthen was (as always) really helpful by pointing me at the udev/devicekit equivalent for info.desktop.icon and info.desktop.name: DKD_PRESENTATION_NAME and DKD_PRESENTATION_ICON_NAME. I also cooked up some variable names in a LIBGPOD namespace to have an udev equivalent to the stuff provided by podsleuth, let me know if it could be useful in your projects, it can be changed to fit your needs :)

End result

And here is the end result after plugging in my iPod:

# udevadm info --query=env --name=sdb2

DKD_PRESENTATION_ICON_NAME=multimedia-player-apple-ipod-color
LIBGPOD_VERSION=1
LIBGPOD_IS_UNKNOWN=0
LIBGPOD_FIREWIRE_ID=000A270002BAD546
LIBGPOD_SERIAL_NUMBER=JQ446FN4R5Q
LIBGPOD_FIRMWARE_VERSION=1.2.1
LIBGPOD_IMAGES_ALBUM_ART_SUPPORTED=1
LIBGPOD_IMAGES_PHOTOS_SUPPORTED=1
LIBGPOD_IMAGES_CHAPTER_IMAGES_SUPPORTED=1
LIBGPOD_DEVICE_CLASS=color
LIBGPOD_MODEL_GENERATION=4.000000
LIBGPOD_MODEL_SHELL_COLOR=white
LIBGPOD_PRODUCTION_FACTORY_ID=JQ
LIBGPOD_PRODUCTION_YEAR=2004
LIBGPOD_PRODUCTION_WEEK=46
LIBGPOD_PRODUCTION_INDEX=20272
LIBGPOD_MODEL_CONTROL_PATH=/iPod_Control


Et voilà! The code is available from the devicekit branch of my libgpod git repo
and will be committed to libgpod svn soon.

First news of 2009 :)

New job

In my last post, I was looking for a job. The good news is that I started working at Mandriva 2 weeks ago. I'll be working on the core distro to (partially) replace pixel which means I'll hack on urpmi and bootloaders among tons of other things. Thanks to everyone who helped me with job opportunities!

libgpod

After far too much time, we finally released libgpod 0.7. It contains tons of improvements, the most noticeable being support for the latest Nano and iPod Classic, but there were lots of other improvements: writing of compact artwork files making the iPod more responsive, chapter data support, better iPod model detection, updated python bindings, improved API documentation, ...

There aren't many changes in this release for iPhone/iPod Touch support. However, in addition to the great work from the iFuse team, marcan figured out how to add songs to a jailbroken iPhone using amarok/gtkpod/rhythmbox/. If you add to that the iPod Touch is on its way to being jailbroken, this means that people really insisting on buying those devices will at least be able to use them without iTunes.

FOSDEM

FOSDEM is only a few weeks away now (7th and 8th of February), the GNOME and Freedesktop devrooms are booked and their schedule has been published. You should definitely plan a trip in Brussels if it's not done already :) And don't forget our group picture on Saturday if you are a GNOME hacker!

Autumn update

Rhythmbox

In the last weeks, I finally landed a few patches for Rhythmbox iPod plugins that had been waiting for too long.

The first one adds a dialog to setup new (empty) ipods, courtesy of gtkpod. This was bug #358029 and the initial patch was made by James Doc Livingston.

The second patch was written by Ed Catmur and Paul Drain. Its purpose is to avoid copying duplicate tracks to the ipod. It's currently a bit simplistic, 2 songs are considered duplicates if they have the same title/album/artist. And the code doing that is factored in a generic function in the RBRemovableMediaSource class so other DAP sources can easily get the same behaviour. The fact that the duplicate check is made in a single function also means that we can easily replace this check with a more elaborated one (using audio fingerprints for example) in the future.

The 3rd (and last) patch has been made by John Daiker and adds a property dialog to the iPod source to see how much space is available on the ipod, to see the ipod serial number, device name, ... This code is specific to the iPod source for now, but seeing bug #561955 which asks for the same feature for MTP devices, there is probably some factoring to be done...

And, more or less related to this last bug, in the last days I worked on porting the Banshee SegmentedBar widget to C to be able to reuse it in Rhythmbox. After writing the GObject boilerplate for a new GtkWidget, the porting was pretty straightforward since Cairo/Pango mono bindings are really close to their C counterparts. The end result can be seen on this screenshot :



The code can be found in bug #561955 as a patch to Rhythmbox source. However, the widget code doesn't depend on any Rhythmbox code which means it should be easy to reuse in any GTK+/C project if you need such a widget. If you do so, please let me know about any bug or API issues you encounter since the code is really young and hasn't been tested much :)


Libgpod

On the libgpod front, not too many news. svn trunk is in a satisfying state and is probably releasable as is. The lastest ipod nanos and ipod classic are supported (with artwork and photos). As always, any testing with the latest ipods is welcome. We are also missing a few serial numbers for the new 16GB nanos, so if you own one, it would be great if you could send us the last 3 digits of its serial number (printed on the back) as well as its color.

The iphone 3g and the ipod touch with 2.x firmwares are not supported at this time, the song database needs to be checksummed to be valid, and noone has reverse engineered the hashing scheme yet. Moreover, Apple is making things difficult through code obfuscation and legal hurdles.

UPDATE: the EFF published this really interesting analysis of the issue

On a brighter side, the iFuse guys have made great progress to make USB communication to non jail-broken devices to work. Before their work, one had to jailbreak his phone and to talk to it through wifi to do anything with it. Now that's no longer necessary thanks to those guys.


Personal life

After a few years working as a team leader on Anevia's Video-on-Demand server, it's time for me to move on and to look for new exciting job opportunities. So if you are looking for an experienced C/C++ Linux developper, you are very welcome to contact me. My resume is available here.

libgpod callout improvements

The other day, I mentioned that libgpod hal callout was setting volume.label to the iPod name to get a nicer name displayed for iPods in Nautilus (among other apps). What I didn't say is that I wasn't really sure that overriding that property with something that has nothing to do with the actual filesystem label was such a good idea.

And it wasn't. After asking David Zeuthen about that on IRC, he kindly told me about info.desktop.name which was added explicitly for that purpose. And I also learnt about info.desktop.icon which is to be preferred to info.icon_name. So I made these 2 changes and pushed them to the podsleuth branch of my git repository.

While I was at it, I worked on the few things that are still in the way to a libgpod 0.7 release, ie I cleaned up the exported symbols to make sure what we export make sense from an API point of view, and I added some missing API doc and made a few fixes in the existing one (some functions were renamed and the API doc wasn't properly updated).


On an unrelated note, I'm glad to see that some people find this blog worth some comments, thanks ;)

And my other project is...

After describing my latest work on Rhythmbox yesterday, here's what I did on libgpod in the last month. libgpod is a cross-platform library used by many different projects (amarok, gtkpod, rhythmbox, songbird to name a few) to access and modify your ipod content.

Extensive SysInfoExtended parsing

As has been known for a while now, the iPod can be queried about its capabilities using SCSI commands and returns XML data describing the iPod (serial number, firmware version, ...) and what it can do can do (podcast support, video formats supported, image formats that it knows how to display, ...). When we released libgpod 0.6, we introduced a hal callout to send the appropriate SCSI query to the iPod and to dump the returned XML data to a file that we named SysInfoExtended. Normal users aren't guaranteed to be able to send raw SCSI commands to a device, hence the use of a hal callout and the dumping of the information to a regular file. However, in libgpod 0.6, we only had a very basic parser for that file which only knew how to read the only SysInfoExtended field we needed. Most of the information about the iPod capabilities was hard-coded into per-ipod model tables, and libgpod had to be told the iPod model before being able to (for example) being able to write artwork to an iPod.

For the next release of libgpod, I decided that we had to be able to use the information from SysInfoExtended to its fullest. I started by writing a generic plist (which is the XML subset SysInfoExtended is in) to GValue parser using libxml instead of GMarkup. Then, I extracted the data I was interested in from the GValue collection the parser gave me to a nice C struct. To make the addition of new fields easy, most of the work is driven from a table indicating the field name in the plist file, the type we want to assign that field data and the offset we want to put that data in the resulting struct. Modifying that table and the struct definition are the only things that need to be done if we want to read additionnal fields from SysInfoExtended.

With that being done , I had everything I needed to have libgpod use the information provided by the device to write artwork to the iPod instead of relying on hard-coded tables. Some refactoring was needed to make it possible to use the artwork data from the iPod (there were some assumptions here and there that the formats supported by the iPod were known at compile time) but now that it's done, the code feels much more natural and maintainable than before.

Getting the iPod model from SysInfoExtended

With the aforementioned work, writing artwork to the iPod has been made much more flexible, but libgpod was still unable to automatically guess the iPod model/color/... from the device without asking the user. This is due to the fact that to do that, libgpod relied on the iPod "ModelNum" which used to be present in a file on the iPod filesystem but for quite some time now, the only way to get that model number is to read it on the iPod box, which is not really easy to do from software :)

But for all recent iPod models, there's another way to guess the iPod model, this is by parsing the iPod serial number. And this serial number is precisely one of the things that we can read from SysInfoExtended! So all we had to do to be able to automatically detect the model/color/.. of a plugged iPod was to properly parse the iPod serial number and to infer the iPod physical features from that serial number, just as what podsleuth does with that table.

Podsleuth

Given that libgpod had already installed a hal callout, and after all the work done to parse the SysInfoExtended data, I realized that libgpod had gathered all the pieces to build a podsleuth clone, so I decided to try to write one just for fun and to test the new API added to libgpod in real-world situations. This led to the work which can be found in the podsleuth branch of my libgpod git repository.

Even though I haven't tested it against banshee, I compared the properties exported by podsleuth and by this experimental stuff, so this code can probably be used as a drop-in replacement to podsleuth. Writing it also made me realize that podsleuth doesn't export enough information about artwork formats compared to what libgpod needs. I'm also not a big fan of how podsleuth exposes the artwork formats: it parses the XML to get the artwork data to immediatly serialize it again to a string. It's then up to the app using podsleuth to parse that string (again) to get the artwork formats supported by the ipod.

Anyway, since I now have hacked this nice tool, it's now up to me to experiment a bit with all of that and to make suggestions as to how things could be improved :) By the way, I already used that code to see how iPod integration with the desktop could be improved: it sets the volume.label HAL property to the name of the iPod as extracted from the iPod database which results in a nicer name for the iPod on your Nautilus desktop.