iPhone syncing on Linux, part 2

Last post was more along the lines of an announcement, so here’s a more concrete guide. There have been new releases of most parts of the software stack in the past few days, so now is the time to grab them if you’re interested. Libgpod is the exception, since it’s still being worked on to get proper support for the iPhone OS 3.x database. That said, if you’re interested in being an early adopter of iPhone sync support and helping us test it, here’s how. If you are a new or inexperienced user, or you are not used to compiling things from source, editing configuration files, and doing some basic troubleshooting, stop here. This isn’t ready for you. In the not too distant future, distros will package this, it will be delivered with your usual dose of healthy and working open source software, and it will all work automagically as soon as you plug in your device. A few notes:

• Keep in mind that if you’re using distro packages for anything I’m going to mention here, you’ll need the development headers too (-dev package or similar).
• When linking to packages below, I’ll link the name to the homepage and the version to a download link or something close to it: if you’re reading this post soon after it was posted, then the versions should be up to date. If it’s been more than a few days, you should check for newer versions.
• You’ll also need cmake to build some things (2.6 or newer); your distro should provide a package for it.
• Most packages will install to /usr/local by default. This should be fine, but you may have to uninstall any older distro packages that live at /usr, and apps may or may not pick up libraries on /usr/local. If you want, you can change the prefix to /usr. You should already know how to do this for autotools-based packages. For CMake-based ones, pass something like -DCMAKE_INSTALL_PREFIX=/usr when calling cmake.

• The libiphone/ifuse page has links to distro repositories with packaged versions of everything here. Yay, you’ve just saved a bunch of time! Do follow the steps below, but you can use these packages instead of building from source. Possible pitfalls:

• If you’ve had prior versions of usbmuxd/libiphone/etc, make sure they’re gone, especially any udev rules files in /etc/udev/rules.d.

• If you’ve been syncing via SSH, and you log in as root, your permissions might be messed up. Run chown -R mobile /var/mobile/Media on the phone, and make sure you don’t get any permission errors when changing files via an ifuse mount.

• There’s a udev bug that causes usbmuxd not to respond to signals, which means it will never quit if you remove all devices, and it might not detect any new devices after you’ve plugged in the one that caused it to start up. As a workaround, kill -9 it, and run it from the console. A workaround for this bug is also in the usbmuxd Git repo.

Step 1

Install libusb-1.0.3 (or newer). This should be straightforward. If your distro carries it, that’s fine, just make sure it’s 1.0.3 at least (if it isn’t, the repos I mentioned above probably have it).

Step 2

Install usbmuxd-1.0.0. Make sure you read the README file, especially the part about creating a user named usbmux with USB access permissions (does not apply if you’re using distro packages, hopefully). After installing usbmuxd, you should increase the syslog debug level by editing /lib/udev/rules.d/85-usbmuxd.rules and adding -v -v flags to the end of both RUN statements, like this: RUN+="/usr/sbin/usbmuxd -u -U -v -v" (and similarly for the other line). This will get you useful debug information in syslog without flooding you with messages. Have syslog open (tail -f, xconsole, whatever). Plug in (or replug) your device. You should see something like this:

usb 3-4.3: new high speed USB device using ehci_hcd and address 68
[...]
usbmuxd[12705]: [3] usbmux v1.0.0 starting up
usbmuxd[12707]: [4] Creating socket
usbmuxd[12707]: [3] Successfully dropped privileges to 'usbmux'
usbmuxd[12707]: [4] Initializing USB
usbmuxd[12707]: [4] Found new device with v/p 05ac:1290 at 3-68
usbmuxd[12707]: [4] Using wMaxPacketSize=512 for device 3-68
usbmuxd[12707]: [3] Connecting to new device on location 0x30044 as ID 1
usbmuxd[12707]: [4] 1 device detected
usbmuxd[12707]: [3] Initialization complete
usbmuxd[12707]: [3] Connected to v1.0 device 1 on location 0x30044 with serial number be2975afb30b6db9025f95261b9e0a7041044661

If you don’t see anything related to usbmuxd, either you forgot the -v -v or you need to reload your udev rules: $ sudo udevadm control --reload-rules. As a last resort, try rebooting. If it still doesn’t work, something’s wrong. You can continue by running usbmuxd manually as explained in the README, but you should report a bug. If you have a jailbroken phone with SSH, try running $ iproxy 2222 22 and then SSH to localhost, port 2222 on another console. You should be able to SSH into your phone. Unplug your phone. If usbmuxd was started via udev (not manually), it should quit. If it doesn’t, report a bug.

Step 3

Install [libplist-0.16 and then libiphone-0.9.4 and ifuse-0.9.4. Now you should be able to mount your phone by simply running $ ifuse <mountpoint>. You should be able to see the media folder structure and even grab photos and other stuff. If you have a jailbroken phone, you can use the \--root option to mount the root filesystem with root privileges (I usually prefer ssh/sshfs for this, though). Do NOT use the \--root option to sync music, it will not work and may mess up your permissions. If you have access to more than one iPhone or iPod Touch, we could use some multiple-device testing! Usbmuxd should happily start up when you plug in the first device, keep running while you plug in any other devices, and stop once all devices have been removed. Usbmuxd will report the device UUIDs (it calls them “serial numbers”) as they are connected (if you’re using -v -v). You should be able to pass the \--uuid=<uuid> option to ifuse to mount a specific device. Please test this for us!

Intermission

If you are not used to alpha software, checking out development sources, testing patches, and working with developers, stop here. Now you’ve got all the underlying stuff working, so you’re probably better off waiting for some time until libgpod stabilizes some and things are merged and tested. If you are a developer willing to help, a very enthusiastic power user, or simply a masochist, feel free to read on. Be warned: pain lies ahead, don’t even think about using the following for day-to-day use at this point in time. From here on there are no distro packages and no nice source tarballs. Experiments and feedback only. This Will Eat Your Music Collection For Dinner And Replace It With Hours And Hours Of .

Step 4

Clone teuf’s sandbox Git repository for libgpod, and switch to the iphone30 branch the libgpod git repository at SourceForge, then compile and install it:

$ git clone git://gtkpod.git.sourceforge.net/gitroot/gtkpod/libgpod
$ cd libgpod
$ CFLAGS="-g -O0" sh autogen.sh --prefix=/usr
$ make
$ sudo make install

Step 5

Mount the device and create the iTunes_Control/Device directory. Then, get your UUID (it should be in the syslog from usbmuxd, or you can find it by running $ lsusb -v | grep -i iSerial). It should be 40 characters long. Then, run $ tools/ipod-read-sysinfo-extended <uuid> <mountpoint>. This should generate a file named iTunes_Control/Device/SysInfoExtended. Make sure it’s not empty and whatnot; it should be a large-ish plist (XML file) with a bunch of info.

Step 6

Make sure your device already contains a valid music database created by iTunes. If it doesn’t, drop me a line on IRC and we can try some cool new stuff. New databases can be created, but there might be issues and you need some way of generating the HashInfo file (more on this later).

Step 7

Open an application that uses libgpod to manage iPods. Gtkpod is recommended at this stage, as its what I use to test. Other apps (Rhythmbox, Amarok, etc) theoretically should work. Theoretically, because Amarok 1.4 crashes/hangs pretty badly for me, for example, and I’m not sure who is to blame. I’d like to hear any success or failure stories. Please run your app from a console so you can see any debug messages.

First, load your device. Once the application has accessed the music library, a new file called HashInfo should have popped up in the iTunes_Control/Device directory. If it isn’t there, you’ve got a problem: check stdout for any messages and and preferably ping me on IRC or at least e-mail me your UUID and your iTunes-created iTunes_Control/iTunes/iTunesCDB file (it shouldn’t have been modified yet).

Make some changes (add a song, make a playlist, etc), save, and cross your fingers. With any luck, you’ll get some debug spew, any songs will transfer, the iPhone/iPod will show a pretty “syncing” screen, the database will be written, the “syncing” screen will go away, and you’ll be able to launch the iPod application and see your changes and play any new songs. There will be bugs at this stage, but please do report any issues along with expected behavior.

Step 8

Plug your device into a computer running iTunes (unmount it on your Linux computer first!) and check that it can see the changes made with libgpod. This is very important, because iTunes and the device use different databases: the data managed by iTunes and libgpod is maintained in an iTunesCDB (a compressed version of the old, classic iTunesDB format), but it is converted to a new SQLite database format for the iPod app to use. So both libgpod and iTunes read the iTunesCDB, but they write out the iTunesCDB and the SQLite databases. You should be able to make changes in iTunes and swap your device from iTunes to libgpod and back and see them, etc.

Bugs

Tons. The only things tested to work are managing simple music and working with simple playlists. There’s a known annoying bug where the iTunes special playlists (Ringtones/etc) are converted to normal playlists by libgpod, and then iTunes recreates them if you use it again, creating duplicates each time you switch between using libgpod and iTunes. The DB schema is outdated (3.0 instead of 3.1), so you’ll see a (hopefully short) “Updating” step when you launch the iPod app. Please report issues, but keep in mind that they’re to be expected. This is very very very alpha code. I probably screwed up something on this post even, so chances are you’ll get stuck somewhere. Tough luck, it’s 7am and I need to catch some sleep 

Hope you had fun! Keep in mind that you can usually find us on #gtkpod on freenode if you need to report an issue. Comments on this post are fine too, for now.