My new Sony Vaio 11.6″ laptop runs an E-350 processor. I’ve upgraded the RAM to 8Gb and threw in a OCZ Vertex 2 SSD. Even using LUKS under LVM2, it feels insanely fast.

Everything I’ve tried works perfectly on it out of the box, including the card reader, HDMI port, Gigabit NIC, USB ports, sound card, wireless, Bluetooth, volume/brightness/video output function keys, etc. Well, almost everything… the graphics is one glaring exception.

For instance, when I go to log out or drop to a virtual terminal, I get all kinds of screen corruption and can’t see a thing. I always need to hit CTRL+ALT+F2 to switch to a virtual terminal, and CTRL+ALT+DEL to restart it to get picture back. Fortunately the SSD means that’s under a 30 second wait to shutdown and boot up again into the login screen, but after owning this machine for just over two weeks now it was starting to get old.

Further, 3D acceleration via Mesa3D was painful. Games like the Penumbra Collection ran at about 1 FPS. It looks like the Phoronix guys have tested newer versions with Gallium3D which look slightly more pleasing, however none of that’s in Debian and I don’t want to waste lots of time recompiling everything and potentially causing new updates to break. Further, while I could watch 720p video reasonably well, 1080p video would occasionally cause slowdowns and prevent the viewing experience from being completely enjoyable.

Time to do something I’d rather not… and install the proprietary driver. 11.3 is currently in the testing repos, however it still isn’t efficient enough to allow 1080p video to play properly. Downloading the new 11.5 Catalyst driver also fails to properly install due to a DKMS compilation error.

Fortunately, Mindwerks has instructions on how to fix this issue under Ubuntu 11.04. With a few little tweaks they can be made to work with Debian too.

Instructions:

Debian Wheezy (7.0) users can finally get fglrx playing nicely together with X.Org X Server 1.9.5. We can also make the latest driver work well with the 2.6.39 kernel.

The Minderks custom build procedure follows, adapted for Debian Wheezy (7.0) users. Note that the commands are for 64-bit, but the only change 32-bit users likely need to make is to download an i386 package instead for step 8. Also, the following commands assume you use the sudo package to gain root.

1. Install the latest 2.6.39 kernel revision from kernel.org. eg.

   $ wget http://www.kernel.org/pub/linux/kernel/v2.6/testing/linux-2.6.39-rc7.tar.bz2
   $ wget http://www.kernel.org/pub/linux/kernel/v2.6/testing/linux-2.6.39-rc7.tar.bz2.sign
   $ gpg --verify linux-2.6.39-rc7.tar.bz2.sign linux-2.6.39-rc7.tar.bz2
   $ tar jxf linux-2.6.39-rc7.tar.bz2
   $ cd linux-2.6.39-rc7
   $ make-kpkg -j2 --rootcmd fakeroot --initrd --bzimage kernel_headers kernel_image
   $ sudo dpkg -i ../*.deb

   and boot it.

2. Download the AMD Catalyst 11.5 64-bit driver installer.
3. Install the Debian VA-API packages. Note that some say to get them from the Splitted Desktop website, however I tried them and didn’t notice any benefit in doing so so reverted to the following:

   $ sudo apt-get install libva1 libva-x11-1 libva-tpi1 libva-glx1 libva-dev

4. Extract the files from the package:

   $ sh ./ati-driver-installer-11-5-x86.x86_64.run --extract ati

5. For 2.6.39 support, download this additional patch: 2.6.39_bkl.patch

   $ wget http://www.mindwerks.net/wp-content/uploads/2011/03/2.6.39_bkl.patch

6. Check for Big Kernel Lock usage:

   $ cat /lib/modules/$(uname -r)/build/.config | grep -c CONFIG_BKL=y

   If the result of this command is 0, then download no_bkl.patch as well. For stock kernels you should get 0 and will need the patch – which is probably the main reason you are here. :)

   $ wget http://www.mindwerks.net/wp-content/uploads/2011/03/no_bkl.patch

7. then apply them:

   $ cd ati; for i in ../*.patch; do patch -p1 < $i; done

8. Build your new ati/fglrx driver:

   $ sudo ./ati-installer.sh 8.85.6 --install

9. Since we’re using the proprietary drivers, we might as well make the most of them. Grab the latest XvBA backend library for VA-API:

   $ wget http://www.splitted-desktop.com/~gbeauchesne/xvba-video/xvba-video_0.7.8-1_amd64.deb
   $ dpkg -i xvba-video_0.7.8-1_amd64.deb
   $ cd /usr/lib/dri && sudo ln -s ../va/drivers/xvba_drv_video.so fglrx_drv_video.so

10. If your /etc/X11/xorg.conf is missing you will need to run:

    $ sudo aticonfig --initial

    and then reboot.

That newly created package should work for the entire 2.6.39 series.

These steps are really useful for AMD Fusion users at the moment. Without VA-API, multi-threaded mplayer will occupy 100% of available CPU (both cores) and drop frames when playing a test 1080p MKV file I have (containing H.264+AAC+AAS). It’s not unwatchable, but it’s annoying. With VA-API-patched mplayer, CPU usage never hits 15% when playing the same test video!

It’s a pity that the Catalyst driver doesn’t currently correctly generate deb packages for Debian, but the fglrx packages are not listed as a depenency for xvba-video so it’s not a major problem.

Update (2011-05-17, 12:41am): Using the new kernel is not necessary. Although there appear to be a few AMD Fusion fixes in there, I haven’t noticed any benefit in using 2.6.39-rc7 – possibly because I’m using the proprietary drivers. In fact, I’m not completely convinced the Catalyst 11.5 drivers are required either as I was having problems getting mplayer to use VA-API when I was testing 11.3. I’d be interested to know if just steps 3, 9 and 10 would be sufficient for hardware decoding using the patched mplayer and the default fglrx-atieventsd fglrx-control fglrx-driver fglrx-glx fglrx-glx-ia32 fglrx-modules-dkms package versions. I might very well have done much more than required.

Update (2011-05-22, 9:04pm): You do not need Catalyst version 11.5 – you can use the version in the repositories. In fact, I recommend it. The reason being (aside from having the drivers managed via apt) is that the compiling and installing 11.5 via the instructions above won’t enable 32-bit compatibility. I tried running a few games under WINE yesterday, but WINE always complained that OpenGL (and hence DirectX) was not available to it. Presumably I needed whatever the fglrx-glx-ia32 package included. Possibly I was just missing something as simple as a system link before, but I didn’t investigate – I’d rather use the standard Debian packages wherever possible if they do the job. Also, you don’t need to manually fetch the Splitted Desktop xvba-video package – xvba-va-driver is basically the same thing, only A) xvba-va-driver is in the official Debian repositories and B) it specifies the fglrx packages as a required dependency.

So in summary:

1. Don’t touch your kernel.

2. $ sudo apt-get install libva1 libva-x11-1 libva-tpi1 libva-glx1 libva-dev xvba-va-driver fglrx-atieventsd fglrx-control fglrx-driver fglrx-glx fglrx-glx-ia32 fglrx-modules-dkms

3. $ sudo aticonfig --initial
4. Reboot
5. Use the VA-API-patched mplayer for watching movies.

It’s easy when you know how. :)

Update (2011-07-03, 12:11am): If you run gnome-mplayer (perhaps so you can make use of gecko-mediaplayer for libva-accelerated video in Firefox), be aware that Debian currently ships no later than 0.9.9.2 – even in sid. Thus, you will encounter a problem with a filter gnome-mplayer automatically uses which is incompatible with vaapi – and will automatically disable it. I can verify that 1.0.4 avoids this issue, and allows gecko-mediaplayer to work wonders when combined with the ViewTube and YouTube HTML5 forever Greasemonkey scripts.