July 21, 2016

Using modern gettext

gettext has seen quite some enhancements in recent years, after Daiki Ueno started maintaining it. It can now extract (and merge back) strings from diverse file formats, including many of the formats that are important for desktop applications. With gettext 0.19.8, there is really no need  anymore to use intltool or GLib’s dated gettext glue (AM_GLIB_GNU_GETTEXT and glib-gettextize).

Since intltool still sticks around in quite a few projects, I thought that I should perhaps explain some of the dos and don’ts for how to get by with plain gettext. Javier Jardon has been tirelessly fighting a battle for using upstream gettext, maybe this will help him reaching the finish line with that project.

Extracting strings

xgettext is the tool used to extract strings from sources into .pot files.

In addition to programming languages such as C, C++, Java, Scheme, etc, it recognizes the following files by their typical file extensions (and it is smart enough to disregard a .in extension):

    • Desktop files: .desktop
    • GSettings schemas: .gschema.xml
    • GtkBuilder ui files: .ui
    • Appdata files: .appdata.xml and .metainfo.xml

You can just add these files to POTFILES.in, without the extra type hints that intltool requires.

One important advantage of xgettext’s xml support, compared to intltool, is that you can install .in files that are valid; no more tag mutilation like <_description> required.

Merging translations

The trickier part is merging translations back into the various file formats. Sometimes that is not necessary, since the file has a reference to the gettext domain, and consumers know to use gettext at runtime: that is the case for GSettings schemas and GtkBuilder ui files, for example.

But in other cases, the translations need to be merged back into  the original file before installing it. In these cases, the original file from which the strings are extracted often has an extra .in extension. The tool that is doing this task is msgfmt.

Intltool installs autotools glue which can define make rules for many of these cases, such as @INTLTOOL_DESKTOP_RULE@. Gettext does not provide this kind of glue, but the msgfmt tool is versatile enough that you can write your own rules fairly easily, for example:

%.desktop: %.desktop.in
        msgfmt --desktop -d $(top_srcdir)/po \
               --template $< -o $@

Extending gettext

Gettext can be extended to understand new xml formats. To do so, you install .its and .loc files. The syntax for these files is explained in detail in the gettext docs. Libraries are expected to install these files for ‘their’ formats (GLib and GTK+ already do, and  PolicyKit will do the same soon.

If you don’t want to wait for your favorite format to come with built-in its support, you can also include its files with your application; gettext will look for such files in $XDG_DATA_DIRS/gettext/its/.

 

July 20, 2016

libinput is done

Don't panic. Of course it isn't. Stop typing that angry letter to the editor and read on. I just picked that title because it's clickbait and these days that's all that matters, right?

With the release of libinput 1.4 and the newest feature to add tablet pad mode switching, we've now finished the TODO list we had when libinput was first conceived. Let's see what we have in libinput right now:

  • keyboard support (actually quite boring)
  • touchscreen support (actually quite boring too)
  • support for mice, including middle button emulation where needed
  • support for trackballs including the ability to use them rotated and to use button-based scrolling
  • touchpad support, most notably:
    • proper multitouch support on touchpads [1]
    • two-finger scrolling and edge scrolling
    • tapping, tap-to-drag and drag-lock (all configurable)
    • pinch and swipe gestures
    • built-in palm and thumb detection
    • smart disable-while-typing without the need for an external process like syndaemon
    • more predictable touchpad behaviours because everything is based on physical units [2]
    • a proper API to allow for kinetic scrolling on a per-widget basis
  • tracksticks work with middle button scrolling and communicate with the touchpad where needed
  • tablet support, most notably:
    • each tool is a separate entity with its own capabilities
    • the pad itself is a separate entity with its own capabilities and events
    • mode switching is exported by the libinput API and should work consistently across callers
  • a way to identify if multiple kernel devices belong to the same physical device (libinput device groups)
  • a reliable test suite
  • Documentation!
The side-effect of libinput is that we are also trying to fix the rest of the stack where appropriate. Mostly this meant pushing stuff into systemd/udev so far, with the odd kernel fix as well. Specifically the udev bits means we
  • know the DPI density of a mouse
  • know whether a touchpad is internal or external
  • fix up incorrect axis ranges on absolute devices (mostly touchpads)
  • try to set the trackstick sensitivity to something sensible
  • know when the wheel click is less/more than the default 15 degrees
And of course, the whole point of libinput is that it can be used from any Wayland compositor and take away most of the effort of implementing an input stack. GNOME, KDE and enlightenment already uses libinput, and so does Canonical's Mir. And some distribution use libinput as the default driver in X through xf86-input-libinput (Fedora 22 was the first to do this). So overall libinput is already quite a success.

The hard work doesn't stop of course, there are still plenty of areas where we need to be better. And of course, new features come as HW manufacturers bring out new hardware. I already have touch arbitration on my todo list. But it's nice to wave at this big milestone as we pass it into the way to the glorious future of perfect, bug-free input. At this point, I'd like to extend my thanks to all our contributors: Andreas Pokorny, Benjamin Tissoires, Caibin Chen, Carlos Garnacho, Carlos Olmedo Escobar, David Herrmann, Derek Foreman, Eric Engestrom, Friedrich Schöller, Gilles Dartiguelongue, Hans de Goede, Jackie Huang, Jan Alexander Steffens (heftig), Jan Engelhardt, Jason Gerecke, Jasper St. Pierre, Jon A. Cruz, Jonas Ådahl, JoonCheol Park, Kristian Høgsberg, Krzysztof A. Sobiecki, Marek Chalupa, Olivier Blin, Olivier Fourdan, Peter Frühberger, Peter Hutterer, Peter Korsgaard, Stephen Chandler Paul, Thomas Hindoe Paaboel Andersen, Tomi Leppänen, U. Artie Eoff, Velimir Lisec.

Finally: libinput was started by Jonas Ådahl in late 2013, so it's already over 2.5 years old. And the git log shows we're approaching 2000 commits and a simple LOCC says over 60000 lines of code. I would also like to point out that the vast majority of commits were done by Red Hat employees, I've been working on it pretty much full-time since 2014 [3]. libinput is another example of Red Hat putting money, time and effort into the less press-worthy plumbing layers that keep our systems running. [4]

[1] Ironically, that's also the biggest cause of bugs because touchpads are terrible. synaptics still only does single-finger with a bit of icing and on bad touchpads that often papers over hardware issues. We now do that in libinput for affected hardware too.
[2] The synaptics driver uses absolute numbers, mostly based on the axis ranges for Synaptics touchpads making them unpredictable or at least different on other touchpads.
[3] Coincidentally, if you see someone suggesting that input is easy and you can "just do $foo", their assumptions may not match reality
[4] No, Red Hat did not require me to add this. I can pretty much write what I want in this blog and these opinions are my own anyway and don't necessary reflect Red Hat yadi yadi ya. The fact that I felt I had to add this footnote to counteract whatever wild conspiracy comes up next is depressing enough.

July 19, 2016

radv: initial hacking on a vulkan driver for AMD VI GPUs
(email sent to mesa-devel list).

I was waiting for an open source driver to appear when I realised I should really just write one myself, some talking with Bas later, and we decided to see where we could get.

This is the point at which we were willing to show it to others, it's not really a vulkan driver yet, so far it's a vulkan triangle demos driver.

It renders the tri and cube demos from the vulkan loader,
and the triangle demo from Sascha Willems demos
and the Vulkan CTS smoke tests (all 4 of them one of which draws a triangle).

There is a lot of work to do, and it's at the stage where we are seeing if anyone else wants to join in at the start, before we make too many serious design decisions or take a path we really don't want to.

So far it's only been run on Tonga and Fiji chips I think, we are hoping to support radeon kernel driver for SI/CIK at some point, but I think we need to get things a bit further on VI chips first.

The code is currently here:
https://github.com/airlied/mesa/tree/semi-interesting

There is a not-interesting branch which contains all the pre-history which might be useful for someone else bringing up a vulkan driver on other hardware.

The code is pretty much based on the Intel anv driver, with the winsys ported from gallium driver,
and most of the state setup from there. Bas wrote the code to connect NIR<->LLVM IR so we could reuse it in the future for SPIR-V in GL if required. It also copies AMD addrlib over, (this should be shared).

Also we don't do SPIR-V->LLVM direct. We use NIR as it has the best chance for inter shader stage optimisations (vertex/fragment combined) which neither SPIR-V or LLVM handles for us, (nir doesn't do it yet but it can).

If you want to submit bug reports, they will only be taken seriously if accompanied by working patches at this stage, and we've no plans to merge to master yet, but open to discussion on when we could do that and what would be required.
GUADEC Flatpak contest
I will be presenting a lightning talk during this year's GUADEC, and running a contest related to what I will be presenting.

Contest

To enter the contest, you will need to create a Flatpak for a piece of software that hasn't been flatpak'ed up to now (application, runtime or extension), hosted in a public repository.

You will have to send me an email about the location of that repository.

I will choose a winner amongst the participants, on the eve of the lightning talks, depending on, but not limited to, the difficulty of packaging, the popularity of the software packaged and its redistributability potential.

You can find plenty of examples (and a list of already packaged applications and runtimes) on this Wiki page.

Prize

A piece of hardware that you can use to replicate my presentation (or to replicate my attempts at a presentation, depending ;). You will need to be present during my presentation at GUADEC to claim your prize.

Good luck to one and all!

July 18, 2016

REMINDER! systemd.conf 2016 CfP Ends in Two Weeks!

Please note that the systemd.conf 2016 Call for Participation ends in less than two weeks, on Aug. 1st! Please send in your talk proposal by then! We’ve already got a good number of excellent submissions, but we are interested in yours even more!

We are looking for talks on all facets of systemd: deployment, maintenance, administration, development. Regardless of whether you use it in the cloud, on embedded, on IoT, on the desktop, on mobile, in a container or on the server: we are interested in your submissions!

In addition to proposals for talks for the main conference, we are looking for proposals for workshop sessions held during our Workshop Day (the first day of the conference). The workshop format consists of a day of 2-3h training sessions, that may cover any systemd-related topic you'd like. We are both interested in submissions from the developer community as well as submissions from organizations making use of systemd! Introductory workshop sessions are particularly welcome, as the Workshop Day is intended to open up our conference to newcomers and people who aren't systemd gurus yet, but would like to become more fluent.

For further details on the submissions we are looking for and the CfP process, please consult the CfP page and submit your proposal using the provided form!

And keep in mind:

REMINDER: Please sign up for the conference soon! Only a limited number of tickets are available, hence make sure to secure yours quickly before they run out! (Last year we sold out.) Please sign up here for the conference!

AND OF COURSE: We are also looking for more sponsors for systemd.conf! If you are working on systemd-related projects, or make use of it in your company, please consider becoming a sponsor of systemd.conf 2016! Without our sponsors we couldn't organize systemd.conf 2016!

Thank you very much, and see you in Berlin!

July 15, 2016

Why synclient does not work anymore

More and more distros are switching to libinput by default. That's a good thing but one side-effect is that the synclient tool does not work anymore [1], it just complains that "Couldn't find synaptics properties. No synaptics driver loaded?"

What is synclient? A bit of history first. Many years ago the only way to configure input devices was through xorg.conf options, there was nothing that allowed for run-time configuration. The Xorg synaptics driver found a solution to that: the driver would initialize a shared memory segment that kept the configuration options and a little tool, synclient (synaptics client), would know about that segment. Calling synclient with options would write to that SHM segment and thus toggle the various options at runtime. Driver and synclient had to be of the same version to know the layout of the segment and it's about as secure as you expect it to be. In 2008 I added input device properties to the server (X Input Extension 1.5 and it's part of 2.0 as well of course). Rather than the SHM segment we now had a generic API to talk to the driver. The API is quite simple, you effectively have two keys (device ID and property number) and you can set any value(s). Properties literally support just about anything but drivers restrict what they allow on their properties and which value maps to what. For example, to enable left-finger tap-to-click in synaptics you need to set the 5th byte of the "Synaptics Tap Action" property to 1.

xinput, a commandline tool and debug helper, has a generic API to change those properties so you can do things like xinput set-prop "device name" "property name" 1 [2]. It does a little bit under the hood but generally it's pretty stupid. You can run xinput set-prop and try to set a value that's out of range, or try to switch from int to float, or just generally do random things.

We were able to keep backwards compatibility in synclient, so where before it would use the SHM segment it would now use the property API, without the user interface changing (except the error messages are now standard Xlib errors complaining about BadValue, BadMatch or BadAccess). But synclient and xinput use the same API to talk to the server and the server can't tell the difference between the two.

Fast forward 8 years and now we have libinput, wrapped by the xf86-input-libinput driver. That driver does the same as synaptics, the config toggles are exported as properties and xinput can read and change them. Because really, you do the smart work by selecting the right property names and values and xinput just passes on the data. But synclient is broken now, simply because it requires the synaptics driver and won't work with anything else. It checks for a synaptics-specific property ("Synaptics Edges") and if that doesn't exists it complains with "Couldn't find synaptics properties. No synaptics driver loaded?". libinput doesn't initialise that property, it has its own set of properties. We did look into whether it's possible to have property-compatibility with synaptics in the libinput driver but it turned out to be a huge effort, flaky reliability at best (not all synaptics options map into libinput options and vice versa) and the benefit was quite limited. Because, as we've been saying since about 2009 - your desktop environment should take over configuration of input devices, hand-written scripts are dodo-esque.

So if you must insist on shellscripts to configure your input devices use xinput instead. synclient is like fsck.ext2, on that glorious day you switch to btrfs it won't work because it was only designed with one purpose in mind.

[1] Neither does syndaemon btw but it's functionality is built into libinput so that doesn't matter.
[2] xinput set-prop --type=int --format=32 "device name" "hey I have a banana" 1 2 3 4 5 6 and congratulations, you've just created a new property for all X clients to see. It doesn't do anything, but you could use those to attach info to devices. If anything was around to read that.

July 14, 2016

xinput resolves device names and property names

xinput is a commandline tool to change X device properties. Specifically, it's a generic interface to change X input driver configuration at run-time, used primarily in the absence of a desktop environment or just for testing things. But there's a feature of xinput that many don't appear to know: it resolves device and property names correctly. So plenty of times you see advice to run a command like this:


xinput set-prop 15 281 1
This is bad advice, it's almost impossible to figure out what this is supposed to do, it depends on the device ID never changing (spoiler: it will) and the property number never changing (spoiler: it will). Worst case, you may suddenly end up setting a different property on a different device and you won't even notice. Instead, just use the built-in name resolution features of xinput:

xinput set-prop "SynPS/2 Synaptics TouchPad" "libinput Tapping Enabled" 1
This command will work regardless of the device ID for the touchpad and regardless of the property number. Plus it's self-documenting. This has been possible for many many years, so please stop using the number-only approach.

July 11, 2016

libinput and graphics tablet mode support

In an earlier post, I explained how we added graphics tablet pad support to libinput. Read that article first, otherwise this article here will be quite confusing.

A lot of tablet pads have mode-switching capabilities. Specifically, they have a set of LEDs and pressing one of the buttons cycles the LEDs. And software is expected to map the ring, strip or buttons to different functionality depending on the mode. A common configuration for a ring or strip would be to send scroll events in mode 1 but zoom in/out when in mode 2. On the Intuos Pro series tablets that mode switch button is the one in the center of the ring. On the Cintiq 21UX2 there are two sets of buttons, one left and one right and one mode toggle button each. The Cintiq 24HD is even more special, it has three separate buttons on each side to switch to a mode directly (rather than just cycling through the modes).

In the upcoming libinput 1.4 we will have mode switching support in libinput, though modes themselves have no real effect within libinput, it is merely extra information to be used by the caller. The important terms here are "mode" and "mode group". A mode is a logical set of button, strip and ring functions, as interpreted by the compositor or the client. How they are used is up to them as well. The Wacom control panels for OS X and Windows allow mode assignment only to the strip and rings while the buttons remain in the same mode at all times. We assign a mode to each button so a caller may provide differing functionality on each button. But that's optional, having a OS X/Windows-style configuration is easy, just ignore the button modes.

A mode group is a physical set of buttons, strips and rings that belong together. On most tablets there is only one mode group but tablets like the Cintiq 21UX2 and the 24HD have two independently controlled mode groups - one left and one right. That's all there is to mode groups, modes are a function of mode groups and can thus be independently handled. Each button, ring or strip belongs to exactly one mode group. And finally, libinput provides information about which button will toggle modes or whether a specific event has toggled the mode. Documentation and a starting point for which functions to look at is available in the libinput documentation.

Mode switching on Wacom tablets is actually software-controlled. The tablet relies on some daemon running to intercept button events and write to the right sysfs files to toggle the LEDs. In the past this was handled by e.g. a callout by gnome-settings-daemon. The first libinput draft implementation took over that functionality so we only have one process to handle the events. But there are a few issues with that approach. First, we need write access to the sysfs file that exposes the LED. Second, running multiple libinput instances would result in conflicts during LED access. Third, the sysfs interface is decidedly nonstandard and quite quirky to handle. And fourth, the most recent device, the Express Key Remote has hardware-controlled LEDs.

So instead we opted for a two-factor solution: the non-standard sysfs interface will be deprecated in favour of a proper kernel LED interface (/sys/class/leds/...) with the same contents as other LEDs. And second, the kernel will take over mode switching using LED triggers that are set up to cover the most common case - hitting a mode toggle button changes the mode. Benjamin Tissoires is currently working on those patches. Until then, libinput's backend implementation will just pretend that each tablet only has one mode group with a single mode. This allows us to get the rest of the userstack in place and then, once the kernel patches are in a released kernel, switch over to the right backend.

July 10, 2016

How GNOME Software uses libflatpak

It seems people are interested in adding support for flatpaks into other software centers, and I thought I might be useful to explain how I did this in gnome-software. I’m lucky enough to have a plugin architecture to make all the flatpak code be self contained in one file, but that’s certainly not a requirement.

Flatpak generates AppStream metadata when you build desktop applications. This means it’s possible to use appstream-glib and a few tricks to just load all the enabled remotes into an existing system store. This makes searching the new applications using the (optionally stemmed) token cache trivial. Once per day gnome-software checks the age of the AppStream cache, and if required downloads a new copy using flatpak_installation_update_appstream_sync(). As if by magic, appstream-glib notices the file modification/creation and updates the internal AsStore with the new applications.

When listing the installed applications, a simple call to flatpak_installation_list_installed_refs() returns us the list we need, on which we can easily set other flatpak-specific data like the runtime. This is matched against the AppStream data, which gives us a localized and beautiful application to display in the listview.

At this point we also call flatpak_installation_list_installed_refs_for_update() and then do flatpak_installation_update() with the NO_DEPLOY flag set. This just downloads the data we need, and can be cancelled without anything bad happening. When populating the updates panel I can just call flatpak_installation_list_installed_refs() again to find installed applications that have downloaded updates ready to apply without network access.

For the sources list I’m calling flatpak_installation_list_remotes() then ignoring any set as disabled or noenumerate. Most remotes have a name and title, and this makes the UI feature complete. When collecting information to show in the ui like the size we have the metadata already, but we also add the size of the runtime if it’s not already installed. This is the same idea as flatpak_installation_install(), where we also install any required runtime when installing the main application. There is a slight impedance mismatch between the flatpak many-installed-versions and the AppStream only-one-version model, but it seems to work well enough in the current code. Flatpak splits the deployment into a runtime containing common libraries that can be shared between apps (for instance, GNOME 3.20 or KDE5) and the application itself, so the software center always needs to install the runtime for the application to launch successfully. This is something that is not enforced by the CLI tool. Rather than installing everything for each app, we can also install other so-called extensions. These are typically non-essential like the various translations and any debug information, but are not strictly limited to those things. libflatpak automatically keeps the extensions up to date when updating, so gnome-software doesn’t have to do anything special at all.

Updating single applications is trivial with flatpak_installation_update() and launching applications is just as easy with flatpak_installation_launch(), although we only support launching the newest installed version of an application at the moment. Reading local bundles works well with flatpak_bundle_ref_new(), although we do have to load the gzipped AppStream metadata and the icon ourselves. Reading a .flatpakrepo file is slightly more work, but the data is in keyfile format and trivial to parse with GKeyFile.

Overall I’ve found libflatpak to be surprisingly easy to work with, requiring none of the kludges of all the different package-based systems I’ve worked on developing PackageKit. Full marks to Alex et al.

July 09, 2016

"I recieved a free or discounted product in return for an honest review"
My experiences with Amazon reviewing have been somewhat unusual. A review of a smart switch I wrote received enough attention that the vendor pulled the product from Amazon. At the time of writing, I'm ranked as around the 2750th best reviewer on Amazon despite having a total of 18 reviews. But the world of Amazon reviews is even stranger than that, and the past couple of weeks have given me some insight into it.

Amazon's success is fairly phenomenal. It's estimated that there's over 50 million people in the US paying $100 a year to get free shipping on Amazon purchases, and combined with Amazon's surprisingly customer friendly service there's a lot of people with a very strong preference for choosing Amazon rather than any other retailer. If you're not on Amazon, you're hurting your sales.

And if you're an established brand, this works pretty well. Some people will search for your product directly and buy it, leaving reviews. Well reviewed products appear higher up in search results, so people searching for an item type rather than a brand will still see your product appear early in the search results, in turn driving sales. Some proportion of those customers will leave reviews, which helps keep your product high up in the results. As long as your products aren't utterly dreadful, you'll probably maintain that position.

But if you're a brand nobody's ever heard of, things are more difficult. People are unlikely to search for your product directly, so you're relying on turning up in the results for more generic terms. But if you're selling a more generic kind of item (say, a Bluetooth smart bulb) then there's probably a number of other brands nobody's ever heard of selling almost identical objects. If there's no reason for anybody to choose your product then you're probably not going to get any reviews and you're not going to move up the search rankings. Even if your product is better than the competition, a small number of sales means a tiny number of reviews. By the time that number's large enough to matter, you're probably onto a new product cycle.

In summary: if nobody's ever heard of you, you need reviews but you're probably not getting any.

The old way of doing this was to send review samples to journalists, but nobody's going to run a comprehensive review of 3000 different USB cables and even if they did almost nobody would read it before making a decision on Amazon. You need Amazon reviews, but you're not getting any. The obvious solution is to send review samples to people who will leave Amazon reviews. This is where things start getting more dubious.

Amazon run a program called Vine which is intended to solve this problem. Send samples to Amazon and they'll distribute them to a subset of trusted reviewers. These reviewers write a review as normal, and Amazon tag the review with a "Vine Voice" badge which indicates to readers that the reviewer received the product for free. But participation in Vine is apparently expensive, and so there's a proliferation of sites like Snagshout or AMZ Review Trader that use a different model. There's no requirement that you be an existing trusted reviewer and the product probably isn't free. You sign up, choose a product, receive a discount code and buy it from Amazon. You then have a couple of weeks to leave a review, and if you fail to do so you'll lose access to the service. This is completely acceptable under Amazon's rules, which state "If you receive a free or discounted product in exchange for your review, you must clearly and conspicuously disclose that fact". So far, so reasonable.

In reality it's worse than that, with several opportunities to game the system. AMZ Review Trader makes it clear to sellers that they can choose reviewers based on past reviews, giving customers an incentive to leave good reviews in order to keep receiving discounted products. Some customers take full advantage of this, leaving a giant number of 5 star reviews for products they clearly haven't tested and then (presumably) reselling them. What's surprising is that this kind of cynicism works both ways. Some sellers provide two listings for the same product, the second being significantly more expensive than the first. They then offer an attractive discount for the more expensive listing in return for a review, taking it down to approximately the same price as the original item. Once the reviews are in, they can remove the first listing and drop the price of the second to the original price point.

The end result is a bunch of reviews that are nominally honest but are tied to perverse incentives. In effect, the overall star rating tells you almost nothing - you still need to actually read the reviews to gain any insight into whether the customer actually used the product. And when you do write an honest review that the seller doesn't like, they may engage in heavy handed tactics in an attempt to make the review go away.

It's hard to avoid the conclusion that Amazon's review model is broken, but it's not obvious how to fix it. When search ranking is tied to reviews, companies have a strong incentive to do whatever it takes to obtain positive reviews. What we're left with for now is having to laboriously click through a number of products to see whether their rankings come from thoughtful and detailed reviews or are just a mass of 5 star one liners.

comment count unavailable comments

July 08, 2016

Portals: Using GTK+ in a Flatpak

The time is ripe for new ways to distribute and deploy desktop applications: between snappy, Flatpak, AppImage and others there are quite a few projects in this area.

Sandboxing

Most of these projects involve some notion of sandboxing: isolating the application from the rest of the system.

Snappy does this by setting environment variables like XDG_DATA_DIRS, PATH, etc,  to tell apps where to find their ‘stuff’ and using app-armor to not let them access things they shouldn’t.

Flatpak takes a somewhat different approach: it uses bind mounts and namespaces to construct a separate view of the world for the app in which it can only see what it is supposed to access.

Regardless which approach you take to sandboxing, desktop applications are not very useful without access to the rest of the system.  So, clearly, we need to poke some holes in the walls of the sandbox, since we want apps to interact with the rest of the system.

The important thing to keep in mind is that we always want to give the user control over these interactions and in particular, control over the data that goes in and out of the sandbox.

Portals

The flatpak story for this has been portals. To quickly summarize: Portals are high-level (in the sense that they talk about concepts that are relevant to users) APIs that let sandboxed applications request access to resources outside. Portal calls will (almost) always involve user interaction (basically, a dialog).

We have talked about portals for quite a while, see for example this post by Allan from two years ago. Therefore,  I am very happy to announce that we now have first release of the portal infrastructure :

Note that the portal APIs themselves are desktop-neutral, and we have separated the implementation using GTK+ dialogs into their own module. We are also working on a qt/KDE  implementation.

The  APIs included in these releases cover a lot of basic things:

  • opening files
  • opening uris
  • printing
  • taking screenshots
  • notifications
  • inhibiting screen lock
  • network status
  • proxy information

Using portals in apps

The good news is that most of these will just work transparently in GTK+ applications, since GTK+ and GLib have suitable APIs that can be made to use the portals without any changes required from the sandboxed application. This support is already in place in the master branches; and will be in the 3.22 releases. We are working on a similar level of support for qt/KDE.

In detail, for opening files, you can use GtkFileChooserNative (a file chooser implementation that will also use a native Windows filechooser if you are on that platform). GtkFileChooserButton will do this for you, unless you manually set its dialog to something else. Certain things will not work with the portal, such as previews, or adding extra custom widgets.

For printing, use GtkPrintOperation (again a long-standing GTK+ api that will use a native Windows print dialog if you are on that platform).

For opening uris, use gtk_show_uri or g_app_info_launch_default_for_uri. We also added a new function that works slightly better in this context, gtk_show_uri_on_window().

Testing

As I said, most of the portals will just be transparently used by sandboxed GTK+ applications (provided they use GTK+ from git), but I’ve also written a portal-test application to try all the available portals.

portal-test

All of the tests use the regular GTK+ APIs, with the exception of the Screenshot button. Since we don’t have a screenshooting API in GTK+, that button makes a direct D-Bus call to the portal. Here is how it looks in action:

portal-test2

Whats next ?

For the first release, we’ve focused on portals for toolkit-level functionality. There is obvously a long list of other system integration points that will need to be covered. High up on our list for the near future are access to devices like webcams and microphones, and pulseaudio. If you are interested, the list of issues has some more information.

Update: I’ve been asked: Why do I need a portal, the file chooser in the libreoffice flatpak seems to work just fine ?! Most of the current flatpaks are shipping with a relatively open sandbox configuration that allows the application full access to the host filesystem, or at least to your entire home directory. Portals enable applications to  function in a constrained sandbox that does not have this.

July 07, 2016

Bluetooth LED bulbs
The best known smart bulb setups (such as the Philips Hue and the Belkin Wemo) are based on Zigbee, a low-energy, low-bandwidth protocol that operates on various unlicensed radio bands. The problem with Zigbee is that basically no home routers or mobile devices have a Zigbee radio, so to communicate with them you need an additional device (usually called a hub or bridge) that can speak Zigbee and also hook up to your existing home network. Requests are sent to the hub (either directly if you're on the same network, or via some external control server if you're on a different network) and it sends appropriate Zigbee commands to the bulbs.

But requiring an additional device adds some expense. People have attempted to solve this in a couple of ways. The first is building direct network connectivity into the bulbs, in the form of adding an 802.11 controller. Go through some sort of setup process[1], the bulb joins your network and you can communicate with it happily. Unfortunately adding wifi costs more than adding Zigbee, both in terms of money and power - wifi bulbs consume noticeably more power when "off" than Zigbee ones.

There's a middle ground. There's a large number of bulbs available from Amazon advertising themselves as Bluetooth, which is true but slightly misleading. They're actually implementing Bluetooth Low Energy, which is part of the Bluetooth 4.0 spec. Implementing this requires both OS and hardware support, so older systems are unable to communicate. Android 4.3 devices tend to have all the necessary features, and modern desktop Linux is also fine as long as you have a Bluetooth 4.0 controller.

Bluetooth is intended as a low power communications protocol. Bluetooth Low Energy (or BLE) is even lower than that, running in a similar power range to Zigbee. Most semi-modern phones can speak it, so it seems like a pretty good choice. Obviously you lose the ability to access the device remotely, but given the track record on this sort of thing that's arguably a benefit. There's a couple of other downsides - the range is worse than Zigbee (but probably still acceptable for any reasonably sized house or apartment), and only one device can be connected to a given BLE server at any one time. That means that if you have the control app open while you're near a bulb, nobody else can control that bulb until you disconnect.

The quality of the bulbs varies a great deal. Some of them are pure RGB bulbs and incapable of producing a convincing white at a reasonable intensity[2]. Some have additional white LEDs but don't support running them at the same time as the colour LEDs, so you have the choice between colour or a fixed (and usually more intense) white. Some allow running the white LEDs at the same time as the RGB ones, which means you can vary the colour temperature of the "white" output.

But while the quality of the bulbs varies, the quality of the apps doesn't really. They're typically all dreadful, competing on features like changing bulb colour in time to music rather than on providing a pleasant user experience. And the whole "Only one person can control the lights at a time" thing doesn't really work so well if you actually live with anyone else. I was dissatisfied.

I'd met Mike Ryan at Kiwicon a couple of years back after watching him demonstrate hacking a BLE skateboard. He offered a couple of good hints for reverse engineering these devices, the first being that Android already does almost everything you need. Hidden in the developer settings is an option marked "Enable Bluetooth HCI snoop log". Turn that on and all Bluetooth traffic (including BLE) is dumped into /sdcard/btsnoop_hci.log. Turn that on, start the app, make some changes, retrieve the file and check it out using Wireshark. Easy.

Conveniently, BLE is very straightforward when it comes to network protocol. The only thing you have is GATT, the Generic Attribute Protocol. Using this you can read and write multiple characteristics. Each packet is limited to a maximum of 20 bytes. Most implementations use a single characteristic for light control, so it's then just a matter of staring at the dumped packets until something jumps out at you. A pretty typical implementation is something like:

0x56,r,g,b,0x00,0xf0,0x00,0xaa

where r, g and b are each just a single byte representing the corresponding red, green or blue intensity. 0x56 presumably indicates a "Set the light to these values" command, 0xaa indicates end of command and 0xf0 indicates that it's a request to set the colour LEDs. Sending 0x0f instead results in the previous byte (0x00 in this example) being interpreted as the intensity of the white LEDs. Unfortunately the bulb I tested that speaks this protocol didn't allow you to drive the white LEDs at the same time as anything else - setting the selection byte to 0xff didn't result in both sets of intensities being interpreted at once. Boo.

You can test this out fairly easily using the gatttool app. Run hcitool lescan to look for the device (remember that it won't show up if anything else is connected to it at the time), then do gatttool -b deviceid -I to get an interactive shell. Type connect to initiate a connection, and once connected send commands by doing char-write-cmd handle value using the handle obtained from your hci dump.

I did this successfully for various bulbs, but annoyingly hit a problem with one from Tikteck. The leading byte of each packet was clearly a counter, but the rest of the packet appeared to be garbage. For reasons best known to themselves, they've implemented application-level encryption on top of BLE. This was a shame, because they were easily the best of the bulbs I'd used - the white LEDs work in conjunction with the colour ones once you're sufficiently close to white, giving you good intensity and letting you modify the colour temperature. That gave me incentive, but figuring out the protocol took quite some time. Earlier this week, I finally cracked it. I've put a Python implementation on Github. The idea is to tie it into Ulfire running on a central machine with a Bluetooth controller, making it possible for me to control the lights from multiple different apps simultaneously and also integrating with my Echo.

I'd write something about the encryption, but I honestly don't know. Large parts of this make no sense to me whatsoever. I haven't even had any gin in the past two weeks. If anybody can explain how anything that's being done there makes any sense at all[3] that would be appreciated.

[1] typically via the bulb pretending to be an access point, but also these days through a terrifying hack involving spewing UDP multicast packets of varying lengths in order to broadcast the password to associated but unauthenticated devices and good god the future is terrifying

[2] For a given power input, blue LEDs produce more light than other colours. To get white with RGB LEDs you either need to have more red and green LEDs than blue ones (which costs more), or you need to reduce the intensity of the blue ones (which means your headline intensity is lower). Neither is appealing, so most of these bulbs will just give you a blue "white" if you ask for full red, green and blue

[3] Especially the bit where we calculate something from the username and password and then encrypt that using some random numbers as the key, then send 50% of the random numbers and 50% of the encrypted output to the device, because I can't even

comment count unavailable comments
crashtesting: now 92000 documents

crash testing, now 92000 documents continuously tested

Last August we had a collection of approximately 76000 documents. This July we have over 92000 documents. The corpus is mostly gathered from our bugzilla and other projects bugzillas via our get-bugzilla-attachments-by-mimetype script. For testing purposes we continuously import them all, and then export certain formats to multiple destination formats. For example odts are re-exported to odt, doc and docx and the results of what documents crashed (which includes asserts) are uploaded to the crashtest site
 
I like to imagine that these are typically the type of mean and bitter documents that try to eat innocent office software alive.
 
The get-bugzilla-attachments-by-mimetype script only downloads new attachments from its target bugzillas which are not already downloaded. The last run to refresh the corpus took over two days to complete. Refreshing isn't fast or cheap so it's fairly infrequent.
 
The regular crashtesting run to import and reexport the corpus is comparatively frequent, takes approximately 24 hours and typically gets run every two or three days on the latest 64bit Linux master in headless mode.

July 05, 2016

Flatpak and GNOME Software

I wanted to write a little about how Flatpak apps are treated differently to packages in GNOME Software. We’ve now got two plugins in master, one called flatpak-user and another called flatpak-system. They both share 99% of the same code, only differing in how they are initialised. As you might expect, -user does per-user installation and updating, and the latter does it per-system for all users. Per-user applications that are specific to just a single user account are an amazingly useful concept, as most developers found using tools like jhbuild. We default to installing software at the moment for all users, but there is actually a org.gnome.software.install-bundles-system-wide dconf key that can be used to reverse this on specific systems.

We go to great lengths to interoperate with the flatpak command line tool, so if you install the nightly GTK3 build of GIMP per-user you can install the normal version system-wide and they both show in the installed and updates panel without conflicting. We’ve also got file notifications set up so GNOME Software shows the correct application state straight away if you add a remote or install a flatpak app on the command line. At the moment we show both packages and flatpaks in the search results, but when we suggest apps on the overview page we automatically prefer the flatpak version if both are available. In Ubuntu, snappy results are sorted above package results unconditionally, but I don’t know if this is a good thing to do for flatpaks upstream, comments welcome. I’m sure whatever defaults I choose will mortally offend someone.

Screenshot from 2016-07-05 14-45-35

GNOME Software also supports single-file flatpak bundles like gimp.flatpak – just double click and you’re good to install. These files are somewhat like a package in that all the required files are included and you can install without internet access. These bundles can also install a remote (ie a reference to a flatpak repository) too, which allows them to be kept up to date. Such per-application remotes are only used for the specific application and not the others potentially in the same tree (for the curious, this is called a “noenumerate” remote). We also support the more rarely seen dummy.flatpakrepo files too; these allow a user to install a remote which could contain a number of applications and makes it very easy to set up an add-on remote that allows you browse a different set of apps than shipped, for instance the Endless-specific apps. Each of these files contains all the metadata we need in AppStream format, with translations, icons and all the things you expect from a modern software center. It’s a shame snappy decided not to use AppStream and AppData for application metadata, as this kind of extra data really makes the UI really beautiful.

Screenshot from 2016-07-05 14-54-18

With the latest version of flatpak we also do a much better job of installing the additional extensions the application needs, for instance locales or debug data. Sharing the same code between the upstream command line tool and gnome-software means we always agree on what needs installing and updating. Just like the CLI, gnome-software can update flatpaks safely live (even when the application is running), although we do a little bit extra compared to the CLI and download the data we need to do the update when the session is idle and on suitable unmetered network access. This means you can typically just click the ‘Update’ button in the updates panel for a near-instant live-update. This is what people have wanted for years, and I’ve told each and every bug-report that live updates using packages only works 99.99% of the time, exploding in a huge fireball 0.01% of the time. Once all desktop apps are packaged as flatpaks we will only need to reboot for atomic offline updates for core platform updates like a new glibc or the kernel. That future is very nearly now.

Screenshot from 2016-07-05 14-54-59

June 21, 2016

I've bought some more awful IoT stuff
I bought some awful WiFi lightbulbs a few months ago. The short version: they introduced terrible vulnerabilities on your network, they violated the GPL and they were also just bad at being lightbulbs. Since then I've bought some other Internet of Things devices, and since people seem to have a bizarre level of fascination with figuring out just what kind of fractal of poor design choices these things frequently embody, I thought I'd oblige.

Today we're going to be talking about the KanKun SP3, a plug that's been around for a while. The idea here is pretty simple - there's lots of devices that you'd like to be able to turn on and off in a programmatic way, and rather than rewiring them the simplest thing to do is just to insert a control device in between the wall and the device andn ow you can turn your foot bath on and off from your phone. Most vendors go further and also allow you to program timers and even provide some sort of remote tunneling protocol so you can turn off your lights from the comfort of somebody else's home.

The KanKun has all of these features and a bunch more, although when I say "features" I kind of mean the opposite. I plugged mine in and followed the install instructions. As is pretty typical, this took the form of the plug bringing up its own Wifi access point, the app on the phone connecting to it and sending configuration data, and the plug then using that data to join your network. Except it didn't work. I connected to the plug's network, gave it my SSID and password and waited. Nothing happened. No useful diagnostic data. Eventually I plugged my phone into my laptop and ran adb logcat, and the Android debug logs told me that the app was trying to modify a network that it hadn't created. Apparently this isn't permitted as of Android 6, but the app was handling this denial by just trying again. I deleted the network from the system settings, restarted the app, and this time the app created the network record and could modify it. It still didn't work, but that's because it let me give it a 5GHz network and it only has a 2.4GHz radio, so one reset later and I finally had it online.

The first thing I normally do to one of these things is run nmap with the -O argument, which gives you an indication of what OS it's running. I didn't really need to in this case, because if I just telnetted to port 22 I got a dropbear ssh banner. Googling turned up the root password ("p9z34c") and I was logged into a lightly hacked (and fairly obsolete) OpenWRT environment.

It turns out that here's a whole community of people playing with these plugs, and it's common for people to install CGI scripts on them so they can turn them on and off via an API. At first this sounds somewhat confusing, because if the phone app can control the plug then there clearly is some kind of API, right? Well ha yeah ok that's a great question and oh good lord do things start getting bad quickly at this point.

I'd grabbed the apk for the app and a copy of jadx, an incredibly useful piece of code that's surprisingly good at turning compiled Android apps into something resembling Java source. I dug through that for a while before figuring out that before packets were being sent, they were being handed off to some sort of encryption code. I couldn't find that in the app, but there was a native ARM library shipped with it. Running strings on that showed functions with names matching the calls in the Java code, so that made sense. There were also references to AES, which explained why when I ran tcpdump I only saw bizarre garbage packets.

But what was surprising was that most of these packets were substantially similar. There were a load that were identical other than a 16-byte chunk in the middle. That plus the fact that every payload length was a multiple of 16 bytes strongly indicated that AES was being used in ECB mode. In ECB mode each plaintext is split up into 16-byte chunks and encrypted with the same key. The same plaintext will always result in the same encrypted output. This implied that the packets were substantially similar and that the encryption key was static.

Some more digging showed that someone had figured out the encryption key last year, and that someone else had written some tools to control the plug without needing to modify it. The protocol is basically ascii and consists mostly of the MAC address of the target device, a password and a command. This is then encrypted and sent to the device's IP address. The device then sends a challenge packet containing a random number. The app has to decrypt this, obtain the random number, create a response, encrypt that and send it before the command takes effect. This avoids the most obvious weakness around using ECB - since the same plaintext always encrypts to the same ciphertext, you could just watch encrypted packets go past and replay them to get the same effect, even if you didn't have the encryption key. Using a random number in a challenge forces you to prove that you actually have the key.

At least, it would do if the numbers were actually random. It turns out that the plug is just calling rand(). Further, it turns out that it never calls srand(). This means that the plug will always generate the same sequence of challenges after a reboot, which means you can still carry out replay attacks if you can reboot the plug. Strong work.

But there was still the question of how the remote control works, since the code on github only worked locally. tcpdumping the traffic from the server and trying to decrypt it in the same way as local packets worked fine, and showed that the only difference was that the packet started "wan" rather than "lan". The server decrypts the packet, looks at the MAC address, re-encrypts it and sends it over the tunnel to the plug that registered with that address.

That's not really a great deal of authentication. The protocol permits a password, but the app doesn't insist on it - some quick playing suggests that about 90% of these devices still use the default password. And the devices are all based on the same wifi module, so the MAC addresses are all in the same range. The process of sending status check packets to the server with every MAC address wouldn't take that long and would tell you how many of these devices are out there. If they're using the default password, that's enough to have full control over them.

There's some other failings. The github repo mentioned earlier includes a script that allows arbitrary command execution - the wifi configuration information is passed to the system() command, so leaving a semicolon in the middle of it will result in your own commands being executed. Thankfully this doesn't seem to be true of the daemon that's listening for the remote control packets, which seems to restrict its use of system() to data entirely under its control. But even if you change the default root password, anyone on your local network can get root on the plug. So that's a thing. It also downloads firmware updates over http and doesn't appear to check signatures on them, so there's the potential for MITM attacks on the plug itself. The remote control server is on AWS unless your timezone is GMT+8, in which case it's in China. Sorry, Western Australia.

It's running Linux and includes Busybox and dnsmasq, so plenty of GPLed code. I emailed the manufacturer asking for a copy and got told that they wouldn't give it to me, which is unsurprising but still disappointing.

The use of AES is still somewhat confusing, given the relatively small amount of security it provides. One thing I've wondered is whether it's not actually intended to provide security at all. The remote servers need to accept connections from anywhere and funnel decent amounts of traffic around from phones to switches. If that weren't restricted in any way, competitors would be able to use existing servers rather than setting up their own. Using AES at least provides a minor obstacle that might encourage them to set up their own server.

Overall: the hardware seems fine, the software is shoddy and the security is terrible. If you have one of these, set a strong password. There's no rate-limiting on the server, so a weak password will be broken pretty quickly. It's also infringing my copyright, so I'd recommend against it on that point alone.

comment count unavailable comments
AAA game, indie game, card-board-box
Early bird gets eaten by the Nyarlathotep
 
The more adventurous of you can use those (designed as embeddable) Lua scripts to transform your DRM-free GOG.com downloads into Flatpaks.

The long-term goal would obviously be for this not to be needed, and for online games stores to ship ".flatpak" files, with metadata so we know what things are in GNOME Software, which automatically picks up the right voice/subtitle language, and presents its extra music and documents in the respective GNOME applications.
 
But in the meanwhile, and for the sake of the games already out there, there's flatpak-games. Note that lua-archive is still fiddly.
 
Support for a few Humble Bundle formats (some formats already are), grab-all RPMs and Debs, and those old Loki games is also planned.
 
It's late here, I'll be off to do some testing I think :)

PS: Even though I have enough programs that would fail to create bundles in my personal collection to accept "game donations", I'm still looking for original copies of Loki games. Drop me a message if you can spare one!

June 14, 2016

Dispatches from the GTK+ hackfest

A quick update from the GTK+ hackfest. I don’t really want to talk about the versioning discussion, except for two points:

First, I want to apologize to Allison for encouraging her to post about this – I really didn’t anticipate the amount of uninformed, unreasonable and hateful reactions that we received.

Second, I want to stress that we just at the beginning of this discussion, we will not make any decisions until after Guadec. Everybody who has opinions on this topic should feel free to give us feedback. We are of course particularly interested in the feedback from parties who will be directly affected by changes in GTK+ versioning, e.g. gstreamer and other dependent libraries.

20160613_155602Todays morning discussion was all about portals. Portals are high-level D-Bus apis that will allow sandboxed applications to request access to outside resources, such as files or pictures, or just for showing URIs or launch other applications. We have early implementations of some of these now,  but after the valuable feedback in todays discussion, we will likely make some  changes to it.

An expectation for portals is that there will be a user interaction before an operation is carried out, to keep users in control and let tem cancel requests. The portal UIs will be provided by the desktop session.

We want to have a number of portals implemented during the summer, starting with the most important ones, like file chooser,  ‘open a uri’, application launcher, proxy support, and a few others. The notes from this discussion can be found here.

20160613_105754In the afternoon, the discussion moved to developer documentation, how to improve it, let readers provide feedback and suggestions, and integrate them in gnome-builder.

We also discussed ways to make GTK+ better for responsive designs.

June 08, 2016

spellchecking calendar/calender
A short 2009 article about Microsoft's group with responsibility for spellchecking that mentions the calendar/calender masking problem. Sometimes you probably do want correctly spelled words to be flagged.

June 07, 2016

Modularity and the desktop

shellsThere has been much talk about modularity recently. Fedora even has a working group on this topic. Modularity is such a generic term that  it is a bit hard to figure out what this is all about, but the wiki page gives some hints: …base module… …docker image… …reduced dependencies… …tooling… 

A very high-level reading would be that this is about defining tools and processes to deliver software in modules, which can be larger units than packages, and e.g. come in the form of container images.

Another reading would be that this is an effort to come to grips with the changing role of distributions in a world were containers are the dominant way to run and deploy applications.

Modularity on the desktop

In this post, I want to look at how this modularity effort looks from the desktop perspective, and what we are doing in this area.

Our main goal for a while has been to make it easier to get desktop applications from application developers to users.

In the traditional Linux distribution world, this is a total nightmare: Once you’ve written your application, you need to either learn how to create an Ubuntu .deb, an Arch .pkg, a Fedora .rpm, to name just a few, and then follow lengthy processes to get your packages accepted into these distributions.

And once you’ve done all that, you will get bug reports that your application is broken with the one or other version of one of your dependencies. If not right away, then a few months down the road when the distributions move on to the latest and greatest releases.  So, keeping those packages working requires the constant attention of a package maintainer.

To improve on this sad state of affairs, we need to make applications much less dependent on the OS they run on, and on the libraries that happen to come with the OS.

At this point, you might say: Aha, you want to bundle dependencies! Yes, but read on. Bundling is a bad word in the traditional distribution world. It implies duplication, since multiple applications may bundle the same library. And having multiple copies of the same library (possibly different versions, too), makes it harder to ensure that bug and security fixes get applied to all the copies.  Not to mention that all the duplication consumes bandwidth when you have to download it all.

Bundling everything with the application certainly has its drawbacks. But there are several things we can do to preserve most of the benefits of bundling, while avoiding the worst of the problems.

One takeaway from the modularity discussions mentioned earlier is that going to larger units than individual packages (ie modules) has  advantages: we thin out the dependency graph, and we get a large piece of functionality that can be tested, installed and updated as a unit.

And we can use a smart storage and transport mechanism such as OSTree to limit the impact that duplication has on bandwidth.

Desktop applications commonly depend on similar sets of libraries: the GTK+ stack is a good example, kdelibs is another.  Treating these common library stacks as modules makes a lot of sense.  We decided to call these modules runtimes. Applications can declare a single dependency on the runtime they need, and get a well-defined set of APIs and libraries in return.

What if you need a library that is not in the runtime ? In that case, you can still bundle it with your application. You will be reponsible for the libraries that you bundle, but in return your users get exactly the version that you’ve tested your app with.

By decoupling the runtimes from the OS and by making them available in the same way as the applications, we  make it possible to have applications that can run on different distributions, regardless of the library versions that are included in the OS. Of course, there is limits to what we can achieve: the applications and runtimes still have requirements (e.g. on Linux kernel features, or on session services that are expected to be present). But building an application that can run on Fedora, Ubuntu, Arch, RHEL and other modern distributions is entirely possible.

But what about security updates?  To make this system work, we need to offer well-maintained runtimes that application authors can rely on, and provide updates for them just as we do now with distribution packages.

But that is not enough. We also need to isolate applications from the rest of the system (and your data!) at runtime, in order to limit the damage that they can do. Historically, the Linux desktop has been really bad about this: X lets any client snoop all input, and applications are generally free to read whatever they find in your home directory. Part of our desktop modularity story is to use container technologies and e.g. Wayland to confine (or sandbox) applications at runtime.

Modularity now!

 Bubblewrap cat by dancing_stupidity

We have been working towards this goal for quite a while, with the Wayland port and gnome-software as a non-package-centric installer.

What I’ve outlined above is more or less the architecture of the Flatpak system (formerly known as xdg-app).  It is not 100% fleshed out yet, but enough pieces are in place now that you can try it out and e.g. install a nightly build of libreoffice that works on both Fedora and Ubuntu.

Head over to www.flatpak.org to learn more.

Be wary of heroes
Inspiring change is difficult. Fighting the status quo typically means being able to communicate so effectively that powerful opponents can't win merely by outspending you. People need to read your work or hear you speak and leave with enough conviction that they in turn can convince others. You need charisma. You need to be smart. And you need to be able to tailor your message depending on the audience, even down to telling an individual exactly what they need to hear to take your side. Not many people have all these qualities, but those who do are powerful and you want them on your side.

But the skills that allow you to convince people that they shouldn't listen to a politician's arguments are the same skills that allow you to convince people that they shouldn't listen to someone you abused. The ability that allows you to argue that someone should change their mind about whether a given behaviour is of social benefit is the same ability that allows you to argue that someone should change their mind about whether they should sleep with you. The visibility that gives you the power to force people to take you seriously is the same visibility that makes people afraid to publicly criticise you.

We need these people, but we also need to be aware that their talents can be used to hurt as well as to help. We need to hold them to higher standards of scrutiny. We need to listen to stories about their behaviour, even if we don't want to believe them. And when there are reasons to believe those stories, we need to act on them. That means people need to feel safe in coming forward with their experiences, which means that nobody should have the power to damage them in reprisal. If you're not careful, allowing charismatic individuals to become the public face of your organisation gives them that power.

There's no reason to believe that someone is bad merely because they're charismatic, but this kind of role allows a charismatic abuser both a great deal of cover and a great deal of opportunity. Sometimes people are just too good to be true. Pretending otherwise doesn't benefit anybody but the abusers.

comment count unavailable comments

May 25, 2016

Blog backlog, Post 3, DisplayLink-based USB3 graphics support for Fedora
Last year, after DisplayLink released the first version of the supporting tools for their USB3 chipsets, I tried it out on my Dell S2340T.

As I wanted a clean way to test new versions, I took Eric Nothen's RPMs, and updated them along with newer versions, automating the creation of 32- and 64-bit x86 versions.

The RPM contains 3 parts, evdi, a GPLv2 kernel module that creates a virtual display, the LGPL library to access it, and a proprietary service which comes with "firmware" files.

Eric's initial RPMs used the precompiled libevdi.so, and proprietary bits, compiling only the kernel module with dkms when needed. I changed this, compiling the library from the upstream repository, using the minimal amount of pre-compiled binaries.

This package supports quite a few OEM devices, but does not work correctly with Wayland, so you'll need to disable Wayland support in /etc/gdm/custom.conf if you want it to work at the login screen, and without having to restart the displaylink.service systemd service after logging in.


 Plugged in via DisplayPort and USB (but I can only see one at a time)


The source for the RPM are on GitHub. Simply clone and run make in the repository to create 32-bit and 64-bit RPMs. The proprietary parts are redistributable, so if somebody wants to host and maintain those RPMs, I'd be glad to pass this on.

May 24, 2016

LVFS Technical White Paper

I spent a good chunk of today writing a technical whitepaper titled Introducing the Linux Vendor Firmware Service — I’d really appreciate any comments, either from people who have seen all progress from the start or who don’t know anything about it at all.

Typos, or more general comments are all welcome and once I’ve got something a bit more polished I’ll be sending this to some important suits in a few well known companies. Thanks for any help!

May 23, 2016

External Plugins in GNOME Software (6)

This is my last post about the gnome-software plugin structure. If you want more, join the mailing list and ask a question. If you’re not sure how something works then I’ve done a poor job on the docs, and I’m happy to explain as much as required.

GNOME Software used to provide a per-process plugin cache, automatically de-duplicating applications and trying to be smarter than the plugins themselves. This involved merging applications created by different plugins and really didn’t work very well. For 3.20 and later we moved to a per-plugin cache which allows the plugin to control getting and adding applications to the cache and invalidating it when it made sense. This seems to work a lot better and is an order of magnitude less complicated. Plugins can trivially be ported to using the cache using something like this:

 
   /* create new object */
   id = gs_plugin_flatpak_build_id (inst, xref);
-  app = gs_app_new (id);
+  app = gs_plugin_cache_lookup (plugin, id);
+  if (app == NULL) {
+     app = gs_app_new (id);
+     gs_plugin_cache_add (plugin, id, app);
+  }

Using the cache has two main benefits for plugins. The first is that we avoid creating duplicate GsApp objects for the same logical thing. This means we can query the installed list, start installing an application, then query it again before the install has finished. The GsApp returned from the second add_installed() request will be the same GObject, and thus all the signals connecting up to the UI will still be correct. This means we don’t have to care about migrating the UI widgets as the object changes and things like progress bars just magically work.

The other benefit is more obvious. If we know the application state from a previous request we don’t have to query a daemon or do another blocking library call to get it. This does of course imply that the plugin is properly invalidating the cache using gs_plugin_cache_invalidate() which it should do whenever a change is detected. Whether a plugin uses the cache for this reason is up to the plugin, but if it does it is up to the plugin to make sure the cache doesn’t get out of sync.

And one last thing: If you’re thinking of building an out-of-tree plugin for production use ask yourself if it actually belongs upstream. Upstream plugins get ported as the API evolves, and I’m already happily carrying Ubuntu and Fedora-specific plugins that either self-disable at runtime or are protected using --enable-foo configure argument.

External Plugins in GNOME Software (5)

This is my penultimate post about the gnome-software plugin structure. If you’ve followed everything so far, well done.

There’s a lot of flexibility in the gnome-software plugin structure; a plugin can add custom applications and handle things like search and icon loading in a totally custom way. Most of the time you don’t care about how search is implemented or how icons are going to be loaded, and you can re-use a lot of the existing code in the appstream plugin. To do this you just save an AppStream-format XML file in either /usr/share/app-info/xmls/, /var/cache/app-info/xmls/ or ~/.local/share/app-info/xmls/. GNOME Software will immediately notice any new files, or changes to existing files as it has set up the various inotify watches.

This allows plugins to care a lot less about how applications are going to be shown. For example, the steam plugin downloads and parses the descriptions from a remote service during gs_plugin_refresh(), and also finds the best icon types and downloads them too. Then it exports the data to an AppStream XML file, saving it to your home directory. This allows all the applications to be easily created (and then refined) using something as simple as gs_app_new("steam:foo.desktop"). All the search tokenisation and matching is done automatically, so it makes the plugin much simpler and faster.

The only extra step the steam plugin needs to do is implement the gs_plugin_adopt_app() function. This is called when an application does not have a management plugin set, and allows the plugin to claim the application for itself so it can handle installation, removal and updating. In the case of steam it could check the ID has a prefix of steam: or could check some other plugin-specific metadata using gs_app_get_metadata_item().

Another good example is the fwupd that wants to handle any firmware we’ve discovered in the AppStream XML. This might be shipped by the vendor in a package using Satellite, or downloaded from the LVFS. It wouldn’t be kind to set a management plugin explicitly in case XFCE or KDE want to handle this in a different way. This adoption function in this case is trivial:

void
gs_plugin_adopt_app (GsPlugin *plugin, GsApp *app)
{
  if (gs_app_get_kind (app) == AS_APP_KIND_FIRMWARE)
    gs_app_set_management_plugin (app, "fwupd");
}

The next (and last!) blog post I’m going to write is about the per-plugin cache that’s available to plugins to help speed up some operations. In related news, we now have a mailing list, so if you’re interested in this stuff I’d encourage you to join and ask questions there. I also released gnome-software 3.21.2 this morning, so if you want to try all this plugin stuff yourself your distro if probably going to be updating packages soon.

May 22, 2016

External Plugins in GNOME Software (4)

After my last post, I wanted to talk more about the refine functionality in gnome-software. As previous examples have shown it’s very easy to add a new application to the search results, updates list or installed list. Some plugins don’t want to add more applications, but want to modify existing applications to add more information depending on what is required by the UI code. The reason we don’t just add everything at once is that for search-as-you-type to work effectively we need to return results in less than about 50ms and querying some data can take a long time. For example, it might take a few hundred ms to work out the download size for an application when a plugin has to also look at what dependencies are already installed. We only need this information once the user has clicked the search results and when the user is in the details panel, so we can save a ton of time not working out properties that are not useful.

Lets looks at another example.

gboolean
gs_plugin_refine_app (GsPlugin *plugin,
                      GsApp *app,
                      GsPluginRefineFlags flags,
                      GCancellable *cancellable,
                      GError **error)
{
  /* not required */
  if ((flags & GS_PLUGIN_REFINE_FLAGS_REQUIRE_LICENSE) == 0)
    return TRUE;

  /* already set */
  if (gs_app_get_license (app) != NULL)
    return TRUE;

  /* FIXME, not just hardcoded! */
  if (g_strcmp0 (gs_app_get_id (app, "chiron.desktop") == 0))
    gs_app_set_license (app, "GPL-2.0 and LGPL-2.0+");

  return TRUE;
}

This is a simple example, but shows what a plugin needs to do. It first checks if the action is required, in this case GS_PLUGIN_REFINE_FLAGS_REQUIRE_LICENSE. This request is more common than you might expect as even the search results shows a non-free label if the license is unspecified or non-free. It then checks if the license is already set, returning with success if so. If not, it checks the application ID and hardcodes a license; in the real world this would be querying a database or parsing an additional config file. As mentioned before, if the license value is freely available without any extra work then it’s best just to set this at the same time as when adding the app with gs_app_list_add(). Think of refine as adding things that cost time to calculate only when really required.

The UI in gnome-software is quite forgiving for missing data, hiding sections or labels as required. Some things are required however, and forgetting to assign an icon or short description will get the application vetoed so that it’s not displayed at all. Helpfully, running gnome-software --verbose on the command line will tell you why an application isn’t shown along with any extra data.

As a last point, a few people have worries that these blogs are perhaps asking for trouble; external plugins have a chequered history in a number of projects and I’m sure gnome-software would be in an even worse position given that the core maintainer team is still so small. Being honest, if we break your external plugin due to an API change in the core you probably should have pushed your changes upstream sooner. There’s a reason you have to build with -DI_KNOW_THE_GNOME_SOFTWARE_API_IS_SUBJECT_TO_CHANGE

External Plugins in GNOME Software (3)

Lots of nice feedback from my last post, so here’s some new stuff. Up now is downloading new metadata and updates in plugins.

The plugin loader supports a gs_plugin_refresh() vfunc that is called in various situations. To ensure plugins have the minimum required metadata on disk it is called at startup, but with a cache age of infinite. This basically means the plugin must just ensure that any data exists no matter what the age.

Usually once per day, we’ll call gs_plugin_refresh() but with the correct cache age set (typically a little over 24 hours) which allows the plugin to download new metadata or payload files from remote servers. The gs_utils_get_file_age() utility helper can help you work out the cache age of file, or the plugin can handle it some other way.

For the Flatpak plugin we just make sure the AppStream metadata exists at startup, which allows us to show search results in the UI. If the metadata did not exist (e.g. if the user had added a remote using the commandline without gnome-software running) then we would show a loading screen with a progress bar before showing the main UI. On fast connections we should only show that for a couple of seconds, but it’s a good idea to try any avoid that if at all possible in the plugin.
Once per day the gs_plugin_refresh() method is called again, but this time with GS_PLUGIN_REFRESH_FLAGS_PAYLOAD set. This is where the Flatpak plugin would download any ostree trees (but not doing the deloy step) so that the applications can be updated live in the details panel without having to wait for the download to complete. In a similar way, the fwupd plugin downloads the tiny LVFS metadata with GS_PLUGIN_REFRESH_FLAGS_METADATA and then downloads the large firmware files themselves only when the GS_PLUGIN_REFRESH_FLAGS_PAYLOAD flag is set.

If the @app parameter is set for gs_plugin_download_file() then the progress of the download is automatically proxied to the UI elements associated with the application, for instance the install button would show a progress bar in the various different places in the UI. For a refresh there’s no relevant GsApp to use, so we’ll leave it NULL which means something is happening globally which the UI can handle how it wants, for instance showing a loading page at startup.

gboolean
gs_plugin_refresh (GsPlugin *plugin,
                   guint cache_age,
                   GsPluginRefreshFlags flags,
                   GCancellable *cancellable,
                   GError **error)
{
  const gchar *metadata_fn = "/var/cache/example/metadata.xml";
  const gchar *metadata_url = "http://www.example.com/new.xml";

  /* this is called at startup and once per day */
  if (flags & GS_PLUGIN_REFRESH_FLAGS_METADATA) {
    g_autoptr(GFile) file = g_file_new_for_path (metadata_fn);

    /* is the metadata missing or too old */
    if (gs_utils_get_file_age (file) > cache_age) {
      if (!gs_plugin_download_file (plugin,
                                    NULL,
                                    metadata_url,
                                    metadata_fn,
                                    cancellable,
                                    error)) {
        /* it's okay to fail here */
        return FALSE;
      }
      g_debug ("successfully downloaded new metadata");
    }
  }

  /* this is called when the session is idle */
  if ((flags & GS_PLUGIN_REFRESH_FLAGS_PAYLOAD) == 0) {
    // FIXME: download any required updates now
  }

  return TRUE;
}

Note, if the downloading fails it’s okay to return FALSE; the plugin loader continues to run all plugins and just logs an error to the console. We’ll be calling into gs_plugin_refresh() again in only a few hours, so there’s no need to bother the user. For actions like gs_plugin_app_install we also do the same thing, but we also save the error on the GsApp itself so that the UI is free to handle that how it wants, for instance showing a GtkDialog window for example.

May 20, 2016

External Plugins in GNOME Software (2)

After quite a lot of positive feedback from my last post I’ll write some more about custom plugins. Next up is returning custom applications into the installed list. The use case here is a proprietary software distribution method that installs custom files into your home directory, but you can use your imagination for how this could be useful.

The example here is all hardcoded, and a true plugin would have to derive the details about the GsApp, for example reading in an XML file or YAML config file somewhere. So, code:

#include <gnome-software.h>

void
gs_plugin_initialize (GsPlugin *plugin)
{
  gs_plugin_add_rule (plugin, GS_PLUGIN_RULE_RUN_BEFORE, "icons");
}

gboolean
gs_plugin_add_installed (GsPlugin *plugin,
                         GsAppList *list,
                         GCancellable *cancellable,
                         GError **error)
{
  g_autofree gchar *fn = NULL;
  g_autoptr(GsApp) app = NULL;
  g_autoptr(AsIcon) icon = NULL;

  /* check if the app exists */
  fn = g_build_filename (g_get_home_dir (), "chiron", NULL);
  if (!g_file_test (fn, G_FILE_TEST_EXISTS))
    return TRUE;

  /* the trigger exists, so create a fake app */
  app = gs_app_new ("example:chiron.desktop");
  gs_app_set_management_plugin (app, "example");
  gs_app_set_kind (app, AS_APP_KIND_DESKTOP);
  gs_app_set_state (app, AS_APP_STATE_INSTALLED);
  gs_app_set_name (app, GS_APP_QUALITY_NORMAL,
                   "Chiron");
  gs_app_set_summary (app, GS_APP_QUALITY_NORMAL,
                      "A teaching application");
  gs_app_set_description (app, GS_APP_QUALITY_NORMAL,
        "Chiron is the name of an application.\n\n"
        "It can be used to demo some of our features");

  /* these are all optional */
  gs_app_set_version (app, "1.2.3");
  gs_app_set_size_installed (app, 2 * 1024 * 1024);
  gs_app_set_size_download (app, 3 * 1024 * 1024);
  gs_app_set_origin_ui (app, "The example plugin");
  gs_app_add_category (app, "Game");
  gs_app_add_category (app, "ActionGame");
  gs_app_add_kudo (app, GS_APP_KUDO_INSTALLS_USER_DOCS);
  gs_app_set_license (app, GS_APP_QUALITY_NORMAL,
                      "GPL-2.0+ and LGPL-2.1+");

  /* create a stock icon (loaded by the 'icons' plugin) */
  icon = as_icon_new ();
  as_icon_set_kind (icon, AS_ICON_KIND_STOCK);
  as_icon_set_name (icon, "input-gaming");
  gs_app_set_icon (app, icon);

  /* return new app */
  gs_app_list_add (list, app);

  return TRUE;
}

This shows a lot of the plugin architecture in action. Some notable points:

  • The application ID (example:chiron.desktop) has a prefix of example which means we can co-exist with any package or flatpak version of the Chiron application, not setting the prefix would make the UI confused if more than one chiron.desktop got added.
  • Setting the management plugin means we can check for this string when working out if we can handle the install or remove action.
  • Most applications want a kind of AS_APP_KIND_DESKTOP to be visible as an application.
  • The origin is where the application originated from — usually this will be something like Fedora Updates.
  • The GS_APP_KUDO_INSTALLS_USER_DOCS means we get the blue “Documentation” award in the details page; there are many kudos to award to deserving apps.
  • Setting the license means we don’t get the non-free warning — removing the 3rd party warning can be done using AS_APP_QUIRK_PROVENANCE
  • The icons plugin will take the stock icon and convert it to a pixbuf of the correct size.

To show this fake application just compile and install the plugin, touch ~/chiron and then restart gnome-software.

Screenshot from 2016-05-20 21-22-38

By filling in the optional details (which can also be filled in using gs_plugin_refine_app() (to be covered in a future blog post) you can also make the details page a much more exciting place. Adding a set of screenshots is left as an exercise to the reader.

Screenshot from 2016-05-20 21-22-46

For anyone interested, I’m also slowly writing up these blog posts into proper docbook and uploading them with the gtk-doc files here. I think this documentation would have been really useful for the Endless and Ubuntu people a few weeks ago, so if anyone sees any typos or missing details please let me know.

May 19, 2016

Your project's RCS history affects ease of contribution (or: don't squash PRs)
Github recently introduced the option to squash commits on merge, and even before then several projects requested that contributors squash their commits after review but before merge. This is a terrible idea that makes it more difficult for people to contribute to projects.

I'm spending today working on reworking some code to integrate with a new feature that was just integrated into Kubernetes. The PR in question was absolutely fine, but just before it was merged the entire commit history was squashed down to a single commit at the request of the reviewer. This single commit contains type declarations, the functionality itself, the integration of that functionality into the scheduler, the client code and a large pile of autogenerated code.

I've got some familiarity with Kubernetes, but even then this commit is difficult for me to read. It doesn't tell a story. I can't see its growth. Looking at a single hunk of this diff doesn't tell me whether it's infrastructural or part of the integration. Given time I can (and have) figured it out, but it's an unnecessary waste of effort that could have gone towards something else. For someone who's less used to working on large projects, it'd be even worse. I'm paid to deal with this. For someone who isn't, the probability that they'll give up and do something else entirely is even greater.

I don't want to pick on Kubernetes here - the fact that this Github feature exists makes it clear that a lot of people feel that this kind of merge is a good idea. And there are certainly cases where squashing commits makes sense. Commits that add broken code and which are immediately followed by a series of "Make this work" commits also impair readability and distract from the narrative that your RCS history should present, and Github present this feature as a way to get rid of them. But that ends up being a false dichotomy. A history that looks like "Commit", "Revert Commit", "Revert Revert Commit", "Fix broken revert", "Revert fix broken revert" is a bad history, as is a history that looks like "Add 20,000 line feature A", "Add 20,000 line feature B".

When you're crafting commits for merge, think about your commit history as a textbook. Start with the building blocks of your feature and make them one commit. Build your functionality on top of them in another. Tie that functionality into the core project and make another commit. Add client support. Add docs. Include your tests. Allow someone to follow the growth of your feature over time, with each commit being a chapter of that story. And never, ever, put autogenerated code in the same commit as an actual functional change.

People can't contribute to your project unless they can understand your code. Writing clear, well commented code is a big part of that. But so is showing the evolution of your features in an understandable way. Make sure your RCS history shows that, otherwise people will go and find another project that doesn't make them feel frustrated.

(Edit to add: Sarah Sharp wrote on the same topic a couple of years ago)

comment count unavailable comments
External plugins in GNOME Software

I’ve just pushed a set of patches to gnome-software master that allow people to compile out-of-tree gnome-software plugins.

In general, building things out-of-tree isn’t something that I think is a very good idea; the API and ABI inside gnome-software is still changing and there’s a huge benefit to getting plugins upstream where they can undergo review and be ported as the API adapts. I’m also super keen to provide configurability in GSettings for doing obviously-useful things, the sort of thing Fleet Commander can set for groups of users. However, now we’re shipping gnome-software in enterprise-class distros we might want to allow customers to ship thier own plugins to make various business-specific changes that don’t make sense upstream. This might involve querying a custom LDAP server and changing the suggested apps to reflect what groups the user is in, or might involve showing a whole new class of applications that does not conform to the Linux-specific “application is a desktop-file” paradigm. This is where a plugin makes sense, and something I’d like to support in future updates to RHEL 7.

At this point it probably makes sense to talk a bit about how the architecture of gnome-software works. At its heart it’s just a big plugin loader that has some GTK UI that gets created for various result types. The idea is we have lots of small plugins that each do one thing and then pass the result onto the other plugins. These are ordered by dependencies against each other at runtime and each one can do things like editing an existing application or adding a new application to the result set. This is how we can add support for things like firmware updating, Steam, GNOME Shell web-apps and flatpak bundles without making big changes all over the source tree.

There are broadly 3 types of plugin methods:

  • Actions: Do something on a specific GsApp; install gimp.desktop
  • Refine: Get details about a specific GsApp; is firefox.desktop installed? or get reviews for inkscape.desktop
  • Adopt: Can this plugin handle this GsApp; can fwupd handle com.hughski.ColorHug2.firmware

You only need to define the vfuncs that the plugin needs, and the name is taken automatically from the suffix of the .so file. So, lets look at a sample plugin one chunk at a time, taking it nice and slow. First the copyright and licence (it only has to be GPLv2+ if it’s headed upstream):

/*
 * Copyright (C) 2016 Richard Hughes 
 * Licensed under the GNU General Public License Version 2
 */

Then, the magic header that sucks in everything that’s exported:

#include <gnome-software.h>

Then we have to define when our plugin is run in reference to other plugins, as we’re such a simple plugin we’re relying on another plugin to run after us to actually make the GsApp “complete”, i.e. adding icons and long descriptions:

void
gs_plugin_initialize (GsPlugin *plugin)
{
  gs_plugin_add_rule (plugin, GS_PLUGIN_RULE_RUN_BEFORE, "appstream");
}

Then we can start to do something useful. In this example I want to show GIMP as a result (from any provider, e.g. flatpak or a distro package) when the user searches exactly for fotoshop. There is no prefixing or stemming being done for simplicity.

gboolean
gs_plugin_add_search (GsPlugin *plugin,
                      gchar **values,
                      GsAppList *list,
                      GCancellable *cancellable,
                      GError **error)
{
  guint i;
  for (i = 0; values[i] != NULL; i++) {
    if (g_strcmp0 (values[i], "fotoshop") == 0) {
      g_autoptr(GsApp) app = gs_app_new ("gimp.desktop");
      gs_app_add_quirk (app, AS_APP_QUIRK_MATCH_ANY_PREFIX);
      gs_app_list_add (list, app);
    }
  }
  return TRUE;
}

We can then easily build and install the plugin using:

gcc -shared -o libgs_plugin_example.so gs-plugin-example.c -fPIC \
 `pkg-config --libs --cflags gnome-software` \
 -DI_KNOW_THE_GNOME_SOFTWARE_API_IS_SUBJECT_TO_CHANGE &&
 sudo cp libgs_plugin_example.so `pkg-config gnome-software --variable=plugindir`

Screenshot from 2016-05-19 10-39-53

I’m going to be cleaning up the exported API and adding some more developer documentation before I release the next tarball, but if this is useful to you please let me know and I’ll do some more blog posts explaining more how the internal architecture of gnome-software works, and how you can do different things with plugins.

May 16, 2016

NetworkManager and WiFi Scans

Recently Dave Täht wrote a blog post investigating latency and WiFi scanning and came across NetworkManager’s periodic scan behavior.  When a WiFi device scans it obviously must change from its current radio channel to other channels and wait for a short amount of time listening for beacons from access points.  That means it’s not passing your traffic.

With a bad driver it can sometimes take 20+ seconds and all your traffic gets dropped on the floor.

With a good driver scanning takes only a few seconds and the driver breaks the scan into chunks, returning to the associated access point’s channel periodically to handle pending traffic.  Even with a good driver, latency-critical applications like VOIP or gaming will clearly suffer while the WiFi device is listening on another channel.

So why does NetworkManager periodically scan for WiFi access points?

Roaming

Whenever your WiFi network has multiple access points with the same SSID (or a dual-band AP with a single SSID) you need roaming to maintain optimal connectivity and speed.  Jumping to a better AP requires that the device know what access points are available, which means doing a periodic scan like NetworkManager does every 2 minutes.  Without periodic scans, the driver must scan at precisely the worst moment: when the signal quality is bad, and data rates are low, and the risk of disconnecting is higher.

Enterprise WiFi setups make the roaming problem much worse because they often have tens or hundreds of access points in the network and because they typically use high-security 802.1x authentication with EAP.  Roaming with 802.1x introduces many more steps to the roaming process, each of which can fail the roaming attempt.  Strategies like pre-authentication and periodic scanning greatly reduce roaming errors and latency.

User responsiveness and Location awareness

The second reason for periodic scanning is to maintain a list of access points around you for presentation in user interfaces and for geolocation in browsers that support it.  Up until a couple years ago, most Linux WiFi applets displayed a drop-down list of access points that you could click on at any time.  Waiting for 5 to 15 seconds for a menu to populate or ‘nmcli dev wifi list’ to return would be annoying.

But with the proliferation of WiFi (often more than 30 or 40 if you live in a flat) those lists became less and less useful, so UIs like GNOME Shell moved to a separate window for WiFi lists.  This reduces the need for a constantly up-to-date WiFi list and thus for periodic scanning.

To help support these interaction models and click-to-scan behaviors like Mac OS X or Maemo, NetworkManager long ago added a D-Bus API method to request an out-of-band WiFi scan.  While it’s pretty trivial to use this API to initiate geolocation or to refresh the WiFi list based on specific user actions, I’m not aware of any clients using it well.  GNOME Shell only requests scans when the network list is empty and plasma-nm only does so when the user clicks a button.  Instead, UIs should simply request scans periodically while the WiFi list is shown, removing the need for yet another click.

WHAT TO DO

If you don’t care about roaming, and I’m assuming David doesn’t, then NetworkManager offers a simple solution: lock your WiFi connection profile to the BSSID of your access point.  When you do this, NetworkManager understands that you do not want to roam and will disable the periodic scanning behavior.  Explicitly requested scans are still allowed.

You can also advocate that your favorite WiFi interface add support for NetworkManager’s RequestScan() API method and begin requesting periodic scans when WiFi lists are shown or when your browser uses geolocation.  When most do this, perhaps NetworkManager could be less aggressive with its own periodic scans, or perhaps remove them altogether in favor of a more general solution.

That general solution might involve disabling periodic roaming when the signal strength is extremely good and start scanning more aggressively when signal strength drops over a threshold.  But signal strength drops for many reasons like turning on a microwave, closing doors, turning on Bluetooth, or even walking to the next room, and triggering a scan then still interrupts your VOIP call or low ping headshot.  This also doesn’t help people who aren’t close to their access point, leading to the same scanning problem David talks about if you’re in the basement but not if you’re in the bedroom.

Another idea would be to disable periodic scanning when latency critical applications are active, but this requires that these applications consistently set the IPv4 TOS field or use the SO_PRIORITY socket option.  Few do so.  This also requires visibility into kernel mac80211 queue depths and would not work for proprietary or non-mac80211-based drivers.  But if all the pieces fell into place on the kernel side, NetworkManager could definitely do this while waiting for applications and drivers to catch up.

If you’ve got other ideas, feel free to propose them.

May 13, 2016

Blutella, a Bluetooth speaker receiver
Quite some time ago, I was asked for a way to use the AV amplifier (which has a fair bunch of speakers connected to it) in our living-room that didn't require turning on the TV to choose a source.

I decided to try and solve this problem myself, as an exercise rather than a cost saving measure (there are good-quality Bluetooth receivers available for between 15 and 20€).

Introducing Blutella



I found this pot of Nutella in my travels (in Europe, smaller quantities are usually in a jar that looks like a mustard glass, with straight sides) and thought it would be a perfect receptacle for a CHIP, to allow streaming via Bluetooth to the amp. I wanted to make a nice how-to for you, dear reader, but best laid plans...

First, the materials:
  • a CHIP
  • jar of Nutella, and "Burnt umber" acrylic paint
  • micro-USB to USB-A and jack 3.5mm to RCA cables
  • Some white Sugru, for a nice finish around the cables
  • bit of foam, a Stanley knife, a CD marker

That's around 10€ in parts (cables always seem to be expensive), not including our salvaged Nutella jar, and the CHIP itself (9$ + shipping).

You'll start by painting the whole of the jar, on the inside, with the acrylic paint. Allow a couple of days to dry, it'll be quite thick.

So, the plan that went awry. Turns out that the CHIP, with the cables plugged in, doesn't fit inside this 140g jar of Nutella. I also didn't make the holes exactly in the right place. The CHIP is tiny, but not small enough to rotate inside the jar without hitting the side, and the groove to screw the cap also have only one position.

Anyway, I pierced two holes in the lid for the audio jack and the USB charging cable, stuffed the CHIP inside, and forced the lid on so it clipped on the jar's groove.

I had nice photos with foam I cut to hold the CHIP in place, but the finish isn't quite up to my standards. I guess that means I can attempt this again with a bigger jar ;)

The software

After flashing the CHIP with Debian, I logged in, and launched a script which I put together to avoid either long how-tos, or errors when I tried to reproduce the setup after a firmware update and reset.

The script for setting things up is in the CHIP-bluetooth-speaker repository. There are a few bugs due to drivers, and lack of integration, but this blog is the wrong place to track them, so check out the issues list.

Apart from those driver problems, I found the integration between PulseAudio and BlueZ pretty impressive, though I wish there was a way for the speaker to reconnect to the phone I streamed from when turned on again, as Bluetooth speakers and headsets do, removing one step from playing back audio.

May 12, 2016

Convenience, security and freedom - can we pick all three?
Moxie, the lead developer of the Signal secure communication application, recently blogged on the tradeoffs between providing a supportable federated service and providing a compelling application that gains significant adoption. There's a set of perfectly reasonable arguments around that that I don't want to rehash - regardless of feelings on the benefits of federation in general, there's certainly an increase in engineering cost in providing a stable intra-server protocol that still allows for addition of new features, and the person leading a project gets to make the decision about whether that's a valid tradeoff.

One voiced complaint about Signal on Android is the fact that it depends on the Google Play Services. These are a collection of proprietary functions for integrating with Google-provided services, and Signal depends on them to provide a good out of band notification protocol to allow Signal to be notified when new messages arrive, even if the phone is otherwise in a power saving state. At the time this decision was made, there were no terribly good alternatives for Android. Even now, nobody's really demonstrated a free implementation that supports several million clients and has no negative impact on battery life, so if your aim is to write a secure messaging client that will be adopted by as many people is possible, keeping this dependency is entirely rational.

On the other hand, there are users for whom the decision not to install a Google root of trust on their phone is also entirely rational. I have no especially good reason to believe that Google will ever want to do something inappropriate with my phone or data, but it's certainly possible that they'll be compelled to do so against their will. The set of people who will ever actually face this problem is probably small, but it's probably also the set of people who benefit most from Signal in the first place.

(Even ignoring the dependency on Play Services, people may not find the official client sufficient - it's very difficult to write a single piece of software that satisfies all users, whether that be down to accessibility requirements, OS support or whatever. Slack may be great, but there's still people who choose to use Hipchat)

This shouldn't be a problem. Signal is free software and anybody is free to modify it in any way they want to fit their needs, and as long as they don't break the protocol code in the process it'll carry on working with the existing Signal servers and allow communication with people who run the official client. Unfortunately, Moxie has indicated that he is not happy with forked versions of Signal using the official servers. Since Signal doesn't support federation, that means that users of forked versions will be unable to communicate with users of the official client.

This is awkward. Signal is deservedly popular. It provides strong security without being significantly more complicated than a traditional SMS client. In my social circle there's massively more users of Signal than any other security app. If I transition to a fork of Signal, I'm no longer able to securely communicate with them unless they also install the fork. If the aim is to make secure communication ubiquitous, that's kind of a problem.

Right now the choices I have for communicating with people I know are either convenient and secure but require non-free code (Signal), convenient and free but insecure (SMS) or secure and free but horribly inconvenient (gpg). Is there really no way for us to work as a community to develop something that's all three?

comment count unavailable comments

May 11, 2016

CfP is now open

The systemd.conf 2016 Call for Participation is Now Open!

We’d like to invite presentation and workshop proposals for systemd.conf 2016!

The conference will consist of three parts:

  • One day of workshops, consisting of in-depth (2-3hr) training and learning-by-doing sessions (Sept. 28th)
  • Two days of regular talks (Sept. 29th-30th)
  • One day of hackfest (Oct. 1st)

We are now accepting submissions for the first three days: proposals for workshops, training sessions and regular talks. In particular, we are looking for sessions including, but not limited to, the following topics:

  • Use Cases: systemd in today’s and tomorrow’s devices and applications
  • systemd and containers, in the cloud and on servers
  • systemd in distributions
  • Embedded systemd and in IoT
  • systemd on the desktop
  • Networking with systemd
  • … and everything else related to systemd

Please submit your proposals by August 1st, 2016. Notification of acceptance will be sent out 1-2 weeks later.

If submitting a workshop proposal please contact the organizers for more details.

To submit a talk, please visit our CfP submission page.

For further information on systemd.conf 2016, please visit our conference web site.

May 09, 2016

Blog backlog, Post 2, xdg-app bundles

I recently worked on creating an xdg-app bundle for GNOME Videos, aka Totem, so it would be built along with other GNOME applications, every night, and made available via the GNOME xdg-app repositories.

There's some functionality that's not working yet though:
  • No support for optical discs
  • The MPRIS plugin doesn't work as we're missing dbus-python (I'm not sure that the plugin will survive anyway, it's more suited to audio players, don't worry though, it's not going to be removed until we have made changes to the sound system in GNOME)
  • No libva/VDPAU hardware acceleration (which would require plugins, and possibly device access of some sort)
However, I created a bundle that extends the freedesktop runtime, that contains gst-libav. We'll need to figure out a way to distribute it in a way that doesn't cause problems for US hosts.

As we also have a recurring problem in Fedora with rpmfusion being out of date, and I sometimes need a third-party movie player to test things out, I put together an mpv manifest, which is the only MPlayer-like with a .desktop and a GUI when launched without any command-line arguments.

Finally, I put together a RetroArch bundle for research into a future project, which uncovered the lack of joystick/joypad support in the xdg-app sandbox.

Hopefully, those few manifests will be useful to other application developers wanting to distribute their applications themselves. There are some other bundles being worked on, and that can be used as examples, linked to in the Wiki.

May 05, 2016

Blog backlog, Post 1, Emoji
Short version


dnf copr enable hadess/emoji
dnf update cairo
dnf install eosrei-emojione-fonts



Long version

A little while ago, I was reading this article, called "Emoji: how do you get from U+1F355 to 🍕?", which said, and I reluctantly quote: "[...] and I don’t know what Linux does, but it’s probably black and white and who cares [...]".

Well. I care. And you probably do as well if your pizza slice above is black and white.

So I set out to check on the status of Behdad Esfahbod (or just "Behdad" as we know him)'s patches to add colour font support to cairo, which he presented at GUADEC in Strasbourg Gothenburg. It adds support for the "bitmap in font" as Android does, and as freetype supports.

It kind of worked, and Matthias Clasen reworked the patches a few times, completing the support. This is probably not the code that will be worked on and will land in cairo, but it's a good enough base for people interested in contributing to use.

After that, we needed something to display using that feature. We ended up using the same font recommended in this article, the Emoji One font.


There's still plenty to be done to support emojis, even after the cairo support is merged. We'd need a way to input emojis (maybe Lalo Martins is listening), and support in a lot of toolkits other than GNOME (Firefox only supports the SVG-in-OTF format, WebKit, Chrome, LibreOffice don't seem to know about colour fonts either).

You can find more information about design interests in GNOME around Emoji on the Wiki.

Update: Behdad's presentation was in Gothenburg, not Strasbourg. You can also see the video on YouTube.

May 04, 2016

The difference between uinput and evdev

A recurring question I encounter is the question whether uinput or evdev should be the approach do implement some feature the user cares about. This question is unfortunately wrongly framed as uinput and evdev have no real overlap and work independent of each other. This post outlines what the differences are. Note that "evdev" here refers to the kernel API, not to the X.Org evdev driver.

First, the easy flowchart: do you have to create a new virtual device that has a set of specific capabilities? Use uinput. Do you have to read and handle events from an existing device? Use evdev. Do you have to create a device and read events from that device? You (probably) need two processes, one doing the uinput bit, one doing the evdev bit.

Ok, let's talk about the difference between evdev and uinput. evdev is the default input API that all kernel input device nodes provide. Each device provides one or more /dev/input/eventN nodes that a process can interact with. This usually means checking a few capability bits ("does this device have a left mouse button?") and reading events from the device. The events themselves are in the form of struct input_event, defined in linux/input.h and consist of a event type (relative, absolute, key, ...) and an event code specific to the type (x axis, left button, etc.). See linux/input-event-codes.h for a list or linux/input.h in older kernels.Specific to evdev is that events are serialised - framed by events of type EV_SYN and code SYN_REPORT. Anything before a SYN_REPORT should be considered one logical hardware event. For example, if you receive an x and y movement within the same SYN_REPORT frame, the device has moved diagonally.

Any event coming from the physical hardware goes into the kernel's input subsystem and is converted to an evdev event that is then available on the event node. That's pretty much it for evdev. It's a fairly simple API but it does have some quirks that are not immediately obvious so I recommend using libevdev whenever you actually need to communicate with a kernel device directly.

uinput is something completely different. uinput is an kernel device driver that provides the /dev/uinput node. A process can open this node, write a bunch of custom commands to it and the kernel then creates a virtual input device. That device, like all others, presents an /dev/input/eventN node. Any event written to the /dev/uinput node will re-appear in that /dev/input/eventN node and a device created through uinput looks just pretty much like a physical device to a process. You can detect uinput-created virtual devices, but usually a process doesn't need to care so all the common userspace (libinput, Xorg) doesn't bother. The evemu tool is one of the most commonly used applications using uinput.

Now, there is one thing that may cause confusion: first, to set up a uinput device you'll have to use the familiar evdev type/code combinations (followed-by a couple of uinput-specific ioctls). Events written to uinput also use the struct input_event form, so looking at uinput code one can easily mistake it for evdev code. Nevertheless, the two serve a completely different purpose. As with evdev, I recommend using libevdev to initalise uinput devices. libevdev has a couple of uinput-related functions that make life easier.

Below is a basic illustration of how things work together. The physical devices send their events through the event nodes and libinput is a process that reads those events. evemu talks to the uinput module and creates a virtual device which then too sends events through its event node - for libinput to read.

April 30, 2016

Yet another GTK+ update

GTK+ 3.20 was released a while ago; we’re up to 3.20.3 now.  As I tried to explain in earlier posts here and here, this was a pretty active development cycle for GTK+. We landed a lot of of new stuff, and many things have changed.

I’m using the neutral term changed here for a reason. How you view changes depends a lot on your perspective. Us, who implemented the changes, are of course convinced that they are great improvements. Others who maintain GTK+ themes or applications may have a different take, since changes often imply that they have to do work to adapt.

What changed in GTK+

A big set of changes is related to the inner workings of GTK+ CSS.

The CSS box model is much better supported in widgets. This includes padding, margins, borders, shadows, and the min-width and min-height properties. Since many widgets are complex, they typically have many CSS boxes. Here is how the box tree GtkNotebook looks:

notebook-nodes

In the past (up to and including GTK+ 3.18), we used a mixture of widget class names (like GtkNotebook), style classes (like .button) and widget names (like #cancel_button) for matching styles to widgets. Now, we are using element names for each box (e.g. header, tabs and tab in the screenshot above). Style classes are still used for optional things and variants.

The themes that are included in GTK+ (Adwaita, Adwaita dark, HighContrast, HighContrastInverse and the win32 theme) have of course been updated to follow this new naming scheme. Third-party themes need and application-specific CSS need to be updated for this too.

To help with this, we have expanded both the general documentation about CSS support in GTK+ here and here, and we have documented the element names, style classes and the node hierarchy for each widget. Here, for example, is the notebook documentation.

The documentation is also a good place to learn about style properties that have been deprecated in favor of equivalent CSS properties, like the secondary cursor color property. We warn about deprecated style properties that are used in themes or custom CSS, so it is easy to find and replace them:

(gtk3-demo:14116): Gtk-WARNING **: Theme parsing error: gtk-contained.css:18:37: The style property GtkWidget:secondary-cursor-color is deprecated and shouldn't be used anymore. It will be removed in a future version

There’s also a number of new features in CSS. We do support the CSS syntax for radial gradients, we let you load and recolor symbolic icons, image() and calc() are supported, as well as the rem (‘root em’) unit.

Beyond CSS, the drag-and-drop code as been rearchitected to move the drag cancel animation and most input handling into GDK, thereby dropping most of the platform-dependent code out of GTK+.  The main reason for doing this was to enable a complete DND implementation for Wayland. As a side-effect, we gained the ability to use non-toplevel widgets as drag icons, and we dropped the X11 specific feature to use RGBA cursors as drag icons.

The Wayland backend has grown most features that it was missing compared to X11:  the already mentioned full DND support, kinetic scrolling, startup notification, primary selection, presenting windows, a bell.

Changes in applications

Here is an unsorted list of issues that may show up in applications with GTK+ 3.20, with some advice on how to handle them.

One of the motivations for the changes is to enable animations and transitions. If you use gtk_style_context_save/restore in your draw() function, that prevents GTK+ from keeping the state that is needed to support animations; so you should avoid it when you can.

There is one place where you need to use gtk_style_context_save(), though: when using “theme colors”.  The function gtk_style_context_get_color() will warn when you pass a state other than the current state of the context. To avoid the warning, save the context and set the state:

gtk_style_context_save (context);
gtk_style_context_set_state (context, state);
gtk_style_context_get_color (context, state, &color);
gtk_style_context_restore (context);

And yes, it has been pointed out repeatedly that this change makes the state parameter of gtk_style_context_get_color() and similar functions largely useless – this API has been around sinc e 3.0, when the CSS machinery was much less developed than it is now. Back then, passing in a different state was not a problem (because animations were not really supported).

Another word of caution about  “theme colors”: CSS has no concept of foreground/background color pairs. The CSS background is just an image, which is why gtk_style_context_get_background_color() is deprecated and we cannot generally make it return a useful color. The proper way to have a theme-provided background in a widget is to call gtk_widget_render_background() in your draw() function.

If you are using type names of GTK+ widgets in your CSS, look up the element names in the documentation and use them instead. For your own widgets, use gtk_widget_class_set_css_name() to give them an element name, and use it in the CSS.

A problem that we’ve seen in some applications is the interaction between size_allocate() and draw(). GTK+’s CSS boxes need to know their size before they can draw. If you derive from a GTK+ widget and override size_allocate without chaining up, then GTK+ does not get a chance to assign sizes to the boxes. This will lead to critical warnings from GTK+’s draw() function if you don’t override it. The possible solutions to this problem are either to chain up in size_allocate or to provide your own draw implementation.

If you are using GTK+ just for themed drawing without using GTK+ widgets, you probably need to make some changes in the way you are getting theme information. We have added a foreing drawing example to gtk3-demo that shows how this can be done. The example was written with the help of libreoffice and firefox developers, and we intend to keep it up-to-date to ensure that this use case is not neglected.

A plea

If you are maintaining a GTK+ application (in particular, a big one like, say, inkscape), and you are looking at porting from GTK+ 2 to GTK+ 3, or updating it to keep up with the changes in 3.20, please let us know about the issues you find. Such feedback will be useful input for us when we get together for a GTK+ hackfest in a few weeks.

Whats coming

One of the big incoming changes for 3.22 is a GL-based renderer and scene graph. Emmanuele has been working on this on-and-off for quite a while – you may have seen some of his earlier presentations. Together with the recent merge of (copies of) clutter and cogl into mutter, this will put clutter on the path towards retirement.

 

April 27, 2016

3rd Party Fedora Repositories and AppStream

I was recently asked how to make 3rd party repositories add apps to GNOME Software. This is relevant if you run a internal private repo for employee tools, or are just kind enough to provide a 3rd party repo for Fedora or RHEL users for your free or non-free applications.

In most cases people are already running something like this to generate the repomd metadata files on a directory of RPM files:

createrepo_c --no-database --simple-md-filenames SRPMS/
createrepo_c --no-database --simple-md-filenames x86_64/

So, we need to actually generate the AppStream XML. This works by exploding any interesting .rpm files and merging together the .desktop file, the .appdata.xml file and preprocessing some icons. Only applications installing AppData files will be shown in GNOME Software, so you might need to fix before you start.

appstream-builder			\
	--origin=yourcompanyname	\
	--basename=appstream		\
	--cache-dir=/tmp/asb-cache	\
	--enable-hidpi			\
	--max-threads=1			\
	--min-icon-size=32		\
	--output-dir=/tmp/asb-md	\
	--packages-dir=x86_64/		\
	--temp-dir=/tmp/asb-icons

This takes a second or two (or 40 minutes if you’re trying to process the entire Fedora archive…) and spits out some files to /tmp/asb-md — you probably want to change some things there to make more sense for your build server.

We then have to take the generated XML and the tarball of icons and add it to the repomd.xml master document so that GNOME Software (via PackageKit) automatically downloads the content for searching. This is as simple as doing:

modifyrepo_c				\
	--no-compress			\
	--simple-md-filenames		\
	/tmp/asb-md/appstream.xml.gz	\
	x86_64/repodata/
modifyrepo_c				\
	--no-compress			\
	--simple-md-filenames		\
	/tmp/asb-md/appstream-icons.tar.gz	\
	x86_64/repodata/

Any questions, please ask. If you’re using a COPR then all these steps are done for you automatically. If you’re using xdg-app already, then this is all magically done for you as well, and automatically downloaded by GNOME Software.

April 22, 2016

Circumventing Ubuntu Snap confinement
Ubuntu 16.04 was released today, with one of the highlights being the new Snap package format. Snaps are intended to make it easier to distribute applications for Ubuntu - they include their dependencies rather than relying on the archive, they can be updated on a schedule that's separate from the distribution itself and they're confined by a strong security policy that makes it impossible for an app to steal your data.

At least, that's what Canonical assert. It's true in a sense - if you're using Snap packages on Mir (ie, Ubuntu mobile) then there's a genuine improvement in security. But if you're using X11 (ie, Ubuntu desktop) it's horribly, awfully misleading. Any Snap package you install is completely capable of copying all your private data to wherever it wants with very little difficulty.

The problem here is the X11 windowing system. X has no real concept of different levels of application trust. Any application can register to receive keystrokes from any other application. Any application can inject fake key events into the input stream. An application that is otherwise confined by strong security policies can simply type into another window. An application that has no access to any of your private data can wait until your session is idle, open an unconfined terminal and then use curl to send your data to a remote site. As long as Ubuntu desktop still uses X11, the Snap format provides you with very little meaningful security. Mir and Wayland both fix this, which is why Wayland is a prerequisite for the sandboxed xdg-app design.

I've produced a quick proof of concept of this. Grab XEvilTeddy from git, install Snapcraft (it's in 16.04), snapcraft snap, sudo snap install xevilteddy*.snap, /snap/bin/xevilteddy.xteddy . An adorable teddy bear! How cute. Now open Firefox and start typing, then check back in your terminal window. Oh no! All my secrets. Open another terminal window and give it focus. Oh no! An injected command that could instead have been a curl session that uploaded your private SSH keys to somewhere that's not going to respect your privacy.

The Snap format provides a lot of underlying technology that is a great step towards being able to protect systems against untrustworthy third-party applications, and once Ubuntu shifts to using Mir by default it'll be much better than the status quo. But right now the protections it provides are easily circumvented, and it's disingenuous to claim that it currently gives desktop users any real security.

comment count unavailable comments

April 21, 2016

You and NetworkManager 1.2 Can Still Ride Together
<figure class="wp-caption alignnone" id="attachment_837" style="width: 640px"><figcaption class="wp-caption-text">You don’t need an Uber, you don’t need a cab (via Casey Bisson CC BY-NC-SA 2.0)</figcaption></figure>

NetworkManager 1.2 was released yesterday, and it’s already built for Fedora (24 and rawhide), a release candidate is in Ubuntu 16.04, and it should appear in other distros soon too.  Lubo wrote a great post on many of the new features, but there’s too many to highlight in one post for our ADD social media 140-character tap-tap generation to handle.  Ready for more?

indicator menus

appletWayland is coming, and it doesn’t support the XEmbed status icons like nm-applet creates.  Desktop environments also want more control over how these status menus appear.  While KDE and GNOME both provide their own network status menus Ubuntu, XFCE, and LXDE use nm-applet.  How do they deal with lack of XEmbed and status icons?

Ubuntu has long patched nm-applet to add App Indicator support, which exposes the applet’s menu structure as D-Bus objects to allow the desktop environment to draw the menu just like it wants.  We enhanced the GTK3 support in libdbusmenu-gtk to handle nm-applet’s icons and then added an indicator mode to nm-applet based off Ubuntu’s work.  We’ve made packager’s lives easier by building both modes into the applet simultaneously and allowing them to be switched at runtime.

IP reconfiguration

Want to add a second IP address or change your DNS servers right away?  With NetworkManager 1.2 you can now change the IP configuration of a device through the D-Bus interface or nmcli without triggering a reconnect.  This lets the network UIs like KDE or GNOME control-center apply changes you make to network configuration immediately without interrupting your network connection.  That might take a cycle  or two to show up in your favorite desktop environment, but the basis is there.

802.1x/WPA Enterprise authentication

An oft-requested feature was the ability to use certificate domain suffix checking to validate an authentication server.  While NetworkManager has supported certificate subject checking for years, this has limitations and isn’t as secure as domain suffix checking.  Both these options help prevent man-in-the-middle attacks where a rogue access point could masquerade as as your normal secure network.  802.1x authentication is still too complicated, and we hope to greatly simplify it in upcoming releases.

Interface stacking

While NM has always been architected to allow bridges-on-bonds-on-VLANs, there were some internal issues that prevented these more complicated configurations from working.  We’ve fixed those bugs, so now layer-cake network setups work in a flash!  Hopefully somebody will come up with a fancy drag-n-drop UI based off Minecraft or CandyCrush with arbitrary interface trees.  Maybe it’ll even have trophies when you finally get a Level 48 active-backup bond.

Old Stable Series

Now that 1.2 is out, the 1.0 series is in maintenance mode.  We’ll fix bugs and any security issues that come up, but typically don’t add new features.  Backporting from 1.2 to 1.0 will be even more difficult due to the removal of dbus-glib, a major feature in 1.2 release.  If you’re on 1.0, 0.9.10, or (gasp!) 0.9.8 I’d urge you to upgrade, and I think you’ll like what you see!

April 20, 2016

Upgrading Fedora 23 to 24 using GNOME Software

I’ve spent the last couple of days fixing up all the upgrade bugs in GNOME Software and backporting them to gnome-3-20. The idea is that we backport gnome-software plus a couple of the deps into Fedora 23 so that we can offer a 100% GUI upgrade experience. It’s the first time we’ve officially transplanted a n+1 GNOME component into an older release (ignoring my unofficial Fedora 20 whole-desktop backport COPR) and so we’re carefully testing for regressions and new bugs.

If you do want to test upgrading from F23 to F24, first make sure you’ve backed up your system. Then, install and enable this COPR and update gnome-software. This should also install a new libhif, libappstream-glib, json-glib and PackageKit and a few other bits. If you’ve not done the update offline using [the old] GNOME Software, you’ll need to reboot at this stage as well.

Fire up the new gnome-software and look at the new UI. Actually, there’s not a lot new to see as we’ve left new features like the ODRS reviewing service and xdg-app as F24-only features, so it should be mostly the same as before but with better search results. Now go to the Updates page which will show any updates you have pending, and it will also download the list of possible distro upgrades to your home directory.

As we’re testing upgrading to a pre-release, we have to convince gnome-software that we’re living in the future. First, open ~/.cache/gnome-software/3.20/upgrades/fedora.json and search for f24. Carefully change the Under Development string to Active then save the file. Log out, back in and the launch gnome-software again or wait for the notification from the shell. If all has gone well you should see a banner telling you about the new upgrade. If you click Download go and get a coffee and start baking a cake, as it’s going to take a long time to download all that new goodness. Once complete just click Install, which prompts a reboot where the packages will be installed. For this step you’ll probably want to bake another cake. We’re not quite in an atomic instant-apply world yet, although I’ll be talking a lot more about that for Fedora 25.

With a bit of luck, after 30 minutes staring at a progressbar the computer should reboot itself into a fresh new Fedora 24 beta installation. Success!

Screenshot_Fedora23-Upgrade_2016-04-20_15:23:27

If you spot any problems or encounter any bugs, please let me know either in bugzilla, email or or IRC. I’ve not backported all the custom CSS for the upgrade banner just yet, but this should be working soon. Thanks!

April 18, 2016

libinput and graphics tablet pad support

When we released graphics tablet support in libinput earlier this year, only tablet tools were supported. So while you could use the pen normally, the buttons, rings and strips on the physical tablet itself (the "pad") weren't detected by libinput and did not work. I have now merged the patches for pad support into libinput.

The reason for the delay was simple: we wanted to get it right [1]. Pads have a couple of properties that tools don't have and we always considered pads to be different to pens and initially focused on a more generic interface (the "buttonset" interface) to accommodate for those. After some coding, we have now arrived at a tablet pad-specific interface instead. This post is a high-level overview of the new tablet pad interface and how we intend it do be used.

The basic sign that a pad is present is when a device has the tablet pad capability. Unlike tools, pads don't have proximity events, they are always considered in proximity and it is up to the compositor to handle the focus accordingly. In most cases, this means tying it to the keyboard focus. Usually a pad is available as soon as a tablet is plugged in, but note that the Wacom ExpressKey Remote (EKR) is a separate, wireless device and may be connected after the physical pad. It is up to the compositor to link the EKR with the correct tablet (if there is more than one).

Pads have three sources of events: buttons, rings and strips. Rings and strips are touch-sensitive surfaces and provide absolute values - rings in degrees, strips in normalized [0.0, 1.0] coordinates. Similar to pointer axis sources we provide a source notification. If that source is "finger", then we send a terminating out-of-range event so that the caller can trigger things like kinetic scrolling.

Buttons on a pad are ... different. libinput usually re-uses the Linux kernel's include/input.h event codes for buttons and keys. But for the pad we decided to use plain sequential button numbering, starting at index 0. So rather than a semantic code like BTN_LEFT, you'd simply get a button 0 event. The reasoning behind this is a caveat in the kernel evdev API: event codes have semantic meaning (e.g. BTN_LEFT) but buttons on a tablet pad don't those meanings. There are some generic event ranges (e.g. BTN_0 through to BTN_9) and the Wacom tablets use those but once you have more than 10 buttons you leak into other ranges. The ranges are simply too narrow so we end up with seemingly different buttons even though all buttons are effectively the same. libinput's pad support undoes that split and combines the buttons into a simple sequential range and leaves any semantic mapping of buttons to the caller. Together with libwacom which describes the location of the buttons a caller can get a relatively good idea of how the layout looks like.

Mode switching is a commonly expected feature on tablet. One button is designated as mode switch button and toggles all other buttons between the available modes. On the Intuos Pro series tablets, that button is usually the button inside the ring. Button mapping and thus mode switching is however a feature we leave up to the caller, if you're working on a compositor you will have to implemented mode switching there.

Other than that, pad support is relatively simple and straightforward and should not cause any big troubles.

[1] or at least less wrong than in the past
[2] They're actually linux/input-event-codes.h in recent kernels

One more attempt at SATA power management
Around a year ago I wrote some patches in an attempt to improve power management on Haswell and Broadwell systems by configuring Serial ATA power management appropriately. I got a couple of reports of them triggering SATA errors for some users, couldn't reproduce them myself and so didn't have a lot of confidence in them. Time passed.

I've been working on power management stuff again this week, so it seemed like a good opportunity to revisit these. I've made a few changes and pushed a couple of trees - one against master and one against 4.5.

First, these probably only have relevance to users of mobile Intel parts in the U or S range (/proc/cpuinfo will tell you - you're looking for a four-digit number that starts with 4 (Haswell), 5 (Broadwell) or 6 (Skylake) and ends with U or S), and won't do anything unless you have SATA drives (including PCI-based SATA). To test them, first disable anything like TLP that might alter your SATA link power management policy. Then check powertop - you should only be getting to PC3 at best. Build a kernel with these patches and boot it. /sys/class/scsi_host/*/link_power_management_policy should read "firmware". Check powertop and see whether you're getting into deeper PC states. Now run your system for a while and check the kernel log for any SATA errors that you didn't see before.

Let me know if you see SATA errors and are willing to help debug this, and leave a comment if you don't see any improvement in PC states.

comment count unavailable comments

April 16, 2016

My first xdg-app

A few days ago, I set out to get some experience with building an application as an xdg-app.  In this post, I’m collecting some of the lessons I learned.

Since I didn’t want pick a too easy test case, I chose terminix, a promising terminal emulator for GNOME. Terminix uses GTK+ and vte, which means that most dependencies are already present in the GNOME runtime.

Terminix

However, terminix is written in D, and the GNOME sdk does not include D support.  So, the challenge here is to build a compiler and runtime for this language, and any other required language-specific utilities. The DMD D compiler is written in D (naturally), so some bootstrapping was required.

Build tools

xdg-app comes with low-level build support in the form of various xdg-app build commands. But you really want to use the newer xdg-app-builder tool. It is really nice.

xdg-app-builder downloads and builds the application and all its dependencies, according to a JSON manifest.  Thats par for the course for modern build tools, of course. But xdg-app-builder also has smart caching: It keeps git checkouts  of all sources (if they are in git), and only rebuilds them when they change. It also keeps the results of each modules’ build in an ostree repository, so reusing a previous build is really fast.

All the caches are kept in .xdg-app-builder/ in the directory where you run the build. If you have many dependencies, this hidden directory can grow large, so you might want to keep an eye on it and clean it out every now and then (remember, it is just a cache).

You can take a look at the JSON file I came up with.

Build API

Similar to the GNOME Continuous build system, xdg-app-builder assumes that each module in your JSON supports the ‘build api’ which consists of configure & make & make install. The world is of course more diverse than that, and rolling your own build system is irresistable for some.

Here is a way to quickly add the required build api support to almost any module (I’ve stolen this setup from the pitivi xdg-app build scripts):

Place a foo-configure script next to your JSON recipe that looks like this (note that Makefile syntax requires tabs that were eaten when I pasted this content in here):

#!/bin/sh

cat <<EOF >Makefile
all:
        ...do whatever is needed to build foo

install:
        ...commands to install foo go here

EOF

In the JSON  fragment for the foo module, you add this file as an extra source (we are taking advantage of the fact that xdg-app-builder allows multiple sources for a module):

"modules": [
    {
        "name": "foo",
        "sources": [
            {
                "type": "git",
                 "url": "http://foo.org/foo.git",
                 "branch": "master"
            },
            {
                "type": "file",
                "path": "foo-configure",
                "dest-filename": "configure"
            }
        ]
    }

I guess you could just as well have the Makefile as yet another source; this approach is following the traditional role of configure scripts to produce Makefiles.

Network access

As I mentioned already, the first step in my build was to build a D compiler written in D. Thankfully, the build script of the dmd compiler is prepared for handling this kind of bootstrapping. It does so by downloading a pre-built D compiler that is used to build the sources.

Downloading things during the build is not great for trusted and repeatable builds. And xdg-app’s build support is set up to produce such builds by running the build in a controlled, sandboxed environment, which doesn’t have network access.

So, in order to get the D compiler built, had to weaken the sandboxing for this module, and grant it network access.  It took me a little while to find out that the build-args field in the build-options does this job:

"modules": [
    {
        "name": "dmd",
        "build-options":
            {
                "build-args": ["--share=network"]
            },
        ...

Shedding weight

After navigating around other hurdles, I eventually succeeded in having a build of my entire JSON recipe run through the end. Yay! But I quickly discovered that the build directory was quite heavy. It came to over 200M, a bit much for a terminal.

xdg-app-builder creates the final build by combining the build results from all the modules in the JSON recipe. That means my terminix build included a D compiler, static and shared libraries for the D runtime, build utilties, etc.

To fix this, I added a couple of cleanup commands to the JSON. These are run after all the modules have been built, and can remove things that are no longer needed.

"cleanup-commands": ["rm -rf /app/lib",
                     "rm -rf /app/src",
                      rm -rf /app/share/nautilus-python",
                      "rm /app/bin/dmd",
                      ...

Note that the paths start with /app, which is the prefix that xdg-app apps are put in (to avoid interference with /usr).

After these cleanups, my build weighed less than a megabyte, which is more acceptable.

Trying it out

The best way to distribute an xdg-app is via an OSTree repository. Since I don’t have a good place to put one, and Terminix is developed on github, I decided to turn my xdg-app into a bundle, using this command:

xdg-app build-bundle ~/xdg-app-repos/terminix \
                     terminix.x86_64.xdgapp \
                     com.gexperts.Terminix \ 
                     master

Since github has a concept of releases, I’ve just put the bundle there:

https://github.com/matthiasclasen/terminix/releases/tag/2016-04-15

Enjoy!

April 15, 2016

David MacKay
The first time I was paid to do software development came as something of a surprise to me. I was working as a sysadmin in a computational physics research group when a friend asked me if I'd be willing to talk to her PhD supervisor. I had nothing better to do, so said yes. And that was how I started the evening having dinner with David MacKay, and ended the evening better fed, a little drunker and having agreed in principle to be paid to write free software.

I'd been hired to work on Dasher, an information-efficient text entry system. It had been developed by one of David's students as a practical demonstration of arithmetic encoding after David had realised that presenting a visualisation of an effective compression algorithm allowed you to compose text without having to enter as much information into the system. At first this was merely a neat toy, but it soon became clear that the benefits of Dasher had a great deal of overlap with good accessibility software. It required much less precision of input, it made it easy to correct mistakes (you merely had to reverse direction in order to start zooming back out of the text you had entered) and it worked with a variety of input technologies from mice to eye tracking to breathing. My job was to take this codebase and turn it into a project that would be interesting to external developers.

In the year I worked with David, we turned Dasher from a research project into a well-integrated component of Gnome, improved its support for Windows, accepted code from an external contributor who ported it to OS X (using an OpenGL canvas!) and wrote ports for a range of handheld devices. We added code that allowed Dasher to directly control the UI of other applications, making it possible for people to drive word processors without having to leave Dasher. We taught Dasher to speak. We strove to avoid the mistakes present in so many other pieces of accessibility software, such as configuration that could only be managed by an (expensive!) external consultant. And we visited Dasher users and learned how they used it and what more they needed, then went back home and did what we could to provide that.

Working on Dasher was an incredible opportunity. I was involved in the development of exciting code. I spoke on it at multiple conferences. I became part of the Gnome community. I visited the USA for the first time. I entered people's homes and taught them how to use Dasher and experienced their joy as they realised that they could now communicate up to an order of magnitude more quickly. I wrote software that had a meaningful impact on the lives of other people.

Working with David was certainly not easy. Our weekly design meetings were, charitably, intense. He had an astonishing number of ideas, and my job was to figure out how to implement them while (a) not making the application overly complicated and (b) convincing David that it still did everything he wanted. One memorable meeting involved me gradually arguing him down from wanting five new checkboxes to agreeing that there were only two combinations that actually made sense (and hence a single checkbox) - and then admitting that this was broadly equivalent to an existing UI element, so we could just change the behaviour of that slightly without adding anything. I took the opportunity to delete an additional menu item in the process.

I was already aware of the importance of free software in terms of developers, but working with David made it clear to me how important it was to users as well. A community formed around Dasher, helping us improve it and allowing us to develop support for new use cases that made the difference between someone being able to type at two words per minute and being able to manage twenty. David saw that this collaborative development would be vital to creating something bigger than his original ideas, and it succeeded in ways he couldn't have hoped for.

I spent a year in the group and then went back to biology. David went on to channel his strong feelings about social responsibility into issues such as sustainable energy, writing a freely available book on the topic. He served as chief adviser to the UK Department of Energy and Climate Change for five years. And earlier this year he was awarded a knighthood for his services to scientific outreach.

David died yesterday. It's unlikely that I'll ever come close to what he accomplished, but he provided me with much of the inspiration to try to do so anyway. The world is already a less fascinating place without him.

comment count unavailable comments

April 13, 2016

Skylake's power management under Linux is dreadful and you shouldn't buy one until it's fixed
(Edit to add: this issue is restricted to the mobile SKUs. Desktop parts have very different power management behaviour)

Linux 4.5 seems to have got Intel's Skylake platform (ie, 6th-generation Core CPUs) to the point where graphics work pretty reliably, which is great progress (4.4 tended to lose all my windows every so often, especially over suspend/resume). I'm even running Wayland happily. Unfortunately one of the reasons I have a laptop is that I want to be able to do things like use it on battery, and power consumption's an important part of that. Skylake continues the trend from Haswell of moving to an SoC-type model where clock and power domains are shared between components that were previously entirely independent, and so you can't enter deep power saving states unless multiple components all have the correct power management configuration. On Haswell/Broadwell this manifested in the form of Serial ATA link power management being involved in preventing the package from going into deep power saving states - setting that up correctly resulted in a reduction in full-system power consumption of about 40%[1].

I've now got a Skylake platform with a nice shiny NVMe device, so Serial ATA policy isn't relevant (the platform doesn't even expose a SATA controller). The deepest power saving state I can get into is PC3, despite Skylake supporting PC8 - so I'm probably consuming about 40% more power than I should be. And nobody seems to know what needs to be done to fix this. I've found no public documentation on the power management dependencies on Skylake. Turning on everything in Powertop doesn't improve anything. My battery life is pretty poor and the system is pretty warm.

The best thing about this is the following statement from page 64 of the 6th Generation Intel ® Processor Datasheet for U-Platforms:

Caution: Long term reliability cannot be assured unless all the Low-Power Idle States are enabled.

which is pretty concerning. Without support for states deeper than PC3, Linux is running in a configuration that Intel imply may trigger premature failure. That's obviously not good. Until this situation is improved, you probably shouldn't buy any Skylake systems if you're planning on running Linux.

[1] These patches never went upstream. Someone reported that they resulted in their SSD throwing errors and I couldn't find anybody with deeper levels of SATA experience who was interested in working on the problem. Intel's AHCI drivers for Windows do the right thing, but I couldn't find anybody at Intel who could get any information from their Windows driver team.

comment count unavailable comments

April 11, 2016

Making it easier to deploy TPMTOTP on non-EFI systems
I've been working on TPMTOTP a little this weekend. I merged a pull request that adds command-line argument handling, which includes the ability to choose the set of PCRs you want to seal to without rebuilding the tools, and also lets you print the base32 encoding of the secret rather than the qr code so you can import it into a wider range of devices. More importantly it also adds support for setting the expected PCR values on the command line rather than reading them out of the TPM, so you can now re-seal the secret against new values before rebooting.

I also wrote some new code myself. TPMTOTP is designed to be usable in the initramfs, allowing you to validate system state before typing in your passphrase. Unfortunately the initramfs itself is one of the things that's measured. So, you end up with something of a chicken and egg problem - TPMTOTP needs access to the secret, and the obvious thing to do is to put the secret in the initramfs. But the secret is sealed against the hash of the initramfs, and so you can't generate the secret until after the initramfs. Modify the initramfs to insert the secret and you change the hash, so the secret is no longer released. Boo.

On EFI systems you can handle this by sticking the secret in an EFI variable (there's some special-casing in the code to deal with the additional metadata on the front of things you read out of efivarfs). But that's not terribly useful if you're not on an EFI system. Thankfully, there's a way around this. TPMs have a small quantity of nvram built into them, so we can stick the secret there. If you pass the -n argument to sealdata, that'll happen. The unseal apps will attempt to pull the secret out of nvram before falling back to looking for a file, so things should just magically work.

I think it's pretty feature complete now, other than TPM2 support? That's on my list.

comment count unavailable comments

April 07, 2016

Why libinput doesn't have a lot of config options

Most days, at least one of the bugs I deal with requests something along the lines of "just add $FOO as a config option". In this post, I'll explain why this is usually a bad solution. First, read http://www.islinuxaboutchoice.com/ and keep those arguments in mind. Generally, there are two groups of configuration options - hardware options and user options. Hardware options are those that deal with specific quirks needed on some hardware, but not on other hardware. User options are those that deal with user preferences such as tapping or two-finger vs. edge scrolling.

In the old synaptics driver, we added options whenever something new came up and we tried to make those options generic. This was a big mistake. The driver now has over 70 configuration options resulting in a test matrix with a googolplex of combinations. In other words, it's completely untestable. To make a device work users often have to find the right combination of options from somewhere, write out a root-owned config file and then hope this works. Why do we still think this is acceptable? Even worse: some options are very specific to hardware but still spread in user forum examples like an STD during spring break.

In libinput, we're having none of that. When hardware doesn't work we expect a user to file a bug, we get it fixed upstream for the specific model and thus automatically fix it for all users of that device. We're leaning heavily on udev's hwdb which we have extended to correct devices when the firmware announces wrong information. This has the advantage that there is only one authoritative source of quirks a device needs to work. And we can update this as time goes by without having to worry about stale configuration options. One good example here is the custom acceleration profile that Lenovo X230 touchpads have in libinput. All in all, there is little pushback for the lack of hardware-specific configuration options and most users are fine with it once they accept the initial waiting period to get the patch into their distribution.

User-specific options are more contentious. In our opinion, some features should be configurable and others should not. Where to draw that line is of course quite undefined. For example, tapping on or off was one of the first configuration options available and that was never a cause for arguments either way (except whether the default should be on or off). Other options are more contentious. Clickpad software buttons are always on the bottom edge and their size is hardcoded (synaptics allowed almost free positioning of those buttons). Other features such as changing a two-finger tap to some other button event is not supported at all in libinput. This effectively comes down to cost. You see, whenever you write "it's just 5 lines of code to make this an option", what I think is "once the patch is reviewed and applied, I'll spend two days to write test cases and documentation. I'll need to handle any bug reports related to this, and I'm expected to make sure this option works indefinitely. Any addition of another feature may conflict with this option, so I need to make sure the right combination is possible and test cases are written." So your work ends after writing a 5 line patch, my work as maintainer merely starts. And unless it pays off long-term, the effort is not worth it. Some features make that cut, others don't if they are too much of a niche feature.

All this is of course nothing new and every software project needs to make these decisions. Input isn't even a special case here, it pales in comparison with e.g. the decisions UI designers need to make. However, in FOSS we have a tendency to think that because something is possible, it should be done. Legally, you have freedom to do almost anything with the software, so you can maintain a local fork of libinput with that extra feature applied. If that isn't acceptable, why would it be acceptable to merge the patch and expect others to shoulder the costs?

April 05, 2016

libinput now has a touchpad software middle button

I just pushed a patch to libinput master to enable a middle button on the clickpad software buttons. Until now, our stance was that clickpads only get a left and right software button, split at the 50% mark. The reasoning is simple: touchpads only have markings for left and right buttons (if any!) and the middle button's extents are not easily discoverable if there is no visual or haptic feedback. A middle button event could however be triggered through middle button emulation, i.e. by clicking the touchpad with a finger on the left and right software button area (see the instructions here).

This is nice in theory but, as usual, reality gets in the way. Most interactions with the middle button are quick and short-lived, i.e. clicking the button once to paste. This interaction is what many touchpads are spectacularly bad at. For middle button emulation to be handled correctly, both fingers must be registered before the physical button press. The scanout rate on a touchpad is often too low and on touchpads with extremely light resistance like the T440 it's common to register the physical click before we know that there's a second finger on the touchpad. But even on a T450 and an X220 with much higher clickpad resistance I barely managed to get above 7 out of 10 correctly registered middle button events. That is simply not good enough.

So the patch I just pushed out to master enables a middle software button between the left and the right button. The exact width of the button scales with the touchpad but it's usually around 20-25mm and it's centered on the touchpad so despite the lack of visual or haptic guides it should be reliable to hit. The new behaviour is hard-coded and for now middle button emulation continues to work on touchpads. In the future, I expect I will remove middle button emulation on touchpads or at least disable it by default.

The middle button will be available in libinput 1.3.

There's more than one way to exploit the commons
There's a piece of software called XScreenSaver. It attempts to fill two somewhat disparate roles:
  • Provide a functioning screen lock on systems using the X11 windowing system, a job made incredibly difficult due to a variety of design misfeatures in said windowing system[1]
  • Provide cute graphical output while the screen is locked
XScreenSaver does an excellent job of the second of these[2] and is pretty good at the first, which is to say that it only suffers from a disastrous security flaw once very few years and as such is certainly not appreciably worse than any other piece of software.

Debian ships an operating system that prides itself on stability. The Debian definition of stability is a very specific one - rather than referring to how often the software crashes or misbehaves, it refers to how often the software changes behaviour. Debian is very reluctant to upgrade software that is part of a stable release, to the extent that developers will attempt to backport individual security fixes to the version they shipped rather than upgrading to a release that contains all those security fixes but also adds a new feature. The argument here is that the new release may also introduce new bugs, and Debian's users desire stability (in the "things don't change" sense) more than new features. Backporting security fixes keeps them safe without compromising the reason they're running Debian in the first place.

This all makes plenty of sense at a theoretical level, but reality is sometimes less convenient. The first problem is that security bugs are typically also, well, bugs. They may make your software crash or misbehave in annoying but apparently harmless ways. And when you fix that bug you've also fixed a security bug, but the ability to determine whether a bug is a security bug or not is one that involves deep magic and a fanatical devotion to the cause so given the choice between maybe asking for a CVE and dealing with embargoes and all that crap when perhaps you've actually only fixed a bug that makes the letter "E" appear in places it shouldn't and not one that allows the complete destruction of your intergalactic invasion fleet means people will tend to err on the side of "Eh fuckit" and go drinking instead. So new versions of software will often fix security vulnerabilities without there being any indication that they do so[3], and running old versions probably means you have a bunch of security issues that nobody will ever do anything about.

But that's broadly a technical problem and one we can apply various metrics to, and if somebody wanted to spend enough time performing careful analysis of software we could have actual numbers to figure out whether the better security approach is to upgrade or to backport fixes. Conversations become boring once we introduce too many numbers, so let's ignore that problem and go onto the second, which is far more handwavy and social and so significantly more interesting.

The second problem is that upstream developers remain associated with the software shipped by Debian. Even though Debian includes a tool for reporting bugs against packages included in Debian, some users will ignore that and go straight to the upstream developers. Those upstream developers then have to spend at least 15 or so seconds telling the user that the bug they're seeing has been fixed for some time, and then figure out how to explain that no sorry they can't make Debian include a fixed version because that's not how things work. Worst case, the stable release of Debian ends up including a bug that makes software just basically not work at all and everybody who uses it assumes that the upstream author is brutally incompetent, and they end up quitting the software industry and I don't know running a nightclub or something.

From the Debian side of things, the straightforward solution is to make it more obvious that users should file bugs with Debian and not bother the upstream authors. This doesn't solve the problem of damaged reputation, and nor does it entirely solve the problem of users contacting upstream developers. If a bug is filed with Debian and doesn't get fixed in a timely manner, it's hardly surprising that users will end up going upstream. The Debian bugs list for XScreenSaver does not make terribly attractive reading.

So, coming back to the title for this entry. The most obvious failure of the commons is where a basically malicious actor consumes while giving nothing back, but if an actor with good intentions ends up consuming more than they contribute that may still be a problem. An upstream author releases a piece of software under a free license. Debian distributes this to users. Debian's policies result in the upstream author having to do more work. What does the upstream author get out of this exchange? In an ideal world, plenty. The author's software is made available to more people. A larger set of developers is willing to work on making improvements to the software. In a less ideal world, rather less. The author has to deal with bug mail about already fixed bugs. The author's reputation may be harmed by user exposure to said fixed bugs. The author may get less in the way of useful bug fixes or features because people are running old versions rather than fixing new ones. If the balance tips towards the latter, the author's decision to release their software under a free license has made their life more difficult.

Most discussions about Debian's policies entirely ignore the latter scenario, focusing more on the fact that the author chose to release their software under a free license to begin with. If the author is unwilling to handle the consequences of that, goes the argument, why did they do it in the first place? The unfortunate logical conclusion to that argument is that the author realises that they made a huge mistake and never does so again, and woo uh oops.

The irony here is that one of Debian's foundational documents, the Debian Free Software Guidelines, makes allowances for this. Section 4 allows for distribution of software in Debian even if the author insists that modified versions[4] are renamed. This allows for an author to make a choice - allow themselves to be associated with the Debian version of their work and increase (a) their userbase and (b) their support load, or try to distinguish what Debian ship from their identity. But that document was ratified in 1997 and people haven't really spent much time since then thinking about why it says what it does, and so this tradeoff is rarely considered.

Free software doesn't benefit from distributions antagonising their upstreams, even if said upstream is a cranky nightclub owner. Debian's users are Debian's highest priority, but those users are going to suffer if developers decide that not using free licenses improves their quality of life. Kneejerk reactions around specific instances aren't helpful, but now is probably a good time to start thinking about what value Debian bring to its upstream authors and how that can be increased. Failing to do so doesn't serve users, Debian itself or the free software community as a whole.

[1] The X server has no fundamental concept of a screen lock. This is implemented by an application asking that the X server send all keyboard and mouse input to it rather than to any other application, and then that application creating a window that fills the screen. Due to some hilarious design decisions, opening a pop-up menu in an application prevents any other application from being able to grab input and so it is impossible for the screensaver to activate if you open a menu and then walk away from your computer. This is merely the most obvious problem - there are others that are more subtle and more infuriating. The only fix in this case is to nuke the site from orbit.

[2] There's screenshots here. My favourites are the one that emulate the electrical characteristics of an old CRT in order to present a more realistic depiction of the output of an Apple 2 and the one that includes a complete 6502 emulator.

[3] And obviously new versions of software will often also introduce new security vulnerabilities without there being any indication that they do so, because who would ever put that in their changelog. But the less ethically challenged members of the security community are more likely to be looking at new versions of software than ones released three years ago, so you're probably still tending towards winning overall

[4] There's a perfectly reasonable argument that all packages distributed by Debian are modified in some way

comment count unavailable comments