May 25, 2016

Blog backlog, Post 3, DisplayLink-based USB3 graphics support for Fedora
Last year, after DisplayLink released the first version of the supporting tools for their USB3 chipsets, I tried it out on my Dell S2340T.

As I wanted a clean way to test new versions, I took Eric Nothen's RPMs, and updated them along with newer versions, automating the creation of 32- and 64-bit x86 versions.

The RPM contains 3 parts, evdi, a GPLv2 kernel module that creates a virtual display, the LGPL library to access it, and a proprietary service which comes with "firmware" files.

Eric's initial RPMs used the precompiled libevdi.so, and proprietary bits, compiling only the kernel module with dkms when needed. I changed this, compiling the library from the upstream repository, using the minimal amount of pre-compiled binaries.

This package supports quite a few OEM devices, but does not work correctly with Wayland, so you'll need to disable Wayland support in /etc/gdm/custom.conf if you want it to work at the login screen, and without having to restart the displaylink.service systemd service after logging in.


 Plugged in via DisplayPort and USB (but I can only see one at a time)


The source for the RPM are on GitHub. Simply clone and run make in the repository to create 32-bit and 64-bit RPMs. The proprietary parts are redistributable, so if somebody wants to host and maintain those RPMs, I'd be glad to pass this on.

May 24, 2016

LVFS Technical White Paper

I spent a good chunk of today writing a technical whitepaper titled Introducing the Linux Vendor Firmware Service — I’d really appreciate any comments, either from people who have seen all progress from the start or who don’t know anything about it at all.

Typos, or more general comments are all welcome and once I’ve got something a bit more polished I’ll be sending this to some important suits in a few well known companies. Thanks for any help!

May 23, 2016

External Plugins in GNOME Software (6)

This is my last post about the gnome-software plugin structure. If you want more, join the mailing list and ask a question. If you’re not sure how something works then I’ve done a poor job on the docs, and I’m happy to explain as much as required.

GNOME Software used to provide a per-process plugin cache, automatically de-duplicating applications and trying to be smarter than the plugins themselves. This involved merging applications created by different plugins and really didn’t work very well. For 3.20 and later we moved to a per-plugin cache which allows the plugin to control getting and adding applications to the cache and invalidating it when it made sense. This seems to work a lot better and is an order of magnitude less complicated. Plugins can trivially be ported to using the cache using something like this:

 
   /* create new object */
   id = gs_plugin_flatpak_build_id (inst, xref);
-  app = gs_app_new (id);
+  app = gs_plugin_cache_lookup (plugin, id);
+  if (app == NULL) {
+     app = gs_app_new (id);
+     gs_plugin_cache_add (plugin, id, app);
+  }

Using the cache has two main benefits for plugins. The first is that we avoid creating duplicate GsApp objects for the same logical thing. This means we can query the installed list, start installing an application, then query it again before the install has finished. The GsApp returned from the second add_installed() request will be the same GObject, and thus all the signals connecting up to the UI will still be correct. This means we don’t have to care about migrating the UI widgets as the object changes and things like progress bars just magically work.

The other benefit is more obvious. If we know the application state from a previous request we don’t have to query a daemon or do another blocking library call to get it. This does of course imply that the plugin is properly invalidating the cache using gs_plugin_cache_invalidate() which it should do whenever a change is detected. Whether a plugin uses the cache for this reason is up to the plugin, but if it does it is up to the plugin to make sure the cache doesn’t get out of sync.

And one last thing: If you’re thinking of building an out-of-tree plugin for production use ask yourself if it actually belongs upstream. Upstream plugins get ported as the API evolves, and I’m already happily carrying Ubuntu and Fedora-specific plugins that either self-disable at runtime or are protected using --enable-foo configure argument.

External Plugins in GNOME Software (5)

This is my penultimate post about the gnome-software plugin structure. If you’ve followed everything so far, well done.

There’s a lot of flexibility in the gnome-software plugin structure; a plugin can add custom applications and handle things like search and icon loading in a totally custom way. Most of the time you don’t care about how search is implemented or how icons are going to be loaded, and you can re-use a lot of the existing code in the appstream plugin. To do this you just save an AppStream-format XML file in either /usr/share/app-info/xmls/, /var/cache/app-info/xmls/ or ~/.local/share/app-info/xmls/. GNOME Software will immediately notice any new files, or changes to existing files as it has set up the various inotify watches.

This allows plugins to care a lot less about how applications are going to be shown. For example, the steam plugin downloads and parses the descriptions from a remote service during gs_plugin_refresh(), and also finds the best icon types and downloads them too. Then it exports the data to an AppStream XML file, saving it to your home directory. This allows all the applications to be easily created (and then refined) using something as simple as gs_app_new("steam:foo.desktop"). All the search tokenisation and matching is done automatically, so it makes the plugin much simpler and faster.

The only extra step the steam plugin needs to do is implement the gs_plugin_adopt_app() function. This is called when an application does not have a management plugin set, and allows the plugin to claim the application for itself so it can handle installation, removal and updating. In the case of steam it could check the ID has a prefix of steam: or could check some other plugin-specific metadata using gs_app_get_metadata_item().

Another good example is the fwupd that wants to handle any firmware we’ve discovered in the AppStream XML. This might be shipped by the vendor in a package using Satellite, or downloaded from the LVFS. It wouldn’t be kind to set a management plugin explicitly in case XFCE or KDE want to handle this in a different way. This adoption function in this case is trivial:

void
gs_plugin_adopt_app (GsPlugin *plugin, GsApp *app)
{
  if (gs_app_get_kind (app) == AS_APP_KIND_FIRMWARE)
    gs_app_set_management_plugin (app, "fwupd");
}

The next (and last!) blog post I’m going to write is about the per-plugin cache that’s available to plugins to help speed up some operations. In related news, we now have a mailing list, so if you’re interested in this stuff I’d encourage you to join and ask questions there. I also released gnome-software 3.21.2 this morning, so if you want to try all this plugin stuff yourself your distro if probably going to be updating packages soon.

May 22, 2016

External Plugins in GNOME Software (4)

After my last post, I wanted to talk more about the refine functionality in gnome-software. As previous examples have shown it’s very easy to add a new application to the search results, updates list or installed list. Some plugins don’t want to add more applications, but want to modify existing applications to add more information depending on what is required by the UI code. The reason we don’t just add everything at once is that for search-as-you-type to work effectively we need to return results in less than about 50ms and querying some data can take a long time. For example, it might take a few hundred ms to work out the download size for an application when a plugin has to also look at what dependencies are already installed. We only need this information once the user has clicked the search results and when the user is in the details panel, so we can save a ton of time not working out properties that are not useful.

Lets looks at another example.

gboolean
gs_plugin_refine_app (GsPlugin *plugin,
                      GsApp *app,
                      GsPluginRefineFlags flags,
                      GCancellable *cancellable,
                      GError **error)
{
  /* not required */
  if ((flags & GS_PLUGIN_REFINE_FLAGS_REQUIRE_LICENSE) == 0)
    return TRUE;

  /* already set */
  if (gs_app_get_license (app) != NULL)
    return TRUE;

  /* FIXME, not just hardcoded! */
  if (g_strcmp0 (gs_app_get_id (app, "chiron.desktop") == 0))
    gs_app_set_license (app, "GPL-2.0 and LGPL-2.0+");

  return TRUE;
}

This is a simple example, but shows what a plugin needs to do. It first checks if the action is required, in this case GS_PLUGIN_REFINE_FLAGS_REQUIRE_LICENSE. This request is more common than you might expect as even the search results shows a non-free label if the license is unspecified or non-free. It then checks if the license is already set, returning with success if so. If not, it checks the application ID and hardcodes a license; in the real world this would be querying a database or parsing an additional config file. As mentioned before, if the license value is freely available without any extra work then it’s best just to set this at the same time as when adding the app with gs_app_list_add(). Think of refine as adding things that cost time to calculate only when really required.

The UI in gnome-software is quite forgiving for missing data, hiding sections or labels as required. Some things are required however, and forgetting to assign an icon or short description will get the application vetoed so that it’s not displayed at all. Helpfully, running gnome-software --verbose on the command line will tell you why an application isn’t shown along with any extra data.

As a last point, a few people have worries that these blogs are perhaps asking for trouble; external plugins have a chequered history in a number of projects and I’m sure gnome-software would be in an even worse position given that the core maintainer team is still so small. Being honest, if we break your external plugin due to an API change in the core you probably should have pushed your changes upstream sooner. There’s a reason you have to build with -DI_KNOW_THE_GNOME_SOFTWARE_API_IS_SUBJECT_TO_CHANGE

External Plugins in GNOME Software (3)

Lots of nice feedback from my last post, so here’s some new stuff. Up now is downloading new metadata and updates in plugins.

The plugin loader supports a gs_plugin_refresh() vfunc that is called in various situations. To ensure plugins have the minimum required metadata on disk it is called at startup, but with a cache age of infinite. This basically means the plugin must just ensure that any data exists no matter what the age.

Usually once per day, we’ll call gs_plugin_refresh() but with the correct cache age set (typically a little over 24 hours) which allows the plugin to download new metadata or payload files from remote servers. The gs_utils_get_file_age() utility helper can help you work out the cache age of file, or the plugin can handle it some other way.

For the Flatpak plugin we just make sure the AppStream metadata exists at startup, which allows us to show search results in the UI. If the metadata did not exist (e.g. if the user had added a remote using the commandline without gnome-software running) then we would show a loading screen with a progress bar before showing the main UI. On fast connections we should only show that for a couple of seconds, but it’s a good idea to try any avoid that if at all possible in the plugin.
Once per day the gs_plugin_refresh() method is called again, but this time with GS_PLUGIN_REFRESH_FLAGS_PAYLOAD set. This is where the Flatpak plugin would download any ostree trees (but not doing the deloy step) so that the applications can be updated live in the details panel without having to wait for the download to complete. In a similar way, the fwupd plugin downloads the tiny LVFS metadata with GS_PLUGIN_REFRESH_FLAGS_METADATA and then downloads the large firmware files themselves only when the GS_PLUGIN_REFRESH_FLAGS_PAYLOAD flag is set.

If the @app parameter is set for gs_plugin_download_file() then the progress of the download is automatically proxied to the UI elements associated with the application, for instance the install button would show a progress bar in the various different places in the UI. For a refresh there’s no relevant GsApp to use, so we’ll leave it NULL which means something is happening globally which the UI can handle how it wants, for instance showing a loading page at startup.

gboolean
gs_plugin_refresh (GsPlugin *plugin,
                   guint cache_age,
                   GsPluginRefreshFlags flags,
                   GCancellable *cancellable,
                   GError **error)
{
  const gchar *metadata_fn = "/var/cache/example/metadata.xml";
  const gchar *metadata_url = "http://www.example.com/new.xml";

  /* this is called at startup and once per day */
  if (flags & GS_PLUGIN_REFRESH_FLAGS_METADATA) {
    g_autoptr(GFile) file = g_file_new_for_path (metadata_fn);

    /* is the metadata missing or too old */
    if (gs_utils_get_file_age (file) > cache_age) {
      if (!gs_plugin_download_file (plugin,
                                    NULL,
                                    metadata_url,
                                    metadata_fn,
                                    cancellable,
                                    error)) {
        /* it's okay to fail here */
        return FALSE;
      }
      g_debug ("successfully downloaded new metadata");
    }
  }

  /* this is called when the session is idle */
  if ((flags & GS_PLUGIN_REFRESH_FLAGS_PAYLOAD) == 0) {
    // FIXME: download any required updates now
  }

  return TRUE;
}

Note, if the downloading fails it’s okay to return FALSE; the plugin loader continues to run all plugins and just logs an error to the console. We’ll be calling into gs_plugin_refresh() again in only a few hours, so there’s no need to bother the user. For actions like gs_plugin_app_install we also do the same thing, but we also save the error on the GsApp itself so that the UI is free to handle that how it wants, for instance showing a GtkDialog window for example.

May 20, 2016

External Plugins in GNOME Software (2)

After quite a lot of positive feedback from my last post I’ll write some more about custom plugins. Next up is returning custom applications into the installed list. The use case here is a proprietary software distribution method that installs custom files into your home directory, but you can use your imagination for how this could be useful.

The example here is all hardcoded, and a true plugin would have to derive the details about the GsApp, for example reading in an XML file or YAML config file somewhere. So, code:

#include <gnome-software.h>

void
gs_plugin_initialize (GsPlugin *plugin)
{
  gs_plugin_add_rule (plugin, GS_PLUGIN_RULE_RUN_BEFORE, "icons");
}

gboolean
gs_plugin_add_installed (GsPlugin *plugin,
                         GsAppList *list,
                         GCancellable *cancellable,
                         GError **error)
{
  g_autofree gchar *fn = NULL;
  g_autoptr(GsApp) app = NULL;
  g_autoptr(AsIcon) icon = NULL;

  /* check if the app exists */
  fn = g_build_filename (g_get_home_dir (), "chiron", NULL);
  if (!g_file_test (fn, G_FILE_TEST_EXISTS))
    return TRUE;

  /* the trigger exists, so create a fake app */
  app = gs_app_new ("example:chiron.desktop");
  gs_app_set_management_plugin (app, "example");
  gs_app_set_kind (app, AS_APP_KIND_DESKTOP);
  gs_app_set_state (app, AS_APP_STATE_INSTALLED);
  gs_app_set_name (app, GS_APP_QUALITY_NORMAL,
                   "Chiron");
  gs_app_set_summary (app, GS_APP_QUALITY_NORMAL,
                      "A teaching application");
  gs_app_set_description (app, GS_APP_QUALITY_NORMAL,
        "Chiron is the name of an application.\n\n"
        "It can be used to demo some of our features");

  /* these are all optional */
  gs_app_set_version (app, "1.2.3");
  gs_app_set_size_installed (app, 2 * 1024 * 1024);
  gs_app_set_size_download (app, 3 * 1024 * 1024);
  gs_app_set_origin_ui (app, "The example plugin");
  gs_app_add_category (app, "Game");
  gs_app_add_category (app, "ActionGame");
  gs_app_add_kudo (app, GS_APP_KUDO_INSTALLS_USER_DOCS);
  gs_app_set_license (app, GS_APP_QUALITY_NORMAL,
                      "GPL-2.0+ and LGPL-2.1+");

  /* create a stock icon (loaded by the 'icons' plugin) */
  icon = as_icon_new ();
  as_icon_set_kind (icon, AS_ICON_KIND_STOCK);
  as_icon_set_name (icon, "input-gaming");
  gs_app_set_icon (app, icon);

  /* return new app */
  gs_app_list_add (list, app);

  return TRUE;
}

This shows a lot of the plugin architecture in action. Some notable points:

  • The application ID (example:chiron.desktop) has a prefix of example which means we can co-exist with any package or flatpak version of the Chiron application, not setting the prefix would make the UI confused if more than one chiron.desktop got added.
  • Setting the management plugin means we can check for this string when working out if we can handle the install or remove action.
  • Most applications want a kind of AS_APP_KIND_DESKTOP to be visible as an application.
  • The origin is where the application originated from — usually this will be something like Fedora Updates.
  • The GS_APP_KUDO_INSTALLS_USER_DOCS means we get the blue “Documentation” award in the details page; there are many kudos to award to deserving apps.
  • Setting the license means we don’t get the non-free warning — removing the 3rd party warning can be done using AS_APP_QUIRK_PROVENANCE
  • The icons plugin will take the stock icon and convert it to a pixbuf of the correct size.

To show this fake application just compile and install the plugin, touch ~/chiron and then restart gnome-software.

Screenshot from 2016-05-20 21-22-38

By filling in the optional details (which can also be filled in using gs_plugin_refine_app() (to be covered in a future blog post) you can also make the details page a much more exciting place. Adding a set of screenshots is left as an exercise to the reader.

Screenshot from 2016-05-20 21-22-46

For anyone interested, I’m also slowly writing up these blog posts into proper docbook and uploading them with the gtk-doc files here. I think this documentation would have been really useful for the Endless and Ubuntu people a few weeks ago, so if anyone sees any typos or missing details please let me know.

May 19, 2016

Your project's RCS history affects ease of contribution (or: don't squash PRs)
Github recently introduced the option to squash commits on merge, and even before then several projects requested that contributors squash their commits after review but before merge. This is a terrible idea that makes it more difficult for people to contribute to projects.

I'm spending today working on reworking some code to integrate with a new feature that was just integrated into Kubernetes. The PR in question was absolutely fine, but just before it was merged the entire commit history was squashed down to a single commit at the request of the reviewer. This single commit contains type declarations, the functionality itself, the integration of that functionality into the scheduler, the client code and a large pile of autogenerated code.

I've got some familiarity with Kubernetes, but even then this commit is difficult for me to read. It doesn't tell a story. I can't see its growth. Looking at a single hunk of this diff doesn't tell me whether it's infrastructural or part of the integration. Given time I can (and have) figured it out, but it's an unnecessary waste of effort that could have gone towards something else. For someone who's less used to working on large projects, it'd be even worse. I'm paid to deal with this. For someone who isn't, the probability that they'll give up and do something else entirely is even greater.

I don't want to pick on Kubernetes here - the fact that this Github feature exists makes it clear that a lot of people feel that this kind of merge is a good idea. And there are certainly cases where squashing commits makes sense. Commits that add broken code and which are immediately followed by a series of "Make this work" commits also impair readability and distract from the narrative that your RCS history should present, and Github present this feature as a way to get rid of them. But that ends up being a false dichotomy. A history that looks like "Commit", "Revert Commit", "Revert Revert Commit", "Fix broken revert", "Revert fix broken revert" is a bad history, as is a history that looks like "Add 20,000 line feature A", "Add 20,000 line feature B".

When you're crafting commits for merge, think about your commit history as a textbook. Start with the building blocks of your feature and make them one commit. Build your functionality on top of them in another. Tie that functionality into the core project and make another commit. Add client support. Add docs. Include your tests. Allow someone to follow the growth of your feature over time, with each commit being a chapter of that story. And never, ever, put autogenerated code in the same commit as an actual functional change.

People can't contribute to your project unless they can understand your code. Writing clear, well commented code is a big part of that. But so is showing the evolution of your features in an understandable way. Make sure your RCS history shows that, otherwise people will go and find another project that doesn't make them feel frustrated.

(Edit to add: Sarah Sharp wrote on the same topic a couple of years ago)

comment count unavailable comments
External plugins in GNOME Software

I’ve just pushed a set of patches to gnome-software master that allow people to compile out-of-tree gnome-software plugins.

In general, building things out-of-tree isn’t something that I think is a very good idea; the API and ABI inside gnome-software is still changing and there’s a huge benefit to getting plugins upstream where they can undergo review and be ported as the API adapts. I’m also super keen to provide configurability in GSettings for doing obviously-useful things, the sort of thing Fleet Commander can set for groups of users. However, now we’re shipping gnome-software in enterprise-class distros we might want to allow customers to ship thier own plugins to make various business-specific changes that don’t make sense upstream. This might involve querying a custom LDAP server and changing the suggested apps to reflect what groups the user is in, or might involve showing a whole new class of applications that does not conform to the Linux-specific “application is a desktop-file” paradigm. This is where a plugin makes sense, and something I’d like to support in future updates to RHEL 7.

At this point it probably makes sense to talk a bit about how the architecture of gnome-software works. At its heart it’s just a big plugin loader that has some GTK UI that gets created for various result types. The idea is we have lots of small plugins that each do one thing and then pass the result onto the other plugins. These are ordered by dependencies against each other at runtime and each one can do things like editing an existing application or adding a new application to the result set. This is how we can add support for things like firmware updating, Steam, GNOME Shell web-apps and flatpak bundles without making big changes all over the source tree.

There are broadly 3 types of plugin methods:

  • Actions: Do something on a specific GsApp; install gimp.desktop
  • Refine: Get details about a specific GsApp; is firefox.desktop installed? or get reviews for inkscape.desktop
  • Adopt: Can this plugin handle this GsApp; can fwupd handle com.hughski.ColorHug2.firmware

You only need to define the vfuncs that the plugin needs, and the name is taken automatically from the suffix of the .so file. So, lets look at a sample plugin one chunk at a time, taking it nice and slow. First the copyright and licence (it only has to be GPLv2+ if it’s headed upstream):

/*
 * Copyright (C) 2016 Richard Hughes 
 * Licensed under the GNU General Public License Version 2
 */

Then, the magic header that sucks in everything that’s exported:

#include <gnome-software.h>

Then we have to define when our plugin is run in reference to other plugins, as we’re such a simple plugin we’re relying on another plugin to run after us to actually make the GsApp “complete”, i.e. adding icons and long descriptions:

void
gs_plugin_initialize (GsPlugin *plugin)
{
  gs_plugin_add_rule (plugin, GS_PLUGIN_RULE_RUN_BEFORE, "appstream");
}

Then we can start to do something useful. In this example I want to show GIMP as a result (from any provider, e.g. flatpak or a distro package) when the user searches exactly for fotoshop. There is no prefixing or stemming being done for simplicity.

gboolean
gs_plugin_add_search (GsPlugin *plugin,
                      gchar **values,
                      GsAppList *list,
                      GCancellable *cancellable,
                      GError **error)
{
  guint i;
  for (i = 0; values[i] != NULL; i++) {
    if (g_strcmp0 (values[i], "fotoshop") == 0) {
      g_autoptr(GsApp) app = gs_app_new ("gimp.desktop");
      gs_app_add_quirk (app, AS_APP_QUIRK_MATCH_ANY_PREFIX);
      gs_app_list_add (list, app);
    }
  }
  return TRUE;
}

We can then easily build and install the plugin using:

gcc -shared -o libgs_plugin_example.so gs-plugin-example.c -fPIC \
 `pkg-config --libs --cflags gnome-software` \
 -DI_KNOW_THE_GNOME_SOFTWARE_API_IS_SUBJECT_TO_CHANGE &&
 sudo cp libgs_plugin_example.so `pkg-config gnome-software --variable=plugindir`

Screenshot from 2016-05-19 10-39-53

I’m going to be cleaning up the exported API and adding some more developer documentation before I release the next tarball, but if this is useful to you please let me know and I’ll do some more blog posts explaining more how the internal architecture of gnome-software works, and how you can do different things with plugins.

May 16, 2016

NetworkManager and WiFi Scans

Recently Dave Täht wrote a blog post investigating latency and WiFi scanning and came across NetworkManager’s periodic scan behavior.  When a WiFi device scans it obviously must change from its current radio channel to other channels and wait for a short amount of time listening for beacons from access points.  That means it’s not passing your traffic.

With a bad driver it can sometimes take 20+ seconds and all your traffic gets dropped on the floor.

With a good driver scanning takes only a few seconds and the driver breaks the scan into chunks, returning to the associated access point’s channel periodically to handle pending traffic.  Even with a good driver, latency-critical applications like VOIP or gaming will clearly suffer while the WiFi device is listening on another channel.

So why does NetworkManager periodically scan for WiFi access points?

Roaming

Whenever your WiFi network has multiple access points with the same SSID (or a dual-band AP with a single SSID) you need roaming to maintain optimal connectivity and speed.  Jumping to a better AP requires that the device know what access points are available, which means doing a periodic scan like NetworkManager does every 2 minutes.  Without periodic scans, the driver must scan at precisely the worst moment: when the signal quality is bad, and data rates are low, and the risk of disconnecting is higher.

Enterprise WiFi setups make the roaming problem much worse because they often have tens or hundreds of access points in the network and because they typically use high-security 802.1x authentication with EAP.  Roaming with 802.1x introduces many more steps to the roaming process, each of which can fail the roaming attempt.  Strategies like pre-authentication and periodic scanning greatly reduce roaming errors and latency.

User responsiveness and Location awareness

The second reason for periodic scanning is to maintain a list of access points around you for presentation in user interfaces and for geolocation in browsers that support it.  Up until a couple years ago, most Linux WiFi applets displayed a drop-down list of access points that you could click on at any time.  Waiting for 5 to 15 seconds for a menu to populate or ‘nmcli dev wifi list’ to return would be annoying.

But with the proliferation of WiFi (often more than 30 or 40 if you live in a flat) those lists became less and less useful, so UIs like GNOME Shell moved to a separate window for WiFi lists.  This reduces the need for a constantly up-to-date WiFi list and thus for periodic scanning.

To help support these interaction models and click-to-scan behaviors like Mac OS X or Maemo, NetworkManager long ago added a D-Bus API method to request an out-of-band WiFi scan.  While it’s pretty trivial to use this API to initiate geolocation or to refresh the WiFi list based on specific user actions, I’m not aware of any clients using it well.  GNOME Shell only requests scans when the network list is empty and plasma-nm only does so when the user clicks a button.  Instead, UIs should simply request scans periodically while the WiFi list is shown, removing the need for yet another click.

WHAT TO DO

If you don’t care about roaming, and I’m assuming David doesn’t, then NetworkManager offers a simple solution: lock your WiFi connection profile to the BSSID of your access point.  When you do this, NetworkManager understands that you do not want to roam and will disable the periodic scanning behavior.  Explicitly requested scans are still allowed.

You can also advocate that your favorite WiFi interface add support for NetworkManager’s RequestScan() API method and begin requesting periodic scans when WiFi lists are shown or when your browser uses geolocation.  When most do this, perhaps NetworkManager could be less aggressive with its own periodic scans, or perhaps remove them altogether in favor of a more general solution.

That general solution might involve disabling periodic roaming when the signal strength is extremely good and start scanning more aggressively when signal strength drops over a threshold.  But signal strength drops for many reasons like turning on a microwave, closing doors, turning on Bluetooth, or even walking to the next room, and triggering a scan then still interrupts your VOIP call or low ping headshot.  This also doesn’t help people who aren’t close to their access point, leading to the same scanning problem David talks about if you’re in the basement but not if you’re in the bedroom.

Another idea would be to disable periodic scanning when latency critical applications are active, but this requires that these applications consistently set the IPv4 TOS field or use the SO_PRIORITY socket option.  Few do so.  This also requires visibility into kernel mac80211 queue depths and would not work for proprietary or non-mac80211-based drivers.  But if all the pieces fell into place on the kernel side, NetworkManager could definitely do this while waiting for applications and drivers to catch up.

If you’ve got other ideas, feel free to propose them.

May 13, 2016

Blutella, a Bluetooth speaker receiver
Quite some time ago, I was asked for a way to use the AV amplifier (which has a fair bunch of speakers connected to it) in our living-room that didn't require turning on the TV to choose a source.

I decided to try and solve this problem myself, as an exercise rather than a cost saving measure (there are good-quality Bluetooth receivers available for between 15 and 20€).

Introducing Blutella



I found this pot of Nutella in my travels (in Europe, smaller quantities are usually in a jar that looks like a mustard glass, with straight sides) and thought it would be a perfect receptacle for a CHIP, to allow streaming via Bluetooth to the amp. I wanted to make a nice how-to for you, dear reader, but best laid plans...

First, the materials:
  • a CHIP
  • jar of Nutella, and "Burnt umber" acrylic paint
  • micro-USB to USB-A and jack 3.5mm to RCA cables
  • Some white Sugru, for a nice finish around the cables
  • bit of foam, a Stanley knife, a CD marker

That's around 10€ in parts (cables always seem to be expensive), not including our salvaged Nutella jar, and the CHIP itself (9$ + shipping).

You'll start by painting the whole of the jar, on the inside, with the acrylic paint. Allow a couple of days to dry, it'll be quite thick.

So, the plan that went awry. Turns out that the CHIP, with the cables plugged in, doesn't fit inside this 140g jar of Nutella. I also didn't make the holes exactly in the right place. The CHIP is tiny, but not small enough to rotate inside the jar without hitting the side, and the groove to screw the cap also have only one position.

Anyway, I pierced two holes in the lid for the audio jack and the USB charging cable, stuffed the CHIP inside, and forced the lid on so it clipped on the jar's groove.

I had nice photos with foam I cut to hold the CHIP in place, but the finish isn't quite up to my standards. I guess that means I can attempt this again with a bigger jar ;)

The software

After flashing the CHIP with Debian, I logged in, and launched a script which I put together to avoid either long how-tos, or errors when I tried to reproduce the setup after a firmware update and reset.

The script for setting things up is in the CHIP-bluetooth-speaker repository. There are a few bugs due to drivers, and lack of integration, but this blog is the wrong place to track them, so check out the issues list.

Apart from those driver problems, I found the integration between PulseAudio and BlueZ pretty impressive, though I wish there was a way for the speaker to reconnect to the phone I streamed from when turned on again, as Bluetooth speakers and headsets do, removing one step from playing back audio.

May 12, 2016

Convenience, security and freedom - can we pick all three?
Moxie, the lead developer of the Signal secure communication application, recently blogged on the tradeoffs between providing a supportable federated service and providing a compelling application that gains significant adoption. There's a set of perfectly reasonable arguments around that that I don't want to rehash - regardless of feelings on the benefits of federation in general, there's certainly an increase in engineering cost in providing a stable intra-server protocol that still allows for addition of new features, and the person leading a project gets to make the decision about whether that's a valid tradeoff.

One voiced complaint about Signal on Android is the fact that it depends on the Google Play Services. These are a collection of proprietary functions for integrating with Google-provided services, and Signal depends on them to provide a good out of band notification protocol to allow Signal to be notified when new messages arrive, even if the phone is otherwise in a power saving state. At the time this decision was made, there were no terribly good alternatives for Android. Even now, nobody's really demonstrated a free implementation that supports several million clients and has no negative impact on battery life, so if your aim is to write a secure messaging client that will be adopted by as many people is possible, keeping this dependency is entirely rational.

On the other hand, there are users for whom the decision not to install a Google root of trust on their phone is also entirely rational. I have no especially good reason to believe that Google will ever want to do something inappropriate with my phone or data, but it's certainly possible that they'll be compelled to do so against their will. The set of people who will ever actually face this problem is probably small, but it's probably also the set of people who benefit most from Signal in the first place.

(Even ignoring the dependency on Play Services, people may not find the official client sufficient - it's very difficult to write a single piece of software that satisfies all users, whether that be down to accessibility requirements, OS support or whatever. Slack may be great, but there's still people who choose to use Hipchat)

This shouldn't be a problem. Signal is free software and anybody is free to modify it in any way they want to fit their needs, and as long as they don't break the protocol code in the process it'll carry on working with the existing Signal servers and allow communication with people who run the official client. Unfortunately, Moxie has indicated that he is not happy with forked versions of Signal using the official servers. Since Signal doesn't support federation, that means that users of forked versions will be unable to communicate with users of the official client.

This is awkward. Signal is deservedly popular. It provides strong security without being significantly more complicated than a traditional SMS client. In my social circle there's massively more users of Signal than any other security app. If I transition to a fork of Signal, I'm no longer able to securely communicate with them unless they also install the fork. If the aim is to make secure communication ubiquitous, that's kind of a problem.

Right now the choices I have for communicating with people I know are either convenient and secure but require non-free code (Signal), convenient and free but insecure (SMS) or secure and free but horribly inconvenient (gpg). Is there really no way for us to work as a community to develop something that's all three?

comment count unavailable comments

May 11, 2016

CfP is now open

The systemd.conf 2016 Call for Participation is Now Open!

We’d like to invite presentation and workshop proposals for systemd.conf 2016!

The conference will consist of three parts:

  • One day of workshops, consisting of in-depth (2-3hr) training and learning-by-doing sessions (Sept. 28th)
  • Two days of regular talks (Sept. 29th-30th)
  • One day of hackfest (Oct. 1st)

We are now accepting submissions for the first three days: proposals for workshops, training sessions and regular talks. In particular, we are looking for sessions including, but not limited to, the following topics:

  • Use Cases: systemd in today’s and tomorrow’s devices and applications
  • systemd and containers, in the cloud and on servers
  • systemd in distributions
  • Embedded systemd and in IoT
  • systemd on the desktop
  • Networking with systemd
  • … and everything else related to systemd

Please submit your proposals by August 1st, 2016. Notification of acceptance will be sent out 1-2 weeks later.

If submitting a workshop proposal please contact the organizers for more details.

To submit a talk, please visit our CfP submission page.

For further information on systemd.conf 2016, please visit our conference web site.

May 09, 2016

Blog backlog, Post 2, xdg-app bundles

I recently worked on creating an xdg-app bundle for GNOME Videos, aka Totem, so it would be built along with other GNOME applications, every night, and made available via the GNOME xdg-app repositories.

There's some functionality that's not working yet though:
  • No support for optical discs
  • The MPRIS plugin doesn't work as we're missing dbus-python (I'm not sure that the plugin will survive anyway, it's more suited to audio players, don't worry though, it's not going to be removed until we have made changes to the sound system in GNOME)
  • No libva/VDPAU hardware acceleration (which would require plugins, and possibly device access of some sort)
However, I created a bundle that extends the freedesktop runtime, that contains gst-libav. We'll need to figure out a way to distribute it in a way that doesn't cause problems for US hosts.

As we also have a recurring problem in Fedora with rpmfusion being out of date, and I sometimes need a third-party movie player to test things out, I put together an mpv manifest, which is the only MPlayer-like with a .desktop and a GUI when launched without any command-line arguments.

Finally, I put together a RetroArch bundle for research into a future project, which uncovered the lack of joystick/joypad support in the xdg-app sandbox.

Hopefully, those few manifests will be useful to other application developers wanting to distribute their applications themselves. There are some other bundles being worked on, and that can be used as examples, linked to in the Wiki.

May 05, 2016

Blog backlog, Post 1, Emoji
Short version


dnf copr enable hadess/emoji
dnf update cairo
dnf install eosrei-emojione-fonts



Long version

A little while ago, I was reading this article, called "Emoji: how do you get from U+1F355 to 🍕?", which said, and I reluctantly quote: "[...] and I don’t know what Linux does, but it’s probably black and white and who cares [...]".

Well. I care. And you probably do as well if your pizza slice above is black and white.

So I set out to check on the status of Behdad Esfahbod (or just "Behdad" as we know him)'s patches to add colour font support to cairo, which he presented at GUADEC in Strasbourg Gothenburg. It adds support for the "bitmap in font" as Android does, and as freetype supports.

It kind of worked, and Matthias Clasen reworked the patches a few times, completing the support. This is probably not the code that will be worked on and will land in cairo, but it's a good enough base for people interested in contributing to use.

After that, we needed something to display using that feature. We ended up using the same font recommended in this article, the Emoji One font.


There's still plenty to be done to support emojis, even after the cairo support is merged. We'd need a way to input emojis (maybe Lalo Martins is listening), and support in a lot of toolkits other than GNOME (Firefox only supports the SVG-in-OTF format, WebKit, Chrome, LibreOffice don't seem to know about colour fonts either).

You can find more information about design interests in GNOME around Emoji on the Wiki.

Update: Behdad's presentation was in Gothenburg, not Strasbourg. You can also see the video on YouTube.

May 04, 2016

The difference between uinput and evdev

A recurring question I encounter is the question whether uinput or evdev should be the approach do implement some feature the user cares about. This question is unfortunately wrongly framed as uinput and evdev have no real overlap and work independent of each other. This post outlines what the differences are. Note that "evdev" here refers to the kernel API, not to the X.Org evdev driver.

First, the easy flowchart: do you have to create a new virtual device that has a set of specific capabilities? Use uinput. Do you have to read and handle events from an existing device? Use evdev. Do you have to create a device and read events from that device? You (probably) need two processes, one doing the uinput bit, one doing the evdev bit.

Ok, let's talk about the difference between evdev and uinput. evdev is the default input API that all kernel input device nodes provide. Each device provides one or more /dev/input/eventN nodes that a process can interact with. This usually means checking a few capability bits ("does this device have a left mouse button?") and reading events from the device. The events themselves are in the form of struct input_event, defined in linux/input.h and consist of a event type (relative, absolute, key, ...) and an event code specific to the type (x axis, left button, etc.). See linux/input-event-codes.h for a list or linux/input.h in older kernels.Specific to evdev is that events are serialised - framed by events of type EV_SYN and code SYN_REPORT. Anything before a SYN_REPORT should be considered one logical hardware event. For example, if you receive an x and y movement within the same SYN_REPORT frame, the device has moved diagonally.

Any event coming from the physical hardware goes into the kernel's input subsystem and is converted to an evdev event that is then available on the event node. That's pretty much it for evdev. It's a fairly simple API but it does have some quirks that are not immediately obvious so I recommend using libevdev whenever you actually need to communicate with a kernel device directly.

uinput is something completely different. uinput is an kernel device driver that provides the /dev/uinput node. A process can open this node, write a bunch of custom commands to it and the kernel then creates a virtual input device. That device, like all others, presents an /dev/input/eventN node. Any event written to the /dev/uinput node will re-appear in that /dev/input/eventN node and a device created through uinput looks just pretty much like a physical device to a process. You can detect uinput-created virtual devices, but usually a process doesn't need to care so all the common userspace (libinput, Xorg) doesn't bother. The evemu tool is one of the most commonly used applications using uinput.

Now, there is one thing that may cause confusion: first, to set up a uinput device you'll have to use the familiar evdev type/code combinations (followed-by a couple of uinput-specific ioctls). Events written to uinput also use the struct input_event form, so looking at uinput code one can easily mistake it for evdev code. Nevertheless, the two serve a completely different purpose. As with evdev, I recommend using libevdev to initalise uinput devices. libevdev has a couple of uinput-related functions that make life easier.

Below is a basic illustration of how things work together. The physical devices send their events through the event nodes and libinput is a process that reads those events. evemu talks to the uinput module and creates a virtual device which then too sends events through its event node - for libinput to read.

April 30, 2016

Yet another GTK+ update

GTK+ 3.20 was released a while ago; we’re up to 3.20.3 now.  As I tried to explain in earlier posts here and here, this was a pretty active development cycle for GTK+. We landed a lot of of new stuff, and many things have changed.

I’m using the neutral term changed here for a reason. How you view changes depends a lot on your perspective. Us, who implemented the changes, are of course convinced that they are great improvements. Others who maintain GTK+ themes or applications may have a different take, since changes often imply that they have to do work to adapt.

What changed in GTK+

A big set of changes is related to the inner workings of GTK+ CSS.

The CSS box model is much better supported in widgets. This includes padding, margins, borders, shadows, and the min-width and min-height properties. Since many widgets are complex, they typically have many CSS boxes. Here is how the box tree GtkNotebook looks:

notebook-nodes

In the past (up to and including GTK+ 3.18), we used a mixture of widget class names (like GtkNotebook), style classes (like .button) and widget names (like #cancel_button) for matching styles to widgets. Now, we are using element names for each box (e.g. header, tabs and tab in the screenshot above). Style classes are still used for optional things and variants.

The themes that are included in GTK+ (Adwaita, Adwaita dark, HighContrast, HighContrastInverse and the win32 theme) have of course been updated to follow this new naming scheme. Third-party themes need and application-specific CSS need to be updated for this too.

To help with this, we have expanded both the general documentation about CSS support in GTK+ here and here, and we have documented the element names, style classes and the node hierarchy for each widget. Here, for example, is the notebook documentation.

The documentation is also a good place to learn about style properties that have been deprecated in favor of equivalent CSS properties, like the secondary cursor color property. We warn about deprecated style properties that are used in themes or custom CSS, so it is easy to find and replace them:

(gtk3-demo:14116): Gtk-WARNING **: Theme parsing error: gtk-contained.css:18:37: The style property GtkWidget:secondary-cursor-color is deprecated and shouldn't be used anymore. It will be removed in a future version

There’s also a number of new features in CSS. We do support the CSS syntax for radial gradients, we let you load and recolor symbolic icons, image() and calc() are supported, as well as the rem (‘root em’) unit.

Beyond CSS, the drag-and-drop code as been rearchitected to move the drag cancel animation and most input handling into GDK, thereby dropping most of the platform-dependent code out of GTK+.  The main reason for doing this was to enable a complete DND implementation for Wayland. As a side-effect, we gained the ability to use non-toplevel widgets as drag icons, and we dropped the X11 specific feature to use RGBA cursors as drag icons.

The Wayland backend has grown most features that it was missing compared to X11:  the already mentioned full DND support, kinetic scrolling, startup notification, primary selection, presenting windows, a bell.

Changes in applications

Here is an unsorted list of issues that may show up in applications with GTK+ 3.20, with some advice on how to handle them.

One of the motivations for the changes is to enable animations and transitions. If you use gtk_style_context_save/restore in your draw() function, that prevents GTK+ from keeping the state that is needed to support animations; so you should avoid it when you can.

There is one place where you need to use gtk_style_context_save(), though: when using “theme colors”.  The function gtk_style_context_get_color() will warn when you pass a state other than the current state of the context. To avoid the warning, save the context and set the state:

gtk_style_context_save (context);
gtk_style_context_set_state (context, state);
gtk_style_context_get_color (context, state, &color);
gtk_style_context_restore (context);

And yes, it has been pointed out repeatedly that this change makes the state parameter of gtk_style_context_get_color() and similar functions largely useless – this API has been around sinc e 3.0, when the CSS machinery was much less developed than it is now. Back then, passing in a different state was not a problem (because animations were not really supported).

Another word of caution about  “theme colors”: CSS has no concept of foreground/background color pairs. The CSS background is just an image, which is why gtk_style_context_get_background_color() is deprecated and we cannot generally make it return a useful color. The proper way to have a theme-provided background in a widget is to call gtk_widget_render_background() in your draw() function.

If you are using type names of GTK+ widgets in your CSS, look up the element names in the documentation and use them instead. For your own widgets, use gtk_widget_class_set_css_name() to give them an element name, and use it in the CSS.

A problem that we’ve seen in some applications is the interaction between size_allocate() and draw(). GTK+’s CSS boxes need to know their size before they can draw. If you derive from a GTK+ widget and override size_allocate without chaining up, then GTK+ does not get a chance to assign sizes to the boxes. This will lead to critical warnings from GTK+’s draw() function if you don’t override it. The possible solutions to this problem are either to chain up in size_allocate or to provide your own draw implementation.

If you are using GTK+ just for themed drawing without using GTK+ widgets, you probably need to make some changes in the way you are getting theme information. We have added a foreing drawing example to gtk3-demo that shows how this can be done. The example was written with the help of libreoffice and firefox developers, and we intend to keep it up-to-date to ensure that this use case is not neglected.

A plea

If you are maintaining a GTK+ application (in particular, a big one like, say, inkscape), and you are looking at porting from GTK+ 2 to GTK+ 3, or updating it to keep up with the changes in 3.20, please let us know about the issues you find. Such feedback will be useful input for us when we get together for a GTK+ hackfest in a few weeks.

Whats coming

One of the big incoming changes for 3.22 is a GL-based renderer and scene graph. Emmanuele has been working on this on-and-off for quite a while – you may have seen some of his earlier presentations. Together with the recent merge of (copies of) clutter and cogl into mutter, this will put clutter on the path towards retirement.

 

April 27, 2016

3rd Party Fedora Repositories and AppStream

I was recently asked how to make 3rd party repositories add apps to GNOME Software. This is relevant if you run a internal private repo for employee tools, or are just kind enough to provide a 3rd party repo for Fedora or RHEL users for your free or non-free applications.

In most cases people are already running something like this to generate the repomd metadata files on a directory of RPM files:

createrepo_c --no-database --simple-md-filenames SRPMS/
createrepo_c --no-database --simple-md-filenames x86_64/

So, we need to actually generate the AppStream XML. This works by exploding any interesting .rpm files and merging together the .desktop file, the .appdata.xml file and preprocessing some icons. Only applications installing AppData files will be shown in GNOME Software, so you might need to fix before you start.

appstream-builder			\
	--origin=yourcompanyname	\
	--basename=appstream		\
	--cache-dir=/tmp/asb-cache	\
	--enable-hidpi			\
	--max-threads=1			\
	--min-icon-size=32		\
	--output-dir=/tmp/asb-md	\
	--packages-dir=x86_64/		\
	--temp-dir=/tmp/asb-icons

This takes a second or two (or 40 minutes if you’re trying to process the entire Fedora archive…) and spits out some files to /tmp/asb-md — you probably want to change some things there to make more sense for your build server.

We then have to take the generated XML and the tarball of icons and add it to the repomd.xml master document so that GNOME Software (via PackageKit) automatically downloads the content for searching. This is as simple as doing:

modifyrepo_c				\
	--no-compress			\
	--simple-md-filenames		\
	/tmp/asb-md/appstream.xml.gz	\
	x86_64/repodata/
modifyrepo_c				\
	--no-compress			\
	--simple-md-filenames		\
	/tmp/asb-md/appstream-icons.tar.gz	\
	x86_64/repodata/

Any questions, please ask. If you’re using a COPR then all these steps are done for you automatically. If you’re using xdg-app already, then this is all magically done for you as well, and automatically downloaded by GNOME Software.

April 22, 2016

Circumventing Ubuntu Snap confinement
Ubuntu 16.04 was released today, with one of the highlights being the new Snap package format. Snaps are intended to make it easier to distribute applications for Ubuntu - they include their dependencies rather than relying on the archive, they can be updated on a schedule that's separate from the distribution itself and they're confined by a strong security policy that makes it impossible for an app to steal your data.

At least, that's what Canonical assert. It's true in a sense - if you're using Snap packages on Mir (ie, Ubuntu mobile) then there's a genuine improvement in security. But if you're using X11 (ie, Ubuntu desktop) it's horribly, awfully misleading. Any Snap package you install is completely capable of copying all your private data to wherever it wants with very little difficulty.

The problem here is the X11 windowing system. X has no real concept of different levels of application trust. Any application can register to receive keystrokes from any other application. Any application can inject fake key events into the input stream. An application that is otherwise confined by strong security policies can simply type into another window. An application that has no access to any of your private data can wait until your session is idle, open an unconfined terminal and then use curl to send your data to a remote site. As long as Ubuntu desktop still uses X11, the Snap format provides you with very little meaningful security. Mir and Wayland both fix this, which is why Wayland is a prerequisite for the sandboxed xdg-app design.

I've produced a quick proof of concept of this. Grab XEvilTeddy from git, install Snapcraft (it's in 16.04), snapcraft snap, sudo snap install xevilteddy*.snap, /snap/bin/xevilteddy.xteddy . An adorable teddy bear! How cute. Now open Firefox and start typing, then check back in your terminal window. Oh no! All my secrets. Open another terminal window and give it focus. Oh no! An injected command that could instead have been a curl session that uploaded your private SSH keys to somewhere that's not going to respect your privacy.

The Snap format provides a lot of underlying technology that is a great step towards being able to protect systems against untrustworthy third-party applications, and once Ubuntu shifts to using Mir by default it'll be much better than the status quo. But right now the protections it provides are easily circumvented, and it's disingenuous to claim that it currently gives desktop users any real security.

comment count unavailable comments

April 21, 2016

You and NetworkManager 1.2 Can Still Ride Together
<figure class="wp-caption alignnone" id="attachment_837" style="width: 640px"><figcaption class="wp-caption-text">You don’t need an Uber, you don’t need a cab (via Casey Bisson CC BY-NC-SA 2.0)</figcaption></figure>

NetworkManager 1.2 was released yesterday, and it’s already built for Fedora (24 and rawhide), a release candidate is in Ubuntu 16.04, and it should appear in other distros soon too.  Lubo wrote a great post on many of the new features, but there’s too many to highlight in one post for our ADD social media 140-character tap-tap generation to handle.  Ready for more?

indicator menus

appletWayland is coming, and it doesn’t support the XEmbed status icons like nm-applet creates.  Desktop environments also want more control over how these status menus appear.  While KDE and GNOME both provide their own network status menus Ubuntu, XFCE, and LXDE use nm-applet.  How do they deal with lack of XEmbed and status icons?

Ubuntu has long patched nm-applet to add App Indicator support, which exposes the applet’s menu structure as D-Bus objects to allow the desktop environment to draw the menu just like it wants.  We enhanced the GTK3 support in libdbusmenu-gtk to handle nm-applet’s icons and then added an indicator mode to nm-applet based off Ubuntu’s work.  We’ve made packager’s lives easier by building both modes into the applet simultaneously and allowing them to be switched at runtime.

IP reconfiguration

Want to add a second IP address or change your DNS servers right away?  With NetworkManager 1.2 you can now change the IP configuration of a device through the D-Bus interface or nmcli without triggering a reconnect.  This lets the network UIs like KDE or GNOME control-center apply changes you make to network configuration immediately without interrupting your network connection.  That might take a cycle  or two to show up in your favorite desktop environment, but the basis is there.

802.1x/WPA Enterprise authentication

An oft-requested feature was the ability to use certificate domain suffix checking to validate an authentication server.  While NetworkManager has supported certificate subject checking for years, this has limitations and isn’t as secure as domain suffix checking.  Both these options help prevent man-in-the-middle attacks where a rogue access point could masquerade as as your normal secure network.  802.1x authentication is still too complicated, and we hope to greatly simplify it in upcoming releases.

Interface stacking

While NM has always been architected to allow bridges-on-bonds-on-VLANs, there were some internal issues that prevented these more complicated configurations from working.  We’ve fixed those bugs, so now layer-cake network setups work in a flash!  Hopefully somebody will come up with a fancy drag-n-drop UI based off Minecraft or CandyCrush with arbitrary interface trees.  Maybe it’ll even have trophies when you finally get a Level 48 active-backup bond.

Old Stable Series

Now that 1.2 is out, the 1.0 series is in maintenance mode.  We’ll fix bugs and any security issues that come up, but typically don’t add new features.  Backporting from 1.2 to 1.0 will be even more difficult due to the removal of dbus-glib, a major feature in 1.2 release.  If you’re on 1.0, 0.9.10, or (gasp!) 0.9.8 I’d urge you to upgrade, and I think you’ll like what you see!

April 20, 2016

Upgrading Fedora 23 to 24 using GNOME Software

I’ve spent the last couple of days fixing up all the upgrade bugs in GNOME Software and backporting them to gnome-3-20. The idea is that we backport gnome-software plus a couple of the deps into Fedora 23 so that we can offer a 100% GUI upgrade experience. It’s the first time we’ve officially transplanted a n+1 GNOME component into an older release (ignoring my unofficial Fedora 20 whole-desktop backport COPR) and so we’re carefully testing for regressions and new bugs.

If you do want to test upgrading from F23 to F24, first make sure you’ve backed up your system. Then, install and enable this COPR and update gnome-software. This should also install a new libhif, libappstream-glib, json-glib and PackageKit and a few other bits. If you’ve not done the update offline using [the old] GNOME Software, you’ll need to reboot at this stage as well.

Fire up the new gnome-software and look at the new UI. Actually, there’s not a lot new to see as we’ve left new features like the ODRS reviewing service and xdg-app as F24-only features, so it should be mostly the same as before but with better search results. Now go to the Updates page which will show any updates you have pending, and it will also download the list of possible distro upgrades to your home directory.

As we’re testing upgrading to a pre-release, we have to convince gnome-software that we’re living in the future. First, open ~/.cache/gnome-software/3.20/upgrades/fedora.json and search for f24. Carefully change the Under Development string to Active then save the file. Log out, back in and the launch gnome-software again or wait for the notification from the shell. If all has gone well you should see a banner telling you about the new upgrade. If you click Download go and get a coffee and start baking a cake, as it’s going to take a long time to download all that new goodness. Once complete just click Install, which prompts a reboot where the packages will be installed. For this step you’ll probably want to bake another cake. We’re not quite in an atomic instant-apply world yet, although I’ll be talking a lot more about that for Fedora 25.

With a bit of luck, after 30 minutes staring at a progressbar the computer should reboot itself into a fresh new Fedora 24 beta installation. Success!

Screenshot_Fedora23-Upgrade_2016-04-20_15:23:27

If you spot any problems or encounter any bugs, please let me know either in bugzilla, email or or IRC. I’ve not backported all the custom CSS for the upgrade banner just yet, but this should be working soon. Thanks!

April 18, 2016

libinput and graphics tablet pad support

When we released graphics tablet support in libinput earlier this year, only tablet tools were supported. So while you could use the pen normally, the buttons, rings and strips on the physical tablet itself (the "pad") weren't detected by libinput and did not work. I have now merged the patches for pad support into libinput.

The reason for the delay was simple: we wanted to get it right [1]. Pads have a couple of properties that tools don't have and we always considered pads to be different to pens and initially focused on a more generic interface (the "buttonset" interface) to accommodate for those. After some coding, we have now arrived at a tablet pad-specific interface instead. This post is a high-level overview of the new tablet pad interface and how we intend it do be used.

The basic sign that a pad is present is when a device has the tablet pad capability. Unlike tools, pads don't have proximity events, they are always considered in proximity and it is up to the compositor to handle the focus accordingly. In most cases, this means tying it to the keyboard focus. Usually a pad is available as soon as a tablet is plugged in, but note that the Wacom ExpressKey Remote (EKR) is a separate, wireless device and may be connected after the physical pad. It is up to the compositor to link the EKR with the correct tablet (if there is more than one).

Pads have three sources of events: buttons, rings and strips. Rings and strips are touch-sensitive surfaces and provide absolute values - rings in degrees, strips in normalized [0.0, 1.0] coordinates. Similar to pointer axis sources we provide a source notification. If that source is "finger", then we send a terminating out-of-range event so that the caller can trigger things like kinetic scrolling.

Buttons on a pad are ... different. libinput usually re-uses the Linux kernel's include/input.h event codes for buttons and keys. But for the pad we decided to use plain sequential button numbering, starting at index 0. So rather than a semantic code like BTN_LEFT, you'd simply get a button 0 event. The reasoning behind this is a caveat in the kernel evdev API: event codes have semantic meaning (e.g. BTN_LEFT) but buttons on a tablet pad don't those meanings. There are some generic event ranges (e.g. BTN_0 through to BTN_9) and the Wacom tablets use those but once you have more than 10 buttons you leak into other ranges. The ranges are simply too narrow so we end up with seemingly different buttons even though all buttons are effectively the same. libinput's pad support undoes that split and combines the buttons into a simple sequential range and leaves any semantic mapping of buttons to the caller. Together with libwacom which describes the location of the buttons a caller can get a relatively good idea of how the layout looks like.

Mode switching is a commonly expected feature on tablet. One button is designated as mode switch button and toggles all other buttons between the available modes. On the Intuos Pro series tablets, that button is usually the button inside the ring. Button mapping and thus mode switching is however a feature we leave up to the caller, if you're working on a compositor you will have to implemented mode switching there.

Other than that, pad support is relatively simple and straightforward and should not cause any big troubles.

[1] or at least less wrong than in the past
[2] They're actually linux/input-event-codes.h in recent kernels

One more attempt at SATA power management
Around a year ago I wrote some patches in an attempt to improve power management on Haswell and Broadwell systems by configuring Serial ATA power management appropriately. I got a couple of reports of them triggering SATA errors for some users, couldn't reproduce them myself and so didn't have a lot of confidence in them. Time passed.

I've been working on power management stuff again this week, so it seemed like a good opportunity to revisit these. I've made a few changes and pushed a couple of trees - one against master and one against 4.5.

First, these probably only have relevance to users of mobile Intel parts in the U or S range (/proc/cpuinfo will tell you - you're looking for a four-digit number that starts with 4 (Haswell), 5 (Broadwell) or 6 (Skylake) and ends with U or S), and won't do anything unless you have SATA drives (including PCI-based SATA). To test them, first disable anything like TLP that might alter your SATA link power management policy. Then check powertop - you should only be getting to PC3 at best. Build a kernel with these patches and boot it. /sys/class/scsi_host/*/link_power_management_policy should read "firmware". Check powertop and see whether you're getting into deeper PC states. Now run your system for a while and check the kernel log for any SATA errors that you didn't see before.

Let me know if you see SATA errors and are willing to help debug this, and leave a comment if you don't see any improvement in PC states.

comment count unavailable comments

April 16, 2016

My first xdg-app

A few days ago, I set out to get some experience with building an application as an xdg-app.  In this post, I’m collecting some of the lessons I learned.

Since I didn’t want pick a too easy test case, I chose terminix, a promising terminal emulator for GNOME. Terminix uses GTK+ and vte, which means that most dependencies are already present in the GNOME runtime.

Terminix

However, terminix is written in D, and the GNOME sdk does not include D support.  So, the challenge here is to build a compiler and runtime for this language, and any other required language-specific utilities. The DMD D compiler is written in D (naturally), so some bootstrapping was required.

Build tools

xdg-app comes with low-level build support in the form of various xdg-app build commands. But you really want to use the newer xdg-app-builder tool. It is really nice.

xdg-app-builder downloads and builds the application and all its dependencies, according to a JSON manifest.  Thats par for the course for modern build tools, of course. But xdg-app-builder also has smart caching: It keeps git checkouts  of all sources (if they are in git), and only rebuilds them when they change. It also keeps the results of each modules’ build in an ostree repository, so reusing a previous build is really fast.

All the caches are kept in .xdg-app-builder/ in the directory where you run the build. If you have many dependencies, this hidden directory can grow large, so you might want to keep an eye on it and clean it out every now and then (remember, it is just a cache).

You can take a look at the JSON file I came up with.

Build API

Similar to the GNOME Continuous build system, xdg-app-builder assumes that each module in your JSON supports the ‘build api’ which consists of configure & make & make install. The world is of course more diverse than that, and rolling your own build system is irresistable for some.

Here is a way to quickly add the required build api support to almost any module (I’ve stolen this setup from the pitivi xdg-app build scripts):

Place a foo-configure script next to your JSON recipe that looks like this (note that Makefile syntax requires tabs that were eaten when I pasted this content in here):

#!/bin/sh

cat <<EOF >Makefile
all:
        ...do whatever is needed to build foo

install:
        ...commands to install foo go here

EOF

In the JSON  fragment for the foo module, you add this file as an extra source (we are taking advantage of the fact that xdg-app-builder allows multiple sources for a module):

"modules": [
    {
        "name": "foo",
        "sources": [
            {
                "type": "git",
                 "url": "http://foo.org/foo.git",
                 "branch": "master"
            },
            {
                "type": "file",
                "path": "foo-configure",
                "dest-filename": "configure"
            }
        ]
    }

I guess you could just as well have the Makefile as yet another source; this approach is following the traditional role of configure scripts to produce Makefiles.

Network access

As I mentioned already, the first step in my build was to build a D compiler written in D. Thankfully, the build script of the dmd compiler is prepared for handling this kind of bootstrapping. It does so by downloading a pre-built D compiler that is used to build the sources.

Downloading things during the build is not great for trusted and repeatable builds. And xdg-app’s build support is set up to produce such builds by running the build in a controlled, sandboxed environment, which doesn’t have network access.

So, in order to get the D compiler built, had to weaken the sandboxing for this module, and grant it network access.  It took me a little while to find out that the build-args field in the build-options does this job:

"modules": [
    {
        "name": "dmd",
        "build-options":
            {
                "build-args": ["--share=network"]
            },
        ...

Shedding weight

After navigating around other hurdles, I eventually succeeded in having a build of my entire JSON recipe run through the end. Yay! But I quickly discovered that the build directory was quite heavy. It came to over 200M, a bit much for a terminal.

xdg-app-builder creates the final build by combining the build results from all the modules in the JSON recipe. That means my terminix build included a D compiler, static and shared libraries for the D runtime, build utilties, etc.

To fix this, I added a couple of cleanup commands to the JSON. These are run after all the modules have been built, and can remove things that are no longer needed.

"cleanup-commands": ["rm -rf /app/lib",
                     "rm -rf /app/src",
                      rm -rf /app/share/nautilus-python",
                      "rm /app/bin/dmd",
                      ...

Note that the paths start with /app, which is the prefix that xdg-app apps are put in (to avoid interference with /usr).

After these cleanups, my build weighed less than a megabyte, which is more acceptable.

Trying it out

The best way to distribute an xdg-app is via an OSTree repository. Since I don’t have a good place to put one, and Terminix is developed on github, I decided to turn my xdg-app into a bundle, using this command:

xdg-app build-bundle ~/xdg-app-repos/terminix \
                     terminix.x86_64.xdgapp \
                     com.gexperts.Terminix \ 
                     master

Since github has a concept of releases, I’ve just put the bundle there:

https://github.com/matthiasclasen/terminix/releases/tag/2016-04-15

Enjoy!

April 15, 2016

David MacKay
The first time I was paid to do software development came as something of a surprise to me. I was working as a sysadmin in a computational physics research group when a friend asked me if I'd be willing to talk to her PhD supervisor. I had nothing better to do, so said yes. And that was how I started the evening having dinner with David MacKay, and ended the evening better fed, a little drunker and having agreed in principle to be paid to write free software.

I'd been hired to work on Dasher, an information-efficient text entry system. It had been developed by one of David's students as a practical demonstration of arithmetic encoding after David had realised that presenting a visualisation of an effective compression algorithm allowed you to compose text without having to enter as much information into the system. At first this was merely a neat toy, but it soon became clear that the benefits of Dasher had a great deal of overlap with good accessibility software. It required much less precision of input, it made it easy to correct mistakes (you merely had to reverse direction in order to start zooming back out of the text you had entered) and it worked with a variety of input technologies from mice to eye tracking to breathing. My job was to take this codebase and turn it into a project that would be interesting to external developers.

In the year I worked with David, we turned Dasher from a research project into a well-integrated component of Gnome, improved its support for Windows, accepted code from an external contributor who ported it to OS X (using an OpenGL canvas!) and wrote ports for a range of handheld devices. We added code that allowed Dasher to directly control the UI of other applications, making it possible for people to drive word processors without having to leave Dasher. We taught Dasher to speak. We strove to avoid the mistakes present in so many other pieces of accessibility software, such as configuration that could only be managed by an (expensive!) external consultant. And we visited Dasher users and learned how they used it and what more they needed, then went back home and did what we could to provide that.

Working on Dasher was an incredible opportunity. I was involved in the development of exciting code. I spoke on it at multiple conferences. I became part of the Gnome community. I visited the USA for the first time. I entered people's homes and taught them how to use Dasher and experienced their joy as they realised that they could now communicate up to an order of magnitude more quickly. I wrote software that had a meaningful impact on the lives of other people.

Working with David was certainly not easy. Our weekly design meetings were, charitably, intense. He had an astonishing number of ideas, and my job was to figure out how to implement them while (a) not making the application overly complicated and (b) convincing David that it still did everything he wanted. One memorable meeting involved me gradually arguing him down from wanting five new checkboxes to agreeing that there were only two combinations that actually made sense (and hence a single checkbox) - and then admitting that this was broadly equivalent to an existing UI element, so we could just change the behaviour of that slightly without adding anything. I took the opportunity to delete an additional menu item in the process.

I was already aware of the importance of free software in terms of developers, but working with David made it clear to me how important it was to users as well. A community formed around Dasher, helping us improve it and allowing us to develop support for new use cases that made the difference between someone being able to type at two words per minute and being able to manage twenty. David saw that this collaborative development would be vital to creating something bigger than his original ideas, and it succeeded in ways he couldn't have hoped for.

I spent a year in the group and then went back to biology. David went on to channel his strong feelings about social responsibility into issues such as sustainable energy, writing a freely available book on the topic. He served as chief adviser to the UK Department of Energy and Climate Change for five years. And earlier this year he was awarded a knighthood for his services to scientific outreach.

David died yesterday. It's unlikely that I'll ever come close to what he accomplished, but he provided me with much of the inspiration to try to do so anyway. The world is already a less fascinating place without him.

comment count unavailable comments

April 13, 2016

Skylake's power management under Linux is dreadful and you shouldn't buy one until it's fixed
(Edit to add: this issue is restricted to the mobile SKUs. Desktop parts have very different power management behaviour)

Linux 4.5 seems to have got Intel's Skylake platform (ie, 6th-generation Core CPUs) to the point where graphics work pretty reliably, which is great progress (4.4 tended to lose all my windows every so often, especially over suspend/resume). I'm even running Wayland happily. Unfortunately one of the reasons I have a laptop is that I want to be able to do things like use it on battery, and power consumption's an important part of that. Skylake continues the trend from Haswell of moving to an SoC-type model where clock and power domains are shared between components that were previously entirely independent, and so you can't enter deep power saving states unless multiple components all have the correct power management configuration. On Haswell/Broadwell this manifested in the form of Serial ATA link power management being involved in preventing the package from going into deep power saving states - setting that up correctly resulted in a reduction in full-system power consumption of about 40%[1].

I've now got a Skylake platform with a nice shiny NVMe device, so Serial ATA policy isn't relevant (the platform doesn't even expose a SATA controller). The deepest power saving state I can get into is PC3, despite Skylake supporting PC8 - so I'm probably consuming about 40% more power than I should be. And nobody seems to know what needs to be done to fix this. I've found no public documentation on the power management dependencies on Skylake. Turning on everything in Powertop doesn't improve anything. My battery life is pretty poor and the system is pretty warm.

The best thing about this is the following statement from page 64 of the 6th Generation Intel ® Processor Datasheet for U-Platforms:

Caution: Long term reliability cannot be assured unless all the Low-Power Idle States are enabled.

which is pretty concerning. Without support for states deeper than PC3, Linux is running in a configuration that Intel imply may trigger premature failure. That's obviously not good. Until this situation is improved, you probably shouldn't buy any Skylake systems if you're planning on running Linux.

[1] These patches never went upstream. Someone reported that they resulted in their SSD throwing errors and I couldn't find anybody with deeper levels of SATA experience who was interested in working on the problem. Intel's AHCI drivers for Windows do the right thing, but I couldn't find anybody at Intel who could get any information from their Windows driver team.

comment count unavailable comments

April 11, 2016

Making it easier to deploy TPMTOTP on non-EFI systems
I've been working on TPMTOTP a little this weekend. I merged a pull request that adds command-line argument handling, which includes the ability to choose the set of PCRs you want to seal to without rebuilding the tools, and also lets you print the base32 encoding of the secret rather than the qr code so you can import it into a wider range of devices. More importantly it also adds support for setting the expected PCR values on the command line rather than reading them out of the TPM, so you can now re-seal the secret against new values before rebooting.

I also wrote some new code myself. TPMTOTP is designed to be usable in the initramfs, allowing you to validate system state before typing in your passphrase. Unfortunately the initramfs itself is one of the things that's measured. So, you end up with something of a chicken and egg problem - TPMTOTP needs access to the secret, and the obvious thing to do is to put the secret in the initramfs. But the secret is sealed against the hash of the initramfs, and so you can't generate the secret until after the initramfs. Modify the initramfs to insert the secret and you change the hash, so the secret is no longer released. Boo.

On EFI systems you can handle this by sticking the secret in an EFI variable (there's some special-casing in the code to deal with the additional metadata on the front of things you read out of efivarfs). But that's not terribly useful if you're not on an EFI system. Thankfully, there's a way around this. TPMs have a small quantity of nvram built into them, so we can stick the secret there. If you pass the -n argument to sealdata, that'll happen. The unseal apps will attempt to pull the secret out of nvram before falling back to looking for a file, so things should just magically work.

I think it's pretty feature complete now, other than TPM2 support? That's on my list.

comment count unavailable comments

April 07, 2016

Why libinput doesn't have a lot of config options

Most days, at least one of the bugs I deal with requests something along the lines of "just add $FOO as a config option". In this post, I'll explain why this is usually a bad solution. First, read http://www.islinuxaboutchoice.com/ and keep those arguments in mind. Generally, there are two groups of configuration options - hardware options and user options. Hardware options are those that deal with specific quirks needed on some hardware, but not on other hardware. User options are those that deal with user preferences such as tapping or two-finger vs. edge scrolling.

In the old synaptics driver, we added options whenever something new came up and we tried to make those options generic. This was a big mistake. The driver now has over 70 configuration options resulting in a test matrix with a googolplex of combinations. In other words, it's completely untestable. To make a device work users often have to find the right combination of options from somewhere, write out a root-owned config file and then hope this works. Why do we still think this is acceptable? Even worse: some options are very specific to hardware but still spread in user forum examples like an STD during spring break.

In libinput, we're having none of that. When hardware doesn't work we expect a user to file a bug, we get it fixed upstream for the specific model and thus automatically fix it for all users of that device. We're leaning heavily on udev's hwdb which we have extended to correct devices when the firmware announces wrong information. This has the advantage that there is only one authoritative source of quirks a device needs to work. And we can update this as time goes by without having to worry about stale configuration options. One good example here is the custom acceleration profile that Lenovo X230 touchpads have in libinput. All in all, there is little pushback for the lack of hardware-specific configuration options and most users are fine with it once they accept the initial waiting period to get the patch into their distribution.

User-specific options are more contentious. In our opinion, some features should be configurable and others should not. Where to draw that line is of course quite undefined. For example, tapping on or off was one of the first configuration options available and that was never a cause for arguments either way (except whether the default should be on or off). Other options are more contentious. Clickpad software buttons are always on the bottom edge and their size is hardcoded (synaptics allowed almost free positioning of those buttons). Other features such as changing a two-finger tap to some other button event is not supported at all in libinput. This effectively comes down to cost. You see, whenever you write "it's just 5 lines of code to make this an option", what I think is "once the patch is reviewed and applied, I'll spend two days to write test cases and documentation. I'll need to handle any bug reports related to this, and I'm expected to make sure this option works indefinitely. Any addition of another feature may conflict with this option, so I need to make sure the right combination is possible and test cases are written." So your work ends after writing a 5 line patch, my work as maintainer merely starts. And unless it pays off long-term, the effort is not worth it. Some features make that cut, others don't if they are too much of a niche feature.

All this is of course nothing new and every software project needs to make these decisions. Input isn't even a special case here, it pales in comparison with e.g. the decisions UI designers need to make. However, in FOSS we have a tendency to think that because something is possible, it should be done. Legally, you have freedom to do almost anything with the software, so you can maintain a local fork of libinput with that extra feature applied. If that isn't acceptable, why would it be acceptable to merge the patch and expect others to shoulder the costs?

April 05, 2016

libinput now has a touchpad software middle button

I just pushed a patch to libinput master to enable a middle button on the clickpad software buttons. Until now, our stance was that clickpads only get a left and right software button, split at the 50% mark. The reasoning is simple: touchpads only have markings for left and right buttons (if any!) and the middle button's extents are not easily discoverable if there is no visual or haptic feedback. A middle button event could however be triggered through middle button emulation, i.e. by clicking the touchpad with a finger on the left and right software button area (see the instructions here).

This is nice in theory but, as usual, reality gets in the way. Most interactions with the middle button are quick and short-lived, i.e. clicking the button once to paste. This interaction is what many touchpads are spectacularly bad at. For middle button emulation to be handled correctly, both fingers must be registered before the physical button press. The scanout rate on a touchpad is often too low and on touchpads with extremely light resistance like the T440 it's common to register the physical click before we know that there's a second finger on the touchpad. But even on a T450 and an X220 with much higher clickpad resistance I barely managed to get above 7 out of 10 correctly registered middle button events. That is simply not good enough.

So the patch I just pushed out to master enables a middle software button between the left and the right button. The exact width of the button scales with the touchpad but it's usually around 20-25mm and it's centered on the touchpad so despite the lack of visual or haptic guides it should be reliable to hit. The new behaviour is hard-coded and for now middle button emulation continues to work on touchpads. In the future, I expect I will remove middle button emulation on touchpads or at least disable it by default.

The middle button will be available in libinput 1.3.

There's more than one way to exploit the commons
There's a piece of software called XScreenSaver. It attempts to fill two somewhat disparate roles:
  • Provide a functioning screen lock on systems using the X11 windowing system, a job made incredibly difficult due to a variety of design misfeatures in said windowing system[1]
  • Provide cute graphical output while the screen is locked
XScreenSaver does an excellent job of the second of these[2] and is pretty good at the first, which is to say that it only suffers from a disastrous security flaw once very few years and as such is certainly not appreciably worse than any other piece of software.

Debian ships an operating system that prides itself on stability. The Debian definition of stability is a very specific one - rather than referring to how often the software crashes or misbehaves, it refers to how often the software changes behaviour. Debian is very reluctant to upgrade software that is part of a stable release, to the extent that developers will attempt to backport individual security fixes to the version they shipped rather than upgrading to a release that contains all those security fixes but also adds a new feature. The argument here is that the new release may also introduce new bugs, and Debian's users desire stability (in the "things don't change" sense) more than new features. Backporting security fixes keeps them safe without compromising the reason they're running Debian in the first place.

This all makes plenty of sense at a theoretical level, but reality is sometimes less convenient. The first problem is that security bugs are typically also, well, bugs. They may make your software crash or misbehave in annoying but apparently harmless ways. And when you fix that bug you've also fixed a security bug, but the ability to determine whether a bug is a security bug or not is one that involves deep magic and a fanatical devotion to the cause so given the choice between maybe asking for a CVE and dealing with embargoes and all that crap when perhaps you've actually only fixed a bug that makes the letter "E" appear in places it shouldn't and not one that allows the complete destruction of your intergalactic invasion fleet means people will tend to err on the side of "Eh fuckit" and go drinking instead. So new versions of software will often fix security vulnerabilities without there being any indication that they do so[3], and running old versions probably means you have a bunch of security issues that nobody will ever do anything about.

But that's broadly a technical problem and one we can apply various metrics to, and if somebody wanted to spend enough time performing careful analysis of software we could have actual numbers to figure out whether the better security approach is to upgrade or to backport fixes. Conversations become boring once we introduce too many numbers, so let's ignore that problem and go onto the second, which is far more handwavy and social and so significantly more interesting.

The second problem is that upstream developers remain associated with the software shipped by Debian. Even though Debian includes a tool for reporting bugs against packages included in Debian, some users will ignore that and go straight to the upstream developers. Those upstream developers then have to spend at least 15 or so seconds telling the user that the bug they're seeing has been fixed for some time, and then figure out how to explain that no sorry they can't make Debian include a fixed version because that's not how things work. Worst case, the stable release of Debian ends up including a bug that makes software just basically not work at all and everybody who uses it assumes that the upstream author is brutally incompetent, and they end up quitting the software industry and I don't know running a nightclub or something.

From the Debian side of things, the straightforward solution is to make it more obvious that users should file bugs with Debian and not bother the upstream authors. This doesn't solve the problem of damaged reputation, and nor does it entirely solve the problem of users contacting upstream developers. If a bug is filed with Debian and doesn't get fixed in a timely manner, it's hardly surprising that users will end up going upstream. The Debian bugs list for XScreenSaver does not make terribly attractive reading.

So, coming back to the title for this entry. The most obvious failure of the commons is where a basically malicious actor consumes while giving nothing back, but if an actor with good intentions ends up consuming more than they contribute that may still be a problem. An upstream author releases a piece of software under a free license. Debian distributes this to users. Debian's policies result in the upstream author having to do more work. What does the upstream author get out of this exchange? In an ideal world, plenty. The author's software is made available to more people. A larger set of developers is willing to work on making improvements to the software. In a less ideal world, rather less. The author has to deal with bug mail about already fixed bugs. The author's reputation may be harmed by user exposure to said fixed bugs. The author may get less in the way of useful bug fixes or features because people are running old versions rather than fixing new ones. If the balance tips towards the latter, the author's decision to release their software under a free license has made their life more difficult.

Most discussions about Debian's policies entirely ignore the latter scenario, focusing more on the fact that the author chose to release their software under a free license to begin with. If the author is unwilling to handle the consequences of that, goes the argument, why did they do it in the first place? The unfortunate logical conclusion to that argument is that the author realises that they made a huge mistake and never does so again, and woo uh oops.

The irony here is that one of Debian's foundational documents, the Debian Free Software Guidelines, makes allowances for this. Section 4 allows for distribution of software in Debian even if the author insists that modified versions[4] are renamed. This allows for an author to make a choice - allow themselves to be associated with the Debian version of their work and increase (a) their userbase and (b) their support load, or try to distinguish what Debian ship from their identity. But that document was ratified in 1997 and people haven't really spent much time since then thinking about why it says what it does, and so this tradeoff is rarely considered.

Free software doesn't benefit from distributions antagonising their upstreams, even if said upstream is a cranky nightclub owner. Debian's users are Debian's highest priority, but those users are going to suffer if developers decide that not using free licenses improves their quality of life. Kneejerk reactions around specific instances aren't helpful, but now is probably a good time to start thinking about what value Debian bring to its upstream authors and how that can be increased. Failing to do so doesn't serve users, Debian itself or the free software community as a whole.

[1] The X server has no fundamental concept of a screen lock. This is implemented by an application asking that the X server send all keyboard and mouse input to it rather than to any other application, and then that application creating a window that fills the screen. Due to some hilarious design decisions, opening a pop-up menu in an application prevents any other application from being able to grab input and so it is impossible for the screensaver to activate if you open a menu and then walk away from your computer. This is merely the most obvious problem - there are others that are more subtle and more infuriating. The only fix in this case is to nuke the site from orbit.

[2] There's screenshots here. My favourites are the one that emulate the electrical characteristics of an old CRT in order to present a more realistic depiction of the output of an Apple 2 and the one that includes a complete 6502 emulator.

[3] And obviously new versions of software will often also introduce new security vulnerabilities without there being any indication that they do so, because who would ever put that in their changelog. But the less ethically challenged members of the security community are more likely to be looking at new versions of software than ones released three years ago, so you're probably still tending towards winning overall

[4] There's a perfectly reasonable argument that all packages distributed by Debian are modified in some way

comment count unavailable comments

April 04, 2016

TPMs, event logs, fine-grained measurements and avoiding fragility in remote-attestation
Trusted Platform Modules are fairly unintelligent devices. They can do some crypto, but they don't have any ability to directly monitor the state of the system they're attached to. This is worked around by having each stage of the boot process "measure" state into registers (Platform Configuration Registers, or PCRs) in the TPM by taking the SHA1 of the next boot component and performing an extend operation. Extend works like this:

New PCR value = SHA1(current value||new hash)

ie, the TPM takes the current contents of the PCR (a 20-byte register), concatenates the new SHA1 to the end of that in order to obtain a 40-byte value, takes the SHA1 of this 40-byte value to obtain a 20-byte hash and sets the PCR value to this. This has a couple of interesting properties:
  • You can't directly modify the contents of the PCR. In order to obtain a specific value, you need to perform the same set of writes in the same order. If you replace the trusted bootloader with an untrusted one that runs arbitrary code, you can't rewrite the PCR to cover up that fact
  • The PCR value is predictable and can be reconstructed by replaying the same series of operations
But how do we know what those operations were? We control the bootloader and the kernel and we know what extend operations they performed, so that much is easy. But the firmware itself will have performed some number of operations (the firmware itself is measured, as is the firmware configuration, and certain aspects of the boot process that aren't in our control may also be measured) and we may not be able to reconstruct those from scratch.

Thankfully we have more than just the final PCR data. The firmware provides an interface to log each extend operation, and you can read the event log in /sys/kernel/security/tpm0/binary_bios_measurements. You can pull information out of that log and use it to reconstruct the writes the firmware made. Merge those with the writes you performed and you should be able to reconstruct the final TPM state. Hurrah!

The problem is that a lot of what you want to measure into the TPM may vary between machines or change in response to configuration changes or system updates. If you measure every module that grub loads, and if grub changes the order that it loads modules in, you also need to update your calculations of the end result. Thankfully there's a way around this - rather than making policy decisions based on the final TPM value, just use the final TPM value to ensure that the log is valid. If you extract each hash value from the log and simulate an extend operation, you should end up with the same value as is present in the TPM. If so, you know that the log is valid. At that point you can examine individual log entries without having to care about the order that they occurred in, which makes writing your policy significantly easier.

But there's another source of fragility. Imagine that you're measuring every command executed by grub (as is the case in the CoreOS grub). You want to ensure that no inappropriate commands have been run (such as ones that would allow you to modify the loaded kernel after it's been measured), but you also want to permit certain variations - for instance, you might have a primary root filesystem and a fallback root filesystem, and you're ok with either being passed as a kernel argument. One approach would be to write two lines of policy, but there's an even more flexible approach. If the bootloader logs the entire command into the event log, when replaying the log we can verify that the event description hashes to the value that was passed to the TPM. If it does, rather than testing against an explicit hash value, we can examine the string itself. If the event description matches a regular expression provided by the policy then we're good.

This approach makes it possible to write TPM policies that are resistant to changes in ordering and permit fine-grained definition of acceptable values, and which can cleanly separate out local policy, generated policy values and values that are provided by the firmware. The split between machine-specific policy and OS policy allows for the static machine-specific policy to be merged with OS-provided policy, making remote attestation viable even over automated system upgrades.

We've integrated an implementation of this kind of policy into the TPM support code we'd like to integrate into Kubernetes, and CoreOS will soon be generating known-good hashes at image build time. The combination of these means that people using Distributed Trusted Computing under Tectonic will be able to validate the state of their systems with nothing more than a minimal machine-specific policy description.

The support code for all of this should also start making it into other distributions in the near future (the grub code is already in Fedora 24), so with luck we can define a cross-distribution policy format and make it straightforward to handle this in a consistent way even in hetrogenous operating system environments. Remote attestation is a powerful tool for ensuring that your systems are in a valid state, but the difficulty of policy management has been a significant factor in making it difficult for people to deploy in their data centres. Making it easier for people to shield themselves against low-level boot attacks is a big step forward in improving the security of distributed workloads and makes bare-metal hosting a much more viable proposition.

comment count unavailable comments

April 03, 2016

Announcing systemd.conf 2016

Announcing systemd.conf 2016

We are happy to announce the 2016 installment of systemd.conf, the conference of the systemd project!

After our successful first conference 2015 we’d like to repeat the event in 2016 for the second time. The conference will take place on September 28th until October 1st, 2016 at betahaus in Berlin, Germany. The event is a few days before LinuxCon Europe, which also is located in Berlin this year. This year, the conference will consist of two days of presentations, a one-day hackfest and one day of hands-on training sessions.

The website is online now, please visit https://conf.systemd.io/.

Tickets at early-bird prices are available already. Purchase them at https://ti.to/systemdconf/systemdconf-2016.

The Call for Presentations will open soon, we are looking forward to your submissions! A separate announcement will be published as soon as the CfP is open.

systemd.conf 2016 is a organized jointly by the systemd community and kinvolk.io.

We are looking for sponsors! We’ve got early commitments from some of last year’s sponsors: Collabora, Pengutronix & Red Hat. Please see the web site for details about how your company may become a sponsor, too.

If you have any questions, please contact us at info@systemd.io.

March 13, 2016

why libinput does not have a configuration storage system

A question that pops up with some regularity is whether libinput has a global configuration storage system, and, subsequently, why it doesn't have one. Comparisons are drawn to the X-specific tool xinput that allows to trigger all configuration options (see below though).

As maintainer of libinput, I can state that libinput will not get a configuration storage system. That job is squarely in the realm of the caller (i.e. usually the compositor and/or the desktop environment). Here are a few reasons:

First, you'd get conflicts with the caller. You need to prioritise which configuration has precendence to decide what to do when libinput and the caller disagree on a configuration item. You (the user) also have to figure why a configuration doesn't work when it's clearly enabled in one of those systems. Ok, so you can work around this by putting in a warning somewhere, provided that you make sure that the warning shows up in the right place and users know where to look. And to know when to send the warning, because again, that requires libinput to know which config has priority.

This is amplified by the lack of an autoritative support system. Speaking from X experience, the number of posts on, say, the ubuntu forums that advocate setting configuration options that haven't existed for years is quite sad. Or users advocate config snippets that set everything but the feature they claim it enables. That gets copy-pasted, etc.

Some configuration options can be incompatible with each other. If you have conflicting configuration systems it gets harder because each configuration system cannot make certain options an either/or anymore. We don't have any of these options just yet, but they may come in the future.

Over time, supported features will change. A setting may be exposed in the magic libinput config system and, a few months later, it is now also exposed by, say, GNOME. All the documentation that points to the libinput configuration is now obsolete because GNOME overrides it. Unless you're running version Foo.Bar, or maybe the patched version from $DISTRIBUTION. It gets messy quickly.

How fine-grained do you need the configuration? libinput's config API applies separately for each device. For most desktop environments it is sufficient to have configuration per device type (touchpad vs mouse vs tablet, etc.). But some exceptions may apply and a newer generic touchpad configuration may need to override device-specific older configuration. Because the newer config is designed around the new libinput version, but the older config is something you copied off a forum post from 2 years ago.

Oh, implicit in the request for a configuration system in libinput is usually also: please write and maintain that system for free, and fix any bugs in a timely manner. And I can't shake the feeling that a large number of these requests are only triggered by "my desktop doesn't expose this setting, if we get it in libinput I can force it into any desktop".

Ironically enough, the tool that is usually used as an example for how it could work is xinput. The only reason xinput works is because the xf86-input-libinput driver exposes properties and maps changes in those to the libinput API calls. i.e. everything is handled by libinput's caller, not libinput itself. Just as it should be.

Lest I be accused of only shooting down ideas here is a constructive idea: write a library or DBus-daemon that exposes some configuration storage and makes it easy to query. Then convince the various desktop environments to use that system instead their existing solutions, and bang, you have a generic configuration storage system for libinput.

March 11, 2016

I stayed in a hotel with Android lightswitches and it was just as bad as you'd imagine
I'm in London for Kubecon right now, and the hotel I'm staying at has decided that light switches are unfashionable and replaced them with a series of Android tablets. <figure>A tablet displaying the text UK_bathroom isn't responding. Do you want to close it?</figure> One was embedded in the wall, but the two next to the bed had convenient looking ethernet cables plugged into the wall. So.

I managed to borrow a couple of USB ethernet adapters, set up a transparent bridge (brctl addbr br0; brctl addif br0 enp0s20f0u1; brctl addif br0 enp0s20f0u2; ifconfig br0 up) and then stuck my laptop between the tablet and the wall. tcpdump -i br0 showed traffic, and wireshark revealed that it was Modbus over TCP. Modbus is a pretty trivial protocol, and notably has no authentication whatsoever. tcpdump showed that traffic was being sent to 172.16.207.14, and pymodbus let me start controlling my lights, turning the TV on and off and even making my curtains open and close. What fun!

And then I noticed something. My room number is 714. The IP address I was communicating with was 172.16.207.14. They wouldn't, would they?

I mean yes obviously they would.

It's basically as bad as it could be - once I'd figured out the gateway, I could access the control systems on every floor and query other rooms to figure out whether the lights were on or not, which strongly implies that I could control them as well. Jesus Molina talked about doing this kind of thing a couple of years ago, so it's not some kind of one-off - instead, hotels are happily deploying systems with no meaningful security, and the outcome of sending a constant stream of "Set room lights to full" and "Open curtain" commands at 3AM seems fairly predictable.

We're doomed.

(edited: this previously claimed I could only access systems on my own floor, but it turns out that each floor is a separate broadcast domain and I just needed to set a gateway to access the others)

(further edit: I'm deliberately not naming the hotel. They were receptive to my feedback and promised to do something about the issue.)

comment count unavailable comments

March 08, 2016

Relatório da Hackatona de Design
Durante semana, no mês de janeiro, eu estive no Brasil, no Rio de Janeiro para uma hackatona de design, com os designers de Endless e do projeto GNOME.

Que é o produto de Endless?

O maior produto de Endless é um sistema operativo para mini computadores que eles fazem, o Endless Mini e o Endless (Maxi?). O sistema operativo usa Linux e uma versão de GNOME com algumas changes (mudanças). O uso principal desses computadores é de ter muitas informações sem acesso à Internet. Por exemplo, tem muitos aplicativos sobre viagens, animais e etc que são diretamente dentro do computador, usando Wikipedia como fonte, e uma outra aplicação de receitas, com uma outra terceira fonte.

A hackatona em si

Os dois primeiros dias foram para viajar e visitar os usuários “beta” do Endless computadores, um dia na Rocinha, uma favela do Rio. E um outro dia em Magé, uma cidade rural do estado do Rio.
Os três últimos dias foram para discussões no escritório de Endless.

Observações

É uma coisa para fazer testes de usabilidade nos EUA e na Europa, e uma outra coisa de fazer isso num país sem habitude de usar “computadores pessoais” com Windows o MacOS X, mas muita mais habitude com celulares.

Por exemplo:
- Se se tem um mouse, vão dar dublo clique. Não é um problema com teclados sensíveis.
- Dividir a tela para ter um aplicativo ao lado de uma outra é difícil também.
- Se não se tem um acesso à Internet, não vão tentar instalar o acessar outros aplicativos que estão já no computador.
- Não estão acostumados a fechar aplicativos que não usam mais. Um sistema operativo de celular vai fechar os aplicativos antigos de maneira transparente.

Conclusões

Muitas coisas que o Endless ou GNOME podem mudar ou melhorar.

- GNOME tem alguns vídeos para explicar o “overview”. Um jogo ou tutorial podem ser melhor para explicar e ter certeza de que os usuários entendem.
- GNOME precisa melhorar a integração de modems celulares. ModemManager tem as funções que GNOME não usa.
- “Web” precisa de integração com detecção de malware, que ele não tem agora, mas foi uma ideia para o Summer Of Code dos anos precedentes.
- GNOME pode melhorar a primeira tela de todos os aplicativos e do sistema também, especialmente se o usuário não tem Internet para baixar conteúdo.

Muito obrigado a fundação GNOME pelas minhas passagens. Obrigado ao Endless e o Allan Day pela a organizacão. Obrigado ao meu empregador Red Hat pela oportunidade. E, enfim, obrigado à Caro pela correcção!



March 07, 2016

Age Ratings in GNOME Software: Introducing OARS?

In GNOME Software we show lots of applications ranging from games aimed at pre-schoolers to applications explicitly designed to download, well, porn. A concept that is fairly well understood by parents is age ratings, and there are well known and trusted ratings bodies such as the ESRB and PEGI, as well as other country-specific schemes. Parents can use the ratings to control what kind of content is available to install, and vendors can use the ratings as a legal (or common-sense) control who gets to purchase what.

The ratings systems between countries are varied, varying from descriptions such as “M” which will be familiar for US users, “R” for Australian users to the slightly more obvious “18+” rating for European users. The differing ratings authorities define what is allowed in each category in slightly different way, some allowing mild profanity for a “7+” rating, and others none at all. Some countries consider drug taking in a video game to be no more dangerous as to mild cursing, other countries consider this on the same level as sexual violence.

OARS

So, we’re sunk, right? Nearly. There exists a group called “International Age Rating Coalition” which allows developers to register (sometimes for free), answer a simple questionnaire and out pops the ratings they should use for various countries. The IARC is made up of the regulatory bodies all over the planet, and so you can use the actual trademarked age rating images for your product. ish.

If you want to build a a software center, say GNOME Software for example, you have to pay a license fee. A $100,000 annual fee, plus extra per application shown in the software center. This is prohibitive for us, and would mean we couldn’t have the same functionality in other software center interfaces.

We could easily provide in the AppData files details about the application/game. This can be combined with a rule engine specific to the country of viewing, which would pop out a rating. I think the ESRB would be hard pushed to trademark “M” as an age rating, although I completely agree they have correctly and sensibly trademarked the stylized logo for the PG rating, along with the “ESRB” name itself. I don’t think this should stop us using an “PG” or “M” rating in the software center as long as we avoid these trademarks and copyrights.

I’m happy to work on a new system to both generate the AppData upstream information from a questionaire, and the rule engines that processes these rules and pops out a rating. The question then becomes, is this useful? Is this something that people would actually want? Comments welcome.

p.s. OARS: “Open Age Rating System”, name is work in process.

March 04, 2016

Why Wayland anyway ?

The Fedora Workstation working group decided this week that we’re not quite there yet for making the Wayland session the default in Fedora 24.

That is a bit of a disappointment for me, since we have worked very hard this cycle to close the gaps;  you can see the progress we’ve made here: primary selection, kinetic scrolling, drag-and-drop,  startup notification, pointer confinement have all landed this cycle. Not to mention countless smaller bug fixes and robustness improvements. But gaps are gaps, so we will take one more cycle to address them.

winter

In any case, the Wayland session in Fedora 24 will be the best Wayland session we’ve ever had, and I really encourage everybody to try it out when F24 released.

Chances are that you won’t be able to tell the difference between the X and Wayland sessions. That is of course intentional – we’ve put a lot of effort into making sure that things work as well or better than before. But it is also a bit of a dilemma for advertising the Wayland work.

Why do it if everything is the same in the end ?

Isolation

One reason is that Wayland is designed from the ground up to isolate clients from each other.  There is no shared coordinate space. Wayland clients cannot snoop on each others input or inject fake input events. They can’t draw on each others windows or cover up windows with fake replicas.

All of these things and many other exploits are possible for malicious X clients, because the X protocol wasn’t designed for untrusted clients.

This makes Wayland a much better choice of display protocol when sandboxing untrusted applications, like xdg-app does.

Baggage

Another reason for using Wayland is that the Wayland protocol is a much better fit for a modern, compositing-based display system.

The X protocol contains many pieces that are simply no longer relevant today, like core fonts or core rendering. Modern toolkits and applications don’t use most of the protocol, but the obsolete parts have to stay if you want to claim to implement the X11 protocol.

Some problematic parts of the X protocol, such as grabs, are simply not present under Wayland.

Extensibility

A third reason for Wayland is that it will be a good basis to enable features that are hard to support under X, such as input transformation or smooth transitions between composited desktop and fullscreen clients.

Currently, we are still playing catch-up with the features of the traditional X session, so we haven’t really done much work on such new features yet. But as a small example,  we’ve already seen that it was relatively easy to add touchpad gesture support to Wayland, while X doesn’t have them yet.

There are more reasons for Wayland beyond these three, and you can read more about them e.g.  here or here.

Try it

Again, if you haven’t tried Wayland yet, the F24 release will be a great time to do so. Simply select the Wayland session (currently just called GNOME) in the session chooser on the login screen:

login

March 01, 2016

libinput and graphics tablet support

Last week's libinput 1.2 release included the new graphics tablet support. This work, originally started as Lyude's 2014 GSoC project enables the use of drawing tablets through libinput (think Wacom tablets, not iPad/Android tablet).

Wacom tablets provide three input types: pen-like tools, buttons and touch rings/strips on the tablet itself, and touch. libinput's tablet support work focuses on the tool support only, pad buttons are still work in progress. Touch is already supported, either through the touchpad interfaces for external tablets (e.g. Intuos) or touchscreen interfaces for direct-touch tablets (e.g. Cintiq). So the below only talks about how to get events from tools, the pad events will be covered in a separate post when it's ready.

How do things work in the xf86-input-wacom driver, the current standard for tablet support in Linux? The driver checks the kernel device node for capabilities and creates multiple X devices, usually pen, eraser, cursor, and pad. When a pen comes into proximity the driver sends events through the pen device, etc. The pen device has all possible axes available but some (e.g. rotation) won't be used unless you're actually using an Wacom Art Pen. Unless specifically configured, all pens send through the same device, all erasers send through the same device, etc.

The libinput tablet API is a notable departure from this approach. In libinput, each tool is a separate entity generating events. A client doesn't wait for events from the tablet, it waits for events from a tool. The tablet itself is little more than an idle device. This has a couple of effects:

  • A struct libinput_tablet_tool is created on-the-fly as a tool comes into proximity and its this struct that events are tied to.
  • This means we default to per-tool handling. Two pens will always be separate and never multiplexed through one device. [1]
  • The tool can be uniquely identified [1]. It's easy to track a tool across two tablets even though this is quite a niche case.
  • A client can query the tool struct for capabilities, but not the tablet. Hence you cannot know what capabilities are available until a tool is in proximity.
  • The tool-based approach theoretically enables us to have multiple tools in proximity simultaneously, though no current hardware supports this.
Now, the advantages for the professional use-case where artists have multiple tools and/or multiple tablets is quite obvious. But it also changes some things for the average user with a single tool, specifically: the data provided by libinput is now precise and reflects the tool you're using. No more fake rotation axis on your standard pen. But you cannot query that information until the pen has been in proximity at least once. This is a change that clients will have to eventually deal with.

I just pushed the tablet support for the xf86-input-libinput driver and this driver reflects the new approach that libinput takes. When a tablet is detected, we create one device that has no axes and serves as the parent device. Once the first tool comes into proximity, a new device is created with the exact capabilities that the tool provides and the serial number in the name:


$> xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
[...]
⎜ ↳ Wacom Intuos5 touch M Pen Pen (0x99800b93) id=21 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
[...]
↳ Wacom Intuos5 touch M Pen id=11 [slave keyboard (3)]
Device 11 is effectively mute (in the future this may become the Pad device, not sure yet). Device 21 appeared once I moved the tool into proximity. That tool has all the properties for configuration as they appear on the device. So far this works, but it means static configuration becomes more difficult. If you are still using xinit scripts or other scripts that only run once on login and not once per new device then the options may not apply correctly.

[1] provided the hardware can uniquely identify the pens

February 28, 2016

native gtk3 menubar in libreoffice
Following quickly on native gtk3 popup context menus is implementation of native gtk3 menubar and menus.


For comparison here's the (not utterly awful) emulated look prior to this. You can compare the spacing of elements in the menubar, menu separator rendering, distance of checkmarks to the following text, the display of the short cuts in different font attributes with different positioning, and menu entry line spacing.


This is mostly achieved by reusing the export of the toplevel menubar and menu structure via GMenuModel work by Antonio Fernandez and Bjoern Michaelsen and just stuffing a GtkMenuBar into a GtkGrid in the toplevel widget and moving the LibreOffice "guts" widget a level down with no LibreOffice menubar visible.

February 25, 2016

I bought some awful light bulbs so you don't have to
I maintain an application for bridging various non-Hue lighting systems to something that looks enough like a Hue that an Amazon Echo will still control them. One thing I hadn't really worked on was colour support, so I picked up some cheap bulbs and a bridge. The kit is badged as an iSuper iRainbow001, and it's terrible.

Things seemed promising enough at first, although the bulbs were alarmingly heavy (there's a significant chunk of heatsink built into them, which seems to get a lot warmer than I'd expect from something that claims a 7W power consumption). The app was a bit clunky, but eh - I wasn't planning on using it for long. I pressed the button on the bridge, launched the app and could control the bulbs. The first thing I noticed was that they had a separate "white" and "colour" mode. White mode was pretty bright, but colour mode massively less so - presumably the white LEDs are entirely independent of the RGB ones, and much higher intensity. Still, potentially useful as mood lighting.

Anyway. Next step was to start playing with the protocol, which meant finding the device on my network. I checked anything that had picked up a DHCP lease recently and nmapped them. The OS detection reported Linux, which wasn't hugely surprising - there was no GPL notice or source code included with the box, but I'm way past the point of shock at that. It also reported that there was a telnet daemon running. I connected and got a login prompt. And then I typed admin as the username and admin as the password and got a root prompt. So, there's that. The copy of Busybox included even came with tftp, so it was easy to get copies of tcpdump and strace on there to see what was up.

So. Next step. Protocol sniffing. I wanted to know how discovery worked, so reset the device to factory and watched what happens. The app on my phone sent out a discovery packet on UDP port 18602 which looked like this:

INLAN:CLIP:23.21.212.48:CLPT:22345:MAC:02:00:00:00:00:00

The CLIP and CLPT fields refer to the cloud server that allows for management when you're not on the local network. The mac field contains an utterly fake address. If you send out a discovery packet and your mac hasn't been registered with the device, you get a denial back. If your mac has been (and by "mac" here I mean "utterly fake mac that's the same for all devices"), you get back a response including the device serial number and IP address. And if you just leave out the mac field entirely, you get back a response no matter whether your address is registered or not. So, that's a start. Once you've registered one copy of the app with the device, anything can communicate with it by just using the same fake mac in the discovery packets.

Onwards. The command format turns out to be simple. They start ##, are followed by two ascii digits encoding a command, four ascii digits containing a bulb id, two ascii digits containing the number of following bytes and then the command data (in ascii). An example is:

##05010002ff

which decodes as command 5 (set white intensity) on bulb 1 with two bytes of data following, each of which is an f. This translates as "Set bulb 1 to full white intensity". I worked out the rest pretty quickly - command 03 sets the RGB colour of the bulb, 0A asks the bridge to search for new bulbs, 0B tells you which bulbs are available and their capabilities and 0E gives you the MAC addresses of the bulbs(‽). 0C crashes the server process, and 06 spews a bunch of garbage back at you and potentially crashes the bulb in a hilarious way that involves it flickering at about 15Hz. It turns out that 06 is actually the "Rename bulb" command, and if you send it less data than it's expecting something goes hilariously wrong in string parsing and everything is awful.

Ok. Easy enough, but not hugely exciting. What about the remote protocol? This turns out to involve sending a login packet and then a wrapped command packet. The login has some length data, a header of "MQIsdp", a long bunch of ascii-encoded hex, a username and a password.

The username is w13733 and the password is gbigbi01. These are hardcoded in the app. The ascii-encoded hex can be replaced with 0s and the login succeeds anyway.

Ok. What about the wrapping on the command? The login never indicated which device we wanted to talk to, so presumably there's some sort of protection going on here oh wait. The command packet is a length, the serial number of the bridge and then a normal command packet. As long as you know the serial number of the device (which it tells you in response to a discovery packet, even if you're unauthenticated), you can use the cloud service to send arbitrary commands to the device (including the one that crashes the service). One of which involves the device then doing some kind of string parsing that doesn't appear to work that well. What could possibly go wrong?

Ok, so that seemed to be the entirety of the network protocol. Anything else to do? Some poking around on the bridge revealed (a) that it had an active wireless device and (b) a running DHCP server. They wouldn't, would they?

Yes. Yes, they would.

The devices all have a hardcoded SSID of "iRainbow", although they don't broadcast it. There's no security - anybody can associate. It'll then hand out an IP address. It's running telnetd on that interface as well. You can bounce through there to the owner's internal network.

So, in summary: it's a device that infringes my copyright, gives you root access in response to trivial credentials, has access control that depends entirely on nobody ever looking at the packets, is sufficiently poorly implemented that you can crash both it and the bulbs, has a cloud access protocol that has no security whatsoever and also acts as an easy mechanism for people to circumvent your network security. This may be the single worst device I've ever bought.

I called the manufacturer and they insisted that the device was designed in 2012, was no longer manufactured or supported, that they had no source code to give me and there would be no security fixes. The vendor wants me to pay for shipping back to them and reserves the right to deduct the cost of the original "free" shipping from the refund. Everything is awful, which is why I just ordered four more random bulbs to play with over the weekend.

comment count unavailable comments

February 24, 2016

current LibreOffice native gtk3 elements
LibreOffice typically basically has just one gtk widget per top level window and draws everything you see itself, using the gtk themeing apis to make what it draws look like they do in gtk.

But there are some truly native gtk elements. Some of them new.

 

Tooltips


Tooltips, like the above transparent example, are real gtk tooltips now. Here's what they used to look like

 

Popovers

We recently in master began using GtkPopovers for informational messages such as the above calc formula formatting help.

 

Context Menus

As of today, context menus are native Gtk menus. This has the advantage of making managing overlarge menu scrolling all a gtk issue instead of faffing around endless in vcl to emulate what a native gtk menu would do. Flipping on/off mnemonics depending on keyboard input or whether the menu was launched by keyboard instead of mouse similarly becomes "someone elses problem". Here's the same menu launched via the menu key, automatically with mnemonics on start.
The menubar and menus launched from that menubar are currently still emulated ones, but they're the next target for nativization. We have mechanisms in place for the MacOSX and Unity menu integration that can probably be adapted and extended to support driving a native GtkMenuBar hosted in the GtkFixed-alike toplevel widget that fills each toplevel window.



February 22, 2016

Freedom, the US Government, and why Apple are still bad
The US Government is attempting to force Apple to build a signed image that can be flashed onto an iPhone used by one of the San Bernardino shooters. To their credit, Apple have pushed back against this - there's an explanation of why doing so would be dangerous here. But what's noteworthy is that Apple are arguing that they shouldn't do this, not that they can't do this - Apple (and many other phone manufacturers) have designed their phones such that they can replace the firmware with anything they want.

In order to prevent unauthorised firmware being installed on a device, Apple (and most other vendors) verify that any firmware updates are signed with a trusted key. The FBI don't have access to Apple's firmware signing keys, and as a result they're unable to simply replace the software themselves. That's why they're asking Apple to build a new firmware image, sign it with their private key and provide it to the FBI.

But what do we mean by "unauthorised firmware"? In this case, it's "not authorised by Apple" - Apple can sign whatever they want, and your iPhone will happily accept that update. As owner of the device, there's no way for you to reconfigure it such that it will accept your updates. And, perhaps worse, there's no way to reconfigure it such that it will reject Apple's.

I've previously written about how it's possible to reconfigure a subset of Android devices so that they trust your images and nobody else's. Any attempt to update the phone using the Google-provided image will fail - instead, they must be re-signed using the keys that were installed in the device. No matter what legal mechanisms were used against them, Google would be unable to produce a signed firmware image that could be installed on the device without your consent. The mechanism I proposed is complicated and annoying, but this could be integrated into the standard vendor update process such that you simply type a password to unlock a key for re-signing.

Why's this important? Sure, in this case the government is attempting to obtain the contents of a phone that belonged to an actual terrorist. But not all cases governments bring will be as legitimate, and not all manufacturers are Apple. Governments will request that manufacturers build new firmware that allows them to monitor the behaviour of activists. They'll attempt to obtain signing keys and use them directly to build backdoors that let them obtain messages sent to journalists. They'll be able to reflash phones to plant evidence to discredit opposition politicians.

We can't rely on Apple to fight every case - if it becomes politically or financially expedient for them to do so, they may well change their policy. And we can't rely on the US government only seeking to obtain this kind of backdoor in clear-cut cases - there's a risk that these techniques will be used against innocent people. The only way for Apple (and all other phone manufacturers) to protect users is to allow users to remove Apple's validation keys and substitute their own. If Apple genuinely value user privacy over Apple's control of a device, it shouldn't be a difficult decision to make.

comment count unavailable comments

February 19, 2016

Die dbus-glib, die!

no-dbus-glib

dbus-glib debuted in 2002 and was the first usable D-Bus client library for GLib-based applications.  NetworkManager used it since the earliest commits in mid-2004.

Why do you care?

Apps and services built on it and enabled the modern desktop.  If you’ve ever tried to code directly to libdbus to add a D-Bus service or talk to one you know what a pain that is.  dbus-glib made that process infinitely easier but had a ton of quirks that didn’t fit well with an evolving GLib. But in the past 5 years the GLib authors recognized that D-Bus needed first-class support and they wrote GDBus with much better GLib integration and a modern asynchronous API.  dbus-glib has been on life-support for years.  It was time to move on.

The upcoming NetworkManager 1.2 release will finally remove use of dbus-glib for the daemon, command-line/TUI utilities, nm-applet/editor, and VPN plugins if you build with legacy components disabled.  Almost nobody noticed this in-flight rebuild of the airplane and that’s exactly how we like it.  I cannot understate how much work this was and how much careful planning we (well, mostly Dan Winship) did to ensure we didn’t break backwards compatibility of either the utility libraries or the D-Bus interface.

To be fair, this is a slight lie; we can’t maintain C API/ABI compatibility in the legacy client libraries by removing dbus-glib and we cannot drop them yet either.  But we did write the new libnm which much more closely matches GLib coding practices and API, including GCancellables and asynchronous functions.  Its API is close enough to the old libnm-util and libnm-glib that porting to libnm is not a huge task, and porting will actually remove code from your application too.  Seriously: the libnm port for nm-connection-editor was +772/-1203 lines of code and nm-applet was +697/-990.  Killing code feels great.

We’ve already ported over all the NetworkManager tools like nmcli, nmtui, nm-applet, nm-connection-editor, and all the VPN plugins.  In the process, we cleaned up a ton of code, created better client APIs, wrote more testcases, reduced the dependency chain, and paid down most of our technical debt.  When kdbus (or bus1 or whatever it’s called) eventually hits, NetworkManager will be ready.

The future is dbus-glib-free.

February 18, 2016

Canonical, Ubuntu and why I seem so upset about them all the time
I had no access to the internet for most of my childhood. Nobody in my area knew anything about programming. I learned a great deal from a number of CDs that included free software source code and archives of project mailing lists. When I got to university, I learned even more by being able to develop a Debian-based OS for use in our computer facilities. That gave me the experience and knowledge that I needed to become involved in Debian development, which in turn gave me the background required to be able to help Ubuntu become the first free software operating system to work out of the box on modern laptops. From there, I've been able to build my career around developing free software.

Ubuntu can be translated as "I am who I am because of who we all are". I am who I am because people made the choice to release their software under licenses that permitted examination, modification and redistribution. I am who I am because I was able to participate in communities that took advantages of those freedoms to produce new and better software. I am who I am because when my priorities differed from those of existing communities, it was still possible for me to benefit from their work and for them to benefit from mine.

Free software doesn't mean that the software is entirely free of restrictions. While a core aspect is the right to distribute modified versions of code, it has never been fundamental to free software that you be able to do so while still claiming that the code is the original version. Various approaches have been taken to make it possible for users to distinguish modified versions, ranging from simply including license terms that require modified versions be marked as such, to licenses that require that you change the name of the package if you modify it. However, what's probably the most effective approach has been to apply trademark law to the problem. Mozilla's trademark policy is an example of this - if you modify the code in ways that aren't approved by Mozilla, you aren't entitled to use the trademarks.

A requirement that you avoid use of trademarks in an infringing way is reasonable. Mozilla products include support for building with branding disabled, which makes it very straightforward for a user to build a modified version of Firefox that can be redistributed without any trademark issues. Red Hat have a similar policy for Fedora and RHEL[1] - you simply replace the packages that contain the branding and you're done.

Canonical's IP policy around Ubuntu is fundamentally different. While Mozilla make it clear that you simply no longer have a right to use the trademarks under trademark law, Canonical appear to require that you remove all trademarks entirely even if using them wouldn't be a violation of trademark law. While Mozilla restrict the redistribution of modified binaries that include their trademarks, Canonical insist that you rebuild everything even if the package doesn't contain any trademarks. And while Mozilla give you a single build option that creates binaries that conform with their trademark requirements, Canonical will refuse to tell you what you have to do.

When asked about this at SCALE earlier this year, Mark Shuttleworth claimed that Ubuntu's policy was consistent with that of other projects. This is inaccurate. Nobody else requires that you rebuild every package before you can redistribute it in a modified distribution - such a restriction is a violation of freedom 2 of the Free Software Definition, and as a result the binary distributions of Ubuntu are not free software. Nobody else refuses to discuss whether you're required to remove non-infringing trademarks in order to be able to redistribute. Nobody else responds to offers to make it easier for users to produce non-infringing derivatives with a flat refusal.

Mark claims that I'm only raising this issue because I work for a competitor and wish to harm Canonical. Nothing could be further from the truth. I began discussing this before working for my current employers - my previous employers had no meaningful market overlap with Canonical at all. The reason I care is because I care about free software. I care about people being able to derive new and interesting things from existing code. I care about a small team of people being able to take Ubuntu and make something better in the same way that Ubuntu did with Debian. I care about ensuring that users receive the freedom to do this without having to jump through a significant number of hoops in the process. Ubuntu has been a spectacularly successful vehicle for getting free software into the hands of users. Mark's generosity in funding this experiment has undoubtedly made the world a better place. Canonical employs a large number of talented developers writing high quality software, many of whom I'm fortunate enough to be able to call friends. And Canonical are squandering that by restricting the rights of their users and alienating the free software community.

I want others to be who they are because of my work and the work of all the others like me. Anything that makes that more difficult saddens me, and so I do what I can to fix it. I criticise Canonical's policies in the hope that we, as a community, can convince Canonical to agree that this kind of artificial barrier to modification hurts us more than it helps them. In many ways, Canonical remain one of our best hopes for broadening the reach of free software, and this is why it's unfortunate that they do so in a way that makes it more difficult for people to have the same experiences that I did.

[1] While it's easy to turn a trademark infringing version of RHEL into a non-infringing one, Red Hat don't provide publicly available binary packages for RHEL. If you get hold of them somehow you're entitled to redistribute them freely, but Red Hat's subscriber agreement indicates that if you do this as a Red Hat customer you will lose access to further binary updates - a provision that I find utterly repugnant. Its inclusion reduces my respect for Red Hat and my enthusiasm for working with them, and given the official Red Hat support for CentOS it appears to make no sense whatsoever. Red Hat should drop it.

comment count unavailable comments

February 16, 2016

Endless and GNOME

Hello! Long time, no post.

Endless + GNOME = Love

Credit to Georges Stavracas

As you may have heard, Endless joined the GNOME Foundation Advisory Board last week.  We appreciate all the kind words of welcome we have received and are looking forward to strengthening our ties with this community. This has been a coming for a bit, and I’m looking forward for us to contribute more over the coming year!

On a personal note, this is really exciting to me. It’s been a couple years since I’ve had the pleasure of being on the Advisory Board, and I’m looking forward to working with the Board again. I’ve wanted to bring the two organizations closer for a while, and we got a glimpse of what was possible at the Design Hackfest in Rio last month.

Who we are and what we do:

Quite simply, we are trying to empower the world! Our mission is to make computing universally accessible, and to solve the barriers of cost and connectivity to the entire world. It’s a bold and ambitious goal, but we are absolutely serious about accomplishing this. There are billions of people who don’t have access to a computer, and would love to have one.

What’s that mean in practical terms? It means we are making a great computer that works at as low a price as we can make. It means constantly keeping an eye on cost while still providing a great value and experience to our users. It means that we want to provide a great OS and fantastic desktop to our users.

20160204140930

Wiring in Rocinha.

And it means that we have to think about a lot of things that many of us don’t think of: Things like connectivity, cost, and robustness are paramount in that environment, and that drives everything we do.

Endless is Hiring!

We can’t do this alone. We are looking for some great engineers to join our team. If this mission sounds great and you’re interested in working with us, let us know! We are looking for people who are passionate about bringing a great desktop to the rest of the world while developing some high-quality Free Software. We have a number of openings available:

  • Application Software Developer: Part of providing a great computer experience is to provide offline applications and content to users.
  • Cloud / Distributed Systems Architect: Help build the service that powers our offline content.
  • Kernel Engineer: We want to run on as many different types of cheap devices as possible. This job will help bring us to laptops, desktops, and all sorts of crazy ARM devices.
  • Internal Tooling Developer: We need someone to help maintain and build our internal build tools and create some new ones.
  • Software Engineer: Don’t see the job you’re looking for? We’re always looking for fantastic engineers of all types!

Feel free to mail me or ping me on IRC if you have any questions!

gtk3 and LibreOffice
Couple of changes to the gtk3 support in LibreOffice master recently.

Theming stuff

The gtk3 support in master LibreOffice is now updated to support the relatively recent changes in gtk 3.19's style mechanism.

With mclasen's help I've added simple demos to the foreign-drawing demo of gtk-demo for the different elements that we render via those rendering apis. So what we attempt to do in LibreOffice is basically documented in gtk-demo now. So from 3.20 onwards if there are styling flaws in LibreOffice, then gtk-demo can be checked if it has the same problem or not.

Drag and Drop

Drag and Drop is now implemented in the gtk3 backend. DnD here includes moving blocks of cells around a calc spreadsheet using the mouse alone.

Wayland gstreamer video playback

If the gtksink gstreamer element is present (e.g. gstreamer >= 1.7.1 as in Fedora 24) then the gtksink gtk widget integration will be used to play video. Which means support for video playback in LibreOffice under wayland.

February 15, 2016

virglrenderer project gets mailing-list and hosting
Just FYI:

There is now a mailing list for virglrenderer library hosted at freedesktop. It is to be used for development discussion of the virgl->GL renderer library and for patches to it.

https://lists.freedesktop.org/mailman/listinfo/virglrenderer-devel

The git tree is also now hosted at:
git://anongit.freedesktop.org/git/virglrenderer
https://cgit.freedesktop.org/virglrenderer

My personal repo will only be used for my own development stuff.

I'll also get a freedesktop patchwork instance set up for this asap.

I'm also contemplating bugzilla vs phabricator.

Dave.

February 11, 2016

Moderate reviews in GNOME Software

I’m pondering adding something like this for GNOME Software:

Screenshot from 2016-02-11 20-31-28

The idea is it would be launched using a special geeky-user-only gnome-software --mode=moderate command line, so that someone can up or down-vote any reviews that are compatible with their country code. This would be used by super-users using community distros like Fedora, and perhaps people in a QA team for distros like RHEL. The reason I wanted to try to use gnome-software rather than just doing it in a web-app was that you have access to the user machine hash, so you can hide reviews the user has already voted on without requiring a username and password that has to be matched to the local machine. We also have all the widgets and code in place, so it is really just a couple of hundred lines of code for this sekret panel. The server keeps track of who voted on what and so reviewers can just close the app, open it a few weeks later and just continue only moderating the reviews that came in since then.

I can’t imagine the tool would be used by many people, but it does make reviewing comments easy. Comments welcome.

February 10, 2016

Comments are live

With a huge amount of help from Robert Ancell for a lot of the foundations for the new code, I’ve pushed a plugin today to allow anonymous rating of applications.

Screenshot from 2016-02-10 17-16-04

If people abuse or spam this I’ll take the feature away until we can have OpenID logins in GNOME Online Accounts, but I’m kinda hoping people won’t be evil. The server is live and accepting reviews and votes, but the API isn’t set in stone.

February 08, 2016

Anonymous reviews in GNOME Software

Choosing an application to install is hard when there are lots of possible projects matching a specific search term. We already list applications based on the integration level and with useful metrics like “is it translated in my language” and this makes sure that high quality applications are listed near the top of the results. For more information about an application we often want a more balanced view than the PR speak or unfounded claims of the upstream project. This is where user-contributed reviews come in.

review-submit

To get a user to contribute a review (which takes time) we need to make the process as easy as possible. Making the user create a user account on yet-another-webservice will make this much harder and increase the barrier to participation to the point that very few people would contribute reviews. If anonymous reviewing does not work the plan is to use some kind of attestation service so you can use a GMail or Facebook for confirming your identity. At this point I’m hoping people will just be nice to each other and not abuse the service although this reviewing facility will go away if it starts being misused.

Designing an anonymous service is hard when you have to be resilient against a socially awkward programmer with specific political ideologies. If you don’t know any people that match this description you have obviously never been subscribed to fedora-devel or memo-list.

Obviously when contacting a web service you share your IP address. This isn’t enough to uniquely identify a machine and user, which we want for the following reasons:

  • Allowing users to retract only their own reviews
  • Stopping users up or down-voting the same review multiple times

A compromise would be to send a hash of two things that identify the user and machine. In GNOME Software we’re using a SHA1 hash of the machine-id and the UNIX username along with a salt, although this “user_id” is only specified as a string and the format is not checked.

For projects like RHEL where we care very much what comments are shown to paying customers we definitely want reviews to be pre-approved and checked before showing to customers. For distros like Fedora we don’t have this luxury and so we’re going to rely on the community to self-regulate reviews. Reviews are either up-voted or down-voted according how useful they are along with the nuclear option of marking the review as abusive.

app-page

By specifying the users current locale we can sort the potential application reviews according to a heuristic that we’re still working on. Generally we want to prefer useful reviews in the users locale and hide ones that have been marked as abusive, and we also want to indicate the users self-review so they can remove it later if required. We also want to prioritize reviews for the current application version compared to really old versions of these applications.

Comments welcome!