February 11, 2016

Moderate reviews in GNOME Software

I’m pondering adding something like this for GNOME Software:

Screenshot from 2016-02-11 20-31-28

The idea is it would be launched using a special geeky-user-only gnome-software --mode=moderate command line, so that someone can up or down-vote any reviews that are compatible with their country code. This would be used by super-users using community distros like Fedora, and perhaps people in a QA team for distros like RHEL. The reason I wanted to try to use gnome-software rather than just doing it in a web-app was that you have access to the user machine hash, so you can hide reviews the user has already voted on without requiring a username and password that has to be matched to the local machine. We also have all the widgets and code in place, so it is really just a couple of hundred lines of code for this sekret panel. The server keeps track of who voted on what and so reviewers can just close the app, open it a few weeks later and just continue only moderating the reviews that came in since then.

I can’t imagine the tool would be used by many people, but it does make reviewing comments easy. Comments welcome.

February 10, 2016

Comments are live

With a huge amount of help from Robert Ancell for a lot of the foundations for the new code, I’ve pushed a plugin today to allow anonymous rating of applications.

Screenshot from 2016-02-10 17-16-04

If people abuse or spam this I’ll take the feature away until we can have OpenID logins in GNOME Online Accounts, but I’m kinda hoping people won’t be evil. The server is live and accepting reviews and votes, but the API isn’t set in stone.

February 08, 2016

Anonymous reviews in GNOME Software

Choosing an application to install is hard when there are lots of possible projects matching a specific search term. We already list applications based on the integration level and with useful metrics like “is it translated in my language” and this makes sure that high quality applications are listed near the top of the results. For more information about an application we often want a more balanced view than the PR speak or unfounded claims of the upstream project. This is where user-contributed reviews come in.

review-submit

To get a user to contribute a review (which takes time) we need to make the process as easy as possible. Making the user create a user account on yet-another-webservice will make this much harder and increase the barrier to participation to the point that very few people would contribute reviews. If anonymous reviewing does not work the plan is to use some kind of attestation service so you can use a GMail or Facebook for confirming your identity. At this point I’m hoping people will just be nice to each other and not abuse the service although this reviewing facility will go away if it starts being misused.

Designing an anonymous service is hard when you have to be resilient against a socially awkward programmer with specific political ideologies. If you don’t know any people that match this description you have obviously never been subscribed to fedora-devel or memo-list.

Obviously when contacting a web service you share your IP address. This isn’t enough to uniquely identify a machine and user, which we want for the following reasons:

  • Allowing users to retract only their own reviews
  • Stopping users up or down-voting the same review multiple times

A compromise would be to send a hash of two things that identify the user and machine. In GNOME Software we’re using a SHA1 hash of the machine-id and the UNIX username along with a salt, although this “user_id” is only specified as a string and the format is not checked.

For projects like RHEL where we care very much what comments are shown to paying customers we definitely want reviews to be pre-approved and checked before showing to customers. For distros like Fedora we don’t have this luxury and so we’re going to rely on the community to self-regulate reviews. Reviews are either up-voted or down-voted according how useful they are along with the nuclear option of marking the review as abusive.

app-page

By specifying the users current locale we can sort the potential application reviews according to a heuristic that we’re still working on. Generally we want to prefer useful reviews in the users locale and hide ones that have been marked as abusive, and we also want to indicate the users self-review so they can remove it later if required. We also want to prioritize reviews for the current application version compared to really old versions of these applications.

Comments welcome!

February 05, 2016

programs won’t start

So recently I got pointed to an aging blocker bug that needed attention, since it negatively affected some rawhide users: they weren’t able to launch certain applications. Three known broken applications were gnome-terminal, nautilus, and gedit. Other applications worked, and even these 3 applications worked in wayland, but not Xorg. The applications failed with messages like:

Gtk-WARNING **: cannot open display:

and

org.gnome.Terminal[2246]: Failed to parse arguments: Cannot open display:

left in the log. These messages means that the programs are unable to create a connection to the X server. There are only a few reasons this error message could get displayed:

    — The socket associated with the X server has become unavailable. In the old days this could happen if, for instance, the socket file got deleted from /tmp. Adam Jackson fixed the X server a number of years ago, to also listen on abstract sockets to avoid that problem. This could also happen if SELinux was blocking access to the socket, but users reported seeing the problem even with SELinux put in permissive mode.
    — The X server isn’t running. In our case, clearly the X server is running since the user can see their desktop and launch other programs
    — The X server doesn’t allow the user to connect because that user wasn’t given access, or that user isn’t providing credentials. These programs are getting run as the same user who started the session, so that user definitely has access.
    — GDM doesn’t require users to provide separate credentials to use the X server, so that’s not it either.
    — $DISPLAY isn’t set, so the client doesn’t know which X server to connect to. This is the only likely cause of the problem. Somehow $DISPLAY isn’t getting put in the environment of these programs.

So the next question is, what makes these applications “special”? Why isn’t $DISPLAY set for them, but other applications work fine? Every application has a .desktop file associated with it, which is a small config file giving information about the application (name, icon, how to run it, etc). When a program is run by gnome-shell, gnome-shell uses the desktop file of that program to figure out how to run it. Most of the malfunctioning programs have this in their desktop files:


DBusActivatable=true

That means that the shell shouldn’t try to run the program directly, instead it should ask the dbus-daemon to run the program on the shell’s behalf. Incidentally, the dbus-daemon then asks systemd to run the program on the dbus-daemon’s behalf. That has lots of nice advantages, like automatically integrating program output to the journal, and putting each service in its own cgroup for resource management. More and more programs are becoming dbus activatable because it’s an important step toward integrating systemd’s session management features into the desktop (though we’re not fully there yet, that initiative should become a priority at some point in the near-to-mid future). So clearly the issue is that the dbus-daemon doesn’t have $DISPLAY in its activation environment, and so programs that rely on D-Bus activation aren’t able to open a display connection to the X server. But why?

When a user logs in, GDM will start a dbus-daemon for that user before it starts the user session. It explicitly makes sure that DISPLAY is in the environment when it starts the dbus-daemon so things should be square. They’re obviously not, though, so I decided to try to reproduce the problem. I turned off my wayland session and instead started up an Xorg (actually I used a livecd since I knew for sure the livecd could reproduce the problem) and then looked at a process listing for the dbus-daemon:


/usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation

This wasn’t run by GDM ! GDM uses different command line arguments that these when it starts the dbus-daemon. Okay, so if it wasn’t getting started by GDM it had to be getting started by the systemd during the PAM conversation right before GDM starts the session. I knew this, because there isn’t really thing other than systemd that runs after the user hits enter at the login screen before gdm starts the user’s session. Also, the command line arguments above in the dbus-daemon instance say ‘–systemd-activation’ which is pretty telling. Furthermore, if a dbus-daemon is already running GDM will avoid starting a second one, so this all adds up. I was surprised that we were using the so called “user bus” instead of session bus already in rawhide. But, indeed, running


$ systemctl --user status dbus.service
● dbus.service - D-Bus User Message Bus
Loaded: loaded (/usr/lib/systemd/user/dbus.service; static; vendor preset: enabled)
Active: active (running) since Tue 2016-02-02 15:04:41 EST; 2 days ago

show’s we’re clearly starting the dbus-daemon before GDM starts the session. Of course, this poses the problem. The dbus-daemon can’t possibly have DISPLAY set in its environment if it’s started before the X server is started. Even if it “wanted” to set DISPLAY it couldn’t even know what value to use, since there’s no X server running yet to tell us the DISPLAY !

So what’s the solution? Many years ago I added a feature to D-Bus to allow a client to change the environment of future programs started by the dbus-daemon. This D-Bus method call, UpdateActivationEnvironment, takes a list of key-value pairs that are just environment variables which get put in the environment of programs before they’re activated. So, the fix is simple, GDM just needs to update the bus activation environment to include DISPLAY as soon as it has a DISPLAY to include.

Special thanks to Sebastian Keller who who figured out the problem before I got around to investigating the issue.

February 01, 2016

leaking buffers in wayland

So in my last blog post I mentioned Matthias was getting SIGBUS when using wayland for a while. You may remember that I guessed the problem was that his /tmp was filling up, and so I produced a patch to stop using /tmp and use memfd_create instead. This resolved the SIGBUS problem for him, but there was something gnawing at me: why was his /tmp filling up? I know gnome-terminal stores its unlimited scrollback buffer in an unlinked file in /tmp so that was one theory. I also have seen, in some cases firefox downloading files to /tmp. Neither explanation sat well with me. scrollback buffers don’t get that large very quickly and Matthias was seeing the problem several times a day. I also doubted he was downloading large files in firefox several times a day. Nonetheless, I shrugged, and moved on to other things…

…until Thursday. Kevin Fenzi mentioned on IRC that he was experiencing a 12GB leak in gnome-shell. That piqued my interest and seemed pretty serious, so I started to troubleshoot with him. My first question was “Are you using the proprietary nvidia driver?”. I asked this because I know the nvidia driver has in the past had issues with leaking memory and gnome-shell. When Kevin responded that he was on intel hardware I then asked him to post the output of /proc/$(pidof gnome-shell)/maps so we could see the make up of the lost memory. Was it the heap? or some other memory mapped regions? To my surprise it was the memfd_create’d shared memory segments from my last post ! So window pixel data was getting leaked. This explains why /tmp was getting filled up for Matthias before, too. Previously, the shared memory segments resided in /tmp after all, so it wouldn’t have taken long for them to use up /tmp.

Of course, the compositor doesn’t create the leaked segments, the clients do, and then those clients share them with the compositor. So we probed a little deeper and found the origin of the leaking segments; they were coming from gnome-terminal. My next thought was to try to reproduce. After a few minutes I found out that typing:


$ while true; do echo; done

into my terminal and then switching focus to and from the terminal window made it leak a segment every time focus changed. So I had a reproducer and just needed to spend some time to debug it. Unfortunately, it was the end of the day and I had to get my daughter from daycare, so I shelved it for the evening. I did notice before I left, though, one oddity in the gtk+ wayland code: it was calling a function named _gdk_wayland_shm_surface_set_busy that contained a call to cairo_surface_reference. You would expect a function called set_something to be idempotent. That is to say, if you call it multiple times it shouldn’t add a new reference to a cairo surface each time. Could it be the surface was getting set “busy” when it was already set busy, causing it to leak a reference to the cairo surface associated with the shared memory, keeping it from getting cleaned up later?

I found out the next day, that indeed, was the case. That’s when I came up with a patch to make sure we never call set_busy when the surface was already busy. Sure enough, it fixed the leak. I wasn’t fully confident in it, though. I didn’t have a full big picture understanding of the whole workflow between compositor and gtk+, and it wasn’t clear to me if set_busy was supposed to ever get called when the surface was busy. I got in contact with the original author of the code, Jasper St. Pierre, to get his take. He thought the patch was okay (modulo some small style changes), but also said that part of the existing code needed to be redone.

The point of the busy flag was to mark a shared memory region as currently being read by the compositor. If the buffer was busy, then the gtk+ couldn’t draw to it without risking stepping on the compositors toes. If gtk+ needed to draw to a busy surface, it instead allocated a temporary buffer to do the drawing and then composited that temporary buffer back to the shared buffer at a later time. The problem was, as written, the “later time” wasn’t necessarily when the shared buffer was available again. The temporary buffer was created right before the toolkit staged some pixel updates, and copied back to the shared buffer after the toolkit was done with that one draw operation. The temporary buffer was scoped to the drawing operation, but the shared buffer wouldn’t be available for new contents until the next frame event some milliseconds later.

So my plan, after conferring with Matthias, was to change the code to not rely on getting the shared buffer back. We’d allocate a “staging” buffer, do all draw operations to it, hand it off to the compositor when we’re done doing updates and forget about it. If we needed to do new drawing we’d allocate a new staging buffer, and so on. One downside of this approach is the new staging buffer has to be initialized with the contents of the previously handed off buffer. This is because, the next drawing operation may only update a small part of the window (say to blink a cursor), and we need the rest of the window to properly get drawn in that. This read back operation isn’t ideal, since it means copying around megabytes of pixel data. Thankfully, the wayland protocol has a mechanism in place to avoid the costly copy in most cases:


→ If a client receives a release event before the frame callback
→ requested in the same wl_surface.commit that attaches this
→ wl_buffer to a surface, then the client is immediately free to
→ re-use the buffer and its backing storage, and does not need a
→ second buffer for the next surface content update.

So that’s our out. If we get a release event on the buffer before the next frame event, the compositor is giving us the buffer back and we can reuse it as the next staging buffer directly. We would only need to allocate a new staging buffer if the compositor was tardy in returning the buffer to us. Alright, I had plan and hammered out a patch on friday. It didn’t leak, and from playing with the machine for while, everything seemed to function, but there was one hiccup: i set a breakpoint in gdb to see if the buffer release event was coming in and it wasn’t. That meant we were always doing the expensive copy operation. Again, I had to go, so I posted the patch to bugzilla and didn’t look at it again until the weekend. That’s when I discovered mutter wasn’t sending the release event for one buffer until it got replaced by another. I fixed mutter to send the release event as soon as it uploaded the pixel data to the gpu and then everything started working great, so I posted the finalized version of the gtk+ patch with a proper commit message, etc.

There’s still some optimization that could be done for compositors that don’t handle early buffer release. Rather than initializing the staging buffer using cairo, we could get away with doing a lone memcpy() call. We know the buffer is linear and each row is right next to the previous in memory, so memcpy might be faster than going through all the cairo/pixman machinery. Alternatively, rather than initializing the staging buffer up front with the contents of the old buffer, we could wait until drawing is complete, and then only draw the parts of the buffer that haven’t been overwritten. Hard to say what the right way to go is without profiling, but both weston on gl and mutter support the early release feature now, so maybe not worth spending too much time on anyway.

January 25, 2016

AppData and the gettext domain

When users are searching for software in GNOME Software it is very important to answer the the question “Is this localized in my language?” If you can only speak Swedish then an application talking just in American English is not much use at all. The way we calculate this in the AppStream builder is to look at the compiled .mo files, breaking them apart and then using statistics to work out what locales are included.

When we’re processing distro packages we usually extract them one at a time. We first try for a gettext domain (the .mo file name) of the distro package name, and if that’s not found then we just try and find the first .mo file in any of the locale directories. This works about 70% of the time (which is good) but fails about 30% of the time (which is bad). For xdg-app we build the application in a special prefix, along with any dependent libraries. We don’t have a distro package name for the bundle (only the application ID) and so the “first .mo file we can find” heuristic fails more often that it works. We clearly need some more information about the gettext domain from the upstream project.

AppData to the rescue. By adding this in the AppData file informs the AppStream generation code in the xdg-app builder what gettext domain to use for an application. To use this you just need to add:

  <translation type="gettext">the_gettext_domain_here</translation>

under the <component> tag. The gettext domain is normally set in the configure.ac file with the GETTEXT_PACKAGE define. If you don’t have this extra data in your application then appstream-util validate is soon going to fail, and your application isn’t going to get the language metadata and so will be lower in the search results for users using GNOME Software in a non-C locale. If your GNOME application is available in jhbuild the good news is that I’ve automatically added the <translation> tag to 104 projects semi-automatically today. For XFCE and KDE I’m going to be sending emails to the development mailing lists tomorrow. For all other applications I’m going to be using the <update_contact> email address set in the AppData file for another mass-emailing.

Although it seems I’m asking people to do more things again and again I can assure you that slowly we’re putting the foundations in place for an awesome software installer experience. Just today I merged in the xdg-app branch into gnome-software and so I’m hoping to have a per-user xdg-app preview available in Fedora 24. Exciting times. :)

libinput and semi-mt touchpads

libinput 1.1.5 has a change in how we deal with semi-mt touchpads, in particular: interpretation of touch points will cease and we will rely on the single touch position and the BTN_TOOL_* flags instead to detect multi-finger interaction. For most of you this will have little effect, even if you have a semi-mt touchpad. As a reminder: semi-mt touchpads are those that can detect the bounding box of two-finger interactions but cannot identify which finger is which. This provides some ambiguity, a pair of touch points at x1/y1 and x2/y2 could be a physical pair of touches at x1/y2 and x2/y1. More importantly, we found issues with semi-mt touchpads that go beyond the ambiguity and reduce the usability of the touchpoints.

Some devices have an extremely low resolution when two-fingers are down (see Bug 91135), the data is little better than garbage. We have had 2-finger scrolling disabled on these touchpads since before libinput 1.0. More recently, Bug 93583 showed that some semi-mt touchpads do not assign the finger positions for some fingers, especially when three fingers are down. This results in touches defaulting to position 0/0 which triggers palm detection or results in scroll jumps, neither of which are helpful. Other semi-mt touchpads assign a straightforward 0/0 as position data and don't update until several events later (see Red Hat Bug 1295073). libinput is not particularly suited to handle this, and even if it did, the touchpad's reaction to a three-finger tap would be noticeably delayed.

In light of these problems, and since these affect all three big semi-mt touchpad manufacturers we decided to drop back and handle semi-mt touchpads as single-finger touchpads with extra finger capability. This means we track only one touchpoint but detect two- and three-finger interactions. Two-finger scrolling is still possible and so is two- and three-finger tapping or the clickfinger behaviour. What isn't possible anymore are pinch gestures and some of the built-in palm detection is deactivated. As mentioned above, this is unlikely to affect you too much, but if you're wondering why gestures don't work on your semi-mt device: the data is garbage.

Is Wayland ready yet?

This question turns up a lot, on the irc channel, mailing lists, forums, your local Stammtisch and at weddings. The correct answer is: this is the wrong question. And I'll explain why in this post. Note that I'll be skipping over a couple of technical bits, if you notice those then you're probably not the person that needs to ask the question in the first place.

On your current Linux desktop, right now, you have at least three processes running: the X server, a window manager/compositor and your web browser. The X server is responsible for rendering things to the screen and handling your input. The window manager is responsible for telling the X server where to render the web browser window. Your web browser is responsible for displaying this post. The X server and the window manager communicate over the X protocol, the X server and the web browser do so too. The browser and the window manager communicate through X properties using the X server as a middle man. That too is done via the X protocol. Note: This is of course a very simplified view.

Wayland is a protocol and it replaces the X protocol. Under Wayland, you only need two processes: a compositor and your web browser. The compositor is effectively equivalent to the X server and window manager merged into one thing, and it communicates with the web browser over the Wayland protocol. For this to work you need the compositor and the web browser to be able to understand the Wayland protocol.

This is why the question "is wayland ready yet" does not make a lot of sense. Wayland is the communication protocol and says very little about the implementation of the two sides that you want to communicate.

Let's assume a scenario where we all decide to switch from English to French because it sounds nicer and English was designed in the 80s when ASCII was king so it doesn't support those funky squiggles that the French like to put on every second character. In this scenario, you wouldn't ask "Is French ready yet?" If no-one around you speaks French yet, then that's not the language not being ready, the implementation (i.e. the humans) aren't ready. Maybe you can use French in a restaurant, but not yet in the supermarket. Maybe one waiter speaks both English and French, but the other one French only. So whether you can use French depends very much on the situation. But everyone agrees that eventually we'll all speak French, even though English will hang around for ages until it finally falls out of use. And those squiggles are so cute!

Wayland is the same. The protocol is stable and has been for a while. But not every compositor and/or toolkit/application speak Wayland yet, so it may not be sufficient for your use-case. So rather than asking "Is Wayland ready yet", you should be asking: "Can I run GNOME/KDE/Enlightenment/etc. under Wayland?" That is the right question to ask, and the answer is generally "It depends what you expect to work flawlessly." This also means "people working on Wayland" is often better stated as "people working on Wayland support in ....".

An exception to the above: Wayland as a protocol defines what you can talk about. As a young protocol (compared to X with 30 years worth of extensions) there are things that should be defined in the protocol but aren't yet. For example, Wacom tablet support is currently missing. Those are the legitimate cases where you can say Wayland isn't ready yet and where people are "working on Wayland". Of course, once the protocol is agreed on, you fall back to the above case: both sides of the equation need to implement the new protocol before you can make use of it.

Update 25/01/15: Matthias' answer to Is GNOME on Wayland ready yet?

January 22, 2016

A Wayland status update

Peter argues that the question “Is Wayland ready yet?” is not the best question to ask.  Then maybe this is a better question:

Is GNOME on Wayland ready yet ?

It has been our goal for a while to get to a point where the Wayland port can be declared complete and ready to be enabled by default. We’ve come a long way since we started the porting effort in September 2013. In fact, we feel that we’re close enough that we can  aim for Wayland by default in Fedora 24.

But the last mile is always the longest, and there’s still a few steps to take before we’re there.  With this weeks releases of Wayland 1.9.91 and the GNOME 3.19.4 releases, we’ve taken a couple of the steps:

  • A lot of work has gone into fixing the positioning of dialogs, menus and other popups in applications. When it was first introduced, the GTK+ Wayland backend was using heuristics based on window type hints for this; now we are more strict about it: just setting a transient parent should be enough to ensure proper placement. We also try to handle dialogs without transient parent as good as we can.
  • Kinetic scrolling now works as well under Wayland as under X11 – or even better (at least as far as the protocol is involved; Wayland has explicit support for this while we are relying on driver-specific heuristics under X). The relevant Wayland protocol additions needed  for this are the axis stop events.
  • Drag-and-Drop under Wayland is now comparable to X11. This is the culmination of multiple efforts:  The Wayland protocol gained some necessary dnd events and supports actions now.  On the GTK+ side, we’ve moved drag icon creation and input handling to the GDK backends, where it can be done in a backend-specific manner.

Whats next ?

  • We should see initial support for Wacom  tablets in Wayland 1.10.
  • A replacement for the X11 primary selection (“middle click paste”) is also in the works; I hope we can reach agreement on the protocol soon.
  • Inside GTK+, menu positioning is being reworked in a similar way to DND: Pushing it down to GDK, where it can be implemented in a backend-specific manner. This is being driven by the team working on the Mir backend, but it will benefit Wayland just as well.

And here is a sneak preview of Wayland remoting that Jonas has been working on for a while:

<video class="wp-video-shortcode" controls="controls" height="266" id="video-1551-1" preload="metadata" width="474"><source src="https://blogs.gnome.org/mclasen/files/2016/01/Screencast-GNOME-Wayland-VNC.webm?_=1" type="video/webm">https://blogs.gnome.org/mclasen/files/2016/01/Screencast-GNOME-Wayland-VNC.webm</video>

January 20, 2016

Linux Foundation quietly drops community representation
The Linux Foundation is an industry organisation dedicated to promoting, protecting and standardising Linux and open source software[1]. The majority of its board is chosen by the member companies - 10 by platinum members (platinum membership costs $500,000 a year), 3 by gold members (gold membership costs $100,000 a year) and 1 by silver members (silver membership costs between $5,000 and $20,000 a year, depending on company size). Up until recently individual members ($99 a year) could also elect two board members, allowing for community perspectives to be represented at the board level.

As of last Friday, this is no longer true. The by-laws were amended to drop the clause that permitted individual members to elect any directors. Section 3.3(a) now says that no affiliate members may be involved in the election of directors, and section 5.3(d) still permits at-large directors but does not require them[2]. The old version of the bylaws are here - the only non-whitespace differences are in sections 3.3(a) and 5.3(d).

These changes all happened shortly after Karen Sandler announced that she planned to stand for the Linux Foundation board during a presentation last September. A short time later, the "Individual membership" program was quietly renamed to the "Individual supporter" program and the promised benefit of being allowed to stand for and participate in board elections was dropped (compare the old page to the new one). Karen is the executive director of the Software Freedom Conservancy, an organisation involved in the vitally important work of GPL enforcement. The Linux Foundation has historically been less than enthusiastic about GPL enforcement, and the SFC is funding a lawsuit against one of the Foundation's members for violating the terms of the GPL. The timing may be coincidental, but it certainly looks like the Linux Foundation was willing to throw out any semblance of community representation just to ensure that there was no risk of someone in favour of GPL enforcement ending up on their board.

Much of the code in Linux is written by employees paid to do this work, but significant parts of both Linux and the huge range of software that it depends on are written by community members who now have no representation in the Linux Foundation. Ignoring them makes it look like the Linux Foundation is interested only in promoting, protecting and standardising Linux and open source software if doing so benefits their corporate membership rather than the community as a whole. This isn't a positive step.

[1] Article II of the bylaws
[2] Other than in the case of the TAB representative, an individual chosen by a board elected via in-person voting at a conference

comment count unavailable comments
xdg-app and GNOME Software

With a huge amount of support from alex, installing applications using xdg-app in GNOME Software is working. There are still a few rough edges, but it’s getting there quickly now.

Screenshot from 2016-01-20 16-50-02

X.Org the project vs X.Org the Foundation

In light of recent general confusion between X.Org the technical project and X.Org the Foundation here's a little overview.

X.Org the project

X.Org is the current reference implementation of the X Window System which has been around since the mid-80s. Its most prominent members is the X server and the related drivers but we put a whole bunch of other things under the same umbrella, e.g. mesa, drm, and - yes - wayland. Like most free software projects it is loosely organised and very few developers are involved in everything, everybody has their niche. If you're running Linux or a BSD and you can see a desktop environment in front of you, X.Org the technical project is somewhere in that stack.

X.Org the Foundation

The foundation is a non-profit organisation tasked with the stewardship of the X Window System, particularly the X.Org implementation. The most important thing is: the X.Org Foundation does not control the technical direction, it acts in a supporting role only. X.Org has a 501(c)3 tax code in the US which means that donations can be tax deducted (though we haven't collected donations in years). It also means that how we can spend money is very restricted. These days the Foundation's supporting roles are largely: sponsoring the annual X Developers Conference (XDC), providing travel sponsorship to XDC attendees and be the organisation to participate in the Google Summer of Code. Oh, and did I mention that the X.Org Foundation does not control the technical direction?

What does it matter?

The difference matters, especially for well-nuanced and thought-out statements like "X must die" in response to articles about the X.Org Foundation. If you want the Foundation to cease to exist, you're essentially saying "XDC and X.Org's GSoC participation must die". Given that a significant percentage of those two are now Wayland-related that may have some unintended side-effects. If you want the technical project to die, it may be wise to consider the side-effects. Wayland isn't quite ready yet, much of the work that is done under the umbrella of X benefits Wayland (libinput, graphics driver work, etc.).

Now if you excuse me, there's a windmill that needs tilting at. Rocinante, where are you?

January 18, 2016

NetworkManager 1.2 Has Better Wi-Fi Scanning

dlink
Almost since the beginning of time, NetworkManager kept an internal list of access points found in the last 3 scans.  Since the background scan were triggered at least every two minutes, an access point could stay in the list for up to 6 minutes.  This was a compromise between mobility, unreliable drivers, and an unreliable medium (eg, air).  Even when you’re not moving the closest access point may not show up in every scan.  So NetworkManager attempted to compensate by keeping access points around for a longer time.

Obviously that approach has problems if you’re driving, on a train, or on a bus.  You can end up with a huge list of access points that are obviously no longer in range.  If you turn off an access point, it could stay in the list a long time.

Ubuntu contributed a patch that exposes the “last seen time” for each access point, which allows the user-interface to decide for itself which access points to show.  A location service (like Firefox or Geoclue) may want a more complete list of access points than the live Wi-Fi network list does, for example, which is why NetworkManager keeps the list in the first place instead of only showing the results of the most recent (and potentially unreliable) scan.

But in the end this behavior needed to change, and with recent versions of wpa_supplicant it was possible to make NetworkManager’s scanning behavior better.  The supplicant also contains a scan list from which NetworkManager built it’s.  Wouldn’t it be great if there was one list instead of two?

So we threw away the internal NetworkManager list and just followed the supplicant’s list.  When the supplicant decides that an access point is no longer visible, NetworkManager removes it too.  This works better because the supplicant has more information than NetworkManager does and can make smarter decisions.  NetworkManager tweaks the supplicant’s behavior through the BSSExpireAge and BSSExpireCount properties so that any access point seen more than 4 minutes ago, or not seen in the past two scans, will be dropped.

When scans happen more often, like when a Wi-Fi network list is displayed, the two-scan limit removes access points after 20 or 30 seconds in the best case.  The supplicant performs backgrounds scans to facilitate faster roaming, which can be triggered on signal strength, which also helps remove old access points when they are out of range.

Tracking the Current BSS

Along with the scanning cleanup, NetworkManager delegates tracking the access point you’re currently associated with to wpa_supplicant’s CurrentBSS property.  Previously NetworkManager periodically asked the Wi-Fi driver what the current access point was, but this was inconsistently implemented between drivers and required hacky workarounds to smooth out intermittent results.

The supplicant’s CurrentBSS property tracks the access point the supplicant wants to be associated with, not what the driver currently is associated with, but these are almost always the same thing, and there’s no point in telling the user that they are momentarily disconnected from their access point during a scan when there is no actual interruption in traffic due to 802.11 protocol mechanisms like powersave buffering.  This was another huge cleanup in the NetworkManager codebase.

Death of dbus-glib

Finally, along with these changes all communication with wpa_supplicant was switched to use GDBus instead of the old, deprecated, and unmaintained dbus-glib library.  This advances our goal of removing all use of dbus-glib from NetworkManager, which was one of the first major users of the library in 2004, and is likely the last one too.  GDBus provides much better integration with glib and the GIO model, is fully supported, and has an saner API.

And even better, through the painstaking work of Dan Winship, Jirka Klimes, Thomas Haller, Lubomir Rintel, me, and others, all of NetworkManager 1.2 was ported to GDBus without changing our public D-Bus API.  Welcome to the future!

January 10, 2016

Support for "Airplane mode" keys
As we were working on audio jack notifications, and were wondering whether the type of notification we'd pop up in this case could be reused in other cases, I encountered a feature request that could now be solved easily with the rfkill D-Bus service we added to gnome-settings-daemon for the 3.10 release.

If you have keyboard buttons on your laptop to enable or disable Bluetooth, or Airplane mode, you can now use them. Note that the "UWB" toggle key will toggle the whole airplane mode mainly because no in-kernel driver uses it, and nobody remembers what UWB is.

Note that the labels and icons used are still subject to changes. In particular as you can see that the labels are too long for lower resolutions.






January 09, 2016

gom is now usable from JavaScript/gjs
Prodded by me while I snoozed on his sofa and with his cat warming me up, a day before the Content Applications hackfest, Florian Müllner started working on fixing a long-standing gjs bug that made it impossible to use gom in GNOME/JavaScript applications. The result of that initial research came a few days later, and is now part of the latest gjs release.

This also fixes using GtkBuilder and json-glib when the libraries create new objects for the benefit of the JavaScript code.

We can finally use gom to store user data in applications like Bolso. Thanks Florian!

January 07, 2016

The importance of Keywords for the software center

In the software center we allow the user to search using case-insensitive keywords, for instance searching for ‘excel’ could match Libreoffice Calc or many other free software spreadsheet applications. At the moment we use the translated keywords set in the desktop file, any extra <keyword> entries in the AppData file, and then fall back to generating tokens from the name, summary and description using a heuristic. This heuristic works most of the time, but a human can often do much better when we know what the most important words are. I’ve started emailing maintainers who do not have any keywords in their application (using the <update_contact> details in the AppData file), but figured I should also write something here.

So, what do I want you to do? If you have no existing keywords, I would like you to add some keywords in the desktop file or the AppData file. If you want the keywords to be used by GNOME Shell as well (which you probably do), the best place to put any search terms is in the keywords section of the desktop file. This can also be marked as translatable so non-English users can search in their own language. This would look something like Keywords=3D;printer; (remember the trailing semicolon!)

The alternative is to put the keywords in the AppData file so that they are only used by the software center and not the desktop shell. You can of course combine putting keywords in both places. The AppData keywords can also be translated, and would look like this:

<keywords>
<keyword>3D</keyword>
<keyword>printer</keyword>
</keywords>

Of course, you don’t have to do a release with this fix straight away, and if you have a stable branch it would be a good thing to backport this as well if it does not add translated strings or you have no string freeze policy. Nothing bad will happen if you ignore this request, but please be aware that matches from keywords are ordered higher in the search results than other partial matches from the name or summary. You also don’t have to add keywords that are the same as the application name or package name, as these are automatically added as case insensitive search tokens. If you don’t have any keywords then your application will still be visible in the various software centers, but it may be harder to find.

Comments welcome.

January 01, 2016

The current state of boot security
I gave a presentation at 32C3 this week. One of the things I said was "If any of you are doing seriously confidential work on Apple laptops, stop. For the love of god, please stop." I didn't really have time to go into the details of that at the time, but right now I'm sitting on a plane with a ridiculous sinus headache and the pseudoephedrine hasn't kicked in yet so here we go.

The basic premise of my presentation was that it's very difficult to determine whether your system is in a trustworthy state before you start typing your secrets (such as your disk decryption passphrase) into it. If it's easy for an attacker to modify your system such that it's not trustworthy at the point where you type in a password, it's easy for an attacker to obtain your password. So, if you actually care about your disk encryption being resistant to anybody who can get temporary physical possession of your laptop, you care about it being difficult for someone to compromise your early boot process without you noticing.

There's two approaches to this. The first is UEFI Secure Boot. If you cryptographically verify each component of the boot process, it's not possible for a user to compromise the boot process. The second is a measured boot. If you measure each component of the boot process into the TPM, and if you use these measurements to control access to a secret that allows the laptop to prove that it's trustworthy (such as Joanna Rutkowska's Anti Evil Maid or my variant on the theme), an attacker can compromise the boot process but you'll know that they've done so before you start typing.

So, how do current operating systems stack up here?

Windows: Supports UEFI Secure Boot in a meaningful way. Supports measured boot, but provides no mechanism for the system to attest that it hasn't been compromised. Good, but not perfect.

Linux: Supports UEFI Secure Boot[1], but doesn't verify signatures on the initrd[2]. This means that attacks such as Evil Abigail are still possible. Measured boot isn't in a good state, but it's possible to incorporate with a bunch of manual work. Vulnerable out of the box, but can be configured to be better than Windows.

Apple: Ha. Snare talked about attacking the Apple boot process in 2012 - basically everything he described then is still possible. Apple recently hired the people behind Legbacore, so there's hope - but right now all shipping Apple hardware has no firmware support for UEFI Secure Boot and no TPM. This makes it impossible to provide any kind of boot attestation, and there's no real way you can verify that your system hasn't been compromised.

Now, to be fair, there's attacks that even Windows and properly configured Linux will still be vulnerable to. Firmware defects that permit modification of System Management Mode code can still be used to circumvent these protections, and the Management Engine is in a position to just do whatever it wants and fuck all of you. But that's really not an excuse to just ignore everything else. Improving the current state of boot security makes it more difficult for adversaries to compromise a system, and if we ever do get to the point of systems which aren't running any hidden proprietary code we'll still need this functionality. It's worth doing, and it's worth doing now.

[1] Well, except Ubuntu's signed bootloader will happily boot unsigned kernels which kind of defeats the entire point of the exercise
[2] Initrds are built on the local machine, so we can't just ship signed images

comment count unavailable comments

December 23, 2015

GNOME Software and xdg-app

Here’s a little Christmas present. This is GNOME Software working with xdg-app to allow live updates of apps and runtimes.

Screenshot from 2015-12-22 15-06-44

This is very much a prototype and needs a lot more work, but seems to work for me with xdg-app from git master (compile with --enable-libxdgapp). If any upstream projects needed any more encouragement, not including an AppData file means the application gets marked as nonfree as we don’t have any project licensing information. Inkscape, I’m looking at you.

December 22, 2015

CSS boxes in GTK+

In my last update, I talked about CSS nodes in GTK+, which are used to match theme style information to widgets, and to hold state that is needed to handle e.g. animations.

Today, I’ll focus on CSS boxes in GTK+. This is where we take size information from the theme, such as margins, padding, and minimum sizes, and apply them to the widget layout. The internal name we’ve chosen for the objects that encapsulate this information is gadgets . I’m well aware that we’re not exactly breaking new ground with this name, but the name isn’t really important (as none of this is currently exposed as public API). For the purpose of this post, you can think of gadgets simply as CSS boxes that make up widgets.

Lets start with a simple example widget, to see what this is about: a radio button (all of the screenshots here are showing GtkInspector, which is available in any GTK+ application, so you can easily do these experiments yourself).

radiobutton1
We start by making the border of the widget’s box visible. The CSS snipplet shown below is using the element name radiobutton, which GtkRadioButton sets on its main CSS node, so this selector will match.

Radio button
This is how it looks. If you compare carefully with the earlier screenshot, you can see that GTK+ has made the widget bigger to make room for the 6 pixel border, while giving the same size as before to the content of the widget.

Radiobutton
The CSS box model has more than just a border, it also allows for padding (inside the border) and margins (outside the border). In fact, the box model is much more complicated than this, and we’re only scraping the surface here. The GTK+ implementation of the box model handles most of the important parts, but doesn’t cover some things that don’t fit well in our situation.

So, lets add a margin and padding. To make this visible, we use some features of CSS background rendering: We specify two backgrounds (a solid blue one and a solid yellow one), and we let one of it be clipped to the size of the ‘content box’ (the area given to the widget content) and one to the size of the ‘padding box’ (which is the content area plus the padding around it).

Radiobutton
This is how it looks. The margin is not very visible here, but you can clearly see the padding (yellow) and the content (blue). Note again how the widget has gotten larger to accommodate the padding and margin.

Radiobutton
When I talked about CSS nodes, I mentioned how widgets can have extra nodes for their ‘components’. This extends to gadgets: each of the widgets components gets their own CSS box, with its own margin, padding, background and whatnot.

So, lets continue the coloring exercise by making our CSS snipplet match not just the radiobutton itself, but also the label and the indicator. Their CSS names are label and radio.

Radiobutton
This is how it looks. Here, we can actually see the margin of the label have an effect: it causes the content area (in blue) to be bigger than it would otherwise be.

Radiobutton
I hope by now it is obvious that this is giving a lot of expressive power to theme authors, who can use all of this to affect the layout and rendering of all the widgets and their components. And there are a lot of them:

Boxes
The GtkInspector is a really useful tool for exploring what is possible with all of this. It gives easy access to both the CSS nodes of each widget:

Nodes
…and to the CSS properties of each node, which is basically the outcome of matching the theme CSS information to the node tree, also taking into account inherited properties and other complications:

CSS properties

Whats next ? All of what I’ve shown here  is already available in GTK+ 3.19.5. We’ve made good progress on converting most widgets to use gadgets internally (‘progress’ is a bit of an understatement, it was a herculean effort, mainly by Benjamin Otte, Cosimo Cecchi and myself). But we are not done yet, so we will continue working on completing the conversion, and on documenting GTKs CSS capabilities better.

When that is done,  we can look at adding other interesting bits of the CSS spec, like calc() or border collapsing. CSS is a huge standard, and theme designers always ask for more.

December 21, 2015

GPL enforcement is a social good
The Software Freedom Conservancy is currently running a fundraising program in an attempt to raise enough money to continue funding GPL compliance work. If they don't gain enough supporters, the majority of their compliance work will cease. And, since SFC are one of the only groups currently actively involved in performing GPL compliance work, that basically means that there will be nobody working to ensure that users have the rights that copyright holders chose to give them.

Why does this matter? More people are using GPLed software than at any point in history. Hundreds of millions of Android devices were sold this year, all including GPLed code. An unknowably vast number of IoT devices run Linux. Cameras, Blu Ray players, TVs, light switches, coffee machines. Software running in places that we would never have previously imagined. And much of it abandoned immediately after shipping, gently rotting, exposing an increasingly large number of widely known security vulnerabilities to an increasingly hostile internet. Devices that become useless because of protocol updates. Toys that have a "Guaranteed to work until" date, and then suddenly Barbie goes dead and you're forced to have an unexpected conversation about API mortality with your 5-year old child.

We can't fix all of these things. Many of these devices have important functionality locked inside proprietary components, released under licenses that grant no permission for people to examine or improve them. But there are many that we can. Millions of devices are running modern and secure versions of Android despite being abandoned by their manufacturers, purely because the vendor released appropriate source code and a community grew up to maintain it. But this can only happen when the vendor plays by the rules.

Vendors who don't release their code remove that freedom from their users, and the weapons users have to fight against that are limited. Most users hold no copyright over the software in the device and are unable to take direct action themselves. A vendor's failure to comply dooms them to having to choose between buying a new device in 12 months or no longer receiving security updates. When yet more examples of vendor-supplied malware are discovered, it's more difficult to produce new builds without them. The utility of the devices that the user purchased is curtailed significantly.

The Software Freedom Conservancy is one of the only organisations actively fighting against this, and if they're forced to give up their enforcement work the pressure on vendors to comply with the GPL will be reduced even further. If we want users to control their devices, to be able to obtain security updates even after the vendor has given up, we need to keep that pressure up. Supporting the SFC's work has a real impact on the security of the internet and people's lives. Please consider giving them money.

comment count unavailable comments

December 15, 2015

libinput and the Lenovo x220 touchpad - after a firmware update to version 8.1

This post only applies to users of the Lenovo x220 laptop experiencing issues when using the touchpad. Specifically, the touchpad is imprecise and "jumpy" after a firmware update, as outlined in Fedora bug 1264453. The cause is buggy touchpad firmware, identifiable by the string "fw: 8.1" in the dmesg output for the touchpad:


[ +0.005261] psmouse serio1: synaptics: Touchpad model: 1, fw: 8.1,
id: 0x1e2b1, caps: 0xd002a3/0x940300/0x123800, board id: 1611, fw id: 1099905
If you are experiencing these touchpad issues and your dmesg shows the 8.1 firmware version, please read on for a solution. By default, the x220 shipped with version 8.0 so unless you updated the firmware as part of a Lenovo update, you are not affected by this bug.

The touchpad issues seem identical as the ones seen on the Lenovo x230 model which has the same physical hardware and also ships with a firmware version 8.1. The root cause as seen by libinput is that the touchpad only sends events once the finger moves approximately 50 device units in either direction. The touchpad advertises a resolution of 65 units/mm horizontally and 136 units/mm vertically, but the effective resolution is reduced by roughly 75% and 30% This bugzilla attachment 1082925 shows the recording, you can easily see that while the pressure is upgraded with high granularity, the motion coordinates jump from one position to the next. From what we know this was introduced by the touchpad firmware v8.1, presumably as part of a filter to reduce the jitter some x230 users saw.

libinput automatically detects the x230 and enables a custom acceleration function for just that model. That same acceleration function works for the x220 v8.1, but unfortunately we cannot automatically detect it. As of libinput 1.1.3, libinput recognises a special udev tag, LIBINPUT_MODEL_LENOVO_X220_TOUCHPAD_FW81, to mark such an updated x220 and enable a better pointer behaviour. To apply this tag, please do the following:

  1. Create a new file /etc/udev/hwdb.d/90-libinput-x220-fw8.1.hwdb
  2. Look for X220 in the 90-libinput-model-quirks.hwdb file, copy the match and the property assignment into the file. As of the time of writing, the two lines are as below, but make sure you take the latest from your locally installed libinput version or the link above.

    libinput:name:SynPS/2 Synaptics TouchPad:dmi:*svnLENOVO:*:pvrThinkPadX220*
    LIBINPUT_MODEL_LENOVO_X220_TOUCHPAD_FW81=1
  3. Update the udev hwdb with sudo udevadm hwdb --update
  4. Verify the tag shows up with sudo udevadm test /sys/class/input/event4 (adjust the event node if necessary)
  5. Reboot
The touchpad is now marked as requiring special treatment and libinput will apply a different pointer acceleration for this touchpad.

Note that any udev property starting with LIBINPUT_MODEL_ is private API and subject to change at any time. We will never break the meaning of the LIBINPUT_MODEL_LENOVO_X220_TOUCHPAD_FW81 property, but the exact behaviour of the property is implementation-dependent and may change at any time. Do not use it for any other purpose than marking the touchpad on a Lenovo x220 with an updated touchpad firmware version v8.1.

December 10, 2015

The Linux Vendor Firmware Service Welcomes Dell

I’m finally able to talk about one of the large vendors who have been trialing the LVFS service for the last few months. Dell have been uploading embargoed UEFI firmware files with metadata for a while, testing the process and the workflow ready for upcoming new models. Mario (Dell) and myself (Red Hat) have been working on fixing all the issues that pop up on real hardware and making the web service both secure and easy to use.

Screenshot from 2015-12-10 08-43-08

The Dell Edge Gateway will be available for purchase soon. When it goes on sale, firmware updates in Linux will work out-of-the-box. I’ve been told that Dell are considering expanding out the LVFS support to all new models supporting UEFI updates. In order to prioritize what models to work on first, I’ve been asked to share this anonymous survey on what Dell hardware people are using on Linux and to gauge if people actually care about being able to upgrade the firmware easily in Linux.

In November, 224 firmware files were installed onto client systems using fwupd. At the moment to update the firmware metadata you need to manually click the refresh button in the updates page, which so far 40,000 people have done. Given that the ColorHug hardware is the only released hardware with firmware on the LVFS, the 224 downloads is about what I expected. When we have major vendors like Dell (and other vendors I can’t talk about yet) shipping real consumer hardware with UEFI update capability the number of files provided should go up by orders of magnitude.

For Fedora 24 we’ll be downloading the firmware metadata automatically (rather than requiring a manual refresh in the updates panel) and we’ve been using the Fedora 23 users as a good way of optimizing the service so we know we can handle the load when we get hundreds of thousands of automatic requests a month. Fedora 24 will also be the first release able to do updates on DFU USB devices, and also the first release with system upgrade capabilities inside GNOME Software so it’s quite exciting from my point of view.

With Dell on board, I’m hoping it will give some of the other vendors enough confidence in the LVFS to talk about distributing their own firmware in public. The LVFS is something I run for all distributions free of charge, but of course Red Hat pays for my time to develop and run the service. I’m looking forward to working with more Red Hat partners and OpenHardware vendors adding even more firmware for even more types of device in the future.

December 08, 2015

Contents Apps Hackfest 2015
As you might already have noticed from the posts on Planet GNOME, and can find again on the hackfest's page, we spent some time in the MediaLab Prado discussing and hacking on Content Apps.

Music

Following discussions about Music's state, I did my bit trying to gather more contributors by porting it to grilo 0.3, and thus bringing it back into the default jhbuild target.

Videos

I made some progress on Videos' "series grouping" feature. Loads of backend code written, but not much in the way of UI for now. We however made some progress discussing said UI with Allan.

I also took the opportunity to fix a few low-hanging fruit^Wbugs.

Documents

This is where the majority of my energy went. After getting a new enough version of LibreOffice going on my machine (Fedora users, that lives in rawhide only right), no thanks to COPR, I tested Pranav's LibreOfficeKit integration into gnome-documents, after Cosimo rebased it.

You can test it now by checking out the wip/lokdocview-rebase branch of gnome-documents, grabbing the above mentioned version of LibreOffice, and running:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/libreoffice/program/  gjs org.gnome.Documents

After a number of fixes, and bugs filed in the Document Foundation bugzilla, we should be able to land this so that you can preview and edit word processing documents, presentations and spreadsheets without going through the heavy PDF preview.


A picture, which doubles the length of my blog post

And the side-effect of this work is that we can start adding new "views" to the application without too much trouble, like, say, an epub view.

Thanks

Many thanks to the GNOME Foundation for sponsoring my travel, the MediaLab Prado for hosting us, and Allan and Florian for organising the hackfest.


December 02, 2015

OpenHardware and code signing (update)

I posted a few weeks ago about the difficulty of providing device-side verification of firmware updates, at the same time remaining OpenHardware and thus easily hackable. The general consensus was that allowing anyone to write any kind of firmware to the device without additional authentication was probably a bad idea, even for OpenHardware devices. I think I’ve come up with an acceptable compromise I can write up as a recommendation, as per usual using the ColorHug+ as an example. For some background, I’ve sold nearly 3,000 original ColorHug devices, and in the last 4 years just three people wanted help writing custom firmware, so I hope you can see the need to protect the majority is so much larger than making the power users happy.

ColorHug+ will be supplied with a bootloader that accepts only firmware encrypted with the secret XTEA key I that I’m using for my devices. XTEA is an acceptable compromise between something as secure as ECC, but that’s actually acceptable in speed and memory usage for a 8-bit microcontroller running at 6MHz with 8k of ROM. Flashing a DIY or modified firmware isn’t possible, and by the same logic flashing a malicious firmware will also not work.

To unlock the device (and so it stays OpenHardware) you just have to remove the two screws, and use a paper-clip to connect TP5 and GND while the device is being plugged into the USB port. Both lights will come on, and stay on for 5 seconds and then the code protection is turned off. This means you can now flash any home-made or malicious firmware to the device as you please.

There are downsides to unlocking; you can’t re-lock the hardware so it supports official updates again. I don’t know if this is a huge problem; flashing home-made firmware could damage the device (e.g. changing the pin mapping from input to output and causing something to get hot). If this is a huge problem I can fix CH+ to allow re-locking and fix up the guidelines, although I’m erring on unlocking being a one way operation.

Comments welcome.

November 29, 2015

What is hacker culture?
Eric Raymond, author of The Cathedral and the Bazaar (an important work describing the effectiveness of open collaboration and development), recently wrote a piece calling for "Social Justice Warriors" to be ejected from the hacker community. The primary thrust of his argument is that by calling for a removal of the "cult of meritocracy", these SJWs are attacking the central aspect of hacker culture - that the quality of code is all that matters.

This argument is simply wrong.

Eric's been involved in software development for a long time. In that time he's seen a number of significant changes. We've gone from computers being the playthings of the privileged few to being nearly ubiquitous. We've moved from the internet being something you found in universities to something you carry around in your pocket. You can now own a computer whose CPU executes only free software from the moment you press the power button. And, as Eric wrote almost 20 years ago, we've identified that the "Bazaar" model of open collaborative development works better than the "Cathedral" model of closed centralised development.

These are huge shifts in how computers are used, how available they are, how important they are in people's lives, and, as a consequence, how we develop software. It's not a surprise that the rise of Linux and the victory of the bazaar model coincided with internet access becoming more widely available. As the potential pool of developers grew larger, development methods had to be altered. It was no longer possible to insist that somebody spend a significant period of time winning the trust of the core developers before being permitted to give feedback on code. Communities had to change in order to accept these offers of work, and the communities were better for that change.

The increasing ubiquity of computing has had another outcome. People are much more aware of the role of computing in their lives. They are more likely to understand how proprietary software can restrict them, how not having the freedom to share software can impair people's lives, how not being able to involve themselves in software development means software doesn't meet their needs. The largest triumph of free software has not been amongst people from a traditional software development background - it's been the fact that we've grown our communities to include people from a huge number of different walks of life. Free software has helped bring computing to under-served populations all over the world. It's aided circumvention of censorship. It's inspired people who would never have considered software development as something they could be involved in to develop entire careers in the field. We will not win because we are better developers. We will win because our software meets the needs of many more people, needs the proprietary software industry either can not or will not satisfy. We will win because our software is shaped not only by people who have a university degree and a six figure salary in San Francisco, but because our contributors include people whose native language is spoken by so few people that proprietary operating system vendors won't support it, people who live in a heavily censored regime and rely on free software for free communication, people who rely on free software because they can't otherwise afford the tools they would need to participate in development.

In other words, we will win because free software is accessible to more of society than proprietary software. And for that to be true, it must be possible for our communities to be accessible to anybody who can contribute, regardless of their background.

Up until this point, I don't think I've made any controversial claims. In fact, I suspect that Eric would agree. He would argue that because hacker culture defines itself through the quality of contributions, the background of the contributor is irrelevant. On the internet, nobody knows that you're contributing from a basement in an active warzone, or from a refuge shelter after escaping an abusive relationship, or with the aid of assistive technology. If you can write the code, you can participate.

Of course, this kind of viewpoint is overly naive. Humans are wonderful at noticing indications of "otherness". Eric even wrote about his struggle to stop having a viscerally negative reaction to people of a particular race. This happened within the past few years, so before then we can assume that he was less aware of the issue. If Eric received a patch from someone whose name indicated membership of this group, would there have been part of his subconscious that reacted negatively? Would he have rationalised this into a more critical analysis of the patch, increasing the probability of rejection? We don't know, and it's unlikely that Eric does either.

Hacker culture has long been concerned with good design, and a core concept of good design is that code should fail safe - ie, if something unexpected happens or an assumption turns out to be untrue, the desirable outcome is the one that does least harm. A command that fails to receive a filename as an argument shouldn't assume that it should modify all files. A network transfer that fails a checksum shouldn't be permitted to overwrite the existing data. An authentication server that receives an unexpected error shouldn't default to granting access. And a development process that may be subject to unconscious bias should have processes in place that make it less likely that said bias will result in the rejection of useful contributions.

When people criticise meritocracy, they're not criticising the concept of treating contributions based on their merit. They're criticising the idea that humans are sufficiently self-aware that they will be able to identify and reject every subconscious prejudice that will affect their treatment of others. It's not a criticism of a desirable goal, it's a criticism of a flawed implementation. There's evidence that organisations that claim to embody meritocratic principles are more likely to reward men than women even when everything else is equal. The "cult of meritocracy" isn't the belief that meritocracy is a good thing, it's the belief that a project founded on meritocracy will automatically be free of bias.

Projects like the Contributor Covenant that Eric finds so objectionable exist to help create processes that (at least partially) compensate for our flaws. Review of our processes to determine whether we're making poor social decisions is just as important as review of our code to determine whether we're making poor technical decisions. Just as the bazaar overtook the cathedral by making it easier for developers to be involved, inclusive communities will overtake "pure meritocracies" because, in the long run, these communities will produce better output - not just in terms of the quality of the code, but also in terms of the ability of the project to meet the needs of a wider range of people.

The fight between the cathedral and the bazaar came from people who were outside the cathedral. Those fighting against the assumption that meritocracies work may be outside what Eric considers to be hacker culture, but they're already part of our communities, already making contributions to our projects, already bringing free software to more people than ever before. This time it's Eric building a cathedral and decrying the decadent hordes in their bazaar, Eric who's failed to notice the shift in the culture that surrounds him. And, like those who continued building their cathedrals in the 90s, it's Eric who's now irrelevant to hacker culture.

(Edited to add: for two quite different perspectives on why Eric's wrong, see Tim's and Coraline's posts)

comment count unavailable comments

November 20, 2015

Better polygon rendering in LibreOffice's Gtk3 Support

Above is how LibreOffice's "svp" backend rendered rotated text outlines in chart where the text is represented by polygon paths. Because the gtk3 backend is based on that svp backend that's what you got with the gtk3 support enabled.

After today's work above is how the svp backend now renders those paths when rendering to a cairo-compatible surface such as the gtk3 support provides.

If we mandate that "svp" only operates on cairo compatible surfaces, then we can get this niceness into android and online too, and can ditch our non-cairo text rendering code paths.
A GTK+ update

You may have noticed that GTK+ master has a large number of changes in the CSS area. As some like to put it:

Oh NO! they’re breaking themes again!

Which is certainly one way to look at it, but I think it misses the point a little bit – since the effort is actually meant to make life easier for anybody who wants to change the appearance of GTK+ widgets.

What changes are we making ?

As a first step, we are introducing CSS nodes. A CSS node has an element name, a state and it can have style classes.  Each widget has one or more CSS nodes, and they are organized in a tree.  GTK+’s CSS machinery matches CSS selectors on this CSS node tree, and as a result, each node carries a full set of CSS properties.

The transition to CSS nodes is mostly done in GTK+ 3.19.2.

In a  second step, we will integrate CSS nodes into size allocation and rendering. This will bring consistent support for margins, padding and min-width/height for all widgets. This step is currently being prepared on the wip/otte/gadget branch.

None of this is exposed as API yet  (with some small exceptions, such as gtk_widget_path_iter_set_object_name ) . It needs more time to prove itself before we are ready to offer this as stable API.

Why are we doing this ?

There are a number of reasons for doing these changes. In no particular order,

  • Element names and style classes use by widgets for their CSS nodes (and their tree relationships) are now documented and provide a stable interface for themes and custom CSS.
  • Element names provide a level of abstraction over the direct use of type names, and will allow reuse of CSS. You wil be able to add a “check” node to your own widget, whereas you can’t call your own widget GtkCheckButton.
  • CSS nodes are permanent. This implies that they can carry state, e.g. for running CSS animations. They can also notify on state changes.Spinning
  • CSS nodes can explored and manipulated in GtkInspector.Inspect!
  • CSS nodes (in particular, the second phase) force us to clean up all the irregularities of the current widget rendering and move to a much more regular model, where every node draws a background and a  frame. Fancy Switches
  • Positional selectors such as :first-child now work everywhere, including parts of a widget. You can e.g., use :nth-child(even) to style notebook tabs.Fancy Tabs

What do you have to do ?

If you maintain a theme or custom CSS for an application,  update your selectors to use the documented element names and classes. The changes in Adwaita can serve as an example for the kind of changes that are needed.

If you use GTK+’s style machinery to render non-widget content (like mutter does for window decorations), use
gtk_widget_path_iter_set_object_name to make your widget paths look like the widgets they imitate.

If you have code that pokes colors out of GtkStyleContext to do your own rendering, you should really port to use the gtk_render APIs and custom CSS. If you want to keep your existing code working for now, you need to minimally fix gtk_style_context_get_color calls to use the correct state.  Ie. go from

gtk_style_context_get_color (context,
                        GTK_STATE_FLAG_SELECTED,
                        &color);

to

gtk_style_context_save (context);
gtk_style_context_set_state (contexts,
                        GTK_STATE_FLAG_SELECTED);
gtk_style_context_get_color (context,
          gtk_style_context_get_state (context),
          &color);
gtk_style_context_restore (context);

What else is new ?

Support for help overlays Shortcuts got merged into GTK+ and is available in 3.19.1. We’ve gotten some feedback from early adopters and added a few more features, so things should be in good shape for wider adoption: please add it to your favorite GTK+ application, and update this page if you do so.

Wayland support is progressing. So far in 3.19, the focus has been on straightening out issues with window and popup sizing and positioning. Many of the remaining gaps will be fixed by integrating the protocols that have been under discussion for a while now: DND actions, tablet support, pointer lock,…  Jonas just revamped the Wayland protocol development process, which should lead to faster progress on this front. For more details, see Jonas’ mail.

November 19, 2015

If it's not practical to redistribute free software, it's not free software in practice
I've previously written about Canonical's obnoxious IP policy and how Mark Shuttleworth admits it's deliberately vague. After spending some time discussing specific examples with Canonical, I've been explicitly told that while Canonical will gladly give me a cost-free trademark license permitting me to redistribute unmodified Ubuntu binaries, they will not tell me what Any redistribution of modified versions of Ubuntu must be approved, certified or provided by Canonical if you are going to associate it with the Trademarks. Otherwise you must remove and replace the Trademarks and will need to recompile the source code to create your own binaries actually means.

Why does this matter? The free software definition requires that you be able to redistribute software to other people in either unmodified or modified form without needing to ask for permission first. This makes it clear that Ubuntu itself isn't free software - distributing the individual binary packages without permission is forbidden, even if they wouldn't contain any infringing trademarks[1]. This is obnoxious, but not inherently toxic. The source packages for Ubuntu could still be free software, making it fairly straightforward to build a free software equivalent.

Unfortunately, while true in theory, this isn't true in practice. The issue here is the apparently simple phrase you must remove and replace the Trademarks and will need to recompile the source code. "Trademarks" is defined later as being the words "Ubuntu", "Kubuntu", "Juju", "Landscape", "Edubuntu" and "Xubuntu" in either textual or logo form. The naive interpretation of this is that you have to remove trademarks where they'd be infringing - for instance, shipping the Ubuntu bootsplash as part of a modified product would almost certainly be clear trademark infringement, so you shouldn't do that. But that's not what the policy actually says. It insists that all trademarks be removed, whether they would embody an infringement or not. If a README says "To build this software under Ubuntu, install the following packages", a literal reading of Canonical's policy would require you to remove or replace the word "Ubuntu" even though failing to do so wouldn't be a trademark infringement. If an @ubuntu.com email address is present in a changelog, you'd have to change it. You wouldn't be able to ship the juju-core package without renaming it and the application within. If this is what the policy means, it's so impractical to be able to rebuild Ubuntu that it's not free software in any meaningful way.

This seems like a pretty ludicrous interpretation, but it's one that Canonical refuse to explicitly rule out. Compare this to Red Hat's requirements around Fedora - if you replace the fedora-logos, fedora-release and fedora-release-notes packages with your own content, you're good. A policy like this satisfies the concerns that Dustin raised over people misrepresenting their products, but still makes it easy for users to distribute modified code to other users. There's nothing whatsoever stopping Canonical from adopting a similarly unambiguous policy.

Mark has repeatedly asserted that attempts to raise this issue are mere FUD, but he won't answer you if you ask him direct questions about this policy and will insist that it's necessary to protect Ubuntu's brand. The reality is that if Debian had had an identical policy in 2004, Ubuntu wouldn't exist. The effort required to strip all Debian trademarks from the source packages would have been immense[2], and this would have had to be repeated for every release. While this policy is in place, nobody's going to be able to take Ubuntu and build something better. It's grotesquely hypocritical, especially when the Ubuntu website still talks about their belief that people should be able to distribute modifications without licensing fees.

All that's required for Canonical to deal with this problem is to follow Fedora's lead and isolate their trademarks in a small set of packages, then tell users that those packages must be replaced if distributing a modified version of Ubuntu. If they're serious about this being a branding issue, they'll do it. And if I'm right that the policy is deliberately obfuscated so Canonical can encourage people to buy licenses, they won't. It's easy for them to prove me wrong, and I'll be delighted if they do. Let's see what happens.

[1] The policy is quite clear on this. If you want to distribute something other than an unmodified Ubuntu image, you have two choices:
  1. Gain approval or certification from Canonical
  2. Remove all trademarks and recompile the source code
Note that option 2 requires you to rebuild even if there are no trademarks to remove.

[2] Especially when every source package contains a directory called "debian"…

comment count unavailable comments

November 18, 2015

Introducing sd-event

The Event Loop API of libsystemd

When we began working on systemd we built it around a hand-written ad-hoc event loop, wrapping Linux epoll. The more our project grew the more we realized the limitations of using raw epoll:

  • As we used timerfd for our timer events, each event source cost one file descriptor and we had many of them! File descriptors are a scarce resource on UNIX, as RLIMIT_NOFILE is typically set to 1024 or similar, limiting the number of available file descriptors per process to 1021, which isn't particularly a lot.

  • Ordering of event dispatching became a nightmare. In many cases, we wanted to make sure that a certain kind of event would always be dispatched before another kind of event, if both happen at the same time. For example, when the last process of a service dies, we might be notified about that via a SIGCHLD signal, via an sd_notify() "STATUS=" message, and via a control group notification. We wanted to get these events in the right order, to know when it's safe to process and subsequently release the runtime data systemd keeps about the service or process: it shouldn't be done if there are still events about it pending.

  • For each program we added to the systemd project we noticed we were adding similar code, over and over again, to work with epoll's complex interfaces. For example, finding the right file descriptor and callback function to dispatch an epoll event to, without running into invalidated pointer issues is outright difficult and requires non-trivial code.

  • Integrating child process watching into our event loops was much more complex than one could hope, and even more so if child process events should be ordered against each other and unrelated kinds of events.

Eventually, we started working on sd-bus. At the same time we decided to seize the opportunity, put together a proper event loop API in C, and then not only port sd-bus on top of it, but also the rest of systemd. The result of this is sd-event. After almost two years of development we declared sd-event stable in systemd version 221, and published it as official API of libsystemd.

Why?

sd-event.h, of course, is not the first event loop API around, and it doesn't implement any really novel concepts. When we started working on it we tried to do our homework, and checked the various existing event loop APIs, maybe looking for candidates to adopt instead of doing our own, and to learn about the strengths and weaknesses of the various implementations existing. Ultimately, we found no implementation that could deliver what we needed, or where it would be easy to add the missing bits: as usual in the systemd project, we wanted something that allows us access to all the Linux-specific bits, instead of limiting itself to the least common denominator of UNIX. We weren't looking for an abstraction API, but simply one that makes epoll usable in system code.

With this blog story I'd like to take the opportunity to introduce you to sd-event, and explain why it might be a good candidate to adopt as event loop implementation in your project, too.

So, here are some features it provides:

  • I/O event sources, based on epoll's file descriptor watching, including edge triggered events (EPOLLET). See sd_event_add_io(3).

  • Timer event sources, based on timerfd_create(), supporting the CLOCK_MONOTONIC, CLOCK_REALTIME, CLOCK_BOOTIME clocks, as well as the CLOCK_REALTIME_ALARM and CLOCK_BOOTTIME_ALARM clocks that can resume the system from suspend. When creating timer events a required accuracy parameter may be specified which allows coalescing of timer events to minimize power consumption. For each clock only a single timer file descriptor is kept, and all timer events are multiplexed with a priority queue. See sd_event_add_time(3).

  • UNIX process signal events, based on signalfd(2), including full support for real-time signals, and queued parameters. See sd_event_add_signal(3).

  • Child process state change events, based on waitid(2). See sd_event_add_child(3).

  • Static event sources, of three types: defer, post and exit, for invoking calls in each event loop, after other event sources or at event loop termination. See sd_event_add_defer(3).

  • Event sources may be assigned a 64bit priority value, that controls the order in which event sources are dispatched if multiple are pending simultanously. See sd_event_source_set_priority(3).

  • The event loop may automatically send watchdog notification messages to the service manager. See sd_event_set_watchdog(3).

  • The event loop may be integrated into foreign event loops, such as the GLib one. The event loop API is hence composable, the same way the underlying epoll logic is. See sd_event_get_fd(3) for an example.

  • The API is fully OOM safe.

  • A complete set of documentation in UNIX man page format is available, with sd-event(3) as the entry page.

  • It's pretty widely available, and requires no extra dependencies. Since systemd is built on it, most major distributions ship the library in their default install set.

  • After two years of development, and after being used in all of systemd's components, it has received a fair share of testing already, even though we only recently decided to declare it stable and turned it into a public API.

Note that sd-event has some potential drawbacks too:

  • If portability is essential to you, sd-event is not your best option. sd-event is a wrapper around Linux-specific APIs, and that's visible in the API. For example: our event callbacks receive structures defined by Linux-specific APIs such as signalfd.

  • It's a low-level C API, and it doesn't isolate you from the OS underpinnings. While I like to think that it is relatively nice and easy to use from C, it doesn't compromise on exposing the low-level functionality. It just fills the gaps in what's missing between epoll, timerfd, signalfd and related concepts, and it does not hide that away.

Either way, I believe that sd-event is a great choice when looking for an event loop API, in particular if you work on system-level software and embedded, where functionality like timer coalescing or watchdog support matter.

Getting Started

Here's a short example how to use sd-event in a simple daemon. In this example, we'll not just use sd-event.h, but also sd-daemon.h to implement a system service.

#include <alloca.h>
#include <endian.h>
#include <errno.h>
#include <netinet/in.h>
#include <signal.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <unistd.h>

#include <systemd/sd-daemon.h>
#include <systemd/sd-event.h>

static int io_handler(sd_event_source *es, int fd, uint32_t revents, void *userdata) {
        void *buffer;
        ssize_t n;
        int sz;

        /* UDP enforces a somewhat reasonable maximum datagram size of 64K, we can just allocate the buffer on the stack */
        if (ioctl(fd, FIONREAD, &sz) < 0)
                return -errno;
        buffer = alloca(sz);

        n = recv(fd, buffer, sz, 0);
        if (n < 0) {
                if (errno == EAGAIN)
                        return 0;

                return -errno;
        }

        if (n == 5 && memcmp(buffer, "EXIT\n", 5) == 0) {
                /* Request a clean exit */
                sd_event_exit(sd_event_source_get_event(es), 0);
                return 0;
        }

        fwrite(buffer, 1, n, stdout);
        fflush(stdout);
        return 0;
}

int main(int argc, char *argv[]) {
        union {
                struct sockaddr_in in;
                struct sockaddr sa;
        } sa;
        sd_event_source *event_source = NULL;
        sd_event *event = NULL;
        int fd = -1, r;
        sigset_t ss;

        r = sd_event_default(&event);
        if (r < 0)
                goto finish;

        if (sigemptyset(&ss) < 0 ||
            sigaddset(&ss, SIGTERM) < 0 ||
            sigaddset(&ss, SIGINT) < 0) {
                r = -errno;
                goto finish;
        }

        /* Block SIGTERM first, so that the event loop can handle it */
        if (sigprocmask(SIG_BLOCK, &ss, NULL) < 0) {
                r = -errno;
                goto finish;
        }

        /* Let's make use of the default handler and "floating" reference features of sd_event_add_signal() */
        r = sd_event_add_signal(event, NULL, SIGTERM, NULL, NULL);
        if (r < 0)
                goto finish;
        r = sd_event_add_signal(event, NULL, SIGINT, NULL, NULL);
        if (r < 0)
                goto finish;

        /* Enable automatic service watchdog support */
        r = sd_event_set_watchdog(event, true);
        if (r < 0)
                goto finish;

        fd = socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0);
        if (fd < 0) {
                r = -errno;
                goto finish;
        }

        sa.in = (struct sockaddr_in) {
                .sin_family = AF_INET,
                .sin_port = htobe16(7777),
        };
        if (bind(fd, &sa.sa, sizeof(sa)) < 0) {
                r = -errno;
                goto finish;
        }

        r = sd_event_add_io(event, &event_source, fd, EPOLLIN, io_handler, NULL);
        if (r < 0)
                goto finish;

        (void) sd_notifyf(false,
                          "READY=1\n"
                          "STATUS=Daemon startup completed, processing events.");

        r = sd_event_loop(event);

finish:
        event_source = sd_event_source_unref(event_source);
        event = sd_event_unref(event);

        if (fd >= 0)
                (void) close(fd);

        if (r < 0)
                fprintf(stderr, "Failure: %s\n", strerror(-r));

        return r < 0 ? EXIT_FAILURE : EXIT_SUCCESS;
}

The example above shows how to write a minimal UDP/IP server, that listens on port 7777. Whenever a datagram is received it outputs its contents to STDOUT, unless it is precisely the string EXIT\n in which case the service exits. The service will react to SIGTERM and SIGINT and do a clean exit then. It also notifies the service manager about its completed startup, if it runs under a service manager. Finally, it sends watchdog keep-alive messages to the service manager if it asked for that, and if it runs under a service manager.

When run as systemd service this service's STDOUT will be connected to the logging framework of course, which means the service can act as a minimal UDP-based remote logging service.

To compile and link this example, save it as event-example.c, then run:

$ gcc event-example.c -o event-example `pkg-config --cflags --libs libsystemd`

For a first test, simply run the resulting binary from the command line, and test it against the following netcat command line:

$ nc -u localhost 7777

For the sake of brevity error checking is minimal, and in a real-world application should, of course, be more comprehensive. However, it hopefully gets the idea across how to write a daemon that reacts to external events with sd-event.

For further details on the functions used in the example above, please consult the manual pages: sd-event(3), sd_event_exit(3), sd_event_source_get_event(3), sd_event_default(3), sd_event_add_signal(3), sd_event_set_watchdog(3), sd_event_add_io(3), sd_notifyf(3), sd_event_loop(3), sd_event_source_unref(3), sd_event_unref(3).

Conclusion

So, is this the event loop to end all other event loops? Certainly not. I actually believe in "event loop plurality". There are many reasons for that, but most importantly: sd-event is supposed to be an event loop suitable for writing a wide range of applications, but it's definitely not going to solve all event loop problems. For example, while the priority logic is important for many usecase it comes with drawbacks for others: if not used carefully high-priority event sources can easily starve low-priority event sources. Also, in order to implement the priority logic, sd-event needs to linearly iterate through the event structures returned by epoll_wait(2) to sort the events by their priority, resulting in worst case O(n*log(n)) complexity on each event loop wakeup (for n = number of file descriptors). Then, to implement priorities fully, sd-event only dispatches a single event before going back to the kernel and asking for new events. sd-event will hence not provide the theoretically possible best scalability to huge numbers of file descriptors. Of course, this could be optimized, by improving epoll, and making it support how todays's event loops actually work (after, all, this is the problem set all event loops that implement priorities -- including GLib's -- have to deal with), but even then: the design of sd-event is focussed on running one event loop per thread, and it dispatches events strictly ordered. In many other important usecases a very different design is preferable: one where events are distributed to a set of worker threads and are dispatched out-of-order.

Hence, don't mistake sd-event for what it isn't. It's not supposed to unify everybody on a single event loop. It's just supposed to be a very good implementation of an event loop suitable for a large part of the typical usecases.

Note that our APIs, including sd-bus, integrate nicely into sd-event event loops, but do not require it, and may be integrated into other event loops too, as long as they support watching for time and I/O events.

And that's all for now. If you are considering using sd-event for your project and need help or have questions, please direct them to the systemd mailing list.

November 14, 2015

fwupd and DFU

For quite a long time fwupd has supported updating the system ‘BIOS’ using the UpdateCapsule UEFI mechanism. This open specification allows vendors provide a single update suitable for Windows and Linux, and the mechanism for applying it is basically the same for all vendors. Although there are only a few systems in the wild supporting capsule updates, a lot of vendors are planning new models next year, and a few of the major ones have been trialing the LVFS service for quite a while too. With capsule updates, fwupd and the LVFS we now have a compelling story for how to distribute and securely install system BIOS updates automatically.

It’s not such a rosy story for USB devices. In theory, everything should be using the DFU specification which has been endorsed by the USB consortium, but for a number of reasons quite a few vendors don’t use this. I’m guilty as charged for the ColorHug devices, as I didn’t know of the existance of DFU when designing the hardware. For ColorHug I just implemented a vendor-specific HID bootloader with a few custom commands as so many other vendors have done; it works well, but every vendor does things a slightly different way which needs having vendor specific update tools and fairly random firmware file formats.

With DFU, what’s supposed to happen is there are two modes for the device, a normal application runtime which is doing whatever the device is supposed to be doing, and another DFU mode which is really just an EEPROM programmer. By ‘detaching’ the application firmware using a special interface you can program the device and then return to normal operation.

So, what to do? For fwupd I want to ask vendors of removable hardware to implement DFU so that we don’t need to write code for each device type in fwupd. To make this a compelling prospect I’ve spent a good chunk of time of last week:

  • Creating a GObjectIntrospectable and cancellable host-side library called libdfu
  • Writing a reference GPLv3+ device-side implementation for a commonly used USB stack for PIC microcontrollers
  • Writing the interface code in fwupd to support DFU files wrapped in .cab files for automatic deployment

At the moment libdfu supports reading and writing raw, DFU and DfuSe file types, and supports reading and writing to DFU 1.1 devices. I’ve not yet implemented writing to ST devices (a special protocol extension invented by ST Microsystems) although that’s only because I’m waiting for someone to lend me a device with a STM32F107 included (e.g. DSO Nano). I’ve hopefully made the code flexible enough to make this possible without breaking API, although the libdfu library is currently private to fwupd until it’s had some proper review. You can of course use the dependable dfu-util tool to flash firmware, but this wasn’t suitable for use inside fwupd for various reasons.

Putting my money where my mouth is, I’ve converted the (not-yet-released) ColorHug+ bootloader and firmware to use DFU; excluding all the time I spent writing the m-stack patch and the libdfu support in fwupd it only took a couple of hours to build and test. Thanks to Christoph Brill, I’ll soon be getting some more hardware (a Neo FreeRunner) to verify this new firmware update mechanism on a real device with multiple implemented DFU interfaces. If anyone else has any DFU-capable hardware (especially Arduino-style devices) I’d be glad of any donations.

Once all this new code has settled down I’m going to be re-emailing a lot of the vendors who were unwilling to write vendor-specific code in fwupd. I’m trying to make the barrier to automatic updates on Linux as low as possible.

Comments welcome.

November 13, 2015

Insert Special Character in Spelling Dialog
LibreOffice 5.1 spelling dialog now has a little toolbar to enable inserting special characters into the spelling editing widget. Also Added paste, so the insert icon isn't lonely.

November 08, 2015

systemd.conf 2015 Summary

systemd.conf 2015 is Over Now!

Last week our first systemd.conf conference took place at betahaus, in Berlin, Germany. With almost 100 attendees, a dense schedule of 23 high-quality talks stuffed into a single track on just two days, a productive hackfest and numerous consumed Club-Mates I believe it was quite a success!

If you couldn't attend the conference, you may watch all talks on our YouTube Channel. The slides are available online, too.

Many photos from the conference are available on the Google Events Page. Enjoy!

I'd specifically like to thank Daniel Mack, Chris Kühl and Nils Magnus for running the conference, and making sure that it worked out as smoothly as it did! Thank you very much, you did a fantastic job!

I'd also specifically like to thank the CCC Video Operation Center folks for the excellent video coverage of the conference. Not only did they implement a live-stream for the entire talks part of the conference, but also cut and uploaded videos of all talks to our YouTube Channel within the same day (in fact, within a few hours after the talks finished). That's quite an impressive feat!

The folks from LinuxTag e.V. put a lot of time and energy in the organization. It was great to see how well this all worked out! Excellent work!

(BTW, LinuxTag e.V. and the CCC Video Operation Center folks are willing to help with the organization of Free Software community events in Germany (and Europe?). Hence, if you need an entity that can do the financial work and other stuff for your Free Software project's conference, consider pinging LinuxTag, they might be willing to help. Similar, if you are organizing such an event and are thinking about providing video coverage, consider pinging the the CCC VOC folks! Both of them get our best recommendations!)

I'd also like to thank our conference sponsors! Specifically, we'd like to thank our Gold Sponsors Red Hat and CoreOS for their support. We'd also like to thank our Silver Sponsor Codethink, and our Bronze Sponsors Pengutronix, Pantheon, Collabora, Endocode, the Linux Foundation, Samsung and Travelping, as well as our Cooperation Partners LinuxTag and kinvolk.io, and our Media Partner Golem.de.

Last but not least I'd really like to thank our speakers and attendees for presenting and participating in the conference. Of course, the conference we put together specifically for you, and we really hope you had as much fun at it as we did!

Thank you all for attending, supporting, and organizing systemd.conf 2015! We are looking forward to seeing you and working with you again at systemd.conf 2016!

Thanks!

November 06, 2015

Why improving kernel security is important
The Washington Post published an article today which describes the ongoing tension between the security community and Linux kernel developers. This has been roundly denounced as FUD, with Rob Graham going so far as to claim that nobody ever attacks the kernel.

Unfortunately he's entirely and demonstrably wrong, it's not FUD and the state of security in the kernel is currently far short of where it should be.

An example. Recent versions of Android use SELinux to confine applications. Even if you have full control over an application running on Android, the SELinux rules make it very difficult to do anything especially user-hostile. Hacking Team, the GPL-violating Italian company who sells surveillance software to human rights abusers, found that this impeded their ability to drop their spyware onto targets' devices. So they took advantage of the fact that many Android devices shipped a kernel with a flawed copy_from_user() implementation that allowed them to copy arbitrary userspace data over arbitrary kernel code, thus allowing them to disable SELinux.

If we could trust userspace applications, we wouldn't need SELinux. But we assume that userspace code may be buggy, misconfigured or actively hostile, and we use technologies such as SELinux or AppArmor to restrict its behaviour. There's simply too much userspace code for us to guarantee that it's all correct, so we do our best to prevent it from doing harm anyway.

This is significantly less true in the kernel. The model up until now has largely been "Fix security bugs as we find them", an approach that fails on two levels:

1) Once we find them and fix them, there's still a window between the fixed version being available and it actually being deployed
2) The forces of good may not be the first ones to find them

This reactive approach is fine for a world where it's possible to push out software updates without having to perform extensive testing first, a world where the only people hunting for interesting kernel vulnerabilities are nice people. This isn't that world, and this approach isn't fine.

Just as features like SELinux allow us to reduce the harm that can occur if a new userspace vulnerability is found, we can add features to the kernel that make it more difficult (or impossible) for attackers to turn a kernel bug into an exploitable vulnerability. The number of people using Linux systems is increasing every day, and many of these users depend on the security of these systems in critical ways. It's vital that we do what we can to avoid their trust being misplaced.

Many useful mitigation features already exist in the Grsecurity patchset, but a combination of technical disagreements around certain features, personality conflicts and an apparent lack of enthusiasm on the side of upstream kernel developers has resulted in almost none of it landing in the kernels that most people use. Kees Cook has proposed a new project to start making a more concerted effort to migrate components of Grsecurity to upstream. If you rely on the kernel being a secure component, either because you ship a product based on it or because you use it yourself, you should probably be doing what you can to support this.

Microsoft received entirely justifiable criticism for the terrible state of security on their platform. They responded by introducing cutting-edge security features across the OS, including the kernel. Accusing anyone who says we need to do the same of spreading FUD is risking free software being sidelined in favour of proprietary software providing more real-world security. That doesn't seem like a good outcome.

comment count unavailable comments
Gadget reviews
Not that I'm really running after more gadgets, but sometimes, there is a need that could only be soothed through new hardware.

Bluetooth UE roll

Got this for my wife, to play music when staying out on the quays of the Rhône, playing music in the kitchen (from a phone or computer), or when she's at the photo lab.

It works well with iOS, MacOS X and Linux. It's very easy to use, with whether it's paired, connected completely obvious, and the charging doesn't need specific cables (USB!).

I'll need to borrow it to add battery reporting for those devices though. You can find a full review on Ars Technica.

Sugru (!)

Not a gadget per se, but I bought some, used it to fix up a bunch of cables, repair some knickknacks, and do some DIY. Highly recommended, especially given the current price of their starter packs.

15-pin to USB Joystick adapter

It's apparently from Ckeyin, but you'll find the exact same box from other vendors. Made my old Gravis joystick work, in the hope that I can make it work with DOSBox and my 20-year old copy of X-Wing vs. Tie Fighter.

Microsoft Surface ARC Mouse

That one was given to me, for testing, works well with Linux. Again, we'll need to do some work to report the battery. I only ever use it when travelling, as the batteries last for absolute ages.

Logitech K750 keyboard

Bought this nearly two years ago, and this is one of my best buys. My desk is close to a window, so it's wireless but I never need to change the batteries or think about charging it. GNOME also supports showing the battery status in the Power panel.

Logitech T650 touchpad

Got this one in sale (17€), to replace my Logitech trackball (one of its buttons broke...). It works great, and can even get you shell gestures when run in Wayland. I'm certainly happy to have one less cable running across my desk, and reuses the same dongle as the keyboard above.

If you use more than one devices, you might be interested in this bug to make it easier to support multiple Logitech "Unifying" devices.

ClicLite charger

Got this from a design shop in Berlin. It should probably have been cheaper than what I paid for it, but it's certainly pretty useful. Charges up my phone by about 20%, it's small, and charges up at the same time as my keyboard (above).

Dell S2340T

Bought about 2 years ago, to replace the monitor I had in an all-in-one (Lenovo all-in-ones, never buy that junk).

Nowadays, the resolution would probably be considered a bit on the low side, and the touchscreen mesh would show for hardcore photography work. It's good enough for videos though and the speaker reaches my sitting position.

It's only been possible to use the USB cable for graphics for a couple of months, and it's probably not what you want to lower CPU usage on your machine, but it works for Fedora with this RPM I made. Talk to me if you can help get it into RPMFusion.

Shame about the huge power brick, but a little bonus for the builtin Ethernet adapter.

Surface 3

This is probably the biggest ticket item. Again, I didn't pay full price for it, thanks to coupons, rewards, and all. The work to getting Linux and GNOME to play well with it is still ongoing, and rather slow.

I won't comment too much on Windows either, but rather as what it should be like once Linux runs on it.

I really enjoy the industrial design, maybe even the slanted edges, but one as to wonder why they made the USB power adapter not sit flush with the edge when plugged in.

I've used it a couple of times (under Windows, sigh) to read Pocket as I do on my iPad 1 (yes, the first one), or stream videos to the TV using Flash, without the tablet getting hot, or too slow either. I also like the fact that there's a real USB(-A) port that's separate from the charging port. The micro SD card port is nicely placed under the kickstand, hard enough to reach to avoid it escaping the tablet when lugged around.

The keyboard, given the thickness of it, and the constraints of using it as a cover, is good enough for light use, when travelling for example, and the layout isn't as awful as on, say, a Thinkpad Carbon X1 2nd generation. The touchpad is a bit on the small side though it would have been hard to make it any bigger given the cover's dimensions.

I would however recommend getting a Surface Pro if you want things to work right now (or at least soon). The one-before-last version, the Surface Pro 3, is probably a good target.

November 05, 2015

attribute((cleanup)), mixed declarations and code, and goto.

One of the cool features of recent GLib is g_autoptr() and g_autofree. It’s liberating to be able to write:

g_autofree char *filename = g_strdup_printf("%s/%d.txt", dir, count);

And be sure that will be freed no matter how your function returns. But as I started to use it, I realized that I wasn’t very sure about some details about the behavior, especially when combined with mixing declarations and code as allowed by C99.

Internally g_autofree uses __attribute__((cleanup)), which is supported by GCC and clang. The definition of g_autofree is basically:

static inline void
g_autoptr_cleanup_generic_gfree (void *p)
{
  void **pp = (void**)p;
  g_free (*pp);
}

#define g_autofree __attribute__((cleanup(g_autoptr_cleanup_generic_gfree)))

Look at the following examples:

int count1(int arg)
{
  g_autofree char *str;

  if (arg < 0)
    return -1;

  str = g_strdup_printf("%d", arg);

  return strlen(str);
}


int count2(int arg)
{
  if (arg < 0)
    return -1;

  g_autofree char *str = g_strdup_printf("%d", arg);

  return strlen(str);
}

int count3(int arg)
{
  if (arg < 0)
    goto out;

  g_autofree char *str = g_strdup_printf("%d", arg);

  return strlen(str);

out:
  return -1;
}

int count4(int arg)
{
  if (arg < 0)
    goto out;

  {
    g_autofree char *str = g_strdup_printf("%d", arg);

    return strlen(str);
  }

out:
  return 0;
}

Which ones of these do you think work as intended, and which ones are buggy? (I’m not recommending this as a way counting the digits in a number – the example is artificial.)

count1() is pretty clearly buggy – the cleanup function will run in the error
path and try to free an uninitialized string. Slightly, more subtly, count3() is also buggy – because the goto jumps over the initialization. But count2() and count4() work as intended.

To understand why this is the case, it’s worth looking at how attribute((cleanup)) is described in the GCC manual – all it says is “the ‘cleanup’ attribute runs a function when the variable goes out of scope.” I first thought that this was a completely insufficient definition – not complete enough to allow figuring out what was supposed to happen in the above cases, but thinking about it a bit, it’s actually a precise definition.

To recall, the scope of a variable in C is from the point of the declaration of the variable to the end of the enclosing block. What the definition is saying is that any time a variable is in scope, and then goes out of scope, there is an implicit call to the cleanup function.

In the early return in count1() and at the return that is jumped to in count3(), the variable ‘str’ is in scope, so the cleanup function will be called, even though the variable is not initialized in either case. In the corresponding places in count2() and count4() the variable ‘str’ is not in scope, so the cleanup function will not be called.

The coding style takewaways from this are 1) Don’t use the g_auto* attributes on a variable that is not initialzed at the time of definition 2) be very careful if combining goto with g_auto*.

It should be noted that GCC is quite good at warning about it if you get it wrong, but it’s still better to understand the rules and get it right from the start.


October 30, 2015

C.H.I.P. flashing on Fedora
You might have heard of the C.H.I.P., the 9$ computer. After contributing to their Kickstarter, and with no intent on hacking on more kernel code than is absolutely necessary, I requested the "final" devices, when chumps like me can read loads of docs and get accessories for it easily.

Turns out that our old friend the Realtek 8723BS chip is the Wi-Fi/Bluetooth chip in the nano computer. NextThingCo got in touch, and sent me a couple of early devices (as well as to the "Kernel hacker" backers), with their plan being to upstream all the drivers and downstream hacks into the upstream kernel.

Before being able to hack on the kernel driver though, we'll need to get some software on it, and find a way to access it. The docs website has instructions on how to flash the device using Ubuntu, but we don't use that here.

You'll need a C.H.I.P., a jumper cable, and the USB cable you usually use for charging your phone/tablet/e-book reader.

First, let's install a few necessary packages:


dnf install -y sunxi-tools uboot-tools python3-pyserial moserial

You might need other things, like git and gcc, but I kind of expect you to already have that installed if you're software hacking. You will probably also need to get sunxi-tools from Koji to get a new enough version that will support the C.H.I.P.

Get your jumper cable out, and make the connection as per the NextThingCo docs. I've copied the photo from the docs to keep this guide stand-alone.



Let's install the tools, modified to work with Fedora's newer, upstreamer, version of the sunxi-tools.


$ git clone https://github.com/hadess/CHIP-tools.git
$ cd CHIP-tools
$ make
$ sudo ./chip-update-firmware.sh -d

If you've followed the instructions, you haven't plugged in the USB cable yet. Plug in the USB cable now, to the micro USB power supply on one end, and to your computer on the other.

You should see the little "OK" after the "waiting for fel" message:


== upload the SPL to SRAM and execute it ==
waiting for fel........OK

At this point, you can unplug the jumper cable, something not mentioned in the original docs. If you don't do that, when the device reboots, it will reboot in flashing mode again, and we obviously don't want that.

At this point, you'll just need to wait a while. It will verify the installation when done, and turn off the device. Unplug, replug, and launch moserial as root. You should be able to access the C.H.I.P. through /dev/ttyACM0 with a baudrate of 115200. The root password is "chip".

Obligatory screenshot of our new computer:


Next step, testing out our cleaned up Realtek driver, Fedora on the C.H.I.P., and plenty more.

October 23, 2015

finding UI crashes by fuzzing input events with american fuzzy lop
As mentioned previously I've been experimenting using afl as a fuzzing engine to fuzz a stream of serialized keyboard events which LibreOffice reads and dispatches.

Performance is still pretty poor, but by tweaking our headless mode to allow dialogs to be created, then using that headless mode for eventtesting and then hacking out the actual rendering of the UI to the headless backend I've got something that performs reasonably well enough to enable me to set a far higher limit of 50 input events per cycle and start to discover real bugs in impress.
  1. 5.1 only crash in impress sidebar under some circumstances
  2. null marked obj still in impress mark cache
  3. another null deref in impress sidebar panels
  4. crash in impress if you exit while the annotation window is open
  5. divide by zero in an impress sidebar panel
  6. another annotation window null deref issue
  7. crash on sending a keystroke to an empty impress page list widget
  8. missing dispose on annotation windows
  9. missing dispose on alive by unshown panels
  10. crash if frame is destroyed before keystroke gets to it
  11. crash if you close impress main frame while slideshow is running

all of which is encouraging, though some of these are possibly very unlikely in real world use. But the prized find is

intermittent crash on undo of insert slide

because I've seen that happen plenty in the real world, and is the problem I was hoping to find.

Turns out its been so difficult to track down because there's a timer involved which is triggered by earlier modifications to the document. To get it to crash by undoing insert slide you have to modify an object in the document, which triggers an object-modified timer, and then very quickly, before the modified-timer fires, undo insert slide. Which has the effect of impress not registering that the slide has been deleted. Some time later, in far away code, impress will crash on use of the deleted slide.

afl-eventtesting was able to find a sequence of keystrokes (which isn't a huge surprise seeing as I primed it with some insert and undo slide sequences so it didn't have to mutate things too far before it modified a document object after an insert and before the undo) to trigger the crash and the eventtesting + headless modes of LibreOffice gave a reproducible platform where the same events happen in the same sequence without any intermediate system-ui events to throw off the delicate timing. Once the thing is reliably reproducible then its just a matter of grinding through the debugging.


October 18, 2015

Second Round of systemd.conf 2015 Sponsors

Second Round of systemd.conf 2015 Sponsors

We are happy to announce the second round of systemd.conf 2015 sponsors! In addition to those from the first announcement, we have:

Our second Gold sponsor is Red Hat!

What began as a better way to build software—openness, transparency, collaboration—soon shifted the balance of power in an entire industry. The revolution of choice continues. Today Red Hat® is the world's leading provider of open source solutions, using a community-powered approach to provide reliable and high-performing cloud, virtualization, storage, Linux®, and middleware technologies.

A Bronze sponsor is Samsung:

From the beginning we have established a very fast pace and are currently one of the biggest and fastest growing modern-technology R&D centers in East-Central Europe. We have started with designing subsystems for digital satellite television, however, we have quickly expanded the scope of our interest. Currently, it includes advanced systems of digital television, platform convergence, mobile systems, smart solutions, and enterprise solutions. Also a vital role in our activity plays the quality and certification center, which controls the conformity of Samsung Electronics products with the highest standards of quality and reliability.

A Bronze sponsor is travelping:

Travelping is passionate about networks, communications and devices. We empower our customers to deploy and operate networks using our state of the art products, solutions and services. Our products and solutions are based on our industry proven physical and virtual appliance platforms. These purpose built platforms ensure best in class performance, scalability and reliability combined with consistent end to end management capabilities. To build this products, Travelping has developed a own embedded, cross platform Linux distribution called CAROS.io which incorporates the systemd service manager and tools.

A Bronze sponsor is Collabora:

Collabora has over 10 years of experience working with top tier OEMs & silicon manufacturers worldwide to develop products based on Open Source software. Through the use of Open Source technologies and methodologies, Collabora helps clients in multiple market segments gain faster time to market and save millions of dollars in licensing and maintenance costs. Collabora has already brought to market several products relying on systemd extensively.

A Bronze sponsor is Endocode:

Endocode AG. An employee-owned, software engineering company from Berlin. Open Source is our heart and soul.

A Bronze sponsor is the Linux Foundation:

The Linux Foundation advances the growth of Linux and offers its collaborative principles and practices to any endeavor.

We are Cooperating with LinuxTag e.V. on the organization:

LinuxTag is Europe's leading organizer of Linux and Open Source events. Born of the community and in business for 20 years, we organize LinuxTag, an annual conference and exhibition attracting thousands of visitors. We also participate and cooperate in organizing workshops, tutorials, seminars, and other events together with and for the Open Source community. Selected events include non-profit workshops, the German Kernel Summit at FrOSCon, participation in the Open Tech Summit, and others. We take care of the organizational framework of systemd.conf 2015. LinuxTag e.V. is a non-profit organization and welcomes donations of ideas and workforce.

A Media Partner is Golem:

Golem.de is an up to date online-publication intended for professional computer users. It provides technology insights of the IT and telecommunications industry. Golem.de offers profound and up to date information on significant and trending topics. Online- and IT-Professionals, marketing managers, purchasers, and readers inspired by technology receive substantial information on product, market and branding potentials through tests, interviews und market analysis.

We'd like to thank our sponsors for their support! Without sponsors our conference would not be possible!

The Conference s SOLD OUT since a few weeks. We no longer accept registrations, nor paper submissions.

For further details about systemd.conf consult the conference website.

See the the first round of sponsor announcements!

See you in Berlin!

October 15, 2015

fuzzing LibreOffice input events with american fuzzy lop

<iframe allowfullscreen="" frameborder="0" height="344" src="https://www.youtube.com/embed/x3YwRaUh0ek" width="459"></iframe>

This is a demo of using the american fuzzy lop fuzzer as an engine to drive LibreOffice UI testing by fuzzing serialized keyboard input events.

For demo purposes the UI is visible here, but it can be run headlessly too. Given enough time afl can fuzz the initial input text of hello into keyboard shortcuts to enter menus and launch dialogs.

I think the concept is pretty neat and fun, but this is still experimental stage stuff that I'm playing with and trying to up the performance.

October 13, 2015

view/edit shape control points
In LibreOffice 5.1 the shape properties dialog for enhanced shapes now lists and enables editing the control points. This is in addition to the preexisting mechanism of selecting with the mouse the yellow control handle of the shape, but enables viewing and fine control over the control values

October 12, 2015

Star ratings in GNOME Software

A long time ago, GNOME software used to show star ratings as popularity next to the application using the fedora-tagger application. This wasn’t a good idea for several reasons:

  • People can’t agree on a scale. Is an otherwise flawless application with one translation issue 5 stars or 4? Is a useful computational fluid dynamics application that crashes on startup but can be run manually on the command line 1 star or 3 stars?
  • It only worked on Fedora, and there was no real policy on how to share data, or the privacy implications of clicking a star
  • People could “game” the ratings system, for example hardcore KDE users could go through all the GNOME apps and give then one star. We then limited this to only rate applications that you have installed, but it was really a cat and mouse thing.

So, lets go two steps back. What is the star rating trying to convey to the user? When I look at a star rating, I want to see a proportional number of stars to how awesome it is to me. The rest of this blog tries to define awesomeness.

As part of the AppStream generation process we explode various parts of the distro binary package and try to build metadata by merging various sources together, for example AppData, desktop files and icons. As part of this we also have access to the finished binary and libraries, and so can also run tools on them to get a metric of awesomeness. So far, the metrics of awesomeness (here-on known as “kudos”) are:

  • AppMenu — has an application menu in line with the GNOME 3 HIG
  • HiDpiIcon — installs a 128×128 or larger application icon
  • HighContrast — installs hicontrast icons for visually impaired users
  • ModernToolkit — uses a modern toolkit like Gtk-3 or QT-5
  • Notifications — registers desktop notifications
  • SearchProvider — provides a search provider for GNOME Shell or KDE Plasma
  • UserDocs — provides user documentation

These attempt to define how tightly the application is integrated with the platform, which is usually a pretty good metric of awesomeness. Of course, some applications like Blender are an island in terms of integration, but of course are awesome. We still need new ideas for this, so ideas are very much welcome.

There are some other “run-time” kudos used as well. These are not encoded by the builder as they require some user information or are too specific to GNOME Software. These include:

  • FeaturedRecommended — One of the GNOME Software design team chose to feature this
  • HasKeywords — there are keywords in the desktop file used for searching
  • HasScreenshots — more than one screenshot is supplied
  • MyLanguage — has a populated translation in my locale, or a locale fallback
  • PerfectScreenshots — screenshots are perfectly sized, in 16:9 aspect
  • Popular — lots of people have downloaded this (only available on Fedora)
  • RecentRelease — there been an upstream release in the last year

When added together, the number of stars will correspond roughtly to the number of kudos the application has.

You can verify the kudos your application is getting by doing something like:

killall gnome-software
gnome-software --verbose

and then navigating to the details for an application you’ll see on the console:

 id-kind:         desktop
 state:           available
 id:              blender.desktop
 kudo:            recent-release
 kudo:            featured-recommended
 kudo:            has-screenshots
 kudo:            popular
 kudo-percentage: 60

Comments (as always) are welcome, as are new ideas on how to test for awesomeness.

Boston GNOME Summit update

The Boston GNOME summit this year is a small, focused hackfest.

Breakfast

The first day was filled with discussions and planning, with one of the central topics being how to make gnome-builder, xdg-app and gnome-continuous play well together. You can find notes and conclusions from this discussion here.

Topics

In the afternoon, Christian Hergert gave us an extensive demo of gnome-builder, with led to lots of discussion around future plans.

The second day was entirely devoted to hacking. Everybody got something done:

  • Cosimo added support for operating with elevated privileges to gvfs.  Here is a screencast showing this in action:
    <video class="wp-video-shortcode" controls="controls" height="267" id="video-1475-2" preload="metadata" width="474"><source src="https://blogs.gnome.org/mclasen/files/2015/10/Screencast-from-10-11-2015-05_09_42-PM.webm?_=2" type="video/webm">https://blogs.gnome.org/mclasen/files/2015/10/Screencast-from-10-11-2015-05_09_42-PM.webm</video>

    The code for this is already in gvfs git.

  • Owen debugged and fixed various issues with gnome-builder under Wayland: mispositioned code completion popups and window stacking problems with the project chooser.
  • Christian asked everybody in the room what missing features kept them from using gnome-builder, and implemented several of the answers he got. One example is better indentation support in both emacs and vi modes.
  • Giovanni added support to the gnome-continuous build machinery to produce xdg-app bundles, and used it to produce an xdg-app for gnome-weather.
  • Myself, I took the shortcuts overlay implementation from gnome-builder and adapted it for GTK+.
    Shortctus overlay
    This code can be found in gtk+ git.

Thanks to everybody who participated so far. The Summit will continue tomorrow with more discussion and hopefully more productive hacking.

I’d also like to thank  Walter Bender and the MIT for hosting us, and Red Hat for sponsoring our breakfasts.

October 06, 2015

Going my own way
Reaction to Sarah's post about leaving the kernel community was a mixture of terrible and touching, but it's still one of those things that almost certainly won't end up making any kind of significant difference. Linus has made it pretty clear that he's fine with the way he behaves, and nobody's going to depose him. That's unfortunate, because earlier today I was sitting in a presentation at Linuxcon and remembering how much I love the technical side of kernel development. "Remembering" is a deliberate choice of word - it's been increasingly difficult to remember that, because instead I remember having to deal with interminable arguments over the naming of an interface because Linus has an undying hatred of BSD securelevel, or having my name forever associated with the deepthroating of Microsoft because Linus couldn't be bothered asking questions about the reasoning behind a design before trashing it.

In the end it's a mixture of just being tired of dealing with the crap associated with Linux development and realising that by continuing to put up with it I'm tacitly encouraging its continuation, but I can't be bothered any more. And, thanks to the magic of free software, it turns out that I can avoid putting up with the bullshit in the kernel community and get to work on the things I'm interested in doing. So here's a kernel tree with patches that implement a BSD-style securelevel interface. Over time it'll pick up some of the power management code I'm still working on, and we'll see where it goes from there. But, until there's a significant shift in community norms on LKML, I'll only be there when I'm being paid to be there. And that's improved my mood immeasurably.

(Edited to add a context link for the "deepthroating of Microsoft" reference)

comment count unavailable comments

September 30, 2015

impress save background image
Impress has a "Set Background Image" option in its slide context menu for a while. For 5.1 I've added a matching "Save Background Image" to save the current background image to file.

September 25, 2015

crash testing and coverity, conference report
Slides for this morning's Crash Testing and Coverity numbers presentation. Summary, all ok, numbers ~0. If I'm analysing this right, then the highest quality is achieved at the height of the holiday season.
vertical text columns preview
My small contribution to last night's LibreOffice conference hack-fest. In vertical text mode, the column view for pages now previews in the correct direction.
Philips Wireless, modernised
I've wanted a stand-alone radio in my office for a long time. I've been using a small portable radio, but it ate batteries quickly (probably a 4-pack of AA for a bit less of a work week's worth of listening), changing stations was cumbersome (hello FM dials) and the speaker was a bit teeny.

A couple of years back, I had a Raspberry Pi-based computer on pre-order (the Kano, highly recommended for kids, and beginners) through a crowd-funding site. So I scoured « brocantes » (imagine a mix of car boot sale and antiques fair, in France, with people emptying their attics) in search of a shell for my small computer. A whole lot of nothing until my wife came back from a week-end at a friend's with this:


Photo from Radio Historia

A Philips Octode Super 522A, from 1934, when SKUs were as superlative-laden and impenetrable as they are today.

Let's DIY



I started by removing the internal parts of the radio, without actually turning it on. When you get such old electronics, they need to be checked thoroughly before being plugged, and as I know nothing about tube radios, I preferred not to. And FM didn't exist when this came out, so not sure what I would have been able to do with it anyway.



Roomy, and dirty. The original speaker was removed, the front buttons didn't have anything holding them any more, and the nice backlit screen went away as well.

To replace the speaker, I went through quite a lot of research, looking for speakers that were embedded, rather than get a speaker in box that I would need to extricate from its container. Visaton make speakers that can be integrated into ceiling, vehicles, etc. That also allowed me to choose one that had a good enough range, and would fit into the one hole in my case.

To replace the screen, I settled on an OLED screen that I knew would work without too much work with the Raspberry Pi, a small AdaFruit SSD1306 one. Small amount of soldering that was up to my level of skills.

It worked, it worked!

Hey, soldering is easy. So because of the size of the speaker I selected, and the output power of the RPi, I needed an amp. The Velleman MK190 kit was cheap (€10), and should just be able to work with the 5V USB power supply I planned to use. Except that the schematics are really not good enough for an electronics starter. I spent a couple of afternoons verifying, checking on the Internet for alternate instructions, re-doing the solder points, to no avail.

'Sup Tiga!

So much wasted time, and got a cheap car amp with a power supply. You can probably find cheaper.

Finally, I got another Raspberry Pi, and SD card, so that the Kano, with its super wireless keyboard, could find a better home (it went to my godson, who seemed to enjoy the early game of Pong, and being a wizard).

Putting it all together

We'll need to hold everything together. I got a bit of help for somebody with a Dremel tool for the piece of wood that will hold the speaker, and another one that will stick three stove bolts out of the front, to hold the original tuning, mode and volume buttons.

A real joiner

I fast-forwarded the machine by a couple of years with a « Philips » figure-of-8 plug at the back, so machine's electrics would be well separated from the outside.



Screws into the side panel for the amp, blu-tack to hold the OLED screen for now, RPi on a few leftover bits of wood.

Software

My first attempt at getting something that I could control on this small computer was lcdgrilo. Unfortunately, I would have had to write a Web UI for it (remember, my buttons are just stuck on, for now at least), and probably port the SSD1306 OLED screen's driver from Python, so not a good fit.

There's no proper Fedora support for Raspberry Pis, and while one can use a nearly stock Debian with a few additional firmware files on Raspberry Pis, Fedora chose not to support that slightly older SoC at all, which is obviously disappointing for somebody working on Fedora as a day job.

Looking for other radio retrofits, and there are plenty of quality ones on the Internet, and for various connected speakers backends, I found PiMusicBox. It's a Debian variant with Mopidy builtin, and a very easy to use initial setup: edit a settings file on the SD card image, boot and access the interface via a browser. Tada!

Once I had tested playback, I lowered the amp's volume to nearly zero, raised the web UI's volume to the maximum, and raised the amp's volume to the maximum bearable for the speaker. As I won't be able to access the amp's dial, we'll have this software only solution.

Wrapping up

I probably spent a longer time looking for software and hardware than actually making my connected radio, but it was an enjoyable couple of afternoons of work, and the software side isn't quite finished.

First, in terms of hardware support, I'll need to make this OLED screen work, how lazy of me. The audio setup is currently just the right speaker, as I'd like both the radios and AirPlay streams to be downmixed.

Secondly, Mopidy supports plugins to extend its sources, uses GStreamer, so would be a right fit for Grilo, making it easier for Mopidy users to extend through Lua.

Do note that the Raspberry Pi I used is a B+ model. For B models, it's recommended to use a separate DAC, because of the bad audio quality, even if the B+ isn't that much better. Testing out use the HDMI output with an HDMI to VGA+jack adapter might be a way to cut costs as well.

Possible improvements could include making the front-facing dials work (that's going to be a tough one), or adding RFID support, so I can wave items in front of it to turn it off, or play a particular radio.

In all, this radio cost me:
- 10 € for the radio case itself
- 36.50 € for the Raspberry Pi and SD card (I already had spare power supplies, and supported Wi-Fi dongle)
- 26.50 € for the OLED screen plus various cables
- 20 € for the speaker
- 18 € for the amp
- 21 € for various cables, bolts, planks of wood, etc.

I might also count the 14 € for the soldering iron, the 10 € for the Velleman amp, and about 10 € for adapters, cables, and supplies I didn't end up using.

So between 130 and 150 €, and a number of afternoons, but at the end, a very flexible piece of hardware that didn't really stretch my miniaturisation skills, and a completely unique piece of furniture.



In the future, I plan on playing with making my own 3-button keyboard, and making a remote speaker to plug in the living room's 5.1 amp with a C.H.I.P computer.

Happy hacking!

September 24, 2015

Filling in the holes in Linux boot chain measurement, and the TPM measurement log
When I wrote about TPM attestation via 2FA, I mentioned that you needed a bootloader that actually performed measurement. I've now written some patches for Shim and Grub that do so.

The Shim code does a couple of things. The obvious one is to measure the second-stage bootloader into PCR 9. The perhaps less expected one is to measure the contents of the MokList and MokSBState UEFI variables into PCR 14. This means that if you're happy simply running a system with your own set of signing keys and just want to ensure that your secure boot configuration hasn't been compromised, you can simply seal to PCR 7 (which will contain the UEFI Secure Boot state as defined by the UEFI spec) and PCR 14 (which will contain the additional state used by Shim) and ignore all the others.

The grub code is a little more complicated because there's more ways to get it to execute code. Right now I've gone for a fairly extreme implementation. On BIOS systems, the grub stage 1 and 2 will be measured into PCR 9[1]. That's the only BIOS-specific part of things. From then on, any grub modules that are loaded will also be measured into PCR 9. The full kernel image will be measured into PCR10, and the full initramfs will be measured into PCR11. The command line passed to the kernel is in PCR12. Finally, each command executed by grub (including those in the config file) is measured into PCR 13.

That's quite a lot of measurement, and there are probably fairly reasonable circumstances under which you won't want to pay attention to all of those PCRs. But you've probably also noticed that several different things may be measured into the same PCR, and that makes it more difficult to figure out what's going on. Thankfully, the spec designers have a solution to this in the form of the TPM measurement log.

Rather than merely extending a PCR with a new hash, software can extend the measurement log at the same time. This is stored outside the TPM and so isn't directly cryptographically protected. In the simplest form, it contains a hash and some form of description of the event associated with that hash. If you replay those hashes you should end up with the same value that's in the TPM, so for attestation purposes you can perform that verification and then merely check that specific log values you care about are correct. This makes it possible to have a system perform an attestation to a remote server that contains a full list of the grub commands that it ran and for that server to make its attestation decision based on a subset of those.

No promises as yet about PCR allocation being final or these patches ever going anywhere in their current form, but it seems reasonable to get them out there so people can play. Let me know if you end up using them!

[1] The code for this is derived from the old Trusted Grub patchset, by way of Sirrix AG's Trusted Grub 2 tree.

comment count unavailable comments

September 23, 2015

GNOME 3.18, here we go
As I'm known to do, a focus on the little things I worked on during the just released GNOME 3.18 development cycle.

Hardware support

The accelerometer support in GNOME now uses iio-sensor-proxy. This daemon also now supports ambient light sensors, which Richard used to implement the automatic brightness adjustment, and compasses, which are used in GeoClue and gnome-maps.

In kernel-land, I've fixed the detection of some Bosch accelerometers, added support for another Kyonix one, as used in some tablets.

I've also added quirks for out-of-the-box touchscreen support on some cheaper tablets using the goodix driver, and started reviewing a number of patches for that same touchscreen.

With Larry Finger, of Realtek kernel drivers fame, we've carried on cleaning up the Realtek 8723BS driver used in the majority of Windows-compatible tablets, in the Endless computer, and even in the $9 C.H.I.P. Linux computer.

Bluetooth UI changes



The Bluetooth panel now has better « empty states », explaining how to get Bluetooth working again when a hardware killswitch is used, or it's been turned off by hand. We've also made receiving files through OBEX Push easier, and builtin to the Bluetooth panel, so that you won't forget to turn it off when done, and won't have trouble finding it, as is the case for settings that aren't used often.


Videos

GNOME Videos has seen some work, mostly in the stabilisation, and bug fixing department, most of those fixes were also landed in the 3.16 version.

We've also been laying the groundwork in grilo for writing ever less code in C for plugin sources. Grilo Lua plugins can now use gnome-online-accounts to access keys for specific accounts, which we've used to re-implement the Pocket videos plugin, as well as the Last.fm cover art plugin.

All those changes should allow implementing OwnCloud support in gnome-music in GNOME 3.20.

My favourite GNOME 3.18 features

You can call them features, or bug fixes, but the overall improvements in the Wayland and touchpad/touchscreen support are pretty exciting. Do try it out when you get a GNOME 3.18 installation, and file bugs, it's coming soon!

Talking of bug fixes, this one means that I don't need to put in my password by hand when I want to access work related resources. Connect to the VPN, and I'm authenticated to Kerberos.

I've also got a particular attachment to the GeoClue GPS support through phones. This allows us to have more accurate geolocation support than any desktop environments around.

A few for later

The LibreOfficeKit support that will be coming to gnome-documents will help us get support for EPubs in gnome-books, as it will make it easier to plug in previewers other than the Evince widget.

Victor Toso has also been working through my Grilo bugs to allow us to implement a preview page when opening videos. Work has already started on that, so fingers crossed for GNOME 3.20!

September 22, 2015

systemd.conf close to being sold out!

Only 14 tickets still available!

systemd.conf 2015 is close to being sold out, there are only 14 tickets left now. If you haven't bought your ticket yet, now is the time to do it, because otherwise it will be too late and all tickets will be gone!

Why attend? At this conference you'll get to meet everybody who is involved with the systemd project and learn what they are working on, and where the project will go next. You'll hear from major users and projects working with systemd. It's the primary forum where you can make yourself heard and get first hand access to everybody who's working on the future of the core Linux userspace!

To get an idea about the schedule, please consult our preliminary schedule.

In order to register for the conference, please visit the registration page.

We are still looking for sponsors. If you'd like to join the ranks of systemd.conf 2015 sponsors, please have a look at our Becoming a Sponsor page!

For further details about systemd.conf consult the conference website.