Fedora desktop Planet

libinput beginner project - disabling touchpads on lid close

Posted by Peter Hutterer on December 07, 2016 10:49 PM

Update: Dec 08 2016: someone's working on this project. Sorry about the late update, but feel free to pick other projects you want to work on.

Interested in hacking on some low-level stuff and implementing a feature that's useful to a lot of laptop owners out there? We have a feature on libinput's todo list but I'm just constantly losing my fight against the ever-growing todo list. So if you already know C and you're interested in playing around with some low-level bits of software this may be the project for you.

Specifically: within libinput, we want to disable certain devices based on a lid state. In the first instance this means that when the lid switch is toggled to closed, the touchpad and trackpoint get silently disabled to not send events anymore. [1] Since it's based on a switch state, this also means that we'll now have to listen to switch events and expose those devices to libinput users.

The things required to get all this working are:

  • Designing a switch interface plus the boilerplate code required (I've done most of this bit already)
  • Extending the current evdev backend to handle devices with EV_SW and exposing their events
  • Hooking up the switch devices to internal touchpads/trackpoints to disable them ad-hoc
  • Handle those devices where lid switch is broken in the hardware (more details on this when we get to this point)

You get to dabble with libinput and a bit of udev and the kernel. Possibly Xorg stuff, but that's unlikely at this point. This project is well suited for someone with a few spare weekends ahead. It's great for someone who hasn't worked with libinput before, but it's not a project to learn C, you better know that ahead of time. I'd provide the mentoring of course (I'm in UTC+10, so expect IRC/email). If you're interested let me know. Riches and fame may happen but are not guaranteed.

[1] A number of laptops have a hw issue where either device may send random events when the lid is closed

xinput is not a configuration UI

Posted by Peter Hutterer on December 07, 2016 02:58 AM

xinput is a tool to query and modify X input device properties (amongst other things). Every so-often someone-complains about it's non-intuitive interface, but this is where users are mistaken: xinput is a not a configuration UI. It is a DUI - a developer user interface [1] - intended to test things without having to write custom (more user-friendly) for each new property. It is nothing but a tool to access what is effectively a key-value store. To use it you need to know not only the key name(s) but also the allowed formats, some of which are only documented in header files. It is intended to be run under user supervision, anything it does won't survive device hotplugging. Relying on xinput for configuration is the same as relying on 'echo' to toggle parameters in /sys for kernel configuration. It kinda possibly maybe works most of the time but it's not pretty. And it's not intended to be, so please don't complain to me about the arcane user interface.

[1] don't do it, things will be a bit confusing, you may not do the right thing, you can easily do damage, etc. A lot of similarities... ;)

Avoiding CVE-2016-8655 with systemd

Posted by Lennart Poettering on December 06, 2016 11:00 PM

Avoiding CVE-2016-8655 with systemd

Just a quick note: on recent versions of systemd it is relatively easy to block the vulnerability described in CVE-2016-8655 for individual services.

Since systemd release v211 there's an option RestrictAddressFamilies= for service unit files which takes away the right to create sockets of specific address families for processes of the service. In your unit file, add RestrictAddressFamilies=~AF_PACKET to the [Service] section to make AF_PACKET unavailable to it (i.e. a blacklist), which is sufficient to close the attack path. Safer of course is a whitelist of address families whch you can define by dropping the ~ character from the assignment. Here's a trivial example:

…
[Service]
ExecStart=/usr/bin/mydaemon
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
…

This restricts access to socket families, so that the service may access only AF_INET, AF_INET6 or AF_UNIX sockets, which is usually the right, minimal set for most system daemons. (AF_INET is the low-level name for the IPv4 address family, AF_INET6 for the IPv6 address family, and AF_UNIX for local UNIX socket IPC).

Starting with systemd v232 we added RestrictAddressFamilies= to all of systemd's own unit files, always with the minimal set of socket address families appropriate.

With the upcoming v233 release we'll provide a second method for blocking this vulnerability. Using RestrictNamespaces= it is possible to limit which types of Linux namespaces a service may get access to. Use RestrictNamespaces=yes to prohibit access to any kind of namespace, or set RestrictNamespaces=net ipc (or similar) to restrict access to a specific set (in this case: network and IPC namespaces). Given that user namespaces have been a major source of security vulnerabilities in the past months it's probably a good idea to block namespaces on all services which don't need them (which is probably most of them).

Of course, ideally, distributions such as Fedora, as well as upstream developers would turn on the various sandboxing settings systemd provides like these ones by default, since they know best which kind of address families or namespaces a specific daemon needs.

New udev property: XKB_FIXED_LAYOUT for keyboards that must not change layouts

Posted by Peter Hutterer on December 06, 2016 02:44 AM

This post mostly affects developers of desktop environments/Wayland compositors. A systemd pull request was merged to add two new properties to some keyboards: XKB_FIXED_LAYOUT and XKB_FIXED_VARIANT. If set, the device must not be switched to a user-configured layout but rather the one set in the properties. This is required to make fake keyboard devices work correctly out-of-the-box. For example, Yubikeys emulate a keyboard and send the configured passwords as key codes matching a US keyboard layout. If a different layout is applied, then the password may get mangled by the client.

Since udev and libinput are sitting below the keyboard layout there isn't much we can do in this layer. This is a job for those parts that handle keyboard layouts and layout configurations, i.e. GNOME, KDE, etc. I've filed a bug for gnome here, please do so for your desktop environment.

If you have a device that falls into this category, please submit a systemd patch/file a bug and cc me on it (@whot).

The future of xinput, xmodmap, setxkbmap, xsetwacom and other tools under Wayland

Posted by Peter Hutterer on December 05, 2016 08:42 PM

This post applies to most tools that interface with the X server and change settings in the server, including xinput, xmodmap, setxkbmap, xkbcomp, xrandr, xsetwacom and other tools that start with x. The one word to sum up the future for these tools under Wayland is: "non-functional".

An X window manager is little more than an innocent bystander when it comes to anything input-related. Short of handling global shortcuts and intercepting some mouse button presses (to bring the clicked window to the front) there is very little a window manager can do. It's a separate process to the X server and does not receive most input events and it cannot affect what events are being generated. When it comes to input device configuration, any X client can tell the server to change it - that's why general debugging tools like xinput work.

A Wayland compositor is much more, it is a window manager and the display server merged into one process. This gives the compositor a lot more power and responsibility. It handles all input events as they come out of libinput and also manages device's configuration. Oh, and instead of the X protocol it speaks Wayland protocol.

The difference becomes more obvious when you consider what happens when you toggle a setting in the GNOME control center. In both Wayland and X, the control center toggles a gsettings key and waits for some other process to pick it up. In both cases, mutter gets notified about the change but what happens then is quite different. In GNOME(X), mutter tells the X server to change a device property, the server passes that on to the xf86-input-libinput driver and from there the setting is toggled in libinput. In GNOME(Wayland), mutter toggles the setting directly in libinput.

Since there is no X server in the stack, the various tools can't talk to it. So to get the tools to work they would have to talk to the compositor instead. But they only know how to speak X protocol, and no Wayland protocol extension exists for input device configuration. Such a Wayland protocol extension would most likely have to be a private one since the various compositors expose device configuration in different ways. Whether this extension will be written and added to compositors is uncertain, I'm not aware of any plans or even intentions to do so (it's a very messy problem). But either way, until it exists, the tools will merely shout into the void, without even an echo to keep them entertained. Non-functional is thus a good summary.

libinput now requires axis resolutions for graphics tablets

Posted by Peter Hutterer on December 05, 2016 01:52 AM

I pushed the patch to require resolution today, expect this to hit the general public with libinput 1.6. If your graphics tablet does not provide axis resolution we will need to add a hwdb entry. Please file a bug in systemd and CC me on it (@whot).

How do you know if your device has resolution? Run sudo evemu-describe against the device node and look for the ABS_X/ABS_Y entries:


# Event code 0 (ABS_X)
# Value 2550
# Min 0
# Max 3968
# Fuzz 0
# Flat 0
# Resolution 13
# Event code 1 (ABS_Y)
# Value 1323
# Min 0
# Max 2240
# Fuzz 0
# Flat 0
# Resolution 13
if the Resolution value is 0 you'll need a hwdb entry or your tablet will stop working in libinput 1.6. You can file the bug now and we can get it fixed, that way it'll be in place once 1.6 comes out.

Please don't use pastebins in bugs

Posted by Peter Hutterer on December 05, 2016 01:51 AM

pastebins are useful for dumping large data sets whenever the medium of conversation doesn't make this easy or useful. IRC is one example, or audio/video conferencing. But pastebins only work when the other side looks at the pastebin before it expires, and the default expiry date for a pastebin may only be a few days.

This makes them effectively useless for bugs where it may take a while for the bug to be triaged and the assignee to respond. It may take even longer to figure out the source of the bug, and if there's a regression it can take months to figure it out. Once the content disappears we have to re-request the data from the reporter. And there is a vicious dependency too: usually, logs are more important for difficult bugs. Difficult bugs take longer to fix. Thus, with pastebins, the more difficult the bug, the more likely the logs become unavailable.

All useful bug tracking systems have an attachment facility. Use that instead, it's archived with the bug and if a year later we notice a regression, we still have access to the data.

If you got here because I pasted the link to this blog post, please do the following: download the pastebin content as raw text, then add it as attachment to the bug (don't paste it as comment). Once that's done, we can have a look at your bug again.

Protected: Another GTK+ update

Posted by Matthias Clasen on December 02, 2016 03:07 PM

This content is password protected. To view it please enter your password below:

GNOME loves to cook

Posted by Matthias Clasen on December 02, 2016 02:53 PM

GNOME needs a recipe app, since we all love to cook.org-gnome-recipesThis is not a new idea. Looking all the way back to 2007, the idea of a GNOME cook book already existed back then. For one reason or another, we never quite got there. But the idea has stuck around.

screenshot-from-2016-12-02-08-39-17

With the upcoming 20th birthday of GNOME next year, some of us thought that we should make another attempt at this application, maybe as a birthday gift to all of GNOME.

Shortly after GUADEC, I got my hands on some existing designs and started to toy around with implementing them over a few weekends and evenings. The screenshots in this post show how far I got since then.

screenshot-from-2016-12-02-08-39-33

Why did I start to write this app from scratch (instead of e.g. trying to give a face-lift to venerable gourmet)  ?

Beyond the obvious reason that I love to code as much as I love to cook, I wanted to give GNOME Builder a more serious test by starting an application from scratch. And I find it very useful to take a look at GTK+ from the application developer side, every now and then.  In both of these regards, the endeavor was already successful and has yielded improvements to both GNOME Builder and GTK+.  For the cooking part, you can judge that for yourself.

screenshot-from-2016-12-02-08-46-17

The main reason for writing this post is that we are at a point now where we need contributions to make progress. The idea is that we will include a decent set of recipes from GNOME contributors all over the world with the application.

Therefore, we need your recipes, ideally with good photos.

All the photos and text you see in the screenshots here are just test data that I’ve used during development, and need to be replaced with actual content.

So, how can you contribute your favorite recipes ?

Add your recipe to the app, and when you are satisfied with how it looks, you can use the Export button on the details page to create an archive with the recipe and related information, such as images. The archive also includes information about the author of the recipe (ie yourself), so make sure to provide some information for that in the Preferences dialog.

We just created a bugzilla project for recipes, so you can just attach the archive to a bug:

https://bugzilla.gnome.org/page.cgi?id=browse.html&product=recipes

Please make it clear in the bug that all the included images are your own and that we are allowed to ship it with the app.

screenshot-from-2016-12-02-08-44-40

Beyond recipes, there are plenty of other ways to contribute, of course. While I’ve tried hard to get many of the features in the initial design implemented, there is a lot more that can be done: for example, unit conversion, or a way to easily share recipes, or to print a shopping list.

screenshot-from-2016-12-02-08-39-43

Where do you get it ?

The project started out on github, but it is also available on git.gnome.org now. The design materials are collected on the GNOME wiki.

If you just want to try it out without building it yourself,  you can just use Flatpak:

flatpak install --from https://alexlarsson.github.io/test-releases/gnome-recipes.flatpakref

Of course, I did not get this far on my own. Thanks are due to several people. First and foremost, Emel Elvin Yıldız, for the designs and feedback on the implementation, Jakub Steiner for the icon and visuals, and Christian Hergert for keeping this idea alive and for making GNOME builder work great.

Ubuntu still isn't free software

Posted by Matthew Garrett on December 02, 2016 09:37 AM
Mark Shuttleworth just blogged about their stance against unofficial Ubuntu images. The assertion is that a cloud hoster is providing unofficial and modified Ubuntu images, and that these images are meaningfully different from upstream Ubuntu in terms of their functionality and security. Users are attempting to make use of these images, are finding that they don't work properly and are assuming that Ubuntu is a shoddy product. This is an entirely legitimate concern, and if Canonical are acting to reduce user confusion then they should be commended for that.

The appropriate means to handle this kind of issue is trademark law. If someone claims that something is Ubuntu when it isn't, that's probably an infringement of the trademark and it's entirely reasonable for the trademark owner to take action to protect the value associated with their trademark. But Canonical's IP policy goes much further than that - it can be interpreted as meaning[1] that you can't distribute works based on Ubuntu without paying Canonical for the privilege, even if you call it something other than Ubuntu.

This remains incompatible with the principles of free software. The freedom to take someone else's work and redistribute it is a vital part of the four freedoms. It's legitimate for Canonical to insist that you not pass it off as their work when doing so, but their IP policy continues to insist that you remove all references to Canonical's trademarks even if their use would not infringe trademark law.

If you ask a copyright holder if you can give a copy of their work to someone else (assuming it doesn't infringe trademark law), and they say no or insist you need an additional contract, it's not free software. If they insist that you recompile source code before you can give copies to someone else, it's not free software. Asking that you remove trademarks that would otherwise infringe trademark law is fine, but if you can't use their trademarks in non-infringing ways, that's still not free software.

Canonical's IP policy continues to impose restrictions on all of these things, and therefore Ubuntu is not free software.

[1] And by "interpreted as meaning" I mean that's what it says and Canonical refuse to say otherwise

comment count unavailable comments

Impress LibreOffice OpenGL Slide Transitions under Wayland via GTK3

Posted by Caolán McNamara on December 01, 2016 04:58 PM
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/mVVh_R7R2f8" width="560"></iframe>
Impress LibreOffice OpenGL Slide Transitions under Wayland via GTK3 (GtkGlArea).

So I've implemented enough to get this working on my machine now. I've demoed "static", "glitter" and "honeycomb" above from my -O0 debugging build. I'll work on merging this to master now, patches are in our gerrit instance. Porting from glew to epoxy is a necessary step, I know it builds on Windows and Mac, but that's utterly untested.

Linux communities, we need your help!

Posted by Richard Hughes on November 28, 2016 11:23 AM

There are a lot of Linux communities all over the globe filled with really nice people who just want to help others. Typically these people either can’t (or don’t feel comfortable) coding, and I’d love to harness some of that potential by adding a huge number of new application reviews to the ODRS. At the moment we have about 1100 reviews, mostly covering the more popular applications, and also mostly written in English.

What I would love is for a few groups of people to come together for their next LUG/outreach/InstallFest and sit down together somewhere cozy and write a few reviews. Bonus points if you use a less-well-known application, and even more points if you can write in a language other than English. Submitting a review is easy; just open up GNOME Software, find the application, and click ‘Write a Review‘ at the bottom of the page.

Application reviews help new users what to install, and the star ratings you give means we can return useful search results full of great applications. Please write an email, ask about helping the ODRS, and perhaps you can help a lot of new users next time you meet with your Linuxy friends.

Thanks!

Last batch of ColorHugALS

Posted by Richard Hughes on November 21, 2016 11:43 AM

I’ve got 9 more ColorHugALS devices in stock and then when they are sold they will be no more for sale. With all the supplier costs going recently up my “sell at cost price” has turned into “make a small loss on each one” which isn’t sustainable. It’s all OpenHardware, both hardware design and the firmware itself so if someone wanted to start building them for sale they would be doing it with my blessing. Of course, I’m happy to continue supporting the existing sold devices into the distant future.

colorhug-als1-large

In part the original goal is fixed, the kernel and userspace support for the new SensorHID protocol works great and ambient light functionality works out of the box for more people on more hardware. I’m slightly disappointed more people didn’t get involved in making the ambient lighting algorithms more smart, but I guess it’s quite a niche area of development.

Plus, in the Apple product development sense, killing off one device lets me start selling something else OpenHardware in the future. :)

Fedora - retiring xorg-x11-drv-synaptics

Posted by Peter Hutterer on November 20, 2016 03:57 AM

The Fedora Change to retire the synaptics driver was approved by FESCO. This will apply to Fedora 26 and is part of a cleanup to, ironically, make the synaptics driver easier to install.

Since Fedora 22, xorg-x11-drv-libinput is the preferred input driver. For historical reasons, almost all users have the xorg-x11-drv-synaptics package installed. But to actually use the synaptics driver over xorg-x11-drv-libinput requires a manually dropped xorg.conf.d snippet. And that's just not ideal. Unfortunately, in DNF/RPM we cannot just say "replace the xorg-x11-drv-synaptics package with xorg-x11-drv-libinput on update but still allow users to install xorg-x11-drv-synaptics after that".

So the path taken is a package rename. Starting with Fedora 26, xorg-x11-drv-libinput's RPM will Provide/Obsolete [1] xorg-x11-drv-synaptics and thus remove the old package on update. Users that need the synaptics driver then need to install xorg-x11-drv-synaptics-legacy. This driver will then install itself correctly without extra user intervention and will take precedence over the libinput driver. Removing xorg-x11-drv-synaptics-legacy will remove the driver assignment and thus fall back to libinput for touchpads. So aside from the name change, everything else works smoother now. Both packages are now updated in Rawhide and should be available from your local mirror soon.

What does this mean for you as a user? If you are a synaptics user, after an update/install, you need to now manually install xorg-x11-drv-synaptics-legacy. You can remove any xorg.conf.d snippets assigning the synaptics driver unless they also include other custom configuration.

See the Fedora Change page for details. Note that this is a Fedora-specific change only, the upstream change for this is already in place.

[1] "Provide" in RPM-speak means the package provides functionality otherwise provided by some other package even though it may not necessarily provide the code from that package. "Obsolete" means that installing this package replaces the obsoleted package.

Help, my app icon is missing!

Posted by Matthias Clasen on November 15, 2016 02:55 PM

I was in this situation recently: An application icon would not show up in GNOME shell, even after I double- and triple-checked that I did all the right things:

  • Installed the desktop file with the right name in /usr/share/applications
  • Installed the icon with the right name in the hicolor icon theme
  • Make sure that Icon key in the desktop file has the right name
  • Restart GNOME shell (this is necessary due to a bug where GNOME shell will not reliably pick up  desktop file changes)

Still, I just get the generic executable icon: missing-icon1

GNOME shell to the rescue

I asked GNOME shell maintainer Florian Müllner for help. He said: GNOME shell is probably picking up a stale desktop file from somewhere.

But from where ? I checked all the locations listed in XDG_DATA_DIRS to no avail.  At this point, I was getting desperate,  so I went back to Florian. He said: Just use looking glass to find out! If you don’t remember, looking glass is a pretty handy debugging console that is built right into GNOME shell. You open it by typing the command

lg

Into the Alt-F2 run dialog.

missing-icon2This brings up the looking glass. Here is how it helped me solve my problem:

missing-icon3I switched to the window tab, then clicked on the application in question, and hit the ‘Insert’ button. That binds the object to a variable named r(x) (for some number x)  in the Javascript evaluator.

Going there, I then typed the command

r(3).app_info.get_filename()

And that showed the problematic desktop file:

missing-icon4I removed that file and restarted the shell once more, and now everything is working as it should:

missing-icon5Thanks, looking glass!

Lyon GNOME Bug day #1

Posted by Bastien Nocera on November 15, 2016 09:48 AM
Last Friday, both a GNOME bug day and a bank holiday, a few of us got together to squash some bugs, and discuss GNOME and GNOME technologies.

Guillaume, a new comer in our group, tested the captive portal support for NetworkManager and GNOME in Gentoo, and added instructions on how to enable it to their Wiki. He also tested a gateway related configuration problem, the patch for which I merged after a code review. Near the end of the session, he also rebuilt WebKitGTK+ to test why Google Docs was not working for him anymore in Web. And nobody believed that he could build it that quickly. Looks like opinions based on past experiences are quite hard to change.

Mathieu worked on removing jhbuild's .desktop file as nobody seems to use it, and it was creating the Sundry category for him, in gnome-shell. He also spent time looking into the tracker blocker that is Mozilla's Focus, based on disconnectme's block lists. It's not as effective as uBlock when it comes to blocking adverts, but the memory and performance improvements, and the slow churn rate, could make it a good default blocker to have in Web.

Haïkel looked into using Emeus, potentially the new GTK+ 4.0 layout manager, to implement the series properties page for Videos.

Finally, I added Bolso to jhbuild, and struggled to get gnome-online-accounts/gnome-keyring to behave correctly in my installation, as the application just did not want to log in properly to the service. I also discussed Fedora's privacy policy (inappropriate for Fedora Workstation, as it doesn't cover the services used in the default installation), a potential design for Flatpak support of joypads and removable devices in general, as well as the future design of the Network panel.

Does $FEATURE work under Wayland?

Posted by Peter Hutterer on November 14, 2016 12:45 AM

I've written more extensively about this here but here's an analogy that should get the point across a bit better: Wayland is just a protocol, just like HTTP. In both cases, you have two sides with very different roles and functionality. In the HTTP case, you have the server (e.g. Apache) and the client (a browser, e.g. Firefox). The communication protocol is HTTP but both sides make a lot of decisions unrelated to the protocol. The server decides what data is sent, the client decides how the data is presented to the user. Wayland is very similar. The server, called the "compositor", decides what data is sent (also: which of the clients even gets the data). The client renders the data [1] and decides what to do with input like key strokes, etc.

Asking Does $FEATURE work under Wayland? is akin to asking Does $FEATURE work under HTTP?. The only answer is: it depends on the compositor and on the client. It's the wrong question. You should ask questions related to the compositor and the client instead, e.g. "does $FEATURE work in GNOME?" or "does $FEATURE work in GTK applications?". That's a question that can be answered.

Of course, there are some cases where the fault is really the protocol itself. But often enough, it's not.

[1] albeit it does so by telling the compositor to display it. The analogy with HTTP only works to some extent... :)

Tor, TPMs and service integrity attestation

Posted by Matthew Garrett on November 10, 2016 08:48 PM
One of the most powerful (and most scary) features of TPM-based measured boot is the ability for remote systems to request that clients attest to their boot state, allowing the remote system to determine whether the client has booted in the correct state. This involves each component in the boot process writing a hash of the next component into the TPM and logging it. When attestation is requested, the remote site gives the client a nonce and asks for an attestation, the client OS passes the nonce to the TPM and asks it to provide a signed copy of the hashes and the nonce and sends them (and the log) to the remote site. The remoteW site then replays the log to ensure it matches the signed hash values, and can examine the log to determine whether the system is trustworthy (whatever trustworthy means in this context).

When this was first proposed people were (justifiably!) scared that remote services would start refusing to work for users who weren't running (for instance) an approved version of Windows with a verifiable DRM stack. Various practical matters made this impossible. The first was that, until fairly recently, there was no way to demonstrate that the key used to sign the hashes actually came from a TPM[1], so anyone could simply generate a set of valid hashes, sign them with a random key and provide that. The second is that even if you have a signature from a TPM, you have no way of proving that it's from the TPM that the client booted with (you can MITM the request and either pass it to a client that did boot the appropriate OS or to an external TPM that you've plugged into your system after boot and then programmed appropriately). The third is that, well, systems and configurations vary so much that outside very controlled circumstances it's impossible to know what a "legitimate" set of hashes even is.

As a result, so far remote attestation has tended to be restricted to internal deployments. Some enterprises use it as part of their VPN login process, and we've been working on it at CoreOS to enable Kubernetes clusters to verify that workers are in a trustworthy state before running jobs on them. While useful, this isn't terribly exciting for most people. Can we do better?

Remote attestation has generally been thought of in terms of remote systems requiring that clients attest. But there's nothing that requires things to be done in that direction. There's nothing stopping clients from being able to request that a server attest to its state, allowing clients to make informed decisions about whether they should provide confidential data. But the problems that apply to clients apply equally well to servers. Let's work through them in reverse order.

We have no idea what expected "good" values are

Yes, and this is a problem. CoreOS ships with an expected set of good values, and we had general agreement at the Linux Plumbers Conference that other distributions would start looking at what it would take to do the same. But how do we know that those values are themselves trustworthy? In an ideal world this would involve reproducible builds, allowing anybody to grab the source code for the OS, build it locally and verify that they have the same hashes.

Ok. So we're able to verify that the booted OS was good. But how about the services? The rkt container runtime supports measuring each container into the TPM, which means we can verify which container images were started. If container images are also built in such a way that they're reproducible, users can grab the source code, rebuild the container locally and again verify that it has the same hashes. Users can then be sure that the remote site is running the code they're looking at.

Or can they? Not really - a general purpose OS has all kinds of ways to inject code into containers, so an admin could simply replace the binaries inside the container after it's been measured, or ptrace() the server, or modify rkt so it generates correct measurements regardless of the image or, well, there's lots they could do. So a general purpose OS is probably a bad idea here. Instead, let's imagine an immutable OS that does nothing other than bring up networking and then reads a config file that tells it which container images to download and run. This reduces the amount of code that needs to support reproducible builds, making it easier for a client to verify that the source corresponds to the code the remote system is actually running.

Is this sufficient? Eh sadly no. Even if we know the valid values for the entire OS and every container, we don't know the legitimate values for the system firmware. Any modified firmware could tamper with the rest of the trust chain, making it possible for you to get valid OS values even if the OS has been subverted. This isn't a solved problem yet, and really requires hardware vendor support. Let's handwave this for now, or assert that we'll have some sidechannel for distributing valid firmware values.

Avoiding TPM MITMing

This one's more interesting. If I ask the server to attest to its state, it can simply pass that through to a TPM running on another system that's running a trusted stack and happily serve me content from a compromised stack. Suboptimal. We need some way to tie the TPM identity and the service identity to each other.

Thankfully, we have one. Tor supports running services in the .onion TLD. The key used to identify the service to the Tor network is also used to create the "hostname" of the system. I wrote a pretty hacky implementation that generates that key on the TPM, tying the service identity to the TPM. You can ask the TPM to prove that it generated a key, and that allows you to tie both the key used to run the Tor service and the key used to sign the attestation hashes to the same TPM. You now know that the attestation values came from the same system that's running the service, and that means you know the TPM hasn't been MITMed.

How do you know it's a TPM at all?

This is much easier. See [1].



There's still various problems around this, including the fact that we don't have this immutable minimal container OS, that we don't have the infrastructure to ensure that container builds are reproducible, that we don't have any known good firmware values and that we don't have a mechanism for allowing a user to perform any of this validation. But these are all solvable, and it seems like an interesting project.

"Interesting" isn't necessarily the right metric, though. "Useful" is. And I think this is very useful. If I'm about to upload documents to a SecureDrop instance, it seems pretty important that I be able to verify that it is a SecureDrop instance rather than something pretending to be one. This gives us a mechanism.

The next few years seem likely to raise interest in ensuring that people have secure mechanisms to communicate. I'm not emotionally invested in this one, but if people have better ideas about how to solve this problem then this seems like a good time to talk about them.

[1] More modern TPMs have a certificate that chains from the TPM's root key back to the TPM manufacturer, so as long as you trust the TPM manufacturer to have kept control of that you can prove that the signature came from a real TPM

comment count unavailable comments

Searching in GNOME Software

Posted by Richard Hughes on November 03, 2016 05:10 PM

I’ve spent a few days profiling GNOME Software on ARM, mostly for curiosity but also to help our friends at Endless. I’ve merged a few patches that make the existing --profile code more useful to profile start up speed. Already there have been some big gains, over 200ms of startup time and 12Mb of RSS, but there’s plenty more that we want to fix to make GNOME Software run really nicely on resource constrained devices.

One of the biggest delays is constructing the search token cache at startup. This is where we look at all the fields of the .desktop files, the AppData files and the AppStream files and split them in a UTF8-sane way into search tokens, adding them into a big hash table after stemming them. We do it with 4 threads by default as it’s trivially parallelizable. With the search cache, when we search we just ask all the applications in the store “do you have this search term” and if so it gets added to the search results and ordered according to how good the match is. This takes 225ms on my super-fast Intel laptop (and much longer on ARM), and this happens automatically the very first time you search for anything in GNOME Software.

At the moment we add (for each locale, including fallbacks) the package name, the app ID, the app name, app single line description, the app keywords and the application long description. The latter is the multi-paragraph long description that’s typically prose. We use 90% of the time spent loading the token cache just splitting and adding the words in the description. As the description is prose, we have to ignore quite a few words e.g. “and”, “the”, “is” and “can” are some of the most frequent, useless words. Just the nature of the text itself (long non-technical prose) it doesn’t actually add many useful keywords to the search cache, and the ones that is does add are treated with such low priority other more important matches are ordered before them.

My proposal: continue to consume everything else for the search cache, and drop using the description. This means we start way quicker, use less memory, but it does require upstream actually adds some [localized] Keywords=foo;bar;baz in either the desktop file or <keywords> in the AppData file. At the moment most do, especially after I sent ~160 emails to the maintainers that didn’t have any defined keywords in the Fedora 25 Alpha, so I think it’s fairly safe at this point. Comments?

Another GTK+ update

Posted by Matthias Clasen on November 02, 2016 04:44 PM

The GTK+ 4 work is continuing at full speed, and today I want to show one of the first concrete benefits from  the GSK merge: We can now record and replay frames. If you ever wondered why your animation does not look quite right, this might be just the tool for you.

<video class="wp-video-shortcode" controls="controls" height="267" id="video-1662-1" preload="metadata" width="474"><source src="https://blogs.gnome.org/mclasen/files/2016/11/Screencast-from-11-02-2016-122859-PM.webm?_=1" type="video/webm">https://blogs.gnome.org/mclasen/files/2016/11/Screencast-from-11-02-2016-122859-PM.webm</video>

To try this new tool, find the Recorder tab in the GTK+ inspector, and use the record button. As you can see, we capture and save the render node tree for each frame that the application draws, as long as recording is enabled (you want to be a bit careful, this is quickly eating up a lot of memory).

The leftmost pane lets you select one of the recorded frames. The middle pane shows the render node tree, and if you select a node there, the rendering on the right is updated to show only that subtree’s effect on the frame.

This is pretty nifty, and will hopefully be very useful in improving the GSK integration in GTK+, as well as helpful for debugging rendering problems in applications.

libinput touchpad pointer acceleration - laptop model names needed

Posted by Peter Hutterer on November 01, 2016 09:50 PM

I finally have a bit of time to look at touchpad pointer acceleration in libinput. But when I did, I found a great total of 5 bugs across freedesktop.org and Red Hat's bugzilla, despite this being the first thing anyone brings up whenever libinput is mentioned. 5 bugs - that's not much to work on. Note that over time there were also a lot of bugs where pointer acceleration was fixed once the touchpad's axis ranges were corrected which usually is a two-liner for the udev hwdb.

Anyway, point of this post: if you're still having issues with pointer acceleration on your touchpad in libinput, please file a bug against libinput and make it block the new tracker bug 98535. The libinput documentation has instructions on how to report a touchpad bug, but amongst the various things I need is your laptop model name.

Don't complain about it on reddit, phoronix, HN, or in some random forum, because you're just wasting bytes there and it won't get fixed that way.

Flatpak cross-compilation support: Epilogue

Posted by Bastien Nocera on October 31, 2016 12:00 PM
You might remember my attempts at getting an easy to use cross-compilation for ARM applications on my x86-64 desktop machine.

With Fedora 25 approaching, I'm happy to say that the necessary changes to integrate the feature have now rolled into Fedora 25.

For example, to compile the GNU Hello Flatpak for ARM, you would run:

$ flatpak install gnome org.freedesktop.Platform/arm org.freedesktop.Sdk/arm
Installing: org.freedesktop.Platform/arm/1.4 from gnome
[...]
$ sudo dnf install -y qemu-user-static
[...]
$ TARGET=arm ./build.sh

For other applications, add the --arch=arm argument to the flatpak-builder command-line.

This example also works for 64-bit ARM with the architecture name aarch64.

Of course smart homes are targets for hackers

Posted by Matthew Garrett on October 28, 2016 05:23 PM
The Wirecutter, an in-depth comparative review site for various electrical and electronic devices, just published an opinion piece on whether users should be worried about security issues in IoT devices. The summary: avoid devices that don't require passwords (or don't force you to change a default and devices that want you to disable security, follow general network security best practices but otherwise don't worry - criminals aren't likely to target you.

This is terrible, irresponsible advice. It's true that most users aren't likely to be individually targeted by random criminals, but that's a poor threat model. As I've mentioned before, you need to worry about people with an interest in you. Making purchasing decisions based on the assumption that you'll never end up dating someone with enough knowledge to compromise a cheap IoT device (or even meeting an especially creepy one in a bar) is not safe, and giving advice that doesn't take that into account is a huge disservice to many potentially vulnerable users.

Of course, there's also the larger question raised by the last week's problems. Insecure IoT devices still pose a threat to the wider internet, even if the owner's data isn't at risk. I may not be optimistic about the ease of fixing this problem, but that doesn't mean we should just give up. It is important that we improve the security of devices, and many vendors are just bad at that.

So, here's a few things that should be a minimum when considering an IoT device:
  • Does the vendor publish a security contact? (If not, they don't care about security)
  • Does the vendor provide frequent software updates, even for devices that are several years old? (If not, they don't care about security)
  • Has the vendor ever denied a security issue that turned out to be real? (If so, they care more about PR than security)
  • Is the vendor able to provide the source code to any open source components they use? (If not, they don't know which software is in their own product and so don't care about security, and also they're probably infringing my copyright)
  • Do they mark updates as fixing security bugs? (If not, they care more about hiding security issues than fixing them)
  • Has the vendor ever threatened to prosecute a security researcher? (If so, again, they care more about PR than security)
  • Does the vendor provide a public minimum support period for the device? (If not, they don't care about security or their users)

    I've worked with big name vendors who did a brilliant job here. I've also worked with big name vendors who responded with hostility when I pointed out that they were selling a device with arbitrary remote code execution. Going with brand names is probably a good proxy for many of these requirements, but it's insufficient.

    So here's my recommendations to The Wirecutter - talk to a wide range of security experts about the issues that users should be concerned about, and figure out how to test these things yourself. Don't just ask vendors whether they care about security, ask them what their processes and procedures look like. Look at their history. And don't assume that just because nobody's interested in you, everybody else's level of risk is equal.


  • comment count unavailable comments

    Deckard and LibreOffice

    Posted by Caolán McNamara on October 27, 2016 12:44 PM
    LibreOffice reuses the same ui format that gtk uses. This suggests that deckard could be used to preview translations of them.

    Testing this out shows (as above) that it can be made to work. A few problems though:

    1. We have various placeholder widgets which don't work in deckard because the widgets don't exist in gtk so dialogs that use them can't display as something falls over with e.g. "Invalid object type 'SvSimpleTableContainer'" I had hoped I'd get placeholders by default on failure.
    2. Our .po translation entries for the dialogs strings all have autogenerated msgctxt fields which don't correspond to the blank default of the .ui so the msgctxt fields have to be removed, then msguniq to remove duplicates, and the result can the be run through msgfmt to create a .mo that works with deckard to show web-previews

    Dual-GPU integration in GNOME

    Posted by Bastien Nocera on October 26, 2016 01:37 PM
    Thanks to the work of Hans de Goede and many others, dual-GPU (aka NVidia Optimus or AMD Hybrid Graphics) support works better than ever in Fedora 25.

    On my side, I picked up some work I originally did for Fedora 24, but ended up being blocked by hardware support. This brings better integration into GNOME.

    The Details Settings panel now shows which video cards you have in your (most likely) laptop.

    dual-GPU Graphics

    The second feature is what Blender and 3D video games users have been waiting for: a contextual menu item to launch the application on the more powerful GPU in your machine.

    Mooo Powaa!

    This demonstration uses a slightly modified GtkGLArea example, which shows which of the GPUs is used to render the application in the title bar.

    on the integrated GPU

    on the discrete GPU

    Behind the curtain

    Behind those 2 features, we have a simple D-Bus service, which runs automatically on boot, and stays running to offer a single property (HasDualGpu) that system components can use to detect what UI to present. This requires the "switcheroo" driver to work on the machine in question.

    Because of the way applications are launched on the discrete GPU, we cannot currently support D-Bus activated applications, but GPU-heavy D-Bus-integrated applications are few and far between right now.

    Future plans

    There's plenty more to do in this area, to polish the integration. We might want applications to tell us whether they'd prefer being run on the integrated or discrete GPU, as live switching between renderers is still something that's out of the question on Linux.

    Wayland dual-GPU support, as well as support for the proprietary NVidia drivers are also things that will be worked on, probably by my colleagues though, as the graphics stack really isn't my field.

    And if the hardware becomes more widely available, we'll most certainly want to support hardware with hotpluggable graphics support (whether gaming laptop "power-ups" or workstation docks).

    Availability

    All the patches necessary to make this work are now available in GNOME git (targeted at GNOME 3.24), and backports are integrated in Fedora 25, due to be released shortly.

    GTK+ happenings

    Posted by Matthias Clasen on October 22, 2016 03:57 PM

    I haven’t written about GTK+ development in some time. But now there are some exciting things happening that are worth writing about.

    Plans

    Back in June, a good number of GTK+ developers came together for a hackfest in Toronto,  It was a very productive gathering. One of the topics we discussed there was the (lack of) stability of GTK+ 3 and versioning. We caused a bit of a firestorm by blogging about this right away… so we went back to the drawing board and had another long discussion about the pros and cons of various versioning schemes at GUADEC.

    <figure class="wp-caption alignnone" style="width: 1024px"><figcaption class="wp-caption-text">GTK+ BOF in Karlsruhe</figcaption></figure>

    The final, agreed-on plan was published on the GTK+ blog, and you can read it there.

    Actions

    Fast-forward to today, and we’ve made good progress on putting this plan into place.

    GTK+ has been branched for 3.22, and all future GTK+ 3 releases will come from this branch. This is very similar to GTK+ 2, where we have the forever-stable 2.24 branch.  We plan to maintain the 3.22 branch for several years, so applications can rely on a stable GTK+ 3.

    One activity that you can see in the branch currently is that we are deprecating APIs that will go away in GTK+ 4. Most deprecations have been in place for a while (some even from 3.0!),  but some functions have just been forgotten. Marking them as deprecated now will make it easier to port to GTK+ 4 in the future. Keep in mind that deprecations are an optional service – you don’t have to rush to act on them unless you want to port to the next version.

    To avoid unnecessary heartburn and build breakage, we’ve switched  jhbuild, GNOME continuous and the flatpak runtimes over to using the 3.22 branch before opening the master branch for  new development, and did the necessary work to make the two branches parallel-installable.

    With all these preparations in place, Benjamin and Timm went to work and did a big round of deprecation cleanup. Altogether,  this removed some 80.000 lines of code. Next, we’ve merged Emmanueles GSK work.  And there is a lot more work queued up, from modernizing the GDK layer, to redoing input handling, to building with meson.

    The current git version of GTK+ calls itself 3.89, and we’re aiming to do a 3.90 release in spring, ideally keeping the usual 6 months cadence.

    …and you

    We hope that at least some of the core GNOME applications will switch to using 3.90 by next spring, since we need testing and validation. But… currently things are a still a bit rough in master. The GSK port will need some more time to shake out rendering issues and make it as fast as it should be.

    Therefore, we recommend that you stick with the 3.22 branch until we do a 3.89.1 release. By that time, the documentation should also have a 3 → 4 migration guide to help you with porting.

    If you are eager to get ready for GTK+ 4 now, you can prepare your application by eliminating the deprecations that show up when you build against the latest 3.22 release.

    Summary

    This is an exciting time for GTK+ ! We will post regular updates as things are landing, but just following the weekly updates on the GTK+ blog should give you a good idea of what is going on.

    Fixing the IoT isn't going to be easy

    Posted by Matthew Garrett on October 22, 2016 05:14 AM
    A large part of the internet became inaccessible today after a botnet made up of IP cameras and digital video recorders was used to DoS a major DNS provider. This highlighted a bunch of things including how maybe having all your DNS handled by a single provider is not the best of plans, but in the long run there's no real amount of diversification that can fix this - malicious actors have control of a sufficiently large number of hosts that they could easily take out multiple providers simultaneously.

    To fix this properly we need to get rid of the compromised systems. The question is how. Many of these devices are sold by resellers who have no resources to handle any kind of recall. The manufacturer may not have any kind of legal presence in many of the countries where their products are sold. There's no way anybody can compel a recall, and even if they could it probably wouldn't help. If I've paid a contractor to install a security camera in my office, and if I get a notification that my camera is being used to take down Twitter, what do I do? Pay someone to come and take the camera down again, wait for a fixed one and pay to get that put up? That's probably not going to happen. As long as the device carries on working, many users are going to ignore any voluntary request.

    We're left with more aggressive remedies. If ISPs threaten to cut off customers who host compromised devices, we might get somewhere. But, inevitably, a number of small businesses and unskilled users will get cut off. Probably a large number. The economic damage is still going to be significant. And it doesn't necessarily help that much - if the US were to compel ISPs to do this, but nobody else did, public outcry would be massive, the botnet would not be much smaller and the attacks would continue. Do we start cutting off countries that fail to police their internet?

    Ok, so maybe we just chalk this one up as a loss and have everyone build out enough infrastructure that we're able to withstand attacks from this botnet and take steps to ensure that nobody is ever able to build a bigger one. To do that, we'd need to ensure that all IoT devices are secure, all the time. So, uh, how do we do that?

    These devices had trivial vulnerabilities in the form of hardcoded passwords and open telnet. It wouldn't take terribly strong skills to identify this at import time and block a shipment, so the "obvious" answer is to set up forces in customs who do a security analysis of each device. We'll ignore the fact that this would be a pretty huge set of people to keep up with the sheer quantity of crap being developed and skip straight to the explanation for why this wouldn't work.

    Yeah, sure, this vulnerability was obvious. But what about the product from a well-known vendor that included a debug app listening on a high numbered UDP port that accepted a packet of the form "BackdoorPacketCmdLine_Req" and then executed the rest of the payload as root? A portscan's not going to show that up[1]. Finding this kind of thing involves pulling the device apart, dumping the firmware and reverse engineering the binaries. It typically takes me about a day to do that. Amazon has over 30,000 listings that match "IP camera" right now, so you're going to need 99 more of me and a year just to examine the cameras. And that's assuming nobody ships any new ones.

    Even that's insufficient. Ok, with luck we've identified all the cases where the vendor has left an explicit backdoor in the code[2]. But these devices are still running software that's going to be full of bugs and which is almost certainly still vulnerable to at least half a dozen buffer overflows[3]. Who's going to audit that? All it takes is one attacker to find one flaw in one popular device line, and that's another botnet built.

    If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards? From an economic perspective, demanding that vendors ship security updates whenever a vulnerability is discovered no matter how old the device is is just not going to work. Many of these vendors are small enough that it'd be more cost effective for them to simply fold the company and reopen under a new name than it would be to put the engineering work into fixing a decade old codebase. And how does this actually help? So far the attackers building these networks haven't been terribly competent. The first thing a competent attacker would do would be to silently disable the firmware update mechanism.

    We can't easily fix the already broken devices, we can't easily stop more broken devices from being shipped and we can't easily guarantee that we can fix future devices that end up broken. The only solution I see working at all is to require ISPs to cut people off, and that's going to involve a great deal of pain. The harsh reality is that this is almost certainly just the tip of the iceberg, and things are going to get much worse before they get any better.

    Right. I'm off to portscan another smart socket.

    [1] UDP connection refused messages are typically ratelimited to one per second, so it'll take almost a day to do a full UDP portscan, and even then you have no idea what the service actually does.

    [2] It's worth noting that this is usually leftover test or debug code, not an overtly malicious act. Vendors should have processes in place to ensure that this isn't left in release builds, but ha well.

    [3] My vacuum cleaner crashes if I send certain malformed HTTP requests to the local API endpoint, which isn't a good sign

    comment count unavailable comments

    Office Binary Document RC4 CryptoAPI Encryption

    Posted by Caolán McNamara on October 21, 2016 10:35 AM
    In LibreOffice we've long supported Microsoft Office's "Office Binary Document RC4 Encryption" for decrypting xls, doc and ppt. But somewhere along the line the Microsoft Office encryption scheme was replaced by a new one, "Office Binary Document RC4 CryptoAPI Encryption", which we didn't support. This is what the error dialog of...

    "The encryption method used in this document is not supported. Only Microsoft Office 97/2000 compatible password encryption is supported."

    ...from LibreOffice is telling you when you open, for example, an encrypted xls saved by a contemporary Microsoft Excel version.

    I got the newer scheme working this morning for xls, so from LibreOffice 5-3 onwards (I may backport to upstream 5-2 and Fedora 5-1) these variants can be successfully decrypted and viewed in LibreOffice.

    systemd.conf 2016 Over Now

    Posted by Lennart Poettering on October 04, 2016 10:00 PM

    systemd.conf 2016 is Over Now!

    A few days ago systemd.conf 2016 ended, our second conference of this kind. I personally enjoyed this conference a lot: the talks, the atmosphere, the audience, the organization, the location, they all were excellent!

    I'd like to take the opportunity to thanks everybody involved. In particular I'd like to thank Chris, Daniel, Sandra and Henrike for organizing the conference, your work was stellar!

    I'd also like to thank our sponsors, without which the conference couldn't take place like this, of course. In particular I'd like to thank our gold sponsor, Red Hat, our organizing sponsor Kinvolk, as well as our silver sponsors CoreOS and Facebook. I'd also like to thank our bronze sponsors Collabora, OpenSUSE, Pantheon, Pengutronix, our supporting sponsor Codethink and last but not least our media sponsor Linux Magazin. Thank you all!

    I'd also like to thank the Video Operation Center ("VOC") for their amazing work on live-streaming the conference and making all talks available on YouTube. It's amazing how efficient the VOC is, it's simply stunning! Thank you guys!

    In case you missed this year's iteration of the conference, please have a look at our YouTube Channel. You'll find all of this year's talks there, as well the ones from last year. (For example, my welcome talk is available here). Enjoy!

    We hope to see you again next year, for systemd.conf 2017 in Berlin!

    The importance of paying attention in building community trust

    Posted by Matthew Garrett on October 03, 2016 05:14 PM
    Trust is important in any kind of interpersonal relationship. It's inevitable that there will be cases where something you do will irritate or upset others, even if only to a small degree. Handling small cases well helps build trust that you will do the right thing in more significant cases, whereas ignoring things that seem fairly insignificant (or saying that you'll do something about them and then failing to do so) suggests that you'll also fail when there's a major problem. Getting the small details right is a major part of creating the impression that you'll deal with significant challenges in a responsible and considerate way.

    This isn't limited to individual relationships. Something that distinguishes good customer service from bad customer service is getting the details right. There are many industries where significant failures happen infrequently, but minor ones happen a lot. Would you prefer to give your business to a company that handles those small details well (even if they're not overly annoying) or one that just tells you to deal with them?

    And the same is true of software communities. A strong and considerate response to minor bug reports makes it more likely that users will be patient with you when dealing with significant ones. Handling small patch contributions quickly makes it more likely that a submitter will be willing to do the work of making more significant contributions. These things are well understood, and most successful projects have actively worked to reduce barriers to entry and to be responsive to user requests in order to encourage participation and foster a feeling that they care.

    But what's often ignored is that this applies to other aspects of communities as well. Failing to use inclusive language may not seem like a big thing in itself, but it leaves people with the feeling that you're less likely to do anything about more egregious exclusionary behaviour. Allowing a baseline level of sexist humour gives the impression that you won't act if there are blatant displays of misogyny. The more examples of these "insignificant" issues people see, the more likely they are to choose to spend their time somewhere else, somewhere they can have faith that major issues will be handled appropriately.

    There's a more insidious aspect to this. Sometimes we can believe that we are handling minor issues appropriately, that we're acting in a way that handles people's concerns, while actually failing to do so. If someone raises a concern about an aspect of the community, it's important to discuss solutions with them. Putting effort into "solving" a problem without ensuring that the solution has the desired outcome is not only a waste of time, it alienates those affected even more - they're now not only left with the feeling that they can't trust you to respond appropriately, but that you will actively ignore their feelings in the process.

    It's not always possible to satisfy everybody's concerns. Sometimes you'll be left in situations where you have conflicting requests. In that case the best thing you can do is to explain the conflict and why you've made the choice you have, and demonstrate that you took this issue seriously rather than ignoring it. Depending on the issue, you may still alienate some number of participants, but it'll be fewer than if you just pretend that it's not actually a problem.

    One warning, though: while building trust in this way enhances people's willingness to join your community, it also builds expectations. If a significant issue does arise, and if you fail to handle it well, you'll burn a lot of that trust in the process. The fact that you've built that trust in the first place may be what saves your community from disintegrating completely, but people will feel even more betrayed if you don't actively work to rebuild it. And if there's a pattern of mishandling major problems, no amount of getting the details right will matter.

    Communities that ignore these issues are, long term, likely to end up weaker than communities that pay attention to them. Making sure you get this right in the first place, and setting expectations that you will pay attention to your contributors, is a vital part of building a meaningful relationship between your community and its members.

    comment count unavailable comments

    radv: status update or is Talos Principle rendering yet?

    Posted by Dave Airlie on September 27, 2016 04:33 AM
    The answer is YES!!

    I fixed the last bug with instance rendering and Talos renders great on radv now.

    Also with the semi-interesting branch vkQuake also renders, there are some upstream bugs that needs fixing in spirv/nir that I'm awaiting and upstream resolution on, but I've included some prelim fixes in semi-interesting for now, that'll go away when upstream fixes are decided on.

    Here's a screenshot:

    Comments about OARS and CSM age ratings

    Posted by Richard Hughes on September 22, 2016 07:40 AM

    I’ve had quite a few comments from people stating that using age rating classification values based on American culture is wrong. So far I’ve been using the Common Sense Media research (and various other psychology textbooks) to essentially clean-room implement a content-rating to appropriate age algorithm.

    Whilst I do agree that other cultures have different sensitivities (e.g. Smoking in Uganda, references to Nazis in Germany) there doesn’t appear to be much research on the suggested age ratings for different categories for those specific countries. Lots of things are outright banned for sale for various reasons (which the populous may completely ignore), but there doesn’t seem to be many statistics that back up the various anecdotal statements. For instance, are there any US-specific guidelines that say that the age rating for playing a game that involves taking illegal drugs should be 18, rather than the 14 which is inferred from CSM? Or the age rating should be 25+ for any game that features drinking alcohol in Saudi Arabia?

    Suggestions (especially references) welcome. Thanks!

    Microsoft aren't forcing Lenovo to block free operating systems

    Posted by Matthew Garrett on September 21, 2016 05:09 PM
    Update: Patches to fix this have been posted

    There's a story going round that Lenovo have signed an agreement with Microsoft that prevents installing free operating systems. This is sensationalist, untrue and distracts from a genuine problem.

    The background is straightforward. Intel platforms allow the storage to be configured in two different ways - "standard" (normal AHCI on SATA systems, normal NVMe on NVMe systems) or "RAID". "RAID" mode is typically just changing the PCI IDs so that the normal drivers won't bind, ensuring that drivers that support the software RAID mode are used. Intel have not submitted any patches to Linux to support the "RAID" mode.

    In this specific case, Lenovo's firmware defaults to "RAID" mode and doesn't allow you to change that. Since Linux has no support for the hardware when configured this way, you can't install Linux (distribution installers will boot, but won't find any storage device to install the OS to).

    Why would Lenovo do this? I don't know for sure, but it's potentially related to something I've written about before - recent Intel hardware needs special setup for good power management. The storage driver that Microsoft ship doesn't do that setup. The Intel-provided driver does. "RAID" mode prevents the Microsoft driver from binding and forces the user to use the Intel driver, which means they get the correct power management configuration, battery life is better and the machine doesn't melt.

    (Why not offer the option to disable it? A user who does would end up with a machine that doesn't boot, and if they managed to figure that out they'd have worse power management. That increases support costs. For a consumer device, why would you want to? The number of people buying these laptops to run anything other than Windows is miniscule)

    Things are somewhat obfuscated due to a statement from a Lenovo rep:This system has a Signature Edition of Windows 10 Home installed. It is locked per our agreement with Microsoft. It's unclear what this is meant to mean. Microsoft could be insisting that Signature Edition systems ship in "RAID" mode in order to ensure that users get a good power management experience. Or it could be a misunderstanding regarding UEFI Secure Boot - Microsoft do require that Secure Boot be enabled on all Windows 10 systems, but (a) the user must be able to manage the key database and (b) there are several free operating systems that support UEFI Secure Boot and have appropriate signatures. Neither interpretation indicates that there's a deliberate attempt to prevent users from installing their choice of operating system.

    The real problem here is that Intel do very little to ensure that free operating systems work well on their consumer hardware - we still have no information from Intel on how to configure systems to ensure good power management, we have no support for storage devices in "RAID" mode and we have no indication that this is going to get better in future. If Intel had provided that support, this issue would never have occurred. Rather than be angry at Lenovo, let's put pressure on Intel to provide support for their hardware.

    comment count unavailable comments

    GNOME Software and Age Ratings

    Posted by Richard Hughes on September 21, 2016 09:57 AM

    After all the tarballs for GNOME 3.22 the master branch of gnome-software is now open to new features. Along with the usual cleanups and speedups one new feature I’ve been working on is finally merging the age ratings work.

    screenshot-from-2016-09-21-10-22-36

    The age ratings are provided by the upstream-supplied OARS metadata in the AppData file (which can be generated easily online) and then an age classification is generated automatically using the advice from the appropriately-named Common Sense Media group. At the moment I’m not doing any country-specific mapping, although something like this will be required to show appropriate ratings when handling topics like alcohol and drugs.

    At the moment the only applications with ratings in Fedora 26 will be Steam games, but I’ve also emailed any maintainer that includes an <update_contact> email address in the appdata file that also identifies as a game in the desktop categories. If you ship an application with an AppData and you think you should have an age rating please use the generator and add the extra few lines to your AppData file. At the moment there’s no requirement for the extra data, although that might be something we introduce just for games in the future.

    I don’t think many other applications will need the extra application metadata, but if you know of any adult only applications (e.g. in Fedora there’s an application for the sole purpose of downloading p0rn) please let me know and I’ll contact the maintainer and ask what they think about the idea. Comments, as always, welcome. Thanks!

    libinput and the Lenovo T460 series trackstick

    Posted by Peter Hutterer on September 20, 2016 06:43 AM

    First a definition: a trackstick is also called trackpoint, pointing stick, or "that red knob between G, H, and B". I'll be using trackstick here, because why not.

    This post is the continuation of libinput and the Lenovo T450 and T460 series touchpads where we focused on a stalling pointer when moving the finger really slowly. Turns out the T460s at least, possibly others in the *60 series have another bug that caused a behaviour that is much worse but we didn't notice for ages as we were focusing on the high-precision cursor movement. Specifically, the pointer would just randomly stop moving for a short while (spoiler alert: 300ms), regardless of the movement speed.

    libinput has built-in palm detection and one of the things it does is to disable the touchpad when the trackstick is in use. It's not uncommon to rest the hand near or on the touchpad while using the trackstick and any detected touch would cause interference with the pointer motion. So events from the touchpad are ignored whenever the trackpoint sends events. [1]

    On (some of) the T460s the trackpoint sends spurious events. In the recording I have we have random events at 9s, then again 3.5s later, then 14s later, then 2s later, etc. Each time, our palm detection could would assume the trackpoint was in use and disable the touchpad for 300ms. If you were using the touchpad while this was happening, the touchpad would suddenly stop moving for 300ms and then continue as normal. Depending on how often these spurious events come in and the user's current caffeination state, this was somewhere between odd, annoying and infuriating.

    The good news is: this is fixed in libinput now. libinput 1.5 and the upcoming 1.4.3 releases will have a fix that ignores these spurious events and makes the touchpad stalls a footnote of history. Hooray.

    [1] we still allow touchpad physical button presses, and trackpoint button clicks won't disable the touchpad

    Understanding evdev

    Posted by Peter Hutterer on September 19, 2016 01:33 PM

    This post explains how the evdev protocol works. After reading this post you should understand what evdev is and how to interpret evdev event dumps to understand what your device is doing. The post is aimed mainly at users having to debug a device, I will thus leave out or simplify some of the technical details. I'll be using the output from evemu-record as example because that is the primary debugging tool for evdev.

    What is evdev?

    evdev is a Linux-only generic protocol that the kernel uses to forward information and events about input devices to userspace. It's not just for mice and keyboards but any device that has any sort of axis, key or button, including things like webcams and remote controls. Each device is represented as a device node in the form of /dev/input/event0, with the trailing number increasing as you add more devices. The node numbers are re-used after you unplug a device, so don't hardcode the device node into a script. The device nodes are also only readable by root, thus you need to run any debugging tools as root too.

    evdev is the primary way to talk to input devices on Linux. All X.Org drivers on Linux use evdev as protocol and libinput as well. Note that "evdev" is also the shortcut used for xf86-input-evdev, the X.Org driver to handle generic evdev devices, so watch out for context when you read "evdev" on a mailing list.

    Communicating with evdev devices

    Communicating with a device is simple: open the device node and read from it. Any data coming out is a struct input_event, defined in /usr/include/linux/input.h:


    struct input_event {
    struct timeval time;
    __u16 type;
    __u16 code;
    __s32 value;
    };
    I'll describe the contents later, but you can see that it's a very simple struct.

    Static information about the device such as its name and capabilities can be queried with a set of ioctls. Note that you should always use libevdevto interact with a device, it blunts the few sharp edges evdev has. See the libevdev documentation for usage examples.

    evemu-record, our primary debugging tool for anything evdev is very simple. It reads the static information about the device, prints it and then simply reads and prints all events as they come in. The output is in machine-readable format but it's annotated with human-readable comments (starting with #). You can always ignore the non-comment bits. There's a second command, evemu-describe, that only prints the description and exits without waiting for events.

    Relative devices and keyboards

    The top part of an evemu-record output is the device description. This is a list of static properties that tells us what the device is capable of. For example, the USB mouse I have plugged in here prints:


    # Input device name: "PIXART USB OPTICAL MOUSE"
    # Input device ID: bus 0x03 vendor 0x93a product 0x2510 version 0x110
    # Supported events:
    # Event type 0 (EV_SYN)
    # Event code 0 (SYN_REPORT)
    # Event code 1 (SYN_CONFIG)
    # Event code 2 (SYN_MT_REPORT)
    # Event code 3 (SYN_DROPPED)
    # Event code 4 ((null))
    # Event code 5 ((null))
    # Event code 6 ((null))
    # Event code 7 ((null))
    # Event code 8 ((null))
    # Event code 9 ((null))
    # Event code 10 ((null))
    # Event code 11 ((null))
    # Event code 12 ((null))
    # Event code 13 ((null))
    # Event code 14 ((null))
    # Event type 1 (EV_KEY)
    # Event code 272 (BTN_LEFT)
    # Event code 273 (BTN_RIGHT)
    # Event code 274 (BTN_MIDDLE)
    # Event type 2 (EV_REL)
    # Event code 0 (REL_X)
    # Event code 1 (REL_Y)
    # Event code 8 (REL_WHEEL)
    # Event type 4 (EV_MSC)
    # Event code 4 (MSC_SCAN)
    # Properties:
    The device name is the one (usually) set by the manufacturer and so are the vendor and product IDs. The bus is one of the "BUS_USB" and similar constants defined in /usr/include/linux/input.h. The version is often quite arbitrary, only a few devices have something meaningful here.

    We also have a set of supported events, categorised by "event type" and "event code" (note how type and code are also part of the struct input_event). The type is a general category, and /usr/include/linux/input-event-codes.h defines quite a few of those. The most important types are EV_KEY (keys and buttons), EV_REL (relative axes) and EV_ABS (absolute axes). In the output above we can see that we have EV_KEY and EV_REL set.

    As a subitem of each type we have the event code. The event codes for this device are self-explanatory: BTN_LEFT, BTN_RIGHT and BTN_MIDDLE are the left, right and middle button. The axes are a relative x axis, a relative y axis and a wheel axis (i.e. a mouse wheel). EV_MSC/MSC_SCAN is used for raw scancodes and you can usually ignore it. And finally we have the EV_SYN bits but let's ignore those, they are always set for all devices.

    Note that an event code cannot be on its own, it must be a tuple of (type, code). For example, REL_X and ABS_X have the same numerical value and without the type you won't know which one is which.

    That's pretty much it. A keyboard will have a lot of EV_KEY bits set and the EV_REL axes are obviously missing (but not always...). Instead of BTN_LEFT, a keyboard would have e.g. KEY_ESC, KEY_A, KEY_B, etc. 90% of device debugging is looking at the event codes and figuring out which ones are missing or shouldn't be there.

    Exercise: You should now be able to read a evemu-record description from any mouse or keyboard device connected to your computer and understand what it means. This also applies to most special devices such as remotes - the only thing that changes are the names for the keys/buttons. Just run sudo evemu-describe and pick any device in the list.

    The events from relative devices and keyboards

    evdev is a serialised protocol. It sends a series of events and then a synchronisation event to notify us that the preceeding events all belong together. This synchronisation event is EV_SYN SYN_REPORT, is generated by the kernel, not the device and hence all EV_SYN codes are always available on all devices.

    Let's have a look at a mouse movement. As explained above, half the line is machine-readable but we can ignore that bit and look at the human-readable output on the right.


    E: 0.335996 0002 0000 0001 # EV_REL / REL_X 1
    E: 0.335996 0002 0001 -002 # EV_REL / REL_Y -2
    E: 0.335996 0000 0000 0000 # ------------ SYN_REPORT (0) ----------
    This means that within one hardware event, we've moved 1 device unit to the right (x axis) and two device units up (y axis). Note how all events have the same timestamp (0.335996).

    Let's have a look at a button press:


    E: 0.656004 0004 0004 589825 # EV_MSC / MSC_SCAN 589825
    E: 0.656004 0001 0110 0001 # EV_KEY / BTN_LEFT 1
    E: 0.656004 0000 0000 0000 # ------------ SYN_REPORT (0) ----------
    E: 0.727002 0004 0004 589825 # EV_MSC / MSC_SCAN 589825
    E: 0.727002 0001 0110 0000 # EV_KEY / BTN_LEFT 0
    E: 0.727002 0000 0000 0000 # ------------ SYN_REPORT (0) ----------
    For button events, the value 1 signals button pressed, button 0 signals button released.

    And key events look like this:


    E: 0.000000 0004 0004 458792 # EV_MSC / MSC_SCAN 458792
    E: 0.000000 0001 001c 0000 # EV_KEY / KEY_ENTER 0
    E: 0.000000 0000 0000 0000 # ------------ SYN_REPORT (0) ----------
    E: 0.560004 0004 0004 458976 # EV_MSC / MSC_SCAN 458976
    E: 0.560004 0001 001d 0001 # EV_KEY / KEY_LEFTCTRL 1
    E: 0.560004 0000 0000 0000 # ------------ SYN_REPORT (0) ----------
    [....]
    E: 1.172732 0001 001d 0002 # EV_KEY / KEY_LEFTCTRL 2
    E: 1.172732 0000 0000 0001 # ------------ SYN_REPORT (1) ----------
    E: 1.200004 0004 0004 458758 # EV_MSC / MSC_SCAN 458758
    E: 1.200004 0001 002e 0001 # EV_KEY / KEY_C 1
    E: 1.200004 0000 0000 0000 # ------------ SYN_REPORT (0) ----------
    Mostly the same as button events. But wait, there is one difference: we have a value of 2 as well. For key events, a value 2 means "key repeat". If you're on the tty, then this is what generates repeat keys for you. In X and Wayland we ignore these repeat events and instead use XKB-based key repeat.

    Now look at the keyboard events again and see if you can make sense of the sequence. We have an Enter release (but no press), then ctrl down (and repeat), followed by a 'c' press - but no release. The explanation is simple - as soon as I hit enter in the terminal, evemu-record started recording so it captured the enter release too. And it stopped recording as soon as ctrl+c was down because that's when it was cancelled by the terminal. One important takeaway here: the evdev protocol is not guaranteed to be balanced. You may see a release for a key you've never seen the press for, and you may be missing a release for a key/button you've seen the press for (this happens when you stop recording). Oh, and there's one danger: if you record your keyboard and you type your password, the keys will show up in the output. Security experts generally reocmmend not publishing event logs with your password in it.

    Exercise: You should now be able to read a evemu-record events list from any mouse or keyboard device connected to your computer and understand the event sequence.This also applies to most special devices such as remotes - the only thing that changes are the names for the keys/buttons. Just run sudo evemu-record and pick any device listed.

    Absolute devices

    Things get a bit more complicated when we look at absolute input devices like a touchscreen or a touchpad. Yes, touchpads are absolute devices in hardware and the conversion to relative events is done in userspace by e.g. libinput. The output of my touchpad is below. Note that I've manually removed a few bits to make it easier to grasp, they will appear later in the multitouch discussion.


    # Input device name: "SynPS/2 Synaptics TouchPad"
    # Input device ID: bus 0x11 vendor 0x02 product 0x07 version 0x1b1
    # Supported events:
    # Event type 0 (EV_SYN)
    # Event code 0 (SYN_REPORT)
    # Event code 1 (SYN_CONFIG)
    # Event code 2 (SYN_MT_REPORT)
    # Event code 3 (SYN_DROPPED)
    # Event code 4 ((null))
    # Event code 5 ((null))
    # Event code 6 ((null))
    # Event code 7 ((null))
    # Event code 8 ((null))
    # Event code 9 ((null))
    # Event code 10 ((null))
    # Event code 11 ((null))
    # Event code 12 ((null))
    # Event code 13 ((null))
    # Event code 14 ((null))
    # Event type 1 (EV_KEY)
    # Event code 272 (BTN_LEFT)
    # Event code 325 (BTN_TOOL_FINGER)
    # Event code 328 (BTN_TOOL_QUINTTAP)
    # Event code 330 (BTN_TOUCH)
    # Event code 333 (BTN_TOOL_DOUBLETAP)
    # Event code 334 (BTN_TOOL_TRIPLETAP)
    # Event code 335 (BTN_TOOL_QUADTAP)
    # Event type 3 (EV_ABS)
    # Event code 0 (ABS_X)
    # Value 2919
    # Min 1024
    # Max 5112
    # Fuzz 0
    # Flat 0
    # Resolution 42
    # Event code 1 (ABS_Y)
    # Value 3711
    # Min 2024
    # Max 4832
    # Fuzz 0
    # Flat 0
    # Resolution 42
    # Event code 24 (ABS_PRESSURE)
    # Value 0
    # Min 0
    # Max 255
    # Fuzz 0
    # Flat 0
    # Resolution 0
    # Event code 28 (ABS_TOOL_WIDTH)
    # Value 0
    # Min 0
    # Max 15
    # Fuzz 0
    # Flat 0
    # Resolution 0
    # Properties:
    # Property type 0 (INPUT_PROP_POINTER)
    # Property type 2 (INPUT_PROP_BUTTONPAD)
    # Property type 4 (INPUT_PROP_TOPBUTTONPAD)
    We have a BTN_LEFT again and a set of other buttons that I'll explain in a second. But first we look at the EV_ABS output. We have the same naming system as above. ABS_X and ABS_Y are the x and y axis on the device, ABS_PRESSURE is an (arbitrary) ranged pressure value.

    Absolute axes have a bit more state than just a simple bit. Specifically, they have a minimum and maximum (not all hardware has the top-left sensor position on 0/0, it can be an arbitrary position, specified by the minimum). Notable here is that the axis ranges are simply the ones announced by the device - there is no guarantee that the values fall within this range and indeed a lot of touchpad devices tend to send values slightly outside that range. Fuzz and flat can be safely ignored, but resolution is interesting. It is given in units per millimeter and thus tells us the size of the device. in the above case: (5112 - 1024)/42 means the device is 97mm wide. The resolution is quite commonly wrong, a lot of axis overrides need the resolution changed to the correct value.

    The axis description also has a current value listed. The kernel only sends events when the value changes, so even if the actual hardware keeps sending events, you may never see them in the output if the value remains the same. In other words, holding a finger perfectly still on a touchpad creates plenty of hardware events, but you won't see anything coming out of the event node.

    Finally, we have properties on this device. These are used to indicate general information about the device that's not otherwise obvious. In this case INPUT_PROP_POINTER tells us that we need a pointer for this device (it is a touchpad after all, a touchscreen would instead have INPUT_PROP_DIRECT set). INPUT_PROP_BUTTONPAD means that this is a so-called clickpad, it does not have separate physical buttons but instead the whole touchpad clicks. Ignore INPUT_PROP_TOPBUTTONPAD because it only applies to the Lenovo *40 series of devices.

    Ok, back to the buttons: aside from BTN_LEFT, we have BTN_TOUCH. This one signals that the user is touching the surface of the touchpad (with some in-kernel defined minimum pressure value). It's not just for finger-touches, it's also used for graphics tablet stylus touchpes (so really, it's more "contact" than "touch" but meh).

    The BTN_TOOL_FINGER event tells us that a finger is in detectable range. This gives us two bits of information: first, we have a finger (a tablet would have e.g. BTN_TOOL_PEN) and second, we may have a finger in proximity without touching. On many touchpads, BTN_TOOL_FINGER and BTN_TOUCH come in the same event, but others can detect a finger hovering over the touchpad too (in which case you'd also hope for ABS_DISTANCE being available on the touchpad).

    Finally, the BTN_TOOL_DOUBLETAP up to BTN_TOOL_QUINTTAP tell us whether the device can detect 2 through to 5 fingers on the touchpad. This doesn't actually track the fingers, it merely tells you "3 fingers down" in the case of BTN_TOOL_TRIPLETAP.

    Exercise: Look at your touchpad's description and figure out if the size of the touchpad is correct based on the axis information [1]. Check how many fingers your touchpad can detect and whether it can do pressure or distance detection.

    The events from absolute devices

    Events from absolute axes are not really any different than events from relative devices which we already covered. The same type/code combination with a value and a timestamp, all framed by EV_SYN SYN_REPORT events. Here's an example of me touching the touchpad:


    E: 0.000001 0001 014a 0001 # EV_KEY / BTN_TOUCH 1
    E: 0.000001 0003 0000 3335 # EV_ABS / ABS_X 3335
    E: 0.000001 0003 0001 3308 # EV_ABS / ABS_Y 3308
    E: 0.000001 0003 0018 0069 # EV_ABS / ABS_PRESSURE 69
    E: 0.000001 0001 0145 0001 # EV_KEY / BTN_TOOL_FINGER 1
    E: 0.000001 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +0ms
    E: 0.021751 0003 0018 0070 # EV_ABS / ABS_PRESSURE 70
    E: 0.021751 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +21ms
    E: 0.043908 0003 0000 3334 # EV_ABS / ABS_X 3334
    E: 0.043908 0003 0001 3309 # EV_ABS / ABS_Y 3309
    E: 0.043908 0003 0018 0065 # EV_ABS / ABS_PRESSURE 65
    E: 0.043908 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +22ms
    E: 0.052469 0001 014a 0000 # EV_KEY / BTN_TOUCH 0
    E: 0.052469 0003 0018 0000 # EV_ABS / ABS_PRESSURE 0
    E: 0.052469 0001 0145 0000 # EV_KEY / BTN_TOOL_FINGER 0
    E: 0.052469 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +9ms
    In the first event you see BTN_TOOL_FINGER and BTN_TOUCH set (this touchpad doesn't detect hovering fingers). An x/y coordinate pair and a pressure value. The pressure changes in the second event, the third event changes pressure and location. Finally, we have BTN_TOOL_FINGER and BTN_TOUCH released on finger up, and the pressure value goes back to 0. Notice how the second event didn't contain any x/y coordinates? As I said above, the kernel only sends updates on absolute axes when the value changed.

    Ok, let's look at a three-finger tap (again, minus the ABS_MT_ bits):


    E: 0.000001 0001 014a 0001 # EV_KEY / BTN_TOUCH 1
    E: 0.000001 0003 0000 2149 # EV_ABS / ABS_X 2149
    E: 0.000001 0003 0001 3747 # EV_ABS / ABS_Y 3747
    E: 0.000001 0003 0018 0066 # EV_ABS / ABS_PRESSURE 66
    E: 0.000001 0001 014e 0001 # EV_KEY / BTN_TOOL_TRIPLETAP 1
    E: 0.000001 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +0ms
    E: 0.034209 0003 0000 2148 # EV_ABS / ABS_X 2148
    E: 0.034209 0003 0018 0064 # EV_ABS / ABS_PRESSURE 64
    E: 0.034209 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +34ms
    [...]
    E: 0.138510 0003 0000 4286 # EV_ABS / ABS_X 4286
    E: 0.138510 0003 0001 3350 # EV_ABS / ABS_Y 3350
    E: 0.138510 0003 0018 0055 # EV_ABS / ABS_PRESSURE 55
    E: 0.138510 0001 0145 0001 # EV_KEY / BTN_TOOL_FINGER 1
    E: 0.138510 0001 014e 0000 # EV_KEY / BTN_TOOL_TRIPLETAP 0
    E: 0.138510 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +23ms
    E: 0.147834 0003 0000 4287 # EV_ABS / ABS_X 4287
    E: 0.147834 0003 0001 3351 # EV_ABS / ABS_Y 3351
    E: 0.147834 0003 0018 0037 # EV_ABS / ABS_PRESSURE 37
    E: 0.147834 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +9ms
    E: 0.157151 0001 014a 0000 # EV_KEY / BTN_TOUCH 0
    E: 0.157151 0003 0018 0000 # EV_ABS / ABS_PRESSURE 0
    E: 0.157151 0001 0145 0000 # EV_KEY / BTN_TOOL_FINGER 0
    E: 0.157151 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +10ms
    In the first event, the touchpad detected all three fingers at the same time. So get BTN_TOUCH, x/y/pressure and BTN_TOOL_TRIPLETAP set. Note that the various BTN_TOOL_* bits are mutually exclusive. BTN_TOOL_FINGER means "exactly 1 finger down" and you can't have exactly 1 finger down when you have three fingers down. In the second event x and pressure update (y has no event, it stayed the same).

    In the event after the break, we switch from three fingers to one finger. BTN_TOOL_TRIPLETAP is released, BTN_TOOL_FINGER is set. That's very common. Humans aren't robots, you can't release all fingers at exactly the same time, so depending on the hardware scanout rate you have intermediate states where one finger has left already, others are still down. In this case I released two fingers between scanouts, one was still down. It's not uncommon to see a full cycle from BTN_TOOL_FINGER to BTN_TOOL_DOUBLETAP to BTN_TOOL_TRIPLETAP on finger down or the reverse on finger up.

    Exercise: test out the pressure values on your touchpad and see how close you can get to the actual announced range. Check how accurate the multifinger detection is by tapping with two, three, four and five fingers. (In both cases, you'll likely find that it's very much hit and miss).

    Multitouch and slots

    Now we're at the most complicated topic regarding evdev devices. In the case of multitouch devices, we need to send multiple touches on the same axes. So we need an additional dimension and that is called multitouch slots (there is another, older multitouch protocol that doesn't use slots but it is so rare now that you don't need to bother).

    First: all axes that are multitouch-capable are repeated as ABS_MT_foo axis. So if you have ABS_X, you also get ABS_MT_POSITION_X and both axes have the same axis ranges and resolutions. The reason here is backwards-compatibility: if a device only sends multitouch events, older programs only listening to the ABS_X etc. events won't work. Some axes may only be available for single-touch (ABS_MT_TOOL_WIDTH in this case).

    Let's have a look at my touchpad, this time without the axes removed:


    # Input device name: "SynPS/2 Synaptics TouchPad"
    # Input device ID: bus 0x11 vendor 0x02 product 0x07 version 0x1b1
    # Supported events:
    # Event type 0 (EV_SYN)
    # Event code 0 (SYN_REPORT)
    # Event code 1 (SYN_CONFIG)
    # Event code 2 (SYN_MT_REPORT)
    # Event code 3 (SYN_DROPPED)
    # Event code 4 ((null))
    # Event code 5 ((null))
    # Event code 6 ((null))
    # Event code 7 ((null))
    # Event code 8 ((null))
    # Event code 9 ((null))
    # Event code 10 ((null))
    # Event code 11 ((null))
    # Event code 12 ((null))
    # Event code 13 ((null))
    # Event code 14 ((null))
    # Event type 1 (EV_KEY)
    # Event code 272 (BTN_LEFT)
    # Event code 325 (BTN_TOOL_FINGER)
    # Event code 328 (BTN_TOOL_QUINTTAP)
    # Event code 330 (BTN_TOUCH)
    # Event code 333 (BTN_TOOL_DOUBLETAP)
    # Event code 334 (BTN_TOOL_TRIPLETAP)
    # Event code 335 (BTN_TOOL_QUADTAP)
    # Event type 3 (EV_ABS)
    # Event code 0 (ABS_X)
    # Value 5112
    # Min 1024
    # Max 5112
    # Fuzz 0
    # Flat 0
    # Resolution 41
    # Event code 1 (ABS_Y)
    # Value 2930
    # Min 2024
    # Max 4832
    # Fuzz 0
    # Flat 0
    # Resolution 37
    # Event code 24 (ABS_PRESSURE)
    # Value 0
    # Min 0
    # Max 255
    # Fuzz 0
    # Flat 0
    # Resolution 0
    # Event code 28 (ABS_TOOL_WIDTH)
    # Value 0
    # Min 0
    # Max 15
    # Fuzz 0
    # Flat 0
    # Resolution 0
    # Event code 47 (ABS_MT_SLOT)
    # Value 0
    # Min 0
    # Max 1
    # Fuzz 0
    # Flat 0
    # Resolution 0
    # Event code 53 (ABS_MT_POSITION_X)
    # Value 0
    # Min 1024
    # Max 5112
    # Fuzz 8
    # Flat 0
    # Resolution 41
    # Event code 54 (ABS_MT_POSITION_Y)
    # Value 0
    # Min 2024
    # Max 4832
    # Fuzz 8
    # Flat 0
    # Resolution 37
    # Event code 57 (ABS_MT_TRACKING_ID)
    # Value 0
    # Min 0
    # Max 65535
    # Fuzz 0
    # Flat 0
    # Resolution 0
    # Event code 58 (ABS_MT_PRESSURE)
    # Value 0
    # Min 0
    # Max 255
    # Fuzz 0
    # Flat 0
    # Resolution 0
    # Properties:
    # Property type 0 (INPUT_PROP_POINTER)
    # Property type 2 (INPUT_PROP_BUTTONPAD)
    # Property type 4 (INPUT_PROP_TOPBUTTONPAD)
    We have an x and y position for multitouch as well as a pressure axis. There are also two special multitouch axes that aren't really axes. ABS_MT_SLOT and ABS_MT_TRACKING_ID. The former specifies which slot is currently active, the latter is used to track touch points.

    Slots are a static property of a device. My touchpad, as you can see above ony supports 2 slots (min 0, max 1) and thus can track 2 fingers at a time. Whenever the first finger is set down it's coordinates will be tracked in slot 0, the second finger will be tracked in slot 1. When the finger in slot 0 is lifted, the second finger continues to be tracked in slot 1, and if a new finger is set down, it will be tracked in slot 0. Sounds more complicated than it is, think of it as an array of possible touchpoints.

    The tracking ID is an incrementing number that lets us tell touch points apart and also tells us when a touch starts and when it ends. The two values are either -1 or a positive number. Any positive number means "new touch" and -1 means "touch ended". So when you put two fingers down and lift them again, you'll get a tracking ID of 1 in slot 0, a tracking ID of 2 in slot 1, then a tracking ID of -1 in both slots to signal they ended. The tracking ID value itself is meaningless, it simply increases as touches are created.

    Let's look at a single tap:


    E: 0.000001 0003 0039 0387 # EV_ABS / ABS_MT_TRACKING_ID 387
    E: 0.000001 0003 0035 2560 # EV_ABS / ABS_MT_POSITION_X 2560
    E: 0.000001 0003 0036 2905 # EV_ABS / ABS_MT_POSITION_Y 2905
    E: 0.000001 0003 003a 0059 # EV_ABS / ABS_MT_PRESSURE 59
    E: 0.000001 0001 014a 0001 # EV_KEY / BTN_TOUCH 1
    E: 0.000001 0003 0000 2560 # EV_ABS / ABS_X 2560
    E: 0.000001 0003 0001 2905 # EV_ABS / ABS_Y 2905
    E: 0.000001 0003 0018 0059 # EV_ABS / ABS_PRESSURE 59
    E: 0.000001 0001 0145 0001 # EV_KEY / BTN_TOOL_FINGER 1
    E: 0.000001 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +0ms
    E: 0.021690 0003 003a 0067 # EV_ABS / ABS_MT_PRESSURE 67
    E: 0.021690 0003 0018 0067 # EV_ABS / ABS_PRESSURE 67
    E: 0.021690 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +21ms
    E: 0.033482 0003 003a 0068 # EV_ABS / ABS_MT_PRESSURE 68
    E: 0.033482 0003 0018 0068 # EV_ABS / ABS_PRESSURE 68
    E: 0.033482 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +12ms
    E: 0.044268 0003 0035 2561 # EV_ABS / ABS_MT_POSITION_X 2561
    E: 0.044268 0003 0000 2561 # EV_ABS / ABS_X 2561
    E: 0.044268 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +11ms
    E: 0.054093 0003 0035 2562 # EV_ABS / ABS_MT_POSITION_X 2562
    E: 0.054093 0003 003a 0067 # EV_ABS / ABS_MT_PRESSURE 67
    E: 0.054093 0003 0000 2562 # EV_ABS / ABS_X 2562
    E: 0.054093 0003 0018 0067 # EV_ABS / ABS_PRESSURE 67
    E: 0.054093 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +10ms
    E: 0.064891 0003 0035 2569 # EV_ABS / ABS_MT_POSITION_X 2569
    E: 0.064891 0003 0036 2903 # EV_ABS / ABS_MT_POSITION_Y 2903
    E: 0.064891 0003 003a 0059 # EV_ABS / ABS_MT_PRESSURE 59
    E: 0.064891 0003 0000 2569 # EV_ABS / ABS_X 2569
    E: 0.064891 0003 0001 2903 # EV_ABS / ABS_Y 2903
    E: 0.064891 0003 0018 0059 # EV_ABS / ABS_PRESSURE 59
    E: 0.064891 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +10ms
    E: 0.073634 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID -1
    E: 0.073634 0001 014a 0000 # EV_KEY / BTN_TOUCH 0
    E: 0.073634 0003 0018 0000 # EV_ABS / ABS_PRESSURE 0
    E: 0.073634 0001 0145 0000 # EV_KEY / BTN_TOOL_FINGER 0
    E: 0.073634 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +9ms
    We have a tracking ID (387) signalling finger down, as well as a position plus pressure. then some updates and eventually a tracking ID of -1 (signalling finger up). Notice how there is no ABS_MT_SLOT here - the kernel buffers those too so while you stay in the same slot (0 in this case) you don't see any events for it. Also notice how you get both single-finger as well as multitouch in the same event stream. This is for backwards compatibility [2]

    Ok, time for a two-finger tap:


    E: 0.000001 0003 0039 0496 # EV_ABS / ABS_MT_TRACKING_ID 496
    E: 0.000001 0003 0035 2609 # EV_ABS / ABS_MT_POSITION_X 2609
    E: 0.000001 0003 0036 3791 # EV_ABS / ABS_MT_POSITION_Y 3791
    E: 0.000001 0003 003a 0054 # EV_ABS / ABS_MT_PRESSURE 54
    E: 0.000001 0003 002f 0001 # EV_ABS / ABS_MT_SLOT 1
    E: 0.000001 0003 0039 0497 # EV_ABS / ABS_MT_TRACKING_ID 497
    E: 0.000001 0003 0035 3012 # EV_ABS / ABS_MT_POSITION_X 3012
    E: 0.000001 0003 0036 3088 # EV_ABS / ABS_MT_POSITION_Y 3088
    E: 0.000001 0003 003a 0056 # EV_ABS / ABS_MT_PRESSURE 56
    E: 0.000001 0001 014a 0001 # EV_KEY / BTN_TOUCH 1
    E: 0.000001 0003 0000 2609 # EV_ABS / ABS_X 2609
    E: 0.000001 0003 0001 3791 # EV_ABS / ABS_Y 3791
    E: 0.000001 0003 0018 0054 # EV_ABS / ABS_PRESSURE 54
    E: 0.000001 0001 014d 0001 # EV_KEY / BTN_TOOL_DOUBLETAP 1
    E: 0.000001 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +0ms
    E: 0.012909 0003 002f 0000 # EV_ABS / ABS_MT_SLOT 0
    E: 0.012909 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID -1
    E: 0.012909 0003 002f 0001 # EV_ABS / ABS_MT_SLOT 1
    E: 0.012909 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID -1
    E: 0.012909 0001 014a 0000 # EV_KEY / BTN_TOUCH 0
    E: 0.012909 0003 0018 0000 # EV_ABS / ABS_PRESSURE 0
    E: 0.012909 0001 014d 0000 # EV_KEY / BTN_TOOL_DOUBLETAP 0
    E: 0.012909 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +12ms
    This was a really quick two-finger tap that illustrates the tracking IDs nicely. In the first event we get a touch down, then an ABS_MT_SLOT event. This tells us that subsequent events belong to the other slot, so it's the other finger. There too we get a tracking ID + position. In the next event we get an ABS_MT_SLOT to switch back to slot 0. Tracking ID of -1 means that touch ended, and then we see the touch in slot 1 ended too.

    Time for a two-finger scroll:


    E: 0.000001 0003 0039 0557 # EV_ABS / ABS_MT_TRACKING_ID 557
    E: 0.000001 0003 0035 2589 # EV_ABS / ABS_MT_POSITION_X 2589
    E: 0.000001 0003 0036 3363 # EV_ABS / ABS_MT_POSITION_Y 3363
    E: 0.000001 0003 003a 0048 # EV_ABS / ABS_MT_PRESSURE 48
    E: 0.000001 0003 002f 0001 # EV_ABS / ABS_MT_SLOT 1
    E: 0.000001 0003 0039 0558 # EV_ABS / ABS_MT_TRACKING_ID 558
    E: 0.000001 0003 0035 3512 # EV_ABS / ABS_MT_POSITION_X 3512
    E: 0.000001 0003 0036 3028 # EV_ABS / ABS_MT_POSITION_Y 3028
    E: 0.000001 0003 003a 0044 # EV_ABS / ABS_MT_PRESSURE 44
    E: 0.000001 0001 014a 0001 # EV_KEY / BTN_TOUCH 1
    E: 0.000001 0003 0000 2589 # EV_ABS / ABS_X 2589
    E: 0.000001 0003 0001 3363 # EV_ABS / ABS_Y 3363
    E: 0.000001 0003 0018 0048 # EV_ABS / ABS_PRESSURE 48
    E: 0.000001 0001 014d 0001 # EV_KEY / BTN_TOOL_DOUBLETAP 1
    E: 0.000001 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +0ms
    E: 0.027960 0003 002f 0000 # EV_ABS / ABS_MT_SLOT 0
    E: 0.027960 0003 0035 2590 # EV_ABS / ABS_MT_POSITION_X 2590
    E: 0.027960 0003 0036 3395 # EV_ABS / ABS_MT_POSITION_Y 3395
    E: 0.027960 0003 003a 0046 # EV_ABS / ABS_MT_PRESSURE 46
    E: 0.027960 0003 002f 0001 # EV_ABS / ABS_MT_SLOT 1
    E: 0.027960 0003 0035 3511 # EV_ABS / ABS_MT_POSITION_X 3511
    E: 0.027960 0003 0036 3052 # EV_ABS / ABS_MT_POSITION_Y 3052
    E: 0.027960 0003 0000 2590 # EV_ABS / ABS_X 2590
    E: 0.027960 0003 0001 3395 # EV_ABS / ABS_Y 3395
    E: 0.027960 0003 0018 0046 # EV_ABS / ABS_PRESSURE 46
    E: 0.027960 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +27ms
    E: 0.051720 0003 002f 0000 # EV_ABS / ABS_MT_SLOT 0
    E: 0.051720 0003 0035 2609 # EV_ABS / ABS_MT_POSITION_X 2609
    E: 0.051720 0003 0036 3447 # EV_ABS / ABS_MT_POSITION_Y 3447
    E: 0.051720 0003 002f 0001 # EV_ABS / ABS_MT_SLOT 1
    E: 0.051720 0003 0036 3080 # EV_ABS / ABS_MT_POSITION_Y 3080
    E: 0.051720 0003 0000 2609 # EV_ABS / ABS_X 2609
    E: 0.051720 0003 0001 3447 # EV_ABS / ABS_Y 3447
    E: 0.051720 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +24ms
    [...]
    E: 0.272034 0003 002f 0000 # EV_ABS / ABS_MT_SLOT 0
    E: 0.272034 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID -1
    E: 0.272034 0003 002f 0001 # EV_ABS / ABS_MT_SLOT 1
    E: 0.272034 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID -1
    E: 0.272034 0001 014a 0000 # EV_KEY / BTN_TOUCH 0
    E: 0.272034 0003 0018 0000 # EV_ABS / ABS_PRESSURE 0
    E: 0.272034 0001 014d 0000 # EV_KEY / BTN_TOOL_DOUBLETAP 0
    E: 0.272034 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +30ms
    Note that "scroll" is something handled in userspace, so what you see here is just a two-finger move. Everything in there i something we've already seen, but pay attention to the two middle events: as updates come in for each finger, the ABS_MT_SLOT changes before the upates are sent. The kernel filter for identical events is still in effect, so in the third event we don't get an update for the X position on slot 1. The filtering is per-touchpoint, so in this case this means that slot 1 position x is still on 3511, just as it was in the previous event.

    That's all you have to remember, really. If you think of evdev as a serialised way of sending an array of touchpoints, with the slots as the indices then it should be fairly clear. The rest is then just about actually looking at the touch positions and making sense of them.

    Exercise: do a pinch gesture on your touchpad. See if you can track the two fingers moving closer together. Then do the same but only move one finger. See how the non-moving finger gets less updates.

    That's it. There are a few more details to evdev but much of that is just more event types and codes. The few details you really have to worry about when processing events are either documented in libevdev or abstracted away completely. The above should be enough to understand what your device does, and what goes wrong when your device isn't working. Good luck.

    [1] If not, file a bug against systemd's hwdb and CC me so we can put corrections in
    [2] We treat some MT-capable touchpads as single-touch devices in libinput because the MT data is garbage

    systemd.conf 2016 Workshop Tickets Available

    Posted by Lennart Poettering on September 17, 2016 10:00 PM

    Tickets for systemd 2016 Workshop day still available!

    We still have a number of ticket for the workshop day of systemd.conf 2016 available. If you are a newcomer to systemd, and would like to learn about various systemd facilities, or if you already know your way around, but would like to know more: this is the best chance to do so. The workshop day is the 28th of September, one day before the main conference, at the betahaus in Berlin, Germany. The schedule for the day is available here. There are five interesting, extensive sessions, run by the systemd hackers themselves. Who better to learn systemd from, than the folks who wrote it?

    Note that the workshop day and the main conference days require different tickets. (Also note: there are still a few tickets available for the main conference!).

    Buy a ticket here.

    See you in Berlin!

    synaptics pointer acceleration

    Posted by Peter Hutterer on September 16, 2016 06:00 AM

    libinput's touchpad acceleration is the cause for a few bugs and outcry from a quite vocal (maj|in)ority. A common suggestion is "make it like the synaptics driver". So I spent a few hours going through the pointer acceleration code to figure out what xf86-input-synaptics actually does (I don't think anyone knows at this point) [1].

    If you just want the TLDR: synaptics doesn't use physical distances but works in device units coupled with a few magic factors, also based on device units. That pretty much tells you all that's needed.

    Also a disclaimer: the last time some serious work was done on acceleration was in 2008/2009. A lot of things have changed since and since the server is effectively un-testable, we ended up with the mess below that seems to make little sense. It probably made sense 8 years ago and given that most or all of the patches have my signed-off-by it must've made sense to me back then. But now we live in the glorious future and holy cow it's awful and confusing.

    Synaptics has three options to configure speed: MinSpeed, MaxSpeed and AccelFactor. The first two are not explained beyond "speed factor" but given how accel usually works let's assume they all somewhoe should work as a multiplication on the delta (so a factor of 2 on a delta of dx/dy gives you 2dx/2dy). AccelFactor is documented as "acceleration factor for normal pointer movements", so clearly the documentation isn't going to help clear any confusion.

    I'll skip the fact that synaptics also has a pressure-based motion factor with four configuration options because oh my god what have we done. Also, that one is disabled by default and has no effect unless set by the user. And I'll also only handle default values here, I'm not going to get into examples with configured values.

    Also note: synaptics has a device-specific acceleration profile (the only driver that does) and thus the acceleration handling is split between the server and the driver.

    Ok, let's get started. MinSpeed and MaxSpeed default to 0.4 and 0.7. The MinSpeed is used to set constant acceleration (1/min_speed) so we always apply a 2.5 constant acceleration multiplier to deltas from the touchpad. Of course, if you set constant acceleration in the xorg.conf, then it overwrites the calculated one.

    MinSpeed and MaxSpeed are mangled during setup so that MaxSpeed is actually MaxSpeed/MinSpeed and MinSpeed is always 1.0. I'm not 100% why but the later clipping to the min/max speed range ensures that we never go below a 1.0 acceleration factor (and thus never decelerate).

    The AccelFactor default is 200/diagonal-in-device-coordinates. On my T440s it's thus 0.04 (and will be roughly the same for most PS/2 Synaptics touchpads). But on a Cyapa with a different axis range it is 0.125. On a T450s it's 0.035 when booted into PS2 and 0.09 when booted into RMI4. Admittedly, the resolution halfs under RMI4 so this possibly maybe makes sense. Doesn't quite make as much sense when you consider the x220t which also has a factor of 0.04 but the touchpad is only half the size of the T440s.

    There's also a magic constant "corr_mul" which is set as:


    /* synaptics seems to report 80 packet/s, but dix scales for
    * 100 packet/s by default. */
    pVel->corr_mul = 12.5f; /*1000[ms]/80[/s] = 12.5 */
    It's correct that the frequency is roughly 80Hz but I honestly don't know what the 100packet/s reference refers to. Either way, it means that we always apply a factor of 12.5, regardless of the timing of the events. Ironically, this one is hardcoded and not configurable unless you happen to know that it's the X server option VelocityScale or ExpectedRate (both of them set the same variable).

    Ok, so we have three factors. 2.5 as a function of MaxSpeed, 12.5 because of 80Hz (??) and 0.04 for the diagonal.

    When the synaptics driver calculates a delta, it does so in device coordinates and ignores the device resolution (because this code pre-dates devices having resolutions). That's great until you have a device with uneven resolutions like the x220t. That one has 75 and 129 units/mm for x and y, so for any physical movement you're going to get almost twice as many units for y than for x. Which means that if you move 5mm to the right you end up with a different motion vector (and thus acceleration) than when you move 5mm south.

    The core X protocol actually defines who acceleration is supposed to be handled. Look up the man page for XChangePointerControl(), it sets a threshold and an accel factor:

    The XChangePointerControl function defines how the pointing device moves. The acceleration, expressed as a fraction, is a multiplier for movement. For example, specifying 3/1 means the pointer moves three times as fast as normal. The fraction may be rounded arbitrarily by the X server. Acceleration only takes effect if the pointer moves more than threshold pixels at once and only applies to the amount beyond the value in the threshold argument.
    Of course, "at once" is a bit of a blurry definition outside of maybe theoretical physics. Consider the definition of "at once" for a gaming mouse with 500Hz sampling rate vs. a touchpad with 80Hz (let us fondly remember the 12.5 multiplier here) and the above description quickly dissolves into ambiguity.

    Anyway, moving on. Let's say the server just received a delta from the synaptics driver. The pointer accel code in the server calculates the velocity over time, basically by doing a hypot(dx, dy)/dtime-to-last-event. Time in the server is always in ms, so our velocity is thus in device-units/ms (not adjusted for device resolution).

    Side-note: the velocity is calculated across several delta events so it gets more accurate. There are some checks though so we don't calculate across random movements: anything older than 300ms is discarded, anything not in the same octant of movement is discarded (so we don't get a velocity of 0 for moving back/forth). And there's two calculations to make sure we only calculate while the velocity is roughly the same and don't average between fast and slow movements. I have my doubts about these, but until I have some more concrete data let's just say this is accurate (altough since the whole lot is in device units, it probably isn't).

    Anyway. The velocity is multiplied with the constant acceleration (2.5, see above) and our 12.5 magic value. I'm starting to think that this is just broken and would only make sense if we used a delta of "event count" rather than milliseconds.

    It is then passed to the synaptics driver for the actual acceleration profile. The first thing the driver does is remove the constant acceleration again, so our velocity is now just v * 12.5. According to the comment this brings it back into "device-coordinate based velocity" but this seems wrong or misguided since we never changed into any other coordinate system.

    The driver applies the accel factor (0.04, see above) and then clips the whole lot into the MinSpeed/MaxSpeed range (which is adjusted to move MinSpeed to 1.0 and scale up MaxSpeed accordingly, remember?). After the clipping, the pressure motion factor is calculated and applied. I skipped this above but it's basically: the harder you press the higher the acceleration factor. Based on some config options. Amusingly, pressure motion has the potential to exceed the MinSpeed/MaxSpeed options. Who knows what the reason for that is...

    Oh, and btw: the clipping is actually done based on the accel factor set by XChangePointerControl into the acceleration function here. The code is


    double acc = factor from XChangePointerControl();
    double factor = the magic 0.04 based on the diagonal;

    accel_factor = velocity * accel_factor;
    if (accel_factor > MaxSpeed * acc)
    accel_factor = MaxSpeed * acc;
    So we have a factor set by XChangePointerControl() but it's only used to determine the maximum factor we may have, and then we clip to that. I'm missing some cross-dependency here because this is what the GUI acceleration config bits hook into. Somewhere this sets things and changes the acceleration by some amount but it wasn't obvious to me.

    Alrighty. We have a factor now that's returned to the server and we're back in normal pointer acceleration land (i.e. not synaptics-specific). Woohoo. That factor is averaged across 4 events using the simpson's rule to smooth out aprupt changes. Not sure this really does much, I don't think we've ever done any evaluation on that. But it looks good on paper (we have that in libinput as well).

    Now the constant accel factor is applied to the deltas. So far we've added the factor, removed it (in synaptics), and now we're adding it again. Which also makes me wonder whether we're applying the factor twice to all other devices but right now I'm past the point where I really want to find out . With all the above, our acceleration factor is, more or less:


    f = units/ms * 12.5 * (200/diagonal) * (1.0/MinSpeed)
    and the deltas we end up using in the server are

    (dx, dy) = f * (dx, dy)
    But remember, we're still in device units here (not adjusted for resolution).

    Anyway. You think we're finished? Oh no, the real fun bits start now. And if you haven't headdesked in a while, now is a good time.

    After acceleration, the server does some scaling because synaptics is an absolute device (with axis ranges) in relative mode [2]. Absolute devices are mapped into the whole screen by default but when they're sending relative events, you still want a 45 degree line on the device to map into 45 degree cursor movement on the screen. The server does this by adjusting dy in-line with the device-to-screen-ratio (taking device resolution into account too). On my T440s this means:


    touchpad x:y is 1:1.45 (16:11)
    screen is 1920:1080 is 1:177 (16:9)

    dy scaling is thus: (16:11)/(16:9) = 9:11 -> y * 11/9
    dx is left as-is. Now you have the delta that's actually applied to the cursor. Except that we're in device coordinates, so we map the current cursor position to device coordinates, then apply the delta, then map back into screen coordinates (i.e. pixels). You may have spotted the flaw here: when the screen size changes, the dy scaling changes and thus the pointer feel. Plug in another monitor, and touchpad acceleration changes. Also: the same touchpad feels different on laptops when their screen hardware differs.

    Ok, let's wrap this up. Figuring out what the synaptics driver does is... "tricky". It seems much like a glorified random number scheme. I'm not planning to implement "exactly the same acceleration as synaptics" in libinput because this would be insane and despite my best efforts, I'm not that yet. Collecting data from synaptics users is almost meaningless, because no two devices really employ the same acceleration profile (touchpad axis ranges + screen size) and besides, there are 11 configuration options that all influence each other.

    What I do plan though is collect more motion data from a variety of touchpads and see if I can augment the server enough that I can get a clear picture of how motion maps to the velocity. If nothing else, this should give us some picture on how different the various touchpads actually behave.

    But regardless, please don't ask me to "just copy the synaptics code".

    [1] fwiw, I had this really great idea of trying to get behind all this, with diagrams and everything. But then I was printing json data from the X server into the journal to be scooped up by sed and python script to print velocity data. And I questioned some of my life choices.
    [2] why the hell do we do this? because synaptics at some point became a device that announce the axis ranges (seemed to make sense at the time, 2008) and then other things started depending on it and with all the fixes to the server to handle absolute devices in relative mode (for tablets) we painted ourselves into a corner. Synaptics should switch back to being a relative device, but last I tried it breaks pointer acceleration and that a) makes the internets upset and b) restoring the "correct" behaviour is, well, you read the article so far, right?

    LVFS and ODRS are down

    Posted by Richard Hughes on September 14, 2016 04:34 PM

    The LVFS firmware server and ODRS reviews server are down because my credit card registered with OpenShift expired. I’ve updated my credit card details, paid the pending invoice and still can’t start any server. I rang customer service who asked me to send an email and have heard nothing back.

    screenshot-from-2016-09-14-17-34-59

    I have backups a few days old, but this whole situation is terrible on so many levels.

    EDIT: cdaley has got everything back working again, it appears I found a corner case in the code that deals with payments.

    Input threads in the X server

    Posted by Peter Hutterer on September 09, 2016 07:21 AM

    A great new feature has been merged during this 1.19 X server development cycle: we're now using threads for input [1]. Previously, there were two options for how an input driver would pass on events to the X server: polling or from within the signal handler. Polling simply adds all input devices' file descriptors to a select(2) loop that is processed in the mainloop of the server. The downside here is that if the server is busy rendering something, your input is delayed until that rendering is complete. Historically, polling was primarily used by the keyboard driver because it just doesn't matter much when key strokes are delayed. Both because you need the client to render them anyway (which it can't when it's busy) and possibly also because we're just so bloody used to typing delays.

    The signal handler approach circumvented the delays by installing a SIGIO handler for each input device fd and calling that when any input occurs. This effectively interrupts the process until the signal handler completes, regardless of what the server is currently busy with. A great solution to provide immediate visible cursor movement (hence it is used by evdev, synaptics, wacom, and most of the now-retired legacy drivers) but it comes with a few side effects. First of all, because the main process is interrupted, the bit where we read the events must be completely separate to the bit where we process the events. That's easy enough, we've had an input event queue in the server for as long as I've been involved with X.Org development (~2006). The drivers push events into the queue during the signal handler, in the main loop the server reads them and processes them. In a busy server that may be several seconds after the pointer motion was performed on the screen but hey, it still feels responsive.

    The bigger issue with the use of a signal handler is: you can't use malloc [2]. Or anything else useful. Look at the man page for signal(7), it literally has a list of allowed functions. This leads to two weird side-effects: one is that you have to pre-allocate everything you may ever need for event processing, the other is that you need to re-implement any function that is not currently async signal safe. The server actually has its own implementation of printf for this reason (for error logging). Let's just say this is ... suboptimal. Coincidentally, libevdev is mostly async signal safe for that reason too. It also means you can't use any libraries, because no-one [3] is insane enough to make libraries async signal-safe.

    We were still mostly "happy" with it until libinput came along. libinput is a full input stack and expecting it to work within a signal handler is the somewhere between optimistic, masochistic and sadistic. The xf86-input-libinput driver doesn't use the signal handler and the side effect of this is that a desktop with libinput didn't feel as responsive when the server was busy rendering.

    Keith Packard stepped in and switched the server from the signal handler to using input threads. Or more specifically: one input thread on top of the main thread. That thread controls all the input device's file descriptors and continuously reads events off them. It otherwise provides the same functionality the signal handler did before: visible pointer movement and shoving events into the event queue for the main thread to process them later. But of course, once you switch to threads, problems have 2 you now. A signal handler is "threading light", only one code path can be interrupted and you know you continue where you left off. So synchronisation primitives are easier than in threads where both code paths continue independently. Keith replaced the previous xf86BlockSIGIO() calls with corresponding input_lock() and input_unlock() calls and all the main drivers have been switched over. But some interesting race conditions kept happening. But as of today, we think most of these are solved.

    The best test we have at this point is libinput's internal test suite. It creates roughly 5000 devices within about 4 minutes and thus triggers most code paths to do with device addition and removal, especially the overlaps between devices sending events before/during/after they get added and/or removed. This is the largest source of possible errors as these are the code paths with the most amount of actual simultaneous access to the input devices by both threads. But what the test suite can't test is normal everyday use. So until we get some more code maturity, expect the occasional crash and please do file bug reports. They'll be hard to reproduce and detect, but don't expect us to run into the same race conditions by accident.

    [1] Yes, your calendar is right, it is indeed 2016, not the 90s or so
    [2] Historical note: we actually mostly ignored this until about 2010 or so when glibc changed the malloc implementation and the server was just randomly hanging whenever we tried to malloc from within the signal handler. Users claimed this was bad UX, but I think it's right up there with motif.
    [3] yeah, yeah, I know, there's always exceptions.

    Fedora: Cinnamon, MATE and the broken GNOME touchpad panel

    Posted by Peter Hutterer on September 06, 2016 06:44 AM

    On Fedora, if you have mate-desktop or cinnamon-desktop installed, your GNOME touchpad configuration panel won't work (see Bug 1338585). Both packages install a symlink to assign the synaptics driver to the touchpad. But GNOME's control-center does not support synaptics anymore, so no touchpad is detected. Note that the issue occurs regardless of whether you use MATE/Cinnamon, merely installing it is enough.

    Unfortunately, there is no good solution to this issue. Long-term both MATE and Cinnamon should support libinput but someone needs to step up and implement it. We don't support run-time driver selection in the X server, so an xorg.conf.d snippet is the only way to assign a touchpad driver. And this means that you have to decide whether GNOME's or MATE/Cinnamon's panel is broken at X start-up time.

    If you need the packages installed but you're not actually using Mate/Cinnamon itself, remove the following symlinks (whichever is present on your system):


    # rm /etc/X11/xorg.conf.d/99-synaptics-mate.conf
    # rm /etc/X11/xorg.conf.d/99-synaptics-cinnamon.conf
    # rm /usr/share/X11/xorg.conf.d/99-synaptics-mate.conf
    # rm /usr/share/X11/xorg.conf.d/99-synaptics-cinnamon.conf
    The /usr/share paths are the old ones and have been replaced with the /etc/ symlinks in cinnamon-desktop-3.0.2-2.fc25 and mate-desktop-1.15.1-4.fc25 and their F24 equivalents.

    libinput and the Lenovo T450 and T460 series touchpads

    Posted by Peter Hutterer on September 06, 2016 02:00 AM

    I'm using T450 and T460 as reference but this affects all laptops from the Lenovo *50 and *60 series. The Lenovo T450 and T460 have the same touchpad hardware, but unfortunately it suffers from what is probably a firmware issue. On really slow movements, the pointer has a halting motion. That effect disappears when the finger moves faster.

    The observable effect is that of a pointer stalling, then jumping by 20 or so pixels. We have had a quirk for this in libinput since March 2016 (see commit a608d9) and detect this at runtime for selected models. In particular, what we do is look for a sequence of events that only update the pressure values but not the x/y position of the finger. This is a good indication that the bug triggers. While it's possible to trigger pressure changes alone, triggering several in a row without a change in the x/y coordinates is extremely unlikely. Remember that these touchpads have a resolution of ~40 units per mm - you cannot hold your finger that still while changing pressure [1]. Once we see those pressure changes only we reset the motion history we keep for each touch. The next event with an x/y coordinate will thus not calculate the delta to the previous position and not trigger a move. The event after that is handled normally again. This avoids the extreme jumps but there isn't anything we can do about the stalling - we never get the event from the kernel. [2]

    Anyway. This bug popped up again elsewhere so this time I figured I'll analyse the data more closely. Specifically, I wrote a script that collected all x/y coordinates of a touchpad recording [3] and produced a black and white image of all device coordinates sent. This produces a graphic that's interesting but not overly useful:


    Roughly 37000 touchpad events. You'll have to zoom in to see the actual pixels.
    I modified the script to assume a white background and colour any x/y coordinate that was never hit black. So an x coordinate of 50 would now produce a vertical 1 pixel line at 50, a y coordinate of 70 a horizontal line at 70, etc. Any pixel that remains white is a coordinate that is hit at some point, anything black was unreachable. This produced more interesting results. Below is the graphic of a short, slow movement right to left.

    A single short slow finger movement
    You can clearly see the missing x coordinates. More specifically, there are some events, then a large gap, then events again. That gap is the stalling cursor where we didn't get any x coordinates. My first assumption was that it may be a sensor issue and that some areas on the touchpad just don't trigger. So what I did was move my finger around the whole touchpad to try to capture as many x and y coordinates as possible.

    Let's have look at the recording from a T440 first because it doesn't suffer from this issue:


    Sporadic black lines indicating unused coordinates but the center is purely white, indicating every device unit was hit at some point
    Ok, looks roughly ok. The black areas are irregular, on the edges and likely caused by me just not covering those areas correctly. In the center it's white almost everywhere, that's where the most events were generated. And now let's compare this to a T450:

    A visible grid of unreachable device units
    The difference is quite noticeable, especially if you consider that the T440 recording had under 15000 events, the T450 recording had almost 37000. The T450 has a patterned grid of unreachable positions. But why? We currently use the PS/2 protocol to talk to the device but we should be using RMI4 over SMBus instead (which is what Windows has done for a while and luckily the RMI4 patches are on track for kernel 4.9). Once we talk to the device in its native protocol we see a resolution of ~20 units/mm and it looks like the T440 output:

    With RMI4, the grid disappears
    Ok, so the problem is not missing coordinates in the sensor and besides, at the resolution the touchpad has a single 'pixel' not triggering shouldn't be much of a problem anyway.

    Maybe the issue had to do with horizontal movements or something? The next approach was for me to move my finger slowly from one side to the left. That's actually hard to do consistently when you're not a robot, so the results are bound to be slightly different. On the T440:


    The x coordinates are sporadic with many missing ones, but the y coordinates are all covered
    You can clearly see where the finger moved left to right. The big black gaps on the x coordinates mostly reflect me moving too fast but you can see how the distance narrows, indicating slower movements. Most importantly: vertically, the strip is uniformly white, meaning that within that range I hit every y coordinate at least once. And the recording from the T450:

    Only one gap in the y range, sporadic gaps in the x range
    Well, still looks mostly the same, so what is happening here? Ok, last test: This time an extremely slow motion left to right. It took me 87 seconds to cover the touchpad. In theory this should render the whole strip white if all x coordinates are hit. But look at this:

    An extremely slow finger movement
    Ok, now we see the problem. This motion was slow enough that almost every x coordinate should have been hit at least once. But there are large gaps and most notably: larger gaps than in the recording above that was a faster finger movement. So what we have here is not an actual hardware sensor issue but that the firmware is working against us here, filtering things out. Unfortunately, that's also the worst result because while hardware issues can usually be worked around, firmware issues are a lot more subtle and less predictable. We've also verified that newer firmware versions don't fix this and trying out some tweaks in the firmware didn't change anything either.

    Windows is affected by this too and so is the synaptics driver. But it's not really noticeable on either and all reports so far were against libinput, with some even claiming that it doesn't manifest with synaptics. But each time we investigated in more detail it turns out that the issue is still there (synaptics uses the same kernel data after all) but because of different acceleration methods users just don't trigger it. So my current plan is to change the pointer acceleration to match something closer to what synaptics does on these devices. That's hard because synaptics is mostly black magic (e.g. synaptics' pointer acceleration depends on screen resolution) and hard to reproduce. Either way, until that is sorted at least this post serves as a link to point people to.

    Many thanks to Andrew Duggan from Synaptics and Benjamin Tissoires for helping out with the analysis and testing of all this.

    [1] Because pressing down on a touchpad flattens your finger and thus changes the shape slightly. While you can hold a finger still, you cannot control that shape
    [2] Yes, predictive movement would be possible but it's very hard to get this right
    [3] These are events as provided by the kernel and unaffected by anything in the userspace stack

    Fedora 25 and Additional Software Sources

    Posted by Richard Hughes on September 02, 2016 08:02 AM

    I was asked to produce a checklist for applications that we want to show up in GNOME Software in Fedora 25. In this post I’ll refer to applications as graphical programs, rather than other system add-on components like drivers and codecs (which the next post will talk about). There is a big checklist, which really is the bare minimum that the distributor has to provide so that the application is listed correctly. If any of these points is causing problems or is confusing, please let me know and I’ll do my best to help.

    So, these things really have to be done:

    • Verify that you ship a .desktop file for each built application, and that these keys exist: Name, Comment, Icon, Categories, Keywords and Exec and that desktop-file-validate correctly validates the file.
    • Verify that there is a PNG (with transparent background) or SVG icon is installed in /usr/share/icons, /usr/share/icons/hicolor/*/apps/*, or /usr/share/${app_name}/icons/* and is at least 64×64 in size.
    • At least one valid AppData file with the suffix .appdata.xml file must be installed into /usr/share/appdata with an <id> that matches the name of the .desktop file, e.g. gimp.appdata.xml. Ideally the name of both the desktop file and appdata should be reverse DNS, e.g. com.hughski.ColorHug.desktop rather than colorhug-client.desktop although this isn’t critically important.
    • Include several 16:9 aspect screenshots in the AppData file along with a compelling translated description made up of multiple paragraphs. Make sure you follow the style guide, which can be tested using appstream-util validate foo.appdata.xml
    • Make sure that there are not two applications installed with one package; in this case split up the package so that there are multiple subpackages or mark one of the .desktop files as NoDisplay=true. Make sure the application-subpackages depend on any -common subpackage and deal with upgrades (perhaps using a metapackage) if you’ve shipped the application before.
    • Make sure your application is visible in the example.xml.gz file when running appstream-builder on the binary rpm(s).
    • Make sure the AppStream metadata is regenerated when the application is updated in the repo, for more details see an entire blog post on this
    • Ensure that enabled_metadata=1 is set in the .repo file. This means that PackageKit will automatically download just the application metadata even when the repository is disabled.

    New xserver driver sort order - evdev < libinput < (synaptics|wacom|...)

    Posted by Peter Hutterer on August 31, 2016 12:01 AM

    In the X server, the input driver assignment is handled by xorg.conf.d snippets. Each driver assigns itself to the type of devices it can handle and the driver that actually loaded is simply the one that sorts last. Historically, we've had the evdev driver sort low and assign itself to everything. synaptics, wacom and the other few drivers that matter sorted higher than evdev and thus assigned themselves to the respective device.

    When xf86-input-libinput first came out 2 years ago, we used a higher sort order than all other drivers to assign it to (almost) all devices. This was of course intentional because we believe that libinput is the best input stack around, the odd bug non-withstanding. Now it has matured a fair bit and we had a lot more exposure to various types of hardware. We've been quirking and fixing things like crazy and libinput is much better for it.

    Two things were an issue with this approach though. First, overriding xf86-input-libinput required manual intervention, usually either copying or symlinking an xorg.conf.d snippet. Second, even though we were overriding the default drivers, we still had them installed everywhere. Now it's time to start properly retiring the old drivers.

    The upstream approach for this is fairly simple: the xf86-input-libinput xorg.conf.d snippet will drop in sort order to sit above evdev. evdev remains as the fallback driver for miscellaneous devices where evdev's blind "forward everything" approach is sufficient. All other drivers will sort higher than xf86-input-libinput and will thus override the xf86-input-libinput assignment. The new sort order is thus:

    • evdev
    • libinput
    • synaptics, wacom, vmmouse, joystick
    evdev and libinput are generic drivers, the others are for specific devices or use-cases. To use a specific driver other than xf86-input-libinput, you now only have to install it. To fall back to xf86-inputlibinput, you uninstall it. No more manual xorg.conf.d snippets symlinking.

    This has an impact on distributions and users. Distributions should ensure that other drivers are never installed by default unless requested by the user (or some software). And users need to be aware that having a driver other than xf86-input-libinput installed may break things. For example, recent GNOME does not support the synaptics driver anymore, installing it will cause the control panel's touchpad bits to stop working. So there'll be a messy transition period but once things are settled, the solution to most input-related driver bugs will be "install/remove driver $foo" as opposed to the current symlink/copy/write an xorg.conf.d snippet.

    radv: status update or is dota2 working yet?

    Posted by Dave Airlie on August 26, 2016 03:05 AM
    Clickbait titles for the win!

    First up, massive thanks to my major co-conspirator on radv, Bas Nieuwenhuizen, for putting in so much effort on getting radv going.

    So where are we at?

    Well this morning I finally found the last bug that was causing missing rendering on Dota 2. We were missing support for a compressed texture format that dota2 used. So currently dota 2 renders, I've no great performance comparison to post yet because my CPU is 5 years old, and can barely get close to 30fps with GL or Vulkan. I think we know of a couple of places that could be bottlenecking us on the CPU side. The radv driver is currently missing hyper-z (90% done), fast color clears and DCC, which are all GPU side speedups in theory. Also running the phoronix-test-suite dota2 tests works sometimes, hangs in a thread lock sometimes, or crashes sometimes. I think we have some memory corruption somewhere that it collides with.

    Other status bits: the Vulkan CTS test suite contains 114598 tests, a piglit run a few hours before I fixed dota2 was at:
    [114598/114598] skip: 50388, pass: 62932, fail: 1193, timeout: 2, crash: 83 - |/-\

    So that isn't too bad a showing, we know some missing features are accounting for some of fails. A lot of the crashes are an assert in CTS hitting, that I don't think is a real problem.

    We render most of the Sascha Willems demos fine.

    I've tested the Talos Principle as well, the texture fix renders a lot more stuff on the screen, but we are still seeing large chunks of blackness where I think there should be trees in-game, the menus etc all seem to load fine.

    All this work is on the semi-interesting branch of
    https://github.com/airlied/mesa

    It only has been tested on VI AMD GPUs, Polaris worked previously but something derailed it, but we should fix it once we get the finished bisect. CIK GPUs kinda work with the amdgpu kernel driver loaded. SI GPUs are nowhere yet.

    Here's a screenshot:

    Priorities in security

    Posted by Matthew Garrett on August 26, 2016 12:02 AM
    I read this tweet a couple of weeks ago:

    and it got me thinking. Security research is often derided as unnecessary stunt hacking, proving insecurity in things that are sufficiently niche or in ways that involve sufficient effort that the realistic probability of any individual being targeted is near zero. Fixing these issues is basically defending you against nation states (who (a) probably don't care, and (b) will probably just find some other way) and, uh, security researchers (who (a) probably don't care, and (b) see (a)).

    Unfortunately, this may be insufficient. As basically anyone who's spent any time anywhere near the security industry will testify, many security researchers are not the nicest people. Some of them will end up as abusive partners, and they'll have both the ability and desire to keep track of their partners and ex-partners. As designers and implementers, we owe it to these people to make software as secure as we can rather than assuming that a certain level of adversary is unstoppable. "Can a state-level actor break this" may be something we can legitimately write off. "Can a security expert continue reading their ex-partner's email" shouldn't be.

    comment count unavailable comments

    Summer Talks, PurpleEgg

    Posted by Owen Taylor on August 25, 2016 01:41 AM

    I recently gave talks at Flock in Krakow and GUADEC in Karlsruhe:

    Flock: What’s Fedora’s Alternative to vi httpd.conf Video Slides: PDF ODP
    GUADEC: Reworking the desktop distribution Video Slides: PDF ODP

    The topics were different but related: The Flock talk talked about how to make things better for a developer using Fedora Workstation as their development workstation, while the GUADEC talk was about the work we are doing to move Fedora to a model where the OS is immutable and separate from applications. A shared idea of the two talks is that your workstation is not your development environment environment. Installing development tools, language runtimes, and header files as part of your base operating system implies that every project you are developing wants the same development environment, and that simply is not the case.

    At both talks, I demo’ed a small project I’ve been working on with codename of PurpleEgg (I didn’t have that codename yet at Flock – the talk instead talks about “NewTerm” and “fedenv”.) PurpleEgg is about creating easily creating containerized environments dedicated to a project, and about integrating those projects into the desktop user interface in a natural, slick way.

    The command line client to PurpleEgg is called pegg:

    [otaylor@localhost ~]$ pegg create django mydjangosite
    [otaylor@localhost ~]$ cd ~/Projects/mydjangosite
    [otaylor@localhost mydangjosite]$  pegg shell
    [[mydjangosite]]$ python manage.py runserver
    August 24, 2016 - 19:11:36
    Django version 1.9.8, using settings 'mydjangosite.settings'
    Starting development server at http://127.0.0.1:8000/
    Quit the server with CONTROL-C.

    “pegg create” step did the following steps:

    • Created a directory ~/Projects/mydjangosite
    • Created a file pegg.yaml with the following contents:
    base: fedora:24
    packages:
    - python3-virtualenv
    - python3-django
    • Created a Docker image is the Fedora 24 base image plus the specified packages
    • Created a venv/ directory in the specified directory and initialized a virtual environment there
    • Ran ‘django-admin startproject’ to create the standard Django project

    pegg shell

    • Checked to see if the Docker image needed updating
    • Ran a bash prompt inside the Docker image with a customized prompt
    • Activated the virtual environment

    The end result is that, without changing the configuration of the host machine at all, in a few simple commands we got to a place where we can work on a Django project just as it is documented upstream.

    But with the PurpleEgg application installed, you get more: you get search results in the GNOME Activities Overview for your projects, and when you activate a search result, you see a window like:

    PurpleEgg-screenshot

    We have a terminal interface specialized for our project:

    • We already have the pegg environment activated
    • New tabs also open within that environment
    • The prompt is uncluttered with relevant information moved to the header bar
    • If the project is checked into Git, the header bar also tracks the Git branch

    There’s a fair bit more that could be done: a GUI for creating and importing projects as in GNOME Builder, GUI integration for Vagrant and Docker, configuring frequently used commands in pegg.yaml, etc.

    At the most basic, the idea is that server-side development is terminal-centric and also somewhat specialized – different languages and frameworks have different ways of doing things. PurpleEgg embraces working like that, but adds just enough conventions so that we can make things better for the developer – just because the developer wants a terminal doesn’t mean that all we can give them is a big pile of terminals.

    PurpleEgg codedump is here. Not warrantied to be fit for any purpose.


    Getting S3 Statistics using S3stat

    Posted by Richard Hughes on August 24, 2016 10:33 AM

    I’ve been using Amazon S3 as a CDN for the LVFS metadata for a few weeks now. It’s been working really well and we’ve shifted a huge number of files in that time already. One thing that made me very anxious was the bill that I was going to get sent by Amazon, as it’s kinda hard to work out the total when you’re serving cough millions of small files rather than a few large files to a few people. I also needed to keep track of which files were being downloaded for various reasons and the Amazon tools make this needlessly tricky.

    I signed up for the free trial of S3stat and so far I’ve been pleasantly surprised. It seems to do a really good job of graphing the spend per day and also allowing me to drill down into any areas that need attention, e.g. looking at the list of 404 codes various people are causing. It was fairly easy to set up, although did take a couple of days to start processing logs (which is all explained in the set up). Amazon really should be providing something similar.

    Screenshot from 2016-08-24 11-29-51

    For people providing less than 200,000 hits per day it’s only $10, which seems pretty reasonable. For my use case (bazillions of small files) it rises to a little-harder-to-justify $50/month.

    I can’t justify the $50/month for the LVFS, but luckily for me they have a Cheap Bastard Plan (their words, not mine!) which swaps a bit of advertising for a free unlimited license. Sounds like a fair swap, and means it’s available for a lot of projects where $600/yr is better spent elsewhere.

    Devo Firmware Updating

    Posted by Richard Hughes on August 24, 2016 08:32 AM

    Does anybody have a Devo RC transmitter I can borrow for a few weeks? I need model 6, 6S, 7E, 8, 8S, 10, 12, 12S, F7 or F12E — it doesn’t actually have to work, I just need the firmware upload feature for testing various things. Please reshare/repost if you’re in any UK RC groups that could help. Thanks!

    Updating Firmware on 8Bitdo Game Controllers

    Posted by Richard Hughes on August 18, 2016 10:15 AM

    I’ve spent a few days adding support for upgrading the firmware of the various wireless 8Bitdo controllers into fwupd. In my opinion, the 8Bitdo hardware is very well made and reasonably priced, and also really good retro fun.

    Although they use a custom file format for firmware, and also use a custom flashing protocol (seriously hardware people, just use DFU!) it was quite straightforward to integrate into fwupd. I’ve created a few things to make this all work:

    • a small libebitdo library in fwupd
    • a small ebitdo-tool binary that talks to the device and can flash a vendor supplied .dat file
    • a ebitdo fwupd provider that uses libebitdo to flash the device
    • a firmware repo that contains all the extra metadata for the LVFS

    I guess I need to thank the guys at 8Bitdo; after asking a huge number of questions they open sourced their OS-X and Windows flashing tools, and also allowed me to distribute the firmware binary on the LVFS. Doing both of those things made it easy to support the hardware.

    Screenshot from 2016-08-18 10-36-56

    The result of all this is that you can now do fwupd update when the game-pad is plugged in using the USB cable (not just connected via bluetooth) and the firmware will be updated to the latest version. Updates will show in GNOME Software, and the world is one step being closer to being awesome.