Fedora security Planet

On Password Management: Firefox Yes, Chrome No

Posted by Adam Young on November 16, 2017 03:51 PM

Summary: Firefox allows me to store passwords locally, encrypted, and password protected. Chrome wants to store passwords on line, which is not acceptable.

A recent, noticeable slowdown in Firefox (since alleviated) cause me to move more of my work over to Chrome. An essential part of this includes logging in to sites that I have password protected. My usual way to make a password is to run

uuidgen -r

And copy the result into the password field.  Since the output looks like this:


You can imagine that I don’t want to type that in every single time.

And, no, I did not just give you my password.  From the uuidgen man page:

There are two types of UUIDs which uuidgen can generate: time-based UUIDs and random-based UUIDs. By default uuidgen will generate a random-based UUID if a high-quality random number generator is
present. Otherwise, it will choose a time-based UUID. It is possible to force the generation of one of these two UUID types by using the -r or -t options.

-r, –random
Generate a random-based UUID. This method creates a UUID consisting mostly of random bits. It requires that the operating system have a high quality random number generator, such as

When I use Firefox, I store these in my local store.  To be precise, Firefox manages an NSS database.  This is a local, encrypted store that can be password protected.  I make use of that password protection.  That is the essential feature.

My local password to access my NSS Database is long, unique, and took me a long time to memorize, and I do not want it shared on other systems.  It is only used local to my primary work station.  Not a perfect setup, but better than the defaults.

I looked for a comparable feature in Chromium and found this under Manage Passwords:

Auto Sign-in
Automatically sign in to websites using stored credentials. If disabled, you will be asked for confirmation every time before signing in to a website.
Access your passwords from any device at passwords.google.com

So, no.  I do not want to share my password data with a third party.  I do not trust Google any more than I have to.  I do not want to send my password data over the wire for storage.

It is a constant battle to limit exposure.  Social media is a huge part of my life, and I don’t really like that.  I want to keep it as much under my own control as possible, or to split my exposure among different players.  Google owes me nothing more than my Google Play Music, as that is all I have paid them for.  Everything else they use for their own means, and I know and accept that.  But it means I have no reason to trust them, or any other social site.  I do not use off-machine password managers.  If I did, it would be one I installed and ran myself on a system that I paid for.

Not Google.



todo.txt done

Posted by Adam Young on November 15, 2017 10:51 PM

While I like the functionality of the todo.txt structure, I do not like the fact that done tasks stay in my todo list in perpetuity, and I also don’t want to lose them.  So, I’ve made a simple hack that allows me to move done items to a done folder.  Here’s the code:

awk '/^x/ {print $0}' ~/Dropbox/todo/todo.txt >> ~/Dropbox/todo/done.txt 
awk '!/^x/ {print $0}' ~/Dropbox/todo/todo.txt > ~/Dropbox/todo/todo2.txt
mv ~/Dropbox/todo/todo2.txt ~/Dropbox/todo/todo.txt


I call it todo_done.sh.

I copied my original to /tmp/pre in order to test and make sure I have a backup.  After running todo_done.sh I get:


$ diff -u /tmp/pre/todo.txt ~/Dropbox/todo/todo.txt
--- /tmp/pre/todo.txt 2017-11-15 17:46:21.794510999 -0500
+++ /home/ayoung/Dropbox/todo/todo.txt 2017-11-15 17:46:24.584515043 -0500
@@ -7,7 +7,6 @@
 2017-10-02 Expenses
 2017-10-04 Containerize hammer
 2017-10-06 Complete steam setup 
-x 2017-10-12 Trrc time resource reduce cost 
 2017-10-12 Whiteboard training 
 2017-10-14 Subscription manager extensions for skis or products? 
 2017-10-15 Workcenter is made up of 4 things: machine, man, method, measures.


$ diff -u /tmp/pre/done.txt ~/Dropbox/todo/done.txt 
--- /tmp/pre/done.txt 2017-11-15 17:46:17.914505377 -0500
+++ /home/ayoung/Dropbox/todo/done.txt 2017-11-15 17:46:24.580515037 -0500
@@ -26,3 +26,4 @@
 x 2017-10-19 Drs appt? 
 x 2017-11-02 Letter of Support
 x 2017-11-15 2017-09-27 LinkedIn TJX
+x 2017-10-12 Trrc time resource reduce cost

Different CloudForms Catalogs for Different Groups

Posted by Adam Young on November 15, 2017 02:37 AM

One of the largest value propositions of DevOps is the concept of Self Service provisioning. If you can remove human interaction from resource allocation, you can reduce both the response time and the likelihood of error in configuration. Red Hat CloudForms has a self service feature that allows a user to select from predefined services. You may wish to show different users different catalog items. This might be for security reasons, such as the set of credentials required and provided, or merely to reduce clutter and focus the end user on specific catalog items. Perhaps some items are still undergoing testing and are not ready for general consumption.

Obviously, these predefined services may not match your entire user population.

I’ve been working on setting up a CloudForms instance where members of different groups see different service catalogs. Here is what I did.

Tags are the primary tool used to match up users and their service catalogs. Specifically, A user will only see a catalog item if his group definition matches the Provisioning Scope tag of the Catalog Item. While you can make some catalog items to have a Provisioning Scope of All, you probably want to scope other items down to the target audience.

I have a demonstration setup based on IdM and CloudForms integration. When uses log in to the CloudForms appliance, one of the user groups managed by LDAP will be used to select their CloudForms group. The CloudForms group has a modified Provisioning Scope tag that will be used to select items from the service catalog.

I also have a top level tenant named “North America” that is used to manage the scope of the tags later on.  I won’t talk through setting this up, as most CloudForms deployment have something set as a top level tenant.

I’m not going to go through the steps to create a new catalog item.  There are other tutorials with go through this in detail.

My organization is adding support for statisticians.  Specifically, we need to provide support for VMs that are designed to support a customized version of the R programming environment.  All users that need these systems will be members of the stats group in IdM.  We want to be able to tag these instances with the stats Provisioning Scope as well.  The user is in the cloudusers group as well, which is required to provide access to the CloudForms appliance.

We start by having our sample user log in to the web UI.  This has the side effect of prepopulating the user and group data.  We could do this manually, but this way is less error prone, if a bit more of hassle.

My user currently only has a single item in her service catalog; the PostgreSQL appliance we make available to all developers.  This allows us to have a standard development environment for database work.

Log out and log back in as an administrator.  Here comes the obscure part.

Provisioning Scope tags are limited to set of valid values.  These values are, by default All or EVMGroup-user_self_service.  This second value matches a group with the same name.  In order to add an option, we need to modify the tag category associated with this tag.

  1. As an administrator, on the top right corner of the screen, click on your user name, and select the Configuration option from the dropdown.
  2. Select your region, in my case this is region 1.
  3. Across the top of the screen, you  will see Settings Region 1, and a series of tabs, most of which have the name of your tenant  (those of you that know my long standing issue with this term are probably grinning at my discomfort).  Since my top level tenant is “North America” I have a tab called North America Tags which I select. Select accordingly.
  4. Next to Category select “Provisioning Scope” from the drop down and you can see my existing set of custom tag values for Provisioning Scope.  Click on <New Entry> to add a new value, which I will call stats. I also use stats for the description.
  5. Click the Add button to the right.  See Below.

Now we can edit the newly defined “R Project” service to limit it to this provisioning scope.

  1. Navigate to Services->Catalogs->Catalog Items.
  2. Select the “R Project” Service.
  3. Click on the Policy  dropdown and select “Edit Tags”
  4. Click on the drop down to the right of “Select a customer tag to assign” (it is probably set on “Auto Approve -Max CPU *”) and scroll down to Provisioning Scope.
  5. The dropdown to the right, which defaults to “<Select a Value to Assign”>. Select this and scroll down to the new value.  For me, this is stats.  The new item will be added to the list.
  6. Click the Save button in the lower right of the screen.

Your list should look like this:

Finally, create the association between this provisioning scope and the stats group.

  1. From the dropdown on the top right of the screen that has your username, select Configuration.
  2. Expand the Access Control accordian
  3. Select groups.
  4. From the Configuration dropdown, select “Add a new Group”
  5. Select a Role for the user.  I use EvmRole-user_self_service
  6. Select a Project/Tenant for the user.
  7. Click on the checkbox labeled “Look Up External Authentiation Groups”
  8. A new field appears called “User to Look Up.”  I am going to user the “statuser” I created for this example, and click retrieve.
  9. The dropdown under the LDAP Groups for User is now populated.  I select stats.

To assign the tag for this group:

  1. Scroll down to the bottom of the page
  2. find and expand the “Provisioning Scope” tag
  3. Select “stats”
  4. Click the Add button in the bottom right corner of the page.

See Below.

Now when statuser logs in  to the self service web UI, they see both of the services provided:


One Big Caveat that has messed me up a few times:  a user only has one group active at a time.  If a user is a member of two groups, CloudForms will select one of them as the active group.  Services assigned only to the non-active group will not show up in the service catalog.  In my case, I had a group called cloudusers, and since all users are a member of that group, they would only see the Provisioning Scope, and thus the catalog items, for cloudusers, and not the stats group.

The Self Service webUI allows the user to change group to any of the other groups to which they are assigned.

The best option is to try and maintain a one to many relationship between groups and users;  constrain most users to a single group to avoid confusion.

This has been a long post.  The web UI for CloudForms requires a lot of navigation, and the concepts required to get this to work required more explanation than I originally had planned.  As I get more familiar with CloudForms, I’ll try to show how these types of operations can be automated from the command line, converted to Ansible playbooks, and thus checked in to version control.

I’ve also been told that, for simple use cases, it is possible to just put the user groups into separate tenants, and they will see different catalogs.  While that does not allow a single item to be in both catalogs, it is significantly easier to set up.

A Big Thank You to Laurent Domb for editing and corrections.

Episode 70 - The security of Intel ME

Posted by Open Source Security Podcast on November 14, 2017 11:54 PM
Josh and Kurt talk about Intel ME, Equifax salary history, and IoT.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5945821/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes

Creating yubikey SSH and TLS certificates

Posted by William Brown on November 10, 2017 02:00 PM

Creating yubikey SSH and TLS certificates

Recently yubikeys were shown to have a hardware flaw in the way the generated private keys. This affects the use of them to provide PIV identies or SSH keys.

However, you can generate the keys externally, and load them to the key to prevent this issue.


First, we’ll create a new NSS DB on an airgapped secure machine (with disk encryption or in memory storage!)

certutil -N -d . -f pwdfile.txt

Now into this, we’ll create a self-signed cert valid for 10 years.

certutil -S -f pwdfile.txt -d . -t "C,," -x -n "SSH" -g 2048 -s "cn=william,O=ssh,L=Brisbane,ST=Queensland,C=AU" -v 120

We export this now to PKCS12 for our key to import.

pk12util -o ssh.p12 -d . -k pwdfile.txt -n SSH

Next we import the key and cert to the hardware in slot 9c

yubico-piv-tool -s9c -i ssh.p12 -K PKCS12 -aimport-key -aimport-certificate -k

Finally, we can display the ssh-key from the token.

ssh-keygen -D /usr/lib64/opensc-pkcs11.so -e

Note, we can make this always used by ssh client by adding the following into .ssh/config:

PKCS11Provider /usr/lib64/opensc-pkcs11.so

TLS Identities

The process is almost identical for user certificates.

First, create the request:

certutil -d . -R -a -o user.csr -f pwdfile.txt -g 4096 -Z SHA256 -v 24 \
--keyUsage digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment --nsCertType sslClient --extKeyUsage clientAuth \
-s "CN=username,O=Testing,L=example,ST=Queensland,C=AU"

Once the request is signed, we should have a user.crt back. Import that to our database:

certutil -A -d . -f pwdfile.txt -i user.crt -a -n TLS -t ",,"

Import our CA certificate also. Next export this to p12:

pk12util -o user.p12 -d . -k pwdfile.txt -n TLS

Now import this to the yubikey - remember to use slot 9a this time!

yubico-piv-tool -s9a -i user.p12 -K PKCS12 -aimport-key -aimport-certificate -k


Changing the X.509 signature algorithm in FreeIPA

Posted by Fraser Tweedale on November 10, 2017 04:10 AM

X.509 certificates are an application of digital signatures for identity verification. TLS uses X.509 to create a chain of trust from a trusted CA to a service certificate. An X.509 certificate binds a public key to a subject by way of a secure and verifiable signature made by a certificate authority (CA).

A signature algorithm has two parts: a public key signing algorithm (determined by the type of the CA’s signing key) and a collision-resistant hash function. The hash function digests the certified data into a small value that is hard to find collision for, which gets signed.

Computers keep getting faster and attacks on cryptography always get better. So over time older algorithms need to be deprecated, and newer algorithms adopted for use with X.509. In the past the MD5 and SHA-1 digests were often used with X.509, but today SHA-256 (a variant of SHA-2) is the most used algorithm. SHA-256 is also the weakest digest accepted by many programs (e.g. web browsers). Stronger variants of SHA-2 are widely supported.

FreeIPA currently uses the sha256WithRSAEncryption signature algorithm by default. Sometimes we get asked about how to use a stronger digest algorithm. In this article I’ll explain how to do that and discuss the motivations and implications.

Implications of changing the digest algorithm

Unlike re-keying or changing the CA’s Subject DN, re-issuing a certificate signed by the same key, but using a different digest, should Just Work. As long as a client knows about the digest algorithm used, it will be able to verify the signature. It’s fine to have a chain of trust that uses a variety of signature algorithms.

Configuring the signature algorithm in FreeIPA

The signature algorithm is configured in each Dogtag certificate profile. Different profiles can use different signature algorithms. The public key signing algorithm depends on the CA’s key type (e.g. RSA) so you can’t change it; you can only change the digest used.

Modifying certificate profiles

Before FreeIPA 4.2 (RHEL 7.2), Dogtag stored certificate profile configurations as flat files. Dogtag 9 stores them in /var/lib/pki-ca/profiles/ca and Dogtag >= 10 stores them in /var/lib/pki/pki-tomcat/ca/profiles/ca. When Dogtag is using file-based profile storage you must modify profiles on all CA replicas for consistent behaviour. After modifying a profile, Dogtag requires a restart to pick up the changes.

As of FreeIPA 4.2, Dogtag uses LDAP-based profile storage. Changes to profiles get replicated among the CA replicas, so you only need to make the change once. Restart is not required. The ipa certprofile plugin provides commands for importing, exporting and modifying certificate profiles.

Because of the variation among versions, I won’t detail the process of modifying profiles. We’ll look at what modifications to make, but skip over how to apply them.

Profile configuration changes

For service certificates, the profile to modify is caIPAserviceCert. If you want to renew the CA signing cert with a different algorithm, modify the caCACert profile. The relevant profile policy components are signingAlgConstraintImpl and signingAlgDefaultImpl. Look for these components in the profile configuration:

policyset.serverCertSet.8.constraint.name=No Constraint
policyset.serverCertSet.8.default.name=Signing Alg

Update the policyset.<name>.<n>.default.params.signingAlg parameter; replace the - with the desired signing algorithm. (I set it to SHA512withRSA.) Ensure that the algorithm appears in the policyset.<name>.<n>.constraint.params.signingAlgsAllowed parameter (if not, add it).

After applying this change, certificates issued using the modified profile will use the specified algorithm.


After modifying the caIPAserviceCert profile, we can renew the HTTP certificate and see that the new certificate uses SHA512withRSA. Use getcert list to find the Certmonger tracking request ID for this certificate. We find the tracking request in the output:

Request ID '20171109075803':
  status: MONITORING
  stuck: no
  key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
  certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
  issuer: CN=Certificate Authority,O=IPA.LOCAL
  subject: CN=rhel69-0.ipa.local,O=IPA.LOCAL
  expires: 2019-11-10 07:53:11 UTC

So the tracking request ID is 20171109075803. Now resubmit the request:

[root@rhel69-0 ca]# getcert resubmit -i 20171109075803
Resubmitting "20171109075803" to "IPA".

After a few moments, check the status of the request:

[root@rhel69-0 ca]# getcert list -i 20171109075803
Number of certificates and requests being tracked: 8.
Request ID '20171109075803':
  status: MONITORING
  stuck: no
  key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
  certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
  issuer: CN=Certificate Authority,O=IPA.LOCAL
  subject: CN=rhel69-0.ipa.local,O=IPA.LOCAL
  expires: 2019-11-11 00:02:56 UTC

We can see by the expires field that renewal succeeded. Pretty-printing the certificate shows that it is using the new signature algorithm:

[root@rhel69-0 ca]# certutil -d /etc/httpd/alias -L -n 'Server-Cert'
        Version: 3 (0x2)
        Serial Number: 12 (0xc)
        Signature Algorithm: PKCS #1 SHA-512 With RSA Encryption
        Issuer: "CN=Certificate Authority,O=IPA.LOCAL"
            Not Before: Fri Nov 10 00:02:56 2017
            Not After : Mon Nov 11 00:02:56 2019
        Subject: "CN=rhel69-0.ipa.local,O=IPA.LOCAL"

It is using SHA-512/RSA. Mission accomplished.


In this article I showed how to configure the signing algorithm in a Dogtag certificate profile. Details about how to modify profiles in particular versions of FreeIPA was out of scope.

In the example I modified the default service certificate profile caIPAserviceCert to use SHA512withRSA. Then I renewed the HTTP TLS certificate to confirm that the configuration change had the intended effect. To change the signature algorithm on the FreeIPA CA certificate, you would modify the caCACert profile then renew the CA certificate. This would only work if the FreeIPA CA is self-signed. If it is externally-signed, it is up to the external CA what digest to use.

In FreeIPA version 4.2 and later, we support the addition of custom certificate profiles. If you want to use a different signature algorithm for a specific use case, instead of modifying the default profile (caIPAserviceCert) you might add a new profile.

The default signature digest algorithm in Dogtag is currently SHA-256. This is appropriate for the present time. There are few reasons why you would need to use something else. Usually it is because of an arbitrary security decision imposed on FreeIPA administrators. There are currently no plans to make the default signature algorithm configurable. But you can control the signature algorithm for a self-signed FreeIPA CA certificate via the ipa-server-install --ca-signing-algorithm option.

In the introduction I mentioned that the CA’s key type determines the public key signature algorithm. That was hand-waving; some key types support multiple signature algorithms. For example, RSA keys support two signature algorithms: PKCS #1 v1.5 and RSASSA-PSS. The latter is seldom used in practice.

The SHA-2 family of algorithms (SHA-256, SHA-384 and SHA-512) are the "most modern" digest algorithms standardised for use in X.509 (RFC 4055). The Russian GOST R digest and signature algorithms are also supported (RFC 4491) although support is not widespread. In 2015 NIST published SHA-3 (based on the Keccak sponge construction). The use of SHA-3 in X.509 has not yet been standardised. There was an Internet-Draft in 2017, but it expired. The current cryptanalysis of SHA-2 suggests there is no urgency to move to SHA-3. But it took a long time to move from SHA-1 (which is now insecure for applications requiring collision resistance) to SHA-2. Therefore it would be good to begin efforts to standardise SHA-3 in X.509 and add library/client support as soon as possible.

Episode 69 - Actionable security advice

Posted by Open Source Security Podcast on November 07, 2017 08:13 PM
Josh and Kurt talk about Amazon Key and actionable advice.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5920064/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes

What's the problem with NUMA anyway?

Posted by William Brown on November 06, 2017 02:00 PM

What’s the problem with NUMA anyway?

What is NUMA?

Non-Uniform Memory Architecture is a method of seperating ram and memory management units to be associated with CPU sockets. The reason for this is performance - if multiple sockets shared a MMU, they will cause each other to block, delaying your CPU.

To improve this, each NUMA region has it’s own MMU and RAM associated. If a CPU can access it’s local MMU and RAM, this is very fast, and does not prevent another CPU from accessing it’s own. For example:

CPU 0   <-- QPI --> CPU 1
  |                   |
  v                   v
MMU 0               MMU 1
  |                   |
  v                   v
RAM 1               RAM 2

For example, on the following system, we can see 1 numa region:

# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3
node 0 size: 12188 MB
node 0 free: 458 MB
node distances:
node   0
  0:  10

On this system, we can see two:

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33 34 35
node 0 size: 32733 MB
node 0 free: 245 MB
node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 36 37 38 39 40 41 42 43 44 45 46 47
node 1 size: 32767 MB
node 1 free: 22793 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

This means that on the second system there is 32GB of ram per NUMA region which is accessible, but the system has total 64GB.

The problem

The problem arises when a process running on NUMA region 0 has to access memory from another NUMA region. Because there is no direct connection between CPU 0 and RAM 1, we must communicate with our neighbour CPU 1 to do this for us. IE:

CPU 0 --> CPU 1 --> MMU 1 --> RAM 1

Not only do we pay a time delay price for the QPI communication between CPU 0 and CPU 1, but now CPU 1’s processes are waiting on the MMU 1 because we are retrieving memory on behalf of CPU 0. This is very slow (and can be seen by the node distances in the numactl –hardware output).

Today’s work around

The work around today is to limit your Directory Server instance to a single NUMA region. So for our example above, we would limit the instance to NUMA region 0 or 1, and treat the instance as though it only has access to 32GB of local memory.

It’s possible to run two instances of DS on a single server, pinning them to their own regions and using replication between them to provide synchronisation. You’ll need a load balancer to fix up the TCP port changes, or you need multiple addresses on the system for listening.

The future

In the future, we’ll be adding support for better copy-on-write techniques that allow the cores to better cache content after a QPI negotiation - but we still have to pay the transit cost. We can minimise this as much as possible, but there is no way today to avoid this penalty. To use all your hardware on a single instance, there will always be a NUMA cost somewhere.

The best solution is as above: run an instance per NUMA region, and internally provide replication for them. Perhaps we’ll support an automatic configuration of this in the future.

Episode 68 - Ruining the Internet

Posted by Open Source Security Podcast on November 01, 2017 02:14 PM
Josh and Kurt talk about Facebook listening to your microphone, Google Chrome certificate pinning, CAs, 152 ways to stay safe, and Kubernetes.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5899279/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes

Getting a Virtual Machine’s IP Address from virsh

Posted by Adam Young on October 27, 2017 03:25 PM

Ten Years later, and I finally know how to get virsh to tell me the IP address for a VM.

So, I have a virtual machine image called cfme. To launch it, I run

sudo virsh start cfme

That gets me a running virtual machine. To find the domain ID that it has been assigned:

$ sudo virsh list
 Id Name State
 1 cfme running

And to Find Its IP Address I use the domifaddr subcommand:

$ sudo virsh domifaddr 1
 Name MAC address Protocol Address
 vnet0 52:54:00:e5:5d:e3 ipv4

From which I can see that the address is

Wish I knew this years ago.  Ah well.  Better late then never.

Thanks to and this blog post for showing me the way:



In Satellite server, Organization != Organization_ID

Posted by Adam Young on October 25, 2017 04:59 PM

Spent a good chunk of yesterday and some of this morning stumped by a simple mistake I made.  I was trying to use our Satellite server via the Hammer command line tool to launch an instance, and kept getting and error in the content-view section.  I tried to list content views and got the same problem.  Here is a short section from my bash session.

$ hammer content-view list
Couldn't find organization 'DEMO'
$ hammer organization list
 5 | DEMO | DEMO | | DEMO

It turns out I had improperly set the default organization inside my config file.  I had:

$ cat ~/.hammer/defaults.yml 
 :value: DEMO


When I should have had

$ cat ~/.hammer/defaults.yml 
 :value: DEMO

Making this change got me moving ahead again.

Episode 67 - Cyber won

Posted by Open Source Security Podcast on October 24, 2017 06:19 PM
Josh and Kurt talk about hacking back, passwords, honeypots, and conspiracies.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5872803/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes

Where Did That Software Come From?

Posted by Russel Doty on October 20, 2017 05:29 PM

I have an article in the Oct. 9 issue of Military Embedded Systems magazine on software provenance titled Where Did That Software Come From?

Where did the software on your embedded system come from? Can you prove it? Can you safely update systems in the field? Cryptography provides the tools for verifying the integrity and provenance of software and data. There is a process as to how users can verify the source of software, if it was tampered with in transit, and if it was modified after installation.

The article explores how cryptography, especially hashing and code signing, can be use to establish the source and integrity. It examines how source code control systems and automated build systems are a key part of the software provenance story.  (Provenance means “a record of ownership of a work of art or an antique, used as a guide to authenticity or quality.” It is increasingly being applied to software.)

As an interesting side note, the article describes how the git version control system is very similar to a blockchain.

Deliberate Elevation of Privileges

Posted by Adam Young on October 18, 2017 07:31 PM

“Ooops.” — Me, doing something as admin that I didn’t mean to do.

While the sudo mechanism has some warranted criticism, it is still an improvement on doing everything as the root account. The essential addition that sudo provides for the average sys admin is the ability to only grant themselves system admin when they explicitly want it.

I was recently thinking about a FreeIPA based cluster where the users did not realize that they could get admin permissions by adding themselves to the user group admins. One Benefit to the centralized admin account is that a user has to chose to operate as admin to perform the operation. If a hacker gets the users password, they do not get admin. However, the number of attacks and weaknesses in this approach far outweigh the benefits. Multiple people need to know the password, revoking it for one revokes it for everyone, anyone can change the password, locking everyone else out, and so on.

We instead added a few key individuals to the admins group and changed the password on the admin account.

This heightened degree of security supports the audit trail. Now if someone performs and admin operation, we know which user did it. It involves enabling audit on the Directory Server (I need to learn how to do this!).

It got me thinking, though, if there was a mechanism like the sudo approach that we could implement for users to temporarily elevate them to admins status. Something like a short term group membership. The requirements, as I can see are these:

  1. A user has to chose to be admin:  “admin-powers activate!”
  2. A user can downgrade back to non-admin at any point: “admin-powers activate!”
  3. Admin powers wear off.  admin-powers only last an hour
  4. No new password has to be memorized for admin-powers
  5. The mechanism for admin-powers has to be resistant to attack.
    1. customizable enough that someone outside the organization can’t guess what they are.
    2. provide some way to prevent shoulder surfing.

I’m going to provide a straw-man here.

  • A REST API protected via SPNEGO
    • another endpoint with client cert possible, too
  • The REST API is password protected with basic-auth.  This is the group password.
  • The IPA service running the web server has the ability to add anyone that is in the “potentaladmins” group to the “admins” groups”
  • The IPA service also schedules an AT job to remove the user from the group.  If an AT entry already exists, remove the older one, so a user can extend their window.
  • A cron job runs each night to remove anyone from the admin group that does not have a current at job scheduled.

As I said, a strawman, but I think it points in the right direction.  Thoughts?

Episode 66 - Objects in mirror are less terrible than they appear

Posted by Open Source Security Podcast on October 16, 2017 03:59 PM
Josh and Kurt talk about Equifax again, Kaspersky, TLS CAs, coming change, social security numbers, and Minecraft.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5840636/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

Episode 65 - Will aliens overthrow us before AI?

Posted by Open Source Security Podcast on October 09, 2017 12:47 AM
Josh and Kurt talk about Apple, Equifax, passwords, AI, and aliens.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5819992/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

Episode 64 - Networks and Dnsmasq and IoT oh my

Posted by Open Source Security Podcast on October 03, 2017 07:18 PM
Josh and Kurt talk about networks, Dnsmasq, IoT, and our coming security dystopian future.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5804301/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

SELinux blocks loading kernel modules

Posted by Dan Walsh on September 28, 2017 07:12 PM

The kernel has a feature where it will load certain kernel modules for a process, when certain syscalls are made.  For example, loading a kernel module when a process attempts to create a different network socket.  

I wrote a blog on https://medium.com/cri-o explaining how this is probably a bad idea from a containers perspective.  I don't want to allow container processes to trigger modifications of the kernel.  And potentially causing the kernel to load risky modules that could have vulnerabilities in them.  I say, let the Administrator or packagers decide what kernel modules need to be loaded and then make the containers live with what is provided for them.  Here is a link to the blog.


Episode 63 - Shoot, Shovel, and Bury

Posted by Open Source Security Podcast on September 26, 2017 12:52 PM
Josh and Kurt talk about the Equifax breach (again) and what it will mean for all of us. Blueborne comes up, as well as #TrevorForget.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5777841/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

Measuring security: Part 2 - The cost of doing business

Posted by Josh Bressers on September 25, 2017 12:48 AM
If you've not read my last post on measuring security you probably should. It talks about how to measure the security of things that make money. That post is mostly focused on things like products that directly generate revenue. This time we're going to talk about a category I'm calling the cost of doing business.

The term "cost of doing business" is something I made up so I could group these ideas in some sensible way. At least sensible to me. You probably can't use this with other humans in a discussion, they won't know what you're talking about. If I had a line graph of spending I would put revenue generating on one side, the purse cost centers on the other side. The cost of doing business is somewhere in the middle. These are activities that directly support whatever it is the organization does to make new money. Projects and solutions that don't directly make money themselves but do directly support things being built that make money.

The cost of doing business includes things like compliance, sending staff to meetings, maybe regulatory requirements. Things that don't directly generate revenue but you can't move forward if you don't do these things. There's not a lot of options in many cases. If you don't have PCI compliance, you can't process payments, you can't make any money, and the company won't last long. If you don't attend certain meetings nobody can get any work done. Regulated industry must follow their requirements or the company can often just be shut down. Sometimes there are things we have to do, even if we don't want to do them.

In the next post we'll talk about what I call "infrastructure", these are the things that are seen as cost centers and often a commodity service (like electricity or internet access). I just want to clarify the difference. Infrastructure is something where you have choice or can decide not to do it with a possible negative (or positive) consequence. Infrastructure is what keep the lights on at a bare minimum. Cost of doing business must be done to get yourself to the next step in a project, there is no choice, which changes what we measure and how we measure it.

The Example

Let's pick on PCI compliance as it's pretty easy to understand example. If you don't do this it's quite likely your company won't survive, assuming you need to process card payments. If you're building a new web site that will process payments, you have to get through PCI compliance, there is no choice, and the project cannot move forward until this is complete. The goal now isn't so much measuring the return on an investment as it is being a good steward of the resources given to us. PCI requirements and audits are not cheap. If you are seen as making poor decisions and squandering your resources it's quite likely the business will get grumpy with you.

Compliance and security aren't the same thing. There is some overlap but it must be understood that you can be compliant and still get hacked. The overlap of compliance is a great thing to focus on for measuring what we do. Did your compliance program make you more secure? Can you show how another group used a compliance requirement to make something better? What if something compliance required saved some money on how the network was architected? There are a lot of side benefits to pay attention to. Make sure you note the things that are improvements, even if they aren't necessarily security improvements.

I've seen examples where compliance was used to justify 2 factor authentication (2FA) in an organization, There are few things more powerful than 2FA that you can deploy. Showing compliance helped move an initiative like this forward, and also showing how the number of malicious logs drops substantially is a powerful message. Just turning on 2FA isn't enough. Make sure you show why it's better, how the attacks are slowed or stopped. Make sure you can show there were few issues for users (the people who struggle will complain loudly). If there is massive disruption for your users, figure out why you didn't know this would happen, someone screwed something up that means. It's important to measure the good and the bad. We rarely measure failure which is a problem. Nobody has a 100% success rate, learn from your failure.

What about attending a meeting or industry conference? Do you just go, file the expense report, and do nothing? That sounds like a waste of time and money. Make sure you have concrete actions. Write down what happened, why it was important you were there, how you made the situation better, and what you're going to do next. How did the meeting move your project forward? Did you learn something new, or make some plans that will help in the future? Make sure the person paying your bills sees this. Make them happy to be providing you the means to keep the business moving forward.

The Cost

The very first step we have to consider when we want to measure what we're doing is to do your homework and understand cost. Not just upfront cost but cost of machines, disk, people, services, anything you need to keep the business moving forward. If there are certain requirements needed for a solution make sure you understand and document it. If a certain piece of software or service has to be used show why. Show what part of the business can function because of the cost you're providing. Remember this is going to be specific requirements you can't escape. These are not commodity services and solutions. And of course the goal is to move forward.

If you inherit an existing solution take a good look at everything, make sure you know exactly what the resource cost of the solution is. The goal here isn't always to show a return on investment, but to show that the current solution makes sense. Just because something costs less money doesn't mean it's cheaper. If your cut rate services will put the project in jeopardy you're going to be in trouble someday. Be able to show this is a real threat. It's possible a decision will be made to take on this threat, but that's not always your choice. Always be able to answer the questions "if we do this what happens" and "if we don't do this what happens".

This topic is tricky. I keep thinking about it and even as I wrote this post it changed quite a lot from what I started to write. If you have something that makes money it's easy to justify investment. If you have something that's a pure cost center it's easy to minimize cost. This middle ground is tricky. How do you show value for something you have to do but isn't directly generating revenue? If you work for a forward looking business you probably won't have to spend a ton of time getting these projects funded. Growing companies understand the cost of doing business.

I have seen some companies that aren't growing as quickly fail to see value in the cost of doing business. There's nothing wrong with this sometimes, but as a security leader your job is to make your leadership understand what isn't happening because of this lack of investment. Sometimes if you keep a project limping along, barely alive, you end up causing a great deal of damage to the project and your staff. If leadership won't fund something, it means they don't view it as important and neither should you. If you think it is important, you need to sell it to your leadership. Sometimes you can't and won't win though, and then you have to be willing to let it go.

A minor addition for todo.txt

Posted by Adam Young on September 23, 2017 05:57 PM

I had a simple todo list I managed using shell scripts and git, but I wanted something for the Cell phone. The todo.txt application fills that need now. But I was able to reuse something from my old approach to make it a little more command line friendly.

I want to be able to add a todo from the command line without thinking syntax or dates or anything. Here is the code:



TODOLINE=`date +%Y-%m-%d`
until [ -z "$1" ] # Until all parameters used up . . .

pushd $HOME/Dropbox/todo

echo $TODOLINE >> todo.txt

I have it in  ~/bin/add_todo~/bin is a part of $PATH.    To remember something, I type:

add_todo Meet with Jim on Monday

And In the corresponding document I get:

2017-09-23 Meet with Jim on Monday

The thing I like about the script is that it treats each command line argument as part of the message;  no quotes are required.

Why all the DAC_READ_SEARCH AVC messages?

Posted by Dan Walsh on September 22, 2017 04:26 PM

If you followed SELinux policy bugs being reported in bugzilla you might have noticed a spike in messages about random domains being denied DAC_READ_SEARCH.

Let's quickly look at what the DAC_READ_SEARCH capability is.  In Linux the power of "root" was broken down into 64 distinct capabilities.  Things like being able to load kernel modules or bind to ports less then 1024.  Well DAC_READ_SEARCH is one of these.

DAC stands for Discretionary Access Control, which is what most people understand as standard Linux permissions, Every process has owner/group.  All file system objects are  assigned owner, group and permission flags.  DAC_READ_SEARCH allows a privilege process to ignore parts of DAC for read and search.

man capabilities



              * Bypass file read permission checks and directory read and execute permission checks;

There is another CAPABILITY called DAC_OVERRIDE


              Bypass file read, write, and execute permission checks.

As you can see DAC_OVERRIDE is more powerful then DAC_READ_SEARCH, in that it can write and execute content ignoring DAC rules, as opposed to just reading the content.

Well why did we suddenly see a spike in confined domains needing DAC_READ_SEARCH?

Nothing in policy changed but the kernel changed.  Basically the kernel was made a little more secure.  Lets look at an example.  There is a small program called unix_chkpwd (checkpwd_t) which you end up executing when you log into the system.  This program reads /etc/shadow.  On Fedora/RHEL/CentOS and probably other GNU/Linux systems, /etc/shadow has 0000 Mode.  This means NO processes on the system, even if running as root (UID=0), are allowed to read/write /etc/shadow, unless they have a DAC Capability.

Well as policy evolved we saw that chckpwd_t needed to read /etc/shadow, it generated an DAC_OVERIDE AVC, so a policy developer gave that SELinux allow rule to checkpwd_t.  And for years things worked properly but then the kernel changed...

If a process tried to read /etc/shadow, it would be allowed if it had either DAC_OVERRIDE or DAC_READ_SEARCH. 


Older kernel's had pseudo code like


         Read a file with 0000 mode.

New Kernel switched to:


         Read a file with 0000 mode.

Since the chkpwd_t had DAC_OVERRIDE in the older kernels, it never checked DAC_READ_SEARCH and therefore DAC_READ_SEARCH was never added to policy.  Now it is checking DAC_READ_SEARCH first so we see the AVC being generated even though the final access was allowed.

This has generated a lot of noise for people from the SELinux logs, but really nothing got denied.  After the AVC was generated the access was still allowed.

The policy package maintainers have been updating all of the policy to allow these new DAC_READ_SEARCH, and I have suggested to them to start dropping alot of DAC_OVERRIDE policy from domains, since a lot of them including chkpwd_t don't need to write /etc/shadow, so they should be able to get by with only reading it.  This should eventually make SELinux policy more secure.

Learn about binary hardening with hardening-check

Posted by Jason Callaway on September 22, 2017 03:38 PM

In my last post, I wrote about using Radare2 to measure cyclomatic complexity, giving us a metric of the number of “moving parts” in a binary. From a risk quantification point of view, this is important because we assume that a more complicated program will be easier to get to misbehave.

But there are a lot of binaries in a Linux install. Let’s do a find against / and see for ourselves. Again, I find myself on my Macbook, so let’s do this from a docker container because it’ll be quicker than a VM. This assumes you already have Docker installed and configured.

mb:~ jason$ docker run --rm -ti fedora bash
[root@0600e63539d7 /]# dnf install -q -y file findutils
[root@0600e63539d7 /]# find / -exec file {} \; | grep -i elf | wc -l

Over a thousand, even in a small docker image! Now imagine you’re a security researcher, or maybe a 1337 hax0r, and you want to find a new vulnerability in one of those binaries. Which ones do you try to attack?

In this scenario, we need to be selective about which binaries we want to target, because fuzzing is time consuming. So which ones will give us the best bang for our buck, so to speak? We want to look for low-hanging fruit, and in order to do that we need to identify binaries that are, basically, poorly built.

This process — looking at binaries, scoring them in terms of how much effort it would take to successfully fuzz them and find a new 0-day — is what the Cyber-ITL is all about. And this is what we want to duplicate in the open source world with the Fedora Cyber Test Lab.

There are a number of mechanisms that have been built into Linux over the years to frustrate this process. I find a good way to learn about those mechanisms is to look at the code for a tool called hardening-check.

hardening-check was written by an a kernel security engineer at Google, Kees Cook, who, from the look of his Twitter profile, is a bit of a Stand Alone Complex fan. This tool came out around 2009, during which time Kees was working as an Ubuntu security engineer. Since then hardening-check has been picked up by other distros, and is just super-handy.

(In the next several posts, we’ll be referring to the hardening-check Perl source code, which is version controlled here.)

First, let’s install and run the tool to get a feel for it. We’ll also install gcc for the next example.

mb:~ jason$ docker run --rm -ti fedora bash
[root@59c1a1a14181 /]# dnf install -q -y hardening-check gcc
[root@59c1a1a14181 /]# hardening-check /bin/ls
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: yes (some protected functions found)
 Read-only relocations: yes
 Immediate binding: yes

Alright, that looks pretty good. Now let’s write another hello world program, but this time we’re going to deliberately use a notoriously unsafe function.

#include <stdio.h>

int main() {
  char str1[12] = "hello world";
  char str2[12];
  sprintf(str2, "%s", str1);
  return 0;

For now, we compile it without any extra flags, then run hardening check.

[root@59c1a1a14181 /]# gcc hello_world.c
[root@59c1a1a14181 /]# hardening-check a.out
 Position Independent Executable: no, normal executable!
 Stack protected: no, not found!
 Fortify Source functions: no, only unprotected functions found!
 Read-only relocations: yes
 Immediate binding: no, not found!

Not so great. But we can ask gcc to help us out.

[root@59c1a1a14181 /]# gcc -D_FORTIFY_SOURCE=2 -O2 -fpic -pie -z now -fstack-protector-all hello_world.c
[root@59c1a1a14181 /]# hardening-check a.out
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: yes
 Read-only relocations: yes
 Immediate binding: yes

That’s better. Now our silly little program is much harder to leverage in an attack.

This brings up a lot of questions, though. Why isn’t every binary compiled with those flags? Why don’t we just tweak gcc so it always applies those protections by default? For that matter, what do all those checks mean? Can they be defeated by attackers?

Stay tuned as we dig into each of those questions, and explore how they apply to our Cyber Test Lab.

U2F Protocol Overview

Posted by Nathaniel McCallum on September 19, 2017 04:04 PM

U2F Protocol Overview

This document serves as an overview of the U2F multi-factor authentication specifications as published by the FIDO Alliance. It does not replace the offical specifications but seeks to provide a quick run-through of how the protocol is built. U2F basically works via two main commands: register and authenticate.

When a server wants to use a U2F device for authentication, it asks the host (usually the client computer) to issue a register command to the device. This command creates a new signing key pair and the public key is returned to the server along with an opaque handle (for later reference) and an attestation certifciate (proving trust of the U2F device to the server).

Then, when it wishes to authenticate the user, the server asks the host to issue an authenticate command to the device. The device then signs a nonce from the server and returns the signature. This proves that the user is in posession of the key. It also returns a counter, proving to the server that the device has not been cloned.

U2F provides a high degree of security and a user-friendly workflow. It is supported by Chrome (natively) and Firefox (via an extension).

U2F Layers

The U2F Protocol happens across two layers: 1. The Transport Layer 2. The APDU Layer

All multi-byte integers are in network byte order (big endian).

The Transport Layer

The Transport Layer is responsible for breaking up packets into frames for transport from the computer to the U2F device over HID or Bluetooth. U2F-over-NFC does not use a transport layer. A full packet looks like this:

U2F Packet Field Size (Bytes)
Channel Identifier (CID) 4 (HID Only)
Command (CMD) 1
Data Length 2
Data 0+

The Channel Identifier (CID) is only used for the HID transport and is completely omitted over Bluetooth.

The CMD byte always has the first bit set and contains the number assigned to this particular command. Most commonly, this command will be the MSG command.

If the packet size is less than the maximum transmission unit (MTU), then it is sent directly. Otherwise, the packet is broken up into MTU-sized frames. The first frame is just the beginning of the packet truncated to fit in the frame. Subsequent frames are simply the additional fragments of the packet with a prefix SEQ byte (and, over HID, the CID). The SEQ byte always has the first bit unset, which allows you to distinguish continuation frames from the start of a new packet. It begins with a zero value and increments for each frame sent. A subsequent frame looks like this:

U2F Frame Field Size (Bytes)
Channel Identifier (CID) 4 (HID Only)
Sequence (SEQ) 1
Data 1+


Below you will find a survey of the most common Transport Layer commands. For the full details on all available commands, please see the U2F specifications.

MSG (CMD = 0x83)

The MSG command contains an APDU request or reply and represents the next layer in communication. This will be discussed in the APDU Layer section below.

PING (CMD = 0x81)

The PING command simply returns the packet to the sender without modification.

ERROR (CMD = 0x84)

The ERROR command is only ever sent as a response. The data contains a one-byte error code. This is only used to communicate errors at the Transport Layer. The APDU Layer has its own error handling.

INIT (CMD = 0x86, HID Only)

The INIT command is only used for HID. Its purpose is to allow the U2F device to assign a unique CID and to get some information about the hardware device. The request contains an 8-byte nonce. The reply data looks like this (fields are 1 byte unless otherwise noted):

U2F INIT Packet Data Field Size (Bytes)
Nonce 8
Channel Identifier (CID) 4
Protocol Version 1
Major Device Version 1
Minor Device Version 1
Build Device Version 1
Capability Flags 1

The request and reply are sent over the reserved broadcast channel (CID = 0xffffffff).

KEEPALIVE (CMD = 0x82, Bluetooth Only)

The KEEPALIVE command is only ever sent from Bluetooth U2F devices to the host. Its purpose is to identify to the host that the request is still being processed or is waiting for user confirmation. Its data contains a single byte indicating the reason for the delayed response.

The APDU Layer

When a MSG command packet is received by the device, the data contains an APDU request. Likewise, the host expects an APDU reply. I will not describe the APDU format in detail. Let it suffice to say that APDU is similar to the transport layer’s packet. It can represent a command (via the CLA and INS bytes), parameters to that command (the P1 and P2 bytes) and a miscelaneous data payload. Likewise, the APDU Layer has its own error handling for errors that occur within this layer.


VERSION (CLA = 0x00, INS = 0x03)

The VERSION command takes no input parameters and no input data. It simply replies with the version of the U2F Raw Message Format supported. There is currently only one version, which is represented as the UTF-8 encoding of the string U2F_V2 without a NULL terminator.

REGISTER (CLA = 0x00, INS = 0x01)

The REGISTER command takes no input parameters. The input data is in the following format:

U2F REGISTER Request Field Size (Bytes)
Challenge Parameter 32
Application Parameter 32

The U2F device will create a new P-256 signing keypair used for future authentications and will return back the following data:

U2F REGISTER Reply Field Size (Bytes)
Reserved (0x05) 1
Public Key 65
Key Handle Length 1
Key Handle 0-255
Attestation Certificate X.509 (DER)
ECDSA (P-256) Signature 71-73

The Public Key field contains the public key of the newly-created signing key pair. The Key Handle is an opaque field identifying the newly created key pair. The final two fields need some additional explanation.

Remote servers need a way to trust the U2F device. This comes in the form of an attestation key pair which may or may not be different from the newly-created signing key pair. The attestation key pair signs the following payload and inserts the signature into the reply (above):

U2F REGISTER Signature Field Size (Bytes)
Reserved (0x05) 1
Application Parameter 32
Challenge Parameter 32
Key Handle 0-255
Public Key 65

If the remote server trusts the attestation certifciate, then it can also trust the newly-generated signing key pair and key handle by validating the attestation signature. Usually, the server will trust the attestation certificate via an intermediary certification authority.

AUTHENTICATE (CLA = 0x00, INS = 0x02)

The AUTHENTICATE command takes a parameter (in P1) and input data in the following format:

U2F AUTHENTICATE Request Field Size (Bytes)
Challenge Parameter 32
Application Parameter 32
Key Handle Length 1
Key Handle 0-255

If P1 == 0x07 (check only), the device only verifies if the Application Parameter and Key Handle are valid for this U2F device. Otherwise, if P1 == 0x03 (enforce user presence and sign) or P1 == 0x08 (don’t enforce user presence and sign), the U2F device then signs a payload in the following format:

U2F AUTHENTICATE Signature Field Size (Bytes)
Application Parameter 32
User Presence Flags 1
Counter 4
Challenge Parameter 32

The counter is used to detect key theft from the U2F device. It may be global or per key handle. This payload is signed by the signing key pair.

Then a response is returned in the following format:

U2F AUTHENTICATE Reply Field Size (Bytes)
User Presence Flags 1
Counter 4
ECDSA (P-256) Signature 71-73

The User Presence field contains flags identifying the level of user presence testing.

Measuring cyclomatic complexity with Radare2

Posted by Jason Callaway on September 19, 2017 11:50 AM

Cyclomatic complexity is a metric that’s used to measure the complexity of a program. It’s one of many binary metrics tracked by the Cyber-ITL, and calculating it is a good first step in repeating their results.

Radare2 makes this super easy, and with r2pipe we can start building out our Python SDK for the Fedora Cyber Test Lab’s open source implementation of the Cyber-ITL’s approach.

macOS Environment Setup

It probably makes more sense to do this work on my Fedora box, but I happen to be on my Macbook. So we’ll use macOS as our environment for this post, which is, honestly, way more difficult.

First, let’s set up Homebrew. Note that in order to do so cleanly, we want to first give my user full control over /usr/local. I prefer to use facls to do this than changing ownership, which seems clumsy. Be sure to replace “jason” with your username.

sudo chmod -R +a "jason allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity,chown,file_inherit,directory_inherit" /usr/local

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

I also like to use the Homebrew version of Python, so we’ll install that as well.

brew install python virtualenv radare2
virtualenv -p /usr/local/bin/python ENV
source ENV/bin/activate
pip install r2pipe

We’re setting up a virtual environment here so we can isolate our installed libraries, but also to ensure we don’t run into PYTHONPATH issues, which is easy to do on macOS with multiple Python interpreters installed.

Simple analysis with r2

Radare2 (pronounced like “radar”) is a fantastic tool and, to be honest, I’ve only scratched its surface. Watching an experienced reverse engineer use r2 is like watching Michelangelo paint. Ok, I’ve never seen that, but I assume it was hella cool.

So let’s write a simple program and analyze it with r2. This assumes you have Xcode Command Line Tools installed, which is from where we’re getting gcc.

#include <stdio.h>

int main() {
  printf("hello world\n");
  return 0;

That should look familiar to everybody. Now let’s compile it, analyze it, and ask r2 to calculate its cyclomatic complexity.

(ENV) mb:~ jason$ gcc hello_world.c
(ENV) mb:~ jason$ r2 a.out
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
 -- For a full list of commands see `strings /dev/urandom`
[0x100000f60]> aaa
[x] Analyze all flags starting with sym. and entry0 (aa)
[x] Analyze len bytes of instructions for references (aar)
[x] Analyze function calls (aac)
[ ] [*] Use -AA or aaaa to perform additional experimental analysis.
[x] Constructing a function name for fcn.* and sym.func.* functions (aan))
[0x100000f60]> afcc
[0x100000f60]> q

Not super-interesting, there’s just a single path the code can take, so when we use the afcc command, or “analyze functions cyclomatic complexity,” we get back 1.

Using the formula the Wikipedia article, we can sanity check that result.

M = E − N + 2P,
E = the number of edges of the graph.
N = the number of nodes of the graph.
P = the number of connected components.

We get

M = 0 – 1 + 2 * 1
M = 1

For our purposes, P will almost always be 2, so we can treat that like a constant.

Ok, let’s add some more complexity.

#include <stdio.h>

int main() {
	int a = 1;
	if( a == 1 ){
		printf("hello world\n");
	return 0;

Compile it and analyze it.

(ENV) mb:~ jason$ gcc hello_world2.c
(ENV) mb:~ jason$ r2 a.out
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
 -- Rename a function using the 'afr <newname> @ <offset>' command.
[0x100000f50]> aaa
[x] Analyze all flags starting with sym. and entry0 (aa)
[x] Analyze len bytes of instructions for references (aar)
[x] Analyze function calls (aac)
[ ] [*] Use -AA or aaaa to perform additional experimental analysis.
[x] Constructing a function name for fcn.* and sym.func.* functions (aan))
[0x100000f50]> afcc
[0x100000f50]> agv
[0x100000f50]> q

Ok, with a conditional in our code we got the complexity to go up. But let’s use the command “agv” or “analyze graph web/png” to get a nice graphic representation of the function graph.

Now let’s use our formula, M = E – N + 2. Three edges, three nodes.

M = 3 – 3 + 2
M = 2

So that tracks. Now once more with an extra conditional statement.

#include <stdio.h>

int main() {
	int a = 1;
	if( a == 1 ){
		printf("hello world\n");
	if(a == 0){
		printf("goodbye world\n");
	return 0; 

Lather, rinse, repeat.

(ENV) mb:~ jason$ gcc hello_world3.c
(ENV) mb:~ jason$ r2 a.out
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
 -- Use '-e bin.strings=false' to disable automatic string search when loading the binary.
[0x100000f20]> aaa
[x] Analyze all flags starting with sym. and entry0 (aa)
[x] Analyze len bytes of instructions for references (aar)
[x] Analyze function calls (aac)
[ ] [*] Use -AA or aaaa to perform additional experimental analysis.
[x] Constructing a function name for fcn.* and sym.func.* functions (aan))
[0x100000f20]> afcc
[0x100000f20]> agv
[0x100000f20]> q

Verifying once again, six edges, five nodes:

M = 6 – 5 + 2
M = 3

You get the idea. These are really trivial examples, and when you ask r2 to analyze complex binaries, it can take a really long time. For fun, try the docker runtime, and see what a pain it is to deal with statically linked Go binaries.

Now let’s use r2pipe to do this from Python.

#!/usr/bin/env python

import os
import r2pipe
import sys

r2 = r2pipe.open(sys.argv[1])
cc = r2.cmdj("afcc")
print cc

And now we can do it from Python!

(ENV) mb:~ jason$ python r2cc.py a.out
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling

BTW, those errors are a known issue and you can ignore them on macOS for now.

Just for fun, let’s look at some macOS binaries.

(ENV) mb:~ jason$ python r2cc.py /bin/ls
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
(ENV) mb:~ jason$ python r2cc.py /usr/sbin/chown
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
(ENV) mb:~ jason$ python r2cc.py /usr/sbin/postfix
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
(ENV) mb:~ jason$ python r2cc.py /bin/bash
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling

Crazy that ls and chown are more complex that Postfix! And BASH is far more complex than either. Think of this value as a representation of the number of moving parts in a machine. The more complex it is, the more likely it’ll break.

But what about the quality of those parts in the machine? A complex but well-engineered machine will work better than a simple piece of junk. And so far, we’re only examining the machine while it’s turned off. That’s static analysis. We also need to watch the machine work, which is dynamic analysis. The Cyber-ITL has been doing both.

Stay tuned as we follow them down the quantitative analysis rabbit hole.

Introducing the Fedora Red Team

Posted by Jason Callaway on September 18, 2017 01:15 AM

Last week my colleagues and I started a new Special Interest Group under the Fedora Project: the Fedora Red Team.

The Fedora Red Team’s goal is to become Red Hat’s upstream cybersecurity community. But what does that actually mean?

“Cyber” is a fascinating term with a colorful history. I’ve written about it before, but the punchline is that we owe its ubiquity to William Gibson’s Neuromancer and an obscure US Government paper from the late 90s, referred to as The Marsh Report.

The context of The Marsh Report seems to have stuck — cyber has come to refer to the high-stakes interplay of offensive and defensive infosec, often at nation-state scale. If you want to get ontological, you could say “cyber” includes the three sub-domains of Computer Network Operations (CNO) — Computer Network Defense, Computer Network Exploitation, and Computer Network Attack.

So why a Fedora Red Team? My colleagues and I needed a place to work on offensive tooling, exploit curation, standards, and reference architectures — but we wanted to do it “the open source way.” A Fedora SIG gives us a community place to fail-fast these projects, a few of which I’ll mention here.

The Enterprise Linux Exploit Mapper (ELEM): as Mike Bursell wrote in his blog, many system administrators find themselves unable to patch. The CVE scoring system helps admins decide when to really push for patching, but many CVE descriptions contain language like “this vulnerability may allow an attacker to execute arbitrary code.” And there’s a reason for that — many vulnerabilities don’t have workable POCs. But what about the ones that do? ELEM makes it easy to map vulnerabilities on a local system to known exploits out in the wild. From a defensive perspective it creates a sort of super-criticality for admins so they can say to their management, “somebody can download this exploit and root our box right this minute.” A tool like this has good offensive value as well, and could save pentesters time doing these mappings manually.

The Fedora Cyber Test Lab (FCTL): I’ve written before about the Cyber-ITL and how important it is to the future of infosec. My only complaint is that their TTPs are not open source, and are not repeatable. So let’s fix that. This project will be an open source, automated, fully repeatable project for quantifying risk at a per-binary level. It will focus at first on RHEL, CentOS, and Fedora, but we’d love help from the community with adding other OS targets. I have a rudimentary first version ready to push to GitHub, which I’ll be blogging about in the coming days.

Penetration Testing Execution Standard (PTES): I’ve written before about how much I love PTES. In my mind, it’s a really important component of the art of pentesting. How can you tell a client that you’ve tested their security according to industry best practices without a yardstick by which to measure those practices? Without such a standard, pentesting relies too much on personal bona fides and flashy marketing. A standard like PTES can fix that. The only problem is that it hasn’t really been updated since 2014. Last summer, I rejiggered the wiki markup into ReStructured Text and put it on Read the Docs, which makes it easier to build community participation. The project is ready to be resurrected, and I hope that we’ll be able to work with the original team. But worst-case, we can fork it and go from there. This isn’t necessarily bad, and the PTES founders may have their reasons for wanting to let the project stay as it is. The Red Team SIG should know by early October which direction we’ll be taking.

I’m excited about this SIG, and will be haunting #fedora-security on Freenode IRC, as well as the security@lists.fedoraproject.org list. Every first Friday of the month we’ll be having meetings in IRC at 1400 UTC. Please join the conversation, or send us your pull requests for our GitHub projects.

Also, if you’re in Northern Virginia on 3 October 2017, we’ll be presenting ELEM at Defense in Depth. Register here and geek it up with us face-to-face!

Disclaimer: while I am talking about work stuff, this is my personal blog, and these views are my own and not my employer’s.

Episode 62 - All about the Equifax hack

Posted by Open Source Security Podcast on September 11, 2017 02:35 PM
Josh and Kurt talk about the Equifax breach and what it will mean for all of us.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5728237/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

Measuring security: Part 1 - Things that make money

Posted by Josh Bressers on September 11, 2017 12:48 PM
If you read my previous post on measuring security, you know I broke measuring into three categories. I have no good reason to do this other than it's something that made sense to me. There are without question better ways to split these apart, I'm sure there is even overlap, but that's not important. What actually matters is to start a discussion on measuring what we do. The first topic is about measuring security that directly adds to revenue such as a product or service.

The concept of making money is simple enough. You take a resource such as raw materials, money, even people in some instances. Usually it's all three. You take these resources then transform them into something new and better. The new creation is then turned into money, or revenue, for your
business. If you have a business that doesn't make more money than it spends you have a problem. If you have a business that doesn't make any money you have a disaster.

This is easy enough to understand, but let's use a grossly simplified example to make sure we're all on the same page. Let's say you're making widgets. I suppose since this is a security topic we should call them BlockWidgetChain. In our fictional universe you spend $10 on materials and people. Make sure you can track how much something costs, you should be able to determine how much of that $10 is materials and how much is people. You then you sell the BlockWidgetChain for $20. That means you spent $10 to make $20. This should make sense to anyone who understands math (or maths for you English speakers).

Now let's say you have a competitor who makes BlockChainWidgets. They're the same thing basically, but they have no idea how much it costs them to make BlockChainWidgets. They know if they charge more than $20 they can't compete because BlockWidgetChains cost $20. Their solution is to charge $20 and hope the books work out.

I've not only described the business plan for most startups but also a company that's almost certainly in trouble. You have to know how much you spend on resources. If you spend more than you're charging for the product that's a horrible business model. Most of security works like this unfortunately. We have no idea how much a lot of what we do costs, we certainly don't know how much value it adds to the bottom line. In many instances we cannot track spending in a meaningful way.

Measuring security
So now we're on to the idea of measuring security in an environment where the security is responsible for making money. Something like security features in a product. Maybe even a security product in some instances. This is the work that pays my bills. I've been working on product security for a very long time. If you're part of your product team (which you should be, product security doesn't belong anywhere else, more on that another day) then you understand the importance of having features that make a product profitable and useful. For example I would say SSO is a must have in today's environment. If you don't have this feature you can't be as effective in the market. But adding and maintaining features isn't free. If you spend $30 and sell it for $20, you'd make more money just by staying in bed. Sometimes the most profitable decision is to not do something.

Go big or go home
The biggest mistake we like to make is doing too much. It's easy to scope a feature too big. At worst you end up failing completely, at best you end up with what you should have scoped in the first place. But you spend a lot more on failure before you end up where you should have been from the start.

Let's use SSO as our example here. If you were going to scope the best SSO solution in the world, your product would be using SAML, OAuth, PKI, Kerberos, Active Directory, LDAP, and whatever else you manage to think of on planning day. This example is pretty clearly over the top, but I bet a lot of new SSO system scope SAML and OAuth at the same time. The reality is you only need one to start. You can add more later. Firstly having a small scope is important. It shows you want to do one thing and do it well instead of doing 3 things badly. There are few features that are useful in a half finished state. Your sales team has no desire to show off a half finished product.

How to decide
But how do we decide which feature to add? The first thing I do is look at customer feedback. Do the customers clearly prefer one over the other? Setup calls with them, go on visits. Learn what they do and how they do it. If this doesn't give you a clear answer, the next question is always "which feature would sell more product". In the case of something like SAML vs OAuth there might not be a good answer. If you're some sort of cloud service OAuth means you can let customers auth against Google and Facebook. That would probably result in more users.

If you're focused on a lot of on-prem solutions, SAML might be more used. It's even possible SSO isn't what customers are after once you start to dig. I find it's best to make a mental plan of how things should look, then make sure that's not what gets built because whatever I think of first is always wrong ;)

But how much does it cost?
Lastly if there's not a good way to show revenue for a feature, you can look at investment cost. The amount of time and money something will take to implement can really help when deciding what to do. If a feature will take years to develop, that's probably not a feature you want or need. Most industries will be very different in a few years. The expectations of today won't be the expectations of tomorrow.

For example if SAML will take three times as long as OAuth to implement. And both features will result in the same number of sales. OAuth will have a substantially larger return on investment as it's much cheaper to implement. A feature doesn't count for anything until it's on the market. Half done or in development are the same as "doesn't exist". Make sure you track time as part of your costs. Money is easy to measure, but people and time are often just as important.

I really do think this is the easiest security category to measure and justify. That could be because I do it every day, but I think if you can tie actual sales back to security features you'll find yourself in a good place. Your senior leadership will think you're magic if you can show them if they invest resources in X they will get Y. Make sure you track the metrics though. It's not enough to meet expectations, make an effort to exceed your expectations. There's nothing leadership likes better than someone who can over-deliver on a regular basis.

I see a lot of groups that don't do any of this. They wander in circles sometimes adding security features that don't matter, often engineering solutions that customers only need or want 10% of. I'll never forget when I first looked at actual metrics on new features and realized something we wanted to add was going to have a massive cost and generate zero additional revenue (it may have actually detracted in future product sales). On this day I saw the power in metrics. Overnight my group became heroes for saving everyone a lot of work and headaches. Sometimes doing nothing is the most valuable action you can take.

Solutions Architect

Posted by Adam Young on September 05, 2017 03:59 PM

Today is my first day at Red Hat! Well, OK, I’ve been here a few years, but today I move from Engineering to Sales. My new role is “Specialist Solutions Architect” where that specialty is Cloud.

I have a lot to learn, and I will try to use this site to record the most important and interesting details I learn.

What are the Cloud Products? well, according to Red Hat’s site, they are (please mentally prepend Red Hat to all of these) OpenStack Platform, OpenShift, CloudForms, Virtualization, Certificate System, Directory Server, as well as products bundles built out of these. Of these all, I’d guess I have the most to learn about CloudForms, as I’ve only recently started working with that. Really, though, I have a lot to learn across the board. I know that both Ansible Tower and Satellite server are major integration points for management of servers in the large, and I’ll be expected to provide expertise there as well. Plus, everything builds on the other product lines: RHEL and variants, as well as the Storage and Networking solutions.

This is going to be fun. Time to dig in.

Episode 61 - Market driven security

Posted by Open Source Security Podcast on September 05, 2017 12:17 PM
Josh and Kurt talk about our lack of progress in security, economics, and how to interact with peers.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5708244/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

The father of modern security: B. F. Skinner

Posted by Josh Bressers on September 04, 2017 01:43 PM
A lot of what we call security is voodoo. Most of it actually.

What I mean with that statement is our security process is often based on ideas that don't really work. As an industry we have built up a lot of ideas and processes that aren't actually grounded in facts and science. We don't understand why we do certain things, but we know that if we don't do those things something bad will happen! Will it really happen? I heard something will happen. I suspect the answer is no, but it's very difficult to explain this concept sometimes.

I'm going to start with some research B. F. Skinner did as my example here. The very short version is that Skinner did research on pigeons. He had a box that delivered food at random intervals. The birds developed rituals that they would do in order to have their food delivered. If a pigeon decided that spinning around would cause food to be delivered, it would continue to spin around, eventually the food would appear reinforcing the nonsensical behavior. The pigeon believed their ritual was affecting how often the food was delivered. The reality is nothing the pigeon did affected how often food was delivered. The pigeon of course didn't know this, they only knew what they experienced.

My favorite example  to use next to this pigeon experiment is the password policies of old. A long time ago someone made up some rules about what a good password should look like. A good password has letters, and numbers, and special characters, and the name of a tree in it. How often we should change a password was also part of this. Everyone knows you should change passwords as often as possible. Two or three times a day is best. The more you change it the more secure it is!

Today we've decided that all this advice was terrible. The old advice was based on voodoo. It was our ritual that kept us safe. The advice to some people seemed like a fair idea, but there were no facts backing it up. Lots of random characters seems like a good idea, but we didn't know why. Changing your password often seemed like a good idea, but we didn't know why. This wasn't much different than the pigeon spinning around to get more food. We couldn't prove it didn't not work, so we kept doing it because we had to do something.

Do you know why we changed all of our password advice? We changed it because someone did the research around passwords. We found out that very long passwords using real words is substantially better than a nonsense short password. We found out that people aren't good at changing their passwords every 90 days. They end up using horrible passwords and adding a 1 to the end. We measured the effectiveness of these processes and understood they were actually doing the opposite of what we wanted them to do. Without question there are other security ideas we do today that fall into this category.

Even though we have research showing this password advice was terrible we still see a lot of organizations and people who believe the old rituals are the right way to keep passwords safe. Sometimes even when you prove something to someone they can't believe it. They are so invested in their rituals that they are unable to imagine any other way of existing. A lot of security happens this way. How many of our rules and processes are based on bad ideas?

How to measure
Here's where it gets real. It's easy to pick on the password example because it's in the past. We need to focus on the present and the future. You have an organization that's full of policy, ideas, and stuff. How can we try to make a dent in what we have today? What matters? What doesn't work, and what's actually harmful?

I'm going to split everything into 3 possible categories. We'll dive deeper into each in future posts, but we'll talk about them briefly right now.

Things that make money
Number one is things that make money. This is something like a product you sell, or a website that customers use to interact with your company. Every company does something that generates revenue. Measuring things that fit into this category is really easy. You just ask "Will this make more, less, or the same amount of money?" If the answer is less you're wasting your time. I wrote about this a bit a long time ago, the post isn't great, but the graphic I made is useful, print it out and plot your features on it. You can probably start asking this question today without much excitement.

Cost of doing business
The next category is what I call cost of doing business. This would be things like compliance or being a part of a professional organization. Sending staff to conferences and meetings. Things that don't directly generate revenue but can have a real impact on the revenue. If you don't have PCI compliance, you can't process payments, you have no revenue, and the company won't last long. Measuring some of these is really hard. Does sending someone to Black Hat directly generate revenue? No. But it will create valuable connections and they will likely learn new things that will be a benefit down the road. I guess you could think of these as investments in future revenue.

My thoughts on how to measure this one is less mature. I think about these often. I'll elaborate more in a future post.

The last category I'm going to call "infrastructure". This one is a bit harder to grasp what makes sense. It's not unlike the previous question though. In this case we ask ourselves "If I stopped doing this what bad thing would happen?" Now I don't mean movie plot bad thing. Yeah if you stopped using your super expensive keycard entry system a spy from a competitor could break in and steal all your secrets using an super encrypted tor enabled flash drive, but they probably won't. This is the category where you have to consider the cost of an action vs the cost of not doing an action. Not doing things will often have a cost, but doing things also has a cost.

Return on investment is the name of the game here. Nobody likes to spend money they don't have to. This is why cloud is disrupting everything. Why pay for servers you don't need when you can rent only what you do need?

I have some great stories for this category, be sure to come back when I publish this followup article.

The homework for everyone now is to just start thinking about what you do and why you do it. If you don't have a good reason, you need to change your thinking. Changing your thinking is really hard to do as a human though. Many of us like to double down on our old beliefs when presented with facts. Don't be that person, keep an open mind.

Running Keycloak in OpenShift

Posted by Fraser Tweedale on September 04, 2017 06:26 AM

At PyCon Australia in August I gave a presentation about federated and social identity. I demonstrated concepts using Keycloak, an Open Source, feature rich identity broker. Keycloak is deployed in JBoss, so I wasn’t excited about the prospect of setting up Keycloak from scratch. Fortunately there is an official Docker image for Keycloak, so with that as the starting point I took an opportunity to finally learn about OpenShift v3, too.

This post is simply a recounting of how I ran Keycloak on OpenShift. Along the way we will look at how to get the containerised Keycloak to trust a private certificate authority (CA).

One thing that is not discussed is how to get Keycloak to persist configuration and user records to a database. This was not required for my demo, but it will be important in a production deployment. Nevertheless I hope this article is a useful starting point for someone wishing to deploy Keycloak on OpenShift.

Bringing up a local OpenShift cluster

To deploy Keycloak on OpenShift, one must first have an OpenShift. OpenShift Online is Red Hat’s public PaaS offering. Although running the demo on a public PaaS was my first choice, OpenShift Online was experiencing issues at the time I was setting up my demo. So I sought a local solution. This approach would have the additional benefit of not being subject to the whims of conference networks (or, it was supposed to – but that is a story for another day!)

oc cluster up

Next I tried oc cluster up. oc is the official OpenShift client program. On Fedora, it is provided by the origin-clients package. oc cluster up command pulls required images and brings up an OpenShift cluster running on the system’s Docker infrastructure. The command takes no further arguments; it really is that simple! Or is it…?

% oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.5.0 image ...
   Pulling image openshift/origin:v1.5.0
   Pulled 0/3 layers, 3% complete
   Pulled 3/3 layers, 100% complete
   Image pull complete
-- Checking Docker daemon configuration ... FAIL
   Error: did not detect an --insecure-registry argument on the Docker daemon

     Ensure that the Docker daemon is running with the following argument:

OK, so it is not that simple. But it got a fair way along, and (kudos to the OpenShift developers) they have provided actionable feedback about how to resolve the issue. I added --insecure-registry to the OPTIONS variable in /etc/sysconfig/docker, then restarted Docker and tried again:

% oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.5.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using as the server IP
-- Starting OpenShift container ...
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
-- Adding default OAuthClient redirect URIs ... OK
-- Installing registry ... OK
-- Installing router ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ... OK
-- Removing temporary directory ... OK
-- Checking container networking ... OK
-- Server Information ... 
   OpenShift server started.
   The server is accessible via web console at:

   You are logged in as:
       User:     developer
       Password: developer

   To login as administrator:
       oc login -u system:admin

Success! Unfortunately, on my machine with several virtual network, oc cluster up messed a bit too much with the routing tables, and when I deployed Keycloak on this cluster it was unable to communicate with my VMs. No doubt these issues could have been solved, but being short on time and with other approaches to try, I abandoned this approach.


Minishift is a tool that launches a single-node OpenShift cluster in a VM. It supports a variety of operating systems and hypervisors. On GNU+Linux it supports KVM and VirtualBox.

First install docker-machine and docker-machine-driver-kvm. (follow the instructions at the preceding links). Unfortunately these are not yet packaged for Fedora.

Download and extract the Minishift release for your OS from https://github.com/minishift/minishift/releases.

Run minishift start:

% ./minishift start
-- Installing default add-ons ... OK
Starting local OpenShift cluster using 'kvm' hypervisor...
Downloading ISO 'https://github.com/minishift/minishift-b2d-iso/releases/download/v1.0.2/minishift-b2d.iso'

... wait a while ...

It downloads a boot2docker VM image containing the openshift cluster, boots the VM, and the console output then resembles the output of oc cluster up. I deduce that oc cluster up is being executed on the VM.

At this point, we’re ready to go. Before I continue, it is important to note that once you have access to an OpenShift cluster, the user experience of creating and managing applications is essentially the same. The commands in the following sections are relevant, regardless whether you are running your app on OpenShift online, on a cluster running on your workstation, or anything in between.

Preparing the Keycloak image

The JBoss project provides official Docker images, including an official Docker image for Keycloak. This image runs fine in plain Docker but the directory permissions are not correct for running in OpenShift.

The Dockerfile for this image is found in the jboss-dockerfiles/keycloak repository on GitHub. Although they do not publish an official image for it, this repository also contains a Dockerfile for Keycloak on OpenShift! I was able to build that image myself and upload it to my Docker Hub account. The steps were as follows.

First clone the jboss-dockerfiles repo:

% git clone https://github.com/jboss-dockerfiles/keycloak docker-keycloak
Cloning into 'docker-keycloak'...
remote: Counting objects: 1132, done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 1132 (delta 14), reused 17 (delta 8), pack-reused 1102
Receiving objects: 100% (1132/1132), 823.50 KiB | 158.00 KiB/s, done.
Resolving deltas: 100% (551/551), done.
Checking connectivity... done.

Next build the Docker image for OpenShift:

% docker build docker-keycloak/server-openshift
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM jboss/keycloak:latest
 ---> fb3fc6a18e16
Step 2 : USER root
 ---> Running in 21b672e19722
 ---> eea91ef53702
Removing intermediate container 21b672e19722
Step 3 : RUN chown -R jboss:0 $JBOSS_HOME/standalone &&     chmod -R g+rw $JBOSS_HOME/standalone
 ---> Running in 93b7d11f89af
 ---> 910dc6c4a961
Removing intermediate container 93b7d11f89af
Step 4 : USER jboss
 ---> Running in 8b8ccba42f2a
 ---> c21eed109d12
Removing intermediate container 8b8ccba42f2a
Successfully built c21eed109d12

Finally, tag the image into the repo and push it:

% docker tag c21eed109d12 registry.hub.docker.com/frasertweedale/keycloak-openshift

% docker login -u frasertweedale registry.hub.docker.com
Login Succeeded

% docker push registry.hub.docker.com/frasertweedale/keycloak-openshift
... wait for upload ...
latest: digest: sha256:c82c3cc8e3edc05cfd1dae044c5687dc7ebd9a51aefb86a4bb1a3ebee16f341c size: 2623

Adding CA trust

For my demo, I used a local FreeIPA installation to issue TLS certificates for the the Keycloak app. I was also going to carry out a scenario where I configure Keycloak to use that FreeIPA installation’s LDAP server to authenticate users. I wanted to use TLS everywhere (eat your own dog food!) I needed the Keycloak application to trust the CA of one of my local FreeIPA installations. This made it necessary to build another Docker image based on the keycloak-openshift image, with the appropriate CA trust built in.

The content of the Dockerfile is:

FROM frasertweedale/keycloak-openshift:latest
USER root
COPY ca.pem /etc/pki/ca-trust/source/anchors/ca.pem
RUN update-ca-trust
USER jboss

The file ca.pem contains the CA certificate to add. It must be in the same directory as the Dockerfile. The build copies the CA certificate to the appropriate location and executes update-ca-trust to ensure that applications – including Java programs – will trust the CA.

Following the docker build I tagged the new image into my hub.docker.com repository (tag: f25-ca) and pushed it. And with that, we are ready to deploy Keycloak on OpenShift.

Creating the Keycloak application in OpenShift

At this point we have a local OpenShift cluster (via Minishift) and a Keycloak image (frasertweedale/keycloak-openshift:f25-ca) to deploy. When deploying the app we need to set some environment variables:


A username for the Keycloak admin account to be created


Passphrase for the admin user


Because the application will be running behind OpenShift’s HTTP proxy, we need to tell Keycloak to use the "external" hostname when creating hyperlinks, rather than Keycloak’s own view.

Use the oc new-app command to create and deploy the application:

% oc new-app --docker-image frasertweedale/keycloak-openshift:f25-ca \
    --env KEYCLOAK_USER=admin \
    --env KEYCLOAK_PASSWORD=secret123 \
--> Found Docker image 45e296f (4 weeks old) from Docker Hub for "frasertweedale/keycloak-openshift:f25-ca"

    * An image stream will be created as "keycloak-openshift:f25-ca" that will track this image
    * This image will be deployed in deployment config "keycloak-openshift"
    * Port 8080/tcp will be load balanced by service "keycloak-openshift"
      * Other containers can access this service through the hostname "keycloak-openshift"

--> Creating resources ...
    imagestream "keycloak-openshift" created
    deploymentconfig "keycloak-openshift" created
    service "keycloak-openshift" created
--> Success
    Run 'oc status' to view your app.

The app gets created immediately but it is not ready yet. The download of the image and deployment of the container (or pod in OpenShift / Kubernetes terminology) will proceed in the background.

After a little while (depending on how long it takes to download the ~300MB Docker image) oc status will show that the deployment is up and running:

% oc status
In project My Project (myproject) on server

svc/keycloak-openshift -
  dc/keycloak-openshift deploys istag/keycloak-openshift:f25-ca 
    deployment #2 deployed 3 minutes ago - 1 pod

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

(In my case, the first deployment failed because the 10-minute timeout elapsed before the image download completed; hence deployment #2 in the output above.)

Creating a secure route

Now the Keycloak application is running, but we cannot reach it from outside the Keycloak project itself. In order to be able to reach it there must be a route. The oc create route command lets us create a route that uses TLS (so clients can authenticate the service). We will use the domain name keycloak.ipa.local. The public/private keypair and certificate have already been generated (how to do that is outside the scope of this article). The certificate was signed by the CA we added to the image earlier. The service name – visible in the oc status output above – is svc/keycloak-openshift.

% oc create route edge \
  --service svc/keycloak-openshift \
  --hostname keycloak.ipa.local \
  --key /home/ftweedal/scratch/keycloak.ipa.local.key \
  --cert /home/ftweedal/scratch/keycloak.ipa.local.pem
route "keycloak-openshift" created

Assuming there is a DNS entry pointing keycloak.ipa.local to the OpenShift cluster, and that the system trusts the CA that issued the certificate, we can now visit our Keycloak application:

% curl https://keycloak.ipa.local/
  ~ Copyright 2016 Red Hat, Inc. and/or its affiliates
  ~ and other contributors as indicated by the @author tags.
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

    <meta http-equiv="refresh" content="0; url=/auth/" />
    <meta name="robots" content="noindex, nofollow">
    <script type="text/javascript">
        window.location.href = "/auth/"
    If you are not redirected automatically, follow this <a href='/auth'>link</a>.

If you visit in a browser, you will be able to log in using the admin account credentials specified in the KEYCLOAK_USER and KEYCLOAK_PASSWORD environment variables specified when the app was created. And from there you can create and manage authentication realms, but that is beyond the scope of this article.


In this post I discussed how to run Keycloak in OpenShift, from bringing up an OpenShift cluster to building the Docker image and creating the application and route in OpenShift. I recounted that I found OpenShift Online unstable at the time I tried it, and that although oc cluster up did successfully bring up a cluster I had trouble getting the Docker and VM networks to talk to each other. Eventually I tried Minishift which worked well.

We saw that although there is no official Docker image for Keycloak in OpenShift, there is a Dockerfile that builds a working image. It is easy to further extend the image to add trust for private CAs.

Creating the Keycloak app in OpenShift, and adding the routes, is straightforward. There are a few important environment variables that must be set. The oc create route command was used to create a secure route to access the application from the outside.

We did not discuss how to set up Keycloak with a database for persisting configuration and user records. The deployment we created is ephemeral. This satisfied my needs for demonstration purposes but production deployments will require persistence. There are official JBoss Docker images that extend the base Keycloak image and add support for PostgreSQL, MySQL and MongoDB. I have not tried these but I’d suggest starting with one of these images if you are looking to do a production deployment. Keep in mind that these images may not include the changes that are required for deploying in OpenShift.

SE Linux for CentOS Part 3

Posted by Adam Young on September 02, 2017 01:02 AM

After the previous two days debugging, Simo Sorce suggested that I need to tell the OS to show all AVCs, some are hidden by default.

The problem is that not all AVCs are reported. We can disable this.

First I needed to install setools:

sudo yum install setools-console

With that I could confirm that there were hidden AVCs:

sudo seinfo --stats | grep audit Audit
allow: 157 Dontaudit: 8036

I disabled the hiding of the AVCs:

sudo semodule --disable_dontaudit --build

And a bunch more AVCs now show up when I deploy a VM.  But…after a couple iterations, its obvious that the same errors keep showing up. Here’s a sample:

type=AVC msg=audit(1504306852.970:3169): avc: denied { search } for pid=27918 comm="cat" name="27821" dev="proc" ino=275493 scontext=system_u:system_r:svirt_tcg_t:s0:c130,c773 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c0,c5 tclass=dir
type=AVC msg=audit(1504306852.984:3173): avc: denied { search } for pid=27929 comm="nsenter" name="27821" dev="proc" ino=275493 scontext=system_u:system_r:svirt_tcg_t:s0:c130,c773 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c0,c5 tclass=dir
type=AVC msg=audit(1504306852.994:3174): avc: denied { search } for pid=27930 comm="pkill" name="498" dev="proc" ino=12227 scontext=system_u:system_r:svirt_tcg_t:s0:c130,c773 tcontext=system_u:system_r:udev_t:s0-s0:c0.c1023 tclass=dir


Looking at the policy generated by audit2allow:

#!!!! This avc is a constraint violation. You would need to modify the attributes of either the source or target types to allow this access.
#Constraint rule: 
# mlsconstrain dir { ioctl read lock search } ((h1 dom h2 -Fail-) or (t1 != { netlabel_peer_t openshift_t openshift_app_t sandbox_min_t sandbox_x_t sandbox_web_t sandbox_net_t svirt_t svirt_tcg_t svirt_lxc_net_t svirt_qemu_net_t svirt_kvm_net_t } -Fail-) ); Constraint DENIED

The line is commented out, which tells me I should not just blindly enable it.  At the bottom of the policy file I see the comment:

# Possible cause is the source user (system_u) and target user (unconfined_u) are different.
# Possible cause is the source role (system_r) and target role (unconfined_r) are different.
# Possible cause is the source level (s0:c130,c773) and target level (s0-s0:c0.c1023) are different.


SE Linux for CentOS Continued

Posted by Adam Young on September 02, 2017 12:52 AM

Trying to troubleshoot the issues from Yesterday’s SELinux errors.

Immediately after a new deploy of the manifests, I want to look at the context on the qemu file:

$ kubectl get pods libvirt-81sdh 
libvirt-81sdh   2/2       Running   0          28s

Now to look at the file:

$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:unlabeled_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

Running restorecon on it, as the audit2allow output suggests:

[ayoung@drifloon kubevirt]$ kubectl  exec libvirt-81sdh -c libvirtd -- restorecon  /usr/local/bin/qemu-system-x86_64
[ayoung@drifloon kubevirt]$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:bin_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

unlabeled_t became bin_t.

Once again, attempt to deploy a vm, and see what AVCs we get:

$ kubectl apply -f cluster/vm-pxe.yaml 
vm "testvm" created
[ayoung@drifloon kubevirt]$ kubectl delete  -f cluster/vm-pxe.yaml 
vm "testvm" deleted

This is what the audit log showed:

type=AVC msg=audit(1504291091.397:2933): avc:  denied  { transition } for  pid=32273 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-18" ino=31526884 scontext=system_u:system_r:spc_t:s0 tcontext=system_u:system_r:svirt_tcg_t:s0:c322,c373 tclass=process

There were several lines like that, but they were identical except for the pid. What does audit2allow show?

#============= spc_t ==============

#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow spc_t svirt_tcg_t:process transition;

Lets see if the additional parameters make a difference:

$ kubectl  exec libvirt-81sdh -c libvirtd -- restorecon  -R -v  /usr/local/bin/qemu-system-x86_64
$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:bin_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

The original lableeing of system_u:object_r:unlabeled_t:s0 is now system_u:object_r:bin_t:s0, which is the same as it was after maiong the restorecon call without the additional parameters.

How about the additional line, the allow? I can apply it outside of the container by using audit2allow:

cat /tmp/audit.txt | audit2allow -a -M virt-policy
sudo semodule -i virt-policy.pp

Upon deploy, a similar error, with a different context:

type=AVC msg=audit(1504294173.446:3734): avc:  denied  { entrypoint } for  pid=6565 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-18" ino=31526884 scontext=system_u:system_r:svirt_tcg_t:s0:c577,c707 tcontext=system_u:object_r:bin_t:s0 tclass=file

Running this through audit2allow generates

#============= svirt_tcg_t ==============

#!!!! WARNING: 'bin_t' is a base type.
allow svirt_tcg_t bin_t:file entrypoint;

While this is a pretty powerful rule, it might be appropriate for what we are doing with virt. Again, lets apply the policy and see what happens.

$ cat virt-policy-2.txt | audit2allow -a -M virt-policy-2
$ sudo semodule -i virt-policy-2.pp

Now a slew of errors, but different ones:

type=AVC msg=audit(1504294406.893:3797): avc:  denied  { write } for  pid=7236 comm="qemu-system-x86" path="pipe:[423417]" dev="pipefs" ino=423417 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:spc_t:s0 tclass=fifo_file
type=AVC msg=audit(1504294406.893:3797): avc:  denied  { write } for  pid=7236 comm="qemu-system-x86" path="pipe:[423417]" dev="pipefs" ino=423417 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:spc_t:s0 tclass=fifo_file
type=AVC msg=audit(1504294406.894:3798): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="ld.so.cache" dev="dm-18" ino=8388771 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1504294406.894:3799): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3800): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3801): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3802): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3803): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3804): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3805): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3806): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.895:3807): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="libtinfo.so.6" dev="dm-18" ino=29360804 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.900:3808): avc:  denied  { sigchld } for  pid=21975 comm="docker-containe" scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:container_runtime_t:s0 tclass=process

This process is iterative, and I had to go through it 10 times until I came up with a complete set of audit2allow generated files. Here is the sum total of what was generated.

module virt-policy-2 1.0;

require {
	type svirt_tcg_t;
	type bin_t;
	class file entrypoint;

#============= svirt_tcg_t ==============

#!!!! WARNING: 'bin_t' is a base type.
allow svirt_tcg_t bin_t:file entrypoint;

module virt-policy-3 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	type spc_t;
	type container_runtime_t;
	class process sigchld;
	class lnk_file read;
	class fifo_file write;
	class file read;

#============= svirt_tcg_t ==============
allow svirt_tcg_t container_runtime_t:process sigchld;
allow svirt_tcg_t spc_t:fifo_file write;

#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file read;
allow svirt_tcg_t unlabeled_t:lnk_file read;

module virt-policy-4 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file open;

#============= svirt_tcg_t ==============

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/ld.so.cache' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/ld.so.cache
allow svirt_tcg_t unlabeled_t:file open;

module virt-policy-5 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file getattr;

#============= svirt_tcg_t ==============

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/ld.so.cache' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/ld.so.cache
allow svirt_tcg_t unlabeled_t:file getattr;

module virt-policy-6 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file execute;

#============= svirt_tcg_t ==============

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/usr/lib64/libtinfo.so.6.0' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/lib64/libtinfo.so.6.0
allow svirt_tcg_t unlabeled_t:file execute;

module virt-policy-7 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	type spc_t;
	class fifo_file { getattr ioctl };
	class file { execute_no_trans write };

#============= svirt_tcg_t ==============
allow svirt_tcg_t spc_t:fifo_file { getattr ioctl };

#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file { execute_no_trans write };

module virt-policy-8 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	type sysfs_t;
	class capability { setgid setuid };
	class file append;
	class filesystem getattr;

#============= svirt_tcg_t ==============
allow svirt_tcg_t self:capability { setgid setuid };
allow svirt_tcg_t sysfs_t:filesystem getattr;

#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file append;

module virt-policy-9 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file ioctl;
	class dir read;

#============= svirt_tcg_t ==============
allow svirt_tcg_t unlabeled_t:dir read;

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/sudoers' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/sudoers
allow svirt_tcg_t unlabeled_t:file ioctl;

module virt-policy 1.0;

require {
	type svirt_tcg_t;
	type spc_t;
	class process transition;

#============= spc_t ==============

#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow spc_t svirt_tcg_t:process transition;

module virt-policy-10 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class capability { net_admin sys_resource };
	class file lock;
	class netlink_audit_socket create;

#============= svirt_tcg_t ==============
allow svirt_tcg_t self:capability { net_admin sys_resource };
allow svirt_tcg_t self:netlink_audit_socket create;

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/run/utmp' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /run/utmp
allow svirt_tcg_t unlabeled_t:file lock;

Obviously, using permissive would have been a shorter process. Let me restart the VM and try that. Here’s what I generate after one iteration:

module kubevirt-policy 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	type container_runtime_t;
	class capability audit_write;
	class unix_stream_socket connectto;
	class file entrypoint;
	class netlink_audit_socket nlmsg_relay;

#============= svirt_tcg_t ==============

#!!!! The file '/run/docker.sock' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /run/docker.sock
allow svirt_tcg_t container_runtime_t:unix_stream_socket connectto;
allow svirt_tcg_t self:capability audit_write;
allow svirt_tcg_t self:netlink_audit_socket nlmsg_relay;

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow svirt_tcg_t unlabeled_t:file entrypoint;

And…. we start into the same pattern. It takes several runs to get to a set of policies that run cleanly. It seems some of the earlier AVCs mask later ones, and running in permissive mode only reports the first of several. Needless to say, the policy for running a VM Via Kubevirt is going to require some scrutiny.

And even then, the VMs still fail to deploy. Disable SELinux and they run. This mystery continues.

SELinux for Kubevirt on Centos

Posted by Adam Young on August 31, 2017 04:51 PM

Without disabling SELinux enforcement, an attempt to deploy a VM generates the following audit message:

type=AVC msg=audit(1504194626.938:877): avc: denied { transition } for pid=9574 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-19" ino=31526884 scontext=system_u:system_r:spc_t:s0 tcontext=system_u:system_r:svirt_tcg_t:s0:c408,c741 tclass=process

Running this through audit2allow provides a little more visibility into the problem:

#============= spc_t ==============

#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow spc_t svirt_tcg_t:process transition;

This is probably due to running as much of the virtualization machinery in containers. /usr/local/bin/qemu-system-x86_64 comes from inside the libvirt container. It does not exist on the base OS filesystem. Thus, just running restorecon won’t do much.

While it is tempting to make this change and hope that everything works, I’ve learned that SELinux is dilligent enough that, if one thing fails, it is usually a few related things that are actually failing, and just dealing with the first is not enough. settting SELinux into permissive mode ande deploying a VM produces a much longer list of avcs. Runnning these through audit2allow generates the following:

#============= spc_t ==============

#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow spc_t svirt_tcg_t:process transition;

#============= svirt_lxc_net_t ==============
allow svirt_lxc_net_t svirt_tcg_t:dir { getattr open read search };
allow svirt_lxc_net_t svirt_tcg_t:file { getattr open read };

#============= svirt_tcg_t ==============
allow svirt_tcg_t cgroup_t:file { getattr open write };

#!!!! The file '/dev/ptmx' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /dev/ptmx
allow svirt_tcg_t container_devpts_t:chr_file open;

#!!!! The file '/run/docker.sock' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /run/docker.sock
allow svirt_tcg_t container_runtime_t:unix_stream_socket connectto;
allow svirt_tcg_t self:capability { audit_write dac_override net_admin setgid setuid sys_ptrace sys_resource };
allow svirt_tcg_t self:netlink_audit_socket { create nlmsg_relay };
allow svirt_tcg_t spc_t:fifo_file { getattr ioctl write };
allow svirt_tcg_t svirt_lxc_net_t:dir search;
allow svirt_tcg_t svirt_lxc_net_t:file { getattr open read };
allow svirt_tcg_t svirt_lxc_net_t:lnk_file read;
allow svirt_tcg_t sysfs_t:filesystem getattr;

#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:dir { add_name mounton read remove_name write };

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow svirt_tcg_t unlabeled_t:file { append create entrypoint execute execute_no_trans getattr ioctl lock open read unlink write };
allow svirt_tcg_t unlabeled_t:lnk_file read;

While audit2allow will get you going again, it often produces a policy that is too permissive. Before blindly accepting this new policy, we need to look through it and figure out if there are better policy rules that acheive the same ends.

Many of the rules look sane on a first glance. For example,

allow svirt_tcg_t container_runtime_t:unix_stream_socket connectto;

allowing a file with the label svirt_tcg_t and the process runtime label of container_runtime to talk to a unix domain socket is obviously required to perform libvirt based tasks.

I suspect that the right solution involves modifying the libvirt container definition to properly set the SELinux context internally, as well as possibly adding something to the manifest for the daemonset to honor/carry forward the SELinux values.

Security ROI isn't impossible, we suck at measuring

Posted by Josh Bressers on August 30, 2017 04:11 PM
As of late I've been seeing a lot of grumbling that security return on investment (ROI) is impossible. This is of course nonsense. Understanding your ROI is one of the most important things you can do as a business leader. You have to understand if what you're doing makes sense. By the very nature of business, some of the things we do have more value than other things. Some things even have negative value. If we don't know which things are the most important, we're just doing voodoo security.

H. James Harrington once said
Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.
Anyone paying attention to the current state of security will probably shed a tear over that statement. The foundation of the statement results in this truth: we can't control or improve the state of security today. As much as we all like to talk about what's wrong with security and how to fix it. The reality is we don't really know what's broken, which of course means we have no idea how to fix anything.

Measuring security isn't impossible, it's just really hard today. It's really hard because we don't really understand what security is in most instances. Security isn't one thing, it's a lot of little things that don't really have anything to do with each other but we clump them together for some reason. We like to build teams of specialized people and call them the security team. We pretend we're responsible for a lot of little unrelated activities but we often don't have any real accountability. The reality is this isn't a great way to do something that actually works, it's a great way to have a lot smart people fail to live up to their true potential. The best security teams in the world today aren't good at security, they're just really good at marketing themselves so everyone thinks they're good at security.

Security needs to be a part of everything, not a special team that doesn't understand what's happening outside their walls. Think for a minute what an organization would look like if we split groups up by what programming language they knew. Now you have all the python people in one corner and all the C guys in the other corner. They'll of course have a long list of reasons why they're smarter and better than the other group (we'll ignore the perl guys down in the basement). Now if there is a project that needs some C and some python they would have to go to each group and get help. Bless the soul of anyone who needs C and python working together in their project. You know this would just be a massive insane turf war with no winner. It's quite likely the project would never work because the groups wouldn't have a huge incentive to work together. I imagine you can see the problem here. You have two groups that need to work together without proper incentive to actually work together.

Security is a lot like this. Does having a special secure development group outside of the development group make sense? Why does it make sense to have a security operations group that isn't just part of IT? If you're not part of a group do you have an incentive for the group to succeed? If I can make development's life so difficult they can't possibly succeed that's development's problem, not yours. You have no incentive to be a reasonable member of the team. The reality is you're not a member of the team at all. Your incentive is to protect your own turf, not help anyone else.

I'm going to pick on Google's Project Zero for a minute here. Not because they're broken, but because they're really really good at what they do. Project zero does research into how to break things, then they work with the project they broke to make it better. If this was part of a more traditional security thinking group, Project Zero would do research, build patches, then demand everyone uses whatever it is they built and throw a tantrum if they don't. This would of course be crazy, unwelcome, and a waste of time. Project Zero has a razor focus on research. More importantly though they work with other groups when it's time to get the final work done. Their razor focus and ability to work with others gives them a pretty clear metric they can see. How many flaws did they find? How many got fixed? How many new attack vectors did they create? This is easy to measure. Of course some groups won't work with them, but in that case they can publish their advisories and move on. There's no value in picking long horrible fights.

So here's the question you have to ask yourself. How much of what you do directly affects the group you're a part of? I don't mean things like enforcing compliance, compliance is a cost like paying for electricity, think bigger here about things that generate revenue. If you're doing a project with development, do your decisions affect them or do they affect you? If your decisions affect development you probably can't measure what you do. You can really only measure things that affect you directly. Even if you think you can measure someone else, you'll never be as good as they are. And honestly, who cares what someone else is doing, measure yourself first.

It's pretty clear we don't actually understand what we like to call "security" because we have no idea how to measure it. If we did understand it, we could measure it. According to H. James Harrington we can't fix what we can't measure it. I think given everything we've seen over the past few years, this is quite accurate. We will never fix our security problems without first measuring our security ROI.

I'll spend some time in the next few posts discussing how to measure what we do with actual examples. It's not as hard as it sounds.

Episode 60 - The official blockchain episode

Posted by Open Source Security Podcast on August 30, 2017 01:30 PM
Josh and Kurt talk about the eclipse and blockchain.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5690282/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

Helicopter security

Posted by Josh Bressers on August 28, 2017 02:39 PM
After my last post about security spending, I was thinking about how most security teams integrate into the overall business (hint: they don't). As part of this thought experiment I decided to compare traditional security to something that in modern times has come to be called helicopter parenting.

A helicopter parent is someone who won't let their kids do anything on their own. These are the people you hear about who follow their child to college, to sports practice. They yell at teachers and coaches for not respecting how special the child is. The kids are never allowed to take any risks because risk is dangerous and bad. If they climb the tree, while it could be a life altering experience, they could also fall and get hurt. Skateboarding is possibly the most dangerous thing anyone could ever do! We better make sure nothing bad can ever happen.

It's pretty well understood now that this sort of attitude is terrible for the children. They must learn to do things on their own, it's part of the development process. Taking risks and failing is an extremely useful exercise. It's not something we think about often, but you have to learn to fail. Failure is hard to learn. The children of helicopter parents do manage to learn one lesson they can use in their life, they learn to hide what they do from their parents. They get extremely good at finding way to get around all their rules and restrictions. To a degree we all had this problem growing up. At some point we all wanted to do something our parents didn't approve of, which generally meant we did it anyway, we just didn't tell our parents. Now imagine a universe where your parents let you do NOTHING, you're going to be hiding literally everything. Nobody throughout history has ever accepted the fact that they can do nothing, they just make sure the authoritarian doesn't know about it. Getting caught is still better than doing nothing much of the time.

This brings us to traditional security. Most security teams don't try to work with the business counterparts. Security teams often think they can just tell everyone else what to do. Have you ever heard the security team ask "what are you trying to do?" Of course not. They always just say "don't do that" or maybe "do it this way" then move on to tell the next group how to do their job. They don't try to understand what you're doing and why you are doing it. It's quite literally not their job to care what you're doing, which is part of the problem. Things like phishing tests are used to belittle, not teach (they have no value as teaching tools, but we won't discuss that today). Many of the old school security teams see their job as risk aversion, not risk management. They are helicopter security teams.

Now as we know from children, if you prevent someone from doing anything they don't become your obedient servant, they go out of their way to make sure the authority has no idea what's going on. This is basically how shadow IT became a thing. It was far easier to go around the rules than work with the existing machine. Helicopter security is worse than nothing. At least with nothing you can figure out what's going on by asking questions and getting honest answers. In a helicopter security environment information is actively hidden because truth will only get you in trouble.

Can we fix this?
I don't know the answer to this question. A lot of tech people I see (not just security) are soldiers from the last war. With the way we see cloud transforming the universe there are a lot of people who are still stuck in the past. We often hear it's hard to learn new things but it's more than that. Technology, especially security, never stands still. It used to move slow enough you could get by for a few years on old skills, but we're in the middle of disruptive change right now. If you're not constantly questioning your existing skills and way of thinking you're already behind. Some people are so far behind they will never catch up. It's human nature to double down on the status quo when you're not part of the change. Helicopter security is that doubling down.

It's far easier to fight change and hope your old skills will remain useful than it is to learn a new skill. Everything we see in IT today is basically a new skill. Today the most useful thing you can know is how to learn quickly, what you learned a few months ago could be useless today, it will probably be useless in the near future. We are actively fighting change like this in security today. We try to lump everything together and pretend we have some sort of control over it. We never really had any control, it's just a lot more obvious now than it was before. Helicopter security doesn't work, no matter how bad you want it to.

The Next Step
The single biggest thing we need to start doing is measure ourselves. Even if you don't want to learn anything new you can at least try to understand what we're doing today that actually works, which things sort of work, and of course the things that don't work at all. In the next few posts I'm going to discuss how to measure security as well as how to avoid voodoo security. It's a lot harder to justify helicopter security behavior once we understand which of our actions work and which don't.

Deploying Kubevirt on Origin Master

Posted by Adam Young on August 25, 2017 03:14 PM

Now that I have a functional OpenShift Origin built from source, I need to deploy KubeVirt on top of it.

Here are my notes. This is rough, and not production quality yet, but should get you started.


As I said in that last post, in order to Build Kubevirt, I had to upgrade to a later version of go (rpm had 1.6, now I have 1.8).


In order to build the manifests with specific versions, I over-rode some config options as I described here.

Specifically, I used docker_tag=devel to make sure I didn’t accidentally download the released versions from Docker hub, as well as set the master_ip address.

To generate the docker images:

make docker

make manifests


Config Changes

In order to make the configuration changes, and have them stick:

oc cluster down

sudo vi /var/lib/origin/openshift.local.config/master/master-config.yaml



externalIPNetworkCIDRs: [""]

The bring the cluster up with:

oc cluster up --use-existing-config --loglevel=5 --version=413eb73 --host-data-dir=/var/lib/origin/etcd/ | tee /tmp/oc.log 2&gt;&amp;1


Got an error showing
$ Ensure that access to ports tcp/8443, udp/53 and udp/8053 is allowed on

[ayoung@drifloon origin]$ sudo firewall-cmd --zone=public --add-port=8443/tcp
 [sudo] password for ayoung:
 [ayoung@drifloon origin]$ sudo firewall-cmd --zone=public --add-port=8053/udp
 [ayoung@drifloon origin]$ sudo firewall-cmd --zone=public --add-port=53/udp

Those won’t persists as is, so:

[ayoung@drifloon origin]$ sudo firewall-cmd --permanent --zone=public --add-port=8443/tcp
 [ayoung@drifloon origin]$ sudo firewall-cmd --permanent --zone=public --add-port=53/udp
 [ayoung@drifloon origin]$ sudo firewall-cmd --permanent --zone=public --add-port=8053/udp

Log in as admin

Redeploy the cluster and then

$ oc login -u system:admin
 Logged into "" as "system:admin" using existing credentials.

You have access to the following projects and can switch between them with 'oc project ':

 * myproject

Using project "myproject".
 [ayoung@drifloon kubevirt]$ oc project kube-system
 Now using project "kube-system" on server "".

Deploying Manifests

Need Updated Manifests from this branch.

To generate alternate values for the manifest:

$ for MANIFEST in `ls manifests/*yaml` ; do kubectl apply -f $MANIFEST ; done

message: ‘No nodes are available that match all of the following predicates::
MatchNodeSelector (1).’
reason: Unschedulable

Head node is not schedulable.

oc adm manage-node localhost --schedulable=true

And now….

for MANIFEST in `ls manifests/*yaml` ; do kubectl delete -f $MANIFEST ; done

wait a bit

for MANIFEST in `ls manifests/*yaml` ; do kubectl apply -f $MANIFEST ; done
$ kubectl get pods
 haproxy-858199412-m78n5 0/1 CrashLoopBackOff 8 16m
 kubevirt-cockpit-demo-4250553349-gm8qm 0/1 CrashLoopBackOff 224 18h
 spice-proxy-1193136539-gr7b3 0/1 CrashLoopBackOff 225 18h
 virt-api-4068750737-j7bwj 1/1 Running 0 18h
 virt-controller-3722000252-bsbsr 1/1 Running 0 18h

why are those three crashing? Permissions

$ kubectl logs kubevirt-cockpit-demo-4250553349-gm8qm
 cockpit-ws: /etc/cockpit/ws-certs.d/0-self-signed.cert: Failed to open file '/etc/cockpit/ws-certs.d/0-self-signed.cert': Permission denied
 kubectl logs haproxy-858199412-hqmk1
 <7>haproxy-systemd-wrapper: executing /usr/local/sbin/haproxy -p /haproxy/run/haproxy.pid -f /usr/local/etc/haproxy/haproxy.cfg -Ds
 [ALERT] 228/180509 (15) : [/usr/local/sbin/haproxy.main()] Cannot create pidfile /haproxy/run/haproxy.pid
 [ayoung@drifloon kubevirt]$ kubectl logs spice-proxy-1193136539-gr7b3
 FATAL: Unable to open configuration file: /home/proxy/squid.conf: (13) Permission denied

virt-api runs as a strange user:

1000050+ 14464 14450 0 Aug16 ? 00:00:01 /virt-api –port 8183 –spice-proxy
1000050+ is, I am guessing, a uid made up by kubernetes

looks like I am tripping over the fact that Openshift security policy by default prohibits you from running as known users. (thanks claytonc)

Pull Request is merged for changing these in KubeVirt

Service Accounts:

oc create serviceaccount -n kube-system kubevirt
oc adm policy add-scc-to-user privileged -n kube-system -z kubevirt

modify the libvirt and virt-handler manifests like so (this is in the version from the branch above):

+       serviceAccountName: kubevirt
       - name: virt-handler


+       serviceAccountName: kubevirt
       hostNetwork: true
       hostPID: true
       hostIPC: true
         runAsUser: 0

OK a few more notes: Need manifest changes so that the various resources end up in the kube-system namespace as well as run as the appropriate kubevirt or kubevirt-admin users. See this pull request.


Have to manually apply the permissions.yaml file, then add scc to get the daemon pods to schedule:

oc adm policy add-scc-to-user privileged -n kube-system -z kubevirt
oc adm policy add-scc-to-user privileged -n kube-system -z kubevirt-admin


You could also run as default serviceAccountuser and just run:

oc  adm policy add-scc-to-user privileged -n kube-system -z default

But that is not a good long term strategy.


In order to launch a VM, it turns out we need an eth1:  The default network setup by the libvirt image assumes it is there.  The easiest way to get one is to modify the VM to use a second network card.  That requires restarting the cluster.  You can also set the name of the interface  in config-local.sh to the appropriate network device/connection for your system using the primary_nic value.

Disable SELinux

Try to deploy vm with

kubectl apply -f cluster/vm.yaml

Check the status using:

 kubectl get vms testvm -o yaml

VM failed to deploy.

Check libvirt Container log

oc logs libvirt-ztt0w -c libvirtd 
2017-08-25 13:03:07.253+0000: 5155: error : qemuProcessReportLogError:1845 : internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/local/bin/qemu-system-x86_64: Permission denied

For now, disable SELinux. In the future, we’ll need a customer SELinux policy to allow this.

 sudo setenforce 0



Finally, the iscsi pod defined in the manifests trips over a bunch of OpenShift permissions hardening issues.  Prior to working those out, I just wanted to run a PXE bootable VM, so I copied the vm.yaml to vm-pxe.yaml and applied that.

Lesson Learned: SecurityContextConstraints can’t be in manifests.

Using the FOR loop for the manifests won’t work long term. We’ll need to apply the permissions.yaml file first, then run the two oc commands to add the Users to the scc, and finally run the rest of the manifests. Adding users to sccs cannot be done via manifest apply, as it has to modify an existing resource. The ServiceAccounts need to be created prior to any of the daemonset or deployment manifests and added to the sccs, or the node selection criteria will not be met, and no pods will be scheduled.



Running OpenShift Origin built from source

Posted by Adam Young on August 23, 2017 02:52 PM

Kubernetes is moving from Third Party Resources to the Aggregated API Server.  In order to work with this and continue to deploy on OpenShift Origin, we need to move from working with the shipped and stable version that is in Fedora 26 to the development version in git.  Here are my notes to get it up and running.


It took a couple tries to realize that the go build process needs a fairly bulky virtual machine. I ended up using one that has 8 GB Ram and 50 GB disk. In order to minimize churn, I also went with a Centos7 deployment.

Once I had a running VM here are the configuration changes I had to make.

Use nmcli c eth0 on and set ONBOOT. This can also be done via editing the network config files or using older tools.

yum update -y
yum groupinstall "Development Tools"
yum instyall -y origin-clients

Ensure that the Docker daemon is running with the following argument:
–insecure-registry  By editing the file /etc/sysconfig/docker.  My Options line looks like this:

OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false --insecure-registry'

Followed dirs from here In order to set up the development environment.

cd $GOPATH/src/github.com/openshift/origin
hack/env hack/build-base-images.sh
OS_BUILD_ENV_PRESERVE=_output/local/releases hack/env make release

Note that the KubeVirt code I want to run on top of this requires a later version of go, and thus I upgrade to go version go1.8.3 linux/amd64  via the tarball install method.

The Hash that gets generated by the build depends on when you run.   To see the images run

docker images

I expand the terminal to full screen lots of columns of data.  Here is a subset:

openshift/hello-openshift 413eb73 911092241b5a 35 hours ago 5.84 MB
openshift/hello-openshift latest 911092241b5a 35 hours ago 5.84 MB
openshift/openvswitch 413eb73 c53aae019d81 35 hours ago 1.241 GB
openshift/openvswitch latest c53aae019d81 35 hours ago 1.241 GB
openshift/node 413eb73 af6135fc50c9 35 hours ago 1.239 GB
openshift/node latest af6135fc50c9 35 hours ago 1.239 GB

The tag is the second column.  This is what I use in order to install. I don’t use “Latest” as that changes over time, and it might accidentally succeed using a remote package when the local build failed.

I want to be able to edit configuration values.  A also want the etcd store to persist across reboots.  Thus,

sudo mkdir /var/lib/origin/etcd
sudo chown ayoung:ayoung /var/lib/origin/etcd

And then my final command line to bring up the cluster is:

oc cluster up --use-existing-config --loglevel=5 --version=413eb73 --host-data-dir=/var/lib/origin/etcd/ | tee /tmp/oc.log 2>&1


Below are some of my Troubleshooting notes. I am going to leave them in here so they show up on future searches for people that have the same problems.  They are rough, and you don’t need to read them.

hack/env make release errored

[WARNING] Copying _output/local/releases from the container failed!
[WARNING] Error response from daemon: lstat /var/lib/docker/devicemapper/mnt/fb199307b2f95649066c42f55e5487c66eb3421e5407c8bd6d2f0a7058bc8cd5/rootfs/go/src/github.com/openshift/origin/_output/local/releases: no such file or directory

Tried with OS_BUILD_ENV_PRESERVE=_output/local but no difference.

Should have been
OS_BUILD_ENV_PRESERVE=_output/local/releases hack/env make release

This did not work (basename error)
export PATH=”${PATH}:$( source hack/lib/init.sh; echo “${OS_OUTPUT_BINPATH}/$( os::util::host_platform )/” )”

But I was able to do
export PATH=$PATH:$PWD/_output/local/bin/linux/amd64/

and then

oc cluster up –version=latest

failed due to docker error
— Checking Docker daemon configuration … FAIL
Error: did not detect an –insecure-registry argument on the Docker daemon

used https://wiki.centos.org/SpecialInterestGroup/PaaS/OpenShift-Quickstart

To fix

oc seems to be running OK now. But not using my git commit

Told to run:


oc cluster up –version=8d96d48

GSoC 2017 - Mentor Report from 389 Project

Posted by William Brown on August 23, 2017 02:00 PM

GSoC 2017 - Mentor Report from 389 Project

This year I have had the pleasure of being a mentor for the Google Summer of Code program, as part of the Fedora Project organisation. I was representing the 389 Directory Server Project and offered students the oppurtunity to work on our command line tools written in python.


From the start we have a large number of really talented students apply to the project. This was one of the hardest parts of the process was to choose a student, given that I wanted to mentor all of them. Sadly I only have so many hours in the day, so we chose Ilias, a student from Greece. What really stood out was his interest in learning about the project, and his desire to really be part of the community after the project concluded.

The project

The project was very deliberately “loose” in it’s specification. Rather than giving Ilias a fixed goal of you will implement X, Y and Z, I chose to set a “broad and vague” task. Initially I asked him to investigate a single area of the code (the MemberOf plugin). As he investigated this, he started to learn more about the server, ask questions, and open doors for himself to the next tasks of the project. As these smaller questions and self discoveries stacked up, I found myself watching Ilias start to become a really complete developer, who could be called a true part of our community.

Ilias’ work was exceptional, and he has documented it in his final report here .

Since his work is complete, he is now free to work on any task that takes his interest, and he has picked a good one! He has now started to dive deep into the server internals, looking at part of our backend internals and how we dump databases from id2entry to various output formats.

What next?

I will be participating next year - Sadly, I think the python project oppurtunities may be more limited as we have to finish many of these tasks to release our new CLI toolset. This is almost a shame as the python components are a great place to start as they ease a new contributor into the broader concepts of LDAP and the project structure as a whole.

Next year I really want to give this oppurtunity to an under-represented group in tech (female, poc, etc). I personally have been really inspired by Noriko and I hope to have the oppurtunity to pass on her lessons to another aspiring student. We need more engineers like her in the world, and I want to help create that future.

Advice for future mentors

Mentoring is not for everyone. It’s not a task which you can just send a couple of emails and be done every day.

Mentoring is a process that requires engagement with the student, and communication and the relationship is key to this. What worked well was meeting early in the project, and working out what community worked best for us. We found that email questions and responses worked (given we are on nearly opposite sides of the Earth) worked well, along with irc conversations to help fix up any other questions. It would not be uncommon for me to spend at least 1 or 2 hours a day working through emails from Ilias and discussions on IRC.

A really important aspect of this communication is how you do it. You have to balance positive communication and encouragement, along with critcism that is constructive and helpful. Empathy is a super important part of this equation.

My number one piece of advice would be that you need to create an environment where questions are encouraged and welcome. You can never be dismissive of questions. If ever you dismiss a question as “silly” or “dumb”, you will hinder a student from wanting to ask more questions. If you can’t answer the question immediately, send a response saying “hey I know this is important, but I’m really busy, I’ll answer you as soon as I can”.

Over time you can use these questions to help teach lessons for the student to make their own discoveries. For example, when Ilias would ask how something worked, I would send my response structured in the way I approached the problem. I would send back links to code, my thoughts, and how I arrived at the conclusion. This not only answered the question but gave a subtle lesson in how to research our codebase to arrive at your own solutions. After a few of these emails, I’m sure that Ilias has now become self sufficent in his research of the code base.

Another valuable skill is that overtime you can help to build confidence through these questions. To start with Ilias would ask “how to implement” something, and I would answer. Over time, he would start to provide ideas on how to implement a solution, and I would say “X is the right one”. As time went on I started to answer his question with “What do you think is the right solution and why?”. These exchanges and justifications have (I hope) helped him to become more confident in his ideas, the presentation of them, and justification of his solutions. It’s led to this excellent exchange on our mailing lists, where Ilias is discussing the solutions to a problem with the broader community, and working to a really great answer.

Final thoughts

This has been a great experience for myself and Ilias, and I really look forward to helping another student next year. I’m sure that Ilias will go on to do great things, and I’m happy to have been part of his journey.

Spend until you're secure

Posted by Josh Bressers on August 23, 2017 12:20 PM
I was watching a few Twitter conversations about purchasing security last week and had yet another conversation about security ROI. This has me thinking about what we spend money on. In many industries we can spend our way out of problems, not all problems, but a lot of problems. With security if I gave you a blank check and said "fix it", you couldn't. Our problem isn't money, it's more fundamental than that.

Spend it like you got it
First let's think about how some problems can be solved with money. If you need more electricity capacity, or more help during a busy time, or more computing power, it's really easy to add capacity. You need more compute power, you can either buy more computers or just spend $2.15 in the cloud. If you need to dig a big hole, for a publicity stunt on Black Friday, you just pay someone to dig a big hole. It's not that hard.

This doesn't always work though, if you're building a new website, you probably can't buy your way to success. If a project like this falls behind it can be very difficult to catch back up. You can however track progress which I would say is at least a reasonable alternative. You can move development to another group or hire a new consultant if the old one isn't living up to expectations.

More Security
What if we need "more" security. How can we buy our way into more security for our organization? I'd start by asking the question can we show any actual value for our current security investment? If you stopped spending money on security tomorrow do you know what the results would be? If you stopped buying toilet paper for your company tomorrow you can probably understand what will happen (if you have a good facilities department I bet they already know the answer to this).

This is a huge problem in many organizations. If you don't know what would happen if you lowered or increased your security spending you're basically doing voodoo security. You can imagine many projects and processes as having a series of inputs that can be adjusted. Things like money, time, people, computers, the list could go on. You can control these variables and have direct outcomes on the project. More people could mean you can spend less money on contractors, more computers could mean less time spent on rendering or compiling. Ideally you have a way to find the optimal levels for each of these variables resulting in not only a high return on investment, but also happier workers as they can see the results of their efforts.

We can't do this with security today because security is too broad. We often don't know what would happen if we add more staff, or more technology.

Fundamental fundamentals
So this brings us to why we can't spend our way to security. I would argue there are two real problems here. The first being "security" isn't a thing. We pretend security is an industry that means something but it's really a lot of smaller things we've clumped together in such a way that ensures we can only fail. I see security teams claim to own anything that has the word security attached to it. They claim ownership of projects and ideas, but then they don't actually take any actions because they're too busy or lack the skills to do the work. Just because you know how to do secure development doesn't automatically make you an expert at network security. If you're great at network security it doesn't mean you know anything about physical security. Security is a lot of little things, we have to start to understand what those are and how to push responsibility to respective groups. Having a special application security team that's not part of development doesn't work. You need all development teams doing things securely.

The second problem is we don't measure what we do. How many security teams tell IT they have to follow a giant list of security rules, but they have no idea what would happen if one or more of those rules were rolled back? Remember when everyone insisted we needed to use complex passwords? Now that's considered bad advice and we shouldn't make people change their passwords often. It's also a bad idea to insist they use a variety of special characters now. How many millions have been wasted on stupid password rules? The fact that we changed the rules without any fanfare means there was no actual science behind the rules in the first place. If we even tried to measure this I suspect we would have known YEARS ago that it was a terrible idea. Instead we just kept doing voodoo security. How many more of our rules do you think will end up being rolled back in the near future because they don't actually make sense?

If you're in charge of a security program the first bit of advice I'd give out is to look at everything you own and get rid of whatever you can. Your job isn't to do everything, figure out what you have to do, then do it well. One project well done is far better than 12 half finished. The next thing you need to do is figure out how much whatever you do costs, and how much benefit it creates. If you can't figure out the benefit, you can probably stop doing it today. If it costs more than it saves, you can stop that too. We must have a razor focus if we're to understand what our real problems are. Once we understand the problems we can start to solve them.

Customizing the KubeVirt Manifests

Posted by Adam Young on August 21, 2017 05:02 PM

My cloud may not look like your cloud. The contract between the application deployment and the Kubernetes installation is a set of manifest files that guide Kubernetes in selecting, naming, and exposing resources. In order to make the generation of the Manifests sane in KubeVirt, we’ve provided a little bit of build system support.

The manifest files are templatized in a jinja style. I say style, because the actual template string replacement is done using simmple bash scripting. Regardless of the mechanism, it should not be hard for a developer to understand what happens. I’ll assume that you have your source code checked out in $GOPATH/src/kubevirt.io/kubevirt/

The template files exist in the manifests subdirectory. Mine looks like this:

haproxy.yaml.in             squid.yaml.in            virt-manifest.yaml.in
iscsi-demo-target.yaml.in   virt-api.yaml.in         vm-resource.yaml.in
libvirt.yaml.in             virt-controller.yaml.in
migration-resource.yaml.in  virt-handler.yaml.in

The simplest way to generate a set of actual manifest files is to run make manifests

make manifests
$ ls -l manifests/*yaml
-rw-rw-r--. 1 ayoung ayoung  672 Aug 21 10:17 manifests/haproxy.yaml
-rw-rw-r--. 1 ayoung ayoung 2384 Aug 21 10:17 manifests/iscsi-demo-target.yaml
-rw-rw-r--. 1 ayoung ayoung 1707 Aug 21 10:17 manifests/libvirt.yaml
-rw-rw-r--. 1 ayoung ayoung  256 Aug 21 10:17 manifests/migration-resource.yaml
-rw-rw-r--. 1 ayoung ayoung  709 Aug 21 10:17 manifests/squid.yaml
-rw-rw-r--. 1 ayoung ayoung  832 Aug 21 10:17 manifests/virt-api.yaml
-rw-rw-r--. 1 ayoung ayoung  987 Aug 21 10:17 manifests/virt-controller.yaml
-rw-rw-r--. 1 ayoung ayoung  954 Aug 21 10:17 manifests/virt-handler.yaml
-rw-rw-r--. 1 ayoung ayoung 1650 Aug 21 10:17 manifests/virt-manifest.yaml
-rw-rw-r--. 1 ayoung ayoung  228 Aug 21 10:17 manifests/vm-resource.yaml

Looking at the difference between, say the virt-api template and final yaml file:

$ diff -u manifests/virt-api.yaml.in manifests/virt-api.yaml
--- manifests/virt-api.yaml.in	2017-07-20 13:29:00.532916101 -0400
+++ manifests/virt-api.yaml	2017-08-21 10:17:10.533038861 -0400
@@ -7,7 +7,7 @@
     - port: 8183
       targetPort: virt-api
   externalIPs :
-    - "{{ master_ip }}"
+    - ""
     app: virt-api
@@ -23,14 +23,14 @@
       - name: virt-api
-        image: {{ docker_prefix }}/virt-api:{{ docker_tag }}
+        image: kubevirt/virt-api:latest
         imagePullPolicy: IfNotPresent
             - "/virt-api"
             - "--port"
             - "8183"
             - "--spice-proxy"
-            - "{{ master_ip }}:3128"
+            - ""
           - containerPort: 8183
             name: "virt-api"
@@ -38,4 +38,4 @@
         runAsNonRoot: true
-        kubernetes.io/hostname: {{ primary_node_name }}
+        kubernetes.io/hostname: master

make manifests, it turns out, just calls a bash script ./hack/build-manifests.sh. This script uses two files to determine the values to use for template string substitution. First, the defaults: hack/config-default.sh. This is where master_ip get the value of This file also gives priority to the $DOCKER_TAG environment variable. However, if you need to customize values further, you can create and manage them in the file hack/config-local.sh. The goal is that any of the keys from the -default file that are specified in the hack/config-local.sh will use the value from the latter file. The set of keys with their defaults (as of this writing) that you can customize are:

binaries="cmd/virt-controller cmd/virt-launcher cmd/virt-handler cmd/virt-api cmd/virtctl cmd/virt-manifest"
docker_images="cmd/virt-controller cmd/virt-launcher cmd/virt-handler cmd/virt-api cmd/virt-manifest images/haproxy images/iscsi-demo-target-tgtd images/vm-killer images/libvirt-kubevirt images/spice-proxy cmd/virt-migrator cmd/registry-disk-v1alpha images/cirros-registry-disk-demo"
optional_docker_images="cmd/registry-disk-v1alpha images/fedora-atomic-registry-disk-demo"
manifest_templates="`ls manifests/*.in`"

Not all of these are for Manifest files. The docker_images key is used in selecting the set of generating Docker images to generate in a command called from a different section of the Makefile. The network_provider is used in the Vagrant setup, and so on.However, most of the values are used in the manifest files. So, If I want to set a master IP Address of, I would have a hack/config-local.sh file that looks like this:

$  diff -u manifests/virt-api.yaml.in manifests/virt-api.yaml
--- manifests/virt-api.yaml.in	2017-07-20 13:29:00.532916101 -0400
+++ manifests/virt-api.yaml	2017-08-21 10:42:28.434742371 -0400
@@ -7,7 +7,7 @@
     - port: 8183
       targetPort: virt-api
   externalIPs :
-    - "{{ master_ip }}"
+    - ""
     app: virt-api
@@ -23,14 +23,14 @@
       - name: virt-api
-        image: {{ docker_prefix }}/virt-api:{{ docker_tag }}
+        image: kubevirt/virt-api:latest
         imagePullPolicy: IfNotPresent
             - "/virt-api"
             - "--port"
             - "8183"
             - "--spice-proxy"
-            - "{{ master_ip }}:3128"
+            - ""
           - containerPort: 8183
             name: "virt-api"
@@ -38,4 +38,4 @@
         runAsNonRoot: true
-        kubernetes.io/hostname: {{ primary_node_name }}
+        kubernetes.io/hostname: master

IoT Security for Developers [Survive IoT Part 5]

Posted by Russel Doty on August 15, 2017 10:33 PM

Previous articles focused on how to securely design and configure a system based on existing hardware, software, IoT Devices, and networks. If you are developing IoT devices, software, and systems, there is a lot more you can do to develop secure systems.

The first thing is to manage and secure communications with IoT Devices. Your software needs to be able to discover, configure, manage and communicate with IoT devices. By considering security implications when designing and implementing these functions you can make the system much more robust. The basic guideline is don’t trust any device. Have checks to verify that a device is what it claims to be, to verify device integrity, and to validate communications with the devices.

Have a special process for discovering and registering devices and restrict access to it. Do not automatically detect and register any device that pops up on the network! Have a mechanism for pairing devices with the gateway, such as a special pairing mode that must be invoked on both the device and the gateway to pair or a requirement to manually enter a device serial number or address into the gateway as part of the registration process. For industrial applications adding devices is a deliberate process – this is not a good operation to fully automate!

A solid approach to gateway and device identity is to have a certificate provisioned onto the device at the factory, by the system integrator, or at a central facility. It is even better if this certificate is backed by a HW root of trust that can’t be copied or spoofed.

Communications between the gateway and the device should be designed. Instead of a general network connection, which can be used for many purposes, consider using a specialized interface. Messaging interfaces are ideal for many IoT applications. Two of the most popular messaging interfaces are MQTT (Message Queued Telemetry Transport) and CoAP. In addition to their many other advantages, these messaging interfaces only carry IoT data, greatly reducing their capability to be used as an attack vector.

Message based interfaces are also a good approach for connecting the IoT Gateway to backend systems. An enterprise message bus like AMQP is a powerful tool for handling asynchronous inputs from thousands of gateways, routing them, and feeding the data into backend systems. A messaging system makes the total system more reliable, more robust, and more efficient – and makes it much easier to implement large scale systems! Messaging interfaces are ideal for handling exceptions – they allow you to simply send the exception as a regular message and have it properly processed and routed by business logic on the backend.

Messaging systems are also ideal for handling unreliable networks and heavy system loads. A messaging system will queue up messages until the network is available. If a sudden burst of activity causes the network and backend systems to be overloaded the messaging system will automatically queue up the messages and then release them for processing as resources become available. Messaging systems allow you to ensure reliable message delivery, which is critical for many applications. Best of all, messaging systems are easy for a programmer to use and do the hard work of building a robust communications capability for you.

No matter what type of interface you are using it is critical to sanitize your inputs. Never just pass through information from a device – instead, check it to make sure that is properly formatted, that it makes sense, that it does not contain a malicious payload, and that the data has not been corrupted. The overall integrity of an IoT system is greatly enhanced by ensuring the quality of the data it is operating on. Perhaps the best example of this is Little Bobby Tables from XKCD (XKCD.com):

Importance of sanitizing your input.

Importance of sanitizing your input.

On a more serious level, poor input sanitization is responsible for many security issues. Programmers should assume that users can’t be trusted and all interactions are a potential attack.

Episode 59 - The VPN Episode

Posted by Open Source Security Podcast on August 15, 2017 03:14 PM
Josh and Kurt talk about VPNs and the upcoming eclipse.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5644794/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

Installing FreeIPA with an Active Directory subordinate CA

Posted by Fraser Tweedale on August 14, 2017 06:04 AM

FreeIPA is often installed in enterprise environments for managing Unix and Linux hosts and services. Most commonly, enterprises use Microsoft Active Directory for managing users, Windows workstations and Windows servers. Often, Active Directory is deployed with Active Directory Certificate Services (AD CS) which provides a CA and certificate management capabilities. Likewise, FreeIPA includes the Dogtag CA, and when deploying FreeIPA in an enterprise using AD CS, it is often desired to make the FreeIPA CA a subordinate CA of the AD CS CA.

In this blog post I’ll explain what is required to issue an AD sub-CA, and how to do it with FreeIPA, including a step-by-step guide to configuring AD CS.

AD CS certificate template overview

AD CS has a concept of certificate templates, which define the characteristics an issued certificate shall have. The same concept exists in Dogtag and FreeIPA except that in those projects we call them certificate profiles, and the mechanism to select which template/profile to use when issuing a certificate is different.

In AD CS, the template to use is indicated by an X.509 extension in the certificate signing request (CSR). The template specifier can be one of two extensions. The first, older extension has OID and allows you to specify a template by name:

CertificateTemplateName ::= SEQUENCE {
   Name            BMPString

(Note that some documents specify UTF8String instead of BMPString. BMPString works and is used in practice. I am not actually sure if UTF8String even works.)

The second, Version 2 template specifier extension has OID and allows you to specify a template by OID and version:

CertificateTemplate ::= SEQUENCE {
    templateID              EncodedObjectID,
    templateMajorVersion    TemplateVersion,
    templateMinorVersion    TemplateVersion OPTIONAL

TemplateVersion ::= INTEGER (0..4294967295)

Note that some documents also show templateMajorVersion as optional, but it is actually required.

When submitting a CSR for signing, AD CS looks for these extensions in the request, and uses the extension data to select the template to use.

External CA installation in FreeIPA

FreeIPA supports installation with an externally signed CA certificate, via ipa-server-install --external-ca or (for existing CA-less installations ipa-ca-install --external-ca). The installation takes several steps. First, a key is generated and a CSR produced:

$ ipa-ca-install --external-ca

Directory Manager (existing master) password: XXXXXXXX

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/8]: configuring certificate server instance
The next step is to get /root/ipa.csr signed by your CA and re-run /sbin/ipa-ca-install as:
/sbin/ipa-ca-install --external-cert-file=/path/to/signed_certificate --external-cert-file=/path/to/external_ca_certificate

The installation program exits while the administrator submits the CSR to the external CA. After they receive the signed CA certificate, the administrator resumes the installation, giving the installation program the CA certificate and a chain of one or more certificates up to the root CA:

$ ipa-ca-install --external-cert-file ca.crt --external-cert-file ipa.crt
Directory Manager (existing master) password: XXXXXXXX

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [29/29]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).

Recall, however, that if the external CA is AD CS, a CSR must bear one of the certificate template specifier extensions. There is an additional installation program option to add the template specifier:

$ ipa-ca-install --external-ca --external-ca-type=ms-cs

This adds a name-based template specifier to the CSR, with the name SubCA (this is the name of the default sub-CA template in AD CS).

Specifying an alternative AD CS template

Everything discussed so far is already part of FreeIPA. Until now, there is no way to specify a different template to use with AD CS.

I have been working on a feature that allows an alternative AD CS template to be specified. Both kinds of template specifier extension are supported, via the new --external-ca-profile installation program option:

$ ipa-ca-install --external-ca --external-ca-type=ms-cs \

(Note: huge OIDs like the above are commonly used by Active Directory for installation-specific objects.)

To specify a template by name, the --external-ca-profile value should be:


To specify a template by OID, the OID and major version must be given, and optionally the minor version too:


Like --external-ca and --external-ca-type, the new --external-ca-profile option is available with both ipa-server-install and ipa-ca-install.

With this feature, it is now possible to specify an alternative or custom certificate template when using AD CS to sign the FreeIPA CA certificate. The feature has not yet been merged but there an open pull request. I have also made a COPR build for anyone interested in testing the feature.

The remainder of this post is a short guide to configuring Active Directory Certificate Services, defining a custom CA profile, and submitting a CSR to issue a certificate.

Renewing the certificate

FreeIPA provides the ipa-cacert-manage renew command for renewing an externally-signed CA certificate. Like installation with an externally-signed CA, this is a two-step procedure. In the first step, the command prompts Certmonger to generate a new CSR for the CA certificate, and saves the CSR so that the administrator can submit it to the external CA.

For renewing a certificate signed by AD CS, as in the installation case a template specifier extension is needed. Therefore the ipa-cacert-manage renew command has also learned the --external-ca-profile option:

# ipa-cacert-manage renew --external-ca-type ms-cs \
  --external-ca-profile MySubCA
Exporting CA certificate signing request, please wait
The next step is to get /var/lib/ipa/ca.csr signed by your CA and re-run ipa-cacert-manage as:
ipa-cacert-manage renew --external-cert-file=/path/to/signed_certificate --external-cert-file=/path/to/external_ca_certificate
The ipa-cacert-manage command was successful

The the above example the CSR that was generated will contain a version 1 template extension, using the name MySubCA. Like the installation commands, the version 2 extension is also supported.

This part of the feature requires some changes to Certmonger as well as FreeIPA. At time of writing these changes haven’t been merged. There is a Certmonger pull request and a Certmonger COPR build if you’d like to test the feature.

Appendix A: installing and configuring AD CS

Assuming an existing installation of Active Directory, AD CS installation and configuration will take 10 to 15 minutes. Open Server Manager, invoke the Add Roles and Features Wizard and select the AD CS Certification Authority role:


Proceed, and wait for the installation to complete…


After installation has finished, you will see AD CS in the Server Manager sidebar, and upon selecting it you will see a notification that Configuration required for Active Directory Certificate Services.


Click More…, and up will come the All Servers Task Details dialog showing that the Post-deployment Configuration action is pending. Click the action to continue:


Now comes the AD CS Configuration assistant, which contains several steps. Proceed past the Specify credentials to configure role services step.

In the Select Role Services to configure step, select Certification Authority then continue:


In the Specify the setup type of the CA step, choose Enterprise CA then continue:


The Specify the type of the CA step lets you choose whether the AD CS CA will be a root CA or chained to an external CA (just like how FreeIPA lets you create root or subordinate CA!) Installing AD CS as a Subordinate CA is outside the scope of this guide. Choose Root CA and continue:


The next step lets you Specify the type of the private key. You can use an existing private key or Create a new private key, the continue.

The Specify the cryptographic options step lets you specify the Key length and hash algorithm for the signature. Choose a key length of at least 2048 bits, and the SHA-256 digest:


Next, Specify the name of the CA. This sets the Subject Distinguished Name of the CA. Accept defaults and continue.

The next step is to Specify the validity period. CA certificates (especially root CAs) typically need a long validity period. Choose a value like 5 Years, then continue:


Accept defauts for the Specify the database locations step.

Finally, you will reach the Confirmation step, which summarises the chosen configuration options. Review the settings then Configure:


The configuration will take a few moments, then the Results will be displayed:


AD CS is now configured and you can begin issuing certificates.

Appendix B: creating a custom sub-CA certificate template

In this section we look at how to create a new certificate template for sub-CAs by duplicating an existing template, then modifying it.

To manage certificate templates, from Server Manager right-click the server and open the Certification Authority program:


In the sidebar tree view, right-click Certificate Templates then select Manage.


The Certificate Templates Console will open. The default profile for sub-CAs has the Template Display Name Subordinate Certification Authority. Right-click this template and choose Duplicate Template.


The new template is created and the Properties of New Template dialog appears, allowing the administrator to customise the template. You can set a new Template display name, Template name and so on:


You can also change various aspects of certificate issuance including which extensions will appear on the issued certificate, and the values of those extensions. In the following screenshot, we see a new Certificate Policies OID being defined for addition to certificates issued via this template:


Also under Extensions, you can discover the OID for this template by looking at the Certificate Template Information extension description.

Finally, having defined the new certificate template, we have to activate it for use with the AD CA. Back in the Certification Authority management window, right-click Certificate Templates and select Certificate Template to Issue:


This will pop up the Enable Certificate Templates dialog, containing a list of templates available for use with the CA. Select the new template and click OK. The new certificate template is now ready for use.

Appendix C: issuing a certificate

In this section we look at how to use AD CS to issue a certificate. It is assumed that the CSR to be signed exists and Active Directory can access it.

In the Certification Authority window, in the sidebar right-click the CA and select All Tasks >> Submit new request…:


This will bring up a file chooser dialog. Find the CSR and Open it:


Assuming all went well (including the CSR indicating a known certificate template), the certificate is immediately issued and the Save Certificate dialog appear, asking where to save the issued certificate.

But that's not my job!

Posted by Josh Bressers on August 13, 2017 07:45 PM
This week I've been thinking about how security people and non security people interact. Various conversations I have often end up with someone suggesting everyone needs some sort of security responsibility. My suspicion is this will never work.

First some background to think about. In any organization there are certain responsibilities everyone has. Without using security as our specific example just yet, let's consider how a typical building functions. You have people who are tasked with keeping the electricity working, the plumbing, the heating and cooling. Some people keep the building clean, some take care of the elevators. Some work in the building to accomplish some other task. If the company that inhabits the building is a bank you can imagine the huge number of tasks that take place inside.

Now here's where I want our analogy to start. If I work in a building and I see a leaking faucet. I probably would report it. If I didn't, it's likely someone else would see it. It's quite possible if I'm one of the electricians and while accessing some hard to reach place I notice a leaking pipe. That's not my job to fix it, I could tell the plumbers but they're not very nice to me, so who cares. The last time I told them about a leaking pipe they blamed me for breaking it, so I don't really have an incentive here. If I do nothing, it really won't affect me. If I tell someone, at best it doesn't affect me, but in reality I probably will get some level of blame or scrutiny.

This almost certainly makes sense to most of us. I wonder if there are organizations where reporting things like this comes with an incentive. A leaking water pipe could end up causing millions in damage before it's found. Nowhere I've ever worked ever really had an incentive to report things like this. If it's not your job, you don't really have to care, so nobody ever really cared.

Now let's think about phishing in a modern enterprise. You see everything from blaming the user who clicked the link, to laughing at them for being stupid, to even maybe firing someone for losing the company a ton of money. If a user clicks a phishing link, and suspects a problem, they have very little incentive to be proactive. It's not their job. I bet the number of clicked phish links we find out about is much much lower than the total number clicked.

I also hear security folks talking about educating the users on how all this works. Users should know how to spot phishing links! While this won't work for a variety of reasons, at the end of the day, it's not their job so why do we think they should know how to do this? Even more important, why do we think they should care?

The think I keep wondering is should this be the job of everyone or just the job of the security people? I think the quick reaction is "everyone" but my suspicion is it's not. Electricity is a great example. How many stories have you heard of office workers being electrocuted in the office? The number is really low because we've made electricity extremely safe. If we put this in the context of modern security we have a system where the office is covered in bare wires. Imagine wires hanging from the ceiling, some draped on the floor. The bathroom has sparking wires next to the sink. We lost three interns last week, those stupid interns! They should have known which wires weren't safe to accidentally touch. It's up to everyone in the office to know which wires are safe and which are dangerous!

This is of course madness, but it's modern day security. Instead of fixing the wires, we just imagine we can train everyone up on how to spot the dangerous ones.

Docker without sudo on Centos 7

Posted by Adam Young on August 09, 2017 06:28 PM

I have been geting prepped to build the OpenShift origin codebase on Centos 7.  I started from a fairly minimal VM which did not have docker or Development Tools installed.  Once I thought I had all the prerequisites, I kicked off the build and got

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

This seems to be due to the fact that  the ayoung user does not have permissions to read/write on the domain socket.  /var/run/docker.sock

$ ls -la /var/run/docker.sock
srw-rw----. 1 root root 0 Aug 9 09:03 /var/run/docker.sock

Enough other stuff seems to discuss this as well.  How can we set up for non-root and non-sudo access to docker?

On my Fedora system, I have:

$ ls -la /var/run/docker.sock
srw-rw----. 1 root docker 0 Aug 7 09:01 /var/run/docker.sock

I set this up long enough ago that I do not remember if I was the one that did this, or if it was a configuration setup by some other package. The docker group has a pretty random ID:

$ getent group docker

So I probably did that.

Back to the VM:

sudo groupadd docker
 sudo chown root:docker /var/run/docker.sock
 sudo usermod -aG docker ayoung

I exited out and logged back in:

$ groups
ayoung wheel docker

And it worked.  Will the socket stay that way?  Hmm.  After the build completes, I’ll reboot the VM and see what we have.

Yes it did.  Is there a better way to do this?  Let me know if you do.


Episode 58 - Backwards compatibility to the point of insanity

Posted by Open Source Security Podcast on August 09, 2017 01:21 PM
Josh and Kurt talk about MalwareTech, Debian killing off TLS 1.0 and 1.1, auto safety, HBO, and npm not typo squatting.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5624741/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes

Bonding two Ethernet devices

Posted by Adam Young on August 08, 2017 10:15 PM

In my continued investigations of networking stuff, I came across the question “How do you bond two ethernet devices together?”   While I did this years ago on RHEL3, I have pretty much forgotten how, so I decided to research and relearn this.

To start, I cloned my Centos 7 VM so I had a throw-away playground.  Before booting the VM, I added a second ethernet device on the default network, parallel to the existing device.


Then I booted it on up.

To start with, I have two ethernet devices that have both gotten their configuration information via DHCP.  I’m going to trash those to get to a clear starting point:

First, lets do it the easy way, with THe Network Manager Text UI (nmtui).

Create brings up this screen

Next to the box that says “slaves” (that term needs to go) Select add, and select ethernet:

And take all the defaults.

Do this again for the second ethernet device as well, and the result should look like this:

And exit out of nmtui.  You can use ip addr to see the current set up, and use ping to confirm that it works.

Lets do that using the CLI.  First, cleanup.

nmcli co delete "Bond connection 1"
nmcli co delete "Ethernet connection 1"
nmcli co delete "Ethernet connection 2"

Turns out there is a tutorial on the machine already:

man nmcli-examples

Example 6. Adding a bonding master and two slave connection profiles

$ nmcli con add type bond ifname mybond0 mode active-backup
$ nmcli con add type ethernet ifname eth1 master mybond0
$ nmcli con add type ethernet ifname eth2 master mybond0

Of course, that should be eth0 and eth1, but the steps laid out work and, again, you can test with ping once it is done.