Fedora security Planet

Episode 78 - Risk lessons from Hawaii

Posted by Open Source Security Podcast on January 16, 2018 10:40 PM
Josh and Kurt talk about the accidental missile warning in Hawaii. We also discuss general preparedness and risk.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/6156285/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes


Episode 77 - npm and the supply chain

Posted by Open Source Security Podcast on January 11, 2018 02:42 AM
Josh and Kurt talk about the recent npm happenings. What it means for the supply chain, and we end with some thoughts on how maybe none of this matters.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/6134997/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes


Episode 76 - Meltdown aftermath

Posted by Open Source Security Podcast on January 07, 2018 09:11 PM
Josh and Kurt talk about the aftermath of Meltdown. The details of the flaw are probably less interesting than what happens now.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/6123826/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes


NOTICE: BLOG HAS MOVED

Posted by Fraser Tweedale on January 05, 2018 03:18 AM

Due to imminent shutdown of OpenShift Online v2 hosting
environment, this blog has MOVED to
https://frasertweedale.github.io/blog-redhat/

Using b43 firmware on Fedora Atomic Workstation

Posted by William Brown on December 22, 2017 02:00 PM

Using b43 firmware on Fedora Atomic Workstation

My Macbook Pro has a broadcom b43 wireless chipset. This is notorious for being one of the most annoying wireless adapters on linux. When you first install Fedora you don’t even see “wifi” as an option, and unless you poke around in dmesg, you won’t find how to enable b43 to work on your platform.

b43

The b43 driver requires proprietary firmware to be loaded else the wifi chip will not run. There are a number of steps for this process found on the linux wireless page . You’ll note that one of the steps is:

export FIRMWARE_INSTALL_DIR="/lib/firmware"
...
sudo b43-fwcutter -w "$FIRMWARE_INSTALL_DIR" broadcom-wl-5.100.138/linux/wl_apsta.o

So we need to be able to write and extract our firmware to /usr/lib/firmware, and then reboot and out wifi works.

Fedora Atomic Workstation

Atomic WS is similar to atomic server, that it’s a read-only ostree based deployment of fedora. This comes with a number of unique challenges and quirks but for this issue:

sudo touch /usr/lib/firmware/test
/bin/touch: cannot touch '/usr/lib/firmware/test': Read-only file system

So we can’t extract our firmware!

Normally linux also supports reading from /usr/local/lib/firmware (which on atomic IS writeable ...) but for some reason fedora doesn’t allow this path.

Solution: Layered RPMs

Atomic has support for “rpm layering”. Ontop of the ostree image (which is composed of rpms) you can supply a supplemental list of packages that are “installed” at rpm-ostree update time.

This way you still have an atomic base platform, with read-only behaviours, but you gain the ability to customise your system. To achive it, it must be possible to write to locations in /usr during rpm install.

This means our problem has a simple solution: Create a b43 rpm package. Note, that you can make this for yourself privately, but you can’t distribute it for legal reasons.

Get setup on atomic to build the packages:

rpm-ostree install rpm-build createrepo
reboot

RPM specfile:

%define debug_package %{nil}
Summary: Allow b43 fw to install on ostree installs due to bz1512452
Name: b43-fw
Version: 1.0.0
Release: 1
License: Proprietary, DO NOT DISTRIBUTE BINARY FORMS
URL: http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/
Group: System Environment/Kernel

BuildRequires: b43-fwcutter

Source0: http://www.lwfinger.com/b43-firmware/broadcom-wl-5.100.138.tar.bz2

%description
Broadcom firmware for b43 chips.

%prep
%setup -q -n broadcom-wl-5.100.138

%build
true

%install
pwd
mkdir -p %{buildroot}/usr/lib/firmware
b43-fwcutter -w %{buildroot}/usr/lib/firmware linux/wl_apsta.o

%files
%defattr(-,root,root,-)
%dir %{_prefix}/lib/firmware/b43
%{_prefix}/lib/firmware/b43/*

%changelog
* Fri Dec 22 2017 William Brown <william at blackhats.net.au> - 1.0.0
- Initial version

Now you can put this into a folder like so:

mkdir -p ~/rpmbuild/{SPECS,SOURCES}
<editor> ~/rpmbuild/SPECS/b43-fw.spec
wget -O ~/rpmbuild/SOURCES/broadcom-wl-5.100.138.tar.bz2 http://www.lwfinger.com/b43-firmware/broadcom-wl-5.100.138.tar.bz2

We are now ready to build!

rpmbuild -bb ~/rpmbuild/SPECS/b43-fw.spec
createrepo ~/rpmbuild/RPMS/x86_64/

Finally, we can install this. Create a yum repos file:

[local-rpms]
name=local-rpms
baseurl=file:///home/<YOUR USERNAME HERE>/rpmbuild/RPMS/x86_64
enabled=1
gpgcheck=0
type=rpm
rpm-ostree install b43-fw

Now reboot and enjoy wifi on your Fedora Atomic Macbook Pro!

Episode 75 - Security Planner review

Posted by Open Source Security Podcast on December 19, 2017 07:49 PM
Josh and Kurt talk about the Security Planner website. It's pretty good all things considered.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/6065761/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes





Picking the Right Hammer for the Job

Posted by Adam Young on December 18, 2017 08:12 PM

Red Hat Satellite Server is a key tool in the provisioning process for the systems in our Labs.  In one of our labs we have an older deployment running Satellite 6 which maps to the upstream project The Foreman version 1.11.  Since I want to be able to perform repeatable operations on this server, I need to make Web API calls.

The easiest way to do this is to use the Hammer CLI. But it turns out the version of Hammer is somewhat tied to the version of Satellite server; the version I have in Fedora 27 Does not talk to this older Satellite instance.  So, I want to run an older Hammer.

I decided to use this as an opportunity to walk through running an RPM managed application targetted for RHEL 6/EPEL 6 via Docker.

Edit: actually, this might not be the case, but the rest of the learning process was interesting enough that I kept working at it.
Edit2: This was necessary, see the bottom. Also, the 1.11 in the URL refers to the upstream repo for theforeman. I’d use a different repo for building using supported RH RPMs.

Here is what I learned.

It took some trial and error to get to the following Dockerfile

FROM index.docker.io/centos:6
 MAINTAINER Adam Young <ayoung@redhat.com>

COPY foreman.repo /etc/yum.repos.d/
 RUN yum -y install centos-release-scl yum-utils
 RUN yum-config-manager --enable rhel-server-rhscl-6-rpm
 RUN yum -y install rh-ruby22
 RUN yum -y install tfm-rubygem-hammer_cli &&\
 yum -y clean all

CMD /usr/bin/hammer

The reason I have the separate RUN lines as opposed to using the &&\ line continuation technique is that I wanted a faster turnaround time for making changes;

With this docker file, I can build the image using

docker build -t hammer-cli .

And run the CLI using

docker run hammer-cli

which produces the output

Usage:
 hammer [OPTIONS] SUBCOMMAND [ARG] ...

Parameters:
 SUBCOMMAND subcommand
 [ARG] ... subcommand arguments

Subcommands:
 defaults Defaults management
 shell Interactive shell

Options:
 --autocomplete LINE Get list of possible endings
 --csv Output as CSV (same as --output=csv)
 --csv-separator SEPARATOR Character to separate the values
 --interactive INTERACTIVE Explicitly turn interactive mode on/off
 One of true/false, yes/no, 1/0.
 --output ADAPTER Set output format. One of [base, table, silent, csv, yaml, json]
 --show-ids Show ids of associated resources
 --version show version
 -c, --config CFG_FILE path to custom config file
 -d, --debug show debugging output
 -h, --help print help
 -p, --password PASSWORD password to access the remote system
 -r, --reload-cache force reload of Apipie cache
 -s, --server SERVER remote system address
 -u, --username USERNAME username to access the remote system
 -v, --verbose be verbose

If I want to pass command line arguments to the command, I can override the entrypoint.  To get help I can run:

docker run hammer-cli /usr/bin/hammer --help

But trying to run a non-trivial command seems to fail:

$ docker run hammer-cli /usr/bin/hammer host
Error: No such sub-command 'host'

See: 'hammer --help'

Perhaps I need additional plugins?

On a functioning system, I can run

hammer user list

And get a response. What other RPMS are on there:

$ rpmquery -a | grep hammer
 tfm-rubygem-hammer_cli-0.5.1.13-2.el7sat.noarch
 tfm-rubygem-hammer_cli_foreman_remote_execution-0.0.5.3-1.el7sat.noarch
 tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3.3-1.el7sat.noarch
 tfm-rubygem-hammer_cli_foreman_tasks-0.0.10.3-1.el7sat.noarch
 tfm-rubygem-hammer_cli_katello-0.0.22.29-1.el7sat.noarch
 tfm-rubygem-hammer_cli_foreman_discovery-0.0.2.3-1.el7sat.noarch
 tfm-rubygem-hammer_cli_import-0.11.2-1.el7sat.noarch
 tfm-rubygem-hammer_cli_csv-2.2.1.1-1.el7sat.noarch
 tfm-rubygem-hammer_cli_foreman_docker-0.0.6-1.el7sat.noarch
 tfm-rubygem-hammer_cli_foreman-0.5.1.10-1.el7sat.noarch
 tfm-rubygem-hammer_cli_foreman_admin-0.0.5-1.el7sat.noarch

Lets add these in to the docker file and see if it makes a difference;

After much trial and error, it turns out the issue is with a few things, including DNS. I did need the plugins above.  I finally got the command to run like this:

docker run hammer-cli /usr/bin/hammer -p redacted -u tester -s https://10.3.76.6 host list

 

Here is my final dockerfile

FROM index.docker.io/centos:6
MAINTAINER Adam Young <ayoung>

COPY foreman.repo /etc/yum.repos.d/
RUN yum -y install centos-release-scl yum-utils
RUN yum-config-manager --enable rhel-server-rhscl-6-rpm
RUN yum -y install rh-ruby22
RUN yum -y install tfm-rubygem-hammer_cli \
tfm-rubygem-hammer_cli \
tfm-rubygem-hammer_cli_foreman_remote_execution \
tfm-rubygem-hammer_cli_foreman_bootdisk \
tfm-rubygem-hammer_cli_foreman_tasks \
tfm-rubygem-hammer_cli_katello \
tfm-rubygem-hammer_cli_foreman_discovery \
tfm-rubygem-hammer_cli_import \
tfm-rubygem-hammer_cli_csv \
tfm-rubygem-hammer_cli_foreman_docker \
tfm-rubygem-hammer_cli_foreman \
tfm-rubygem-hammer_cli_foreman_admin
RUN yum -y clean all

CMD /usr/bin/hammer

Edit: 2 so I DID need this version of Hammer, as the host create failed using the latest. I was able to createa host using the following command:

docker run  hammer-cli  /usr/bin/hammer  \
       -p redacted -u tester \
       -s https://10.3.76.6 \
       host create \
       --name blade02 \
       --architecture x86_64 \
       --domain mydomain.net \
       --operatingsystem "RedHat 7.4" \
       --partition-table "Kickstart default" \
       --location Phoenix \
       --hostgroup "MYDOMAINHOSTGROUP" \
       --interface="primary=true, \
            mac=D8:D3:85:C3:3C:08" \
       --organization DEMO \
       --root-pass redacted

Episode 74 - Facial recognition and physical security

Posted by Open Source Security Podcast on December 13, 2017 02:21 AM
Josh and Kurt talk about facial recognition, physical security, banking, and Amazon Alexa.

<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/6039839/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes



Episode 73 - Security from Santa

Posted by Open Source Security Podcast on December 06, 2017 01:02 AM
Josh and Kurt talk about basic security metrics and security from Santa. Is Santa GDPR compliant?




<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/6015418/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes



Tracing a Tempest Failure in Keystone

Posted by Adam Young on December 03, 2017 01:54 AM

The tools that we’ve used to develop Keystone have changed a bit over the years.  As I work on some long standing bugs, I’ve had to learn what the latest tools are, and how to use them.  Recently I had to track down the cause of a failed Tempest run.  Here are the steps I went through to find the trace.

First, lets start with a failure of a review that I submitted. This failed, not in the unit tests, but in the functional tests.  I had not run against a full tempest suite, not suspecting that the tests would fail there, but they did.

Note the tempest failure on legacy-tempest-dsvm-py35.  Since this is a voting check, if I don’t get the test to pass, the review will not merge.  This is a clickable link, and clicking on it brings me to the top level page with a little guide to the test run:

This does not give a lot to work with.  What I want is the test output.  You can see that in the past I’ve clicked on logs (it is a more reddish shade than the blue links).  That brings up the cached logs.

Scroll down to the bottom to find the link for tempest logs.

The tempest log output looks like this:

And clicking on the blue “fail” link for expands the stack trace, telling me what actually failed.

Here I can read the stack trace and see it was calling “create_endpoint” which failed due to a policy check.

 

Sending FreeOTP Codes Over Bluetooth

Posted by Nathaniel McCallum on December 02, 2017 04:23 PM

One Time Passwords are everywhere these days. We have great standards, like HOTP and TOTP, which means that all our implementations can be interoperable. We also have open source implementations of these standards, such as FreeOTP. Tons of server applications and web services use OTPs every day for millions of users.

But, currently, one big user experience issue comes to the fore: typing OTP codes is error-prone and time-consuming. Can’t we do better?

On and off for the last few years, I have been working to improve this situation. The result I have come up with is a companion app to FreeOTP called Jelling that allows FreeOTP to send OTP codes directly to paired Bluetooth Low Energy devices. I hope to explain in this post how Jelling works and outline the challenges that still remain. Hopefully you, my most esteemed reader, can help us close some of the gaps that remain.

Bluetooth LE GATT

Bluetooth Low Energy (BLE) has a mode of operation called the Generic Attribute Profile (GATT). Applications define collections of services that contain characteristics which are manipulated by remote entities.1 This means we can define our own protocol using GATT and use this protocol to send a token code from a Sender to the intended Receiver. The Sender will be an Android or iOS version of the FreeOTP application. The Receiver will usually be a Windows, macOS or Linux computer.

First, we have to decide whether the Sender or the Receiver will be the Peripheral or the Central. A Peripheral sends advertisement packets which tell other devices that the specified service is available for use. The Central scans for these advertisements and, when found, can connect to the Peripheral and use its Services. Two principles are important. First, a Peripheral can only have one Central, but a Central can have many Peripherals. Second, scanning uses more power than advertising, so it should be used sparingly.

Second, we need to consider the user experience. Attempting to initialize the token sharing transaction from the Receiver might seem to make the most sense, since the user is already typically doing something on the Receiver when he wants a token code. However, this means we would need to negotiate over BLE about which token code to receive. This negotiation would use a lot of power. Further, because the tokens are authentication credentials, we must confirm on the Sender before sending them to protect from token theft. On the other hand, if the Sender selects the token first and initiates the share we don’t require negotiation or confirmation at all. Therefore, we can reduce the number of GATT requests we need to send to one.2

Third, we need to consider security. We can’t just share a token without being sure about who will receive it. This means that we must use both BLE authentication and encryption. Further, if we implemented the Sender as a Peripheral where it sent characteristic notifications including the token code, we often can’t see who is subscribed to those notifications due to platform API limitations. This doesn’t work for our case since we might care that some OTP codes are only sent to some paired devices.

All of this leads to a simple implementation. The Sender (FreeOTP) operates in Central mode and the Receiver (computer) is a Peripheral that exposes a single service (B670003C-0079-465C-9BA7-6C0539CCD67F) which, in turn, exposes a single, write-only characteristic (F4186B06-D796-4327-AF39-AC22C50BDCA8). We protect this characteristic using both BLE encryption and authentication. The Sender initiates the connection and transfers the OTP by sending a single write to the characteristic containing the OTP code it wishes to share. This means that the Receiver needs to advertise whenever it wants to receive token codes (default: always3). The Sender scans and connects only when it wants to share; this preserves battery life. The Sender chooses which token to share and which Receiver to send it to. Once the Receiver has the token, it emulates a keyboard and types wherever the cursor is. This user experience is simple and intuitive.

Project Status

The Good News

I’ve implemented Receivers for Windows, macOS and Linux. Each is implemented using its platform’s native development tooling and languages (C#, Swift and C, respectively). The Receivers advertise themselves as a BLE Peripheral which FreeOTP can connect to via pairing. Once paired, FreeOTP can send tokens to the selected Receiver.

I’ve also implemented this behavior in FreeOTP. For iOS, this is already merged to master. For Android, this lives in the bt branch which I hope to merge soon. Like always, you can click a token to show the code directly. However, now you can choose to share the token. This pops up a secondary menu which shows the available Receivers within range. Selecting a Receiver shares the token if already paired. Otherwise, it begins the pairing procedure.

The Bad News

Unfortunately, the platform support for this functionality appears to be frustratingly inconsistent. The following chart documents my success and failure. This test reflects my tests with both FreeOTP and LightBlue (a BLE GATT testing application on iOS and Android). This proves (to me) that the problem isn’t my code but rather platform incompatibilities. For more details, see the charts in the README.md of each repository.

Status Matrix

Pixel Nexus 5x iPhone 6+
Windows
macOS
Linux

Windows

Sharing works with my iPhone 6+. It does not work from either my Google Pixel or Nexus 5x. I suspect my Pixel has problems because I wasn’t able to get it to work anywhere. However, on my Nexus 5x the simple, single GATT write never causes the callback to fire in my Windows code. It does work if I disable authentication. This leads me to believe that there is a compatibility issue between Windows and Android during pairing.

macOS

The macOS BLE implementation seems to be the most mature of all the platforms. Sharing works on both my iPhone 6 and Nexus 5x. Like I mentioned above, my Pixel seems horribly broken. I even updated to 8.1 beta to see if it has been fixed. No dice. I will try on another Pixel that has been reset to factory defaults soon.

Linux

Unfortunately, Linux appears to be the most broken of all the platforms. With my iPhone 6+ I am able to successfully see the advertisement and perform service discovery. But when I perform a GATT write, pairing never begins. My Nexus 5x can see the advertisement and connect, but it fails performing service discovery. My Pixel connects only in non-BLE mode (it does seem to connect in BR/ATT mode; but that is useless).

The Call for Help!

I’d really like to bring this functionality to a FreeOTP release in the near future. But we need your help! “How?” you ask.

  1. Test Jelling on your systems. We need tests with all different kinds of phones and laptops. Test instructions are available in the README.md of each repository (Windows, macOS and Linux).

  2. Assist debugging combinations where things are broken. In particular, we need help from Bluez. If you are a Bluez developer, please contact me!

  3. Review the code. I’m not a Windows or macOS developer. If I’m doing something wrong, it is probably my fault. Bug reports are welcome. Patches are even better.


  1. Search for “Bluetooth GATT” to learn more. [return]
  2. “Here is the token code.” [return]
  3. There is a trade-off here between usability, privacy and battery usage. The later is much less of a concern in the Receiver since it has a large battery and is often plugged in. Privacy is a tough one. If you’re advertising you can be used by FreeOTP, you can also be tracked by someone physically nearby. We consider this low-risk. [return]

Episode 72 - Bitcoin: It's over 9000

Posted by Open Source Security Podcast on November 28, 2017 08:47 PM
Josh and Kurt talk about Bitcoin, blockchain, and other cryptocurrencies.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5988551/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes


CloudForms’ Domains Import and Export

Posted by Adam Young on November 28, 2017 03:47 PM

DevOps requires that everything goes into Revision Control. CloudForms’ modifications are no exception. But how do you revision control something that is managed by a GUI and stored in a database? Import and export with the command line. Here’s how.

To get code into and out of CloudForms, you use rake.  A version of version of rake is provided with the appliance.

This requires a rakefile, and to get access to the default one distributed with the appliance, run

cd /var/www/miq/vmdb

I’m using /tmp/cfme as my import and export directory, and the domain is younglogic. Adjust your commands appropriately.

To export code from the appliance to a directory, use the following command:

 /var/www/miq/vmdb/bin/rake evm:automate:export DOMAIN=younglogic EXPORT_DIR=/tmp/cfme OVERWRITE=true

To import code into the CloudForms server, run:

/var/www/miq/vmdb/bin/rake evm:automate:import DOMAIN=younglogic IMPORT_DIR=/tmp/cfme SYSTEM=false ENABLED=true PREVIEW=false OVERWRITE=true

The OVERWRITE parameter is essential:  if you do not include it, you will not see changes in the destination.  It is a a little tricky;  without OVERWRITE explicitly set, either command will work the first time, when there is no code in CloudForms for import or if the target directory is empty for export.

The PREVIEW=false is also essential: The PREVIEW option is used for doing a test run. If PREVIEW is set to true, no changes will be written.

There are a few points to consider when using this for your workflow.

  1. File permissions.  If you run export as root, you will create the target directory as the root user.  This is not usually the preferred account.  However, on the appliance, the root directory /var/www/miq/vmdb and all its children are owned by root:root.  A better approach would allow this code to be run as a non-root user based on group membership.  Locking down sudo for the user such that they can only run /var/www/miq/vmdb/bin/rake might be an acceptable mitigating approach as well.
  2. Running this on a production server.  it might make sense to have a single CFME instance that has almost all roles removed except for those required to import and export code in order to constrain where the import process can be run.
  3. This is not the git repositories owner role.  That is explicitly for managing access to Ansible playbooks.
  4. There is no guarantee that old files will be removed. If you are concerned about old artifacts, you should delete the domain on the server prior to import. This may not be a real concern; It is always possible to delete spurious items post import, too. I’ll update this as I learn more.

With those things in mind, we should be able to build a workflow for Continuous Integration and Deployment of code to our Cloud Forms instances.

Changing a CA’s Subject DN; Part II: FreeIPA

Posted by Fraser Tweedale on November 22, 2017 02:23 AM

In the previous post I explained how the CA Subject DN is an integral part of X.509 any why you should not change it. Doing so can break path validation, CRLs and OCSP, and many programs will not copye with the change. I proposed some alternative approaches that avoid these problems: re-chaining the CA, and creating subordinate CAs.

If you were thinking of changing your CA’s Subject DN, I hope that I dissuaded you. But if I failed, or you absolutely do need to change the Subject DN of your CA, where there’s a will there’s way. The purpose of this post is to explore how to do this in FreeIPA, and discuss the implications.

This is a long post. If you are really changing the CA subject DN, don’t skip anything. Otherwise don’t feel back about skimming or jumping straight to the discussion. Even skimming the article will give you an idea of the steps involved, and how to repair the ensuing breakage.

Changing the FreeIPA CA’s Subject DN

Before writing this post, I had never even attempted to do this. I am unaware of anyone else trying or whether they were successful. But the question of how to do it has come up several times, so I decided to investigate. The format of this post follows my exploration of the topic as I poked and prodded a FreeIPA deployment, working towards the goal.

What was the goal? Let me state the goal, and some assumptions:

  • The goal is to give the FreeIPA CA a new Subject DN. The deployment should look and behave as though it were originally installed with the new Subject.
  • We want to keep the old CA certificate in the relevant certificate stores and databases, alongside the new certificate.
  • The CA is not being re-keyed (I will deal with re-keying in a future article).
  • We want to be able to do this with both self-signed and externally-signed CAs. It’s okay if the process differs.
  • It’s okay to have manual steps that the administrator must perform.

Let’s begin on the deployment’s CA renewal master.

Certmonger (first attempt)

There is a Certmonger tracking request for the FreeIPA CA, which uses the dogtag-ipa-ca-renew-agent CA helper. The getcert resubmit command lets you change the Subject DN when you resubmit a request, via the -N option. I know the internals of the CA helper and I can see that there will be problems after renewing the certificate this way. Storing the certificate in the ca_renewal LDAP container will fail. But the renewal itself might succeed so I’ll try it and see what happens:

[root@f27-2 ~]# getcert resubmit -i 20171106062742 \
  -N 'CN=IPA.LOCAL CA 2017.11.09'
Resubmitting "20171106062742" to "dogtag-ipa-ca-renew-agent".

After waiting about 10 seconds for Certmonger to do its thing, I check the state of the tracking request:

[root@f27-2 ~]# getcert list -i 20171106062742
Request ID '20171106062742':
  status: MONITORING
  CA: dogtag-ipa-ca-renew-agent
  issuer: CN=Certificate Authority,O=IPA.LOCAL 201711061603
  subject: CN=Certificate Authority,O=IPA.LOCAL 201711061603
  expires: 2037-11-06 17:26:21 AEDT
  ... (various fields omitted)

The status and expires fields show that renewal succeeded, but the certificate still has the old Subject DN. This happened because the dogtag-ipa-ca-renew-agent helper doesn’t think it is renewing the CA certificate (which is true!)

Modifying the IPA CA entry

So let’s trick the Certmonger renewal helper. dogtag-ipa-ca-renew-agent looks up the CA Subject DN in the ipaCaSubjectDn attribute of the ipa CA entry (cn=ipa,cn=cas,cn=ca,{basedn}). This attribute is not writeable via the IPA framework but you can change it using regular LDAP tools (details out of scope). If the certificate is self-signed you should also change the ipaCaIssuerDn attribute. After modifying the entry run ipa ca-show to verify that these attributes have the desired values:

[root@f27-2 ~]# ipa ca-show ipa
  Name: ipa
  Description: IPA CA
  Authority ID: cdbfeb5a-64d2-4141-98d2-98c005802fc1
  Subject DN: CN=IPA.LOCAL CA 2017.11.09
  Issuer DN: CN=IPA.LOCAL CA 2017.11.09
  Certificate: MIIDnzCCAoegAwIBAgIBCTANBgkqhkiG9w0...

Certmonger (second attempt)

Now let’s try and renew the CA certificate via Certmonger again:

[root@f27-2 ~]# getcert resubmit -i 20171106062742 \
  -N 'CN=IPA.LOCAL CA 2017.11.09'
Resubmitting "20171106062742" to "dogtag-ipa-ca-renew-agent".

Checking the getcert list output after a short wait:

[root@f27-2 ~]# getcert list -i 20171106062742
Request ID '20171106062742':
  status: MONITORING
  CA: dogtag-ipa-ca-renew-agent
  issuer: CN=Certificate Authority,O=IPA.LOCAL 201711061603
  subject: CN=IPA.LOCAL CA 2017.11.09
  expires: 2037-11-09 16:11:12 AEDT
  ... (various fields omitted)

Progress! We now have a CA certificate with the desired Subject DN. The new certificate has the old (current) issuer DN. We’ll ignore that for now.

Checking server health

Now I need to check the state of the deployment. Did anything go wrong during renewal? Is everything working?

First, I checked the Certmonger journal output to see if there were any problems. The journal contained the following messages (some fields omitted for brevity):

16:11:17 /dogtag-ipa-ca-renew-agent-submit[1662]: Forwarding request to dogtag-ipa-renew-agent
16:11:17 /dogtag-ipa-ca-renew-agent-submit[1662]: dogtag-ipa-renew-agent returned 0
16:11:19 /stop_pkicad[1673]: Stopping pki_tomcatd
16:11:20 /stop_pkicad[1673]: Stopped pki_tomcatd
16:11:22 /renew_ca_cert[1710]: Updating CS.cfg
16:11:22 /renew_ca_cert[1710]: Updating CA certificate failed: no matching entry found
16:11:22 /renew_ca_cert[1710]: Starting pki_tomcatd
16:11:34 /renew_ca_cert[1710]: Started pki_tomcatd
16:11:34 certmonger[2013]: Certificate named "caSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" issued by CA and saved.

We can see that the renewal succeeded and Certmonger saved the new certificate in the NSSDB. Unfortunately there was an error in the renew_ca_cert post-save hook: it failed to store the new certificate in the LDAP certstore. That should be easy to resolve. I’ll make a note of that and continue checking deployment health.

Next, I checked whether Dogtag was functioning. systemctl status pki-tomcatd@pki-tomcat and the CA debug log (/var/log/pki/pki-tomcat/ca/debug) indicated that Dogtag started cleanly. Even better, the Dogtag NSSDB has the new CA certificate with the correct nickname:

[root@f27-2 ~]# certutil -d /etc/pki/pki-tomcat/alias \
  -L -n 'caSigningCert cert-pki-ca'
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 11 (0xb)
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=Certificate Authority,O=IPA.LOCAL 201711061603"
        Validity:
            Not Before: Thu Nov 09 05:11:12 2017
            Not After : Mon Nov 09 05:11:12 2037
        Subject: "CN=IPA.LOCAL CA 2017.11.09"
  ... (remaining lines omitted)

We have not yet confirmed that the Dogtag uses the new CA Subject DN as the Issuer DN on new certificates (we’ll check this later).

Now let’s check the state of IPA itself. There is a problem in communication between the IPA framework and Dogtag:

[root@f27-2 ~]# ipa ca-show ipa
ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500.

A quick look in /var/log/httpd/access_log showed that it was not a general problem but only occurred when accessing a particular resource:

[09/Nov/2017:17:15:09 +1100] "GET https://f27-2.ipa.local:443/ca/rest/authorities/cdbfeb5a-64d2-4141-98d2-98c005802fc1/cert HTTP/1.1" 500 6201

That is a Dogtag lightweight authority resource for the CA identified by cdbfeb5a-64d2-4141-98d2-98c005802fc1. That is the CA ID recorded in the FreeIPA ipa CA entry. This gives a hint about where the problem lies. An ldapsearch reveals more:

[f27-2:~] ftweedal% ldapsearch -LLL \
    -D 'cn=directory manager' -w DM_PASSWORD \
    -b 'ou=authorities,ou=ca,o=ipaca' -s one
dn: cn=cdbfeb5a-64d2-4141-98d2-98c005802fc1,ou=authorities,ou=ca,o=ipaca
authoritySerial: 9
objectClass: authority
objectClass: top
cn: cdbfeb5a-64d2-4141-98d2-98c005802fc1
authorityID: cdbfeb5a-64d2-4141-98d2-98c005802fc1
authorityKeyNickname: caSigningCert cert-pki-ca
authorityEnabled: TRUE
authorityDN: CN=Certificate Authority,O=IPA.LOCAL 201711061603
description: Host authority

dn: cn=008a4ded-fd4b-46fe-8614-68518123c95f,ou=authorities,ou=ca,o=ipaca
objectClass: authority
objectClass: top
cn: 008a4ded-fd4b-46fe-8614-68518123c95f
authorityID: 008a4ded-fd4b-46fe-8614-68518123c95f
authorityKeyNickname: caSigningCert cert-pki-ca
authorityEnabled: TRUE
authorityDN: CN=IPA.LOCAL CA 2017.11.09
description: Host authority

There are now two authority entries when there should be one. During startup, Dogtag makes sure it has an authority entry for the main ("host") CA. It compares the Subject DN from the signing certificate in its NSSDB to the authority entries. If it doesn’t find a match it creates a new entry, and that’s what happened here.

The resolution is straightforward:

  1. Stop Dogtag
  2. Update the authorityDN and authoritySerial attributes of the original host authority entry.
  3. Delete the new host authority entry.
  4. Restart Dogtag.

Now the previous ldapsearch returns one entry, with the original authority ID and correct attribute values:

[f27-2:~] ftweedal% ldapsearch -LLL \
    -D 'cn=directory manager' -w DM_PASSWORD \
    -b 'ou=authorities,ou=ca,o=ipaca' -s one
dn: cn=cdbfeb5a-64d2-4141-98d2-98c005802fc1,ou=authorities,ou=ca,o=ipaca
authoritySerial: 11
authorityDN: CN=IPA.LOCAL CA 2017.11.09
objectClass: authority
objectClass: top
cn: cdbfeb5a-64d2-4141-98d2-98c005802fc1
authorityID: cdbfeb5a-64d2-4141-98d2-98c005802fc1
authorityKeyNickname: caSigningCert cert-pki-ca
authorityEnabled: TRUE
description: Host authority

And the operations that were failing before (e.g. ipa ca-show ipa) now succeed. So we’ve confirmed, or restored, the basic functionality on this server.

LDAP certificate stores

There are two LDAP certificate stores in FreeIPA. The first is cn=ca_renewal,cn=ipa,cn=etc,{basedn}. It is only used for replicating Dogtag CA and system certificates from the CA renewal master to CA replicas. The dogtag-ipa-ca-renew-agent Certmonger helper should update the cn=caSigningCert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,{basedn} entry after renewing the CA certificate. A quick ldapsearch shows that this succeeded, so there is nothing else to do here.

The other certificate store is cn=certificates,cn=ipa,cn=etc,{basedn}. This store contains trusted CA certificates. FreeIPA clients and servers retrieve certificates from this directory when updating their certificate trust stores. Certificates are stored in this container with a cn based on the Subject DN, except for the IPA CA which is stored with cn={REALM-NAME} IPA CA. (In my case, this is cn=IPA.LOCAL IPA CA.)

We discovered the failure to update this certificate store earlier (in the Certmonger journal). Now we must fix it up. We still want to trust certificates with the old Issuer DN, otherwise we would have to reissue all of them. So we need to keep the old CA certificate in the store, alongside the new.

The process to fix up the certificate store is:

  1. Export the new CA certificate from the Dogtag NSSDB to a file:

    [root@f27-2 ~]# certutil -d /etc/pki/pki-tomcat/alias \
       -L -a -n 'caSigningCert cert-pki-ca' > new-ca.crt
  2. Add the new CA certificate to the certificate store:

    [root@f27-2 ~]# ipa-cacert-manage install new-ca.crt
    Installing CA certificate, please wait
    CA certificate successfully installed
    The ipa-cacert-manage command was successful
  3. Rename (modrdn) the existing cn={REALM-NAME} IPA CA entry. The new cn RDN is based on the old CA Subject DN.
  4. Rename the new CA certificate entry. The current cn is the new Subject DN. Rename it to cn={REALM-NAME} IPA CA. I encountered a 389DS attribute uniqueness error when I attempted to do this as a modrdn operation. I’m not sure why it happened. To work around the problem I deleted the entry and added it back with the new cn.

At the end of this procedure the certificate store is as it should be. The CA certificate with new Subject DN is installed as {REALM-NAME} IPA CA and the old CA certificate has been preserved under a different RDN.

Updating certificate databases

The LDAP certificate stores have the new CA certificate. Now we need to update the other certificate databases so that the programs that use them will trust certificates with the new Issuer DN. These databases include:

/etc/ipa/ca.crt

CA trust store used by the IPA framework

/etc/ipa/nssdb

An NSSDB used by FreeIPA

/etc/dirsrv/slapd-{REALM-NAME}

NSSDB used by 389DS

/etc/httpd/alias

NSSDB used by Apache HTTPD

/etc/pki/ca-trust/source/ipa.p11-kit

Adds FreeIPA CA certificates to the system-wide trust store

Run ipa-certupdate to update these databases with the CA certificates from the LDAP CA certificate store:

[root@f27-2 ~]# ipa-certupdate
trying https://f27-2.ipa.local/ipa/json
[try 1]: Forwarding 'schema' to json server 'https://f27-2.ipa.local/ipa/json'
trying https://f27-2.ipa.local/ipa/session/json
[try 1]: Forwarding 'ca_is_enabled/1' to json server 'https://f27-2.ipa.local/ipa/session/json'
[try 1]: Forwarding 'ca_find/1' to json server 'https://f27-2.ipa.local/ipa/session/json'
failed to update IPA.LOCAL IPA CA in /etc/dirsrv/slapd-IPA-LOCAL: Command '/usr/bin/certutil -d /etc/dirsrv/slapd-IPA-LOCAL -A -n IPA.LOCAL IPA CA -t C,, -a -f /etc/dirsrv/slapd-IPA-LOCAL/pwdfile.txt' returned non-zero exit status 255.
failed to update IPA.LOCAL IPA CA in /etc/httpd/alias: Command '/usr/bin/certutil -d /etc/httpd/alias -A -n IPA.LOCAL IPA CA -t C,, -a -f /etc/httpd/alias/pwdfile.txt' returned non-zero exit status 255.
failed to update IPA.LOCAL IPA CA in /etc/ipa/nssdb: Command '/usr/bin/certutil -d /etc/ipa/nssdb -A -n IPA.LOCAL IPA CA -t C,, -a -f /etc/ipa/nssdb/pwdfile.txt' returned non-zero exit status 255.
Systemwide CA database updated.
Systemwide CA database updated.
The ipa-certupdate command was successful
[root@f27-2 ~]# echo $?
0

ipa-certupdate reported that it was successful and it exited cleanly. But a glance at the output shows that not all went well. There were failures added the new CA certificate to several NSSDBs. Running one of the commands manually to see the command output doesn’t give us much more information:

[root@f27-2 ~]# certutil -d /etc/ipa/nssdb -f /etc/ipa/nssdb/pwdfile.txt \
    -A -n 'IPA.LOCAL IPA CA' -t C,, -a < ~/new-ca.crt
certutil: could not add certificate to token or database: SEC_ERROR_ADDING_CERT: Error adding certificate to database.
[root@f27-2 ~]# echo $?
255

At this point I guessed that because there is already a certificate stored with the nickname IPA.LOCAL IPA CA, NSS refuses to add a certificate with a different Subject DN under the same nickname. So I will delete the certificates with this nickname from each of the NSSDBs, then try again. For some reason the nickname appeared twice in each NSSDB:

[root@f27-2 ~]# certutil -d /etc/dirsrv/slapd-IPA-LOCAL -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

CN=alt-f27-2.ipa.local,O=Example Organization                u,u,u
CN=CA,O=Example Organization                                 C,,
IPA.LOCAL IPA CA                                             CT,C,C
IPA.LOCAL IPA CA                                             CT,C,C

So for each NSSDB, to delete the certificate I had to execute the certutil command twice. For the 389DS NSSDB, the command was:

[root@f27-2 ~]# certutil -d /etc/httpd/alias -D -n "IPA.LOCAL IPA CA"

The commands for the other NSSDBs were similar. With the problematic certificates removed, I tried running ipa-certupdate again:

[root@f27-2 ~]# ipa-certupdate
trying https://f27-2.ipa.local/ipa/session/json
[try 1]: Forwarding 'ca_is_enabled/1' to json server 'https://f27-2.ipa.local/ipa/session/json'
[try 1]: Forwarding 'ca_find/1' to json server 'https://f27-2.ipa.local/ipa/session/json'
Systemwide CA database updated.
Systemwide CA database updated.
The ipa-certupdate command was successful
[root@f27-2 ~]# echo $?
0

This time there were no errors. certutil shows an IPA.LOCAL IPA CA certificate in the database, and it’s the right certificate:

[root@f27-2 ~]# certutil -d /etc/dirsrv/slapd-IPA-LOCAL -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

CN=alt-f27-2.ipa.local,O=Example Organization                u,u,u
CN=CA,O=Example Organization                                 C,,
CN=Certificate Authority,O=IPA.LOCAL 201711061603            CT,C,C
CN=Certificate Authority,O=IPA.LOCAL 201711061603            CT,C,C
IPA.LOCAL IPA CA                                             C,,
[root@f27-2 ~]# certutil -d /etc/dirsrv/slapd-IPA-LOCAL -L -n 'IPA.LOCAL IPA CA'
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 11 (0xb)
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=Certificate Authority,O=IPA.LOCAL 201711061603"
        Validity:
            Not Before: Thu Nov 09 05:11:12 2017
            Not After : Mon Nov 09 05:11:12 2037
        Subject: "CN=IPA.LOCAL CA 2017.11.09"
        ...

I also confirmed that the old and new CA certificates are present in the /etc/ipa/ca.crt and /etc/pki/ca-trust/source/ipa.p11-kit files. So all the certificate databases now include the new CA certificate.

Renewing the CA certificate (again)

Observe that (in the self-signed FreeIPA CA case) the Issuer DN of the new CA certificate is the Subject DN of the old CA certificate. So we have not quite reached out goal. The original CA certificate was self-signed, so we want a self-signed certificate with the new Subject.

Renewing the CA certificate one more time should result in a self-signed certificate. The current situation is not likely to result in operational issues. So you can consider this an optional step. Anyhow, let’s give it a go:

[root@f27-2 ~]# getcert list -i 20171106062742 | egrep 'status|issuer|subject'
        status: MONITORING
        issuer: CN=Certificate Authority,O=IPA.LOCAL 201711061603
        subject: CN=IPA.LOCAL CA 2017.11.09
[root@f27-2 ~]# getcert resubmit -i 20171106062742
Resubmitting "20171106062742" to "dogtag-ipa-ca-renew-agent".
[root@f27-2 ~]# sleep 5
[root@f27-2 ~]# getcert list -i 20171106062742 | egrep 'status|issuer|subject'
        status: MONITORING
        issuer: CN=IPA.LOCAL CA 2017.11.09
        subject: CN=IPA.LOCAL CA 2017.11.09

Now we have a self-signed CA cert with the new Subject DN. This step has also confirmed that that the certificate issuance is working fine with the new CA subject.

Renewing FreeIPA service certificates

This is another optional step, because we have kept the old CA certificate in the trust store. I want to check that certificate renewals via the FreeIPA framework are working, and this is a fine way to do that.

I’ll renew the HTTP service certificate. This deployment is using an externally-signed HTTP certificate so first I had to track it:

[root@f27-2 ~]# getcert start-tracking \
  -d /etc/httpd/alias -p /etc/httpd/alias/pwdfile.txt \
  -n 'CN=alt-f27-2.ipa.local,O=Example Organization' \
  -c IPA -D 'f27-2.ipa.local' -K 'HTTP/f27-2.ipa.local@IPA.LOCAL'
New tracking request "20171121071700" added.

Then I resubmitted the tracking request. I had to include the -N <SUBJECT> option because the current Subject DN would be rejected by FreeIPA. I also had to include the -K <PRINC_NAME> option due to a bug in Certmonger.

[root@f27-2 ~]# getcert resubmit -i 20171121073608 \
  -N 'CN=f27-2.ipa.local' \
  -K 'HTTP/f27-2.ipa.local@IPA.LOCAL'
Resubmitting "20171121073608" to "IPA".
[root@f27-2 ~]# sleep 5
[root@f27-2 ~]# getcert list -i 20171121073608 \
  | egrep 'status|error|issuer|subject'
      status: MONITORING
      issuer: CN=IPA.LOCAL CA 2017.11.09
      subject: CN=f27-2.ipa.local,O=IPA.LOCAL 201711061603

The renewal succeeded, proving that certificate issuance via the FreeIPA framework is working.

Checking replica health

At this point, I’m happy with the state of the FreeIPA server. But so far I have only dealt with one server in the topology (the renewal master, whose hostname is f27-2.ipa.local). What about other CA replicas?

I log onto f27-1.ipa.local (a CA replica). As a first step I execute ipa-certupdate. This failed in the same was as on the renewal master, and the steps to resolve were the same.

Next I tell Certmonger to renew the CA certificate. This should not renew the CA certificate, only retrieve the certificate from the LDAP certificate store:

[root@f27-1 ~]# getcert list -i 20171106064548 \
  | egrep 'status|error|issuer|subject'
        status: MONITORING
        issuer: CN=Certificate Authority,O=IPA.LOCAL 201711061603
        subject: CN=Certificate Authority,O=IPA.LOCAL 201711061603
[root@f27-1 ~]# getcert resubmit -i 20171106064548
Resubmitting "20171106064548" to "dogtag-ipa-ca-renew-agent".
[root@f27-1 ~]# sleep 30
[root@f27-1 ~]# getcert list -i 20171106064548 | egrep 'status|error|issuer|subject'
        status: MONITORING
        issuer: CN=Certificate Authority,O=IPA.LOCAL 201711061603
        subject: CN=Certificate Authority,O=IPA.LOCAL 201711061603

Well, that did not work. Instead of retrieving the new CA certificate from LDAP, the CA replica issued a new certificate:

[root@f27-1 ~]# certutil -d /etc/pki/pki-tomcat/alias -L \
    -n 'caSigningCert cert-pki-ca'
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 268369927 (0xfff0007)
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=Certificate Authority,O=IPA.LOCAL 201711061603"
        Validity:
            Not Before: Tue Nov 21 08:18:09 2017
            Not After : Fri Nov 06 06:26:21 2037
        Subject: "CN=Certificate Authority,O=IPA.LOCAL 201711061603"
        ...

This was caused by the first problem we faced when renewing the CA certificate with a new Subject DN. Once again, a mismatch between the Subject DN in the CSR and the FreeIPA CA’s Subject DN has confused the renewal helper.

The resolution in this case is to delete all the certificates with nickname caSigningCert cert-pki-ca or IPA.LOCAl IPA CA from Dogtag’s NSSDB then add the new CA certificate to the NSSDB. Then run ipa-certupdate again. Dogtag must not be running during this process:

[root@f27-1 ~]# systemctl stop pki-tomcatd@pki-tomcat
[root@f27-1 ~]# cd /etc/pki/pki-tomcat/alias
[root@f27-1 ~]# certutil -d . -D -n 'caSigningCert cert-pki-ca'
[root@f27-1 ~]# certutil -d . -D -n 'caSigningCert cert-pki-ca'
[root@f27-1 ~]# certutil -d . -D -n 'caSigningCert cert-pki-ca'
[root@f27-1 ~]# certutil -d . -D -n 'caSigningCert cert-pki-ca'
certutil: could not find certificate named "caSigningCert cert-pki-ca": SEC_ERROR_BAD_DATABASE: security library: bad database.
[root@f27-1 ~]# certutil -d . -D -n 'IPA.LOCAL IPA CA'
[root@f27-1 ~]# certutil -d . -D -n 'IPA.LOCAL IPA CA'
[root@f27-1 ~]# certutil -d . -D -n 'IPA.LOCAL IPA CA'
certutil: could not find certificate named "IPA.LOCAL IPA CA": SEC_ERROR_BAD_DATABASE: security library: bad database.
[root@f27-1 ~]# certutil -d . -A \
    -n 'caSigningCert cert-pki-ca' -t 'CT,C,C' < /root/ipa-ca.pem
[root@f27-1 ~]# ipa-certupdate
trying https://f27-1.ipa.local/ipa/json
[try 1]: Forwarding 'ca_is_enabled' to json server 'https://f27-1.ipa.local/ipa/json'
[try 1]: Forwarding 'ca_find/1' to json server 'https://f27-1.ipa.local/ipa/json'
Systemwide CA database updated.
Systemwide CA database updated.
The ipa-certupdate command was successful
[root@f27-1 ~]# systemctl start pki-tomcatd@pki-tomcat

Dogtag started without issue and I was able to issue a certificate via the ipa cert-request command on this replica.

Discussion

It took a while and required a lot of manual effort, but I reached the goal of changing the CA Subject DN. The deployment seems to be operational, although my testing was not exhaustive and there may be breakage that I did not find.

One of the goals was to define the process for both self-signed and externally-signed CAs. I did not deal with the externally-signed CA case. This article (and the process of writing it) was long enough without it! But much of the process, and problems encountered, will be the same.

There are some important concerns and caveats to be aware of.

First, CRLs generated after the Subject DN change may be bogus. They will be issued by the new CA but will contain serial numbers of revoked certificates that were issued by the old CA. Such assertions are invalid but not harmful in practice because those serial numbers will never be reused with the new CA. This is an implementation detail of Dogtag and not true in general.

But there is a bigger problem related to CRLs. After the CA name change, the old CA will never issue another CRL. This means that revoked certificates with the old Issuer DN will never again appear on a CRL issued by the old CA. Worse, the Dogtag OCSP responder errors when you query the status of a certificate with the old Issuer DN. In sum, this means that there is no way for Dogtag to revoke a certificate with the old Issuer DN. Because many systems "fail open" in the event of missing or invalid CRLs or OCSP errors, this is a potentially severe security issue.

Changing a FreeIPA installation’s CA Subject DN, whether by the procedure outlined in this post or by any other, is unsupported. If you try to do it and break your installation, we (the FreeIPA team) may try to help you recover, to a point. But we can’t guarantee anything. Here be dragons and all that.

If you think you need to change your CA Subject DN and have not read the previous post on this topic, please go and read it. It proposes some alternatives that, if applicable, avoid the messy process and security issues detailed here. Despite showing you how to change a FreeIPA installation’s CA Subject DN, my advice remains: don’t do it. I hope you will heed it.

Episode 71 - GitHub's Security Scanner

Posted by Open Source Security Podcast on November 21, 2017 03:47 PM
Josh and Kurt talk about GitHub's security scanner and Linus' security email. We clarify the esoteric difference between security bugs and non security bugs.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5967189/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes



Changing a CA’s Subject DN; Part I: Don’t Do That

Posted by Fraser Tweedale on November 20, 2017 06:03 AM

When you deploy an X.509 certificate authority (CA), you choose a Subject Distinguished Name for that CA. It is sometimes abbreviated as Subject DN, Subject Name, SDN or just Subject.

The Subject DN cannot be changed; it is "for life". But sometimes someone wants to change it anyway. In this article I’ll speculate why someone might want to change a CA’s Subject DN, discuss why it is problematic to do so, and propose some alternative approaches.

What is the Subject DN?

A distinguished name (DN) is a sequence of sets of name attribute types and values. Common attribute types include Common Name (CN), Organisation (O), Organisational Unit (OU), Country (C) and so on. DNs are encoded in ASN.1, but have a well defined string representation. Here’s an example CA subject DN:

CN=DigiCert Global Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US

All X.509 certificates contain an Issuer DN field and a Subject DN field. If the same value is used for both issuer and subject, it is a self-signed certificate. When a CA issues a certificate, the Issuer DN on the issued certificate shall be the Subject DN of the CA certificate. This relationship is a "link" in the chain of signatures from some root CA to end entity (or leaf) certificate.

The Subject DN uniquely identifies a CA. It is the CA. A CA can have multiple concurrent certificates, possibly with different public keys and key types. But if the Subject DN is the same, they are just different certificates for a single CA. Corollary: if the Subject DN differs, it is a different CA even if the key is the same.

CA Subject DN in FreeIPA

A standard installation of FreeIPA includes a CA. It can be a root CA or it can be signed by some other CA (e.g. the Active Directory CA of the organisation). As of FreeIPA v4.5 you can specify any CA Subject DN. Earlier versions required the subject to start with CN=Certificate Authority.

If you don’t explicitly specify the subject during installation, it defaults to CN=Certificate Authority, O=EXAMPLE.COM (replace EXAMPLE.COM with the actual realm name).

Why change the CA Subject DN?

Why would someone want to change a CA’s Subject DN? Usually it is because there is some organisational or regulatory requirement for the Subject DN to have a particular form. For whatever reason the Subject DN doesn’t comply, and now they want to bring it into compliance. In the FreeIPA case, we often see that the default CA Subject DN was accepted, only to later realise that a different name is needed.

To be fair, the FreeIPA installer does not prompt for a CA Subject DN but rather uses the default form unless explicitly told otherwise via options. Furthermore, the CA Subject DN is not mentioned in the summary of the installation parameters prior to confirming and proceeding with the installation. And there are the aforementioned restrictions in FreeIPA < v4.5. So in most cases where a FreeIPA administrator wants to change the CA Subject DN, it is not because they chose the wrong one, rather they were not given an opportunity to choose the right one.

Implications of changing the CA Subject DN

In the X.509 data model the Subject DN is the essence of a CA. So what happens if we do change it? There are several areas of concern, and we will look at each in turn.

Certification paths

Normally when you renew a CA certificate, you don’t need to keep the old CA certificates around in your trust stores. If the new CA certificate is within its validity period you can just replace the old certificate, and everything will keep working.

But if you change the Subject DN, you need to keep the old certificate around, because previously issued certificates will bear the old Issuer DN. Conceptually this is not a problem, but many programs and libraries cannot cope with multiple subjects using the same key. In this case the only workaround is to reissue every certificate, with the new Issuer DN. This is a nightmare.

CRLs

A certificate revocation list is a signed list of non-expired certificates that have been revoked. A CRL issuer is either the CA itself, or a trusted delegate. A CRL signing delegate has its own signing key and an X.509 certificate issued by the CA, which asserts that the subject is a CRL issuer. Like certificates, CRLs have an Issuer DN field.

So if the CA’s Subject DN changes, then CRLs issued by that CA must use the new name in the Issuer field. But recall that certificates are uniquely identified by the Issuer DN and Serial (think of this as a composite primary key). So if the CRL issuer changes (or the issuer of the CRL issuer), all the old revocation information is invalid. Now you must maintain two CRLs:

  • One for the old CA Subject. Even after the name change, this CRL may grow as certificates that were issued using the old CA subject are revoked.
  • One for the new CA Subject. It will start off empty.

If a CRL signing delegate is used, there is further complexity. You need two separate CRL signing certificates (one with the old Issuer DN, one with the new), and must

Suffice to say, a lot of CA programs do not handle these scenarios nicely or at all.

OCSP

The Online Certificate Status Protocol is a protocol for checking the revocation status of a single certificate. Like CRLs, OCSP responses may be signed by the issuing CA itself, or a delegate.

As in the CRL delegation case, different OCSP delegates must be used depending on which DN was the Issuer of the certificate whose status is being checked. If performing direct OCSP signing, if identifying the Responder ID by name, then the old or new name would be included depending on the Issuer of the certificate.

Performing the change

Most CA programs do not offer a way to change the Subject DN. This is not surprising, given that the operation just doesn’t fit into X.509 at all, to say nothing of the implementation considerations that arise.

It may be possible to change the CA Subject DN with some manual effort. In a follow-up post I’ll demonstrate how to change the CA Subject DN in a FreeIPA deployment.

Alternative approaches

I have outlined reasons why renaming a CA is a Bad Idea. So what other options are there?

Whether any of the follow options are viable depends on the use case or requirements. They might not be viable. If you have any other ideas about this I would love to have your feedback! So, let’s look at a couple of options.

Do nothing

If you only want to change the CA Subject DN for cosmetic reasons, don’t. Unless there is a clear business or organisational imperative, just accept the way things are. Your efforts would be better spent somewhere else, I promise!

Re-chaining your CA

If there is a requirement for your root CA to have a Subject DN of a particular form, you could create a CA that satisfies the requirement somewhere else (e.g. a separate instance of Dogtag or even a standalone OpenSSL CA). Then you can re-chain your FreeIPA CA up to this new external CA. That is, you renew the CA certificate, but the issuer of the new IPA CA certificate is the new external CA.

The new external CA becomes a trusted root CA, and your FreeIPA infrastructure and clients continue to function as normal. The FreeIPA CA is now an intermediate CA. No certificates need to be reissued, although some server configurations may need to be updated to include the new FreeIPA CA in their certificate chains.

Subordinate CA

If certain end-entity certificates have to be issued by a CA whose Subject DN meets certain requirements, you could create a subordinate CA (or sub-CA for short) with a compliant name. That is, the FreeIPA CA issues an intermediate CA certificate with the desired Subject DN, and that CA issues the leaf certificates.

FreeIPA support Dogtag lightweight sub-CAs as of v4.4 and there are no restrictions on the Subject DN (except uniqueness). Dogtag lightweight CAs live within the same Dogtag instance as the main FreeIPA CA. See ipa help ca for plugin documentation. One major caveat is that CRLs are not yet supported for lightweight CAs (there is an open ticket).

You could also use the FreeIPA CA to issue a CA certificate for some other CA program (possible another deployment of Dogtag or FreeIPA).

Conclusion

In this post I explained what a CA’s Subject DN is, and how it is an integral part of how X.509 works. We discussed some of the conceptual and practical issues that arise when you change a CA’s Subject DN. In particular, path validation, CRLs and OCSP are affected, and a lot of software will break when encountering a "same key, different subject" scenario.

The general recommendation for changing a CA’s subject DN is don’t. But if there is a real business reason why the current subject is unsuitable, we looked at a couple of alternative approaches that could help: re-chaining the CA, and creating sub-CAs.

In my next post we will have an in-depth look how to change a FreeIPA CA’s Subject DN: how to do it, and how to deal with the inevitable breakage.

On Password Management: Firefox Yes, Chrome No

Posted by Adam Young on November 16, 2017 03:51 PM

Summary: Firefox allows me to store passwords locally, encrypted, and password protected. Chrome wants to store passwords on line, which is not acceptable.

A recent, noticeable slowdown in Firefox (since alleviated) cause me to move more of my work over to Chrome. An essential part of this includes logging in to sites that I have password protected. My usual way to make a password is to run

uuidgen -r

And copy the result into the password field.  Since the output looks like this:

df413d28-957d-4d17-88f4-39c08b6eb81c

You can imagine that I don’t want to type that in every single time.

And, no, I did not just give you my password.  From the uuidgen man page:

There are two types of UUIDs which uuidgen can generate: time-based UUIDs and random-based UUIDs. By default uuidgen will generate a random-based UUID if a high-quality random number generator is
present. Otherwise, it will choose a time-based UUID. It is possible to force the generation of one of these two UUID types by using the -r or -t options.

OPTIONS
-r, –random
Generate a random-based UUID. This method creates a UUID consisting mostly of random bits. It requires that the operating system have a high quality random number generator, such as
/dev/random.

When I use Firefox, I store these in my local store.  To be precise, Firefox manages an NSS database.  This is a local, encrypted store that can be password protected.  I make use of that password protection.  That is the essential feature.

My local password to access my NSS Database is long, unique, and took me a long time to memorize, and I do not want it shared on other systems.  It is only used local to my primary work station.  Not a perfect setup, but better than the defaults.

I looked for a comparable feature in Chromium and found this under Manage Passwords:

Auto Sign-in
Automatically sign in to websites using stored credentials. If disabled, you will be asked for confirmation every time before signing in to a website.
Access your passwords from any device at passwords.google.com

So, no.  I do not want to share my password data with a third party.  I do not trust Google any more than I have to.  I do not want to send my password data over the wire for storage.

It is a constant battle to limit exposure.  Social media is a huge part of my life, and I don’t really like that.  I want to keep it as much under my own control as possible, or to split my exposure among different players.  Google owes me nothing more than my Google Play Music, as that is all I have paid them for.  Everything else they use for their own means, and I know and accept that.  But it means I have no reason to trust them, or any other social site.  I do not use off-machine password managers.  If I did, it would be one I installed and ran myself on a system that I paid for.

Not Google.

 

 

todo.txt done

Posted by Adam Young on November 15, 2017 10:51 PM

While I like the functionality of the todo.txt structure, I do not like the fact that done tasks stay in my todo list in perpetuity, and I also don’t want to lose them.  So, I’ve made a simple hack that allows me to move done items to a done folder.  Here’s the code:

#!/bin/sh
awk '/^x/ {print $0}' ~/Dropbox/todo/todo.txt >> ~/Dropbox/todo/done.txt 
awk '!/^x/ {print $0}' ~/Dropbox/todo/todo.txt > ~/Dropbox/todo/todo2.txt
mv ~/Dropbox/todo/todo2.txt ~/Dropbox/todo/todo.txt

 

I call it todo_done.sh.

I copied my original to /tmp/pre in order to test and make sure I have a backup.  After running todo_done.sh I get:

 

$ diff -u /tmp/pre/todo.txt ~/Dropbox/todo/todo.txt
--- /tmp/pre/todo.txt 2017-11-15 17:46:21.794510999 -0500
+++ /home/ayoung/Dropbox/todo/todo.txt 2017-11-15 17:46:24.584515043 -0500
@@ -7,7 +7,6 @@
 2017-10-02 Expenses
 2017-10-04 Containerize hammer
 2017-10-06 Complete steam setup 
-x 2017-10-12 Trrc time resource reduce cost 
 2017-10-12 Whiteboard training 
 2017-10-14 Subscription manager extensions for skis or products? 
 2017-10-15 Workcenter is made up of 4 things: machine, man, method, measures.

and

$ diff -u /tmp/pre/done.txt ~/Dropbox/todo/done.txt 
--- /tmp/pre/done.txt 2017-11-15 17:46:17.914505377 -0500
+++ /home/ayoung/Dropbox/todo/done.txt 2017-11-15 17:46:24.580515037 -0500
@@ -26,3 +26,4 @@
 x 2017-10-19 Drs appt? 
 x 2017-11-02 Letter of Support
 x 2017-11-15 2017-09-27 LinkedIn TJX
+x 2017-10-12 Trrc time resource reduce cost

Different CloudForms Catalogs for Different Groups

Posted by Adam Young on November 15, 2017 02:37 AM

One of the largest value propositions of DevOps is the concept of Self Service provisioning. If you can remove human interaction from resource allocation, you can reduce both the response time and the likelihood of error in configuration. Red Hat CloudForms has a self service feature that allows a user to select from predefined services. You may wish to show different users different catalog items. This might be for security reasons, such as the set of credentials required and provided, or merely to reduce clutter and focus the end user on specific catalog items. Perhaps some items are still undergoing testing and are not ready for general consumption.

Obviously, these predefined services may not match your entire user population.

I’ve been working on setting up a CloudForms instance where members of different groups see different service catalogs. Here is what I did.

Tags are the primary tool used to match up users and their service catalogs. Specifically, A user will only see a catalog item if his group definition matches the Provisioning Scope tag of the Catalog Item. While you can make some catalog items to have a Provisioning Scope of All, you probably want to scope other items down to the target audience.

I have a demonstration setup based on IdM and CloudForms integration. When uses log in to the CloudForms appliance, one of the user groups managed by LDAP will be used to select their CloudForms group. The CloudForms group has a modified Provisioning Scope tag that will be used to select items from the service catalog.

I also have a top level tenant named “North America” that is used to manage the scope of the tags later on.  I won’t talk through setting this up, as most CloudForms deployment have something set as a top level tenant.

I’m not going to go through the steps to create a new catalog item.  There are other tutorials with go through this in detail.

My organization is adding support for statisticians.  Specifically, we need to provide support for VMs that are designed to support a customized version of the R programming environment.  All users that need these systems will be members of the stats group in IdM.  We want to be able to tag these instances with the stats Provisioning Scope as well.  The user is in the cloudusers group as well, which is required to provide access to the CloudForms appliance.

We start by having our sample user log in to the web UI.  This has the side effect of prepopulating the user and group data.  We could do this manually, but this way is less error prone, if a bit more of hassle.

My user currently only has a single item in her service catalog; the PostgreSQL appliance we make available to all developers.  This allows us to have a standard development environment for database work.

Log out and log back in as an administrator.  Here comes the obscure part.

Provisioning Scope tags are limited to set of valid values.  These values are, by default All or EVMGroup-user_self_service.  This second value matches a group with the same name.  In order to add an option, we need to modify the tag category associated with this tag.

  1. As an administrator, on the top right corner of the screen, click on your user name, and select the Configuration option from the dropdown.
  2. Select your region, in my case this is region 1.
  3. Across the top of the screen, you  will see Settings Region 1, and a series of tabs, most of which have the name of your tenant  (those of you that know my long standing issue with this term are probably grinning at my discomfort).  Since my top level tenant is “North America” I have a tab called North America Tags which I select. Select accordingly.
  4. Next to Category select “Provisioning Scope” from the drop down and you can see my existing set of custom tag values for Provisioning Scope.  Click on <New Entry> to add a new value, which I will call stats. I also use stats for the description.
  5. Click the Add button to the right.  See Below.

Now we can edit the newly defined “R Project” service to limit it to this provisioning scope.

  1. Navigate to Services->Catalogs->Catalog Items.
  2. Select the “R Project” Service.
  3. Click on the Policy  dropdown and select “Edit Tags”
  4. Click on the drop down to the right of “Select a customer tag to assign” (it is probably set on “Auto Approve -Max CPU *”) and scroll down to Provisioning Scope.
  5. The dropdown to the right, which defaults to “<Select a Value to Assign”>. Select this and scroll down to the new value.  For me, this is stats.  The new item will be added to the list.
  6. Click the Save button in the lower right of the screen.

Your list should look like this:

Finally, create the association between this provisioning scope and the stats group.

  1. From the dropdown on the top right of the screen that has your username, select Configuration.
  2. Expand the Access Control accordian
  3. Select groups.
  4. From the Configuration dropdown, select “Add a new Group”
  5. Select a Role for the user.  I use EvmRole-user_self_service
  6. Select a Project/Tenant for the user.
  7. Click on the checkbox labeled “Look Up External Authentiation Groups”
  8. A new field appears called “User to Look Up.”  I am going to user the “statuser” I created for this example, and click retrieve.
  9. The dropdown under the LDAP Groups for User is now populated.  I select stats.

To assign the tag for this group:

  1. Scroll down to the bottom of the page
  2. find and expand the “Provisioning Scope” tag
  3. Select “stats”
  4. Click the Add button in the bottom right corner of the page.

See Below.

Now when statuser logs in  to the self service web UI, they see both of the services provided:

 

One Big Caveat that has messed me up a few times:  a user only has one group active at a time.  If a user is a member of two groups, CloudForms will select one of them as the active group.  Services assigned only to the non-active group will not show up in the service catalog.  In my case, I had a group called cloudusers, and since all users are a member of that group, they would only see the Provisioning Scope, and thus the catalog items, for cloudusers, and not the stats group.

The Self Service webUI allows the user to change group to any of the other groups to which they are assigned.

The best option is to try and maintain a one to many relationship between groups and users;  constrain most users to a single group to avoid confusion.

This has been a long post.  The web UI for CloudForms requires a lot of navigation, and the concepts required to get this to work required more explanation than I originally had planned.  As I get more familiar with CloudForms, I’ll try to show how these types of operations can be automated from the command line, converted to Ansible playbooks, and thus checked in to version control.

I’ve also been told that, for simple use cases, it is possible to just put the user groups into separate tenants, and they will see different catalogs.  While that does not allow a single item to be in both catalogs, it is significantly easier to set up.

A Big Thank You to Laurent Domb for editing and corrections.

Episode 70 - The security of Intel ME

Posted by Open Source Security Podcast on November 14, 2017 11:54 PM
Josh and Kurt talk about Intel ME, Equifax salary history, and IoT.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5945821/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes



Creating yubikey SSH and TLS certificates

Posted by William Brown on November 10, 2017 02:00 PM

Creating yubikey SSH and TLS certificates

Recently yubikeys were shown to have a hardware flaw in the way the generated private keys. This affects the use of them to provide PIV identies or SSH keys.

However, you can generate the keys externally, and load them to the key to prevent this issue.

SSH

First, we’ll create a new NSS DB on an airgapped secure machine (with disk encryption or in memory storage!)

certutil -N -d . -f pwdfile.txt

Now into this, we’ll create a self-signed cert valid for 10 years.

certutil -S -f pwdfile.txt -d . -t "C,," -x -n "SSH" -g 2048 -s "cn=william,O=ssh,L=Brisbane,ST=Queensland,C=AU" -v 120

We export this now to PKCS12 for our key to import.

pk12util -o ssh.p12 -d . -k pwdfile.txt -n SSH

Next we import the key and cert to the hardware in slot 9c

yubico-piv-tool -s9c -i ssh.p12 -K PKCS12 -aimport-key -aimport-certificate -k

Finally, we can display the ssh-key from the token.

ssh-keygen -D /usr/lib64/opensc-pkcs11.so -e

Note, we can make this always used by ssh client by adding the following into .ssh/config:

PKCS11Provider /usr/lib64/opensc-pkcs11.so

TLS Identities

The process is almost identical for user certificates.

First, create the request:

certutil -d . -R -a -o user.csr -f pwdfile.txt -g 4096 -Z SHA256 -v 24 \
--keyUsage digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment --nsCertType sslClient --extKeyUsage clientAuth \
-s "CN=username,O=Testing,L=example,ST=Queensland,C=AU"

Once the request is signed, we should have a user.crt back. Import that to our database:

certutil -A -d . -f pwdfile.txt -i user.crt -a -n TLS -t ",,"

Import our CA certificate also. Next export this to p12:

pk12util -o user.p12 -d . -k pwdfile.txt -n TLS

Now import this to the yubikey - remember to use slot 9a this time!

yubico-piv-tool -s9a -i user.p12 -K PKCS12 -aimport-key -aimport-certificate -k

Done!

Changing the X.509 signature algorithm in FreeIPA

Posted by Fraser Tweedale on November 10, 2017 04:10 AM

X.509 certificates are an application of digital signatures for identity verification. TLS uses X.509 to create a chain of trust from a trusted CA to a service certificate. An X.509 certificate binds a public key to a subject by way of a secure and verifiable signature made by a certificate authority (CA).

A signature algorithm has two parts: a public key signing algorithm (determined by the type of the CA’s signing key) and a collision-resistant hash function. The hash function digests the certified data into a small value that is hard to find collision for, which gets signed.

Computers keep getting faster and attacks on cryptography always get better. So over time older algorithms need to be deprecated, and newer algorithms adopted for use with X.509. In the past the MD5 and SHA-1 digests were often used with X.509, but today SHA-256 (a variant of SHA-2) is the most used algorithm. SHA-256 is also the weakest digest accepted by many programs (e.g. web browsers). Stronger variants of SHA-2 are widely supported.

FreeIPA currently uses the sha256WithRSAEncryption signature algorithm by default. Sometimes we get asked about how to use a stronger digest algorithm. In this article I’ll explain how to do that and discuss the motivations and implications.

Implications of changing the digest algorithm

Unlike re-keying or changing the CA’s Subject DN, re-issuing a certificate signed by the same key, but using a different digest, should Just Work. As long as a client knows about the digest algorithm used, it will be able to verify the signature. It’s fine to have a chain of trust that uses a variety of signature algorithms.

Configuring the signature algorithm in FreeIPA

The signature algorithm is configured in each Dogtag certificate profile. Different profiles can use different signature algorithms. The public key signing algorithm depends on the CA’s key type (e.g. RSA) so you can’t change it; you can only change the digest used.

Modifying certificate profiles

Before FreeIPA 4.2 (RHEL 7.2), Dogtag stored certificate profile configurations as flat files. Dogtag 9 stores them in /var/lib/pki-ca/profiles/ca and Dogtag >= 10 stores them in /var/lib/pki/pki-tomcat/ca/profiles/ca. When Dogtag is using file-based profile storage you must modify profiles on all CA replicas for consistent behaviour. After modifying a profile, Dogtag requires a restart to pick up the changes.

As of FreeIPA 4.2, Dogtag uses LDAP-based profile storage. Changes to profiles get replicated among the CA replicas, so you only need to make the change once. Restart is not required. The ipa certprofile plugin provides commands for importing, exporting and modifying certificate profiles.

Because of the variation among versions, I won’t detail the process of modifying profiles. We’ll look at what modifications to make, but skip over how to apply them.

Profile configuration changes

For service certificates, the profile to modify is caIPAserviceCert. If you want to renew the CA signing cert with a different algorithm, modify the caCACert profile. The relevant profile policy components are signingAlgConstraintImpl and signingAlgDefaultImpl. Look for these components in the profile configuration:

policyset.serverCertSet.8.constraint.class_id=signingAlgConstraintImpl
policyset.serverCertSet.8.constraint.name=No Constraint
policyset.serverCertSet.8.constraint.params.signingAlgsAllowed=SHA1withRSA,SHA256withRSA,SHA512withRSA,MD5withRSA,MD2withRSA,SHA1withDSA,SHA1withEC,SHA256withEC,SHA384withEC,SHA512withEC
policyset.serverCertSet.8.default.class_id=signingAlgDefaultImpl
policyset.serverCertSet.8.default.name=Signing Alg
policyset.serverCertSet.8.default.params.signingAlg=-

Update the policyset.<name>.<n>.default.params.signingAlg parameter; replace the - with the desired signing algorithm. (I set it to SHA512withRSA.) Ensure that the algorithm appears in the policyset.<name>.<n>.constraint.params.signingAlgsAllowed parameter (if not, add it).

After applying this change, certificates issued using the modified profile will use the specified algorithm.

Results

After modifying the caIPAserviceCert profile, we can renew the HTTP certificate and see that the new certificate uses SHA512withRSA. Use getcert list to find the Certmonger tracking request ID for this certificate. We find the tracking request in the output:

...
Request ID '20171109075803':
  status: MONITORING
  stuck: no
  key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
  certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
  CA: IPA
  issuer: CN=Certificate Authority,O=IPA.LOCAL
  subject: CN=rhel69-0.ipa.local,O=IPA.LOCAL
  expires: 2019-11-10 07:53:11 UTC
  ...
...

So the tracking request ID is 20171109075803. Now resubmit the request:

[root@rhel69-0 ca]# getcert resubmit -i 20171109075803
Resubmitting "20171109075803" to "IPA".

After a few moments, check the status of the request:

[root@rhel69-0 ca]# getcert list -i 20171109075803
Number of certificates and requests being tracked: 8.
Request ID '20171109075803':
  status: MONITORING
  stuck: no
  key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
  certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
  CA: IPA
  issuer: CN=Certificate Authority,O=IPA.LOCAL
  subject: CN=rhel69-0.ipa.local,O=IPA.LOCAL
  expires: 2019-11-11 00:02:56 UTC
  ...

We can see by the expires field that renewal succeeded. Pretty-printing the certificate shows that it is using the new signature algorithm:

[root@rhel69-0 ca]# certutil -d /etc/httpd/alias -L -n 'Server-Cert'
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12 (0xc)
        Signature Algorithm: PKCS #1 SHA-512 With RSA Encryption
        Issuer: "CN=Certificate Authority,O=IPA.LOCAL"
        Validity:
            Not Before: Fri Nov 10 00:02:56 2017
            Not After : Mon Nov 11 00:02:56 2019
        Subject: "CN=rhel69-0.ipa.local,O=IPA.LOCAL"

It is using SHA-512/RSA. Mission accomplished.

Discussion

In this article I showed how to configure the signing algorithm in a Dogtag certificate profile. Details about how to modify profiles in particular versions of FreeIPA was out of scope.

In the example I modified the default service certificate profile caIPAserviceCert to use SHA512withRSA. Then I renewed the HTTP TLS certificate to confirm that the configuration change had the intended effect. To change the signature algorithm on the FreeIPA CA certificate, you would modify the caCACert profile then renew the CA certificate. This would only work if the FreeIPA CA is self-signed. If it is externally-signed, it is up to the external CA what digest to use.

In FreeIPA version 4.2 and later, we support the addition of custom certificate profiles. If you want to use a different signature algorithm for a specific use case, instead of modifying the default profile (caIPAserviceCert) you might add a new profile.

The default signature digest algorithm in Dogtag is currently SHA-256. This is appropriate for the present time. There are few reasons why you would need to use something else. Usually it is because of an arbitrary security decision imposed on FreeIPA administrators. There are currently no plans to make the default signature algorithm configurable. But you can control the signature algorithm for a self-signed FreeIPA CA certificate via the ipa-server-install --ca-signing-algorithm option.

In the introduction I mentioned that the CA’s key type determines the public key signature algorithm. That was hand-waving; some key types support multiple signature algorithms. For example, RSA keys support two signature algorithms: PKCS #1 v1.5 and RSASSA-PSS. The latter is seldom used in practice.

The SHA-2 family of algorithms (SHA-256, SHA-384 and SHA-512) are the "most modern" digest algorithms standardised for use in X.509 (RFC 4055). The Russian GOST R digest and signature algorithms are also supported (RFC 4491) although support is not widespread. In 2015 NIST published SHA-3 (based on the Keccak sponge construction). The use of SHA-3 in X.509 has not yet been standardised. There was an Internet-Draft in 2017, but it expired. The current cryptanalysis of SHA-2 suggests there is no urgency to move to SHA-3. But it took a long time to move from SHA-1 (which is now insecure for applications requiring collision resistance) to SHA-2. Therefore it would be good to begin efforts to standardise SHA-3 in X.509 and add library/client support as soon as possible.

Episode 69 - Actionable security advice

Posted by Open Source Security Podcast on November 07, 2017 08:13 PM
Josh and Kurt talk about Amazon Key and actionable advice.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5920064/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes



What's the problem with NUMA anyway?

Posted by William Brown on November 06, 2017 02:00 PM

What’s the problem with NUMA anyway?

What is NUMA?

Non-Uniform Memory Architecture is a method of seperating ram and memory management units to be associated with CPU sockets. The reason for this is performance - if multiple sockets shared a MMU, they will cause each other to block, delaying your CPU.

To improve this, each NUMA region has it’s own MMU and RAM associated. If a CPU can access it’s local MMU and RAM, this is very fast, and does not prevent another CPU from accessing it’s own. For example:

CPU 0   <-- QPI --> CPU 1
  |                   |
  v                   v
MMU 0               MMU 1
  |                   |
  v                   v
RAM 1               RAM 2

For example, on the following system, we can see 1 numa region:

# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3
node 0 size: 12188 MB
node 0 free: 458 MB
node distances:
node   0
  0:  10

On this system, we can see two:

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33 34 35
node 0 size: 32733 MB
node 0 free: 245 MB
node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 36 37 38 39 40 41 42 43 44 45 46 47
node 1 size: 32767 MB
node 1 free: 22793 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

This means that on the second system there is 32GB of ram per NUMA region which is accessible, but the system has total 64GB.

The problem

The problem arises when a process running on NUMA region 0 has to access memory from another NUMA region. Because there is no direct connection between CPU 0 and RAM 1, we must communicate with our neighbour CPU 1 to do this for us. IE:

CPU 0 --> CPU 1 --> MMU 1 --> RAM 1

Not only do we pay a time delay price for the QPI communication between CPU 0 and CPU 1, but now CPU 1’s processes are waiting on the MMU 1 because we are retrieving memory on behalf of CPU 0. This is very slow (and can be seen by the node distances in the numactl –hardware output).

Today’s work around

The work around today is to limit your Directory Server instance to a single NUMA region. So for our example above, we would limit the instance to NUMA region 0 or 1, and treat the instance as though it only has access to 32GB of local memory.

It’s possible to run two instances of DS on a single server, pinning them to their own regions and using replication between them to provide synchronisation. You’ll need a load balancer to fix up the TCP port changes, or you need multiple addresses on the system for listening.

The future

In the future, we’ll be adding support for better copy-on-write techniques that allow the cores to better cache content after a QPI negotiation - but we still have to pay the transit cost. We can minimise this as much as possible, but there is no way today to avoid this penalty. To use all your hardware on a single instance, there will always be a NUMA cost somewhere.

The best solution is as above: run an instance per NUMA region, and internally provide replication for them. Perhaps we’ll support an automatic configuration of this in the future.

Episode 68 - Ruining the Internet

Posted by Open Source Security Podcast on November 01, 2017 02:14 PM
Josh and Kurt talk about Facebook listening to your microphone, Google Chrome certificate pinning, CAs, 152 ways to stay safe, and Kubernetes.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5899279/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes



Getting a Virtual Machine’s IP Address from virsh

Posted by Adam Young on October 27, 2017 03:25 PM

Ten Years later, and I finally know how to get virsh to tell me the IP address for a VM.

So, I have a virtual machine image called cfme. To launch it, I run

sudo virsh start cfme

That gets me a running virtual machine. To find the domain ID that it has been assigned:

$ sudo virsh list
 Id Name State
----------------------------------------------------
 1 cfme running

And to Find Its IP Address I use the domifaddr subcommand:

$ sudo virsh domifaddr 1
 Name MAC address Protocol Address
-------------------------------------------------------------------------------
 vnet0 52:54:00:e5:5d:e3 ipv4 192.168.122.98/24

From which I can see that the address is 192.168.122.98

Wish I knew this years ago.  Ah well.  Better late then never.

Thanks to and this blog post for showing me the way:

 

 

In Satellite server, Organization != Organization_ID

Posted by Adam Young on October 25, 2017 04:59 PM

Spent a good chunk of yesterday and some of this morning stumped by a simple mistake I made.  I was trying to use our Satellite server via the Hammer command line tool to launch an instance, and kept getting and error in the content-view section.  I tried to list content views and got the same problem.  Here is a short section from my bash session.

$ hammer content-view list
Couldn't find organization 'DEMO'
$ hammer organization list
 5 | DEMO | DEMO | | DEMO

It turns out I had improperly set the default organization inside my config file.  I had:

$ cat ~/.hammer/defaults.yml 
---
:defaults:
 :organization_id:
 :value: DEMO

 

When I should have had

$ cat ~/.hammer/defaults.yml 
---
:defaults:
 :organization:
 :value: DEMO

Making this change got me moving ahead again.

Episode 67 - Cyber won

Posted by Open Source Security Podcast on October 24, 2017 06:19 PM
Josh and Kurt talk about hacking back, passwords, honeypots, and conspiracies.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5872803/height/90/theme/custom/autoplay/no/autonext/no/thumbnail/yes/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="100%"></iframe>

Show Notes



Where Did That Software Come From?

Posted by Russel Doty on October 20, 2017 05:29 PM

I have an article in the Oct. 9 issue of Military Embedded Systems magazine on software provenance titled Where Did That Software Come From?

Where did the software on your embedded system come from? Can you prove it? Can you safely update systems in the field? Cryptography provides the tools for verifying the integrity and provenance of software and data. There is a process as to how users can verify the source of software, if it was tampered with in transit, and if it was modified after installation.

The article explores how cryptography, especially hashing and code signing, can be use to establish the source and integrity. It examines how source code control systems and automated build systems are a key part of the software provenance story.  (Provenance means “a record of ownership of a work of art or an antique, used as a guide to authenticity or quality.” It is increasingly being applied to software.)

As an interesting side note, the article describes how the git version control system is very similar to a blockchain.


Deliberate Elevation of Privileges

Posted by Adam Young on October 18, 2017 07:31 PM

“Ooops.” — Me, doing something as admin that I didn’t mean to do.

While the sudo mechanism has some warranted criticism, it is still an improvement on doing everything as the root account. The essential addition that sudo provides for the average sys admin is the ability to only grant themselves system admin when they explicitly want it.

I was recently thinking about a FreeIPA based cluster where the users did not realize that they could get admin permissions by adding themselves to the user group admins. One Benefit to the centralized admin account is that a user has to chose to operate as admin to perform the operation. If a hacker gets the users password, they do not get admin. However, the number of attacks and weaknesses in this approach far outweigh the benefits. Multiple people need to know the password, revoking it for one revokes it for everyone, anyone can change the password, locking everyone else out, and so on.

We instead added a few key individuals to the admins group and changed the password on the admin account.

This heightened degree of security supports the audit trail. Now if someone performs and admin operation, we know which user did it. It involves enabling audit on the Directory Server (I need to learn how to do this!).

It got me thinking, though, if there was a mechanism like the sudo approach that we could implement for users to temporarily elevate them to admins status. Something like a short term group membership. The requirements, as I can see are these:

  1. A user has to chose to be admin:  “admin-powers activate!”
  2. A user can downgrade back to non-admin at any point: “admin-powers activate!”
  3. Admin powers wear off.  admin-powers only last an hour
  4. No new password has to be memorized for admin-powers
  5. The mechanism for admin-powers has to be resistant to attack.
    1. customizable enough that someone outside the organization can’t guess what they are.
    2. provide some way to prevent shoulder surfing.

I’m going to provide a straw-man here.

  • A REST API protected via SPNEGO
    • another endpoint with client cert possible, too
  • The REST API is password protected with basic-auth.  This is the group password.
  • The IPA service running the web server has the ability to add anyone that is in the “potentaladmins” group to the “admins” groups”
  • The IPA service also schedules an AT job to remove the user from the group.  If an AT entry already exists, remove the older one, so a user can extend their window.
  • A cron job runs each night to remove anyone from the admin group that does not have a current at job scheduled.

As I said, a strawman, but I think it points in the right direction.  Thoughts?

Episode 66 - Objects in mirror are less terrible than they appear

Posted by Open Source Security Podcast on October 16, 2017 03:59 PM
Josh and Kurt talk about Equifax again, Kaspersky, TLS CAs, coming change, social security numbers, and Minecraft.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5840636/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes



Episode 65 - Will aliens overthrow us before AI?

Posted by Open Source Security Podcast on October 09, 2017 12:47 AM
Josh and Kurt talk about Apple, Equifax, passwords, AI, and aliens.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5819992/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes



Episode 64 - Networks and Dnsmasq and IoT oh my

Posted by Open Source Security Podcast on October 03, 2017 07:18 PM
Josh and Kurt talk about networks, Dnsmasq, IoT, and our coming security dystopian future.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5804301/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes



SELinux blocks loading kernel modules

Posted by Dan Walsh on September 28, 2017 07:12 PM

The kernel has a feature where it will load certain kernel modules for a process, when certain syscalls are made.  For example, loading a kernel module when a process attempts to create a different network socket.  

I wrote a blog on https://medium.com/cri-o explaining how this is probably a bad idea from a containers perspective.  I don't want to allow container processes to trigger modifications of the kernel.  And potentially causing the kernel to load risky modules that could have vulnerabilities in them.  I say, let the Administrator or packagers decide what kernel modules need to be loaded and then make the containers live with what is provided for them.  Here is a link to the blog.

https://medium.com/cri-o/cri-o-has-builtin-selinux-support-6ff45b707cf0

Episode 63 - Shoot, Shovel, and Bury

Posted by Open Source Security Podcast on September 26, 2017 12:52 PM
Josh and Kurt talk about the Equifax breach (again) and what it will mean for all of us. Blueborne comes up, as well as #TrevorForget.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5777841/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes



Measuring security: Part 2 - The cost of doing business

Posted by Josh Bressers on September 25, 2017 12:48 AM
If you've not read my last post on measuring security you probably should. It talks about how to measure the security of things that make money. That post is mostly focused on things like products that directly generate revenue. This time we're going to talk about a category I'm calling the cost of doing business.

The term "cost of doing business" is something I made up so I could group these ideas in some sensible way. At least sensible to me. You probably can't use this with other humans in a discussion, they won't know what you're talking about. If I had a line graph of spending I would put revenue generating on one side, the purse cost centers on the other side. The cost of doing business is somewhere in the middle. These are activities that directly support whatever it is the organization does to make new money. Projects and solutions that don't directly make money themselves but do directly support things being built that make money.

The cost of doing business includes things like compliance, sending staff to meetings, maybe regulatory requirements. Things that don't directly generate revenue but you can't move forward if you don't do these things. There's not a lot of options in many cases. If you don't have PCI compliance, you can't process payments, you can't make any money, and the company won't last long. If you don't attend certain meetings nobody can get any work done. Regulated industry must follow their requirements or the company can often just be shut down. Sometimes there are things we have to do, even if we don't want to do them.

In the next post we'll talk about what I call "infrastructure", these are the things that are seen as cost centers and often a commodity service (like electricity or internet access). I just want to clarify the difference. Infrastructure is something where you have choice or can decide not to do it with a possible negative (or positive) consequence. Infrastructure is what keep the lights on at a bare minimum. Cost of doing business must be done to get yourself to the next step in a project, there is no choice, which changes what we measure and how we measure it.

The Example

Let's pick on PCI compliance as it's pretty easy to understand example. If you don't do this it's quite likely your company won't survive, assuming you need to process card payments. If you're building a new web site that will process payments, you have to get through PCI compliance, there is no choice, and the project cannot move forward until this is complete. The goal now isn't so much measuring the return on an investment as it is being a good steward of the resources given to us. PCI requirements and audits are not cheap. If you are seen as making poor decisions and squandering your resources it's quite likely the business will get grumpy with you.

Compliance and security aren't the same thing. There is some overlap but it must be understood that you can be compliant and still get hacked. The overlap of compliance is a great thing to focus on for measuring what we do. Did your compliance program make you more secure? Can you show how another group used a compliance requirement to make something better? What if something compliance required saved some money on how the network was architected? There are a lot of side benefits to pay attention to. Make sure you note the things that are improvements, even if they aren't necessarily security improvements.

I've seen examples where compliance was used to justify 2 factor authentication (2FA) in an organization, There are few things more powerful than 2FA that you can deploy. Showing compliance helped move an initiative like this forward, and also showing how the number of malicious logs drops substantially is a powerful message. Just turning on 2FA isn't enough. Make sure you show why it's better, how the attacks are slowed or stopped. Make sure you can show there were few issues for users (the people who struggle will complain loudly). If there is massive disruption for your users, figure out why you didn't know this would happen, someone screwed something up that means. It's important to measure the good and the bad. We rarely measure failure which is a problem. Nobody has a 100% success rate, learn from your failure.

What about attending a meeting or industry conference? Do you just go, file the expense report, and do nothing? That sounds like a waste of time and money. Make sure you have concrete actions. Write down what happened, why it was important you were there, how you made the situation better, and what you're going to do next. How did the meeting move your project forward? Did you learn something new, or make some plans that will help in the future? Make sure the person paying your bills sees this. Make them happy to be providing you the means to keep the business moving forward.

The Cost

The very first step we have to consider when we want to measure what we're doing is to do your homework and understand cost. Not just upfront cost but cost of machines, disk, people, services, anything you need to keep the business moving forward. If there are certain requirements needed for a solution make sure you understand and document it. If a certain piece of software or service has to be used show why. Show what part of the business can function because of the cost you're providing. Remember this is going to be specific requirements you can't escape. These are not commodity services and solutions. And of course the goal is to move forward.

If you inherit an existing solution take a good look at everything, make sure you know exactly what the resource cost of the solution is. The goal here isn't always to show a return on investment, but to show that the current solution makes sense. Just because something costs less money doesn't mean it's cheaper. If your cut rate services will put the project in jeopardy you're going to be in trouble someday. Be able to show this is a real threat. It's possible a decision will be made to take on this threat, but that's not always your choice. Always be able to answer the questions "if we do this what happens" and "if we don't do this what happens".

Conclusion
This topic is tricky. I keep thinking about it and even as I wrote this post it changed quite a lot from what I started to write. If you have something that makes money it's easy to justify investment. If you have something that's a pure cost center it's easy to minimize cost. This middle ground is tricky. How do you show value for something you have to do but isn't directly generating revenue? If you work for a forward looking business you probably won't have to spend a ton of time getting these projects funded. Growing companies understand the cost of doing business.

I have seen some companies that aren't growing as quickly fail to see value in the cost of doing business. There's nothing wrong with this sometimes, but as a security leader your job is to make your leadership understand what isn't happening because of this lack of investment. Sometimes if you keep a project limping along, barely alive, you end up causing a great deal of damage to the project and your staff. If leadership won't fund something, it means they don't view it as important and neither should you. If you think it is important, you need to sell it to your leadership. Sometimes you can't and won't win though, and then you have to be willing to let it go.

A minor addition for todo.txt

Posted by Adam Young on September 23, 2017 05:57 PM

I had a simple todo list I managed using shell scripts and git, but I wanted something for the Cell phone. The todo.txt application fills that need now. But I was able to reuse something from my old approach to make it a little more command line friendly.

I want to be able to add a todo from the command line without thinking syntax or dates or anything. Here is the code:

 

#!/bin/sh

TODOLINE=`date +%Y-%m-%d`
until [ -z "$1" ] # Until all parameters used up . . .
do
 TODOLINE=$TODOLINE" $1 "
 shift
done

pushd $HOME/Dropbox/todo

echo $TODOLINE >> todo.txt

I have it in  ~/bin/add_todo~/bin is a part of $PATH.    To remember something, I type:

add_todo Meet with Jim on Monday

And In the corresponding document I get:

2017-09-23 Meet with Jim on Monday

The thing I like about the script is that it treats each command line argument as part of the message;  no quotes are required.

Why all the DAC_READ_SEARCH AVC messages?

Posted by Dan Walsh on September 22, 2017 04:26 PM

If you followed SELinux policy bugs being reported in bugzilla you might have noticed a spike in messages about random domains being denied DAC_READ_SEARCH.

Let's quickly look at what the DAC_READ_SEARCH capability is.  In Linux the power of "root" was broken down into 64 distinct capabilities.  Things like being able to load kernel modules or bind to ports less then 1024.  Well DAC_READ_SEARCH is one of these.

DAC stands for Discretionary Access Control, which is what most people understand as standard Linux permissions, Every process has owner/group.  All file system objects are  assigned owner, group and permission flags.  DAC_READ_SEARCH allows a privilege process to ignore parts of DAC for read and search.

man capabilities

...

       CAP_DAC_READ_SEARCH

              * Bypass file read permission checks and directory read and execute permission checks;

There is another CAPABILITY called DAC_OVERRIDE

       CAP_DAC_OVERRIDE

              Bypass file read, write, and execute permission checks.

As you can see DAC_OVERRIDE is more powerful then DAC_READ_SEARCH, in that it can write and execute content ignoring DAC rules, as opposed to just reading the content.

Well why did we suddenly see a spike in confined domains needing DAC_READ_SEARCH?

Nothing in policy changed but the kernel changed.  Basically the kernel was made a little more secure.  Lets look at an example.  There is a small program called unix_chkpwd (checkpwd_t) which you end up executing when you log into the system.  This program reads /etc/shadow.  On Fedora/RHEL/CentOS and probably other GNU/Linux systems, /etc/shadow has 0000 Mode.  This means NO processes on the system, even if running as root (UID=0), are allowed to read/write /etc/shadow, unless they have a DAC Capability.

Well as policy evolved we saw that chckpwd_t needed to read /etc/shadow, it generated an DAC_OVERIDE AVC, so a policy developer gave that SELinux allow rule to checkpwd_t.  And for years things worked properly but then the kernel changed...

If a process tried to read /etc/shadow, it would be allowed if it had either DAC_OVERRIDE or DAC_READ_SEARCH. 

  

Older kernel's had pseudo code like

If DAC_OVERRIDE or DAC_READ_SEARCH:

         Read a file with 0000 mode.

New Kernel switched to:

If DAC_READ_SEARCH or DAC_OVERRIDE

         Read a file with 0000 mode.

Since the chkpwd_t had DAC_OVERRIDE in the older kernels, it never checked DAC_READ_SEARCH and therefore DAC_READ_SEARCH was never added to policy.  Now it is checking DAC_READ_SEARCH first so we see the AVC being generated even though the final access was allowed.

This has generated a lot of noise for people from the SELinux logs, but really nothing got denied.  After the AVC was generated the access was still allowed.

The policy package maintainers have been updating all of the policy to allow these new DAC_READ_SEARCH, and I have suggested to them to start dropping alot of DAC_OVERRIDE policy from domains, since a lot of them including chkpwd_t don't need to write /etc/shadow, so they should be able to get by with only reading it.  This should eventually make SELinux policy more secure.



Learn about binary hardening with hardening-check

Posted by Jason Callaway on September 22, 2017 03:38 PM

In my last post, I wrote about using Radare2 to measure cyclomatic complexity, giving us a metric of the number of “moving parts” in a binary. From a risk quantification point of view, this is important because we assume that a more complicated program will be easier to get to misbehave.

But there are a lot of binaries in a Linux install. Let’s do a find against / and see for ourselves. Again, I find myself on my Macbook, so let’s do this from a docker container because it’ll be quicker than a VM. This assumes you already have Docker installed and configured.

mb:~ jason$ docker run --rm -ti fedora bash
[root@0600e63539d7 /]# dnf install -q -y file findutils
[root@0600e63539d7 /]# find / -exec file {} \; | grep -i elf | wc -l
1113

Over a thousand, even in a small docker image! Now imagine you’re a security researcher, or maybe a 1337 hax0r, and you want to find a new vulnerability in one of those binaries. Which ones do you try to attack?

In this scenario, we need to be selective about which binaries we want to target, because fuzzing is time consuming. So which ones will give us the best bang for our buck, so to speak? We want to look for low-hanging fruit, and in order to do that we need to identify binaries that are, basically, poorly built.

This process — looking at binaries, scoring them in terms of how much effort it would take to successfully fuzz them and find a new 0-day — is what the Cyber-ITL is all about. And this is what we want to duplicate in the open source world with the Fedora Cyber Test Lab.

There are a number of mechanisms that have been built into Linux over the years to frustrate this process. I find a good way to learn about those mechanisms is to look at the code for a tool called hardening-check.

hardening-check was written by an a kernel security engineer at Google, Kees Cook, who, from the look of his Twitter profile, is a bit of a Stand Alone Complex fan. This tool came out around 2009, during which time Kees was working as an Ubuntu security engineer. Since then hardening-check has been picked up by other distros, and is just super-handy.

(In the next several posts, we’ll be referring to the hardening-check Perl source code, which is version controlled here.)

First, let’s install and run the tool to get a feel for it. We’ll also install gcc for the next example.

mb:~ jason$ docker run --rm -ti fedora bash
[root@59c1a1a14181 /]# dnf install -q -y hardening-check gcc
[root@59c1a1a14181 /]# hardening-check /bin/ls
/bin/ls:
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: yes (some protected functions found)
 Read-only relocations: yes
 Immediate binding: yes

Alright, that looks pretty good. Now let’s write another hello world program, but this time we’re going to deliberately use a notoriously unsafe function.

#include <stdio.h>

int main() {
  char str1[12] = "hello world";
  char str2[12];
  sprintf(str2, "%s", str1);
  return 0;
}

For now, we compile it without any extra flags, then run hardening check.

[root@59c1a1a14181 /]# gcc hello_world.c
[root@59c1a1a14181 /]# hardening-check a.out
./a.out:
 Position Independent Executable: no, normal executable!
 Stack protected: no, not found!
 Fortify Source functions: no, only unprotected functions found!
 Read-only relocations: yes
 Immediate binding: no, not found!

Not so great. But we can ask gcc to help us out.

[root@59c1a1a14181 /]# gcc -D_FORTIFY_SOURCE=2 -O2 -fpic -pie -z now -fstack-protector-all hello_world.c
[root@59c1a1a14181 /]# hardening-check a.out
./a.out:
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: yes
 Read-only relocations: yes
 Immediate binding: yes

That’s better. Now our silly little program is much harder to leverage in an attack.

This brings up a lot of questions, though. Why isn’t every binary compiled with those flags? Why don’t we just tweak gcc so it always applies those protections by default? For that matter, what do all those checks mean? Can they be defeated by attackers?

Stay tuned as we dig into each of those questions, and explore how they apply to our Cyber Test Lab.

U2F Protocol Overview

Posted by Nathaniel McCallum on September 19, 2017 04:04 PM

U2F Protocol Overview

This document serves as an overview of the U2F multi-factor authentication specifications as published by the FIDO Alliance. It does not replace the offical specifications but seeks to provide a quick run-through of how the protocol is built. U2F basically works via two main commands: register and authenticate.

When a server wants to use a U2F device for authentication, it asks the host (usually the client computer) to issue a register command to the device. This command creates a new signing key pair and the public key is returned to the server along with an opaque handle (for later reference) and an attestation certifciate (proving trust of the U2F device to the server).

Then, when it wishes to authenticate the user, the server asks the host to issue an authenticate command to the device. The device then signs a nonce from the server and returns the signature. This proves that the user is in posession of the key. It also returns a counter, proving to the server that the device has not been cloned.

U2F provides a high degree of security and a user-friendly workflow. It is supported by Chrome (natively) and Firefox (via an extension).

U2F Layers

The U2F Protocol happens across two layers: 1. The Transport Layer 2. The APDU Layer

All multi-byte integers are in network byte order (big endian).

The Transport Layer

The Transport Layer is responsible for breaking up packets into frames for transport from the computer to the U2F device over HID or Bluetooth. U2F-over-NFC does not use a transport layer. A full packet looks like this:

U2F Packet Field Size (Bytes)
Channel Identifier (CID) 4 (HID Only)
Command (CMD) 1
Data Length 2
Data 0+

The Channel Identifier (CID) is only used for the HID transport and is completely omitted over Bluetooth.

The CMD byte always has the first bit set and contains the number assigned to this particular command. Most commonly, this command will be the MSG command.

If the packet size is less than the maximum transmission unit (MTU), then it is sent directly. Otherwise, the packet is broken up into MTU-sized frames. The first frame is just the beginning of the packet truncated to fit in the frame. Subsequent frames are simply the additional fragments of the packet with a prefix SEQ byte (and, over HID, the CID). The SEQ byte always has the first bit unset, which allows you to distinguish continuation frames from the start of a new packet. It begins with a zero value and increments for each frame sent. A subsequent frame looks like this:

U2F Frame Field Size (Bytes)
Channel Identifier (CID) 4 (HID Only)
Sequence (SEQ) 1
Data 1+

Commands

Below you will find a survey of the most common Transport Layer commands. For the full details on all available commands, please see the U2F specifications.

MSG (CMD = 0x83)

The MSG command contains an APDU request or reply and represents the next layer in communication. This will be discussed in the APDU Layer section below.

PING (CMD = 0x81)

The PING command simply returns the packet to the sender without modification.

ERROR (CMD = 0x84)

The ERROR command is only ever sent as a response. The data contains a one-byte error code. This is only used to communicate errors at the Transport Layer. The APDU Layer has its own error handling.

INIT (CMD = 0x86, HID Only)

The INIT command is only used for HID. Its purpose is to allow the U2F device to assign a unique CID and to get some information about the hardware device. The request contains an 8-byte nonce. The reply data looks like this (fields are 1 byte unless otherwise noted):

U2F INIT Packet Data Field Size (Bytes)
Nonce 8
Channel Identifier (CID) 4
Protocol Version 1
Major Device Version 1
Minor Device Version 1
Build Device Version 1
Capability Flags 1

The request and reply are sent over the reserved broadcast channel (CID = 0xffffffff).

KEEPALIVE (CMD = 0x82, Bluetooth Only)

The KEEPALIVE command is only ever sent from Bluetooth U2F devices to the host. Its purpose is to identify to the host that the request is still being processed or is waiting for user confirmation. Its data contains a single byte indicating the reason for the delayed response.

The APDU Layer

When a MSG command packet is received by the device, the data contains an APDU request. Likewise, the host expects an APDU reply. I will not describe the APDU format in detail. Let it suffice to say that APDU is similar to the transport layer’s packet. It can represent a command (via the CLA and INS bytes), parameters to that command (the P1 and P2 bytes) and a miscelaneous data payload. Likewise, the APDU Layer has its own error handling for errors that occur within this layer.

Commands

VERSION (CLA = 0x00, INS = 0x03)

The VERSION command takes no input parameters and no input data. It simply replies with the version of the U2F Raw Message Format supported. There is currently only one version, which is represented as the UTF-8 encoding of the string U2F_V2 without a NULL terminator.

REGISTER (CLA = 0x00, INS = 0x01)

The REGISTER command takes no input parameters. The input data is in the following format:

U2F REGISTER Request Field Size (Bytes)
Challenge Parameter 32
Application Parameter 32

The U2F device will create a new P-256 signing keypair used for future authentications and will return back the following data:

U2F REGISTER Reply Field Size (Bytes)
Reserved (0x05) 1
Public Key 65
Key Handle Length 1
Key Handle 0-255
Attestation Certificate X.509 (DER)
ECDSA (P-256) Signature 71-73

The Public Key field contains the public key of the newly-created signing key pair. The Key Handle is an opaque field identifying the newly created key pair. The final two fields need some additional explanation.

Remote servers need a way to trust the U2F device. This comes in the form of an attestation key pair which may or may not be different from the newly-created signing key pair. The attestation key pair signs the following payload and inserts the signature into the reply (above):

U2F REGISTER Signature Field Size (Bytes)
Reserved (0x05) 1
Application Parameter 32
Challenge Parameter 32
Key Handle 0-255
Public Key 65

If the remote server trusts the attestation certifciate, then it can also trust the newly-generated signing key pair and key handle by validating the attestation signature. Usually, the server will trust the attestation certificate via an intermediary certification authority.

AUTHENTICATE (CLA = 0x00, INS = 0x02)

The AUTHENTICATE command takes a parameter (in P1) and input data in the following format:

U2F AUTHENTICATE Request Field Size (Bytes)
Challenge Parameter 32
Application Parameter 32
Key Handle Length 1
Key Handle 0-255

If P1 == 0x07 (check only), the device only verifies if the Application Parameter and Key Handle are valid for this U2F device. Otherwise, if P1 == 0x03 (enforce user presence and sign) or P1 == 0x08 (don’t enforce user presence and sign), the U2F device then signs a payload in the following format:

U2F AUTHENTICATE Signature Field Size (Bytes)
Application Parameter 32
User Presence Flags 1
Counter 4
Challenge Parameter 32

The counter is used to detect key theft from the U2F device. It may be global or per key handle. This payload is signed by the signing key pair.

Then a response is returned in the following format:

U2F AUTHENTICATE Reply Field Size (Bytes)
User Presence Flags 1
Counter 4
ECDSA (P-256) Signature 71-73

The User Presence field contains flags identifying the level of user presence testing.

Measuring cyclomatic complexity with Radare2

Posted by Jason Callaway on September 19, 2017 11:50 AM

Cyclomatic complexity is a metric that’s used to measure the complexity of a program. It’s one of many binary metrics tracked by the Cyber-ITL, and calculating it is a good first step in repeating their results.

Radare2 makes this super easy, and with r2pipe we can start building out our Python SDK for the Fedora Cyber Test Lab’s open source implementation of the Cyber-ITL’s approach.

macOS Environment Setup

It probably makes more sense to do this work on my Fedora box, but I happen to be on my Macbook. So we’ll use macOS as our environment for this post, which is, honestly, way more difficult.

First, let’s set up Homebrew. Note that in order to do so cleanly, we want to first give my user full control over /usr/local. I prefer to use facls to do this than changing ownership, which seems clumsy. Be sure to replace “jason” with your username.

sudo chmod -R +a "jason allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity,chown,file_inherit,directory_inherit" /usr/local

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

I also like to use the Homebrew version of Python, so we’ll install that as well.

brew install python virtualenv radare2
virtualenv -p /usr/local/bin/python ENV
source ENV/bin/activate
pip install r2pipe

We’re setting up a virtual environment here so we can isolate our installed libraries, but also to ensure we don’t run into PYTHONPATH issues, which is easy to do on macOS with multiple Python interpreters installed.

Simple analysis with r2

Radare2 (pronounced like “radar”) is a fantastic tool and, to be honest, I’ve only scratched its surface. Watching an experienced reverse engineer use r2 is like watching Michelangelo paint. Ok, I’ve never seen that, but I assume it was hella cool.

So let’s write a simple program and analyze it with r2. This assumes you have Xcode Command Line Tools installed, which is from where we’re getting gcc.

#include <stdio.h>

int main() {
  printf("hello world\n");
  return 0;
}

That should look familiar to everybody. Now let’s compile it, analyze it, and ask r2 to calculate its cyclomatic complexity.

(ENV) mb:~ jason$ gcc hello_world.c
(ENV) mb:~ jason$ r2 a.out
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
 -- For a full list of commands see `strings /dev/urandom`
[0x100000f60]> aaa
[x] Analyze all flags starting with sym. and entry0 (aa)
[x] Analyze len bytes of instructions for references (aar)
[x] Analyze function calls (aac)
[ ] [*] Use -AA or aaaa to perform additional experimental analysis.
[x] Constructing a function name for fcn.* and sym.func.* functions (aan))
[0x100000f60]> afcc
1
[0x100000f60]> q

Not super-interesting, there’s just a single path the code can take, so when we use the afcc command, or “analyze functions cyclomatic complexity,” we get back 1.

Using the formula the Wikipedia article, we can sanity check that result.

M = E − N + 2P,
where
E = the number of edges of the graph.
N = the number of nodes of the graph.
P = the number of connected components.

We get

M = 0 – 1 + 2 * 1
M = 1

For our purposes, P will almost always be 2, so we can treat that like a constant.

Ok, let’s add some more complexity.

#include <stdio.h>

int main() {
	int a = 1;
	if( a == 1 ){
		printf("hello world\n");
	}
	return 0;
}

Compile it and analyze it.

(ENV) mb:~ jason$ gcc hello_world2.c
(ENV) mb:~ jason$ r2 a.out
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
 -- Rename a function using the 'afr <newname> @ <offset>' command.
[0x100000f50]> aaa
[x] Analyze all flags starting with sym. and entry0 (aa)
[x] Analyze len bytes of instructions for references (aar)
[x] Analyze function calls (aac)
[ ] [*] Use -AA or aaaa to perform additional experimental analysis.
[x] Constructing a function name for fcn.* and sym.func.* functions (aan))
[0x100000f50]> afcc
2
[0x100000f50]> agv
[0x100000f50]> q

Ok, with a conditional in our code we got the complexity to go up. But let’s use the command “agv” or “analyze graph web/png” to get a nice graphic representation of the function graph.

Now let’s use our formula, M = E – N + 2. Three edges, three nodes.

M = 3 – 3 + 2
M = 2

So that tracks. Now once more with an extra conditional statement.

#include <stdio.h>

int main() {
	int a = 1;
	if( a == 1 ){
		printf("hello world\n");
	}
	if(a == 0){
		printf("goodbye world\n");
	}
	return 0; 
}

Lather, rinse, repeat.

(ENV) mb:~ jason$ gcc hello_world3.c
(ENV) mb:~ jason$ r2 a.out
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
 -- Use '-e bin.strings=false' to disable automatic string search when loading the binary.
[0x100000f20]> aaa
[x] Analyze all flags starting with sym. and entry0 (aa)
[x] Analyze len bytes of instructions for references (aar)
[x] Analyze function calls (aac)
[ ] [*] Use -AA or aaaa to perform additional experimental analysis.
[x] Constructing a function name for fcn.* and sym.func.* functions (aan))
[0x100000f20]> afcc
3
[0x100000f20]> agv
[0x100000f20]> q

Verifying once again, six edges, five nodes:

M = 6 – 5 + 2
M = 3

You get the idea. These are really trivial examples, and when you ask r2 to analyze complex binaries, it can take a really long time. For fun, try the docker runtime, and see what a pain it is to deal with statically linked Go binaries.

Now let’s use r2pipe to do this from Python.

#!/usr/bin/env python

import os
import r2pipe
import sys

r2 = r2pipe.open(sys.argv[1])
r2.cmd("aaa")
cc = r2.cmdj("afcc")
print cc

And now we can do it from Python!

(ENV) mb:~ jason$ python r2cc.py a.out
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
3

BTW, those errors are a known issue and you can ignore them on macOS for now.

Just for fun, let’s look at some macOS binaries.

(ENV) mb:~ jason$ python r2cc.py /bin/ls
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
36
(ENV) mb:~ jason$ python r2cc.py /usr/sbin/chown
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
31
(ENV) mb:~ jason$ python r2cc.py /usr/sbin/postfix
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
24
(ENV) mb:~ jason$ python r2cc.py /bin/bash
syntax error: error in error handling
syntax error: error in error handling
syntax error: error in error handling
177

Crazy that ls and chown are more complex that Postfix! And BASH is far more complex than either. Think of this value as a representation of the number of moving parts in a machine. The more complex it is, the more likely it’ll break.

But what about the quality of those parts in the machine? A complex but well-engineered machine will work better than a simple piece of junk. And so far, we’re only examining the machine while it’s turned off. That’s static analysis. We also need to watch the machine work, which is dynamic analysis. The Cyber-ITL has been doing both.

Stay tuned as we follow them down the quantitative analysis rabbit hole.

Introducing the Fedora Red Team

Posted by Jason Callaway on September 18, 2017 01:15 AM

Last week my colleagues and I started a new Special Interest Group under the Fedora Project: the Fedora Red Team.

The Fedora Red Team’s goal is to become Red Hat’s upstream cybersecurity community. But what does that actually mean?

“Cyber” is a fascinating term with a colorful history. I’ve written about it before, but the punchline is that we owe its ubiquity to William Gibson’s Neuromancer and an obscure US Government paper from the late 90s, referred to as The Marsh Report.

The context of The Marsh Report seems to have stuck — cyber has come to refer to the high-stakes interplay of offensive and defensive infosec, often at nation-state scale. If you want to get ontological, you could say “cyber” includes the three sub-domains of Computer Network Operations (CNO) — Computer Network Defense, Computer Network Exploitation, and Computer Network Attack.

So why a Fedora Red Team? My colleagues and I needed a place to work on offensive tooling, exploit curation, standards, and reference architectures — but we wanted to do it “the open source way.” A Fedora SIG gives us a community place to fail-fast these projects, a few of which I’ll mention here.

The Enterprise Linux Exploit Mapper (ELEM): as Mike Bursell wrote in his blog, many system administrators find themselves unable to patch. The CVE scoring system helps admins decide when to really push for patching, but many CVE descriptions contain language like “this vulnerability may allow an attacker to execute arbitrary code.” And there’s a reason for that — many vulnerabilities don’t have workable POCs. But what about the ones that do? ELEM makes it easy to map vulnerabilities on a local system to known exploits out in the wild. From a defensive perspective it creates a sort of super-criticality for admins so they can say to their management, “somebody can download this exploit and root our box right this minute.” A tool like this has good offensive value as well, and could save pentesters time doing these mappings manually.

The Fedora Cyber Test Lab (FCTL): I’ve written before about the Cyber-ITL and how important it is to the future of infosec. My only complaint is that their TTPs are not open source, and are not repeatable. So let’s fix that. This project will be an open source, automated, fully repeatable project for quantifying risk at a per-binary level. It will focus at first on RHEL, CentOS, and Fedora, but we’d love help from the community with adding other OS targets. I have a rudimentary first version ready to push to GitHub, which I’ll be blogging about in the coming days.

Penetration Testing Execution Standard (PTES): I’ve written before about how much I love PTES. In my mind, it’s a really important component of the art of pentesting. How can you tell a client that you’ve tested their security according to industry best practices without a yardstick by which to measure those practices? Without such a standard, pentesting relies too much on personal bona fides and flashy marketing. A standard like PTES can fix that. The only problem is that it hasn’t really been updated since 2014. Last summer, I rejiggered the wiki markup into ReStructured Text and put it on Read the Docs, which makes it easier to build community participation. The project is ready to be resurrected, and I hope that we’ll be able to work with the original team. But worst-case, we can fork it and go from there. This isn’t necessarily bad, and the PTES founders may have their reasons for wanting to let the project stay as it is. The Red Team SIG should know by early October which direction we’ll be taking.

I’m excited about this SIG, and will be haunting #fedora-security on Freenode IRC, as well as the security@lists.fedoraproject.org list. Every first Friday of the month we’ll be having meetings in IRC at 1400 UTC. Please join the conversation, or send us your pull requests for our GitHub projects.

Also, if you’re in Northern Virginia on 3 October 2017, we’ll be presenting ELEM at Defense in Depth. Register here and geek it up with us face-to-face!

Disclaimer: while I am talking about work stuff, this is my personal blog, and these views are my own and not my employer’s.

Episode 62 - All about the Equifax hack

Posted by Open Source Security Podcast on September 11, 2017 02:35 PM
Josh and Kurt talk about the Equifax breach and what it will mean for all of us.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5728237/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes



Measuring security: Part 1 - Things that make money

Posted by Josh Bressers on September 11, 2017 12:48 PM
If you read my previous post on measuring security, you know I broke measuring into three categories. I have no good reason to do this other than it's something that made sense to me. There are without question better ways to split these apart, I'm sure there is even overlap, but that's not important. What actually matters is to start a discussion on measuring what we do. The first topic is about measuring security that directly adds to revenue such as a product or service.

Revenue
The concept of making money is simple enough. You take a resource such as raw materials, money, even people in some instances. Usually it's all three. You take these resources then transform them into something new and better. The new creation is then turned into money, or revenue, for your
business. If you have a business that doesn't make more money than it spends you have a problem. If you have a business that doesn't make any money you have a disaster.

This is easy enough to understand, but let's use a grossly simplified example to make sure we're all on the same page. Let's say you're making widgets. I suppose since this is a security topic we should call them BlockWidgetChain. In our fictional universe you spend $10 on materials and people. Make sure you can track how much something costs, you should be able to determine how much of that $10 is materials and how much is people. You then you sell the BlockWidgetChain for $20. That means you spent $10 to make $20. This should make sense to anyone who understands math (or maths for you English speakers).

Now let's say you have a competitor who makes BlockChainWidgets. They're the same thing basically, but they have no idea how much it costs them to make BlockChainWidgets. They know if they charge more than $20 they can't compete because BlockWidgetChains cost $20. Their solution is to charge $20 and hope the books work out.

I've not only described the business plan for most startups but also a company that's almost certainly in trouble. You have to know how much you spend on resources. If you spend more than you're charging for the product that's a horrible business model. Most of security works like this unfortunately. We have no idea how much a lot of what we do costs, we certainly don't know how much value it adds to the bottom line. In many instances we cannot track spending in a meaningful way.

Measuring security
So now we're on to the idea of measuring security in an environment where the security is responsible for making money. Something like security features in a product. Maybe even a security product in some instances. This is the work that pays my bills. I've been working on product security for a very long time. If you're part of your product team (which you should be, product security doesn't belong anywhere else, more on that another day) then you understand the importance of having features that make a product profitable and useful. For example I would say SSO is a must have in today's environment. If you don't have this feature you can't be as effective in the market. But adding and maintaining features isn't free. If you spend $30 and sell it for $20, you'd make more money just by staying in bed. Sometimes the most profitable decision is to not do something.

Go big or go home
The biggest mistake we like to make is doing too much. It's easy to scope a feature too big. At worst you end up failing completely, at best you end up with what you should have scoped in the first place. But you spend a lot more on failure before you end up where you should have been from the start.

Let's use SSO as our example here. If you were going to scope the best SSO solution in the world, your product would be using SAML, OAuth, PKI, Kerberos, Active Directory, LDAP, and whatever else you manage to think of on planning day. This example is pretty clearly over the top, but I bet a lot of new SSO system scope SAML and OAuth at the same time. The reality is you only need one to start. You can add more later. Firstly having a small scope is important. It shows you want to do one thing and do it well instead of doing 3 things badly. There are few features that are useful in a half finished state. Your sales team has no desire to show off a half finished product.

How to decide
But how do we decide which feature to add? The first thing I do is look at customer feedback. Do the customers clearly prefer one over the other? Setup calls with them, go on visits. Learn what they do and how they do it. If this doesn't give you a clear answer, the next question is always "which feature would sell more product". In the case of something like SAML vs OAuth there might not be a good answer. If you're some sort of cloud service OAuth means you can let customers auth against Google and Facebook. That would probably result in more users.

If you're focused on a lot of on-prem solutions, SAML might be more used. It's even possible SSO isn't what customers are after once you start to dig. I find it's best to make a mental plan of how things should look, then make sure that's not what gets built because whatever I think of first is always wrong ;)

But how much does it cost?
Lastly if there's not a good way to show revenue for a feature, you can look at investment cost. The amount of time and money something will take to implement can really help when deciding what to do. If a feature will take years to develop, that's probably not a feature you want or need. Most industries will be very different in a few years. The expectations of today won't be the expectations of tomorrow.

For example if SAML will take three times as long as OAuth to implement. And both features will result in the same number of sales. OAuth will have a substantially larger return on investment as it's much cheaper to implement. A feature doesn't count for anything until it's on the market. Half done or in development are the same as "doesn't exist". Make sure you track time as part of your costs. Money is easy to measure, but people and time are often just as important.

I really do think this is the easiest security category to measure and justify. That could be because I do it every day, but I think if you can tie actual sales back to security features you'll find yourself in a good place. Your senior leadership will think you're magic if you can show them if they invest resources in X they will get Y. Make sure you track the metrics though. It's not enough to meet expectations, make an effort to exceed your expectations. There's nothing leadership likes better than someone who can over-deliver on a regular basis.

I see a lot of groups that don't do any of this. They wander in circles sometimes adding security features that don't matter, often engineering solutions that customers only need or want 10% of. I'll never forget when I first looked at actual metrics on new features and realized something we wanted to add was going to have a massive cost and generate zero additional revenue (it may have actually detracted in future product sales). On this day I saw the power in metrics. Overnight my group became heroes for saving everyone a lot of work and headaches. Sometimes doing nothing is the most valuable action you can take.

Solutions Architect

Posted by Adam Young on September 05, 2017 03:59 PM

Today is my first day at Red Hat! Well, OK, I’ve been here a few years, but today I move from Engineering to Sales. My new role is “Specialist Solutions Architect” where that specialty is Cloud.

I have a lot to learn, and I will try to use this site to record the most important and interesting details I learn.

What are the Cloud Products? well, according to Red Hat’s site, they are (please mentally prepend Red Hat to all of these) OpenStack Platform, OpenShift, CloudForms, Virtualization, Certificate System, Directory Server, as well as products bundles built out of these. Of these all, I’d guess I have the most to learn about CloudForms, as I’ve only recently started working with that. Really, though, I have a lot to learn across the board. I know that both Ansible Tower and Satellite server are major integration points for management of servers in the large, and I’ll be expected to provide expertise there as well. Plus, everything builds on the other product lines: RHEL and variants, as well as the Storage and Networking solutions.

This is going to be fun. Time to dig in.

Episode 61 - Market driven security

Posted by Open Source Security Podcast on September 05, 2017 12:17 PM
Josh and Kurt talk about our lack of progress in security, economics, and how to interact with peers.


<iframe allowfullscreen="" height="90" mozallowfullscreen="" msallowfullscreen="" oallowfullscreen="" scrolling="no" src="http://html5-player.libsyn.com/embed/episode/id/5708244/height/90/width/640/theme/custom/autonext/no/thumbnail/yes/autoplay/no/preload/no/no_addthis/no/direction/backward/render-playlist/no/custom-color/6e6a6a/" style="border: none;" webkitallowfullscreen="" width="640"></iframe>

Show Notes



The father of modern security: B. F. Skinner

Posted by Josh Bressers on September 04, 2017 01:43 PM
A lot of what we call security is voodoo. Most of it actually.

What I mean with that statement is our security process is often based on ideas that don't really work. As an industry we have built up a lot of ideas and processes that aren't actually grounded in facts and science. We don't understand why we do certain things, but we know that if we don't do those things something bad will happen! Will it really happen? I heard something will happen. I suspect the answer is no, but it's very difficult to explain this concept sometimes.

I'm going to start with some research B. F. Skinner did as my example here. The very short version is that Skinner did research on pigeons. He had a box that delivered food at random intervals. The birds developed rituals that they would do in order to have their food delivered. If a pigeon decided that spinning around would cause food to be delivered, it would continue to spin around, eventually the food would appear reinforcing the nonsensical behavior. The pigeon believed their ritual was affecting how often the food was delivered. The reality is nothing the pigeon did affected how often food was delivered. The pigeon of course didn't know this, they only knew what they experienced.

My favorite example  to use next to this pigeon experiment is the password policies of old. A long time ago someone made up some rules about what a good password should look like. A good password has letters, and numbers, and special characters, and the name of a tree in it. How often we should change a password was also part of this. Everyone knows you should change passwords as often as possible. Two or three times a day is best. The more you change it the more secure it is!

Today we've decided that all this advice was terrible. The old advice was based on voodoo. It was our ritual that kept us safe. The advice to some people seemed like a fair idea, but there were no facts backing it up. Lots of random characters seems like a good idea, but we didn't know why. Changing your password often seemed like a good idea, but we didn't know why. This wasn't much different than the pigeon spinning around to get more food. We couldn't prove it didn't not work, so we kept doing it because we had to do something.

Do you know why we changed all of our password advice? We changed it because someone did the research around passwords. We found out that very long passwords using real words is substantially better than a nonsense short password. We found out that people aren't good at changing their passwords every 90 days. They end up using horrible passwords and adding a 1 to the end. We measured the effectiveness of these processes and understood they were actually doing the opposite of what we wanted them to do. Without question there are other security ideas we do today that fall into this category.

Even though we have research showing this password advice was terrible we still see a lot of organizations and people who believe the old rituals are the right way to keep passwords safe. Sometimes even when you prove something to someone they can't believe it. They are so invested in their rituals that they are unable to imagine any other way of existing. A lot of security happens this way. How many of our rules and processes are based on bad ideas?

How to measure
Here's where it gets real. It's easy to pick on the password example because it's in the past. We need to focus on the present and the future. You have an organization that's full of policy, ideas, and stuff. How can we try to make a dent in what we have today? What matters? What doesn't work, and what's actually harmful?

I'm going to split everything into 3 possible categories. We'll dive deeper into each in future posts, but we'll talk about them briefly right now.

Things that make money
Number one is things that make money. This is something like a product you sell, or a website that customers use to interact with your company. Every company does something that generates revenue. Measuring things that fit into this category is really easy. You just ask "Will this make more, less, or the same amount of money?" If the answer is less you're wasting your time. I wrote about this a bit a long time ago, the post isn't great, but the graphic I made is useful, print it out and plot your features on it. You can probably start asking this question today without much excitement.

Cost of doing business
The next category is what I call cost of doing business. This would be things like compliance or being a part of a professional organization. Sending staff to conferences and meetings. Things that don't directly generate revenue but can have a real impact on the revenue. If you don't have PCI compliance, you can't process payments, you have no revenue, and the company won't last long. Measuring some of these is really hard. Does sending someone to Black Hat directly generate revenue? No. But it will create valuable connections and they will likely learn new things that will be a benefit down the road. I guess you could think of these as investments in future revenue.

My thoughts on how to measure this one is less mature. I think about these often. I'll elaborate more in a future post.

Infrastructure
The last category I'm going to call "infrastructure". This one is a bit harder to grasp what makes sense. It's not unlike the previous question though. In this case we ask ourselves "If I stopped doing this what bad thing would happen?" Now I don't mean movie plot bad thing. Yeah if you stopped using your super expensive keycard entry system a spy from a competitor could break in and steal all your secrets using an super encrypted tor enabled flash drive, but they probably won't. This is the category where you have to consider the cost of an action vs the cost of not doing an action. Not doing things will often have a cost, but doing things also has a cost.

Return on investment is the name of the game here. Nobody likes to spend money they don't have to. This is why cloud is disrupting everything. Why pay for servers you don't need when you can rent only what you do need?

I have some great stories for this category, be sure to come back when I publish this followup article.

The homework for everyone now is to just start thinking about what you do and why you do it. If you don't have a good reason, you need to change your thinking. Changing your thinking is really hard to do as a human though. Many of us like to double down on our old beliefs when presented with facts. Don't be that person, keep an open mind.

Running Keycloak in OpenShift

Posted by Fraser Tweedale on September 04, 2017 06:26 AM

At PyCon Australia in August I gave a presentation about federated and social identity. I demonstrated concepts using Keycloak, an Open Source, feature rich identity broker. Keycloak is deployed in JBoss, so I wasn’t excited about the prospect of setting up Keycloak from scratch. Fortunately there is an official Docker image for Keycloak, so with that as the starting point I took an opportunity to finally learn about OpenShift v3, too.

This post is simply a recounting of how I ran Keycloak on OpenShift. Along the way we will look at how to get the containerised Keycloak to trust a private certificate authority (CA).

One thing that is not discussed is how to get Keycloak to persist configuration and user records to a database. This was not required for my demo, but it will be important in a production deployment. Nevertheless I hope this article is a useful starting point for someone wishing to deploy Keycloak on OpenShift.

Bringing up a local OpenShift cluster

To deploy Keycloak on OpenShift, one must first have an OpenShift. OpenShift Online is Red Hat’s public PaaS offering. Although running the demo on a public PaaS was my first choice, OpenShift Online was experiencing issues at the time I was setting up my demo. So I sought a local solution. This approach would have the additional benefit of not being subject to the whims of conference networks (or, it was supposed to – but that is a story for another day!)

oc cluster up

Next I tried oc cluster up. oc is the official OpenShift client program. On Fedora, it is provided by the origin-clients package. oc cluster up command pulls required images and brings up an OpenShift cluster running on the system’s Docker infrastructure. The command takes no further arguments; it really is that simple! Or is it…?

% oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.5.0 image ...
   Pulling image openshift/origin:v1.5.0
   Pulled 0/3 layers, 3% complete
   ...
   Pulled 3/3 layers, 100% complete
   Extracting
   Image pull complete
-- Checking Docker daemon configuration ... FAIL
   Error: did not detect an --insecure-registry argument on the Docker daemon
   Solution:

     Ensure that the Docker daemon is running with the following argument:
        --insecure-registry 172.30.0.0/16

OK, so it is not that simple. But it got a fair way along, and (kudos to the OpenShift developers) they have provided actionable feedback about how to resolve the issue. I added --insecure-registry 172.30.0.0/16 to the OPTIONS variable in /etc/sysconfig/docker, then restarted Docker and tried again:

% oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.5.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using 192.168.0.160 as the server IP
-- Starting OpenShift container ...
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
-- Adding default OAuthClient redirect URIs ... OK
-- Installing registry ... OK
-- Installing router ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ... OK
-- Removing temporary directory ... OK
-- Checking container networking ... OK
-- Server Information ... 
   OpenShift server started.
   The server is accessible via web console at:
       https://192.168.0.160:8443

   You are logged in as:
       User:     developer
       Password: developer

   To login as administrator:
       oc login -u system:admin

Success! Unfortunately, on my machine with several virtual network, oc cluster up messed a bit too much with the routing tables, and when I deployed Keycloak on this cluster it was unable to communicate with my VMs. No doubt these issues could have been solved, but being short on time and with other approaches to try, I abandoned this approach.

Minishift

Minishift is a tool that launches a single-node OpenShift cluster in a VM. It supports a variety of operating systems and hypervisors. On GNU+Linux it supports KVM and VirtualBox.

First install docker-machine and docker-machine-driver-kvm. (follow the instructions at the preceding links). Unfortunately these are not yet packaged for Fedora.

Download and extract the Minishift release for your OS from https://github.com/minishift/minishift/releases.

Run minishift start:

% ./minishift start
-- Installing default add-ons ... OK
Starting local OpenShift cluster using 'kvm' hypervisor...
Downloading ISO 'https://github.com/minishift/minishift-b2d-iso/releases/download/v1.0.2/minishift-b2d.iso'

... wait a while ...

It downloads a boot2docker VM image containing the openshift cluster, boots the VM, and the console output then resembles the output of oc cluster up. I deduce that oc cluster up is being executed on the VM.

At this point, we’re ready to go. Before I continue, it is important to note that once you have access to an OpenShift cluster, the user experience of creating and managing applications is essentially the same. The commands in the following sections are relevant, regardless whether you are running your app on OpenShift online, on a cluster running on your workstation, or anything in between.

Preparing the Keycloak image

The JBoss project provides official Docker images, including an official Docker image for Keycloak. This image runs fine in plain Docker but the directory permissions are not correct for running in OpenShift.

The Dockerfile for this image is found in the jboss-dockerfiles/keycloak repository on GitHub. Although they do not publish an official image for it, this repository also contains a Dockerfile for Keycloak on OpenShift! I was able to build that image myself and upload it to my Docker Hub account. The steps were as follows.

First clone the jboss-dockerfiles repo:

% git clone https://github.com/jboss-dockerfiles/keycloak docker-keycloak
Cloning into 'docker-keycloak'...
remote: Counting objects: 1132, done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 1132 (delta 14), reused 17 (delta 8), pack-reused 1102
Receiving objects: 100% (1132/1132), 823.50 KiB | 158.00 KiB/s, done.
Resolving deltas: 100% (551/551), done.
Checking connectivity... done.

Next build the Docker image for OpenShift:

% docker build docker-keycloak/server-openshift
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM jboss/keycloak:latest
 ---> fb3fc6a18e16
Step 2 : USER root
 ---> Running in 21b672e19722
 ---> eea91ef53702
Removing intermediate container 21b672e19722
Step 3 : RUN chown -R jboss:0 $JBOSS_HOME/standalone &&     chmod -R g+rw $JBOSS_HOME/standalone
 ---> Running in 93b7d11f89af
 ---> 910dc6c4a961
Removing intermediate container 93b7d11f89af
Step 4 : USER jboss
 ---> Running in 8b8ccba42f2a
 ---> c21eed109d12
Removing intermediate container 8b8ccba42f2a
Successfully built c21eed109d12

Finally, tag the image into the repo and push it:

% docker tag c21eed109d12 registry.hub.docker.com/frasertweedale/keycloak-openshift

% docker login -u frasertweedale registry.hub.docker.com
Password:
Login Succeeded

% docker push registry.hub.docker.com/frasertweedale/keycloak-openshift
... wait for upload ...
latest: digest: sha256:c82c3cc8e3edc05cfd1dae044c5687dc7ebd9a51aefb86a4bb1a3ebee16f341c size: 2623

Adding CA trust

For my demo, I used a local FreeIPA installation to issue TLS certificates for the the Keycloak app. I was also going to carry out a scenario where I configure Keycloak to use that FreeIPA installation’s LDAP server to authenticate users. I wanted to use TLS everywhere (eat your own dog food!) I needed the Keycloak application to trust the CA of one of my local FreeIPA installations. This made it necessary to build another Docker image based on the keycloak-openshift image, with the appropriate CA trust built in.

The content of the Dockerfile is:

FROM frasertweedale/keycloak-openshift:latest
USER root
COPY ca.pem /etc/pki/ca-trust/source/anchors/ca.pem
RUN update-ca-trust
USER jboss

The file ca.pem contains the CA certificate to add. It must be in the same directory as the Dockerfile. The build copies the CA certificate to the appropriate location and executes update-ca-trust to ensure that applications – including Java programs – will trust the CA.

Following the docker build I tagged the new image into my hub.docker.com repository (tag: f25-ca) and pushed it. And with that, we are ready to deploy Keycloak on OpenShift.

Creating the Keycloak application in OpenShift

At this point we have a local OpenShift cluster (via Minishift) and a Keycloak image (frasertweedale/keycloak-openshift:f25-ca) to deploy. When deploying the app we need to set some environment variables:

KEYCLOAK_USER=admin

A username for the Keycloak admin account to be created

KEYCLOAK_PASSWORD=secret123

Passphrase for the admin user

PROXY_ADDRESS_FORWARDING=true

Because the application will be running behind OpenShift’s HTTP proxy, we need to tell Keycloak to use the "external" hostname when creating hyperlinks, rather than Keycloak’s own view.

Use the oc new-app command to create and deploy the application:

% oc new-app --docker-image frasertweedale/keycloak-openshift:f25-ca \
    --env KEYCLOAK_USER=admin \
    --env KEYCLOAK_PASSWORD=secret123 \
    --env PROXY_ADDRESS_FORWARDING=true
--> Found Docker image 45e296f (4 weeks old) from Docker Hub for "frasertweedale/keycloak-openshift:f25-ca"

    * An image stream will be created as "keycloak-openshift:f25-ca" that will track this image
    * This image will be deployed in deployment config "keycloak-openshift"
    * Port 8080/tcp will be load balanced by service "keycloak-openshift"
      * Other containers can access this service through the hostname "keycloak-openshift"

--> Creating resources ...
    imagestream "keycloak-openshift" created
    deploymentconfig "keycloak-openshift" created
    service "keycloak-openshift" created
--> Success
    Run 'oc status' to view your app.

The app gets created immediately but it is not ready yet. The download of the image and deployment of the container (or pod in OpenShift / Kubernetes terminology) will proceed in the background.

After a little while (depending on how long it takes to download the ~300MB Docker image) oc status will show that the deployment is up and running:

% oc status
In project My Project (myproject) on server https://192.168.42.214:8443

svc/keycloak-openshift - 172.30.198.217:8080
  dc/keycloak-openshift deploys istag/keycloak-openshift:f25-ca 
    deployment #2 deployed 3 minutes ago - 1 pod

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

(In my case, the first deployment failed because the 10-minute timeout elapsed before the image download completed; hence deployment #2 in the output above.)

Creating a secure route

Now the Keycloak application is running, but we cannot reach it from outside the Keycloak project itself. In order to be able to reach it there must be a route. The oc create route command lets us create a route that uses TLS (so clients can authenticate the service). We will use the domain name keycloak.ipa.local. The public/private keypair and certificate have already been generated (how to do that is outside the scope of this article). The certificate was signed by the CA we added to the image earlier. The service name – visible in the oc status output above – is svc/keycloak-openshift.

% oc create route edge \
  --service svc/keycloak-openshift \
  --hostname keycloak.ipa.local \
  --key /home/ftweedal/scratch/keycloak.ipa.local.key \
  --cert /home/ftweedal/scratch/keycloak.ipa.local.pem
route "keycloak-openshift" created

Assuming there is a DNS entry pointing keycloak.ipa.local to the OpenShift cluster, and that the system trusts the CA that issued the certificate, we can now visit our Keycloak application:

% curl https://keycloak.ipa.local/
<!--
  ~ Copyright 2016 Red Hat, Inc. and/or its affiliates
  ~ and other contributors as indicated by the @author tags.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
    <meta http-equiv="refresh" content="0; url=/auth/" />
    <meta name="robots" content="noindex, nofollow">
    <script type="text/javascript">
        window.location.href = "/auth/"
    </script>
</head>
<body>
    If you are not redirected automatically, follow this <a href='/auth'>link</a>.
</body>
</html>

If you visit in a browser, you will be able to log in using the admin account credentials specified in the KEYCLOAK_USER and KEYCLOAK_PASSWORD environment variables specified when the app was created. And from there you can create and manage authentication realms, but that is beyond the scope of this article.

Conclusion

In this post I discussed how to run Keycloak in OpenShift, from bringing up an OpenShift cluster to building the Docker image and creating the application and route in OpenShift. I recounted that I found OpenShift Online unstable at the time I tried it, and that although oc cluster up did successfully bring up a cluster I had trouble getting the Docker and VM networks to talk to each other. Eventually I tried Minishift which worked well.

We saw that although there is no official Docker image for Keycloak in OpenShift, there is a Dockerfile that builds a working image. It is easy to further extend the image to add trust for private CAs.

Creating the Keycloak app in OpenShift, and adding the routes, is straightforward. There are a few important environment variables that must be set. The oc create route command was used to create a secure route to access the application from the outside.

We did not discuss how to set up Keycloak with a database for persisting configuration and user records. The deployment we created is ephemeral. This satisfied my needs for demonstration purposes but production deployments will require persistence. There are official JBoss Docker images that extend the base Keycloak image and add support for PostgreSQL, MySQL and MongoDB. I have not tried these but I’d suggest starting with one of these images if you are looking to do a production deployment. Keep in mind that these images may not include the changes that are required for deploying in OpenShift.

SE Linux for CentOS Part 3

Posted by Adam Young on September 02, 2017 01:02 AM

After the previous two days debugging, Simo Sorce suggested that I need to tell the OS to show all AVCs, some are hidden by default.

The problem is that not all AVCs are reported. We can disable this.

First I needed to install setools:

sudo yum install setools-console

With that I could confirm that there were hidden AVCs:

sudo seinfo --stats | grep audit Audit
allow: 157 Dontaudit: 8036

I disabled the hiding of the AVCs:

sudo semodule --disable_dontaudit --build

And a bunch more AVCs now show up when I deploy a VM.  But…after a couple iterations, its obvious that the same errors keep showing up. Here’s a sample:

type=AVC msg=audit(1504306852.970:3169): avc: denied { search } for pid=27918 comm="cat" name="27821" dev="proc" ino=275493 scontext=system_u:system_r:svirt_tcg_t:s0:c130,c773 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c0,c5 tclass=dir
type=AVC msg=audit(1504306852.984:3173): avc: denied { search } for pid=27929 comm="nsenter" name="27821" dev="proc" ino=275493 scontext=system_u:system_r:svirt_tcg_t:s0:c130,c773 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c0,c5 tclass=dir
type=AVC msg=audit(1504306852.994:3174): avc: denied { search } for pid=27930 comm="pkill" name="498" dev="proc" ino=12227 scontext=system_u:system_r:svirt_tcg_t:s0:c130,c773 tcontext=system_u:system_r:udev_t:s0-s0:c0.c1023 tclass=dir

 

Looking at the policy generated by audit2allow:

#!!!! This avc is a constraint violation. You would need to modify the attributes of either the source or target types to allow this access.
#Constraint rule: 
# mlsconstrain dir { ioctl read lock search } ((h1 dom h2 -Fail-) or (t1 != { netlabel_peer_t openshift_t openshift_app_t sandbox_min_t sandbox_x_t sandbox_web_t sandbox_net_t svirt_t svirt_tcg_t svirt_lxc_net_t svirt_qemu_net_t svirt_kvm_net_t } -Fail-) ); Constraint DENIED

The line is commented out, which tells me I should not just blindly enable it.  At the bottom of the policy file I see the comment:

# Possible cause is the source user (system_u) and target user (unconfined_u) are different.
# Possible cause is the source role (system_r) and target role (unconfined_r) are different.
# Possible cause is the source level (s0:c130,c773) and target level (s0-s0:c0.c1023) are different.

Curious.

SE Linux for CentOS Continued

Posted by Adam Young on September 02, 2017 12:52 AM

Trying to troubleshoot the issues from Yesterday’s SELinux errors.

Immediately after a new deploy of the manifests, I want to look at the context on the qemu file:

$ kubectl get pods libvirt-81sdh 
NAME            READY     STATUS    RESTARTS   AGE
libvirt-81sdh   2/2       Running   0          28s

Now to look at the file:

$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:unlabeled_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

Running restorecon on it, as the audit2allow output suggests:

[ayoung@drifloon kubevirt]$ kubectl  exec libvirt-81sdh -c libvirtd -- restorecon  /usr/local/bin/qemu-system-x86_64
[ayoung@drifloon kubevirt]$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:bin_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

unlabeled_t became bin_t.

Once again, attempt to deploy a vm, and see what AVCs we get:

$ kubectl apply -f cluster/vm-pxe.yaml 
vm "testvm" created
[ayoung@drifloon kubevirt]$ kubectl delete  -f cluster/vm-pxe.yaml 
vm "testvm" deleted

This is what the audit log showed:

type=AVC msg=audit(1504291091.397:2933): avc:  denied  { transition } for  pid=32273 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-18" ino=31526884 scontext=system_u:system_r:spc_t:s0 tcontext=system_u:system_r:svirt_tcg_t:s0:c322,c373 tclass=process

There were several lines like that, but they were identical except for the pid. What does audit2allow show?

#============= spc_t ==============

#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow spc_t svirt_tcg_t:process transition;

Lets see if the additional parameters make a difference:

$ kubectl  exec libvirt-81sdh -c libvirtd -- restorecon  -R -v  /usr/local/bin/qemu-system-x86_64
$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:bin_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

The original lableeing of system_u:object_r:unlabeled_t:s0 is now system_u:object_r:bin_t:s0, which is the same as it was after maiong the restorecon call without the additional parameters.

How about the additional line, the allow? I can apply it outside of the container by using audit2allow:

cat /tmp/audit.txt | audit2allow -a -M virt-policy
sudo semodule -i virt-policy.pp

Upon deploy, a similar error, with a different context:

type=AVC msg=audit(1504294173.446:3734): avc:  denied  { entrypoint } for  pid=6565 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-18" ino=31526884 scontext=system_u:system_r:svirt_tcg_t:s0:c577,c707 tcontext=system_u:object_r:bin_t:s0 tclass=file

Running this through audit2allow generates

#============= svirt_tcg_t ==============

#!!!! WARNING: 'bin_t' is a base type.
allow svirt_tcg_t bin_t:file entrypoint;

While this is a pretty powerful rule, it might be appropriate for what we are doing with virt. Again, lets apply the policy and see what happens.

$ cat virt-policy-2.txt | audit2allow -a -M virt-policy-2
$ sudo semodule -i virt-policy-2.pp

Now a slew of errors, but different ones:

type=AVC msg=audit(1504294406.893:3797): avc:  denied  { write } for  pid=7236 comm="qemu-system-x86" path="pipe:[423417]" dev="pipefs" ino=423417 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:spc_t:s0 tclass=fifo_file
type=AVC msg=audit(1504294406.893:3797): avc:  denied  { write } for  pid=7236 comm="qemu-system-x86" path="pipe:[423417]" dev="pipefs" ino=423417 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:spc_t:s0 tclass=fifo_file
type=AVC msg=audit(1504294406.894:3798): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="ld.so.cache" dev="dm-18" ino=8388771 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1504294406.894:3799): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3800): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3801): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3802): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3803): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3804): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3805): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3806): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.895:3807): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="libtinfo.so.6" dev="dm-18" ino=29360804 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.900:3808): avc:  denied  { sigchld } for  pid=21975 comm="docker-containe" scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:container_runtime_t:s0 tclass=process

This process is iterative, and I had to go through it 10 times until I came up with a complete set of audit2allow generated files. Here is the sum total of what was generated.

module virt-policy-2 1.0;

require {
	type svirt_tcg_t;
	type bin_t;
	class file entrypoint;
}

#============= svirt_tcg_t ==============

#!!!! WARNING: 'bin_t' is a base type.
allow svirt_tcg_t bin_t:file entrypoint;

module virt-policy-3 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	type spc_t;
	type container_runtime_t;
	class process sigchld;
	class lnk_file read;
	class fifo_file write;
	class file read;
}

#============= svirt_tcg_t ==============
allow svirt_tcg_t container_runtime_t:process sigchld;
allow svirt_tcg_t spc_t:fifo_file write;

#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file read;
allow svirt_tcg_t unlabeled_t:lnk_file read;

module virt-policy-4 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file open;
}

#============= svirt_tcg_t ==============

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/ld.so.cache' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/ld.so.cache
allow svirt_tcg_t unlabeled_t:file open;

module virt-policy-5 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file getattr;
}

#============= svirt_tcg_t ==============

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/ld.so.cache' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/ld.so.cache
allow svirt_tcg_t unlabeled_t:file getattr;

module virt-policy-6 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file execute;
}

#============= svirt_tcg_t ==============

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/usr/lib64/libtinfo.so.6.0' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/lib64/libtinfo.so.6.0
allow svirt_tcg_t unlabeled_t:file execute;

module virt-policy-7 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	type spc_t;
	class fifo_file { getattr ioctl };
	class file { execute_no_trans write };
}

#============= svirt_tcg_t ==============
allow svirt_tcg_t spc_t:fifo_file { getattr ioctl };

#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file { execute_no_trans write };

module virt-policy-8 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	type sysfs_t;
	class capability { setgid setuid };
	class file append;
	class filesystem getattr;
}

#============= svirt_tcg_t ==============
allow svirt_tcg_t self:capability { setgid setuid };
allow svirt_tcg_t sysfs_t:filesystem getattr;

#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file append;

module virt-policy-9 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file ioctl;
	class dir read;
}

#============= svirt_tcg_t ==============
allow svirt_tcg_t unlabeled_t:dir read;

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/sudoers' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/sudoers
allow svirt_tcg_t unlabeled_t:file ioctl;

module virt-policy 1.0;

require {
	type svirt_tcg_t;
	type spc_t;
	class process transition;
}

#============= spc_t ==============

#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow spc_t svirt_tcg_t:process transition;


module virt-policy-10 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	class capability { net_admin sys_resource };
	class file lock;
	class netlink_audit_socket create;
}

#============= svirt_tcg_t ==============
allow svirt_tcg_t self:capability { net_admin sys_resource };
allow svirt_tcg_t self:netlink_audit_socket create;

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/run/utmp' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /run/utmp
allow svirt_tcg_t unlabeled_t:file lock;

Obviously, using permissive would have been a shorter process. Let me restart the VM and try that. Here’s what I generate after one iteration:

module kubevirt-policy 1.0;

require {
	type unlabeled_t;
	type svirt_tcg_t;
	type container_runtime_t;
	class capability audit_write;
	class unix_stream_socket connectto;
	class file entrypoint;
	class netlink_audit_socket nlmsg_relay;
}

#============= svirt_tcg_t ==============

#!!!! The file '/run/docker.sock' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /run/docker.sock
allow svirt_tcg_t container_runtime_t:unix_stream_socket connectto;
allow svirt_tcg_t self:capability audit_write;
allow svirt_tcg_t self:netlink_audit_socket nlmsg_relay;

#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow svirt_tcg_t unlabeled_t:file entrypoint;

And…. we start into the same pattern. It takes several runs to get to a set of policies that run cleanly. It seems some of the earlier AVCs mask later ones, and running in permissive mode only reports the first of several. Needless to say, the policy for running a VM Via Kubevirt is going to require some scrutiny.

And even then, the VMs still fail to deploy. Disable SELinux and they run. This mystery continues.