Distroless Containers for corporate use: Nix Flakes vs Fedora

Distroless Containers for corporate use: Nix Flakes vs Fedora

In this post, I share my personal perspective on distroless containers for corporate use, hence with a view on compliance. I offer an alternative to Nix Flakes using Fedora and Podman.

Logos of Fedora, Podman and NixOS

For several years, Bitnami offered many standard cloud component Container images and Kubernetes Helm charts, e.g. for PostgreSQL, MariaDB, Redis, or MongoDB. I think they were well maintained, used in many production setups, and used with Bitnami subscriptions by paying customers. Also governments used them (e.g. the German government with openDesk and BundesMessenger or the European Commission with SIMPL-Open). In 2019, Bitnami was bought by VM Ware.

In 2025, Bitnami revaluated their business case and decided to discontinue their current offering:

In a time, where many customers find their dependency with VM Ware already problematic, customers are sceptical towards a mere upgrade to the new Bitnami offering. So it comes to no surprise, that the Internet is full of discussions on alternatives. Before I present three of them, let me recall the challenges:

  • supply chain security
  • complexity due to diverging supply chain sources
  • transparency
  • compliance with NIS2 (Network and Information Security (NIS) Directive 2, adopted in November 2022 and in principle effective from 2025), example measures from nis2compliant.org:
    • robust vulnerability handling and disclosure practices
    • Secure supply chain interactions and mitigate risks related to suppliers or service providers, ensuring comprehensive security from end to end
    • incident detection, triage, and response to meet reporting obligations
  • Cybersecurity Regulation for EU institutions, e.g.
    • supply chain security, including security-related aspects concerning the relationships between each Union entity and its direct suppliers or service providers
    • establishment of software supply chain security through criteria for secure software development and evaluation
  • any other applicable national legislation

So what are the options to manage supply chain security with a view of reaching compliance? I believe it is impossible for an individual organisation to take care of the supply chain security directly for all their software in use. On top, any attempt would not be good use of (public) money. Hence, the goal must be to outsource and to seek synergies with organisations having similar or higher requirements.

CloudPirates.io

Some German products may switch from Bitnami to CloudPirates.io. CloudPirates is a company like Bitnami, but it is in Germany and hasn’t been bought by VM Ware (yet). Read their German blog post addressing the Bitnami policy change. I have checked their MariaDB helm chart. It relies on the community MariaDB container.

To ensure compliance when using their work, it may be necessary to introduce obligations for CloudPirates, which they may allow only against a fee.

In alternative would be RedHat Linux, that also offers all kinds of containers with maintenance against a fee.

Self-Made Distroless Containers

Governments could build containers for their own ministries. This is what parts of the German government currently explore. But then, there are many ways how this can be organised.

  • Upstream: The Government could just review the community images that everyone is using. However, the community may not be reactive enough or have diverging standards, etc.
  • Downstream: The Government could maintain their own downstream fork and still collaborate with the community.

In both situations, you have to decide with which community. Consider for instance MariaDB. Relevant communities are:

  • the MariaDB community
  • the Debian community that packages (and patches) MariaDB
  • the Opensuse community that packages (and patches) MariaDB
  • the NixOS community that packages (and patches) MariaDB
  • the Fedora/CentOS Stream/AlmaLinux community that packages (and patches MariaDB)

The US Government got a project on this at https://repo1.dso.mil/dsop, but it consists of many repos, so that I cannot grasp easily their general apporach. Their NodeJS (slim) image relies on Alpine sourced from their own mirror.

The German Government decided to test NixOS Flakes to build containers from Debian packages that contain the bare minimum of software. If the container does not even contain a package management system, then it is called distroless. Read more about it from Google, Docker, RedHat, or Bitnami (minideb).

Find their work at: https://gitlab.opencode.de/open-code/oci

The following list of their requirements is copied over from the (nodejs image README.md):

Base Image Security

  • Minimal base images are used - There is no base image at all, since this build is done using nix and debian packages directly.
  • Base image provenance is verified - There is no base image at all, since this build is done using nix and debian packages directly.
  • Immutable artifact references are used - There is no base image at all, since this build is done using nix and debian packages directly.

  • Base Image can be automatically updates (or after a fixed period of time to avoid being a victim of a supply chain attack) - There is no base image at all, since this build is done using nix and debian packages directly.

Build Process Security

  • Reproducible builds are implemented - Nix is being used to ensure that builds are reproducible, meaning the same source code and build instructions always produce identical container images.

  • Build environment is isolated - Build runs in a Kubernetes GitLab-Runner. Nix is instructed to not do any sandboxing. However, the build environment is isolated from the host system and other builds to prevent contamination.

  • Build provenance is attested - Build process generates cryptographically signed provenance (metadata about who, what, when, and how the artifact was built), ideally at SLSA Level 2 or higher. This creates an auditable trail proving the container came from your legitimate build system and hasn’t been substituted.

  • Containers are signed - Images are signed using Cosign to ensure authenticity and integrity. They can be verified using the cosign.pub public key.

  • Dev / Compile time dependencies are removed - Uses Nix

Component Management & Transparency

  • All components are identified - Have a look at the config json files in the root directory. All components, including their versions, urls and checksums are listed there as input to the build process.

  • Component PURLs can be matched to CVE reports We are using debian packages to match against known CVEs. Besides that, we are downloading Nodejs 24 from nodejs.org. To match that against CVEs, we are using the PURL constructed from their GitHub repository: pkg:github/nodejs/node@<version>. We can match this using DevGuard against CVEs (https://osv.dev/list?q=github.com%2Fnodejs%2Fnode).

  • Component checksums are verified - All checksums are either in config.json or in flake.nix.

  • Regular updates are provided for all components or latest Builds like new nodejs versions - Components receive timely updates and patches from upstream maintainers, and your build process incorporates these updates regularly. This ensures your container stays protected against newly discovered vulnerabilities in its components.

  • Components can be automatically updated in a timely manner (or after a fixed period of time to avoid being a victim of a supply chain attack) - An automated pipeline exists to detect, test, and deploy component updates without manual intervention. Can provenance or signatures be verified for upstream components? This ensures security patches are applied quickly, reducing the window of exposure to known vulnerabilities.

Secrets & Sensitive Data

  • No secrets in images - Credentials, API keys, certificates, and other sensitive data are never embedded in container images; they are injected at runtime via secrets management systems. This prevents secrets from being exposed in image layers, which can be extracted by anyone with access to the image.

Runtime Configuration

  • Resource limits are documented - Since this is an runtime image only, resource limits heavily depend on the application being run.

  • Container runs as non-root user - User 53111 (nonroot) is used as non-root user.

Compliance & Vulnerability Management

  • SBOM is attested - A Software Bill of Materials (SBOM) is generated, accurate, and cryptographically attested to prove the container’s contents. This provides a tamper-proof inventory of components for compliance, license management, security scanning, and incident response.

  • Vulnerability management is done in a timely manner - We are using DevGuard to monitor vulnerabilities in our components. New vulnerabilities are assessed and remediated promptly based on their severity and exploitability.

  • VEX is attested - Vulnerability Exploitability eXchange (VEX) documents are provided and attested, indicating which vulnerabilities are exploitable in the specific container context and which are mitigated. This reduces alert fatigue by documenting which CVEs don’t actually affect your container due to configuration or usage patterns.

Fedora-based distroless Container images

As part of my pet pilot project EU OS, I rely on compiled code (i.e. RPM packages) from Fedora. Fedora (and their downstream stable versions Redhat RHEL, CentOS Stream, AlmaLinux) have technologies in place to cover most if not all Government use cases for open source:

  • operating system for the corporate laptop of the end user (check out EU OS for some inspiration)
  • operating system for cloud servers, including Kubernetes clusters
  • container images for cloud workloads

So if we reuse the compiled code for all purposes, then the supply chain security becomes more managable (but due to such centralisation, vulnerabilities could have a higher impact).

Let us check, how distroless container images can be built from Fedora RPM packages. Fedora described this in a blog post from 2021. Meanwhile, things have changed a little bit and such Fedora distroless images can also be composed with podman and its multi stage builds. That’s my example for a small NodeJS container:

# kate: hl Containerfile;

ARG ROOTFS="/mnt/rootfs"
ARG HOME=/home/nonroot

ARG DNF="dnf"
ARG RELEASEVER="42"
FROM quay.io/fedora/fedora-minimal:42
# alternatively:
# ARG RELEASEVER="9"
# FROM registry.access.redhat.com/ubi9/ubi-minimal
# or
# ARG RELEASEVER="10"
# FROM quay.io/almalinuxorg/10-minimal:10.0

ARG ROOTFS
ARG DNF
ARG RELEASEVER
ARG DNF_OPTS="--installroot=${ROOTFS} --releasever=${RELEASEVER} --noplugins --config=/etc/dnf/dnf.conf --setopt=install_weak_deps=0 --setopt=cachedir=/var/cache/$DNF --setopt=keepcache=1 --setopt=reposdir=/etc/yum.repos.d --setopt=varsdir=/etc/dnf"

USER root

# pinning of software versions possible with https://dnf5.readthedocs.io/en/latest/dnf5_plugins/manifest.8.html
# (see also: https://github.com/rpm-software-management/dnf5/pull/2425)
RUN --mount=type=cache,target=/var/cache/$DNF \
  mkdir -p ${ROOTFS} && \
  $DNF ${DNF_OPTS} -y --nodocs install nodejs22

FROM scratch

ARG ROOTFS
ARG HOME

COPY --from=base ${ROOTFS} /

RUN \
  mkdir -p $HOME && \
  printf "nonroot:x:1001:\n" >> /etc/group && \
  printf "nonroot:x:1001:1001:Nonroot User:/home/nonroot:/sbin/nologin\n" >> /etc/passwd && \
  printf "nonroot:!:20386::::::\n" >> /etc/shadow && \
  chown -R 1001:1001 $HOME && \
  chmod -R g=u $HOME

USER 1001
WORKDIR $HOME
ENTRYPOINT ["/bin/bash"]

How does it compare? What is missing?

  • The container image is not yet reproducible in the sense that it always uses the latest packages at the time of the build. However, with RPM manifests, the Fedora package manager can be instructed to install specific software versions, similar to npm and its package.json files. I could not enable it yet, because the feature is currently disabled, as the library is not yet widely available in the Fedora package repositories.
  • SBOMs can be generated directly from the RPM database. Trivy can list vulnerabilities for a given SBOM, but only for distributions with support (fedora is not; CentOS Stream, RHEL, and AlmaLinux is). Trivy can also generate SBOMs for the container images directly. Renovate can be configured to update RPM manifest files.
  • Podman supports signing containers.
  • The container image also supports a non-root account.
  • The container currently contains still bash, find, sed, grep as those tools are pulled in as a dependency of ca-certificates. The latter is required by nodejs. To remove them, an alternative custom ca-certificates package needs to be prepared that has no such dependencies. See also: https://discussion.fedoraproject.org/t/169906/2

I think the main advantage would be to avoid Nix flakes. Maybe Nix flakes are cool, but the system is apparently still experimental/beta software (see here or here). Also, many developers have not worked yet with Nix flakes. So this is something new to learn. Using Nix flakes doesn’t make podman or Containerfiles redundant. So learning Nix flakes does not replace learning Podman or Containerfiles.

Obviously, this advantage would apply equally to building Podman distroless containers with OpenSUSE RPMs or Debian DEBs. All it takes is a build tool that can install dependencies in a separate folder. For dnf, this is done with the option --installroot. If an organisation has already solved supply chain security for a repository of compiled code, then I believe it is good practice to reuse this repository.

What is your view? Please comment, react on Mastodon or use any other channel.

References

Install Belgian eID on Atomic Fedora 42 (Kinoite/Silverblue)

Belgium provides the Belgian eID software also for Fedora, but on Atomic Fedora, the setup is a bit different. Let’s test eID authentication and PDF signing

To do my tax declaration in Belgium, I have several login methods. One of them is the Belgian eID (eidas). To use it, you need an ID card (or resident card) and a smart card reader. I use the smart card reader CardMan 3121 from OMNIKEY. The setup will also allow you to sign PDF documents and emails with your Belgian ID card. Neat! Other countries would require the purchase of additional certificates, but in Belgium you should have it already – free of charge.

Install Belgian eID on an Atomic Fedora desktop

sudo rpm-ostree install https://eid.belgium.be/sites/default/files/software/eid-archive-fedora-2021-1.noarch.rpm
# reboot now
sudo rpm-ostree install -A eid-viewer eid-mw
# optional reboot

You can check if everything is in order with rpm-ostree status. My output:

State: idle
Deployments:
  fedora:fedora/42/x86_64/kinoite
                  Version: 42.20250429.1 (2025-04-29T19:10:59Z)
               BaseCommit: 530f49cde70f792bb77daa1c0570e1e2e66e2e1ac15c5edcf8e4b2774e452105
                   Commit: b96d42074e4448754bd192650dd5efbdc4192ac004667adb35491db84cb47440
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
                     Diff: 6 added
          LayeredPackages: eid-mw eid-viewer [redacted]
            LocalPackages: eid-archive-fedora-2021-1.noarch

● fedora:fedora/42/x86_64/kinoite
                  Version: 42.20250429.1 (2025-04-29T19:10:59Z)
         BootedBaseCommit: 530f49cde70f792bb77daa1c0570e1e2e66e2e1ac15c5edcf8e4b2774e452105
                   Commit: 1a3e69661f9dbca3cd798c807c59d2c2c28331f7496b9ea0dab6d46986c6b740
               LiveCommit: b96d42074e4448754bd192650dd5efbdc4192ac004667adb35491db84cb47440
                 LiveDiff: 6 added
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
          LayeredPackages: [redacted]
            LocalPackages: eid-archive-fedora-2021-1.noarch
                 Unlocked: transient

Then, you need to install the Firefox plugin from https://addons.mozilla.org/en-US/firefox/addon/belgium-eid/.

Note that on Atomic Fedora desktops, Firefox is (as of May 2025) installed as system application and other browsers (such as Chromium) is installed in a flatpak sandbox. So it is very likely that other browsers than Firefox cannot access the eID setup on the system.

References:

First Test with eid-viewer

You should find now in your application menu eID Viewer. Or you lunch in the terminal eid-viewer. Enter your card. Then you should see the data on your card already.

Login with eID

You can now use the Belgian eID to access a governmental service, such as the tax declaration portal. Go to https://fin.belgium.be/fr/particuliers/declaration-impot/rentrer-declaration and choose eID as your mean for authentication. You will need to provide the PIN code that comes with the ID card. :tada:

Sign PDFs with eID

This is not so clear yet. Okular is usually a flatpak. In order to have gpg find the card reader, I had to restart a service first:

gpg --card-status
# => can't connect to 'socket:///home/rriemann/.gnupg/log-socket': No such file or directory
systemctl restart pcscd
gpg --card-status
# can't connect to 'socket:///home/rriemann/.gnupg/log-socket': No such file or directory
# Reader ...........: OMNIKEY AG CardMan 3121 00 00
# Application ID ...: 534C4090413423078AA5B22712924134
# Application type .: PKCS#15

Okular supports as PDF signature backends both NSS and GnuPG (S/MIME). As it does not work with any option, I check in the app Kleopatra (KDE certificate manager) the smartcards. It turns out I have to configure the trust of various certificates belonging to the Belgian authorities.

Then, I restart Okular again and choose under SettingsConfigure Backends… → PDF backend configuration the option Signature Backend to GnuPG (S/MIME). I get the following feedback:

screenshot of Okular backends config dialogue

When I then choose in the Okular Tools menu the signing option, I end up in a loop with a pinentry-qt dialogue:

Please insert the card with serial number:

[redacted serial number]

It does not work. So close!

An alternative for signing offers the command line tool pdfsig.

With pdfsig -backend GPG -list-nicks, I get a list of fingerprints. One of the hardware ones is for signing, one for authentication. The smartcard tab in the app Kleopatra also displays the names/purposes alongside the fingerprint. So it may be better suited. Otherwise, try out all to find the one for signing. Then, PDFs should be signed with:

pdfsig unsigned.pdf signed.pdf -add-signature -nick [redacted my fingerprint] -reason 'for fun!'

Unfortunately, I only get an error:

signDocument: error getting signature info

We can try briefly the NSS backend with pdfsig. For this, use pdfsig -list-nicks to check nick names:

Certificate nicknames available: BELPIC:Authentication BELPIC:Signature

Then, signing should work with:

pdfsig unsigned.pdf signed.pdf -add-signature -nick BELPIC:Signature -reason 'for fun!'

Then, I get queried for the pin and upon entry, the PDF is signed. This can be checked as follows:

# pdfsig signed.pdf
Digital Signature Info of: signed.pdf
Signature #1:
  - Signature Field Name: 34B8E9A9E274A3BCE18E633ABD5B1ECA
  - Signer Certificate Common Name: Robert Riemann (Signature)
  - Signer full Distinguished Name: CN=Robert Riemann (Signature),serialNumber=[redacted],givenName=Robert,SN=Riemann,C=DE
  - Signing Time: May 29 2025 14:58:22
  - Signing Hash Algorithm: SHA-256
  - Signature Type: adbe.pkcs7.detached
  - Signed Ranges: [0 - 515528], [535530 - 536032]
  - Total document signed
  - Signature Validation: Signature is Valid.
  - Certificate Validation: Certificate issuer isn't Trusted.

It remains yet to determine why the certificate validation fails even though the certificate is marked trusted in Kleopatra. Let me know if you have an answer!

Interview about EU OS: ‘Most public servants have never used something else than Windows’

The Spanish techblog ‘MuyLinux’ has interviewed Robert Riemann on EU OS. Find here the English version of the interview.

The interview has been conducted by Jose Pomeyrol and published originally in Spanish on MuyLinux. Find the English version here below.

EU OS logo The other day I noticed something curious: after updating one of the apps I use regularly, it now shows a bold message when starting up — “Made with ❤️ Europe.” It’s similar to the tagline on the credits page of EU OS, a new Linux distribution being discussed in various tech-focused forums these last days. What do these two projects have in common? Among other things, they are both developed in Europe — or at least, their final form is.

Europe, and the European Union in particular, is preparing to face challenges unprecedented in recent history: tensions with Russia and calls for rearmament among Eurozone members; Trump’s return to the White House and a new wave of protectionist policies; and China’s technological rise, especially in AI. Europe must respond on multiple fronts — and the complexity of these issues doesn’t make things any easier.

To explore all this, we exchanged via email with Robert Riemann, master in physics and PhD in computer science, Head of Digital Transformation in the Technology and Privacy Unit of one body of the EU, and project lead of EU OS, a Linux distribution with institutional ambitions… proudly “Made with ❤️ in Brussels.”

Mastodon: How to configure custom signup questions

Mastodon: How to configure custom signup questions

Mastodon administrators can enable manual review for signups. This blog post shows how to add a custom question to receive relevant input for this manual review.

The Mastodon instance eupolicy.social is geared towards people who identify themselves as part of the EU Bubble. Purposfully, there is no definition provided and indeed the instance is a bit flexible on what EU Bubble would mean.

To help the instance keep this focus (and avoid spam), the admins have opted for approval-based registration of signups. New users can during their signup process provide some text about themselves. To make this more relevant, eupolicy.social decided to customise the text that instructs the users to describe their relation with the instance.

HU Berlin eduroam for Android

If the eduroam Android app to setup the wifi crashes for you, try this alternative path.

I tried to setup eduroam for the Humboldt University of Berlin (Humboldt Universität in Berlin) using the app advertised in the manual: geteduroam

Unfortunately, the app crashes on my Android phone. If this is your case as well, proceed as follows:

  1. download the CA certificate hu-ca-2024.crt1
  2. go to your wifi settings and select eduroam to setup this wifi
  3. use TTLS/PAP (I forgot which one)
  4. add anonymous identity: eduroam@hu-berlin.de
  5. add username as username@hu-berlin.de (use username@physik.hu-berlin.de or username@mathematik.hu-berlin.de if your account is with those faculties)
  6. add as CA certificate the file downloaded before
  7. do not verify this certificate
  8. add as domain: hu-berlin.de

Note that other universities may require other setups.

  1. I have created this certificate file with openssl x509 -inform PEM -outform DER -in CA.pem -out hu-ca-2024.crt using the CA.pem extracted from the eduroam setup for my PC. ↩︎

Pagination