RDO's continuous packaging platform

How to continuously package OpenStack (or other things) for RPM-based Distributions

Fabien Boucher - Haïkel Guémar - Matthieu Huin

Feb. 2017

About RDO

  • the RPM Distribution of OpenStack...

About RDO

  • the RPM Distribution of OpenStack...

  • But also a group of Ridiculously Dedicated OpenStackers...

About RDO

  • the RPM Distribution of OpenStack...

  • But also a group of Ridiculously Dedicated OpenStackers...

About RDO

  • the RPM Distribution of OpenStack...

  • But also a group of Ridiculously Dedicated OpenStackers...

  • 250+ packages maintained, and growing

What's in a Package?

(basically)

<Name>-<Version>-<Release>.rpm

What's in a Package?

(basically)

<Name>-<Version>-<Release>.rpm

Sources

Upstream code

What's in a Package?

(basically)

<Name>-<Version>-<Release>.rpm

Sources

SPEC

Upstream code

  • building steps
  • installation/removal steps
  • dependencies
  • patches
  • ...

Packager=Spec Dev

Packagers have the same needs as open source developers...

  • Version control (distgit), on a public platform, community oriented

Packager=Spec Dev

Packagers have the same needs as open source developers...

  • Version control (distgit), on a public platform, community oriented
  • Peer review

Packager=Spec Dev

Packagers have the same needs as open source developers...

  • Version control (distgit), on a public platform, community oriented
  • Peer review
  • Automated testing & validation of contributions (ie continuous integration), on a controlled testing environment

Packager=Spec Dev

Packagers have the same needs as open source developers...

  • Version control (distgit), on a public platform, community oriented
  • Peer review
  • Automated testing & validation of contributions (ie continuous integration), on a controlled testing environment
  • Smart merge management

Packager=Spec Dev

...With Specificities due to upstream

  • Version control (distgit), on a public platform, community oriented
  • Peer review
  • Automated testing & validation of contributions (ie continuous integration), on a controlled testing environment
  • Smart merge management
  • Follow upstream changes in order to validate spec files against them

Packager=Spec Dev

...With Specificities due to upstream

  • Version control (distgit), on a public platform, community oriented
  • Peer review
  • Automated testing & validation of contributions (ie continuous integration), on a controlled testing environment
  • Smart merge management
  • Follow upstream changes in order to validate spec files against them
  • Release as closely as possible to upstream (esp. vulnerability or security fixes)

Packaging Openstack

It cranks it up to 11

Releases every 6 months, statistics for the last cycle (Newton):

  • 2700+ contributors merged 43000+ commits upstream across 600+ projects, or ~240 commits/day

Packaging Openstack

It cranks it up to 11

Releases every 6 months, statistics for the last cycle (Newton):

  • 2700+ contributors merged 43000+ commits upstream across 600+ projects, or ~240 commits/day
  • Nova (compute component) alone: ~1700 commits / ~10 commits/day

Packaging Openstack

It cranks it up to 11

Releases every 6 months, statistics for the last cycle (Newton):

  • 2700+ contributors merged 43000+ commits upstream across 600+ projects, or ~240 commits/day
  • Up to ~280 commits/day in the last month of the cycle (bug fixes, release candidates)
  • Nova (compute component) alone: ~1700 commits / ~10 commits/day

Packaging Openstack

It cranks it up to 11

Extra constraints for RDO Packagers:

  • Spec files must be validated after each upstream change. Waiting for the tag would induce too much to catch up with

Packaging Openstack

It cranks it up to 11

Extra constraints for RDO Packagers:

  • Spec files must be validated after each upstream change. Waiting for the tag would induce too much to catch up with
  • Strong dependencies between packages: if a core package is modified, all other packages must be tested for compatibility

Packaging Openstack

It cranks it up to 11

Extra constraints for RDO Packagers:

  • Spec files must be validated after each upstream change. Waiting for the tag would induce too much to catch up with
  • Strong dependencies between packages: if a core package is modified, all other packages must be tested for compatibility
  • The test platform must be able to handle heavy surges and parallelize tasks

BuildinG aN RPM FACTORY

BuildinG aN RPM FACTORY

RDO Tools

BuildinG aN RPM FACTORY

RDO Tools

  • DLRN is a tool to continuously test the packaging recipes against OpenStack's upstream master to detect packaging issues early

BuildinG aN RPM FACTORY

RDO Tools

  • DLRN is a tool to continuously test the packaging recipes against OpenStack's upstream master to detect packaging issues early

  • It can generate a full test package repository to test interdependencies, and generally acts as the "master RPM" repository

BuildinG aN RPM FACTORY

RDO Tools

  • DLRN is a tool to continuously test the packaging recipes against OpenStack's upstream master to detect packaging issues early

  • It can generate a full test package repository to test interdependencies, and generally acts as the "master RPM" repository

BuildinG aN RPM FACTORY

RDO Tools

  • DLRN is a tool to continuously test the packaging recipes against OpenStack's upstream master to detect packaging issues early

  • It can generate a full test package repository to test interdependencies, and generally acts as the "master RPM" repository

  • RDOPKG is a CLI used to automate some packaging tasks like flattening patch chains for inclusion in specs (more on that later)

BuildinG aN RPM FACTORY

Community Tools

BuildinG aN RPM FACTORY

Community Tools

  • Koji is a software collection (client & server) that can build and store RPM packages

BuildinG aN RPM FACTORY

Community Tools

  • Koji is a software collection (client & server) that can build and store RPM packages

  • Koji provides a fresh environment (buildroot) for each build

BuildinG aN RPM FACTORY

Community Tools

  • Koji is a software collection (client & server) that can build and store RPM packages

  • Koji provides a fresh environment (buildroot) for each build

BuildinG aN RPM FACTORY

Software Factory

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

  • Code hosting and review (Gerrit)

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

  • Code hosting and review (Gerrit)

  • Jobs orchestration (Zuul & Jenkins)

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

  • Code hosting and review (Gerrit)

  • Jobs orchestration (Zuul & Jenkins)

  • Project dependencies management when building testing environments (Zuul)

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

  • Code hosting and review (Gerrit)

  • Jobs orchestration (Zuul & Jenkins)

  • Project dependencies management when building testing environments (Zuul)

  • Jobs executors spawned & terminated on demand on an OpenStack cloud (Nodepool)

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

  • Code hosting and review (Gerrit)

  • Jobs orchestration (Zuul & Jenkins)

  • Project dependencies management when building testing environments (Zuul)

  • Jobs executors spawned & terminated on demand on an OpenStack cloud (Nodepool)

  • Smart commit gating (Zuul)

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

  • Code hosting and review (Gerrit)

  • Jobs orchestration (Zuul & Jenkins)

  • Project dependencies management when building testing environments (Zuul)

  • Jobs executors spawned & terminated on demand on an OpenStack cloud (Nodepool)

  • Smart commit gating (Zuul)

  • config: (almost) everything as code

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

  • Code hosting and review (Gerrit)

  • Jobs orchestration (Zuul & Jenkins)

  • Project dependencies management when building testing environments (Zuul)

  • Jobs executors spawned & terminated on demand on an OpenStack cloud (Nodepool)

  • Smart commit gating (Zuul)

  • config: (almost) everything as code

  • Flexible workflow, based on reviewing and testing code prior to merging it

BuildinG aN RPM FACTORY

Software Factory

  • SF is a CI/CD platform based on OpenStack's own CI ("Infra")

  • Code hosting and review (Gerrit)

  • Jobs orchestration (Zuul & Jenkins)

  • Project dependencies management when building testing environments (Zuul)

  • Jobs executors spawned & terminated on demand on an OpenStack cloud (Nodepool)

  • Smart commit gating (Zuul)

  • config: (almost) everything as code

  • Flexible workflow, based on reviewing and testing code prior to merging it

  • 💛💛💛

BuildinG aN RPM FACTORY

With our powers combined...

  • The community platform for building RDO
  • A Software Factory deployment branded for RDO
  • Hosts all the distgit repositories + extra patches repositories
  • All upstream changes acted upon with DLRN
  • All distgit changes reviewed via Gerrit
  • All distgit changes tested via Zuul + Jenkins + Nodepool + CBS

workflow overview

Use Cases

1. Change on upstream's Master

Upstream

openstack/nova:master

review.rdoproject.org

nova-distgit:rpm-master

DLRN testing

  • DLRN uses the packaging recipes from review.rdoproject.org's Git repositories (rpm-master branches) to build RPMs

Use Cases

1. Change on upstream's Master

Upstream

openstack/nova:master

review.rdoproject.org

nova-distgit:rpm-master

DLRN testing

  • DLRN uses the packaging recipes from review.rdoproject.org's Git repositories (rpm-master branches) to build RPMs

Use Cases

1. Change on upstream's Master

Upstream

openstack/nova:master

review.rdoproject.org

nova-distgit:rpm-master

DLRN testing

  • DLRN uses the packaging recipes from review.rdoproject.org's Git repositories (rpm-master branches) to build RPMs

  • In case of a build failure, DLRN will submit a placeholder patch through review.rdoproject.org's Gerrit on the corresponding repository(ies) (rpm-master branch)

Use Cases

1. Change on upstream's Master

Upstream

openstack/nova:master

review.rdoproject.org

nova-distgit:rpm-master

DLRN testing

  • DLRN uses the packaging recipes from review.rdoproject.org's Git repositories (rpm-master branches) to build RPMs

  • In case of a build failure, DLRN will submit a placeholder patch through review.rdoproject.org's Gerrit on the corresponding repository(ies) (rpm-master branch)

  • The repository(ies) maintainers are set automatically as reviewers on the patch

  • This empty patch will obviously fail the CI tests on review.rdoproject.org and incite the maintainers to fix the packaging issue

Use Cases

2. Change on a stable branch's distgit

Upstream

nova:stable/newton

review.rdoproject.org

nova-distgit:newton-rdo

CentOS Build System

target: newton

  • New changes on stable branches trigger a test "scratch build" on CBS automatically. The build results are shown:
    • as a CI score on review.rdoproject.org's Gerrit (Verified +1/-1)
    • as a link to the CBS build logs on the review page
    • as built artifacts (RPMs) fetched from CBS and copied to the executor node

RDO Stable CentOS Repository

  • An open review is created on review.rdoproject.org

Use Cases

2. Change on a stable branch's distgit

Upstream

nova:stable/newton

review.rdoproject.org

nova-distgit:newton-rdo

CentOS Build System

target: newton

  • In case of build success:
    • The built RPMs are mashed into the target's repository
    • The change is merged / review closed

RDO Stable CentOS Repository

  • If a verified patch is approved by a core maintainer, it is gated for merge
  • A "non-scratch" build is triggered on CBS

Use Cases

2. Change on a stable branch's distgit

Upstream

nova:stable/newton

review.rdoproject.org

nova-distgit:newton-rdo

CentOS Build System

target: newton

  • In case of build failure:
    • The resulting artifacts are discarded
    • The review gets a Verified -2 score and remains open
    • The patch is not merged

RDO Stable CentOS Repository

  • In case of build success:
    • The built RPMs are mashed into the target's repository
    • The change is merged / review closed
  • If a verified patch is approved by a core maintainer, it is gated for merge
  • A "non-scratch" build is triggered on CBS

Use Cases

3. Distro-specific patches

Upstream

nova:newton

review.rdoproject.org

nova-distgit:newton-rdo

CentOS Build System

  • Patches needed for packaging are stored on Gerrit as permanently open reviews on top of the upstream clone
  • Patches can form a chain of reviews (chain of patches)

review.rdoproject.org

nova:newton-patches

nova:newton-rdo-patches

Use Cases

3. Distro-specific patches

Upstream

nova:newton

review.rdoproject.org

nova-distgit:newton-rdo

CentOS Build System

  • Patches needed for packaging are stored on Gerrit as permanently open reviews on top of the upstream clone
  • Patches can form a chain of reviews (chain of patches)

review.rdoproject.org

nova:newton-patches

nova:newton-rdo-patches

  • Advantages of the review workflow:
    • Rebasing is easier (Gerrit UI)
    • Gerrit keeps the history of the chain
    • Quality ensured by testing the patches chain
    • Dealing with patches this way is easier when multiple packagers work on the same package

Return on experience

  • ~800 commits

RDO stats for the Newton cycle

Return on experience

  • ~800 commits
  • ~70 contributors

RDO stats for the Newton cycle

Return on experience

  • ~800 commits
  • ~70 contributors

RDO stats for the Newton cycle

Return on experience

  • ~800 commits
  • ~70 contributors

RDO stats for the Newton cycle

  • RDO Newton packages were available 10 hours after upstream release

better processes, lower entry costs

  • Automate the distribution pipeline, like continuous delivery

Return on experience

better processes, lower entry costs

  • Automate the distribution pipeline, like continuous delivery

Return on experience

better processes, lower entry costs

  • Automate the distribution pipeline, like continuous delivery

Return on experience

  • Gerrit helps building a community of contributors
    • Code transparency
    • Peer review
    • Faster onboarding

Join us

like Other open source projects hosted on public instances

softwarefactory-project.io

  • Software Factory
  • Distributed CI
  • Skydive

review.rdoproject.org

  • RDO
  • DLRN
  • Opstools

Your projects, on your instance ?

Thank you

  • https://softwarefactory-project.io
  • https://review.rdoproject.org
  • Freenode:
    • #softwarefactory
    • #rdo
  • mailing lists:
    • softwarefactory-dev@redhat.com
    • rdo-list@redhat.com

Keep in touch

RDO's continuous packaging platform

By Matthieu Huin

RDO's continuous packaging platform

How to continuously package OpenStack (or other things) for CentOS