A tool for testing Ceph locally using nested rootless podman containers

ceph-devstack is a tool that can deploy and manage containerized versions of teuthology and its associated services, to test Ceph (or just teuthology) on your local machine. It lets you avoid:

• Accessing Ceph's Sepia lab
• Needing dedicated storage devices to test Ceph OSDs

Basically, the goal is that you can test your Ceph branch locally using containers as storage test nodes.

It is currently under active development and has not yet had a formal release.

☑︎ CentOS 9.Stream should work out of the box

☑︎ CentOS 8.Stream mostly works - but has not yet passed a Ceph test

☐ A recent Fedora should work but has not been tested

☒ Ubuntu does not currently ship a new enough podman

☒ MacOS will require special effort to support since podman operations are done inside a VM

• A supported operating system
• podman 4.0+ using the crun runtime.
  • On CentOS 8, modify /etc/containers/containers.conf to set the runtime
• Linux kernel 5.12+, or 4.15+ and fuse-overlayfs
• cgroup v2
  • On CentOS 8, see ./docs/cgroup_v2.md
• With podman <5.0, podman's DNS plugin, from the podman-plugins package
• A user account that has sudo access and also is a member of the disk group
• The following sysctl settings:
  • fs.aio-max-nr=1048576
  • kernel.pid_max=4194304
• If using SELinux in enforcing mode:
  • setsebool -P container_manage_cgroup=true
  • setsebool -P container_use_devices=true

ceph-devstack doctor will check the above and report any issues along with suggested remedies; its --fix flag will apply them for you.

sudo usermod -a -G disk $(whoami)  # and re-login afterward
git clone https://github.com/ceph/teuthology/
cd teuthology && ./bootstrap
python3 -m venv venv
source ./venv/bin/activate
python3 -m pip install git+https://github.com/zmc/ceph-devstack.git

ceph-devstack 's default configuration is here. It can be extended by placing a file at ~/.config/ceph-devstack/config.toml or by using the --config-file flag.

ceph-devstack config dump will output the current configuration.

To back loop devices with slices of a real NVMe device instead of sparse files, configure [block_pool] in your config. The pool is shared across ceph_node and all testnode_* containers. Each loop device uses that container's loop_device_size for its backing region on the pool parent.

[block_pool]
parent = "/dev/nvme0n1p1"   # or a dedicated whole disk
allow_enroll = true         # first run only; unset afterward

[containers.ceph_node]
loop_device_size = "5G"

[containers.testnode]
loop_device_size = "5G"

Requirements:

• Your user must be in the disk group and able to read/write the parent device directly (enrollment writes a tail marker to the raw device).

• On a Fedora CoreOS podman machine (macOS), the disk group is defined in /usr/lib/group, so usermod -aG disk core silently does nothing until the group line is copied into /etc/group. ceph-devstack doctor --fix does both; by hand, inside podman machine ssh:

  grep -q '^disk:' /etc/group || grep '^disk:' /usr/lib/group | sudo tee -a /etc/group
  sudo usermod -aG disk core

  New podman machine ssh sessions then have the group.

• The parent must be empty, or already enrolled by ceph-devstack (tail marker at the end of the device).

ceph-devstack block-pool status shows current allocations.

All pool device access (sysfs, findmnt, blkid, and the marker and probe reads/writes via dd) runs on the container host, so a pool inside a podman machine (macOS, RemoteHost) should work, but that setup is unverified. The pool state file and its lock always stay on the local machine.

Each container picks host loop devices at create time: it needs loop_device_count devices of loop_device_size, reuses any existing backing files for that container name, then takes the lowest free loop numbers after scanning what is already attached or claimed in any stack's disk_images directory.

If block_pool.json is lost but the on-disk marker remains, start again to reclaim the marker; the device is not reformatted. The old allocations are forgotten, though, so regions that still hold data are refused as not empty until they are wiped.

To run a single-container local Ceph cluster instead of teuthology:

ceph-devstack --stack ceph pull
ceph-devstack --stack ceph create
ceph-devstack --stack ceph start
podman logs -f ceph_node
podman exec ceph_node ceph -s

The default image, quay.ceph.io/ceph-ci/ceph:main, is amd64-only. On arm64 hosts, including Apple Silicon podman machines, OSDs can't be created under emulation (io_setup fails with ENOSYS), so use an arm64 image instead:

[containers.ceph_node]
image = "quay.io/ceph/ceph:v20"

The dashboard settings (port, user, password, SSL) are under [containers.ceph_node]; see the defaults in ceph_devstack/config.toml. Set dashboard_show_password = true to print the password in the container log. Cluster state lives in the stack's cluster directory and loop device backing files in its disk_images directory; remove deletes both. Without a block pool, create also clears an old cluster directory, since the fresh sparse backing files would not match it.

ceph-devstack prepares the cluster directory (entrypoint script, dashboard password) on the local machine, and podman bind-mounts it from the container host, so the path must be visible on both. With a RemoteHost (e.g. a podman machine on macOS) that should work for paths shared into the machine, but it is unverified.

With image_builder = "binary-patch" (the default), the build-ceph stack:

1. Compiles Ceph binaries via src/script/build-with-container.py
2. Patches the compiled binaries into a base container image (base_image, default quay.ceph.io/ceph-ci/ceph:main) using src/script/cpatch.py, producing a local image (default localhost/ceph-cpatch:main)

Point the ceph stack at that image:

[containers.ceph_builder]
repo = "~/src/ceph"
# image_builder = "binary-patch"  # default

[containers.ceph_node]
image = "localhost/ceph-cpatch:main"

Then run the two stacks manually:

ceph-devstack --stack build-ceph start
ceph-devstack --stack ceph start

Or use depends to combine them (see below). This mode is faster for iterative development since it compiles only the changed C++ targets and patches them into an existing image rather than rebuilding RPM packages.

With image_builder = "package-build" under [containers.ceph_builder] (RPM distros such as centos9 only), the build-ceph stack:

1. Builds packages via src/script/build-with-container.py
2. Indexes the RPM tree (createrepo) and serves it briefly over HTTP on the build host (inside the Podman machine on macOS)
3. Runs upstream container/build.sh with CUSTOM_CEPH_REPO_URL pointing at a generated yum .repo file, then tags the result as a local image (default localhost/ceph-devstack:main)

Point the ceph stack at that image:

[containers.ceph_builder]
repo = "~/src/ceph"
image_builder = "package-build"
build_distro = "centos9"

[containers.ceph_node]
image = "localhost/ceph-devstack:main"

Then run the two stacks manually:

ceph-devstack --stack build-ceph start
ceph-devstack --stack ceph start

Or use depends to combine them into a single command. Add this to your user config (~/.config/ceph-devstack/config.toml):

[stacks.ceph]
depends = ["build-ceph"]

Then ceph-devstack --stack ceph start will automatically run the build-ceph stack first, compiling Ceph and producing the runtime image before deploying the cluster. stop and remove do not cascade into dependency stacks, so build caches are preserved.

The teuthology stack includes two services that bridge locally-built artifacts into test jobs:

• package_repo — an HTTP server exposing built RPMs at http://package_repo:8080/rpm/ on the ceph-devstack network
• registry — an OCI distribution registry at registry:5000 for locally-built container images

Both are included in stacks.teuthology.services by default. Whenever they are deployed, ceph-devstack:

1. Mounts the RPM output directory from build-ceph into package_repo
2. Configures testnodes to trust registry:5000 as an insecure registry
3. Pushes the image built from containers.ceph_builder.repo (if set) into the registry once all containers are running

Only with local_artifacts = true set on the stack does teuthology also test them: it deep-merges local artifact settings into teuthology's .teuthology.yaml, so that install tasks pull packages from the local repo and cephadm tasks pull images from the local registry, and it schedules the repo's own commit instead of resolving one upstream.

First, configure build-ceph to compile from your local checkout (~/.config/ceph-devstack/config.toml):

[containers.ceph_builder]
repo = "~/src/ceph"
image_builder = "package-build"
build_distro = "centos9"

[stacks.teuthology]
depends = ["build-ceph"]
local_artifacts = true

Then start the teuthology stack. The depends declaration causes build-ceph to run first, compiling packages and producing a container image:

ceph-devstack --stack teuthology start

After the build completes, package_repo immediately begins serving the RPMs. Once all containers are running, ceph-devstack also pushes the built container image into the registry, under its local tag (e.g. main) and the CI-pattern tags <branch>-<distro>-<arch>-devel, <branch>-<distro> and <sha1>-<distro>; cephadm tasks are pointed at the last one.

Now schedule a test suite that exercises local artifacts:

export TEUTHOLOGY_SUITE=orch:cephadm:smoke-small
ceph-devstack --stack teuthology start

Check that the package repo is serving RPMs:

podman exec package_repo ls /packages/rpm/

Check that the registry has your image:

podman exec registry wget -qO- http://localhost:5000/v2/_catalog

If you prefer to run stacks manually rather than using depends:

ceph-devstack --stack build-ceph start
ceph-devstack --stack teuthology start

If you don't need local artifacts (e.g. testing against upstream packages), the simplest approach is to set local_artifacts = false (the default):

[stacks.teuthology]
local_artifacts = false

This keeps teuthology testing upstream builds (its .teuthology.yaml gets no local artifact settings) while the services keep running for manual use. To also stop the containers from being created, remove package_repo and registry from your teuthology stack config:

[stacks.teuthology]
services = [
    "postgres",
    "paddles",
    "beanstalk",
    "pulpito",
    "teuthology",
    "testnode",
    "archive",
]

When these services are absent, testnodes will not receive the insecure registry config and teuthology's config will not point at them.

As an example, the following configuration will use a local image for paddles with the tag TEST; it will also create ten testnode containers; and will build its teuthology container from the git repo at ~/src/teuthology:

[containers.paddles]
image = "localhost/paddles:TEST"

[containers.testnode]
count = 10

[containers.teuthology]
repo = "~/src/teuthology"
build_image = true

By default, pre-built container images are pulled from quay.io/ceph-infra. The images can be overridden via the config file. It's also possible to build images from on-disk git repositories.

First, you'll want to pull all the images:

ceph-devstack pull

Optional: If building any images from repos:

ceph-devstack build

Next, you can start the containers with:

ceph-devstack start

Once everything is started, a message similar to this will be logged:

View test results at http://smithi065.front.sepia.ceph.com:8081/

This link points to the running Pulpito instance. Test archives are also stored in <data_dir>/teuthology/archive (data_dir defaults to ~/.local/share/ceph-devstack).

To watch teuthology's output, you can:

podman logs -f teuthology

If you want testnode containers to be replaced as they are stopped and destroyed, you can:

ceph-devstack watch

When finished, this command removes all the resources that were created:

ceph-devstack remove

By default, we run the teuthology:no-ceph suite to self-test teuthology. If we wanted to test Ceph itself, we could use the orch:cephadm:smoke-small suite:

export TEUTHOLOGY_SUITE=orch:cephadm:smoke-small

It's possible to skip the automatic suite-scheduling behavior:

export TEUTHOLOGY_SUITE=none

If you need to use "real" testnodes and have access to a lab, there are a few additonal steps to take. We will use the Sepia lab as an example below:

To give the teuthology container access to your SSH private key (via podman secret):

export SSH_PRIVKEY_PATH=$HOME/.ssh/id_rsa

To lock machines from the lab:

ssh teuthology.front.sepia.ceph.com
~/teuthology/virtualenv/bin/teuthology-lock \
  --lock-many 1 \
  --machine-type smithi \
  --desc "teuthology dev testing"

Once you have your machines locked, you need to provide a list of their hostnames and their machine type:

export TEUTHOLOGY_TESTNODES="smithiXXX.front.sepia.ceph.com,smithiYYY.front.sepia.ceph.com"
export TEUTHOLOGY_MACHINE_TYPE="smithi"

1. First fork the repo if you have not done so.
2. Clone your forked repo

git clone https://github.com/<user-name>/ceph-devstack

3. Setup the remote repo as upstream (this will prevent creating additional branches)

git remote add upstream https://github.com/zmc/ceph-devstack

4. Create virtual env in the root directory of ceph-devstack & install python dependencies

python3 -m venv venv
./venv/bin/pip3 install -e .

5. Activate venv

source venv/bin/activate

6. Run doctor command to check & fix the dependencies that you need for ceph-devstack

ceph-devstack -v doctor --fix

7. Build, Create and Start the all containers in ceph-devstack

ceph-devstack -v build
ceph-devstack -v create
ceph-devstack -v start

8. Test the containers by waiting for teuthology to finish and print the logs

ceph-devstack wait teuthology
podman logs -f teuthology