Adding new apt package¶

The following example adds vim to the Airflow image. When adding packages via apt you should switch to the root user when running the apt commands, but do not forget to switch back to the airflow user after installation is complete.

FROM apache/airflow:3.4.0
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         vim \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow

Adding new PyPI packages individually¶

The following example adds lxml python package from PyPI to the image. When adding packages via pip you need to use the airflow user rather than root. Attempts to install pip packages as root will fail with an appropriate error message.

FROM apache/airflow:3.4.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml

Adding packages from requirements.txt¶

The following example adds few python packages from requirements.txt from PyPI to the image. Note that similarly when adding individual packages, you need to use the airflow user rather than root. Attempts to install pip packages as root will fail with an appropriate error message.

FROM apache/airflow:3.4.0
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt

lxml
beautifulsoup4

Embedding dags¶

The following example adds test_dag.py to your image in the /opt/airflow/dags folder.

FROM apache/airflow:3.4.0

COPY --chown=airflow:root test_dag.py /opt/airflow/dags

import datetime

import pendulum

from airflow.models.dag import DAG
from airflow.providers.standard.operators.empty import EmptyOperator

now = pendulum.now(tz="UTC")
now_to_the_hour = (now - datetime.timedelta(0, 0, 0, 0, 0, 3)).replace(minute=0, second=0, microsecond=0)
START_DATE = now_to_the_hour
DAG_NAME = "test_dag_v1"

dag = DAG(
    DAG_NAME,
    schedule="*/10 * * * *",
    default_args={"depends_on_past": True},
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
)

run_this_1 = EmptyOperator(task_id="run_this_1", dag=dag)
run_this_2 = EmptyOperator(task_id="run_this_2", dag=dag)
run_this_2.set_upstream(run_this_1)
run_this_3 = EmptyOperator(task_id="run_this_3", dag=dag)
run_this_3.set_upstream(run_this_2)

Add Airflow configuration with environment variables¶

The following example adds Airflow configuration to the image. airflow.cfg file in $AIRFLOW_HOME directory contains Airflow’s configuration. You can set options with environment variables for those Airflow’s configuration by using this format: AIRFLOW__{SECTION}__{KEY} (note the double underscores).

FROM apache/airflow:3.4.0
ENV AIRFLOW__CORE__LOAD_EXAMPLES=True
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=my_conn_string

Base images¶

There are two types of images you can extend your image from:

1. Regular Airflow image that contains the most common extras and providers, and all supported backend database clients for AMD64 platform and Postgres for ARM64 platform.

2. Slim Airflow image, which is a minimal image, contains all supported backends database clients installed for AMD64 platform and Postgres for ARM64 platform, but contains no extras or providers, except the 4 default providers.

Naming conventions for the images:

• The “latest” image is always the latest released stable version available.

Example of setting own Airflow providers¶

The Airflow Providers are released independently of core Airflow and sometimes you might want to upgrade specific providers only to fix some problems or use features available in that provider version. Here is an example of how you can do it

FROM apache/airflow:3.4.0
RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir apache-airflow-providers-docker==2.5.1

Example of adding Airflow Provider package and apt package¶

The following example adds apache-spark airflow-providers which requires both java and python package from PyPI.

FROM apache/airflow:3.4.0
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         openjdk-21-jre-headless \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow
ENV JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" apache-airflow-providers-apache-spark==5.1.1

Example of adding apt package¶

The following example adds vim to the Airflow image.

FROM apache/airflow:3.4.0
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         vim \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow

Example of adding PyPI package¶

The following example adds lxml python package from PyPI to the image.

FROM apache/airflow:3.4.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml

Example of adding PyPI package with constraints¶

The following example adds lxml python package from PyPI to the image with constraints that were used to install airflow. This allows you to use the version of packages that you know were tested with the given version of Airflow. You can also use it if you do not want to use potentially newer versions that were released after the version of Airflow you are using.

FROM apache/airflow:3.4.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml --constraint "${HOME}/constraints.txt"

Example of adding PyPI package with uv¶

The following example adds lxml python package from PyPI to the image using uv. This is an experimental feature as uv is a very fast but also very new tool in the Python ecosystem.

FROM apache/airflow:3.4.0

# The `uv` tools is Rust packaging tool that is much faster than `pip` and other installers
# Support for uv as installation tool is experimental

RUN uv pip install --no-cache "apache-airflow==${AIRFLOW_VERSION}" lxml

Example of adding packages from requirements.txt¶

The following example adds few python packages from requirements.txt from PyPI to the image. Note that similarly when adding individual packages, you need to use the airflow user rather than root. Attempts to install pip packages as root will fail with an appropriate error message.

FROM apache/airflow:3.4.0
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt

lxml
beautifulsoup4

Example of adding a Python venv for ExternalPythonOperator¶

The following example creates a separate Python virtualenv inside the image and installs packages from requirements.txt into it, leaving Airflow’s own Python environment untouched. This is the typical setup for the ExternalPythonOperator (or the @task.external_python decorator), which executes tasks in a pre-existing, immutable Python environment. Use /opt/airflow/venv/bin/python as the operator’s python argument to run tasks in this venv.

The venv is created without --system-site-packages, so the packages installed inside it are isolated from Airflow’s own dependencies.

FROM apache/airflow:3.4.0
COPY requirements.txt /
RUN python -m venv /opt/airflow/venv \
    && /opt/airflow/venv/bin/pip install --no-cache-dir -r /requirements.txt

colorama==0.4.0

Example when writable directory is needed¶

The following example adds a new directory that is supposed to be writable for any arbitrary user running the container.

FROM apache/airflow:3.4.0
RUN umask 0002; \
    mkdir -p ~/writeable-directory

Example when you add packages requiring compilation¶

The following example adds mpi4py package which requires both build-essential and mpi compiler.

FROM apache/airflow:3.4.0
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         build-essential libopenmpi-dev \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" "mpi4py==4.1.1"

The size of this image is ~ 1.1 GB when build. As you will see further, you can achieve 20% reduction in size of the image in case you use “Customizing” rather than “Extending” the image.

Example when you want to embed dags¶

The following example adds test_dag.py to your image in the /opt/airflow/dags folder.

FROM apache/airflow:3.4.0

COPY --chown=airflow:root test_dag.py /opt/airflow/dags

import datetime

import pendulum

from airflow.models.dag import DAG
from airflow.providers.standard.operators.empty import EmptyOperator

now = pendulum.now(tz="UTC")
now_to_the_hour = (now - datetime.timedelta(0, 0, 0, 0, 0, 3)).replace(minute=0, second=0, microsecond=0)
START_DATE = now_to_the_hour
DAG_NAME = "test_dag_v1"

dag = DAG(
    DAG_NAME,
    schedule="*/10 * * * *",
    default_args={"depends_on_past": True},
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
)

run_this_1 = EmptyOperator(task_id="run_this_1", dag=dag)
run_this_2 = EmptyOperator(task_id="run_this_2", dag=dag)
run_this_2.set_upstream(run_this_1)
run_this_3 = EmptyOperator(task_id="run_this_3", dag=dag)
run_this_3.set_upstream(run_this_2)

Example of changing Airflow configuration using environment variables¶

The following example adds Airflow configuration changes to the Airflow image.

FROM apache/airflow:3.4.0
ENV AIRFLOW__CORE__LOAD_EXAMPLES=True
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=my_conn_string

Building from PyPI packages¶

This is the basic way of building the custom images from sources.

The following example builds the production image for the default Python version with latest PyPI-released Airflow, with default set of Airflow extras and dependencies. The latest PyPI-released Airflow constraints are used automatically.

export DOCKER_BUILDKIT=1

docker build . \
    --tag "my-stable-airflow:0.0.1"

The following example builds the production image for the default Python version with default extras from 2.3.0 Airflow package. The 2.3.0 constraints are used automatically.

export AIRFLOW_VERSION=3.0.3
export DOCKER_BUILDKIT=1

docker build . \
    --build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.13.18-debian13-dev}" \
    --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
    --tag "my-pypi-selected-version:0.0.1"

The following example builds the production image for the default Python version with additional Airflow extras (mssql,hdfs) from 2.3.0 PyPI package, and additional dependency (oauth2client).

export AIRFLOW_VERSION=3.0.3
export DOCKER_BUILDKIT=1

docker build . \
    --pull \
    --build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.13.18-debian13-dev}" \
    --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
    --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" \
    --build-arg ADDITIONAL_PYTHON_DEPS="oauth2client" \
    --tag "my-pypi-extras-and-deps:0.0.1"

The following example adds mpi4py package which requires both build-essential and mpi compiler.

export AIRFLOW_VERSION=3.0.3
export DOCKER_BUILDKIT=1

docker build . \
    --pull \
    --build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.13.18-debian13-dev}" \
    --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
    --build-arg ADDITIONAL_PYTHON_DEPS="mpi4py==4.1.1" \
    --build-arg ADDITIONAL_DEV_APT_DEPS="libopenmpi-dev" \
    --build-arg ADDITIONAL_RUNTIME_APT_DEPS="openmpi-common" \
    --tag "my-build-essential-image:0.0.1"

The above image is equivalent of the “extended” image from previous chapter but its size is only 874 MB. Comparing to 1.1 GB of the “extended image” this is about 230 MB less, so you can achieve ~20% improvement in size of the image by using “customization” vs. extension. The saving can increase in case you have more complex dependencies to build.

Building optimized images¶

The following example builds the production image for the default Python version with additional Airflow extras from PyPI package but it includes additional apt dev and runtime dependencies.

The dev dependencies are those that require build-essential and usually need to involve recompiling of some python dependencies so those packages might require some additional DEV dependencies to be present during recompilation. Those packages are not needed at runtime, so we only install them for the “build” time. They are not installed in the final image, thus producing much smaller images. In this case pandas requires recompilation so it also needs gcc and g++ as dev APT dependencies. The jre-headless does not require recompiling so it can be installed as the runtime APT dependency.

export AIRFLOW_VERSION=3.0.3
export DOCKER_BUILDKIT=1

docker build . \
    --pull \
    --build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.13.18-debian13-dev}" \
    --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
    --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" \
    --build-arg ADDITIONAL_PYTHON_DEPS="pandas==2.1.2" \
    --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" \
    --build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless" \
    --tag "my-pypi-dev-runtime:0.0.1"

Building prod images using UV as the package installer¶

The following example builds the production image in default settings, but uses uv to build the image. This is an experimental feature as uv is a very fast but also very new tool in the Python ecosystem.

export DOCKER_BUILDKIT=1
docker build . \
    --build-arg AIRFLOW_USE_UV="true" \
    --tag "my-custom-use-uv-airflow:0.0.1"

Building images with MySQL client¶

Using custom installation sources¶

You can customize more aspects of the image - such as additional commands executed before apt dependencies are installed, or adding extra sources to install your dependencies from. You can see all the arguments described below but here is an example of rather complex command to customize the image based on example in this comment:

In case you need to use your custom PyPI package indexes, you can also customize PYPI sources used during image build by adding a docker-context-files/pip.conf file when building the image. This pip.conf will not be committed to the repository (it is added to .gitignore) and it will not be present in the final production image. It is added and used only in the build segment of the image. Therefore this pip.conf file can safely contain list of package indexes you want to use, usernames and passwords used for authentication. More details about pip.conf file can be found in the pip configuration.

If you used the .piprc before (some older versions of pip used it for customization), you can put it in the docker-context-files/.piprc file and it will be automatically copied to HOME directory of the airflow user.

Note, that those customizations are only available in the build segment of the Airflow image and they are not present in the final image. If you wish to extend the final image and add custom .piprc and pip.conf, you should add them in your own Dockerfile used to extend the Airflow image.

Such customizations are independent of the way how Airflow is installed.

The following - rather complex - example shows capabilities of:

• Adding Airflow extras (slack, odbc)

• Adding PyPI dependencies (azure-storage-blob, oauth2client, beautifulsoup4, dateparser, rocketchat_API,typeform)

• Adding custom environment variables while installing apt dependencies - both DEV and RUNTIME (ACCEPT_EULA=Y')

• Adding custom curl command for adding keys and configuring additional apt sources needed to install apt dependencies (both DEV and RUNTIME)

• Adding custom apt dependencies, both DEV (msodbcsql17 unixodbc-dev g++) and runtime msodbcsql17 unixodbc git procps vim)

export AIRFLOW_VERSION=3.0.3
export DOCKER_BUILDKIT=1

docker build . -f Dockerfile \
    --pull \
    --platform 'linux/amd64' \
    --build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.13.18-debian13-dev}" \
    --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
    --build-arg ADDITIONAL_AIRFLOW_EXTRAS="slack,odbc" \
    --build-arg ADDITIONAL_PYTHON_DEPS=" \
        azure-storage-blob \
        oauth2client \
        beautifulsoup4 \
        dateparser \
        rocketchat_API \
        typeform" \
    --build-arg ADDITIONAL_DEV_APT_COMMAND="curl https://packages.microsoft.com/keys/microsoft.asc | \
    apt-key add --no-tty - && \
    curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list" \
    --build-arg ADDITIONAL_DEV_APT_ENV="ACCEPT_EULA=Y" \
    --build-arg ADDITIONAL_DEV_APT_DEPS="msodbcsql17 unixodbc-dev g++" \
    --build-arg ADDITIONAL_RUNTIME_APT_COMMAND="curl https://packages.microsoft.com/keys/microsoft.asc | \
    apt-key add --no-tty - && \
    curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list" \
    --build-arg ADDITIONAL_RUNTIME_APT_ENV="ACCEPT_EULA=Y" \
    --build-arg ADDITIONAL_RUNTIME_APT_DEPS="msodbcsql17 unixodbc git procps vim" \
    --tag "my-custom-sources-image:0.0.1"

Build images in security restricted environments¶

You can also make sure your image is only built using local constraint file and locally downloaded wheel files. This is often useful in Enterprise environments where the binary files are verified and vetted by the security teams. It is also the most complex way of building the image. You should be an expert of building and using Dockerfiles in order to use it and have to have specific needs of security if you want to follow that route.

Properties of the hardened base image¶

Since Airflow 3.4.0 the images are built on the Debian 13 “trixie” -dev variant of the Docker Hardened Image for Python (dhi.io/python, mirrored to ghcr.io/apache/airflow/base/python) instead of compiling Python on debian:bookworm-slim. Docker publishes the definitions these images are built from in the docker-hardened-images/catalog repository (package/python for the Python build, image/python for the image). The differences from the previous images that can matter when you extend or run the image are:

• Python is patched. Docker applies its own patches on top of the upstream CPython release (package/python/patch in the catalog). One of them makes pkgutil.get_data() reject resource paths that are absolute or contain .. (CVE-2026-3479) - upstream CPython only documents that restriction and does not enforce it. Libraries that load bundled data through ../ paths fail with ValueError: resource must be a relative path with no parent directory components; for example moto before 5.2.2.

• Threads get a 1 MiB stack. The hardened Python is compiled with -DTHREAD_STACK_SIZE=0x100000, while the previous images used the glibc default (the 8 MiB ulimit -s). On Python 3.12 and 3.13 the interpreter’s C recursion guard is a fixed depth that assumes the larger stack, so deeply nested data - for example a JSON body parsed in a web server worker thread - can crash the process instead of raising RecursionError. The image restores the 8 MiB default for threads started by Python with a airflow-thread-stack-size.pth file in the base Python’s site-packages; delete that file if you need the smaller stacks.

• Python is built without profile-guided optimization. The previous images were built with --enable-optimizations and --with-lto; the hardened Python uses only --with-lto, so CPU-bound Python code can run measurably slower.

• Python lives under /usr. The Debian 13 hardened images install Python as Debian packages (python-3.13, libpython-3.13, …), so it is /usr/bin/python3 with the standard library in /usr/lib/python3.13 (the Debian 12 ones used /opt/python). /usr/python and the /usr/local/bin links point to it, so the paths used before keep working. The standard library ships without .pyc files, so the build compiles it into the image.

• The OS is minimal. The base ships no compiler, curl, wget, git, gzip, which or ldconfig, no dash (/bin/sh is bash) and a stripped /etc - for example without /etc/shells or the PAM common-* files. The build restores what Airflow and the Debian packages it installs need; a custom image that relied on something else being present has to install it.

• Packages also come from Docker’s Debian repository. Next to Debian’s own repositories, the Debian 13 images configure http://dhi.io/deb/debian/main (/etc/apt/sources.list.d/dhi.sources, signed with /usr/share/keyrings/dhi-deb-main.gpg), which serves Docker’s rebuilds of Debian packages. Their versions carry a +dhi<N> suffix, so apt prefers them over Debian’s builds - in the image and in images extending it. The repository can be used anonymously: unlike pulling the dhi.io images, installing packages from it needs no docker login or any other credentials.

• Installing apt packages can pull in a second Python. The Debian 13 images register their own Python with dpkg (python-3.12, libpython-3.12, …), and Docker’s package repository serves the other Python versions as packages too. A Debian package that depends on python3 - directly or, as libenchant-2-dev does, through its build tools - therefore makes apt install another hardened Python version (for example python-3.13 into a Python 3.12 image), and on the Debian 12 images it pulls in Debian’s own python3. The second interpreter does not replace the image’s Python, but it can take over /usr/bin/python3 and its headers, and C extensions built afterwards then compile against the wrong Python. The image build fails when this happens, but a custom image that installs extra apt packages is not checked. Check what apt-get install --simulate would install before adding packages that might depend on python3, and prefer the runtime library packages over their -dev variants.

• The image’s Python packages are held. On the Debian 13 images, Debian’s own python3 packages conflict with the image’s python-3.X packages, so a package that apt can only satisfy with them - for example NodeSource’s nodejs - would make apt remove the image’s Python without asking. The image marks its Python packages with apt-mark hold, so apt reports the python3 conflict and the installation fails instead. Install that software some other way - for Node.js, copy the node binary from the official node image - rather than releasing the hold.

• C extensions are compiled more strictly. The Debian 13 images come with GCC 14, which turns incompatible pointer types, integer conversions and implicit function declarations into errors, and the hardened Python passes -Werror=implicit-function-declaration on to every C extension it builds. Old source distributions that compiled with warnings before - for example xmlsec 1.3.14 or ibm-db 3.2.0 - fail to build. Prefer versions with wheels or newer releases; if you have to build such a package, setting CFLAGS="-Wno-error=incompatible-pointer-types -Wno-error=int-conversion -Wno-error=implicit-function-declaration" for the installation turns the errors back into warnings.

• apt does not refresh the shared library cache. The base images ship libc-bin without its dpkg trigger, so installing a shared library does not run ldconfig, and anything that looks libraries up through the cache - such as ctypes.util.find_library() - does not find it. The image adds /etc/apt/apt.conf.d/99airflow-ldconfig, which runs ldconfig after every dpkg run, including those of images extending it.

• Some OS files are modified. /etc/os-release identifies the system as “Docker Hardened Images (Debian)”, and /etc/debian_version differs from the Debian package’s copy, so an apt-get upgrade that touches base-files would stop at dpkg’s interactive configuration-file prompt. The image ships /etc/dpkg/dpkg.cfg.d/airflow-keep-conffiles (force-confdef and force-confold), so dpkg keeps the image’s copy of a configuration file without asking - also in images that extend it. A package upgrade therefore does not replace a configuration file that differs from the package’s version.

• The tags are rebuilt in place, so the default is pinned to a digest. Docker republishes the same tags as CVEs are fixed, and Airflow refreshes its mirror every six hours, so building from a tag alone can pick up a newer base at any time. The default BASE_IMAGE therefore names the digest as well, and Airflow’s regular dependency upgrade moves it - in a pull request that has to pass the full CI before the new base is used. Pass a tag without a digest to build on the newest mirrored base instead. The mirror also copies the newest Python patch release Docker publishes, and the same upgrade moves the pinned patch level to it, so the images follow Docker’s releases rather than python.org’s.

• Packages come from Docker’s apt repository first. The image keeps stock Debian’s apt sources beside dhi.io, and /etc/apt/preferences.d/airflow-prefer-dhi ranks dhi.io higher, so apt installs Docker’s build of a package whenever dhi.io publishes it and falls back to Debian’s otherwise - also in images that extend it. The priority stays below 1000, so apt still favors dhi.io’s build but can take a newer version from another repository when a dependency needs it (for example PGDG’s libpq5).

Debian trixie and the legacy image flavor¶

Together with the hardened base, the default images moved from Debian 12 “bookworm” to Debian 13 “trixie”. Besides the newer system libraries, this shows up in two places when you extend the image:

• Several library packages were renamed in trixie’s 64-bit time_t transition - for example libssl3 is libssl3t64 and libldap-2.5-0 is libldap2 - so RUNTIME_APT_DEPS or ADDITIONAL_RUNTIME_APT_DEPS that name the bookworm packages have to be updated.

• The MySQL-compatible client comes from MariaDB 11.8, the first MariaDB LTS published for trixie. The mysql named commands are provided by mariadb-client-compat.

For the 3.4.x line Airflow also publishes legacy images, built as before 3.4.0: Python downloaded from python.org, verified with its sigstore signature and compiled from sources on debian:bookworm-slim. They are deprecated and will be removed in Airflow 3.5.0. You can build one with the AIRFLOW_IMAGE_FLAVOR build argument:

docker build . \
  --build-arg AIRFLOW_IMAGE_FLAVOR="legacy" \
  --build-arg BASE_IMAGE="debian:bookworm-slim" \
  --build-arg AIRFLOW_PYTHON_VERSION="3.13" \
  --tag my-image:my-tag

A MAJOR.MINOR AIRFLOW_PYTHON_VERSION builds the latest python.org release of that Python line; pass MAJOR.MINOR.PATCH to build a specific one.

The flavor an image was built with is recorded in its org.apache.airflow.image.flavor label.

Build images in FIPS-compliant environments¶

Airflow images are based on the Docker Hardened Images for Python, and Docker publishes FIPS-validated variants of those images. Those variants are only available with a paid Docker subscription, so they cannot be the default, but you can point the build at one with the BASE_IMAGE build argument.

docker build . --build-arg BASE_IMAGE="dhi.io/python:3.13.16-debian13-fips-dev" --tag my-image:my-tag

This builds below builds the production image with packages and constraints used from the local docker-context-files rather than installed from PyPI or GitHub. It also disables MySQL client installation as it is using external installation method.

Note that as a prerequisite - you need to have downloaded wheel files. In the example below we first download such constraint file locally and then use pip download to get the .whl files needed but in most likely scenario, those wheel files should be copied from an internal repository of such .whl files. Note that AIRFLOW_VERSION_SPECIFICATION is only there for reference, the Apache Airflow .whl file in the right version is part of the .whl files downloaded.

Note that ‘pip download’ will only works on Linux host as some of the packages need to be compiled from sources and you cannot install them providing --platform switch. They also need to be downloaded using the same python version as the target image.

The pip download might happen in a separate environment. The files can be committed to a separate binary repository and vetted/verified by the security team and used subsequently to build images of Airflow when needed on an air-gaped system.

Example of preparing the constraint files and wheel files. Note that mysql dependency is removed as mysqlclient is installed from Oracle’s apt repository and if you want to add it, you need to provide this library from your repository if you want to build Airflow image in an “air-gaped” system.

mkdir -p docker-context-files
export AIRFLOW_VERSION="3.0.3"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true

curl -Lo "docker-context-files/constraints-3.11.txt" \
    "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.11.txt"

echo
echo "Make sure you use the right python version here (should be same as in constraints)!"
echo
python --version

pip download --dest docker-context-files \
    --constraint docker-context-files/constraints-3.11.txt  \
    "apache-airflow[async,celery,cncf.kubernetes,elasticsearch,postgres,redis,ssh,statsd]==${AIRFLOW_VERSION}"

After this step is finished, your docker-context-files folder will contain all the packages that are needed to install Airflow from.

Those downloaded packages and constraint file can be pre-vetted by your security team before you attempt to install the image. You can also store those downloaded binary packages in your private artifact registry which allows for the flow where you will download the packages on one machine, submit only new packages for security vetting and only use the new packages when they were vetted.

On a separate (air-gaped) system, all the PyPI packages can be copied to docker-context-files where you can build the image using the packages downloaded by passing those build args:

• INSTALL_DISTRIBUTIONS_FROM_CONTEXT="true" - to use packages present in docker-context-files

• AIRFLOW_CONSTRAINTS_LOCATION=/docker-context-files/YOUR_CONSTRAINT_FILE.txt - to downloaded constraint files

• (Optional) INSTALL_MYSQL_CLIENT="false" if you do not want to install MySQL client from the Oracle repositories.

• (Optional) INSTALL_MSSQL_CLIENT="false" if you do not want to install MsSQL client from the Microsoft repositories.

• (Optional) INSTALL_POSTGRES_CLIENT="false" if you do not want to install Postgres client from the Postgres repositories.

Note, that the solution we have for installing python packages from local packages, only solves the problem of “air-gaped” python installation. The Docker image also downloads apt dependencies and node-modules. Those types of dependencies are more likely to be available in your “air-gaped” system via transparent proxies and it should automatically reach out to your private registries. However, in the future the solution might be applied to both of those installation steps.

You can also use techniques described in the previous chapter to make docker build use your private apt sources or private PyPI repositories (via .pypirc) available which can be security-vetted.

If you fulfill all the criteria, you can build the image on an air-gaped system by running command similar to the below:

export DOCKER_BUILDKIT=1

docker build . \
    --pull \
    --build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.13.18-debian13-dev}" \
    --build-arg AIRFLOW_INSTALLATION_METHOD="apache-airflow" \
    --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
    --build-arg INSTALL_MYSQL_CLIENT="false" \
    --build-arg INSTALL_MSSQL_CLIENT="false" \
    --build-arg INSTALL_POSTGRES_CLIENT="true" \
    --build-arg DOCKER_CONTEXT_FILES="docker-context-files" \
    --build-arg INSTALL_DISTRIBUTIONS_FROM_CONTEXT="true" \
    --build-arg AIRFLOW_CONSTRAINTS_LOCATION="/docker-context-files/constraints-3.11.txt" \
    --tag airflow-my-restricted-environment:0.0.1

Modifying the Dockerfile¶

The build arg approach is a convenience method if you do not want to manually modify the Dockerfile. Our approach is flexible enough to be able to accommodate most requirements and customizations out-of-the-box. When you use it, you do not need to worry about adapting the image every time a new version of Airflow is released. However, sometimes it is not enough if you have very specific needs and want to build a very custom image. In such case you can simply modify the Dockerfile manually as you see fit and store it in your forked repository. However, you will have to make sure to rebase your changes whenever new version of Airflow is released, because we might modify the approach of our Dockerfile builds in the future and you might need to resolve conflicts and rebase your changes.

There are a few things to remember when you modify the Dockerfile:

• We are using the widely recommended pattern of .dockerignore where everything is ignored by default and only the required folders are added through exclusion (!). This allows to keep docker context small because there are many binary artifacts generated in the sources of Airflow and if they are added to the context, the time of building the image would increase significantly. If you want to add any new folders to be available in the image you must add them here with leading !

  # Ignore everything
  **
  
  # Allow only these directories
  !airflow
  ...
  

• The docker-context-files folder is automatically added to the context of the image, so if you want to add individual files, binaries, requirement files etc you can add them there. The docker-context-files is copied to the /docker-context-files folder of the build segment of the image, so it is not present in the final image - which makes the final image smaller in case you want to use those files only in the build segment. You must copy any files from the directory manually, using COPY command if you want to get the files in your final image (in the main image segment).

Build Args reference¶

The detailed --build-arg reference can be found in Image build arguments reference.

The architecture of the images¶

You can read more details about the images - the context, their parameters and internal structure in the Images documentation.

Dependencies caching¶

To enable faster iteration when building the image locally (especially if you are testing different combination of python packages), pip caching has been enabled. The caching id is based on four different parameters:

1. TARGETARCH: Avoid sharing architecture specific cached package

2. DEPENDENCY_CACHE_EPOCH: Enable invalidating cache by passing DEPENDENCY_CACHE_EPOCH as --build-arg