GeezerVison is showing github.com — verity volumes: attach verified read-only data disks to CVMs by h4x3rotab · Pull Request #752 · Dstack-TEE/dstack
Conversation
h4x3rotab
commented
Jul 3, 2026
•
edited by kvinwang
Loading
Uh oh!
There was an error while loading. Please reload this page.
What this does
Adds generic read-only verity data volumes to CVMs. An operator builds or wraps a filesystem image, places it in the VMM's configured volumes_dir, and attaches it by name at deploy time. The guest discovers the disk, verifies it against the root measured in app-compose.json, and mounts it read-only at the requested absolute path.
How it fits together
dstack verity --dir <path> packs a directory into a reproducible squashfs volume.
dstack verity --fs-image <path> wraps an existing filesystem image.
- Each disk uses a generic
DSTACK_VOLUME envelope in partition 1, filesystem data in partition 2, and the dm-verity superblock/hash tree in partition 3.
- The VMM resolves bare volume names under
cvm.volumes_dir and attaches the raw disks read-only.
- The guest
dstack-volume helper scans /sys/class/block, matches the full root, opens the disk with veritysetup using the measured root, and mounts it before the application starts.
- Attestation binds
verity_root and target through app-compose.json and therefore the measured app_id.
Example:
dstack verity --dir ./models -o models.img
dstack deploy -c docker-compose.yaml \
--volume models.img:<verity-root>:/run/models
See docs/verity-volumes.md.
Why Docker image seeding is not included
An earlier version injected pre-extracted layers directly into Docker's private overlay2 state. That couples the feature to Docker internals (imagedb, layerdb, overlay2, and repository metadata) and creates unsafe lifecycle edge cases: bind mounts disappear on reboot while Docker metadata persists, pulled images may share layers with verity-seeded images, and removing or changing a volume can leave existing images or containers with dangling layer references.
Safely reconciling that state would require ownership tracking and Docker-specific graph cleanup, while a custom containerd snapshotter would be a substantially larger and less mature integration. This PR therefore keeps the trust and lifecycle boundary simple: it only exposes verified, read-only data filesystems and does not modify Docker image state.
Validation
cargo clippy -p dstack-volume --all-targets -- -D warnings
cargo clippy -p dstack-cli --all-targets -- -D warnings
cargo test -p dstack-volume -p dstack-cli -p dstack-types
cargo test -p dstack-vmm --no-fail-fast
- End-to-end local volume build with
dstack verity --dir
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
h4x3rotab
changed the title
verity volumes: reproducible dm-verity image/data pre-seeding for CVMs
verity volumes: pre-load docker images and data into CVMs
Jul 4, 2026
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
kvinwang
changed the title
verity volumes: pre-load docker images and data into CVMs
verity volumes: attach verified read-only data disks to CVMs
Jul 21, 2026
Add read-only verity volumes -- extra virtio-blk disks a CVM can mount
instead of pulling and unpacking their contents. A volume is declared in
app-compose.json as `{ verity_root, target }`, and attached at deploy time
with `--volume <name>`: the vmm looks the name up under cvm.volumes_dir and
attaches it read-only. Because verity_root is part of the measured compose,
the guest can check the bytes it's handed against it.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Just before dockerd starts, dstack-prepare.sh runs the seeding helper. For
each volume declared in the compose it finds the matching disk by opening it
with veritysetup against the measured verity_root, then either seeds docker's
overlay2 store (target "docker") so the images are already present, or mounts
the volume at a path (a data volume). It's fail-safe throughout: a volume
that's missing or doesn't verify is skipped, and its images just pull.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
`dstack verity <image>...` builds a squashfs + dm-verity volume that
pre-extracts docker images (or, with `--dir`, a plain directory) so a CVM can
start without pulling or unpacking them. It fetches images itself through
oci-client -- no docker daemon -- and lays out the overlay2 store
deterministically (each layer's directory id is its chain-id, with a fixed
timestamp and salt). The same inputs always produce the same verity_root, so
anyone can recompute it from the pinned image digests and confirm what's in
the volume without trusting the builder. `dstack deploy --volume` attaches
the result.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…volume
# Conflicts:
# dstack/crates/dstack-cli-core/src/compose.rs
# dstack/crates/dstack-cli/src/main.rs
…volume
# Conflicts:
# dstack/vmm/src/app/qemu.rs
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
1 more reviewer
Reviewers whose approvals may not affect merge requirements
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|