GeezerVison

Pull Request Test Coverage Report for Build 2293926458

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 98.354%

Totals
Change from base Build 2280279923:	0.0%
Covered Lines:	1201
Relevant Lines:	1217

💛 - Coveralls

Although this PR is fully functional, I labeled it as PoC because it has a couple of UX pain points:

Key distribution

• Signing key for CD build job attestation is created locally and stored as GitHub secret (along with decryption password) in a one-time setup, and piped to a file during each CD build, in order for in-toto to use it, which feels cumbersome and risky (layout and maintainer attestation keys, otoh, use gpg and thus benefit from existing key server network).
• While public keys and authorization mappings for attestation signatures are published through the layouts, the layout signing keys and thresholds need to be defined and distributed out-of-band. This PoC assumes that the layout is signed with one maintainer gpg key (see note about threshold below). How to establish root trust in layout signing keys is still unclear. Relying on fingerprints from maintainers.txt in the repo to protect the repo does not seem ideal.

Metadata distribution

• CD build job attestations are published automatically on the GitHub release page, from where they must be downloaded by the verifier.
• There is no mechanism to easily distribute layouts and maintainer attestations for verification. This PoC assumes that the verifier has created layouts and maintainer attestations and they are available locally during verification
• There is also no mechanism to easily distribute layouts for threshold signing. In this PoC the layouts are signed only once.

in-toto quirks

In order for the verifier to link the paper trail of attestations to the actual final product, i.e. the build artifacts, a layout must define a so-called inspection, which in turn must define a shell command that is executed upon verification, even if no command is needed. Besides unnecessarily running a command, this can also fail the entire verification, if the command defined in the layout is not available to the verifier.

Ideally, the inspection command is relevant to the verification. In our case this could be an untar/unzip command to open up the build artifact and, by defining further artifact rules in the layout, match its contents to the tagged sources earlier in the supply chain. To not further bloat the PR, I only defined the dummy inspection as described above.

Another minor ux nuisance is that when the verifier runs the inspection command, it records all files from cwd as materials and products into an inspection attestation, which can then be used to link the final product to the products reported by the build attestation. For this to work, verification has to be run from a clean directory that contains only the build artifact to be verified.

See in-toto/specification#27 for a discussion about (dummy) inspections and linking the final product.

FYI: I also played around with a hatch plugin to generate in-toto attestations transparently by just calling python -m build ..., instead of wrapping the build command with in-toto-run [opts] -- [cmd]: see lukpueh@f89dcbd

For this PoC it made sense to just use the verbose in-toto-run command, but a hatch plugin could benefit the broader community.

(cc @ofek, author of hatch and in-toto adoption veteran)

One more thing I'd like to add to the setup is creating attestations for the cd test runs and mandate them by the layout. But I'll likely do this in a follow-up PR.

FYI: I also played around with a hatch plugin to generate in-toto attestations transparently by just calling python -m build ..., instead of wrapping the build command with in-toto-run [opts] -- [cmd]

I'm still ingesting the whole thing so this is just a comment on this specifically:

The build system plugin feels like a good idea but I do have a fundamental issue with injecting intoto into the python-tuf build system in general like the example does: we're adding a build dependency (an unversioned one even, although I'm sure that's because it's a PoC) for no benefit to the actual build.

I'm not saying the in-toto maintainers are not trustworthy 😛, but I'm not sure that an attestation mechanism should be in the actual build pipeline. I thought I could just use it to document inputs and outputs, without giving the tool the ability to compromise our build system. Is there a real advantage to letting in-toto run the whole build? It seems to me like the build command is just another input to attest -- and in fact in our case it's already attested if cd.yml is part of the source source attestation

Example attestations seem to be here
https://github.com/lukpueh/tuf/releases/tag/v1.2.dev26

I thought I could just use it to document inputs and outputs, without giving the tool the ability to compromise our build system. Is there a real advantage to letting in-toto run the whole build?

No real advantage except for easier invocation maybe, and the ability to automatically record "byproducts" such as standard streams in the attestations, which I am not doing at the moment.

To record an attestation with in-toto, without having it run the command, you can do: in-toto-record start && python -m build && in-toto-record stop, which is what I also tried out at some point, while preparing this PR, because it gives better control over which materials/products should be included in the attestation: see lukpueh/tuf@release-attestations-with-record).

cc @SolidifiedRay

Heck, yes! Now I wonder if you could use TUF (e.g., PEP 480 and/or SigStore Trust Delegations) to somehow bootstrap the whole thing (pointing to in-toto metadata inside)...

We should combine efforts on this and in-toto/in-toto#444 to get them both over the line, especially with bootstrapping in-toto metadata using TUF.

I've given this feedback verbally but maybe should do it in writing too: the core requirement for any build integrity improvement is a verification process that can be and is used in the real world. If the verification is not easily usable and and understandable by humans, then the build integrity work may be cool but is in the end academic. This is the reason I've not made a proposal of e.g. sigstore signing yet: I could easily sign but what's the point if verification requires a crypto & certificate specialist?

The core idea of a formal way to require CD and maintainer attestations is really cool but still, I would like to see a way to easily verify at least some of this in a practically useful way before we merge... Otherwise it feels like adding a lot of new process (and code to maintain) right now for vague potential value some time in the future.

Can you say more about what you mean by "practically useful"?

What I mean is that

1. there is a plan for how this verification really happens -- so we don't just enable verification in theory, we ensure it happens or at least make it likely to happen. Do we have verification automation somewhere? If we don't, will someone else have that automation? clear instructions for contributors and users to verify?
2. the fairly long list of verification todo items in RELEASE_with_in-toto.md is somehow handled -- I don't necessarily mean everything needs to be polished but I'd like to see at least a practical plan for these items

If we merge like this, to me it looks like verification still requires a considerable amount of effort for an in-toto specialist, and is very difficult for a random developer.

I we merge like this, to me it looks like verification still requires a considerable amount of effort for an in-toto specialist, and is very difficult for a random developer.

Good questions, and this is precisely the kind of thing we should at least de facto standardise with, say, PEP 480. Let's actually take something like the Datadog work I did above (which partly inspired the upcoming Scudo, a variant of Uptane with in-toto attestations), and try it with a prototype of PEP 480.

Hey just now seeing this. Yeah a build hook would work. For example, Project Jupyter & IPython are almost done switching their entire packaging ecosystem to be based on a Hatch build hook plugin. I tried to link to most of the changes here: https://hatch.pypa.io/latest/users/

One thing I added with all of you in mind was the extra-metadata wheel option. Basically a dedicated *.dist-info/extra_metadata/ directory to stuff whatever users might want, like signatures 😄

One thing I added with all of you in mind was the extra-metadata wheel option. Basically a dedicated *.dist-info/extra_metadata/ directory to stuff whatever users might want, like signatures 😄

So we can validate and verify in-toto links using layout keys distributed out of band for now 🙂

Converted this back to draft to be used as reference for other release signing solutions discussed in #1966, most notably wrt to UX and security properties.

python-tuf maintainers currently don't feel confident to merge because of the perceived potential for human error in the setup and the complexity of validating its correctness.

for reference, we now get a minimal build attestation for free (example is tuf-on-ci but python-tuf will have the same as soon as we do next release):

$ pypi-attestations verify pypi --repository https://github.com/theupdateframework/tuf-on-ci \
     https://files.pythonhosted.org/packages/5f/0f/c922408cb12ffde512458906804f9800b55e09a9e27e32f911672cf93002/tuf_on_ci_sign-0.16.0-py3-none-any.whl
OK: tuf_on_ci_sign-0.16.0-py3-none-any.whl
$