GeezerVison

Security: Shall we consider enabling some standard security scan (e.g., upstream uses CodeQL https://github.com/aws/aws-lambda-runtime-interface-emulator/actions/workflows/github-code-scanning/codeql) as a quality gate to avoid shipping a dependency update with a known issue?
I don't know whether we need to consider other practices such as cooldown (to mitigate supply chain attacks).

I approved, but I agree - we should have a cooldown defined, and the lifecycle of the lambda images is also a thing to take into account!

Re:cooldown @dfangl @joe4dev
I've pushed this commit 4df9e43
We now have a 3 days cooldown for routine dep bumps but security fixes ship immediately. How does this sound?

a) How do we ensure that we catch any potentially breaking regression through the LocalStack tests before (accidentally) shipping lambda-images?

I share the concern, however do you think it's blocking this PR?

How do we integrate upstream changes now that merging into localstack is blocked?

@joe4dev what about removing the rule that enforces squash only? I've added "automergeStrategy": "squash" to the renovate config to make sure we always squash those: 6a6a2e7

Security: Shall we consider enabling some standard security scan

Added a trivy scan e520701. I think it makes sense to use the same tool we use in https://github.com/localstack/secops wdyt? @joe4dev can you add this to the required checks for a PR merge?

a) How do we ensure that we catch any potentially breaking regression through the LocalStack tests before (accidentally) shipping lambda-images?

I share the concern, however do you think it's blocking this PR?

I don't think we should block this great progress, but let's take some precautions to mitigate the risk of breaking Lambda without prior warning for our enterprise customers using K8. Some ideas for at least better awareness (before we can harden the process): Add a warning to the lambda-images README and/or PR template to wait for 🟢 LocalStack tests before shipping a new RIE in Lambda images.
With https://github.com/localstack/lambda-images/pull/42, the barrier to merge a RIE dependency update is quite low in the lambda-images repo. Such a RenovateBot PR should at least come with a big warning if we don't enforce automated CI checks. We could also think about a ls-ci-green label that needs to be added by a human before merging as an explicit step.

what about removing the rule that enforces squash only?

I don't think that changes anything because the current process requires force-pushing and branch protection disables that (which I think is a good think in general).

Security: I added the new security check. Thanks for adding 👍 Adopting what we use by default sounds reasonable (I'm not up-to-date).

let's take some precautions to mitigate the risk of breaking Lambda without prior warning for our enterprise customers using K8.

Thanks @joe4dev, I understand better your concern now.
What about this:

1. RIE automerges and autoreleases new versions
2. New weekly workflow in localstack-pro that picks up last RIE version and opens a PR
3. localstack-pro PR gets merged when tests are green + we push a new tag to the same RIE version: for example RIE v0.2.1 gets new tag validated-v0.2.1
4. Update lambda-images renovate config so it opens a PR for new RIE version prefixed with validated-*

How does this look @joe4dev @dfangl? I would take care of this as part of my ticket since my goal is to fully automate the RIE CVEs remediation.

what about removing the rule that enforces squash only?

I don't think that changes anything because the current process requires force-pushing and branch protection disables that (which I think is a good think in general).

We could add the aws team to the bypass list, or create a new smaller dedicated team e.g. lambda-upstream-sync?

Thank you @carole-lavillonniere for pushing the RIE CVE remediation 🙏

Using a pre-release might be a temporary option for now. I wanted to share some broader considerations/concerns; good to be aware of but not all in scope here:

What about this:

1. RIE automerges and autoreleases new versions

2. New weekly workflow in `localstack-pro` that picks up last RIE version and opens a PR

3. `localstack-pro` PR gets merged when tests are green + we push a new tag to the same RIE version: for example RIE `v0.2.1` gets new tag `validated-v0.2.1`

4. Update `lambda-images` renovate config so it opens a PR for new RIE version prefixed with `validated-*`

Do I understand correctly that Docker would use a separate release (v0.2.1) than K8 (validated-v0.2.1)?
The RIE version is part of the our Dockerhub tags (e.g., 3.14_v0.2.1) and relevant for customers using LocalStack behind a corporate proxy. I don't think we should expose such an internal validated-* tag to customers. It's a matter of trust and raises questions about whether the regular release is not validated. If we go this route, I would opt for a prerelease (pre-) similar to #104

We might want to (re-)consider the e2e release process:

• We ultimately need Define dependency management for RIE and LocalStack versions in K8 (IMHO: biggest time 💣 and low-hanging fruit ). ATM, shipping a broken release (automatically) requires immediate customer attention, and we have no way of fixing it ourselves due to unmanaged versioning.
• The proposed workflow above makes shipping proper dependency management harder by creating a chicken-egg problem. It assumes the localstack-pro PR runs before the lambda-images PR, but the localstack-pro PR ultimately depends on lambda-images by explicitly referencing an image (unless we treat Docker and K8 staged).
• Ideally (out-of-scope), we want to ensure the K8 tests run successfully before shipping a new lambda-image-RIE combination
• In the Lambda handover (internal Notion link), Alex suggested guarding the RIE release early by running localstack-pro tests in a private fork of the RIE. That's a way to ensure we release only a working version, but it requires (much) more effort to implement. I'm not sure whether that's worth it.

We could add the aws team to the bypass list, or create a new smaller dedicated team e.g. lambda-upstream-sync?

Yeah, we can try that (both work as long as we're careful with force-pushing and don't give RenovateBot force-push permissions)

Would it perhaps be easier not to use dependabot/renovatebot for this, but trigger an init update automatically after the PRO PR is merged, without a new release? I agree that the double release is not optimal.

Thank you @carole-lavillonniere for pushing the RIE CVE remediation 🙏

Using a pre-release might be a temporary option for now. I wanted to share some broader considerations/concerns; good to be aware of but not all in scope here:

What about this:

1. RIE automerges and autoreleases new versions

2. New weekly workflow in `localstack-pro` that picks up last RIE version and opens a PR

3. `localstack-pro` PR gets merged when tests are green + we push a new tag to the same RIE version: for example RIE `v0.2.1` gets new tag `validated-v0.2.1`

4. Update `lambda-images` renovate config so it opens a PR for new RIE version prefixed with `validated-*`

Do I understand correctly that Docker would use a separate release (v0.2.1) than K8 (validated-v0.2.1)? The RIE version is part of the our Dockerhub tags (e.g., 3.14_v0.2.1) and relevant for customers using LocalStack behind a corporate proxy. I don't think we should expose such an internal validated-* tag to customers. It's a matter of trust and raises questions about whether the regular release is not validated. If we go this route, I would opt for a prerelease (pre-) similar to #104

We might want to (re-)consider the e2e release process:

* We ultimately need [Define dependency management for RIE and LocalStack versions in K8](https://linear.app/localstack/issue/DEVX-327/define-clear-dependency-management-for-rie-and-localstack-versions) (IMHO: biggest time 💣 and low-hanging fruit ). ATM, shipping a broken release (automatically) requires immediate customer attention, and we have no way of fixing it ourselves due to unmanaged versioning.

* The proposed workflow above makes shipping proper dependency management harder by creating a chicken-egg problem. It assumes the localstack-pro PR runs before the lambda-images PR, but the localstack-pro PR ultimately depends on lambda-images by explicitly referencing an image (unless we treat Docker and K8 staged).

* Ideally (out-of-scope), we want to ensure the K8 tests run successfully before shipping a new lambda-image-RIE combination

* In the Lambda handover (internal Notion [link](https://app.notion.com/p/localstack/Discuss-how-ownership-of-Lambda-will-be-passed-to-Phoenix-Squad-35ffc2a234318088b239f7ff9424c63b?source=copy_link#35ffc2a234318085803edc3d43e24463)), Alex suggested guarding the RIE release early by running localstack-pro tests in a private fork of the RIE. That's a way to ensure we release only a working version, but it requires (much) more effort to implement. I'm not sure whether that's worth it.

We could add the aws team to the bypass list, or create a new smaller dedicated team e.g. lambda-upstream-sync?

Yeah, we can try that (both work as long as we're careful with force-pushing and don't give RenovateBot force-push permissions)

I did not mean for it to be a "separate release", but one single image that gets a new tag once validated, but @joe4dev your idea with making use of prereleases seems more sensible.

What about this:

1. lambda-runtime-init: renovate automerges non-major dep bumps. A weekly workflow patch-bumps and publishes one release, eg v0.2.1, marked prerelease: true. The binary is built once, here.
2. localstack-pro: a weekly workflow opens a PR bumping LAMBDA_RUNTIME_DEFAULT_VERSION prerelease, with a team as assignee/reviewers. On merge (when CI is green): flip that RIE release to prerelease: false via the API (No second tag, no rebuild)
3. lambda-images: Renovate picks up the now-stable RIE version and opens a PR.

I know we have a call tomorrow so I am not expecting an answer now, just wanted to write this down as a base for discussion.

@joe4dev I did not catch where you see a chicken-egg problem (that did not already exist), but looking forward to hearing your thoughts tomorrow.

Note that this does not solve testing a new RIE with k8s, but as @joe4dev mentioned, this is out-of-scope and a bigger project.

Thank you @carole-lavillonniere for pushing the RIE CVE remediation 🙏

Using a pre-release might be a temporary option for now. I wanted to share some broader considerations/concerns; good to be aware of but not all in scope here:

What about this:

1. RIE automerges and autoreleases new versions

2. New weekly workflow in `localstack-pro` that picks up last RIE version and opens a PR

3. `localstack-pro` PR gets merged when tests are green + we push a new tag to the same RIE version: for example RIE `v0.2.1` gets new tag `validated-v0.2.1`

4. Update `lambda-images` renovate config so it opens a PR for new RIE version prefixed with `validated-*`

Do I understand correctly that Docker would use a separate release (v0.2.1) than K8 (validated-v0.2.1)? The RIE version is part of the our Dockerhub tags (e.g., 3.14_v0.2.1) and relevant for customers using LocalStack behind a corporate proxy. I don't think we should expose such an internal validated-* tag to customers. It's a matter of trust and raises questions about whether the regular release is not validated. If we go this route, I would opt for a prerelease (pre-) similar to #104

We might want to (re-)consider the e2e release process:

* We ultimately need [Define dependency management for RIE and LocalStack versions in K8](https://linear.app/localstack/issue/DEVX-327/define-clear-dependency-management-for-rie-and-localstack-versions) (IMHO: biggest time 💣 and low-hanging fruit ). ATM, shipping a broken release (automatically) requires immediate customer attention, and we have no way of fixing it ourselves due to unmanaged versioning.

* The proposed workflow above makes shipping proper dependency management harder by creating a chicken-egg problem. It assumes the localstack-pro PR runs before the lambda-images PR, but the localstack-pro PR ultimately depends on lambda-images by explicitly referencing an image (unless we treat Docker and K8 staged).

* Ideally (out-of-scope), we want to ensure the K8 tests run successfully before shipping a new lambda-image-RIE combination

* In the Lambda handover (internal Notion [link](https://app.notion.com/p/localstack/Discuss-how-ownership-of-Lambda-will-be-passed-to-Phoenix-Squad-35ffc2a234318088b239f7ff9424c63b?source=copy_link#35ffc2a234318085803edc3d43e24463)), Alex suggested guarding the RIE release early by running localstack-pro tests in a private fork of the RIE. That's a way to ensure we release only a working version, but it requires (much) more effort to implement. I'm not sure whether that's worth it.

We could add the aws team to the bypass list, or create a new smaller dedicated team e.g. lambda-upstream-sync?

Yeah, we can try that (both work as long as we're careful with force-pushing and don't give RenovateBot force-push permissions)

@joe4dev are you able to add the aws team to the bypass list?