GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-115399: Upgrade bundled libexpat to 2.6.0 by sethmlarson · Pull Request #115431 · python/cpython

Conversation

@sethmlarson

@sethmlarson sethmlarson commented Feb 13, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Closes #115399

  • Upgrades libexpat to 2.6.0
  • Adds XML_GE definition to LIBEXPAT_CFLAGS since 2.6.0 requires us to choose one, I set it to 1 to keep general entities support. Our test suite didn't fail either way with this setting but I'm assuming we want to keep support?

cc @hartwork

  • Issue: Please upgrade bundled Expat to 2.6.0 (e.g. for the fix to CVE-2023-52425) #115399

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hartwork reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@sethmlarson
sethmlarson requested review from corona10 and erlend-aasland as code owners February 13, 2024 22:08
@bedevere-app bedevere-app Bot added the awaiting review label Feb 13, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Feb 13, 2024
Please upgrade bundled Expat to 2.6.0 (e.g. for the fix to CVE-2023-52425) #115399
Closed
ambv
ambv reviewed Feb 13, 2024
View reviewed changes
Comment thread configure.ac Outdated
LIBEXPAT_INTERNAL=
], [
LIBEXPAT_CFLAGS="-I\$(srcdir)/Modules/expat"
LIBEXPAT_CFLAGS="-I\$(srcdir)/Modules/expat -DXML_GE=1"

@ambv ambv Feb 13, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's that?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@sethmlarson sethmlarson Feb 13, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to make a choice, error message from the Windows builds that are now failing cuz I need to update them as well:

XML_GE (for general entities) must be defined, non-empty, either 1 or 0 (0 to disable, 1 to enable; 1 is a common default

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hartwork reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@erlend-aasland erlend-aasland Feb 13, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should probably update the default value in line 3719 as well.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@hartwork hartwork Feb 13, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's that?

@ambv "GE" is short for general entities. The flag is documented in detail at https://libexpat.github.io/doc/api/latest/#XML_GE .

@sethmlarson I would have expected (meaning considered likely) a new line #define XML_GE 1 to go into CPython's own expat_config.h. Did you avoid that file on purpose? Its sibling XML_DTD seems to live there.

You should probably update the default value in line 3719 as well.

For anyone else wondering where (or what the line content is), that seems to be:

cpython/configure.ac

Line 3719 in 514b1c9

LIBEXPAT_CFLAGS=${LIBEXPAT_CFLAGS-""}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ambv and gpshead reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

@sethmlarson sethmlarson Feb 13, 2024

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hartwork Thank you for this, I don't know how but my grep XML_DTD turned up no results. That's definitely where it should go.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hartwork reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@hartwork

hartwork commented Feb 13, 2024

Copy link
Copy Markdown
Contributor

[..] I set it to 1 to keep general entities support. Our test suite didn't fail either way with this setting but I'm assuming we want to keep support?

@sethmlarson good call, yes. On a sidenote XML_DTD requires XML_GE so it's the only choice here with XML_DTD active anyway. Good 👍

sethmlarson and gpshead reacted with heart emoji
All reactions
  • ❤️ 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sethmlarson
sethmlarson force-pushed the libexpat-2.6.0 branch from 4c20480 to f41fbfc Compare February 13, 2024 22:39
@sethmlarson

sethmlarson commented Feb 13, 2024

Copy link
Copy Markdown
Contributor Author

Thanks for the guidance @hartwork! I rearranged the configuration to do as you said.

I also split the update itself into a separate commit from the SBOM update (which removes expat/expat_config.h from being a part of the expat project) so it's easier to backport to older branches. This PR can directly be backported to 3.12 but 3.11 and earlier will need only a011b5f

hartwork reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sethmlarson
sethmlarson requested a review from ambv February 13, 2024 22:41
@sethmlarson
sethmlarson requested a review from ericsnowcurrently as a code owner February 13, 2024 23:03
hartwork
hartwork reviewed Feb 13, 2024
View reviewed changes
Comment thread Modules/expat/expat_external.h
Comment on lines -67 to -71

/* Namespace external symbols to allow multiple libexpat version to
co-exist. */
#include "pyexpatns.h"

@hartwork hartwork Feb 13, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sethmlarson could it be that this removal is accidental?

PS: Here's the Dockerfile that made me spot this, based on earlier version #98742 (review):

# Copyright (c) 2022-2024 Sebastian Pipping <[email protected]>
# Licensed under the Apache License version 2.0

FROM alpine
RUN apk add --update \
            diffutils \
            git \
            sed \
        && \
    git clone --depth 1 https://github.com/python/cpython cpython-main \
        && \
    ( cd cpython-main && git rev-parse HEAD ) \
        && \
    git clone --depth 1 --branch libexpat-2.6.0 https://github.com/sethmlarson/cpython cpython-pr \
        && \
    ( cd cpython-pr && git rev-parse HEAD ) \
        && \
    git config --global advice.detachedHead false \
        && \
    git clone --depth 1 --branch R_2_5_0 https://github.com/libexpat/libexpat libexpat_2_5_0 \
        && \
    git clone --depth 1 --branch R_2_6_0 https://github.com/libexpat/libexpat libexpat_2_6_0 \
        && \
    diff -r -u libexpat_2_5_0/expat/lib/ cpython-main/Modules/expat/ | tee 2-5-0.diff \
        && \
    diff -r -u libexpat_2_6_0/expat/lib/ cpython-pr/Modules/expat/ | tee 2-6-0.diff \
        && \
    sed -e '/^Only in /d' -e '/^\(+++\|---\) /d' -e '/^diff /d' -i 2-5-0.diff 2-6-0.diff \
        && \
    diff -u 2-5-0.diff 2-6-0.diff \
        && \
    echo 'Diff is good.'

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@corona10 corona10 Feb 14, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hugovk It feels like now we have to automate this diff comparison. Do you have any good ideas on when to trigger the workflow smartly?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@hartwork hartwork Feb 14, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@corona10 did you mean me or @hugovk?

To automate this, you'd need to extract an old and a new version from the diff between the push HEAD and the base commit and so on. I'm not sure if Expat releases are frequent enough to build automation like that and keep it working.

Pull request #100539 is about something related but different, but maybe still of interest to you.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@hartwork hartwork Feb 14, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@corona10 PS: maybe a cheap version to cover the core need (unless misunderstood) would be to make CI assert that that file still includes file pyexpatns.h using grep? That would be orders of magnitude simpler 😃

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@sethmlarson sethmlarson Feb 14, 2024

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this, I've updated my commit to not remove these lines.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hartwork reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@corona10 corona10 Feb 15, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hartwork
I meant @hugovk since he has good insight into automation workflow in CPython.
But in any case, the link you introduced looks cool to me, I think that it would be worth to take a look at :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@sethmlarson sethmlarson added the needs backport to 3.12 only security fixes label Feb 14, 2024
sethmlarson added 2 commits February 14, 2024 09:38
@sethmlarson
Upgrade bundled libexpat to 2.6.0
9fc6eb9
@sethmlarson
Update SBOM for libexpat 2.6.0
0d3fb31
@sethmlarson
sethmlarson force-pushed the libexpat-2.6.0 branch from b25685e to 0d3fb31 Compare February 14, 2024 15:38
@Yhg1s
Yhg1s enabled auto-merge (squash) February 14, 2024 15:45
Yhg1s
Yhg1s approved these changes Feb 14, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Feb 14, 2024
hartwork
hartwork approved these changes Feb 14, 2024
View reviewed changes
ambv
ambv approved these changes Feb 14, 2024
View reviewed changes
@hartwork hartwork mentioned this pull request Feb 14, 2024
Release Expat 2.6.0 libexpat/libexpat#775
Closed
28 tasks done
@sethmlarson sethmlarson mentioned this pull request Feb 14, 2024
[3.11] Upgrade bundled libexpat to 2.6.0 (GH-115399) #115468
Merged
@sethmlarson

sethmlarson commented Feb 14, 2024

Copy link
Copy Markdown
Contributor Author

Backport to 3.11 and prior of this PR: #115468

3.12 will be handled automatically once this PR merges.

hartwork reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Yhg1s
Yhg1s merged commit 4b2d178 into python:main Feb 14, 2024
@miss-islington-app

miss-islington-app Bot commented Feb 14, 2024

Copy link
Copy Markdown

Thanks @sethmlarson for the PR, and @Yhg1s for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Feb 14, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Feb 14, 2024
@sethmlarson @miss-islington
pythongh-115399: Upgrade bundled libexpat to 2.6.0 (pythonGH-115431)
f601dee
(cherry picked from commit 4b2d178)

Co-authored-by: Seth Michael Larson <[email protected]>
@bedevere-app

bedevere-app Bot commented Feb 14, 2024

Copy link
Copy Markdown

GH-115469 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Feb 14, 2024
@sethmlarson
sethmlarson deleted the libexpat-2.6.0 branch February 14, 2024 18:08
@gpshead gpshead added the type-security A security issue label Feb 14, 2024
gpshead pushed a commit that referenced this pull request Feb 14, 2024
@miss-islington @sethmlarson
[3.12] gh-115399: Upgrade bundled libexpat to 2.6.0 (GH-115431) (#115469
14930ac
)

gh-115399: Upgrade bundled libexpat to 2.6.0 (GH-115431)
(cherry picked from commit 4b2d178)

Co-authored-by: Seth Michael Larson <[email protected]>
@hartwork hartwork mentioned this pull request Mar 13, 2024
Please upgrade bundled Expat to 2.6.2 (e.g. for the fix to CVE-2024-28757) #116741
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ambv ambv ambv approved these changes
@Yhg1s Yhg1s Yhg1s approved these changes
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland
@corona10 corona10 Awaiting requested review from corona10
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
+1 more reviewer
@hartwork hartwork hartwork approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

type-security A security issue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Please upgrade bundled Expat to 2.6.0 (e.g. for the fix to CVE-2023-52425)

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@sethmlarson @hartwork @ambv @Yhg1s @corona10 @erlend-aasland @gpshead
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.