GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-102988: Detect email address parsing errors and return empty tuple to indicate the parsing error (old API) by tdwyer · Pull Request #108250 · python/cpython

Conversation

@tdwyer

@tdwyer tdwyer commented Aug 22, 2023 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This PR is designed to detect parsing errors and return an empty tuple to indicate the parsing error. Additionally, this PR updates the test_email.py to check for these bugs, as well as, adds some other wacky Address Headers that are in the examples of RFC 2822 and makes sure they are being parsed correctly.

I realize that this PR dose not actually track down the bug and fix it. It simply detects the error has happened and returns a parsing error. However, Lib/email/utils.py is a much simpler file than Lib/email/_parseaddr.py, so it is much easier to review this change. Additionally, there are actually multiple bugs which are causing erroneous output. Tracing the code flow for each and fixing them would be prone to error considering all of the wacky stuff that RFC 2822 allows for in Address headers. Finally, this change is actually rather simple.

  • Issue: [CVE-2023-27043] Parsing errors in email/_parseaddr.py lead to incorrect value in email address part of tuple #102988

📚 Documentation preview 📚: https://cpython-previews--108250.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@tdwyer
pythongh-102988: Detect email address parsing errors and return empty…
91b5b37
… tuple to indicate the parsing error (old API)
@tdwyer
tdwyer requested a review from a team as a code owner August 22, 2023 00:16
@bedevere-bot bedevere-bot mentioned this pull request Aug 21, 2023
[CVE-2023-27043] Parsing errors in email/_parseaddr.py lead to incorrect value in email address part of tuple #102988
Closed
@bedevere-bot bedevere-bot added the awaiting review label Aug 22, 2023
@tdwyer

tdwyer commented Aug 22, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Note, I just have the regex looking for both Double Quotes and Single Quotes but I think only Double-Quotes are allowed to specify a Real Name part.

Let me know if Single Quotes can NOT be used to wrap the Real Name e.g. 'Bob Bobby' <[email protected]>

I'll remove that from the regex if so

Additionally, I'm also stripping out escaped commas before counting e.g. \,
Let me know if I should not be doing that.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tdwyer
Also strip out escaped commas before counting
4afc952
theta682
theta682 reviewed Aug 22, 2023
View reviewed changes
Comment thread Lib/email/utils.py
accepted_values = []
for v in email_header_fields:
s = v.replace('\\(', '').replace('\\)', '')
if s.count('(') != s.count(')'):

@theta682 theta682 Aug 22, 2023

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not check )( or any other combination when ) proceeds (. Probably it could be better to iterate through the string and manually increment and decrement verifying that the counter never goes below 0 and it is exactly 0 at the end

Suggested change
if s.count('(') != s.count(')'):
counter = 0
for c in s:
if c == '(':
counter += 1
elif c == ')':
counter -= 1
if counter < 0:
break
if counter != 0:

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@tdwyer tdwyer Aug 22, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is not necessary. The fix already accounts for this and none of those are an issue.

My current solution works, because instead of trying to actually parse things I just detect an error has occurred by making sure that the resulting number of address 2-Tuples matches the number of Headers that were parsed. Which is why I went this direction. It's super hard to fix the actual parsing logic of RFC 2822 headers :P

./python
Python 3.13.0a0 (heads/blah_1:a1cc74c4ee, Aug 21 2023, 18:40:57) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> from email.utils import parseaddr, getaddresses
>>> 
>>> 
>>> parseaddr('[email protected]])(<[email protected]>')
('', '')
>>> 
>>> parseaddr('[email protected])(<[email protected]>')
('', '')
>>> 
>>> parseaddr('[email protected] <[email protected]>')
('', '')
>>> parseaddr('[email protected]..<[email protected]>')
('', '')
>>> 
>>> getaddresses(['[email protected]..<[email protected]>'])
[('', '')]
>>> 
>>> getaddresses(['[email protected]])(<[email protected]>'])
[('', '')]
>>> 
>>> getaddresses(['[email protected])(<[email protected]>'])
[('', '')]

With the un-patched python your attack results in multiple Tuples being returned

>>> getaddresses(['[email protected]])(<[email protected]>'])
[('', '[email protected]'), ('', ''), ('', '')]

A cleaner way to say it is this. There are countless ways to trigger this parsing error but there is only one error e.g. the parser will return an abnormal number of output tuples. So, rather than trying to detect every possible input which could trigger the bug, I just detect the one error that they all result in.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@tdwyer tdwyer Aug 23, 2023

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking at this again, it is interesting that my solution works here… why do I check for matching parentheses at all ?

[email protected])(<bob&@example.com

I have ideas, but I’ll look into it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/email/utils.py Outdated

# When a comma is used in the Real Name part it is not a deliminator
# So strip those out before counting the commas
pattern = r'"[^"]*,[^"]*"|\'[^\']*,[^\']\'*|\\,'

@theta682 theta682 Aug 22, 2023

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this regex support multiple commas inside quotes?

Suggested change
pattern = r'"[^"]*,[^"]*"|\'[^\']*,[^\']\'*|\\,'
pattern = r'''"[^"]*,[^"]*"|'[^']*,[^']'*|\\,'''

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@tdwyer tdwyer Aug 22, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex works for multiple commas because the Not-In-List [^"] allows for other commas. There just simply has to be at least one comma in between the Quotes.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@frenzymadness frenzymadness Aug 23, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just curious. The regex proposed by me simply ignores whatever is in single or double quotes and seems to be much simpler than this one. Is there any benefit in ignoring only the quoted parts containing at least one comma? I think it has no effect on the comma-counting logic.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@tdwyer tdwyer Aug 25, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@frenzymadness I don't see your suggested regex. In any case, perhaps what you are suggesting would be fine. I'll think on it. However, from a security perspective I'd like to be as specific as possible, because things get super tricky and I'd not want to implement a fix that is latter proven to be able to be bypassed. It's only the commas in quotes that are an issue here and should be accounted for.

In ether case, I plan on at least implementing the Triple Single-Quote syntax so I don't need to escape the single-quotes.

Additionally, I want looks some more to make sure that single-quotes are valid chars to isolate the Real Name part of the header. It's looking like only Double-Quotes to that, if so then only commas in Double-Quotes should be stripped.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@frenzymadness frenzymadness Aug 26, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex has been proposed in this comment: #102988 (comment)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@tdwyer tdwyer Aug 31, 2023

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I'll consider that. I've been busy with work but I plan on adding the extra test_email.py and updating the regex over this weekend.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_email/test_email.py
Comment on lines +3347 to +3348
eq(utils.getaddresses(['[email protected]]<[email protected]>']),
[('', '')])

@theta682 theta682 Aug 22, 2023

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
eq(utils.getaddresses(['[email protected]]<[email protected]>']),
[('', '')])
eq(utils.getaddresses(['[email protected]]<[email protected]>']),
[('', '')])
eq(utils.getaddresses(['[email protected]])(<[email protected]>']),
[('', '')])

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@tdwyer tdwyer Aug 22, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, not a bad idea to add extra tests. That way if someone else changes this code in the future these other corner cases will be tested for.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@stratakis

stratakis commented Sep 6, 2023

Copy link
Copy Markdown
Contributor

Any updates on that?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tdwyer
Only account for comma in double-quotes and added tests for this
30d3ed7
@tdwyer

tdwyer commented Sep 9, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

After some investigation, I confirmed that I was being too carful before. Only commas in double-quotes should be accounted for not commas in single-quotes nor commas with prefixed with a backslash

A comma in single-quotes and a comma prefixed by a backslash trigger the bug. However, a comma in double-quotes is parsed correctly. This is how all of those cases behave in:

Python 2.7.17
Python 3.6.9
Python 3.11.5

 s = "'ae.com,' <[email protected]>"
 parseaddr(s)
('', "'ae.com")
 getaddresses([s])
[('', "'ae.com"), ("'", '[email protected]')]

 s = '"ae.com," <[email protected]>'
 getaddresses([s])
[('ae.com,', '[email protected]')]
 parseaddr(s)
('ae.com,', '[email protected]')

 s = 'ae.com\,<[email protected]>'
 parseaddr(s)
('', 'ae.com\\')
 getaddresses([s])
[('', 'ae.com\\'), ('', '[email protected]')]

Additionally, somehow the fix for CVE-2019-16056 regarding [email protected] <[email protected]> seems to have been undone and is exploitable in:

Python 2.7.17
Python 3.6.9
Python 3.11.5

 getaddresses(['[email protected] <[email protected]>']) 
[('', '[email protected]'), ('', '[email protected]')]
  
 parseaddr('[email protected] <[email protected]>')                                         
('', '[email protected]')
  
 parseaddr('[email protected]<[email protected]>')                                          
('', '[email protected]')

So, this PR fixes both CVE-2019-16056 and CVE-2023-27043

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@frenzymadness

frenzymadness commented Sep 20, 2023

Copy link
Copy Markdown
Contributor

What is necessary here to move this PR forward?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner mentioned this pull request Oct 20, 2023
[CVE-2023-27043] gh-102988: Reject malformed addresses in email.parseaddr() #111116
Merged
@vstinner

vstinner commented Oct 20, 2023

Copy link
Copy Markdown
Member

My colleague Lumir Balhar @frenzymadness ran an impact check on this PR on Fedora: in short, there is no impact, the test suite of all Python packages (in Fedora) pass with the change. While there were some build errors, they were unrelated to the email issue. For details, see https://copr.fedorainfracloud.org/coprs/lbalhar/email-CVE/builds/ COPR which as more than 4300 builds.

I created a copy of this PR to add a strict parameter: PR #111116.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ambv

ambv commented Oct 27, 2023

Copy link
Copy Markdown
Contributor

Closing this in favor of Victor's updated version.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ambv ambv closed this Oct 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@theta682 theta682 theta682 left review comments
@frenzymadness frenzymadness frenzymadness left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@tdwyer @stratakis @frenzymadness @vstinner @ambv @theta682 @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.