GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-101703: use `PyOS_snprintf` instead of `sprintf` by sobolevn · Pull Request #101729 · python/cpython

Conversation

@sobolevn

@sobolevn sobolevn commented Feb 9, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Several points of interest:

  1. sprintf is indeed insecure generally: https://rules.sonarsource.com/c/RSPEC-6069 But, for our cases - I was not able to find a single insecure place, so I don't mark this PR as :security:
  2. snprintf is also listed as not secure here: https://clang.llvm.org/docs/analyzer/checkers.html#security-insecureapi-deprecatedorunsafebufferhandling-c
  3. snprintf is reported to be slightly slower than sprintf
  • Issue: Let's get rid of sprintf() #101703

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@sobolevn
sobolevn requested review from abalkin, lysnikolaou, markshannon, pablogsal and pganssle as code owners February 9, 2023 07:28
@bedevere-bot bedevere-bot added the awaiting review label Feb 9, 2023
@bedevere-bot bedevere-bot mentioned this pull request Feb 9, 2023
Let's get rid of sprintf() #101703
Open
@sobolevn sobolevn added the skip news label Feb 9, 2023
@sobolevn
sobolevn marked this pull request as draft February 9, 2023 07:43
@sobolevn
pythongh-101703: use snprintf instead of sprintf
b33fb2d
@sobolevn
sobolevn force-pushed the issue-101703 branch from 38f8e7d to b33fb2d Compare February 9, 2023 07:56
@sobolevn

sobolevn commented Feb 9, 2023

Copy link
Copy Markdown
Member Author

Ok, the first part with constant sized elements is done. Now, more complex ones.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Feb 9, 2023

Copy link
Copy Markdown
Member Author

Looks like we have PyOS_snprintf instead of just snprintf 🤔

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sobolevn added 3 commits February 9, 2023 11:33
@sobolevn
Use PyOS_snprintf
8fb350d
@sobolevn
Convert more constant sized buffers
61eee4d
@sobolevn
Convert ones using direct allocation
ade334c
@sobolevn

sobolevn commented Feb 9, 2023

Copy link
Copy Markdown
Member Author

Almost done!

There are a couple left, but I don't quite understand the code and I will highly appreciate help / advice on these cases:

  • cpython/Modules/_ctypes/_ctypes.c

    Lines 2642 to 2651 in 5839575

    cp += sprintf(cp, "%x", Py_SAFE_DOWNCAST(index, Py_ssize_t, int));
    while (target->b_base) {
    bytes_left = sizeof(string) - (cp - string) - 1;
    /* Hex format needs 2 characters per byte */
    if (bytes_left < sizeof(Py_ssize_t) * 2) {
    PyErr_SetString(PyExc_ValueError,
    "ctypes object structure too deep");
    return NULL;
    }
    cp += sprintf(cp, ":%x", Py_SAFE_DOWNCAST(target->b_index, Py_ssize_t, int));
  • cpython/Objects/unicodeobject.c

    Line 736 in 5839575

    size = sprintf(str, "&#%d;", PyUnicode_READ(kind, data, i));
  • cpython/Parser/string_parser.c

    Line 121 in 5839575

    sprintf(p, "\\U%08x", chr);
  • cpython/Python/pystrtod.c

    Line 1242 in 5839575

    exp_len = sprintf(p, "%+.02d", exp);
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn
Finish!
9bc0864
@sobolevn sobolevn added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2023
@bedevere-bot

bedevere-bot commented Feb 9, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit 9bc0864 🤖

If you want to schedule another build, you need to add the :hammer: test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2023
sobolevn added 7 commits February 9, 2023 14:44
@sobolevn
Fix CI
ca2acae
@sobolevn
Bisect errors to a single place
dff1e73
@sobolevn
Fix Windows x86
6953f30
@sobolevn
Convert unicodeobject.c
6c60488
@sobolevn
Fix unicodeobject.c
cc74727
@sobolevn
Convert string_parser.c
95d6c52
@sobolevn
Convert pystrtod.c
c7dd348
@sobolevn sobolevn added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2023
@bedevere-bot

bedevere-bot commented Feb 9, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit c7dd348 🤖

If you want to schedule another build, you need to add the :hammer: test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn
sobolevn marked this pull request as ready for review February 10, 2023 10:03
@sobolevn

sobolevn commented Feb 10, 2023

Copy link
Copy Markdown
Member Author

@Yhg1s @gvanrossum it is ready to be reviewed :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland
erlend-aasland reviewed Feb 10, 2023
View reviewed changes
Comment thread Objects/unicodeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Yhg1s
Yhg1s reviewed Feb 10, 2023
View reviewed changes
Comment thread Objects/unicodeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn
Address review comments
ae812b6
@sobolevn

sobolevn commented Feb 10, 2023

Copy link
Copy Markdown
Member Author

I've made the requested changes, thanks for the suggestions 👍

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Mar 27, 2023

Copy link
Copy Markdown
Member Author

@Yhg1s can you please review again? :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn
Merge branch 'main' into issue-101703
8a95f08
@sobolevn sobolevn added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 11, 2023
@bedevere-bot

bedevere-bot commented Apr 11, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit 8a95f08 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 11, 2023
@sobolevn

sobolevn commented Apr 11, 2023

Copy link
Copy Markdown
Member Author

I've merged it with the latest main, new round of buildbot tests are required to make sure that all still works correctly.

Does anyone want to take a look once again, please? 😉

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@arhadthedev arhadthedev changed the title gh-101703: use snprintf instead of sprintf gh-101703: use PyOS_snprintf instead of sprintf Apr 24, 2023
@sobolevn sobolevn mentioned this pull request Apr 24, 2023
sprintf deprecation warning on macOS for datetimemodule #103733
Closed
@sobolevn
Merge branch 'main' into issue-101703
3e8e61f
@sobolevn sobolevn added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 24, 2023
@bedevere-bot

bedevere-bot commented Apr 24, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit 3e8e61f 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 24, 2023
@sobolevn
Merge branch 'main' into issue-101703
d82e315
@sobolevn sobolevn added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 27, 2023
@bedevere-bot

bedevere-bot commented Apr 27, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit d82e315 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 27, 2023
AlexWaygood
AlexWaygood requested changes Aug 28, 2023
View reviewed changes

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now has merge conflicts

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting changes and removed awaiting review labels Aug 28, 2023
@bedevere-bot

bedevere-bot commented Aug 28, 2023

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner requested changes Nov 10, 2023
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sprintf is indeed insecure generally: https://rules.sonarsource.com/c/RSPEC-6069 But, for our cases - I was not able to find a single insecure place, so I don't mark this PR as :security:

In the past, I saw people introducing bugs and security vulnerabilities by attempting to make linters happy about getting rid of "dangerous functions".

Python provides advances functions to format strings in a safe way: PyBytes_FromFormat() and PyUnicode_FromFormat(). Moreover, there is also _PyBytesWriter and _PyUnicodeWriter internal API which can be used to "create" a new string in a safe way (without having to handle memory directly).

If a string is longer than expected, PyOS_snprintf() truncates the string, but your change ignores PyOS_snprintf() result and so the string is ignored silently. I am not convinced that these changes are worth it:

  • It doesn't prevent to truncate strings.
  • It's designed to prevent buffer overflow, but it doesn't show examples of possible buffer overfow.
  • From my point of view, I don't see how it makes the code easier to maintain or easier to read.

For example, the PR changes FindAddress() to use PyOS_snprintf(), but it keeps alloca() which is IMO way more problematic. Using PyBytes_FromFormat() would be safer here: the function takes care of memory management and allocates memory on the heap, there is no risk of stack overflow.

If you want me to review this PR, I would prefer smaller PRs with a more precise scope and goal.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner

vstinner commented Nov 10, 2023

Copy link
Copy Markdown
Member

In the past, I saw people introducing bugs and security vulnerabilities by attempting to make linters happy about getting rid of "dangerous functions".

The worst example that I recall:

  • https://bugs.php.net/bug.php?id=55439
  • Change introducing the security issue: php/php-src@7eb5bbb : "Fix more signed 1-bit bitfield, and let's use strlcpy/strlcat instead for these static string copies". Replace "unsafe" strncpy()/strncat() with "safe" strlcpy()/strncat().
  • Fix : php/php-src@4f98090 (as Subversion commit)

Commit message of the fix:

commit 4f980905a0bff94807ea07cb897c0e4cd4e6b83f
Author: Stanislav Malyshev <[email protected]>
Date:   Fri Aug 19 22:49:18 2011 +0000

    Unbreak crypt() (fix bug #55439)
    # If you want to remove static analyser messages, be my guest,
    # but please run unit tests after

Well, at that time, PHP had no automated test suite, and running tests showed the regression. But well, it's surprising how using "safer" function can introduce a major security vulnerability.

The bug allowed to bypass any kind of login page (implemented with crypt()) with any password...

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Nov 10, 2023

Copy link
Copy Markdown
Member Author

I wanted to close this PR anyway, I think that we should do it on a per-usage approach. Easier to review, easier to merge.

vstinner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn sobolevn closed this Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yhg1s Yhg1s Yhg1s left review comments
@erlend-aasland erlend-aasland erlend-aasland left review comments
@vstinner vstinner vstinner requested changes
@AlexWaygood AlexWaygood AlexWaygood requested changes
@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
@pganssle pganssle Awaiting requested review from pganssle pganssle is a code owner
@abalkin abalkin Awaiting requested review from abalkin
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting changes skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@sobolevn @bedevere-bot @vstinner @Yhg1s @erlend-aasland @AlexWaygood
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.