GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Optimize makedirs on Windows · Issue #101358 · python/cpython

Optimize makedirs on Windows #101358

New issue
Copy link
New issue
Copy link
Open
Open
Optimize makedirs on Windows#101358
Copy link
Assignees
mdboom
Labels
OS-windowsperformancePerformance or resource usagePerformance or resource usagetype-featureA feature request or enhancementA feature request or enhancement

Description

@DefaultRyan
DefaultRyan
opened on Jan 26, 2023
Issue body actions

I've been working to improve the performance of some Python workflows on Windows, and I found that a noticeable amount of time was being spent in makedirs.

I some more profiling on the venv create scenario, and learned that roughly 1/3rd of the stat cost can be solely attributed to the exists check in makedirs (src). (Which is being called by compile here, and from my cursory glance at that code, it should simply be passing exist_ok=True rather than catching FileExistsError, right?). What makes this especially wasteful is that this run contained 1451 calls to makedirs and 1455 calls to makedir, implying that the exists check was making things worse, rather than helping. Put another way, exists is taking nearly as much time as mkdir here.

While it's true that if/when we get GetFileInformationByPath(), it should make stat calls much faster, that will only benefit the minority of users running a new version of Windows containing this new API. Therefore, it makes sense for us to improve the state of things for users running an older Windows, when practical.

And in this case, we should allow makedirs on Windows to execute in an idiomatically performant way. And that is to simply call CreateDirectoryW() and selectively ignore ERROR_ALREADY_EXISTS.

CreateDirectoryW() isn't cheap, but in the case of an existing directory, I don't think it's any worse than Windows' stat implementation that uses CreateFileW(). And in the case of when the directory doesn't exist and needs to be created, it will be significantly faster to make the single call.

It's probably worth splitting off a Windows-specific implementation of makedirs into the native side (posixmodule.c ?) so that we don't depend on raising a relatively expensive FileNotFoundError.
The non-Windows implementation should still use the cheap and reliable exists() and isdir().

(Forked from discussion on #101196)

Activity

  1. DefaultRyan
    added
    type-featureA feature request or enhancementA feature request or enhancement
    on Jan 26, 2023
  2. AlexWaygood
    added
    performancePerformance or resource usagePerformance or resource usage
    OS-windows
    on Jan 26, 2023
  3. mdboom
    self-assigned this
    on Feb 28, 2023
  4. serhiy-storchaka
    added this to OS issues 💻on Jul 3, 2026
  5. github-project-automation
    moved this to Todo in OS issues 💻on Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @mdboom
    mdboom

Labels

OS-windowsperformancePerformance or resource usagePerformance or resource usagetype-featureA feature request or enhancementA feature request or enhancement

Projects

  • OS issues 💻
    Status
    Todo
    Show more project fields

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.