GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions by rhettinger · Pull Request #102649 · python/cpython

Conversation

@rhettinger

@rhettinger rhettinger commented Mar 13, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • Use sumprod() which is faster, simpler, and more accurate than rounding each multiplication before summation.
  • For an additional speed-up and simplification, compute the (x_xi - bar) only once instead of multiple times.
  • For Spearman's rank correlation, we can skip the (x_xi - bar) step because the ranks are centered around zero.

Baseline timing

% ./python.exe -m timeit -r11 -s 'from random import expovariate as r' -s 'from statistics import correlation' -s 'n=100' -s 'data = [r() for i in range(n)]' -s 'weights = [r() for i in range(n)]' 'correlation(data, weights)'
10000 loops, best of 11: 21 usec per loop

Timing with PR

% ./python.exe -m timeit -r11 -s 'from random import expovariate as r' -s 'from statistics import correlation' -s 'n=100' -s 'data = [r() for i in range(n)]' -s 'weights = [r() for i in range(n)]' 'correlation(data, weights)'
50000 loops, best of 11: 8.92 usec per loop
  • Issue: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions #102670

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
prithvijitguha and KennedyRichard reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
rhettinger added 8 commits March 13, 2023 11:12
@rhettinger
Use sumprod() to speed-up and improve accuracy of correlation()
3f2fd8d
@rhettinger
Only compute differences once
3241045
@rhettinger
Use sumprod() in covariance().
52c7584
@rhettinger
Use sumprod() in linear_regression().
16b2745
@rhettinger
Can skip centering if xbar or ybar is already zero
b9eeee5
@rhettinger
Factor-out common logic in linear_regression
c51e520
@rhettinger
Factor-out common logic in linear_regression
39be9f8
@rhettinger
Use generator for y array in linear_regression()
a448615
@rhettinger rhettinger added performance Performance or resource usage skip issue skip news 3.12 only security fixes labels Mar 13, 2023
@bedevere-bot bedevere-bot added the awaiting core review label Mar 13, 2023
eendebakpt
eendebakpt approved these changes Mar 13, 2023
View reviewed changes
@rhettinger
Coerce result to a float
c9a4e4d
eendebakpt
eendebakpt reviewed Mar 13, 2023
View reviewed changes
Comment thread Lib/statistics.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@rhettinger
Add test for the result type being a float.
5fe1be5
@rhettinger rhettinger changed the title Use sumprod() to simplify, speed up, and improve accuracy of statistics functions GH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions Mar 13, 2023
@rhettinger
Add blurb
45ffe7e
@rhettinger rhettinger removed skip issue skip news labels Mar 13, 2023
@bedevere-bot bedevere-bot mentioned this pull request Mar 13, 2023
Use sumprod() to simplify, speed up, and improve accuracy of statistics functions #102670
Closed
@rhettinger
rhettinger merged commit 457e4d1 into python:main Mar 14, 2023
@bedevere-bot bedevere-bot removed the awaiting core review label Mar 14, 2023
carljm added a commit to carljm/cpython that referenced this pull request Mar 14, 2023
@carljm
Merge branch 'main' into inlinecomp2
6c5f269 
* main: (50 commits)
  pythongh-102674: Remove _specialization_stats from Lib/opcode.py (python#102685)
  pythongh-102660: Handle m_copy Specially for the sys and builtins Modules (pythongh-102661)
  pythongh-102354: change python3 to python in docs examples (python#102696)
  pythongh-81057: Add a CI Check for New Unsupported C Global Variables (pythongh-102506)
  pythonGH-94851: check unicode consistency of static strings in debug mode (python#102684)
  pythongh-100315: clarification to `__slots__` docs. (python#102621)
  pythonGH-100227: cleanup initialization of global interned dict (python#102682)
  doc: Remove a duplicate 'versionchanged' in library/asyncio-task (pythongh-102677)
  pythongh-102013: Add PyUnstable_GC_VisitObjects (python#102014)
  pythonGH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions (pythonGH-102649)
  pythongh-102627: Replace address pointing toward malicious web page (python#102630)
  pythongh-98831: Use DECREF_INPUTS() more (python#102409)
  pythongh-101659: Avoid Allocation for Shared Exceptions in the _xxsubinterpreters Module (pythongh-102659)
  pythongh-101524: Fix the ChannelID tp_name (pythongh-102655)
  pythongh-102069: Fix `__weakref__` descriptor generation for custom dataclasses (python#102075)
  pythongh-98169 dataclasses.astuple support DefaultDict (python#98170)
  pythongh-102650: Remove duplicate include directives from multiple source files (python#102651)
  pythonGH-100987: Don't cache references to the names and consts array in `_PyEval_EvalFrameDefault`. (python#102640)
  pythongh-87092: refactor assemble() to a number of separate functions, which do not need the compiler struct (python#102562)
  pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (python#102631)
  ...
alexanderGerbik
alexanderGerbik reviewed Mar 23, 2023
View reviewed changes
Comment thread Lib/test/test_statistics.py
@@ -1,4 +1,4 @@
"""Test suite for statistics module, including helper NumericTestCase and
x = """Test suite for statistics module, including helper NumericTestCase and

@alexanderGerbik alexanderGerbik Mar 23, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a typo or some kind of black magic?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Fidget-Spinner pushed a commit to Fidget-Spinner/cpython that referenced this pull request Mar 27, 2023
@rhettinger @Fidget-Spinner
pythonGH-102670: Use sumprod() to simplify, speed up, and improve acc…
5b8c3cb 
…uracy of statistics functions (pythonGH-102649)
warsaw pushed a commit to warsaw/cpython that referenced this pull request Apr 11, 2023
@rhettinger @warsaw
pythonGH-102670: Use sumprod() to simplify, speed up, and improve acc…
f767f4b 
…uracy of statistics functions (pythonGH-102649)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eendebakpt eendebakpt eendebakpt approved these changes
+1 more reviewer
@alexanderGerbik alexanderGerbik alexanderGerbik left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

3.12 only security fixes performance Performance or resource usage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@rhettinger @eendebakpt @alexanderGerbik @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.