GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GitHub - inkloop/quantiles: Optimal Quantile Approximation in Streams

BranchesTags
Open more actions menu
 
 

Latest commit

 

History

28 Commits
28 Commits

Folders and files

NameName
Last commit message
Last commit date
demo
demo
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
buffer.go
buffer.go
 
 
buffer_test.go
buffer_test.go
 
 
stream.go
stream.go
 
 
stream_example_test.go
stream_example_test.go
 
 
stream_test.go
stream_test.go
 
 
summary.go
summary.go
 
 
summary_test.go
summary_test.go
 
 
utils.go
utils.go
 
 

Repository files navigation

quantiles - Optimal Quantile Approximation in Streams

This is a translation of TensorFlow's quantile helper class, it aims to compute approximate quantiles with error bound guarantees for weighted data sets. This implementation is an adaptation of techniques from the following papers:

  • (2001) Space-efficient online computation of quantile summaries.
  • (2004) Power-conserving computation of order-statistics over sensor networks.
  • (2007) A fast algorithm for approximate quantiles in high speed data streams.
  • (2016) XGBoost: A Scalable Tree Boosting System.

The key ideas at play are the following:

  • Maintain an in-memory multi-level quantile summary in a way to guarantee a maximum approximation error of eps * W per bucket where W is the total weight across all points in the input dataset.
  • Two base operations are defined: MERGE and COMPRESS. MERGE combines two summaries guaranteeing a epsNew = max(eps1, eps2). COMPRESS compresses a summary to b + 1 elements guaranteeing epsNew = epsOld + 1/b.
  • b * sizeof(summary entry) must ideally be small enough to fit in an average CPU L2 cache.
  • To distribute this algorithm with maintaining error bounds, we need the worker-computed summaries to have no more than eps / h error where h is the height of the distributed computation graph which is 2 for an MR with no combiner.

We mainly want to max out IO bw by ensuring we're not compute-bound and using a reasonable amount of RAM.

Complexity:

  • Compute: O(n * log(1/eps * log(eps * n))).
  • Memory: O(1/eps * log^2(eps * n)) <- for one worker streaming through the entire dataset.

An epsilon value of zero would make the algorithm extremely inefficent and therefore, is disallowed.

TODO

  • Implement an online estimator without the need of finalizing the stream
  • Add proper documentation
  • Benchmark
  • Add serialization

About

Optimal Quantile Approximation in Streams

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.