GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GitHub - samber/slog-sampling: 🚨 slog sampling: drop repetitive log records

BranchesTags
Open more actions menu

Latest commit

 

History

103 Commits
103 Commits

Folders and files

NameName
Last commit message
Last commit date
.github
.github
 
 
buffer
buffer
 
 
example
example
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
benchmark_test.go
benchmark_test.go
 
 
counters.go
counters.go
 
 
counters_benchmark_test.go
counters_benchmark_test.go
 
 
counters_test.go
counters_test.go
 
 
fuzz_test.go
fuzz_test.go
 
 
go.mod
go.mod
 
 
go.sum
go.sum
 
 
hash.go
hash.go
 
 
hook.go
hook.go
 
 
main_test.go
main_test.go
 
 
matchers.go
matchers.go
 
 
matchers_benchmark_test.go
matchers_benchmark_test.go
 
 
matchers_test.go
matchers_test.go
 
 
middleware_absolute.go
middleware_absolute.go
 
 
middleware_absolute_test.go
middleware_absolute_test.go
 
 
middleware_custom.go
middleware_custom.go
 
 
middleware_custom_test.go
middleware_custom_test.go
 
 
middleware_threshold.go
middleware_threshold.go
 
 
middleware_threshold_test.go
middleware_threshold_test.go
 
 
middleware_uniform.go
middleware_uniform.go
 
 
middleware_uniform_test.go
middleware_uniform_test.go
 
 
random.go
random.go
 
 
random_benchmark_test.go
random_benchmark_test.go
 
 
random_test.go
random_test.go
 
 

Repository files navigation

Slog sampling policy

tag Go Version GoDoc Build Status Go report Coverage Contributors License

A middleware that samples incoming records which caps the CPU and I/O load of logging while attempting to preserve a representative subset of your logs.

Sampling fixes throughput by dropping repetitive log entries.


Sponsored by:
Cast AI
Cut Kubernetes & AI costs, boost application stability

Dash0
100% OpenTelemetry-native observability platform
Simple to use, built on open standards, and designed for full cost control

See also:

  • slog-multi: slog.Handler chaining, fanout, routing, failover, load balancing...
  • slog-formatter: slog attribute formatting
  • slog-sampling: slog sampling policy
  • slog-mock: slog.Handler for test purposes

HTTP middlewares:

  • slog-gin: Gin middleware for slog logger
  • slog-echo: Echo middleware for slog logger
  • slog-fiber: Fiber middleware for slog logger
  • slog-chi: Chi middleware for slog logger
  • slog-http: net/http middleware for slog logger

Loggers:

  • slog-zap: A slog handler for Zap
  • slog-zerolog: A slog handler for Zerolog
  • slog-logrus: A slog handler for Logrus

Log sinks:

  • slog-datadog: A slog handler for Datadog
  • slog-betterstack: A slog handler for Betterstack
  • slog-rollbar: A slog handler for Rollbar
  • slog-loki: A slog handler for Loki
  • slog-sentry: A slog handler for Sentry
  • slog-syslog: A slog handler for Syslog
  • slog-logstash: A slog handler for Logstash
  • slog-fluentd: A slog handler for Fluentd
  • slog-graylog: A slog handler for Graylog
  • slog-quickwit: A slog handler for Quickwit
  • slog-slack: A slog handler for Slack
  • slog-telegram: A slog handler for Telegram
  • slog-mattermost: A slog handler for Mattermost
  • slog-microsoft-teams: A slog handler for Microsoft Teams
  • slog-webhook: A slog handler for Webhook
  • slog-kafka: A slog handler for Kafka
  • slog-nats: A slog handler for NATS
  • slog-parquet: A slog handler for Parquet + Object Storage
  • slog-channel: A slog handler for Go channels

🚀 Install

go get github.com/samber/slog-sampling

Compatibility: go >= 1.21

No breaking changes will be made to exported APIs before v2.0.0.

💡 Usage

GoDoc: https://pkg.go.dev/github.com/samber/slog-sampling

Middlewares

3 strategies are available:

  • Uniform sampling: drop % of logs
  • Threshold sampling: drop % of logs after a threshold
  • Absolute sampling: limit logs throughput to a fixed number of records
  • Custom sampler

The sampling middleware can be used standalone or with the slog-multi helpers.

A combination of multiple sampling strategies can be chained. Eg:

  • drop when a single log message is produced more than 100 times per second
  • drop above 1000 log records per second (globally)

Matchers

Similar log records can be deduplicated and rate-limited using the Matcher API.

Available Matcher:

  • slogsampling.MatchByLevelAndMessage (default)
  • slogsampling.MatchAll
  • slogsampling.MatchByLevel
  • slogsampling.MatchByMessage
  • slogsampling.MatchBySource
  • slogsampling.MatchByAttribute
  • slogsampling.MatchByContextValue

Uniform sampling

type UniformSamplingOption struct {
    // The sample rate for sampling traces in the range [0.0, 1.0].
    Rate float64

    // Optional hooks
    OnAccepted func(context.Context, slog.Record)
    OnDropped  func(context.Context, slog.Record)
}

Example using slog-multi:

import (
    slogmulti "github.com/samber/slog-multi"
    slogsampling "github.com/samber/slog-sampling"
    "log/slog"
)

// Will print 33% of entries.
option := slogsampling.UniformSamplingOption{
	// The sample rate for sampling traces in the range [0.0, 1.0].
    Rate:       0.33,
}

logger := slog.New(
    slogmulti.
        Pipe(option.NewMiddleware()).
        Handler(slog.NewJSONHandler(os.Stdout, nil)),
)

Threshold sampling

type ThresholdSamplingOption struct {
    // This will log the first `Threshold` log entries with the same hash,
    // in a `Tick` interval as-is. Following that, it will allow `Rate` in the range [0.0, 1.0].
    Tick       time.Duration
    Threshold  uint64
    Rate       float64

    // Group similar logs (default: by level and message)
    Matcher func(ctx context.Context, record *slog.Record) string

    // Optional hooks
    OnAccepted func(context.Context, slog.Record)
    OnDropped  func(context.Context, slog.Record)
}

If Rate is zero, the middleware will drop all log entries after the first Threshold records in that interval.

Example using slog-multi:

import (
    slogmulti "github.com/samber/slog-multi"
    slogsampling "github.com/samber/slog-sampling"
    "log/slog"
)

// Will print the first 10 entries having the same level+message, then every 10th messages until next interval.
option := slogsampling.ThresholdSamplingOption{
    Tick:       5 * time.Second,
    Threshold:  10,
    Rate:       0.1,
}

logger := slog.New(
    slogmulti.
        Pipe(option.NewMiddleware()).
        Handler(slog.NewJSONHandler(os.Stdout, nil)),
)

Absolute sampling

type AbsoluteSamplingOption struct {
    // This will log all entries with the same hash until max is reached,
    // in a `Tick` interval as-is. Following that, it will reduce log throughput
    // depending on previous interval.
    Tick time.Duration
    Max  uint64

    // Group similar logs (default: by level and message)
    Matcher Matcher

    // Optional hooks
    OnAccepted func(context.Context, slog.Record)
    OnDropped  func(context.Context, slog.Record)
}

Example using slog-multi:

import (
    slogmulti "github.com/samber/slog-multi"
    slogsampling "github.com/samber/slog-sampling"
    "log/slog"
)

// Will print the first 10 entries during the first 5s, then a fraction of messages during the following intervals.
option := slogsampling.AbsoluteSamplingOption{
    Tick:       5 * time.Second,
    Max:        10,

    Matcher: slogsampling.MatchAll(),
}

logger := slog.New(
    slogmulti.
        Pipe(option.NewMiddleware()).
        Handler(slog.NewJSONHandler(os.Stdout, nil)),
)

Custom sampler

type CustomSamplingOption struct {
    // The sample rate for sampling traces in the range [0.0, 1.0].
    Sampler func(context.Context, slog.Record) float64

    // Optional hooks
    OnAccepted func(context.Context, slog.Record)
    OnDropped  func(context.Context, slog.Record)
}

Example using slog-multi:

import (
    slogmulti "github.com/samber/slog-multi"
    slogsampling "github.com/samber/slog-sampling"
    "log/slog"
)

// Will print 100% of log entries during the night, or 50% of errors, 20% of warnings and 1% of lower levels.
option := slogsampling.CustomSamplingOption{
    Sampler: func(ctx context.Context, record slog.Record) float64 {
        if record.Time.Hour() < 6 || record.Time.Hour() > 22 {
            return 1
        }

        switch record.Level {
        case slog.LevelError:
            return 0.5
        case slog.LevelWarn:
            return 0.2
        default:
            return 0.01
        }
    },
}

logger := slog.New(
    slogmulti.
        Pipe(option.NewMiddleware()).
        Handler(slog.NewJSONHandler(os.Stdout, nil)),
)

🤝 Contributing

  • Ping me on twitter @samuelberthe (DMs, mentions, whatever :))
  • Fork the project
  • Fix open issues or request new features

Don't hesitate ;)

# Install some dev dependencies
make tools

# Run tests
make test
# or
make watch-test

👤 Contributors

Contributors

💫 Show your support

Give a ⭐️ if this project helped you!

GitHub Sponsors

📝 License

Copyright © 2023 Samuel Berthe.

This project is MIT licensed.

About

🚨 slog sampling: drop repetitive log records

pkg.go.dev/github.com/samber/slog-sample

Topics

gogolanghandlerlog-levelloggerloggingmiddlewarerate-limitingsamplingslogstructured-loggingthroughput

Resources

Readme
MIT license

Code of conduct

Code of conduct

Security policy

Security policy
Activity

Stars

120 stars

Watchers

2 watching

Forks

4 forks
Report repository

Releases

Sponsor this project

Used by

Contributors

Languages

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.