A middleware that samples incoming records which caps the CPU and I/O load of logging while attempting to preserve a representative subset of your logs.
Sampling fixes throughput by dropping repetitive log entries.
Sponsored by:
Simple to use, built on open standards, and designed for full cost control
See also:
- slog-multi:
slog.Handlerchaining, fanout, routing, failover, load balancing... - slog-formatter:
slogattribute formatting - slog-sampling:
slogsampling policy - slog-mock:
slog.Handlerfor test purposes
HTTP middlewares:
- slog-gin: Gin middleware for
sloglogger - slog-echo: Echo middleware for
sloglogger - slog-fiber: Fiber middleware for
sloglogger - slog-chi: Chi middleware for
sloglogger - slog-http:
net/httpmiddleware forsloglogger
Loggers:
- slog-zap: A
sloghandler forZap - slog-zerolog: A
sloghandler forZerolog - slog-logrus: A
sloghandler forLogrus
Log sinks:
- slog-datadog: A
sloghandler forDatadog - slog-betterstack: A
sloghandler forBetterstack - slog-rollbar: A
sloghandler forRollbar - slog-loki: A
sloghandler forLoki - slog-sentry: A
sloghandler forSentry - slog-syslog: A
sloghandler forSyslog - slog-logstash: A
sloghandler forLogstash - slog-fluentd: A
sloghandler forFluentd - slog-graylog: A
sloghandler forGraylog - slog-quickwit: A
sloghandler forQuickwit - slog-slack: A
sloghandler forSlack - slog-telegram: A
sloghandler forTelegram - slog-mattermost: A
sloghandler forMattermost - slog-microsoft-teams: A
sloghandler forMicrosoft Teams - slog-webhook: A
sloghandler forWebhook - slog-kafka: A
sloghandler forKafka - slog-nats: A
sloghandler forNATS - slog-parquet: A
sloghandler forParquet+Object Storage - slog-channel: A
sloghandler for Go channels
go get github.com/samber/slog-sampling
Compatibility: go >= 1.21
No breaking changes will be made to exported APIs before v2.0.0.
GoDoc: https://pkg.go.dev/github.com/samber/slog-sampling
3 strategies are available:
- Uniform sampling: drop % of logs
- Threshold sampling: drop % of logs after a threshold
- Absolute sampling: limit logs throughput to a fixed number of records
- Custom sampler
The sampling middleware can be used standalone or with the slog-multi helpers.
A combination of multiple sampling strategies can be chained. Eg:
- drop when a single log message is produced more than 100 times per second
- drop above 1000 log records per second (globally)
Similar log records can be deduplicated and rate-limited using the Matcher API.
Available Matcher:
slogsampling.MatchByLevelAndMessage(default)slogsampling.MatchAllslogsampling.MatchByLevelslogsampling.MatchByMessageslogsampling.MatchBySourceslogsampling.MatchByAttributeslogsampling.MatchByContextValue
type UniformSamplingOption struct { // The sample rate for sampling traces in the range [0.0, 1.0]. Rate float64 // Optional hooks OnAccepted func(context.Context, slog.Record) OnDropped func(context.Context, slog.Record) }
Example using slog-multi:
import ( slogmulti "github.com/samber/slog-multi" slogsampling "github.com/samber/slog-sampling" "log/slog" ) // Will print 33% of entries. option := slogsampling.UniformSamplingOption{ // The sample rate for sampling traces in the range [0.0, 1.0]. Rate: 0.33, } logger := slog.New( slogmulti. Pipe(option.NewMiddleware()). Handler(slog.NewJSONHandler(os.Stdout, nil)), )
type ThresholdSamplingOption struct { // This will log the first `Threshold` log entries with the same hash, // in a `Tick` interval as-is. Following that, it will allow `Rate` in the range [0.0, 1.0]. Tick time.Duration Threshold uint64 Rate float64 // Group similar logs (default: by level and message) Matcher func(ctx context.Context, record *slog.Record) string // Optional hooks OnAccepted func(context.Context, slog.Record) OnDropped func(context.Context, slog.Record) }
If Rate is zero, the middleware will drop all log entries after the first Threshold records in that interval.
Example using slog-multi:
import ( slogmulti "github.com/samber/slog-multi" slogsampling "github.com/samber/slog-sampling" "log/slog" ) // Will print the first 10 entries having the same level+message, then every 10th messages until next interval. option := slogsampling.ThresholdSamplingOption{ Tick: 5 * time.Second, Threshold: 10, Rate: 0.1, } logger := slog.New( slogmulti. Pipe(option.NewMiddleware()). Handler(slog.NewJSONHandler(os.Stdout, nil)), )
type AbsoluteSamplingOption struct { // This will log all entries with the same hash until max is reached, // in a `Tick` interval as-is. Following that, it will reduce log throughput // depending on previous interval. Tick time.Duration Max uint64 // Group similar logs (default: by level and message) Matcher Matcher // Optional hooks OnAccepted func(context.Context, slog.Record) OnDropped func(context.Context, slog.Record) }
Example using slog-multi:
import ( slogmulti "github.com/samber/slog-multi" slogsampling "github.com/samber/slog-sampling" "log/slog" ) // Will print the first 10 entries during the first 5s, then a fraction of messages during the following intervals. option := slogsampling.AbsoluteSamplingOption{ Tick: 5 * time.Second, Max: 10, Matcher: slogsampling.MatchAll(), } logger := slog.New( slogmulti. Pipe(option.NewMiddleware()). Handler(slog.NewJSONHandler(os.Stdout, nil)), )
type CustomSamplingOption struct { // The sample rate for sampling traces in the range [0.0, 1.0]. Sampler func(context.Context, slog.Record) float64 // Optional hooks OnAccepted func(context.Context, slog.Record) OnDropped func(context.Context, slog.Record) }
Example using slog-multi:
import ( slogmulti "github.com/samber/slog-multi" slogsampling "github.com/samber/slog-sampling" "log/slog" ) // Will print 100% of log entries during the night, or 50% of errors, 20% of warnings and 1% of lower levels. option := slogsampling.CustomSamplingOption{ Sampler: func(ctx context.Context, record slog.Record) float64 { if record.Time.Hour() < 6 || record.Time.Hour() > 22 { return 1 } switch record.Level { case slog.LevelError: return 0.5 case slog.LevelWarn: return 0.2 default: return 0.01 } }, } logger := slog.New( slogmulti. Pipe(option.NewMiddleware()). Handler(slog.NewJSONHandler(os.Stdout, nil)), )
- Ping me on twitter @samuelberthe (DMs, mentions, whatever :))
- Fork the project
- Fix open issues or request new features
Don't hesitate ;)
# Install some dev dependencies make tools # Run tests make test # or make watch-test
Give a ⭐️ if this project helped you!
Copyright © 2023 Samuel Berthe.
This project is MIT licensed.