A small Go storefront API that emits logs, metrics and traces continuously in the messy shapes which Bindplane exists to clean up — unparsed text logs, PII in log bodies, multi-line stack traces, floods of low-value DEBUG, metrics whose cardinality you can tune on command. Every one of those knobs is adjustable at runtime over HTTP, while the Bindplane pipeline is running.
flowchart LR
you["curl /admin/chaos"]
subgraph compose["docker compose"]
direction LR
lg["`checkout-loadgen
~5 rps, bursts`"]
app["checkout-service
Go + OTel SDK
:8080"]
vol[("applogs volume
app.json · access.log
runtime.log")]
col["bdot-collector
OTLP
:4317 / :4318"]
lg -->|"HTTP :8080"| app
app -->|"OTLP gRPC
logs · metrics · traces"| col
app -->|"writes"| vol
vol -->|"ro mount
/var/log/checkout"| col
end
cloud["app.bindplane.com"]
dest[("destination")]
you --> app
col <-.->|"`OpAMP wss
pipeline config`"| cloud
col -->|"`per the config
you built`"| dest
classDef ext fill:#f5f5f5,stroke:#999,color:#111,stroke-dasharray:4 3
class cloud,dest ext
| Container | What it does |
|---|---|
checkout-service |
The API, on localhost:8080. Writes log files to the shared applogs volume. |
checkout-loadgen |
~5 rps of mixed traffic, tripling for 20s every 2 minutes so throughput graphs have shape. |
bdot-collector |
BDOT collector. OTLP on 4317/4318, health on 13133, zPages on 55679. Mounts applogs read-only at /var/log/checkout. |
cmd/server the service
cmd/loadgen traffic generator (bursts every 2 minutes)
internal/telemetry OTel SDK setup: resource + logs/metrics/traces exporters
internal/applog slog fan-out: stdout, JSON file, access log, raw text, OTLP
internal/scenario chaos dials + fake customers, products, failures, panics
internal/api HTTP handlers, spans, metric instruments
| Signal | How it leaves the process | What it exercises in Bindplane |
|---|---|---|
| Logs (OTLP) | otlploggrpc → collector :4317 |
OTLP source; structured records, trace-correlated |
| Logs (JSON file) | <LOG_DIR>/app.json |
File source + JSON parsing |
| Logs (text file) | <LOG_DIR>/access.log |
File source + regex / Apache parsing |
| Logs (raw text) | <LOG_DIR>/runtime.log |
File source + multiline / recombine |
| Logs (stdout) | container stdout, JSON | Container / journald source |
| Metrics | otlpmetricgrpc, every 10s |
Order counters, value + latency histograms, inventory, Go runtime |
| Traces | otlptracegrpc |
orders.create → validate → inventory → payment → persist |
Every structured log carries trace_id / span_id, so log↔trace correlation works in whatever backend you route to.
Docker with Compose v2 is the only prerequisite — the Go toolchain lives inside the build image, so you never need it installed.
The collector is BDOT, the Bindplane Distribution for OpenTelemetry — an OTel Collector with extra receivers/processors plus an OpAMP client, so app.bindplane.com can manage it remotely. This repo runs it in managed mode only: nothing else to stand up, and no YAML here to drift out of sync with the UI.
That makes an install secret key the one thing you have to supply. In the UI: Agents → Install Agent — choose any platform, and the command it prints contains the key. Copy just the key:
cp .env.example .env
# .env
OPAMP_ENDPOINT=wss://app.bindplane.com/v1/opamp
OPAMP_SECRET_KEY=<paste it here>
OPAMP_LABELS=app=checkout,env=demo,ephemeral=true
The endpoint and labels are already defaulted in docker-compose.yaml, so in
practice the secret key is the only value you fill in. Then bring up the app, a
load generator, and the collector:
# docker compose up --build -d
make upConfirm it works:
curl -s localhost:8080/healthz curl -sX POST localhost:8080/api/orders -d '{"sku":"SKU-1002","quantity":2}' # tail the collector make logs
The collector should appear in the UI under Agents within a few seconds,
labelled app=checkout, env=demo.
Tear down, log volume included:
# docker compose down -v
make down Every variable is optional. .env.example covers the three OpAMP variables that the
collector needs, and docker-compose.yaml sets the app variables that need
non-default values inside the containers — ERROR_RATE, PII_RATE,
PANIC_RATE and NOISE_LOGS among them, seeding the chaos dials that
/admin/chaos tunes at runtime. The three worth knowing about:
| Variable | Default | Meaning |
|---|---|---|
LOG_DIR |
./logs |
Where the three log files go; empty disables file logging |
LOG_LEVEL |
debug |
debug / info / warn / error |
MANAGER_YAML_PATH |
set in compose | Where the collector persists its identity. Unset drops it into standalone mode, which this setup never wants |
OPAMP_LABELS is how Bindplane targets configurations at fleets, and
OPAMP_SECRET_KEY is the only value you must supply yourself.
GET / service info; 404s every other path
GET /healthz
GET /api/products
POST /api/orders {"sku":"SKU-1002","quantity":2}
GET /api/orders/{id}
GET /admin/chaos
POST /admin/chaos
About 2% of the generated traffic asks for /wp-admin/login.php, so there is a
steady trickle of 404s and route not found WARN records to filter or route on
— status code in the access log, severity everywhere else.
With the stack up and the agent enrolled, everything below happens in the Bindplane UI — none of it lives in this repo.
a. Create the configuration. Configurations → New, platform Linux, then add:
Sources
- OTLP — gRPC
0.0.0.0:4317, HTTP0.0.0.0:4318. Picks up logs, metrics and traces from the Go SDK in one shot. - File — path
/var/log/checkout/app.json. Add a Parse JSON processor on the source, parse frombody, map severity from thelevelfield. - File — path
/var/log/checkout/access.log. Add Parse with Regex; this is the one to do live, because the preview pane shows extracted fields before you deploy anything:^(?P<client_ip>\S+) \S+ (?P<user>\S+) \[(?P<ts>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) (?P<proto>[^"]+)" (?P<status>\d{3}) (?P<bytes>\d+) "(?P<referer>[^"]*)" "(?P<user_agent>[^"]*)" (?P<duration_s>[\d.]+)$userholds the shopper's email address wheneverpii_ratefires and-otherwise, so this source is a masking target as well as a parsing one. - File — path
/var/log/checkout/runtime.log, with multiline enabled and line start pattern^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}. Without this, each panic arrives as a dozen orphaned single-line records. The dump also carries arequest context:line of card numbers and tokens, so masking here only works once recombination does.
Processors worth adding, in this order
| Processor | Why |
|---|---|
| Mask Sensitive Data | Card numbers, SSNs and bearer tokens go downstream redacted while the raw file on disk stays untouched. PII reaches all four surfaces, so it has to be masked on all four |
| Filter by Severity / Sampling | Drop the DEBUG flood and watch the throughput graph fall |
| Delete Attribute / Aggregate Metrics | Strip customer.id back off the metrics it inflates |
| Add Fields | Stamp team, cost_center |
Destinations — Bindplane's own Dev Null or a File destination is enough to demo the mechanics; add a second destination and route by severity to show fan-out.
b. Roll it out. Attach the configuration to the agent and deploy.
To show fleet management, run more than one collector and push a single config
change to all of them by label. Compose refuses to scale a service that has a
fixed container_name, so comment out container_name: bdot-collector in
docker-compose.yaml first, then:
docker compose up -d --scale collector=3
Three agents enroll with the same app=checkout labels; change the
configuration once and watch all three roll. That is the part a hand-written
YAML file on a box cannot do.
POST /admin/chaos takes a partial body, so each call changes only the
dials you name. Change the shape of the data mid-conversation, then show the
processor absorbing it.
| Dial | Meaning | make shortcut |
|---|---|---|
noise_logs_per_request |
Extra DEBUG lines per request | chaos-noise → 60, a ~20× spike; filter / sample / reduce |
pii_rate |
Fraction logging card numbers, SSNs, emails, bearer tokens | chaos-pii → every request; mask sensitive data |
error_rate |
Fraction of orders that fail (declines, timeouts, 429s, DB resets) | chaos-errors → 50% failures, with panic_rate 0.25 |
panic_rate |
Fraction that dump a multi-line Go stack trace | ↑ same target; multiline, severity routing |
extra_latency_ms |
Added to every payment call | chaos-slow → +1.5s; latency histogram moves |
high_cardinality |
Adds customer.id to metric attributes |
chaos-cardinality; delete attribute / aggregate |
make chaos prints the current values, make chaos-reset returns to baseline,
and anything the targets do not cover you can set by hand:
curl -sX POST localhost:8080/admin/chaos -d '{"pii_rate":0.5,"extra_latency_ms":400}'- Start clean.
make up, show the agent enrolling and all three signals arriving. - Parse. Add the
access.logsource and build the regex live in the preview pane. - Mask.
make chaos-pii, show the leak, add the processor, show it gone — without redeploying the app. - Reduce.
make chaos-noise, then filter. Talk about the bill. - Route. Errors to one destination, everything else to cheap archive.
- Fleet. Scale to three collectors, roll a change out by label.
| Symptom | Cause |
|---|---|
required variable OPAMP_SECRET_KEY is missing |
No .env, or the key is blank. cp .env.example .env and paste the key from Agents → Install Agent. |
Collector logs Starting In Managed Mode then unsupported URL |
OPAMP_ENDPOINT is empty or malformed. It must be a full wss://.../v1/opamp URL. |
open /out/...: permission denied |
The collector runs as user otel; it cannot write to a root-owned bind mount. Write inside /etc/otel/storage. |
| Access-log records never reach the destination | A severity filter is dropping them. Parsed access logs have no severity, so a rule like severity_number < SEVERITY_NUMBER_INFO matches UNSPECIFIED (0) and silently eats every one. Scope the filter to the other sources, or guard with severity_number != SEVERITY_NUMBER_UNSPECIFIED and .... |
| Changed the pipeline in Bindplane, nothing changed | The configuration has to be rolled out to the agent, not just saved. |
| No file logs at all | LOG_DIR unset or not writable. In Docker both containers must share the applogs volume. |
| Code change not taking effect | make up passes --build, but a bare docker compose up -d does not rebuild the image. |
make chaos-noise does nothing |
The noise is DEBUG. If LOG_LEVEL is above debug the records are never emitted in the first place. |
- Log files are never rotated. For a long-running demo, cap them or add
logrotate— or leave it and use it as an excuse to talk about how the collector handles file rotation. - Card numbers, SSNs and tokens are randomly generated fakes.
- The collector image is pinned to
bindplane-agent:1.84.0. Bump it if Bindplane Cloud has moved on by the time you demo — a stale agent version shows up as a warning on the Agents page.