GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Refactor benchmarks by fdcastel · Pull Request #1203 · FirebirdSQL/NETProvider

Conversation

@fdcastel

@fdcastel fdcastel commented Oct 26, 2024

Copy link
Copy Markdown
Member

Based on #1200 for now. I will rebase it with latest master in the future.

  • Rename project Perf to FirebirdSql.Data.FirebirdClient.Benchmarks.
  • Update project to use .NET8.
  • Upgrade BenchmarkDotNet to version 0.14.0.
  • Update baseline nuget package to v10.3.1.
  • Add /BenchmarkDotNet.Artifacts folder to .gitignore.
  • Pass command-line arguments to BenchmarkDotNet engine.
  • Apply SQL Formatting. Use raw strings.
  • Add script run-benchmark.ps1.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
This was referenced Oct 26, 2024
Adds .NET distributed tracing instrumentation & metrics. #1200
Open
Update benchmarks cincuranet/FirebirdSql.Data.FirebirdClient#129
Closed
willibrandon
willibrandon reviewed Oct 26, 2024
View reviewed changes
Comment thread src/FirebirdSql.Data.FirebirdClient.Benchmarks/CommandBenchmark.cs Resolved

Uh oh!

There was an error while loading. Please reload this page.

@fdcastel
fdcastel force-pushed the refactor-benchmarks branch from bd2e3a0 to e5e4075 Compare May 18, 2025 01:19
@fdcastel

fdcastel commented May 18, 2025

Copy link
Copy Markdown
Member Author

Rebase with master.

cincuranet reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fdcastel
fdcastel force-pushed the refactor-benchmarks branch 2 times, most recently from 2a8789b to 1ec795d Compare March 23, 2026 03:31
@fdcastel

fdcastel commented Mar 23, 2026

Copy link
Copy Markdown
Member Author

Following up on the initial refactoring, these additional changes were made on the branch:

Infrastructure

  • Extracted a shared BenchmarkConfig : ManualConfig class to eliminate the duplicated inner Config class that existed independently in each benchmark class. Job configuration is now defined in a single place.
  • Extracted a BenchmarkBase abstract class holding the ConnectionString, CreateDatabase(), and GlobalCleanup() members that were duplicated across benchmark classes.
  • The connection string can now be overridden via the FIREBIRD_BENCHMARK_CS environment variable (falls back to the existing localhost default), enabling use in environments with a non-default Firebird setup.

Configuration

  • Updated the benchmark toolchain from .NET 8 to .NET 10, matching the project's TargetFramework.
  • Removed WithPlatform(Platform.X64) and WithJit(Jit.RyuJit) — both are no-ops on .NET 8+ (x64 + RyuJIT is the only supported configuration) and were just noise.
  • Added MarkdownExporter.GitHub so each run automatically writes a GitHub-flavored Markdown table to BenchmarkDotNet.Artifacts/, ready to paste into issues or PRs.
  • Added DefaultOrderer(SummaryOrderPolicy.FastestToSlowest) to make the results table easier to scan.

Benchmark methods

  • Single-value [Params(100)] and [Params(100_000)] replaced with plain constants — a single-value [Params] adds a column to the results table without varying anything.
  • Fetch methods now return the last read value instead of discarding it, to make dead-code elimination by the JIT impossible.
  • Added async variants (ExecuteAsync, FetchAsync, OpenCloseAsync) for all sync benchmarks. Async ADO.NET paths have different allocation profiles (state machines, Task wrappers) and are the primary path in web workloads.

New benchmarks

  • ConnectionBenchmark — measures connection open/close throughput (sync and async), which exercises connection pool logic.
  • LargeFetchBenchmark — added to the committed tree (was previously an untracked file).

Script

  • run-benchmark.ps1 now accepts -Benchmark ConnectionBenchmark in addition to the existing values, and two new optional switches: -Disasm (enables --disasm) and -Profile (enables ETW profiling on Windows).

Documentation

  • Added docs/benchmark.md describing prerequisites, how to run, connection string configuration, available benchmarks, and where results are written.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fdcastel fdcastel mentioned this pull request Mar 23, 2026
(Improvements) Rune operations optimization #1252
Open
@fdcastel

fdcastel commented Mar 23, 2026

Copy link
Copy Markdown
Member Author

Future improvements:

  1. async benchmarks
  2. connection-pool benchmarks
  3. configurable connection string
  4. Remove Duplicated config (use a shared base)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pl752 pl752 mentioned this pull request May 21, 2026
Major performance degradation introduced with 10.3.3 (UTF8 char fetch) #1272
Open
fdcastel added 19 commits May 23, 2026 16:14
@fdcastel
Adds FbTransactionInfo.GetTransactionId().
2b3905c
@fdcastel
Adds .NET distributed tracing instrumentation.
7693055
@fdcastel
Add metrics.
4ef4ceb
@fdcastel
Fix NullReferenceException when StartActivity returns null
caf154c
StartActivity() returns null when no listeners are interested in the
activity. The code accessed activity.IsAllDataRequested without a null
check, causing a NullReferenceException. Use the Npgsql-style pattern
'activity is not { IsAllDataRequested: true }' for early return.
@fdcastel
Fix Stopwatch elapsed time conversion
49ac081
Stopwatch ticks are not TimeSpan ticks when Stopwatch.IsHighResolution
is true (most systems). TimeSpan.FromTicks(stopwatchTicks) produces
incorrect durations. Use Stopwatch.GetElapsedTime() which correctly
handles the frequency conversion.
@fdcastel
Rename db.system to db.system.name per OTel semantic conventions v1.40+
a19bd65
The semantic conventions renamed db.system to db.system.name. The
well-known value for Firebird is 'firebirdsql', not 'firebird'.
Updated in both FbActivitySource (spans) and FbConnection (metrics).
@fdcastel
Set db.namespace attribute on spans from connection database name
259bf5b
The semantic conventions mark db.namespace as Conditionally Required
when available. The connection's Database property is used as the value.
@fdcastel
Set error.type attribute on failure spans
3863d5b
The semantic conventions mark error.type as Conditionally Required on
failure. For FbException, use the SQLSTATE code; for other exceptions,
use the full exception type name.
@fdcastel
Gate db.query.text and db.query.parameter.* behind opt-in flags
43eff96
Per OTel semantic conventions, db.query.text and db.query.parameter.*
are Opt-In level attributes that may expose sensitive data. They should
not be collected by default. Added IsQueryTextTracingEnabled flag and
reuse IsParameterLoggingEnabled for parameter tracing.
@fdcastel
Read ActivitySource/Meter version from assembly metadata
18de3ee
Replace hardcoded '1.0.0' version with the assembly's informational
version, keeping the telemetry version in sync with the NuGet package.
@fdcastel
Fix activity lifecycle: record metrics on error path, prevent stale r…
1556b05
…ecording

TraceCommandException now records metrics immediately and resets
_startedAtTicks so the later TraceCommandStop (called during Release)
does not record a second, inflated duration that includes cleanup time.
@fdcastel
Remove non-standard and deprecated telemetry attributes
b3739dc
- Remove db.transaction_id custom tag (not in OTel spec, causes server
  round-trip per traced command)
- Remove commented-out db.snapshot_id code
- Remove deprecated exception.escaped attribute
- Remove NormalizeDbNull helper, inline DBNull check
@fdcastel
Add server.port, db.stored_procedure.name; fix server.address in metrics
25b2457
- Set server.port on spans when using a non-default port
- Set db.stored_procedure.name for StoredProcedure command types
- Fix server.address in MetricsConnectionAttributes to not concatenate
  the port (server.port is now a separate attribute)
- Clean up leftover placeholder comments
@fdcastel
Add db.query.summary and db.operation.name for Text commands
25a5bbf
Set db.query.summary as the low-cardinality activity name on all spans.
For Text commands, extract the first SQL verb (SELECT, INSERT, etc.)
and use it as db.operation.name and in the activity name.
@fdcastel
Add debug assertions in TraceCommandStop and TraceCommandException
d0e513c
Debug.Assert checks that TraceCommandStart was called before the stop
or exception path, helping catch lifecycle bugs during development.
@fdcastel
Initialize MetricsConnectionAttributes to empty array
8007f3d
Prevents potential NullReferenceException if metrics are recorded
before the connection string is set.
@fdcastel
Avoid dictionary allocation in observable metric callbacks
0cf25aa
Change GetMetrics() to return IEnumerable of tuples via Select instead
of allocating a Dictionary via ToDictionary on every collection cycle.
Also simplify GetConnectionMax to use Select instead of SelectMany.
@fdcastel
Add InstrumentAdvice with histogram bucket boundaries on .NET 9+
55bfb50
Suggest default histogram bucket boundaries for operation duration and
connection create time histograms, improving out-of-the-box resolution.
Guarded by NET9_0_OR_GREATER since InstrumentAdvice is a .NET 9 API.
@fdcastel
Add FbTelemetry public class with ActivitySource and Meter names
6584b87
Expose the telemetry source names as public constants so consumers can
easily subscribe: builder.AddSource(FbTelemetry.ActivitySourceName) and
builder.AddMeter(FbTelemetry.MeterName). Internal code updated to use
the constants.
fdcastel added 17 commits May 23, 2026 16:14
@fdcastel
Add OpenTelemetry integration documentation
3b5e155
Document the ActivitySource and Meter names, available span attributes,
opt-in sensitive attributes, and available metrics with their types and
semantic convention references.
@fdcastel
Fix IndexOfAny compilation on all target frameworks
e1ad719
@fdcastel
Refactor: Benchmarks.
96c596d
- Rename project 'Perf' to 'FirebirdSql.Data.FirebirdClient.Benchmarks'.
- Update project to use .net8.
- Upgrade BenchmarkDotNet to version 0.14.0.
- Update baseline nuget package to v10.3.1.
- Add /BenchmarkDotNet.Artifacts to .gitignore.
- Pass command-line arguments to BenchmarkDotNet engine.
- Apply SQL Formatting. Use raw strings.
- Add script run-benchmark.ps1.
@fdcastel
Updates run-benchmark.ps1.
dfde2ae
@fdcastel
Fix solution file: update benchmark project path and folder name
554807d
@fdcastel
Add LargeFetchBenchmark for bulk read throughput across data types
fedb3c9
@fdcastel
Extract shared BenchmarkConfig to eliminate duplication
bffabe8
@fdcastel
Extract BenchmarkBase with shared ConnectionString and env-var override
d60ebd2
@fdcastel
Update benchmark toolchain from .NET 8 to .NET 10
3ea32fb
@fdcastel
Add GitHub Markdown exporter and fastest-to-slowest ordering
eda44be
@fdcastel
Replace single-value [Params] with constants
a66c5fa
@fdcastel
Return values from Fetch benchmarks to prevent dead-code elimination
a150a18
@fdcastel
Add async benchmark variants for Execute and Fetch
3a8e509
@fdcastel
Add connection open/close benchmark
c56595f
@fdcastel
Add -Disasm and -Profile flags to run-benchmark.ps1
2359fa3
@fdcastel
Fix build errors: remove duplicate braces, add missing using
3e12746
@fdcastel
Add benchmark documentation
9d42b35
@fdcastel
fdcastel force-pushed the refactor-benchmarks branch from 533d57c to 9d42b35 Compare May 23, 2026 20:04
@fdcastel

fdcastel commented May 23, 2026

Copy link
Copy Markdown
Member Author

Rebased onto latest master.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fdcastel
Add CHAR UTF8 coverage to CommandBenchmark and document regression de…
46db5a1
…tection

CommandBenchmark (the default suite) only covered BIGINT and VARCHAR UTF8, so
the fixed-length CHAR(n) UTF8 rune path was untested -- the exact path where
issue FirebirdSQL#1272 (~11.7x allocation regression) lives. That regression surfaced only
via the opt-in LargeFetchBenchmark.

- CommandBenchmark: add "CHAR(100) CHARACTER SET UTF8" param; add SampleValue()
  so Execute sends a column-sized string (exercising the write/validate rune
  count path) instead of a bare int.
- docs/benchmark.md: correct the NuGet baseline description (pinned version plus
  a bump-per-release note), add an "Interpreting results" section noting that
  Alloc Ratio is the primary regression signal (with FirebirdSQL#1272 as a worked example),
  and list the third data type.

Verified locally against the released baseline: CHAR UTF8 Fetch Alloc Ratio
~9.6x and Execute ~1.9x, while BIGINT and VARCHAR stay ~1.0.
@fdcastel

fdcastel commented May 23, 2026

Copy link
Copy Markdown
Member Author

Added a couple of benchmark improvements prompted by #1272 (the UTF8 CHAR fetch allocation regression). The lesson from that investigation: the regression lives only in the fixed-length CHAR(n) CHARACTER SET UTF8 rune path, but the default CommandBenchmark only covered BIGINT and VARCHAR(10) UTF8, so it never surfaced there — it was visible only via the opt-in LargeFetchBenchmark. These changes make the default suite catch that class of regression and make the results easier to read.

What changed (commit 46db5a1)

  • CommandBenchmark: added CHAR(100) CHARACTER SET UTF8 to the DataType params. This covers the read/truncation path in Fetch/FetchAsync automatically.
  • CommandBenchmark.Execute: Execute/ExecuteAsync now send a column-sized string (via a small SampleValue() helper) instead of a bare integer, so the write/validate rune-count path is exercised too — previously it was never benchmarked.
  • docs/benchmark.md: corrected the NuGet baseline description (it claimed "latest published" but the project pins a specific released version; it now documents the pin plus a bump-per-release note), added an Interpreting results section explaining that for these I/O-bound benchmarks the Alloc Ratio (not the noisy Mean) is the primary regression signal — using Major performance degradation introduced with 10.3.3 (UTF8 char fetch) #1272 as a worked example — and listed the new data type.

Effect — the default benchmark now catches #1272

Running CommandBenchmark on this branch, local Core100 vs the released NuGet100 baseline:

DataType Allocated (baseline → local) Alloc Ratio
BIGINT — Fetch 46.3 KB → 50.9 KB 1.10
VARCHAR(10) UTF8 — Fetch 50.2 KB → 54.8 KB 1.09
CHAR(100) UTF8 — Fetch 187.7 KB → 1805.1 KB 9.61
CHAR(100) UTF8 — FetchAsync 209.6 KB → 1825.1 KB 8.71
CHAR(100) UTF8 — Execute 409.5 KB → 759.9 KB 1.86
CHAR(100) UTF8 — ExecuteAsync 977.9 KB → 1329.1 KB 1.36

Only the CHAR UTF8 cases regress; BIGINT and VARCHAR stay at ~1.0, which also matches the root cause (only DbDataType.Char goes through the rune-truncation path). These ratios reflect the current provider code on master — once the fix from #1252 lands they should return to ~1.0.

CI is green across FB30/FB40/FB50.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@willibrandon willibrandon willibrandon left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@fdcastel @willibrandon
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.