GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [O2B-1564] Create background GAQ summary worker by isaachilly · Pull Request #2175 · AliceO2Group/Bookkeeping

Conversation

@isaachilly

@isaachilly isaachilly commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

I have a JIRA ticket

  • branch and/or PR name(s) include(s) JIRA ID
  • issue has "Fix version" assigned
  • issue "Status" is set to "In review"
  • PR labels are selected

Notable changes for users:

  • N/A

Notable changes for developers:

  • A new background worker running all the time at set intervals.

Changes made to the database:

  • N/A

A configurable background worker that runs a function that batch recalculates GAQ summaries that have an invalidatedAt column that is not null.

Dynamically adjusts batch size between a min and a max based on backlog.

Worker can be paused (useful for testing but maybe in prod for whatever reason).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
isaachilly added 30 commits March 5, 2026 14:51
@isaachilly
[O2B-1545] Add GAQ summary models, adapters and migration
bb5a9c3
@isaachilly
[O2B-1545] Add GAQ summary repositories and timestamps
4fade0f
Add GaqSummaryRepository and GaqSummaryInvalidationRepository and export them from the repositories index.

Update migration to replace the previous invalidated_at field with created_at and add updated_at to mirror default Sequelize tables.
@isaachilly
[O2B-1545] Use default Sequelize createdAt/updatedAt instead of compu…
15afdaa
…tedAt
@isaachilly
[O2B-1545] Use mcReproducibleCoverage float instead of boolean
363f09b
Rename and change mcReproducible to be the coverage float not the boolean.
@isaachilly
[O2B-1545] Remove mcReproducible field
002f301
Forgot to remove.
@isaachilly
[O2B-1563] Invalidate GAQ summaries on related changes
5bebbb5
Add GAQ summary invalidation whenever underlying data affecting GAQ changes.

These changes ensure GAQ summary caches/records are marked for recomputation whenever detectors, QC flags, or run QC times that influence GAQ summaries are modified.
@isaachilly
Add GAQ worker and summary recalculation
ee55f59
Introduce background processing for GAQ summary invalidations and wire it into the app scheduler. Changes include:

- Add gaq config to services config.
- Move GaqService to lib/server/services/gaq and update imports across controllers/use-cases/tests.
- Extend GaqService to pop invalidations and recalculate summaries within a transaction.
- Add GaqWorker that guards concurrent runs and calls GaqService to process a batch of invalid summaries.
- Schedule the GaqWorker in application startup when GAQ recalculation is enabled.
- Add a soft-delete filter (where: { deleted: false }) when querying QC flags to map ids to run numbers as otherwise summary invalidations occur even on already deleted flags.

These changes enable periodic recalculation of GAQ summaries when invalidations are queued in the table.
@isaachilly
[O2B-1564] Create computeSummary to separate getting and computing
530216b
@isaachilly
[O2B-1564] Exclude reproducible coverage from badEffectiveRunCoverage
e1f3223
@isaachilly
[O2B-1545] Rename GAQ summary coverage fields
16d3563
@isaachilly
Merge branch 'feature/O2B-1545/Create-GAQ-Summary-Tables' into featur…
9bf8027
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
@isaachilly
Merge branch 'feature/O2B-1563/Create-GAQ-summary-invalidation-mechan…
edcd9f7
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
@isaachilly
[O2B-1564] Rename coverage fields to match table cols in GaqService o…
87afd8c
…utput
@isaachilly
[O2B-1563] Add GAQ summary invalidation trigger tests
170c1b3
Validate GAQ summary invalidation behaviour on QC flag create/verify/delete, deleteAllForDataPass, explicit/default GAQ detector changes, and run QC time updates.
@isaachilly
[O2B-1563] Use dataPass.id in GAQ invalidation TYPO
78a2181
@isaachilly
Merge branch 'feature/O2B-1563/Create-GAQ-summary-invalidation-mechan…
a41b312
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
@isaachilly
Make GAQ worker singleton and add pause/resume
abb263b
Converted gaqWorker to a singleton.

This allows adding of pause() and resume() methods to prevent the worker from processing invalidated summaries during test execution.

Reduced the default GAQ recalculation period from 1 minute to 10 seconds to improve test suite performance.
@isaachilly
[O2B-1564] Add GAQ worker tests; pause worker in DB reset
10fac20
Adds tests that verify the worker removes an invalidation and adds a summary, upserts for an already present summary, batch processes correctly, and doesn't run concurrent recalculations.

Update resetDatabaseContent to pause/resume the GAQ worker to avoid worker failures when the invalidation table is dropped.
@isaachilly
[O2B-1564] Add GaqService tests and register suite
6aa9617
GaqService summary recalculation and invalidation processing functions tested directly circumventing worker.

Tests verify correct computed fields, summary upsert, behaviour when no coverage exists, and batched invalidation processing.
@isaachilly
Add calculationFailed to GAQ summaries
c2ccbd4
Add a calculation_failed boolean column to gaq_summaries. This allows us to know whether a summary has been attempted to be calculated but unsuccessful due to limited data etc.
@isaachilly
Merge branch 'feature/O2B-1545/Create-GAQ-Summary-Tables' into featur…
1e78cac
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
@isaachilly
Merge branch 'feature/O2B-1563/Create-GAQ-summary-invalidation-mechan…
f0dea1f
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
@isaachilly
[O2B-1545] Make GAQ summary columns nullable
35112e5
Remove not-null constraints from several GAQ summary columns in the migration to allow NULL when values are unavailable.
@isaachilly
Merge branch 'feature/O2B-1545/Create-GAQ-Summary-Tables' into featur…
c81cdf6
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
@isaachilly
Merge branch 'feature/O2B-1563/Create-GAQ-summary-invalidation-mechan…
5562afe
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
@isaachilly
[O2B-1564] Persist GAQ calculation failure
83ddf1d
Stop returning early when summary is missing; always upsert a GaqSummary record and include a calculationFailed boolean.

This ensures failed calculations are persisted for monitoring instead of being silently skipped.
@isaachilly
[O2B-1564] Assert calculationFailed in created summary
00c1e2b
Update test to expect that when there is no coverage data for a run, a GAQ summary is stored with calculationFailed set to true.
@isaachilly
[O2B-1545] Add invalidatedAt to gaq_summaries, remove invalidation table
5059082
Store invalidation timestamp directly on gaq_summaries instead of a separate gaq_summary_invalidations table.
@isaachilly
[O2B-1545] Rename calculationFailed to notComputable
9893667
@isaachilly
Merge branch 'feature/O2B-1545/Create-GAQ-Summary-Tables' into featur…
1588cd0
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
isaachilly added 15 commits May 5, 2026 12:46
@isaachilly
[O2B-1564] Improve logging message for GAQWorker
4943091
More details logged to help debugging the state of the GAQ worker.
@isaachilly
[O2B-1564] Add warning message when the num of invalidated summaries …
0586b3e
…consistently exceeds batch size

Also move any GAQ worker related logging to within the GAQ worker itself (separation of concerns).
@isaachilly
[O2B-1545] Add a composite primary key on GAQSummary table
f5f686c
Sequelize model omitted the primaryKey that migration file defines.
@isaachilly
Merge branch 'feature/O2B-1545/Create-GAQ-Summary-Tables' into featur…
3c7bce3
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
@isaachilly
[O2B-1563] Centralise GAQ summary invalidation and skip redundant work
9c1116b
Extract invalidation into helpers.
Guard GAQ updateRun reload so don't run reload unnecessarily.
Add to notComputable field in model allowNull default params.
Add more test coverage for invalidation on run patch.
@isaachilly
Merge branch 'feature/O2B-1563/Create-GAQ-summary-invalidation-mechan…
baa9bb0
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
@isaachilly
[O2B-1564] Overwrite summary values if couldn't be calculated
8992231
@isaachilly
[O2B-1564] Protect invalidatedAt from race condition
7481ec1
Protects invalidatedAt from being set to null if the datapass and runNumber becomes invalid for whatever reason in the middle of computeSummary.

ComputeSummary would overwrite the invalidatedAt column with null at the end.

Now it performs a check to prevent this.
@isaachilly
[O2B-1564] Fixed silent error in test
fe8062c
@isaachilly
[O2B-1564] Modify pause behaviour to account for in-flight recalculation
9851455
@isaachilly
[O2B-1564] Remove extra where option
d7dc453
@isaachilly
[O2B-1564] Make batchSize dynamic between a min and max
35272c7
Add a helper that logs a warning when the env var is set to an invalid value instead of silently falling back to the default.

If batchSize is not enough log the first warning after 5 consecutive overflow ticks, then every 30 ticks while it persists and recovery line when the streak ends so operators can grep for "backlog recovered".

Log at debug level when the worker is paused or resumed.
@isaachilly
Merge branch 'main' into feature/O2B-1563/Create-GAQ-summary-invalida…
ce0ac8c
…tion-mechanism
@isaachilly
Merge branch 'main' into feature/O2B-1545/Create-GAQ-Summary-Tables
f2f5f79
@isaachilly
Revert "Merge branch 'main' into feature/O2B-1563/Create-GAQ-summary-…
23f6e04
…invalidation-mechanism"

This reverts commit ce0ac8c, reversing
changes made to 9c1116b.
@isaachilly
isaachilly requested a review from graduta June 19, 2026 16:14
@isaachilly isaachilly self-assigned this Jun 19, 2026
@isaachilly isaachilly added the backend label Jun 19, 2026
@codecov

codecov Bot commented Jun 19, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.90323% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.49%. Comparing base (838e6a8) to head (79fc244).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
lib/server/services/gaq/GaqWorker.js 88.67% 6 Missing ⚠️
lib/database/adapters/GaqSummaryAdapter.js 50.00% 3 Missing ⚠️
...ons/v1/20260223120000-create-gaq-summary-tables.js 66.66% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2175      +/-   ##
==========================================
+ Coverage   45.82%   46.49%   +0.66%     
==========================================
  Files        1039     1043       +4     
  Lines       17189    17318     +129     
  Branches     3138     3168      +30     
==========================================
+ Hits         7877     8052     +175     
+ Misses       9312     9266      -46     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isaachilly isaachilly changed the title [O2B-1564] Ccreate background GAQ summary worker [O2B-1564] Create background GAQ summary worker Jun 29, 2026
isaachilly added 6 commits June 29, 2026 12:39
@isaachilly
[O2B-1564] Preserve GAQ worker pause state across resetDatabaseContent
cfbc5ac
resetDatabaseContent unconditionally called gaqWorker.resume(), which silently undid any pause set up by a calling test suite.

Capture the worker's paused state before pausing
and only resume it if it wasn't already paused on entry.

Adds an isPaused getter on GaqWorker to expose the state without reaching into the private field.
@isaachilly
[O2B-1564] Remove overkill parsePositiveInt helper and use Number lik…
c7d37b7
…e the rest
@isaachilly
[O2B-1564] Add missing docker env vars to CI dockerfile
4a97602
@isaachilly
Update ca-certificates version in Dockerfile (#2138)
5fee0a4
@isaachilly
Merge branch 'feature/O2B-1545/Create-GAQ-Summary-Tables' into featur…
0b2dab4
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
@isaachilly
Merge branch 'feature/O2B-1563/Create-GAQ-summary-invalidation-mechan…
6ee99b3
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
github-code-quality[bot]
github-code-quality Bot found potential problems Jun 29, 2026
View reviewed changes
Comment thread lib/public/Model.js Fixed

Uh oh!

There was an error while loading. Please reload this page.

Comment thread lib/public/Model.js Fixed

Uh oh!

There was an error while loading. Please reload this page.

Comment thread lib/public/Model.js Fixed

Uh oh!

There was an error while loading. Please reload this page.

Comment thread lib/public/components/common/form/inputs/DateTimeInputModel.js Fixed

Uh oh!

There was an error while loading. Please reload this page.

isaachilly added 2 commits June 29, 2026 18:36
@isaachilly
Reapply "Merge branch 'main' into feature/O2B-1563/Create-GAQ-summary…
467630a
…-invalidation-mechanism"

This reverts commit 23f6e04.
@isaachilly
Merge branch 'feature/O2B-1563/Create-GAQ-summary-invalidation-mechan…
79fc244
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
@graduta
graduta force-pushed the main branch from 544a0ee to 1758c68 Compare July 29, 2026 09:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@graduta graduta Awaiting requested review from graduta
1 more reviewer
@github-code-quality github-code-quality[bot] github-code-quality[bot] left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@isaachilly isaachilly

Labels

backend

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@isaachilly
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.