GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: harden warehouse pushdown and close local/warehouse parity gaps … · Veridelta/veridelta@14e8108

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
  • .cursor/rules
    • 300-security.mdc
  • docs
    • configuration.md
  • pyproject.toml
  • src/veridelta
    • cli.py
    • connectors
      • lakehouse.py
      • sql.py
      • warehouse.py
    • engine.py
    • models.py
    • report.py
  • tests
    • integration
      • test_engine_routing.py
      • test_pushdown_parity.py
    • unit
      • test_cli.py
      • test_connectors.py
      • test_engine.py
      • test_models.py
      • test_report.py
      • test_sql_compiler.py
      • test_warehouse_execution.py
  • uv.lock
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎.cursor/rules/300-security.mdc‎

Copy file name to clipboardExpand all lines: .cursor/rules/300-security.mdc
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Apply these rules when modifying files inside `src/veridelta/`.
1414
- `execute_pushdown` runs compiler-produced `statement` only. Do not concatenate config fields into a second SQL string.
1515
- Identifiers: allowlist each segment (`SQL_IDENTIFIER_SEGMENT` / `SQL_RELATION_PATTERN` in `models.py`) then dialect-quote (Snowflake `"`, Databricks backticks). Fail closed with `ConnectorError` on disallowed characters.
1616
- Do not apply SQL identifier regex to file-source `column_names`, `primary_keys`, credentials, hostnames, tokens, or lakehouse `table_uri`.
17-
- Data literals (`value_map`, `regex_replace`) go through `_literal` (`'` → `''`). Do not treat them as identifiers.
17+
- Data literals (`value_map`, `regex_replace`, datetime formats) go through `_literal`, which applies the dialect's `_LITERAL_ESCAPES`. Snowflake and Databricks read backslash escapes inside quotes, so both double `\` before anything else; Databricks concatenates adjacent literals, so it escapes `'` as `\'` rather than `''`. Do not escape by hand, and do not treat literals as identifiers. The DuckDB harness cannot see these rules; `test_sql_compiler.py` checks them against a reference lexer.
1818
- `null_values` accepts mixed scalars, so it renders through `_sentinel_literal`: strings still go through `_literal`, while numbers and booleans emit unquoted so they cannot break a numeric column's cast. Only `str` can reach `_literal`, so the escaping guarantee is unchanged. Sentinels are also filtered against the probed dtype (`sentinels.py`) before rendering, which is a correctness control, not a security one.
1919
- Numeric SQL operands (`absolute_tolerance`, `relative_tolerance`, `version`, `snapshot_id`, `pad_zeros`) stay strict `float`/`int` fields. Do not coerce strings into SQL or Polars kwargs.
2020
- `cast_to` is a closed `CastTarget` literal rendered through `_CAST_KEYWORDS`. A type name cannot be quoted or bound inside `CAST(x AS ...)`, so the table is the whole control. Never interpolate the config value directly, and never widen the literal without adding a keyword for every dialect.
Collapse file

‎docs/configuration.md‎

Copy file name to clipboardExpand all lines: docs/configuration.md
+35-18Lines changed: 35 additions & 18 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ target:
1919
primary_keys: ["user_id"]
2020
```
2121
22+
`primary_keys` must name at least one column, and together the keys must be unique on each side.
23+
2224
File sources may omit `type` (it defaults to `file`) and continue to use `path`, `format`, and optional `options`.
2325

2426
### File formats
@@ -49,25 +51,29 @@ uv add 'veridelta[iceberg]'
4951
uv add 'veridelta[all]'
5052
```
5153

52-
Do not commit `password` or `access_token` in YAML. Inject secrets from the environment or your orchestrator's secret store at runtime.
54+
Do not commit `password` or `access_token` in YAML. The loader does not expand environment variables, so either render the file from your secret store at runtime, or build the connection in Python (for example `SnowflakeConfig(..., password=os.environ["SNOWFLAKE_PASSWORD"])`) and pass it to `DiffEngine.run_from_configs`.
5355

54-
Same-warehouse SQL pushdown runs only when both sides are Snowflake or both sides are Databricks, the connection fields match (`account`, `user`, `warehouse`, `database`, `schema_name`, `role`, and `password` for Snowflake; `server_hostname`, `http_path`, `access_token`, `catalog`, and `schema_name` for Databricks), and the `table` names differ. Mixed file/lakehouse and warehouse backends, or Snowflake paired with Databricks, raise `ConnectorError`.
56+
Same-warehouse SQL pushdown runs only when both sides are Snowflake or both sides are Databricks, the connection fields match (`account`, `user`, `warehouse`, `database`, `schema_name`, `role`, and `password` for Snowflake; `server_hostname`, `http_path`, `access_token`, `catalog`, and `schema_name` for Databricks), and the `table` names differ; naming the same table twice raises `ConfigError`, since a table compared with itself always matches. Mixed file/lakehouse and warehouse backends, or Snowflake paired with Databricks, raise `ConnectorError`.
5557

56-
`table` must be one to three unquoted identifier segments (`EVENTS`, `schema.table`, or `catalog.schema.table`). Pattern-only `DiffRule` entries are not compiled to SQL; they raise `ConnectorError` on the warehouse path.
58+
`table` must be one to three unquoted identifier segments (`EVENTS`, `schema.table`, or `catalog.schema.table`). Rules that select columns by `pattern` are matched against the probed column names before any SQL is compiled, so they apply in the warehouse exactly as they do locally.
5759

5860
Pushdown issues eight statements per run: a zero-row column probe and a `COUNT(*)` per side, then inner-join mismatches, target-only added rows, source-only removed rows, and a per-column mismatch tally. Those fill every `DiffSummary` field including `column_mismatches`, so `threshold`, `match_rate_percentage`, and the drift report mean the same thing they do for local comparisons.
5961

60-
Every column present on both sides is compared, exactly as it is locally. Columns without an explicit rule inherit the global `default_*` settings, so a `default_absolute_tolerance` applies in the warehouse too. Columns marked `ignore` are excluded, and `rename_to` pairs a source column with its renamed target counterpart.
62+
Every column present on both sides is compared, exactly as it is locally. Columns without an explicit rule inherit the global `default_*` settings, so a `default_absolute_tolerance` applies in the warehouse too. As in a local run, a tolerance only loosens a column that is numeric once normalized, such as a text column with `cast_to: Float64`; text, boolean, and temporal columns are compared exactly. Columns marked `ignore` are excluded, and `rename_to` pairs a source column with its renamed target counterpart.
6163

62-
The column probes enforce `schema_mode` and primary-key existence before any comparison runs, raising `ConfigError` on drift. Probed names are compared exactly as the compiler quotes them, with no case folding, so YAML identifiers must match the stored column case (Snowflake stores unquoted names uppercase).
64+
The column probes enforce `schema_mode` and primary-key existence before any comparison runs, raising `ConfigError` on drift. Probed names are compared exactly as the compiler quotes them, with no case folding, so YAML identifiers must match the stored column case (Snowflake stores unquoted names uppercase). `normalize_column_names` cannot change that: pushdown raises `ConfigError` if it would rename a stored column.
6365

64-
All nine transform stages compile, so a rule means the same thing in a warehouse as it does locally. Two behaviors still differ from the file and lakehouse path:
66+
All nine transform stages compile for compared columns, so a rule means the same thing in a warehouse as it does locally. These behaviors still differ from the file and lakehouse path:
6567

6668
- Artifacts contain primary keys only, since the comparison SQL never projects full rows. They are written as `added_rows_pks_only`, `removed_rows_pks_only`, and `changed_rows_pks_only` so they cannot be confused with local artifacts, which hold complete records.
67-
- Primary-key uniqueness is not verified, so duplicate keys inflate the inner-join mismatch count instead of raising `DataIntegrityError`.
69+
- Primary keys are joined as stored. Stages 1 through 7 normalize compared columns but not key columns, so keys that differ only by a transform (a `case_insensitive` or `pad_zeros` rule, or a global `default_whitespace_mode`) are reported as added and removed rather than matched.
70+
- Primary-key uniqueness is not verified. Duplicate keys inflate the mismatch count, or pass unnoticed when the duplicated rows are identical, instead of raising `DataIntegrityError`.
71+
- `strict_types` applies to local runs only. When the two relations store a column as different types, the warehouse compares them under its own coercion rules.
6872

6973
Parity is verified by a differential test harness that runs both engines over the same frames and compares the results. The harness executes compiled SQL through DuckDB, which catches semantic errors -- null propagation, three-valued logic, operator precedence -- but cannot catch vendor-specific divergence. Snowflake and Databricks spellings are pinned by direct assertions on the emitted SQL instead.
7074

75+
Write `regex_replace` patterns, `value_map` entries, and text `null_values` exactly as you would for a local run. Each is escaped for the target warehouse's string-literal rules, so a backslash in `\d` or `\N` and an apostrophe in `O'Brien` arrive intact; do not double them yourself. Escaping preserves the text, but each warehouse still runs its own regex engine: keep replacements free of capture-group references, which Polars and Databricks write as `$1` and Snowflake as `\1`. Likewise `whitespace_mode` trims only spaces in a warehouse, where Polars also strips tabs and line breaks.
76+
7177
Two stages need explaining:
7278

7379
- `pad_zeros` compiles to a sign-aware, non-truncating expression rather than a bare `LPAD`, which pads in front of a minus sign (`0-12` where Python gives `-012`) and discards characters past the target width.
@@ -123,9 +129,9 @@ veridelta run -c veridelta.yaml --quiet
123129
veridelta run -c veridelta.yaml --html report.html --html-max-rows 1000
124130
```
125131

126-
`--json` prints `DiffSummary` as JSON on stdout. `--quiet` suppresses progress chatter on stderr (the JSON line still prints). Progress chatter always goes to stderr, so `veridelta run --json | jq` does not have to strip anything first. `--html` writes a standalone report with no CDN references, capped at `--html-max-rows` (default 1000) so a large diff cannot produce an unopenable file. Pushdown reports are labeled as primary-keys-only.
132+
`--json` prints `DiffSummary` as JSON on stdout. `--quiet` suppresses progress chatter on stderr (the JSON line still prints). Progress chatter always goes to stderr, so `veridelta run --json | jq` does not have to strip anything first. `--html` writes a standalone report with no CDN references, capped at `--html-max-rows` (zero or more; default 1000) so a large diff cannot produce an unopenable file. Pushdown reports are labeled as primary-keys-only.
127133

128-
Exit codes stay at `0` for a match within `threshold` and `1` for drift or any failure.
134+
Exit codes are `0` for a match within `threshold`, `1` for drift or any failure while running, and `2` for invalid command-line arguments.
129135

130136
```yaml
131137
source:
@@ -224,10 +230,10 @@ Global directives control the strictness of the underlying Polars evaluation eng
224230
| :--- | :--- |
225231
| `schema_mode` | Enforces column structure constraints. Options: `intersection` (default, compares common columns only), `exact`, `allow_additions`, `allow_removals`. |
226232
| `strict_types` | If `false` (default), the engine dynamically soft-casts target columns to source types to prevent execution halts on mismatched types. If `true`, type mismatches automatically fail the row. |
227-
| `normalize_column_names`| If `true`, strips whitespace and lowercases all column headers prior to schema alignment. |
233+
| `normalize_column_names`| If `true`, strips whitespace and lowercases all column headers prior to schema alignment, on every entry point including `DiffEngine(...).run()` and `validate_schemas`. Configured `primary_keys`, `column_names`, and `rename_to` are normalized the same way; a `pattern` is not, so write it against the lowercase names. Headers that collide once normalized raise `ConfigError`. |
228234
| `threshold` | The allowable mismatch ratio (0.0 to 1.0) before the pipeline exits with a failure code. |
229-
| `default_absolute_tolerance` | Global absolute numeric tolerance. A column without its own `absolute_tolerance` inherits this. |
230-
| `default_relative_tolerance` | Global relative numeric tolerance. A column without its own `relative_tolerance` inherits this. |
235+
| `default_absolute_tolerance` | Global absolute numeric tolerance. A column without its own `absolute_tolerance` inherits this. Columns that are not numeric after normalization are compared exactly. |
236+
| `default_relative_tolerance` | Global relative numeric tolerance. A column without its own `relative_tolerance` inherits this. Columns that are not numeric after normalization are compared exactly. |
231237
| `default_treat_null_as_equal` | Global `NULL == NULL` policy. Defaults to `true`. A column rule can override it. |
232238
| `default_whitespace_mode` | Global whitespace stripping: `none` (default), `left`, `right`, or `both`. |
233239
| `default_null_values` | Global sentinel list. Applied only to columns whose type can hold each value. |
@@ -266,14 +272,14 @@ It raises `ConfigError` on a violation and returns nothing otherwise.
266272

267273
## Column-Level Overrides (Rules)
268274

269-
The `rules` array defines granular, per-column or regex-pattern tolerances. A rule selects columns by exact `column_names` or by a regular expression in `pattern`, and every other field is optional. When a column is named by more than one rule, the first rule listing it by exact name wins, then the first whose `pattern` matches.
275+
The `rules` array defines granular, per-column or regex-pattern tolerances. A rule selects columns by exact `column_names` or by a regular expression in `pattern`, and every other field is optional. When a column is named by more than one rule, the first rule listing it by exact name wins, then the first whose `pattern` matches. One rule governs each column, `ignore` included, so an exact-name rule keeps a column that a broader ignore `pattern` would otherwise drop. A renamed column answers to both spellings: a rule listing its target name wins, then the rule listing its source name. Local runs and warehouse pushdown resolve rules the same way.
270276

271277
| Field | Description |
272278
| :--- | :--- |
273279
| `column_names` | Exact source column names this rule governs. |
274-
| `pattern` | Regular expression matched against the start of each column name. Pattern-only rules are not compiled to warehouse SQL and raise `ConnectorError` there. |
275-
| `absolute_tolerance` | Maximum absolute numeric difference. Overrides `default_absolute_tolerance`. |
276-
| `relative_tolerance` | Maximum relative numeric difference (`0.01` is 1%). Overrides `default_relative_tolerance`. |
280+
| `pattern` | Regular expression matched against the start of each column name. |
281+
| `absolute_tolerance` | Maximum absolute numeric difference. Overrides `default_absolute_tolerance`. Must be finite; use `ignore` to stop comparing a column. |
282+
| `relative_tolerance` | Maximum relative numeric difference (`0.01` is 1%). Overrides `default_relative_tolerance`. Must be finite. |
277283
| `case_insensitive` | Lowercase text before comparing. |
278284
| `whitespace_mode` | `none`, `left`, `right`, or `both`. Overrides `default_whitespace_mode`. |
279285
| `regex_replace` | Mapping of regex pattern to replacement, applied in order to text columns. |
@@ -284,7 +290,7 @@ The `rules` array defines granular, per-column or regex-pattern tolerances. A ru
284290
| `datetime_format` | `strptime` pattern that parses text into timestamps. |
285291
| `timezone` | Zone that timezone-aware timestamps are converted to. |
286292
| `cast_to` | `Int64`, `Float64`, `String`, `Boolean`, `Date`, or `Datetime`. |
287-
| `ignore` | Exclude the matched columns from the comparison entirely. |
293+
| `ignore` | Exclude the columns this rule governs from the comparison entirely. |
288294
| `rename_to` | Target name for a single source column. |
289295

290296
### Transform order
@@ -301,7 +307,7 @@ Rules are not applied in the order you write them. Every column follows one fixe
301307
8. Comparison (equality, or numeric tolerance)
302308
9. Null-safe equality (`treat_null_as_equal`)
303309

304-
Stages 1 through 7 normalize each dataset on its own, before any join. That means they apply to primary keys as well: a `case_insensitive` rule on a key column changes how rows are matched, not just how they are compared. If normalizing a key collapses two rows into one, `DataIntegrityError` is raised rather than allowing a join explosion.
310+
Stages 1 through 7 normalize each dataset on its own, before any join. In local runs that means they apply to primary keys as well: a `case_insensitive` rule on a key column changes how rows are matched, not just how they are compared. If normalizing a key collapses two rows into one, `DataIntegrityError` is raised rather than allowing a join explosion. Warehouse pushdown joins keys as stored; see [Warehouse and lakehouse sources](#warehouse-and-lakehouse-sources).
305311

306312
Stages 2, 3, and 4 operate on text and are skipped for non-string columns, so a global `default_whitespace_mode` is safe to set on a mixed schema. Stage 1 is filtered per column instead, as described below.
307313

@@ -401,3 +407,14 @@ rules:
401407
- column_names: ["legacy_customer_id"]
402408
rename_to: "customer_id"
403409
```
410+
411+
The rule's other settings apply to the renamed pair on both sides, so a rename can carry a tolerance or a transform:
412+
413+
```yaml
414+
rules:
415+
- column_names: ["legacy_amt"]
416+
rename_to: "amount"
417+
absolute_tolerance: 0.01
418+
```
419+
420+
`primary_keys` are written with the target spelling, so a renamed key works in local runs. Warehouse pushdown joins on stored column names and raises `ConfigError` for a key that exists on the source only under its old name.
Collapse file

‎pyproject.toml‎

Copy file name to clipboardExpand all lines: pyproject.toml
+3-1Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ Issues = "https://github.com/Veridelta/veridelta/issues"
3535

3636
[project.optional-dependencies]
3737
snowflake = [
38-
"snowflake-connector-python>=3.0.0",
38+
# 3.7.0 added fetch_arrow_all(force_return_table=True). Before it, a result
39+
# with no rows arrives as None, and every pushdown run starts with one.
40+
"snowflake-connector-python>=3.7.0",
3941
"pyarrow>=10.0.1; python_version < '3.14'",
4042
# The driver warns on any pyarrow outside this range once running on 3.14.
4143
"pyarrow>=14.0.1,<24; python_version >= '3.14'",
Collapse file

‎src/veridelta/cli.py‎

Copy file name to clipboardExpand all lines: src/veridelta/cli.py
+22-1Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,27 @@ def _progress(message: str, *, quiet: bool) -> None:
4343
print(message, file=sys.stderr)
4444

4545

46+
def _row_limit(text: str) -> int:
47+
"""Parse `--html-max-rows`, which must be a whole number of zero or more.
48+
49+
Args:
50+
text (str): Raw argument value.
51+
52+
Returns:
53+
int: The row limit.
54+
55+
Raises:
56+
argparse.ArgumentTypeError: If the value is not a non-negative integer.
57+
"""
58+
try:
59+
value = int(text)
60+
except ValueError:
61+
raise argparse.ArgumentTypeError(f"expected a whole number, got {text!r}") from None
62+
if value < 0:
63+
raise argparse.ArgumentTypeError(f"must be zero or more, got {value}")
64+
return value
65+
66+
4667
def run(args: argparse.Namespace) -> int:
4768
"""Executes the comparison workflow based on CLI arguments.
4869
@@ -156,7 +177,7 @@ def build_parser() -> argparse.ArgumentParser:
156177
)
157178
run_parser.add_argument(
158179
"--html-max-rows",
159-
type=int,
180+
type=_row_limit,
160181
default=DEFAULT_MAX_ROWS,
161182
metavar="N",
162183
help=f"Rows to embed per HTML table before truncating (default: {DEFAULT_MAX_ROWS}).",
Collapse file

‎src/veridelta/connectors/lakehouse.py‎

Copy file name to clipboardExpand all lines: src/veridelta/connectors/lakehouse.py
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,8 @@
2626

2727
_UNCONNECTED = "Lakehouse connector is not connected. Call connect() first."
2828
_PUSHDOWN_UNSUPPORTED = "SQL pushdown is warehouse-only; lakehouse connectors use lazy scans."
29-
_DELTA_EXTRA = "Delta Lake extra is not installed. Install it with: uv sync --extra delta"
30-
_ICEBERG_EXTRA = "Iceberg extra is not installed. Install it with: uv sync --extra iceberg"
29+
_DELTA_EXTRA = "Delta Lake extra is not installed. Install it with: uv add 'veridelta[delta]'"
30+
_ICEBERG_EXTRA = "Iceberg extra is not installed. Install it with: uv add 'veridelta[iceberg]'"
3131

3232

3333
class DeltaLakeConnector(VerideltaConnector):

0 commit comments

Comments
0 (0)
You are visitor 01402605 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.