GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Fix DECIMAL parameter cast size for exponent-notation and negative values by winklemad · Pull Request #952 · databricks/databricks-sql-python

Conversation

@winklemad

@winklemad winklemad commented Sep 20, 2026

Copy link
Copy Markdown

Description

DecimalParameter.calculate_decimal_cast_string (parameters/native.py) infers the DECIMAL(precision, scale) to cast a bound Decimal to by string-splitting str(value) on ".". That miscounts two forms str(Decimal) legitimately produces, so the generated type can't hold the bound value.

Exponent notation — str() emits E for many magnitudes (e.g. Decimal("1500").normalize() is Decimal("1.5E+3"); .scaleb(), arithmetic, and scientific input do the same):

cursor.execute("SELECT ?", [Decimal("1500").normalize()])
# infers DECIMAL(5,4)  -- the "E+3" is counted as 4 fractional digits
# but 1500 needs DECIMAL(4,0); DECIMAL(5,4) max is ±9.9999 -> overflow
#   -> error in ANSI mode, or silently NULL in legacy mode

Negative values — the leading - is counted as an integer digit, over-widening every negative by one precision digit, which becomes a hard failure at the boundary:

Decimal("-" + "9" * 38)   # a valid DECIMAL(38,0)
# infers DECIMAL(39,0), which exceeds Databricks' maximum DECIMAL precision of 38
#   -> invalid cast, query fails

Both the cast type and the bound literal come from the same value, so the server can't reconcile them — a perfectly valid Decimal is rejected or silently nulled.

Fix

Derive precision/scale from Decimal.as_tuple() (sign, digits, exponent) — the exact numeric value — instead of the display string, so sign and exponent formatting no longer inflate the counts. Handles exponent notation, negatives, and sub-1 values uniformly.

Tests

The six existing test_calculate_decimal_cast_string cases are unchanged and still pass. Added regression cases for exponent notation (Decimal("1500").normalize() → DECIMAL(4,0), Decimal("1e5") → DECIMAL(6,0)) and negatives (-9…9 (38 digits) → DECIMAL(38,0), Decimal("-12.34") → DECIMAL(4,2)). tests/unit/test_parameters.py passes (60).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@winklemad
Fix DECIMAL parameter cast size for exponent-notation and negative va…
0472626
…lues

DecimalParameter.calculate_decimal_cast_string inferred DECIMAL(precision,
scale) by string-splitting str(value) on ".", which miscounts two forms
that str(Decimal) legitimately produces:

- exponent notation (Decimal("1500").normalize() -> "1.5E+3"): the "E+3"
  was counted as fractional digits, giving DECIMAL(5,4) for 1500, which
  overflows (an error in ANSI mode, NULL in legacy mode).
- a leading minus sign: counted as an extra integer digit, so a negative
  38-digit value produced DECIMAL(39,0), exceeding Databricks' max DECIMAL
  precision of 38 and failing the cast.

Derive precision/scale from Decimal.as_tuple() (digits + exponent), ignoring
sign and display format. Existing cast-string tests are unchanged; added
regression cases for exponent notation and negative values.

Signed-off-by: Madan Kumar <[email protected]>
@aminghadersohi

aminghadersohi commented Sep 26, 2026

Copy link
Copy Markdown

Confirming this is still needed on 4.6.0. Against a SQL warehouse, a bound Decimal("1E-18") (cast as DECIMAL(5,0)) and Decimal("1E-7") are stored as 0 without an error, and a 38-digit negative such as -99999999999999999999.999999999999999999 is rejected as DECIMAL(39,18). Sizing the cast from Decimal.as_tuple() as this PR does makes all of those round-trip exactly, including DECIMAL(38,18) at both extremes, in Core executemany and in SELECT :p. It would be great to get this reviewed.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@winklemad

winklemad commented Sep 26, 2026

Copy link
Copy Markdown
Author

Thanks for checking this against a real warehouse, that is really helpful. Good to know the executemany and SELECT :p paths round-trip too, since those were the two I was least sure about outside the unit tests.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jay-xiao446

jay-xiao446 commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

The enhancement to incorperate Exponent and Negative values generally looks good.
One edge case to consider: For Decimal("0E+38"), as_tuple() returns one digit and exponent 38, producing DECIMAL(39,0). It should be DECIMAL(1,0)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@winklemad
Size a zero DECIMAL with a positive exponent as one digit
60d58f9
Decimal("0E+38") has a single digit and exponent 38, which the integer
branch turned into DECIMAL(39,0). A zero only needs DECIMAL(1,0).
@winklemad

winklemad commented Sep 29, 2026

Copy link
Copy Markdown
Author

@jay-xiao446 good catch, thanks. A zero like 0E+38 has one digit but a large exponent, so the integer branch counted 39. It's now sized as a single digit when the value is zero, with tests for 0E+38, -0E+5 and 0E-5 (the last one still gives DECIMAL(5,5)). Pushed in 60d58f9.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jay-xiao446
jay-xiao446 approved these changes Sep 30, 2026
View reviewed changes

@jay-xiao446 jay-xiao446 left a comment

Copy link
Copy Markdown
Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@leninworld leninworld mentioned this pull request Oct 18, 2026
fix(parameters): honor explicit decimal precision and scale #988
Open
3 of 4 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jay-xiao446 jay-xiao446 jay-xiao446 approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@winklemad @aminghadersohi @jay-xiao446
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.