GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix(postgresql): set query result column nullable if selected column contains json operator or null value in case expression by swallowstalker · Pull Request #3739 · sqlc-dev/sqlc

Conversation

@swallowstalker

@swallowstalker swallowstalker commented Dec 10, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Trying to solve #3710

JSON traversal operator on selected columns

Based on postgresql docs, operator ->> and #>> should return string, and since this is JSON and it could have null value on its field, it should return pgtype.Text. Other JSON operator should return interface{} since it returns JSON object

Refactor AExpr and AConst on selected columns

Since it is used on CaseExpr, I refactored both of them to allow it to be reusable in different location

TypeCast null value checking

If there is any selected columns which has null value to be casted to other type, for example null::text then it should return pgtype.Text to allow nullable text

CaseExpr nullable + column type handling

Previously, sqlc determines column type from default case and uses hardcoded NotNull to true. Relying on default case can be bad since it might not be declared, if user decides not to use ELSE keyword.

This PR will determine column type + nullability from all sub-case, on list of THEN clause combined with ELSE clause considering data types involved on those multiple clause.

  1. If there is only one data type on any case expr clause, then it will be nullable of that data type
  2. If there are multiple data type, it will automatically generate interface{} type
  3. If all case expr clause have null result, then it will generate interface{} type
  4. Case expr without default clause, will be assumed to have null as its ELSE result

and if any of those column are nullable, then the resulting column will be set to nullable.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

veqryn and TheDevMinerTV reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
@swallowstalker swallowstalker mentioned this pull request Dec 10, 2024
Postgres: casting result to text results in sqlc generating a string instead of a nullable string #3710
Open
@swallowstalker swallowstalker changed the title feat: adding nullable text if selected column is json operator and casted as text feat: adding nullable text if selected column has json operator and has cast type Dec 10, 2024
@swallowstalker swallowstalker changed the title feat: adding nullable text if selected column has json operator and has cast type fix(postgresql): adding nullable text if selected column has json operator and has cast type Dec 10, 2024
@swallowstalker swallowstalker changed the title fix(postgresql): adding nullable text if selected column has json operator and has cast type fix(postgresql): set query result column nullable if selected column contains json operator or null value in case expression Dec 12, 2024
@veqryn

veqryn commented Dec 12, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Can we add this test case as well please?

-- name: GetNullable3B :many
SELECT CASE WHEN true THEN 'hello'::text ELSE null::text END 
FROM "mytable";
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@veqryn

veqryn commented Dec 12, 2024

Copy link
Copy Markdown

Where in your tests are we setting the expectation of the generated results?
Like, I would have expected your GetNullable2 to return an any/interface{}, but I don't see the expectation set anywhere. Right now, the generated code appears to return a pgtype.Text, which is odd for something returning integers, floats, and strings.

Or is the committed code the expectation?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@swallowstalker

swallowstalker commented Dec 13, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Can we add this test case as well please?

GetNullable3B added

Like, I would have expected your GetNullable2 to return an any/interface{}

In my PR description I proposed type precedence, for multiple type inside case expression. For example

  • 1st condition result is int, 2nd condition result is float, 3rd condition result is text, and default condition is null, then it should be nullable text, not an interface
  • 1st condition result is int, 2nd condition result is float, and default condition is int, then it should be non-nullable float, not an interface
  • 1st condition result is int, and default condition is null, then it should be nullable int, not an interface

and so on. but on the other side, I think that the user should be given freedom if the return value of case expr has several type variance (in short, just return any or interface{}). @kyleconroy maybe you have feedback on this particular case? 👀

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@swallowstalker

swallowstalker commented Dec 13, 2024

Copy link
Copy Markdown
Contributor Author

Or is the committed code the expectation?

I have generated the result using code from this PR, and then make it as expectation (commited on repo), but to be honest I have a problem in github workflow tests, it seems that the binary that the workflow use is the previous version before I made these changes, so the generated go file would always be different than the one I commited on the repo. Especially on TestReplay endtoend tests. Maybe somebody can help giving me some pointers?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

swallowstalker added 8 commits December 13, 2024 07:41
@swallowstalker
feat: adding nullable text if selected column is json operator and ca…
9ea4ce3
…sted as text
@swallowstalker
test: fix fmt
25274e8
@swallowstalker
test: delete fourth column
f61bfcc
@swallowstalker
test: avoid test format because the reversed query from raw statement…
049fb49
…s printed selected columns without parentheses, it will generate different type on from query.sql
@swallowstalker
feat: case expr handling and json operator handling
ebc8a77
@swallowstalker
fix: change managed-db to base
f154a2f
@swallowstalker
test: add null::text as default case expr
822e73f
@swallowstalker
test: changing author email
e7ddabf
@swallowstalker
swallowstalker force-pushed the text-alias branch from dcf4684 to e7ddabf Compare December 13, 2024 00:43
@veqryn

veqryn commented Dec 13, 2024

Copy link
Copy Markdown

So after thinking about it, I realized we really only care about what Postgres thinks the returned values are, not what we think they should be.
I tried various combinations of your GetNullable2, and found that postgres doesn't really care about the order of which case/when statement comes first, but instead has some kind of preference, which you mentioned.

For example, if you insert at least 4 rows into mytable, then running this query:

SELECT CASE
    WHEN id = 1 THEN '1'
    WHEN id = 2 THEN null
    WHEN id = 3 THEN id
    WHEN id = 4 THEN 4.5
    ELSE '10'
END
FROM "mytable";

Returns:

1
[NULL]
3
4.5
10
10

The strings for '1' and '10' got cast into numbers. id is still an integer/number. The decimal or float of 4.5 got to stay as is. I suppose the correct type for this would be a nullable decimal?

But if you change the query to this:

SELECT CASE
    WHEN id = 1 THEN '1'::text
    WHEN id = 2 THEN null
    WHEN id = 3 THEN id
    WHEN id = 4 THEN 4.5
    ELSE '10'
END
FROM "mytable";

You get an error: SQL Error [42804]: ERROR: CASE types text and bigint cannot be matched

or this:

SELECT CASE
    WHEN id = 1 THEN 'foobar'
    WHEN id = 2 THEN null
    WHEN id = 3 THEN id
    WHEN id = 4 THEN 4.5
    ELSE '10'
END
FROM "mytable";

You get a similar error: SQL Error [22P02]: ERROR: invalid input syntax for type numeric: "foobar"

So to sum up: I think you are on the right track, with forcing these types of queries into a type rather than an any/interface{}

I don't know how realistic these are in the real world. Perhaps we should try to find queries that hit an actual table of data, rather than case statements of literals.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@swallowstalker

swallowstalker commented Dec 19, 2024

Copy link
Copy Markdown
Contributor Author

I agree that it's not quite realistic, I better left it to sqlc user to decide which data type that will be used in case of multiple data type exist on case-when-then-else. In short, I'll make it interface then

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

swallowstalker added 2 commits December 19, 2024 14:02
@swallowstalker
Merge branch 'main' into text-alias
d0cbfe4
# Conflicts:
#	go.sum
@swallowstalker
fix: change type precedence auto-selection to data type any, to allow…
409d2fb
… user more freedom to choose query column type
@dosubot dosubot Bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Dec 19, 2024
swallowstalker
swallowstalker commented Dec 19, 2024
View reviewed changes
Comment thread internal/compiler/output_columns.go
col = &Column{Name: name, DataType: "float", NotNull: true}
case *ast.Boolean:
col = &Column{Name: name, DataType: "bool", NotNull: true}
case *ast.Null:

@swallowstalker swallowstalker Dec 19, 2024

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is new addition to handle null const value. I need to distinguish between any and null, because null in CaseExpr would determine query column result nullability, but does not make it included in column type decision.

for ast.A_Const condition, i have converted DataType null back to any (line 136, outputColumns function) because we don't know its type and to preserve backward compatibility

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
swallowstalker
swallowstalker commented Dec 19, 2024
View reviewed changes
Comment thread internal/compiler/output_columns.go
}
} else {
cols = append(cols, &Column{Name: name, DataType: "any", NotNull: false})
defaultCol = &Column{Name: name, DataType: "null", NotNull: false}

@swallowstalker swallowstalker Dec 19, 2024

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing ELSE clause on CaseExpr is considered to return null value

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@swallowstalker
test: uncommitted test case for GetNullable2G
19ab642
@veqryn

veqryn commented Dec 22, 2024

Copy link
Copy Markdown

Based on the tests, I believe this fixes the issue for me.
How do we get this reviewed by sqlc devs?

swallowstalker reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@swallowstalker

swallowstalker commented Dec 22, 2024

Copy link
Copy Markdown
Contributor Author

from my previous PR, I mentioned @kyleconroy to ask for review. would you mind checking it @kyleconroy? sorry i still can't assign reviewer for this PR

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@swallowstalker

swallowstalker commented Jan 27, 2025

Copy link
Copy Markdown
Contributor Author

bumping this PR, is there anything more i can do for this PR?

veqryn, DrowningToast, and sgielen reacted with thumbs up emoji
All reactions
  • 👍 3 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ken-talltree-io

ken-talltree-io commented Oct 24, 2026

Copy link
Copy Markdown

Tried this on current main; it applies apart from a one-line whitespace hunk. CASE nullability is right now, but a WHEN result that's a column reference types as any:

create table books (id int primary key, title text not null);
create table shelves (id int primary key, label text not null, book_id int);

-- name: Shelf :many
select case when b.id > 0 then s.label else null::text end shelf
from books b left join shelves s on s.book_id = b.id;

With this PR, Shelf returns []any; on main it returns []string (wrongly non-null), and it should be []sql.NullString. convertCaseExprCondToColumn falls back to any for an *ast.ColumnRef; resolving it the way outputColumns resolves a plain column reference would keep the type.

Scalar subqueries (#4262) aren't covered here; I'll propose that fix separately.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ken-talltree-io ken-talltree-io mentioned this pull request Oct 24, 2026
compiler: a scalar subquery's column is nullable #4654
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@swallowstalker @veqryn @ken-talltree-io
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.