GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: Update the pyarrow to latest v14.0.1 regarding the CVE-2023-47248. by shuchu · Pull Request #3835 · feast-dev/feast

Conversation

@shuchu

@shuchu shuchu commented Nov 14, 2023

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:
Update the pyarrow to latest version v14.0.1 which has the fix for CVE-2023-47248

Fixes #3832

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@shuchu
fix: Update the pyarrow to latest v14.0.1 regarding the CVE-2023-47248.
9d671f1 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu

shuchu commented Nov 14, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator Author

A little bit worried about the unit test coverage.

please be aware that I unpin the pyarrow version.

py3.8-requirements.txt and py3.8-ci-requirements.txt were updated manually. (regarding the DASK version issue for python 3.8)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@achals achals added lgtm approved ok-to-test labels Nov 14, 2023
achals
achals reviewed Nov 14, 2023
View reviewed changes
Comment thread sdk/python/requirements/py3.10-ci-requirements.txt Outdated
@@ -1,5 +1,5 @@
#
# This file is autogenerated by pip-compile with Python 3.10
# This file is autogenerated by pip-compile with Python 3.9

@achals achals Nov 14, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is incorrect

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@shuchu shuchu Nov 15, 2023

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right, I need to create a python 3.10 venv and run the command from Makefile. let me fix this.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@shuchu shuchu Nov 15, 2023

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, let's see the testing results.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@shuchu shuchu Nov 15, 2023

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems integration test failed...

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@shuchu shuchu Nov 15, 2023

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

21 fails, the most frequent error is about the wrong format of Timestamp: google.api_core.exceptions.BadRequest: 400 Error while reading data, error message: Invalid timestamp microseconds value 1700011424237000000 of logical type NONE; in column 'created'

let me dig into it and see what's the root cause

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@shuchu shuchu Nov 16, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1, The Google's BigQuery api only accepts "ms" resolution for timestamp, while the Pyarrow.parquet.write_table() will maintain the resolution to the exact original resolution which is "ns" by default.
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@achals achals mentioned this pull request Nov 14, 2023
fix: Update typeguard version to >=4.0.0 #3837
Merged
shuchu added 2 commits November 14, 2023 20:14
@shuchu
fix: Update the requirements file of python3.10.
9b4ec66 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu
fix: force the timestamp's datatype resolution to ms for bigquery whi…
0e0a5ad 
…le write to temporary parquet file.

Signed-off-by: Shuchu Han <[email protected]>
@shuchu
shuchu requested a review from sudohainguyen as a code owner November 16, 2023 05:21
@shuchu
fix: Update the timestamp resolution to us which is exact same as usi…
fe61d0d 
…ng pyarrow v10.0.1

Signed-off-by: Shuchu Han <[email protected]>
@shuchu

shuchu commented Nov 16, 2023

Copy link
Copy Markdown
Collaborator Author

I meet a very interesting problem. I only update the Pyarrow version and snowflake api, the integration test results show me that the timestamp range is error while running Redshift SQL query.

{ error:  Timestamp out of range.\n  
              code:      8001\n  }

it happens while run "get_historical_features()" and the timestamp range were inferenced from the "entity_df":
as in redshift.py::_get_entity_df_event_timestamp_range().
f"SELECT MIN({entity_df_event_timestamp_col}) AS min, MAX({entity_df_event_timestamp_col}) AS max "
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
shuchu added 4 commits November 16, 2023 21:09
@shuchu
fix: Add debug info to check the entity_df before send to redshift.
e357d79 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu
fix: add debug to check the loaded pyarrow table file.
20e55c4 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu
fix: debug the transformation from pyarrow table to pandas df.
fbb5b20 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu
fix: roll pyarrow back.
059914b 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu

shuchu commented Nov 17, 2023

Copy link
Copy Markdown
Collaborator Author

please do not merge this PR. @sudohainguyen
It's in a mess status and is for debugging only now.
sudohainguyen reacted with eyes emoji
All reactions
  • 👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@shuchu
fix: force test fail to check table and pandas df.
f032d7a 
Signed-off-by: Shuchu Han <[email protected]>
@sudohainguyen

sudohainguyen commented Nov 17, 2023

Copy link
Copy Markdown
Collaborator

No worries, looking forward to seeing this works
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
shuchu added 5 commits November 17, 2023 01:33
@shuchu
fix: debug the transformation from parquet file to pyarrow table.
a2e217f 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu
fix: debug the parquet file.
0e43126 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu
fix: test 14.0.1 parquet file.
19d30ef 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu
fix: test the parquet reader of pandas instead of pyarrow.
ffe6255 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu
fix: use datetime resolution us while writing test data into redshift.
dc2630e 
Signed-off-by: Shuchu Han <[email protected]>
@shuchu

shuchu commented Nov 17, 2023

Copy link
Copy Markdown
Collaborator Author

Finally, I found the fix way. It's about the setting of parameter "coerce_timestamps" of "pyarrow.parquet.write_table".

Let me close this PR and create a clean new one.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@sudohainguyen

sudohainguyen commented Nov 17, 2023

Copy link
Copy Markdown
Collaborator

Great @shuchu !!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@shuchu shuchu closed this Nov 17, 2023
@shuchu shuchu mentioned this pull request Nov 17, 2023
fix: Upgrade the pyarrow to latest v14.0.1 for CVE-2023-47248. #3841
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@achals achals achals left review comments
@sudohainguyen sudohainguyen Awaiting requested review from sudohainguyen sudohainguyen is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

approved lgtm ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Security vulnerability of python package: pyarrow (CVE-2023-47248)

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@shuchu @sudohainguyen @achals
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.