GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: Update import logic to remove `pyspark` dependency from Snowflake Offline Store by beauchbum · Pull Request #3397 · feast-dev/feast

Conversation

@beauchbum

@beauchbum beauchbum commented Dec 15, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

  • Moves the pyspark import inside of the relevant method so that the Snowflake offline store does not have a necessary dependency on pyspark
  • Uses typing.TYPE_CHECKING to do the imports while type checking, but not in runtime

Resulting behavior

  • The Snowflake Offline Store can be used without pyspark installed
    • However, when running an environment without pyspark, then calling to_spark_df raises a FeastExtrasDependencyImportError

Which issue(s) this PR fixes:
Fixes this issue

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Ryan Beauchamp added 2 commits December 15, 2022 08:54
Ryan Beauchamp
fix: move pyspark import inside relevant method
19d4141 
Signed-off-by: Ryan Beauchamp <[email protected]>
Ryan Beauchamp
fix: put import inside type checking so that type hints works
14e4bd0 
Signed-off-by: Ryan Beauchamp <[email protected]>
@beauchbum
beauchbum requested a review from sfc-gh-madkins as a code owner December 15, 2022 14:11
@feast-ci-bot feast-ci-bot added size/S size/M and removed size/S labels Dec 15, 2022
@beauchbum
beauchbum marked this pull request as draft December 15, 2022 15:12
@feast-ci-bot feast-ci-bot added the do-not-merge/work-in-progress label Dec 15, 2022
@beauchbum
beauchbum marked this pull request as ready for review December 15, 2022 15:21
@feast-ci-bot feast-ci-bot added size/S and removed do-not-merge/work-in-progress size/M labels Dec 15, 2022
@beauchbum
beauchbum force-pushed the pyspark_import branch from 6c098b4 to 31574f5 Compare December 15, 2022 17:19
@feast-ci-bot feast-ci-bot added size/M and removed size/S labels Dec 15, 2022
Ryan Beauchamp
fix: import future annotations
eae29ad 
Signed-off-by: Ryan Beauchamp <[email protected]>
@beauchbum
beauchbum force-pushed the pyspark_import branch from 31574f5 to eae29ad Compare December 15, 2022 17:30
@feast-ci-bot feast-ci-bot added size/S and removed size/M labels Dec 15, 2022
@beauchbum

beauchbum commented Dec 15, 2022

Copy link
Copy Markdown
Contributor Author

Went back and forth on this a lot to get things to work. Dealing with an optional import (pyspark) that needs to pass mypy type checking was difficult. I found a solution from this post.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
beauchbum
beauchbum commented Dec 15, 2022
View reviewed changes
Comment thread sdk/python/feast/infra/offline_stores/snowflake.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
achals
achals approved these changes Dec 15, 2022
View reviewed changes

@achals achals left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@feast-ci-bot feast-ci-bot assigned achals Dec 15, 2022
@feast-ci-bot feast-ci-bot added lgtm approved labels Dec 15, 2022
@achals achals added ok-to-test and removed lgtm approved labels Dec 15, 2022
Ryan Beauchamp
fix: annotate only the method signature
c04e38c 
Signed-off-by: Ryan Beauchamp <[email protected]>
@feast-ci-bot feast-ci-bot added the approved label Dec 16, 2022
sfc-gh-madkins
sfc-gh-madkins approved these changes Dec 18, 2022
View reviewed changes
@feast-ci-bot

feast-ci-bot commented Dec 18, 2022

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, beauchbum, sfc-gh-madkins

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [achals,sfc-gh-madkins]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@sfc-gh-madkins

sfc-gh-madkins commented Dec 18, 2022

Copy link
Copy Markdown
Collaborator

/lgtm
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@feast-ci-bot feast-ci-bot assigned sfc-gh-madkins Dec 18, 2022
@feast-ci-bot feast-ci-bot added the lgtm label Dec 18, 2022
@sfc-gh-madkins
sfc-gh-madkins enabled auto-merge (squash) December 18, 2022 04:10
@feast-ci-bot
feast-ci-bot merged commit cf073e6 into feast-dev:master Dec 18, 2022
felixwang9817 pushed a commit that referenced this pull request Jan 3, 2023
@feast-ci-bot
chore(release): release 0.28.0
81c3483 
# [0.28.0](v0.27.0...v0.28.0) (2023-01-03)

### Bug Fixes

* Apply billing project when infer schema ([#3417](#3417)) ([4f9ad7e](4f9ad7e))
* Assertion condition when value is 0 ([#3401](#3401)) ([98a24a3](98a24a3))
* Enable registry caching in SQL Registry ([#3395](#3395)) ([2e57376](2e57376))
* Fix bug where SQL registry was incorrectly writing infra config around online stores ([#3394](#3394)) ([6bcf77c](6bcf77c))
* Get all columns with describe table method from RedshiftData-api ([#3377](#3377)) ([fd97254](fd97254))
* ODFV able to handle boolean pandas type ([#3384](#3384)) ([8f242e6](8f242e6))
* Remove PySpark dependency from Snowflake Offline Store ([#3388](#3388)) ([7b160c7](7b160c7))
* Specifies timeout in exception polling ([#3398](#3398)) ([c0ca7e4](c0ca7e4))
* Update import logic to remove `pyspark` dependency from Snowflake Offline Store ([#3397](#3397)) ([cf073e6](cf073e6))

### Features

* Add template for Github Codespaces ([#3421](#3421)) ([41c0537](41c0537))
* Adds description attribute for features/fields ([#3425](#3425)) ([26f4881](26f4881))
* Snowflake skip materialization if no table change ([#3404](#3404)) ([0ab3942](0ab3942))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@achals achals achals approved these changes
@sfc-gh-madkins sfc-gh-madkins sfc-gh-madkins approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@achals achals

@sfc-gh-madkins sfc-gh-madkins

Labels

approved lgtm ok-to-test size/S

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

pyspark dependency breaks Snowflake offline store

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@beauchbum @feast-ci-bot @sfc-gh-madkins @achals
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.