GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Add Snowflake materialization engine by sfc-gh-madkins · Pull Request #2948 · feast-dev/feast

feat: Add Snowflake materialization engine #2948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
achals merged 1 commit into feast-dev:master from sfc-gh-madkins:snowflake_materialize
Aug 18, 2022

Uh oh!

There was an error while loading. Please reload this page.

Merged

feat: Add Snowflake materialization engine #2948

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
5d0cf95
feat: Add Snowflake materialization engine
sfc-gh-madkins Aug 18, 2022
File filter
Filter by extension


Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.

Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.

Diff view
Diff view

There are no files selected for viewing

2 changes: 1 addition & 1 deletion 2 .github/workflows/pr_local_integration_tests.yml
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,4 +75,4 @@ jobs:
IS_TEST: "True"
FEAST_LOCAL_ONLINE_CONTAINER: "True"
FEAST_IS_LOCAL_TEST: "True"
run: pytest -n 8 --cov=./ --cov-report=xml --color=yes --integration -k "not gcs_registry and not s3_registry and not test_lambda_materialization" sdk/python/tests
run: pytest -n 8 --cov=./ --cov-report=xml --color=yes --integration -k "not gcs_registry and not s3_registry and not test_lambda_materialization and not test_snowflake_materialization" sdk/python/tests
2 changes: 2 additions & 0 deletions 2 docs/reference/batch-materialization/README.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@

Please see [Batch Materialization Engine](../../getting-started/architecture-and-components/batch-materialization-engine.md) for an explanation of batch materialization engines.

{% page-ref page="snowflake.md" %}

{% page-ref page="bytewax.md" %}
28 changes: 28 additions & 0 deletions 28 docs/reference/batch-materialization/snowflake.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Snowflake

## Description

The [Snowflake](https://trial.snowflake.com) batch materialization engine provides a highly scalable and parallel execution engine using a Snowflake Warehouse for batch materializations operations (`materialize` and `materialize-incremental`) when using a `SnowflakeSource`.

The engine requires no additional configuration other than for you to supply Snowflake's standard login and context details. The engine leverages custom (automatically deployed for you) Python UDFs to do the proper serialization of your offline store data to your online serving tables.

When using all three options together, `snowflake.offline`, `snowflake.engine`, and `snowflake.online`, you get the most unique experience of unlimited scale and performance + governance and data security.

## Example

{% code title="feature_store.yaml" %}
```yaml
...
offline_store:
type: snowflake.offline
...
batch_engine:
type: snowflake.engine
account: snowflake_deployment.us-east-1
user: user_login
password: user_password
role: sysadmin
warehouse: demo_wh
database: FEAST
```
{% endcode %}
2 changes: 1 addition & 1 deletion 2 docs/reference/offline-stores/snowflake.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Below is a matrix indicating which functionality is supported by `SnowflakeRetri
| export to dataframe | yes |
| export to arrow table | yes |
| export to arrow batches | no |
| export to SQL | no |
| export to SQL | yes |
| export to data lake (S3, GCS, etc.) | yes |
| export to data warehouse | yes |
| export as Spark dataframe | no |
Expand Down
12 changes: 10 additions & 2 deletions 12 docs/tutorials/tutorials-overview/driver-stats-on-snowflake.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: >-
# Drivers stats on Snowflake

In the steps below, we will set up a sample Feast project that leverages Snowflake
as an offline store.
as an offline store + materialization engine + online store.

Starting with data in a Snowflake table, we will register that table to the feature store and define features associated with the columns in that table. From there, we will generate historical training data based on those feature definitions and then materialize the latest feature values into the online store. Lastly, we will retrieve the materialized feature values.

Expand Down Expand Up @@ -46,7 +46,7 @@ The following files will automatically be created in your project folder:

#### Inspect `feature_store.yaml`

Here you will see the information that you entered. This template will use Snowflake as an offline store and SQLite as the online store. The main thing to remember is by default, Snowflake objects have ALL CAPS names unless lower case was specified.
Here you will see the information that you entered. This template will use Snowflake as the offline store, materialization engine, and the online store. The main thing to remember is by default, Snowflake objects have ALL CAPS names unless lower case was specified.

{% code title="feature_store.yaml" %}
```yaml
Expand All @@ -61,6 +61,14 @@ offline_store:
role: ROLE_NAME #case sensitive
warehouse: WAREHOUSE_NAME #case sensitive
database: DATABASE_NAME #case cap sensitive
batch_engine:
type: snowflake.engine
account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com
user: USERNAME
password: PASSWORD
role: ROLE_NAME #case sensitive
warehouse: WAREHOUSE_NAME #case sensitive
database: DATABASE_NAME #case cap sensitive
online_store:
type: snowflake.online
account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com
Expand Down
8 changes: 8 additions & 0 deletions 8 sdk/python/docs/source/feast.infra.materialization.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,14 @@ feast.infra.materialization.local\_engine module
:undoc-members:
:show-inheritance:

feast.infra.materialization.snowflake\_engine module
----------------------------------------------------

.. automodule:: feast.infra.materialization.snowflake_engine
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

Expand Down
8 changes: 0 additions & 8 deletions 8 sdk/python/docs/source/feast.infra.utils.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,6 @@ feast.infra.utils.hbase\_utils module
:undoc-members:
:show-inheritance:

feast.infra.utils.snowflake\_utils module
-----------------------------------------

.. automodule:: feast.infra.utils.snowflake_utils
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

Expand Down
Loading
Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.