GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Add Cassandra/AstraDB online store contribution by hemidactylus · Pull Request #2873 · feast-dev/feast

feat: Add Cassandra/AstraDB online store contribution #2873

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 3 commits into feast-dev:master from hemidactylus:cassandra
Aug 9, 2022

Uh oh!

There was an error while loading. Please reload this page.
Merged

feat: Add Cassandra/AstraDB online store contribution #2873

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
02ac46e
Cassandra online store
kevjumba Jun 28, 2022
4c18d45
remove from main CI path and update entity key serialization in template
adchia Aug 9, 2022
6cd1e90
revert makefile change
adchia Aug 9, 2022
File filter
Filter by extension
Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Diff view
Diff view

There are no files selected for viewing

1 change: 1 addition & 0 deletions 1 CONTRIBUTING.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,6 +312,7 @@ The services with containerized replacements currently implemented are:
- Trino
- HBase
- Postgres
- Cassandra

You can run `make test-python-integration-container` to run tests against the containerized versions of dependencies.

Expand Down
25 changes: 25 additions & 0 deletions 25 Makefile
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,31 @@ test-python-universal-postgres:
not test_universal_types" \
sdk/python/tests

test-python-universal-cassandra:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \
FEAST_USAGE=False \
IS_TEST=True \
python -m pytest -x --integration \
sdk/python/tests

test-python-universal-cassandra-no-cloud-providers:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \
FEAST_USAGE=False \
IS_TEST=True \
python -m pytest -x --integration \
-k "not test_lambda_materialization_consistency and \
not test_apply_entity_integration and \
not test_apply_feature_view_integration and \
not test_apply_entity_integration and \
not test_apply_feature_view_integration and \
not test_apply_data_source_integration and \
not test_nullable_online_store and \
not gcs_registry and \
not s3_registry" \
sdk/python/tests

test-python-universal:
FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 --integration sdk/python/tests

Expand Down
2 changes: 1 addition & 1 deletion 2 README.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
* [x] [Cassandra / AstraDB](https://github.com/datastaxdevs/feast-cassandra-online-store)
* [x] [Cassandra / AstraDB](https://docs.feast.dev/reference/online-stores/cassandra)
* [ ] Bigtable (in progress)
* **Feature Engineering**
* [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#))
Expand Down
4 changes: 4 additions & 0 deletions 4 docs/reference/online-stores/README.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,7 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
{% content-ref url="postgres.md" %}
[postgres.md](postgres.md)
{% endcontent-ref %}

{% content-ref url="cassandra.md" %}
[cassandra.md](cassandra.md)
{% endcontent-ref %}
61 changes: 61 additions & 0 deletions 61 docs/reference/online-stores/cassandra.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Cassandra / Astra DB online store

## Description

The [Cassandra / Astra DB] online store provides support for materializing feature values into an Apache Cassandra / Astra DB database for online features.

* The whole project is contained within a Cassandra keyspace
* Each feature view is mapped one-to-one to a specific Cassandra table
* This implementation inherits all strengths of Cassandra such as high availability, fault-tolerance, and data distribution

An easy way to get started is the command `feast init REPO_NAME -t cassandra`.

### Example (Cassandra)

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: cassandra
hosts:
- 192.168.1.1
- 192.168.1.2
- 192.168.1.3
keyspace: KeyspaceName
port: 9042 # optional
username: user # optional
password: secret # optional
protocol_version: 5 # optional
load_balancing: # optional
local_dc: 'datacenter1' # optional
load_balancing_policy: 'TokenAwarePolicy(DCAwareRoundRobinPolicy)' # optional
```
{% endcode %}

### Example (Astra DB)

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: cassandra
secure_bundle_path: /path/to/secure/bundle.zip
keyspace: KeyspaceName
username: Client_ID
password: Client_Secret
protocol_version: 4 # optional
load_balancing: # optional
local_dc: 'eu-central-1' # optional
load_balancing_policy: 'TokenAwarePolicy(DCAwareRoundRobinPolicy)' # optional

```
{% endcode %}

For a full explanation of configuration options please look at file
`sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md`.

Storage specifications can be found at `docs/specs/online_store_format.md`.
2 changes: 1 addition & 1 deletion 2 docs/roadmap.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
* [x] [Cassandra / AstraDB](https://github.com/datastaxdevs/feast-cassandra-online-store)
* [x] [Cassandra / AstraDB](https://docs.feast.dev/reference/online-stores/cassandra)
* [ ] Bigtable (in progress)
* **Feature Engineering**
* [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#))
Expand Down
80 changes: 80 additions & 0 deletions 80 docs/specs/online_store_format.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,86 @@ Other types of entity keys are not supported in this version of the specificatio

![Datastore Online Example](datastore_online_example.png)

## Cassandra/Astra DB Online Store Format

### Overview

Apache Cassandra™ is a table-oriented NoSQL distributed database. Astra DB is a managed database-as-a-service
built on Cassandra, and will be assimilated to the former in what follows.

In Cassandra, tables are grouped in _keyspaces_ (groups of related tables). Each table is comprised of
_rows_, each containing data for a given set of _columns_. Moreover, rows are grouped in _partitions_ according
to a _partition key_ (a portion of the uniqueness-defining _primary key_ set of columns), so that all rows
with the same values for the partition key are guaranteed to be stored on the same Cassandra nodes, next to each other,
which guarantees fast retrieval times.

This architecture makes Cassandra a good fit for an online feature store in Feast.

### Cassandra Online Store Format

Each project (denoted by its name, called "feature store name" elsewhere) may contain an
arbitrary number of `FeatureView`s: these correspond each to a specific table, and
all tables for a project are to be contained in a single keyspace. The keyspace should
have been created by the Feast user preliminarly and is to be specified in the feature store
configuration `yaml`.

The table for a project `project` and feature view `FeatureView` will have name
`project_FeatureView` (e.g. `feature_repo_driver_hourly_stats`).

All tables have the same structure. Cassandra is schemaful and the columns are strongly typed.
In the following table schema (which also serves as Chebotko diagram) the Python
and Cassandra data types are both specified:

|Table: |`<project>`_`<FeatureView>` | | _(Python type)_ |
|---------------|-----------------------------|--|----------------------|
|`entity_key` |`TEXT` |K | `str` |
|`feature_name` |`TEXT` |C↑| `str` |
|`value` |`BLOB` | | `bytes` |
|`event_ts` |`TIMESTAMP` | | `datetime.datetime` |
|`created_ts` |`TIMESTAMP` | | `datetime.datetime` |

Each row in the table represents a single value for a feature in a feature view,
thus associated to a specific entity. The choice of partitioning ensures that,
within a given feature view (i.e. a single table), for a given entity any number
of features can be retrieved with a single, best-practice-respecting query
(which is what happens in the `online_read` method implementation).


The `entity_key` column is computed as `serialize_entity_key(entityKey).hex()`,
where `entityKey` is of type `feast.protos.feast.types.EntityKey_pb2.EntityKey`.

The value of `feature_name` is the plain-text name of the feature as defined
in the corresponding `FeatureView`.

For `value`, the bytes from `[protoValue].SerializeToString()`
are used, where `protoValue` is of type `feast.protos.feast.types.Value_pb2.Value`.

Column `event_ts` stores the timestamp the feature value refers to, as passed
to the store method. Conversely, column `created_ts`, meant to store the write
time for the entry, is now being deprecated and will be never written by this
online-store implementation. Thanks to the internal storage mechanism of Cassandra,
this does not incur a noticeable performance penalty (hence, for the time being,
the column can be maintained in the schema).

### Example entry

For a project `feature_repo` and feature view named `driver_hourly_stats`,
a typical row in table `feature_repo_driver_hourly_stats` might look like:

|Column |content | notes |
|---------------|-----------------------------------------------------|-------------------------------------------------------------------|
|`entity_key` |`020000006472697665725f69640400000004000000ea030000` | from `"driver_id = 1002"` |
|`feature_name` |`conv_rate` | |
|`value` |`0x35f5696d3f` | from `float_val: 0.9273980259895325`, i.e. `(b'5\xf5im?').hex()` |
|`event_ts` |`2022-07-07 09:00:00.000000+0000` | from `datetime.datetime(2022, 7, 7, 9, 0)` |
|`created_ts` |`null` | not explicitly written to avoid unnecessary tombstones |

### Known Issues

If a `FeatureView` ever gets _re-defined_ in a schema-breaking way, the implementation is not able to rearrange the
schema of the underlying table accordingly (neither dropping all data nor, even less so, keeping it somehow).
This should never occur, lest one encounters all sorts of data-retrieval issues anywhere in Feast usage.

# Appendix

##### Appendix A. Value proto format.
Expand Down
7 changes: 7 additions & 0 deletions 7 sdk/python/docs/index.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,13 @@ HBase Online Store
:members:
:noindex:

Cassandra Online Store
-----------------------

.. automodule:: feast.infra.online_stores.contrib.cassandra_online_store.cassandra_online_store
:members:
:noindex:


Batch Materialization Engine
============================
Expand Down
20 changes: 14 additions & 6 deletions 20 sdk/python/docs/source/feast.infra.offline_stores.contrib.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,26 @@ Subpackages
Submodules
----------

feast.infra.offline\_stores.contrib.contrib\_repo\_configuration module
-----------------------------------------------------------------------
feast.infra.offline\_stores.contrib.postgres\_repo\_configuration module
------------------------------------------------------------------------

.. automodule:: feast.infra.offline_stores.contrib.contrib_repo_configuration
.. automodule:: feast.infra.offline_stores.contrib.postgres_repo_configuration
:members:
:undoc-members:
:show-inheritance:

feast.infra.offline\_stores.contrib.postgres\_repo\_configuration module
------------------------------------------------------------------------
feast.infra.offline\_stores.contrib.spark\_repo\_configuration module
---------------------------------------------------------------------

.. automodule:: feast.infra.offline_stores.contrib.postgres_repo_configuration
.. automodule:: feast.infra.offline_stores.contrib.spark_repo_configuration
:members:
:undoc-members:
:show-inheritance:

feast.infra.offline\_stores.contrib.trino\_repo\_configuration module
---------------------------------------------------------------------

.. automodule:: feast.infra.offline_stores.contrib.trino_repo_configuration
:members:
:undoc-members:
:show-inheritance:
Expand Down
21 changes: 21 additions & 0 deletions 21 ...python/docs/source/feast.infra.online_stores.contrib.cassandra_online_store.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
feast.infra.online\_stores.contrib.cassandra\_online\_store package
===================================================================

Submodules
----------

feast.infra.online\_stores.contrib.cassandra\_online\_store.cassandra\_online\_store module
-------------------------------------------------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.cassandra_online_store.cassandra_online_store
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

.. automodule:: feast.infra.online_stores.contrib.cassandra_online_store
:members:
:undoc-members:
:show-inheritance:
9 changes: 9 additions & 0 deletions 9 sdk/python/docs/source/feast.infra.online_stores.contrib.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,20 @@ Subpackages
.. toctree::
:maxdepth: 4

feast.infra.online_stores.contrib.cassandra_online_store
feast.infra.online_stores.contrib.hbase_online_store

Submodules
----------

feast.infra.online\_stores.contrib.cassandra\_repo\_configuration module
------------------------------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.cassandra_repo_configuration
:members:
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.contrib.hbase\_repo\_configuration module
--------------------------------------------------------------------

Expand Down
8 changes: 8 additions & 0 deletions 8 sdk/python/docs/source/feast.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,14 @@ feast.field module
:undoc-members:
:show-inheritance:

feast.file\_utils module
------------------------

.. automodule:: feast.file_utils
:members:
:undoc-members:
:show-inheritance:

feast.flags\_helper module
--------------------------

Expand Down
7 changes: 7 additions & 0 deletions 7 sdk/python/docs/source/index.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,13 @@ HBase Online Store
:members:
:noindex:

Cassandra Online Store
-----------------------

.. automodule:: feast.infra.online_stores.contrib.cassandra_online_store.cassandra_online_store
:members:
:noindex:


Batch Materialization Engine
============================
Expand Down
2 changes: 1 addition & 1 deletion 2 sdk/python/feast/cli.py
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -590,7 +590,7 @@ def materialize_incremental_command(ctx: click.Context, end_ts: str, views: List
"--template",
"-t",
type=click.Choice(
["local", "gcp", "aws", "snowflake", "spark", "postgres", "hbase"],
["local", "gcp", "aws", "snowflake", "spark", "postgres", "hbase", "cassandra"],
case_sensitive=False,
),
help="Specify a template for the created project",
Expand Down
Loading
Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.