GeezerVison is showing github.com — feat(cli): Updated feast init demo by adding rag template by Vperiodt · Pull Request #5946 · feast-dev/feast
feat(cli): Updated feast init demo by adding rag template
#5946
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement . We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
Show file tree
Hide file tree
Changes from all commits
Commits
File filter
Filter by extension
.md
(1)
.py
(6)
.yaml
(1)
No extension
(1)
All 4 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Hide whitespace
Diff view
Hide whitespace
There are no files selected for viewing
Original file line number
Diff line number
Diff line change
Expand Up
@@ -477,6 +477,7 @@ def materialize_incremental_command(
"milvus",
"ray",
"ray_rag",
"rag",
"pytorch_nlp",
],
case_sensitive=False,
Expand Down
Original file line number
Diff line number
Diff line change
@@ -0,0 +1,99 @@
# City Information Q&A — RAG Demo with Feast
A complete Retrieval-Augmented Generation (RAG) demo using Feast for feature management and Milvus for vector search.
## Project Structure
```
rag/
├── feature_repo/
│ ├── data/
│ │ └── city_wikipedia_summaries_with_embeddings.parquet # Sample data (US cities)
│ ├── example_repo.py # Entity, Feature Views, Feature Service definitions
│ ├── feature_store.yaml # Feast config (Milvus online store, file offline store)
│ └── test_workflow.py # End-to-end demo: apply → materialize → search
└── README.md
```
## Quick Start
### 1. Initialize the template
```bash
feast init -t rag my_city_qa
cd my_city_qa/feature_repo
```
### 2. Install dependencies
```bash
pip install feast torch transformers pymilvus
```
### 3. Apply feature definitions
```bash
feast apply
```
### 4. Explore in the Feast UI
```bash
feast ui
```
### 5. Run the demo workflow
```bash
python test_workflow.py
```
## Key Commands
| Command | Description |
|---------|-------------|
| `feast apply` | Register entities, feature views, and feature services |
| `feast materialize --disable-event-timestamp` | Load parquet data into the online store (Milvus) for vector search. Optionally add `-v city_summary_embeddings -v city_metadata` to materialize only those views. |
| `feast feature-views list` | List registered feature views |
| `feast entities list` | List registered entities |
| `feast feature-services list` | List registered feature services |
| `feast ui` | Start the Feast UI at http://localhost:8888 |
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ City Q&A Pipeline │
├─────────────────────────────────────────────────────────────────┤
│ │
│ User Question │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Embed Query │ (MiniLM 384-dim) │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────┐ │
│ │ city_summary_embeddings (Milvus) │ ← Vector Search │
│ │ - vector (COSINE similarity) │ │
│ │ - sentence_chunks │ │
│ └─────────────────────────────────────┘ │
│ │ │
│ ▼ (top-k city_ids) │
│ ┌─────────────────────────────────────┐ │
│ │ city_metadata (Feast Online Store) │ ← Metadata Lookup │
│ │ - state │ │
│ │ - wiki_summary │ │
│ └─────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ LLM Answer │ (optional: GPT/Claude) │
│ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
Original file line number
Diff line number
Diff line change
@@ -0,0 +1 @@
Original file line number
Diff line number
Diff line change
@@ -0,0 +1,56 @@
def bootstrap():
# Bootstrap() is called from init_repo() during `feast init`
import pathlib
from datetime import datetime
import numpy as np
import pandas as pd
repo_path = pathlib.Path(__file__).parent.absolute() / "feature_repo"
data_path = repo_path / "data"
data_path.mkdir(exist_ok=True)
# Minimal city data with embeddings (384-d to match feature_store embedding_dim)
embedding_dim = 384
now = datetime.now().replace(microsecond=0, tzinfo=None)
cities = [
(
1,
"New York",
"New York",
"New York City is the most populous city in the United States.",
),
(
2,
"Los Angeles",
"California",
"Los Angeles is the second most populous city in the United States.",
),
(
3,
"Chicago",
"Illinois",
"Chicago is the third most populous city in the United States.",
),
]
rows = []
for city_id, city_name, state, wiki_summary in cities:
vec = np.random.randn(embedding_dim).astype(np.float32)
vec = (vec / np.linalg.norm(vec)).tolist()
rows.append(
Comment thread
Comment on lines
+35
to
+40
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more .
[Critical] Missing unit tests for bootstrap functionality
The user explicitly requested unit testing covering all major impact. The bootstrap.py function creates test data and should have comprehensive unit tests to verify data generation, file creation, and error handling scenarios.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
{
"city_id": city_id,
"event_timestamp": pd.Timestamp(now),
"vector": vec,
"sentence_chunks": wiki_summary[:200],
"state": f"{city_name}, {state}",
"wiki_summary": wiki_summary,
}
)
df = pd.DataFrame(rows)
parquet_path = data_path / "city_wikipedia_summaries_with_embeddings.parquet"
df.to_parquet(path=str(parquet_path), index=False)
if __name__ == "__main__":
bootstrap()
Original file line number
Diff line number
Diff line change
@@ -0,0 +1 @@
Original file line number
Diff line number
Diff line change
@@ -0,0 +1,137 @@
from datetime import timedelta
from feast import (
Entity,
FeatureService,
FeatureView,
Field,
FileSource,
PushSource,
)
from feast.data_format import ParquetFormat
from feast.types import Array, Float32, String
from feast.value_type import ValueType
# Entity: Identifies each city document/chunk in the knowledge base
city = Entity(
name="city_id",
value_type=ValueType.INT64,
description="Unique identifier for each city Wikipedia summary (document chunk ID).",
join_keys=["city_id"],
)
# Data Source: Parquet file containing city summaries with pre-computed embeddings
city_summaries_source = FileSource(
name="city_summaries_source",
file_format=ParquetFormat(),
path="./data/city_wikipedia_summaries_with_embeddings.parquet",
timestamp_field="event_timestamp",
description="Wikipedia summaries of US cities (batch).",
)
# Push Source: same schema as batch; allows near real-time ingestion of new/updated docs
city_summaries_push_source = PushSource(
name="city_summaries_push_source",
batch_source=city_summaries_source,
description="Push source for real-time updates to city summaries/embeddings.",
)
# Feature View 1: City embeddings for semantic/vector search (RAG retrieval)
city_summary_embeddings = FeatureView(
name="city_summary_embeddings",
description="City Wikipedia summaries with embeddings for semantic search. ",
entities=[city],
schema=[
Field(
name="vector",
dtype=Array(Float32),
description="384-dimensional sentence embedding for semantic similarity search (MiniLM).",
vector_index=True,
vector_search_metric="COSINE",
),
Field(
name="sentence_chunks",
dtype=String,
description="Chunked sentences from the Wikipedia summary.",
),
],
source=city_summaries_source,
ttl=timedelta(days=1),
online=True,
tags={"team": "ml-platform", "use_case": "city_qa", "type": "vector"},
)
# Feature View 2: City metadata for scalar lookups (no vector search)
city_metadata = FeatureView(
name="city_metadata",
description="City metadata including state and full Wikipedia summary. ",
entities=[city],
schema=[
Field(
name="state",
dtype=String,
description="US state where the city is located (e.g., 'New York, New York').",
),
Field(
name="wiki_summary",
dtype=String,
description="Full Wikipedia summary of the city.",
),
],
source=city_summaries_source,
ttl=timedelta(hours=2),
online=True,
tags={"team": "ml-platform", "use_case": "city_qa", "type": "metadata"},
)
# Feature View 3: Fresh embeddings (PushSource) for near real-time doc updates
city_summary_embeddings_realtime = FeatureView(
name="city_summary_embeddings_realtime",
description="Same as city_summary_embeddings but with real-time ingestion (PushSource).",
entities=[city],
schema=[
Field(
name="vector",
dtype=Array(Float32),
description="384-dimensional sentence embedding for semantic similarity search.",
vector_index=True,
vector_search_metric="COSINE",
),
Field(
name="sentence_chunks",
dtype=String,
description="Chunked sentences from the Wikipedia summary.",
),
],
source=city_summaries_push_source,
ttl=timedelta(hours=2),
online=True,
tags={
"team": "ml-platform",
"use_case": "city_qa",
"type": "vector",
"ingestion": "push",
},
)
# Feature Service: Bundles features for the City Q&A retrieval endpoint
city_qa_v1 = FeatureService(
name="city_qa_v1",
features=[
city_summary_embeddings,
city_metadata,
],
description="Feature service for City Information Q&A. ",
tags={"team": "ml-platform", "version": "v1"},
)
# Feature service that includes push-backed and request-time features
city_qa_v2 = FeatureService(
name="city_qa_v2",
features=[
city_summary_embeddings_realtime,
city_metadata,
],
description="City Q&A with push ingestion and request-time context (query_text, user_id).",
tags={"team": "ml-platform", "version": "v2"},
)
Original file line number
Diff line number
Diff line change
@@ -0,0 +1,20 @@
project: my_project
project_description: |
This project is a smart City Q&A assistant that gives you accurate, detailed answers about any city instantly.
provider: local
registry: data/registry.db
online_store:
type: milvus
path: data/online_store.db
vector_enabled: true
embedding_dim: 384
index_type: "FLAT"
metric_type: "COSINE"
offline_store:
type: file
entity_key_serialization_version: 3
# By default, no_auth for authentication and authorization
auth:
type: no_auth
Loading
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Loading
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.