GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Horizontal scaling support to the Feast operator by ntkathole · Pull Request #6000 · feast-dev/feast

feat: Horizontal scaling support to the Feast operator #6000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ntkathole merged 5 commits into feast-dev:master from ntkathole:operator_hpa
Feb 27, 2026

Uh oh!

There was an error while loading. Please reload this page.
Merged

feat: Horizontal scaling support to the Feast operator #6000

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6656ec1
feat: Horizontal scaling support to the Feast operator
ntkathole Feb 21, 2026
9241ba9
docs: Added blog post on horizontal scaling
ntkathole Feb 23, 2026
dee0cd6
fix: Address comments
ntkathole Feb 26, 2026
101dd55
fix: Server side apply
ntkathole Feb 26, 2026
befdf23
fix: Use autoscalingv2.HorizontalPodAutoscaler
ntkathole Feb 26, 2026
File filter
Filter by extension
Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Diff view
Diff view

There are no files selected for viewing

16 changes: 10 additions & 6 deletions 16 .secrets.baseline
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
6 changes: 4 additions & 2 deletions 6 docs/how-to-guides/feast-on-kubernetes.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ spec:
> _More advanced FeatureStore CR examples can be found in the feast-operator [samples directory](../../infra/feast-operator/config/samples)._

{% hint style="success" %}
Important note: Scaling a Feature Store Deployment should only be done if the configured data store(s) will support it.
**Scaling:** The Feast Operator supports horizontal scaling via static replicas, HPA autoscaling, or external autoscalers like [KEDA](https://keda.sh). Scaling requires DB-backed persistence for all enabled services.

Please check the how-to guide for some specific recommendations on [how to scale Feast](./scaling-feast.md).
See the [Horizontal Scaling with the Feast Operator](./scaling-feast.md#horizontal-scaling-with-the-feast-operator) guide for configuration details, or check the general recommendations on [how to scale Feast](./scaling-feast.md).
{% endhint %}

> _Sample scaling CRs are available at [`v1_featurestore_scaling_static.yaml`](../../infra/feast-operator/config/samples/v1_featurestore_scaling_static.yaml) and [`v1_featurestore_scaling_hpa.yaml`](../../infra/feast-operator/config/samples/v1_featurestore_scaling_hpa.yaml)._
156 changes: 155 additions & 1 deletion 156 docs/how-to-guides/scaling-feast.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,158 @@ However, this process does not scale for large data sets, since it's executed on
Feast supports pluggable [Compute Engines](../getting-started/components/compute-engine.md), that allow the materialization process to be scaled up.
Aside from the local process, Feast supports a [Lambda-based materialization engine](https://rtd.feast.dev/en/master/#alpha-lambda-based-engine), and a [Bytewax-based materialization engine](https://rtd.feast.dev/en/master/#bytewax-engine).

Users may also be able to build an engine to scale up materialization using existing infrastructure in their organizations.
Users may also be able to build an engine to scale up materialization using existing infrastructure in their organizations.

### Horizontal Scaling with the Feast Operator

When running Feast on Kubernetes with the [Feast Operator](./feast-on-kubernetes.md), you can horizontally scale the FeatureStore deployment using `spec.replicas` or HPA autoscaling. The FeatureStore CRD implements the Kubernetes [scale sub-resource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource), so you can also use `kubectl scale`:

```bash
kubectl scale featurestore/my-feast --replicas=3
```

**Prerequisites:** Horizontal scaling requires **DB-backed persistence** for all enabled services (online store, offline store, and registry). File-based persistence (SQLite, DuckDB, `registry.db`) is incompatible with multiple replicas because these backends do not support concurrent access from multiple pods.

#### Static Replicas

Set a fixed number of replicas via `spec.replicas`:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: sample-scaling
spec:
feastProject: my_project
replicas: 3
services:
onlineStore:
persistence:
store:
type: postgres
secretRef:
name: feast-data-stores
registry:
local:
persistence:
store:
type: sql
secretRef:
name: feast-data-stores
```

#### Autoscaling with HPA

Configure a HorizontalPodAutoscaler to dynamically scale based on metrics. HPA autoscaling is configured under `services.scaling.autoscaling` and is mutually exclusive with `spec.replicas > 1`:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: sample-autoscaling
spec:
feastProject: my_project
services:
scaling:
autoscaling:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
onlineStore:
persistence:
store:
type: postgres
secretRef:
name: feast-data-stores
server:
resources:
requests:
cpu: 200m
memory: 256Mi
registry:
local:
persistence:
store:
type: sql
secretRef:
name: feast-data-stores
```

{% hint style="info" %}
When autoscaling is configured, the operator automatically sets the deployment strategy to `RollingUpdate` (instead of the default `Recreate`) to ensure zero-downtime scaling. You can override this by explicitly setting `deploymentStrategy` in the CR.
{% endhint %}

#### Validation Rules

The operator enforces the following rules:
- `spec.replicas > 1` and `services.scaling.autoscaling` are **mutually exclusive** -- you cannot set both.
- Scaling with `replicas > 1` or any `autoscaling` config is **rejected** if any enabled service uses file-based persistence.
- S3 (`s3://`) and GCS (`gs://`) backed registry file persistence is allowed with scaling, since these object stores support concurrent readers.

#### Using KEDA (Kubernetes Event-Driven Autoscaling)

[KEDA](https://keda.sh) is also supported as an external autoscaler. KEDA should target the FeatureStore's scale sub-resource directly (since it implements the Kubernetes scale API). This is the recommended approach because the operator manages the Deployment's replica count from `spec.replicas` — targeting the Deployment directly would conflict with the operator's reconciliation.

When using KEDA, do **not** set `scaling.autoscaling` or `spec.replicas > 1` -- KEDA manages the replica count through the scale sub-resource.

1. **Ensure DB-backed persistence** -- The CRD's CEL validation rules automatically enforce DB-backed persistence when KEDA scales `spec.replicas` above 1 via the scale sub-resource. The operator also automatically switches the deployment strategy to `RollingUpdate` when `replicas > 1`.

2. **Configure the FeatureStore** with DB-backed persistence:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: sample-keda
spec:
feastProject: my_project
services:
onlineStore:
persistence:
store:
type: postgres
secretRef:
name: feast-data-stores
registry:
local:
persistence:
store:
type: sql
secretRef:
name: feast-data-stores
```

3. **Create a KEDA `ScaledObject`** targeting the FeatureStore resource:

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: feast-scaledobject
spec:
scaleTargetRef:
apiVersion: feast.dev/v1
kind: FeatureStore
name: sample-keda
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring.svc:9090
metricName: http_requests_total
query: sum(rate(http_requests_total{service="feast"}[2m]))
threshold: "100"
```

{% hint style="warning" %}
KEDA-created HPAs are not owned by the Feast operator. The operator will not interfere with them, but it also will not clean them up if the FeatureStore CR is deleted. You must manage the KEDA `ScaledObject` lifecycle independently.
{% endhint %}

For the full API reference, see the [FeatureStore CRD reference](../../infra/feast-operator/docs/api/markdown/ref.md).
54 changes: 54 additions & 0 deletions 54 infra/feast-operator/api/v1/featurestore_types.go
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package v1

import (
appsv1 "k8s.io/api/apps/v1"
autoscalingv2 "k8s.io/api/autoscaling/v2"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand Down Expand Up @@ -67,6 +68,10 @@ const (
)

// FeatureStoreSpec defines the desired state of FeatureStore
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)",message="replicas > 1 and services.scaling.autoscaling are mutually exclusive."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) && has(self.services.onlineStore.persistence.store))",message="Scaling requires DB-backed persistence for the online store. Configure services.onlineStore.persistence.store when using replicas > 1 or autoscaling."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (!has(self.services) || !has(self.services.offlineStore) || (has(self.services.offlineStore.persistence) && has(self.services.offlineStore.persistence.store)))",message="Scaling requires DB-backed persistence for the offline store. Configure services.offlineStore.persistence.store when using replicas > 1 or autoscaling."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.registry) && (has(self.services.registry.remote) || (has(self.services.registry.local) && has(self.services.registry.local.persistence) && (has(self.services.registry.local.persistence.store) || (has(self.services.registry.local.persistence.file) && has(self.services.registry.local.persistence.file.path) && (self.services.registry.local.persistence.file.path.startsWith('s3://') || self.services.registry.local.persistence.file.path.startsWith('gs://')))))))",message="Scaling requires DB-backed or remote registry. Configure registry.local.persistence.store or use a remote registry when using replicas > 1 or autoscaling. S3/GCS-backed registry is also allowed."
type FeatureStoreSpec struct {
// +kubebuilder:validation:Pattern="^[A-Za-z0-9][A-Za-z0-9_-]*$"
// FeastProject is the Feast project id. This can be any alphanumeric string with underscores and hyphens, but it cannot start with an underscore or hyphen. Required.
Expand All @@ -76,6 +81,11 @@ type FeatureStoreSpec struct {
AuthzConfig *AuthzConfig `json:"authz,omitempty"`
CronJob *FeastCronJob `json:"cronJob,omitempty"`
BatchEngine *BatchEngineConfig `json:"batchEngine,omitempty"`
// Replicas is the desired number of pod replicas. Used by the scale sub-resource.
// Mutually exclusive with services.scaling.autoscaling.
// +kubebuilder:default=1
// +kubebuilder:validation:Minimum=1
Replicas *int32 `json:"replicas"`
}

// FeastProjectDir defines how to create the feast project directory.
Expand Down Expand Up @@ -301,6 +311,35 @@ type FeatureStoreServices struct {
DisableInitContainers bool `json:"disableInitContainers,omitempty"`
// Volumes specifies the volumes to mount in the FeatureStore deployment. A corresponding `VolumeMount` should be added to whichever feast service(s) require access to said volume(s).
Volumes []corev1.Volume `json:"volumes,omitempty"`
// Scaling configures horizontal scaling for the FeatureStore deployment (e.g. HPA autoscaling).
// For static replicas, use spec.replicas instead.
Scaling *ScalingConfig `json:"scaling,omitempty"`
}

// ScalingConfig configures horizontal scaling for the FeatureStore deployment.
type ScalingConfig struct {
// Autoscaling configures a HorizontalPodAutoscaler for the FeatureStore deployment.
// Mutually exclusive with spec.replicas.
// +optional
Autoscaling *AutoscalingConfig `json:"autoscaling,omitempty"`
}

// AutoscalingConfig defines HPA settings for the FeatureStore deployment.
type AutoscalingConfig struct {
// MinReplicas is the lower limit for the number of replicas. Defaults to 1.
// +kubebuilder:validation:Minimum=1
// +optional
MinReplicas *int32 `json:"minReplicas,omitempty"`
// MaxReplicas is the upper limit for the number of replicas. Required.
// +kubebuilder:validation:Minimum=1
MaxReplicas int32 `json:"maxReplicas"`
// Metrics contains the specifications for which to use to calculate the desired replica count.
// If not set, defaults to 80% CPU utilization.
// +optional
Metrics []autoscalingv2.MetricSpec `json:"metrics,omitempty"`
// Behavior configures the scaling behavior of the target.
// +optional
Behavior *autoscalingv2.HorizontalPodAutoscalerBehavior `json:"behavior,omitempty"`
}

// OfflineStore configures the offline store service
Expand Down Expand Up @@ -690,6 +729,20 @@ type FeatureStoreStatus struct {
FeastVersion string `json:"feastVersion,omitempty"`
Phase string `json:"phase,omitempty"`
ServiceHostnames ServiceHostnames `json:"serviceHostnames,omitempty"`
// Replicas is the current number of ready pod replicas (used by the scale sub-resource).
Replicas int32 `json:"replicas,omitempty"`
// Selector is the label selector for pods managed by the FeatureStore deployment (used by the scale sub-resource).
Selector string `json:"selector,omitempty"`
// ScalingStatus reports the current scaling state of the FeatureStore deployment.
ScalingStatus *ScalingStatus `json:"scalingStatus,omitempty"`
}

// ScalingStatus reports the observed scaling state.
type ScalingStatus struct {
Comment thread
ntkathole marked this conversation as resolved.
Resolved

Uh oh!

There was an error while loading. Please reload this page.
// CurrentReplicas is the current number of pod replicas.
CurrentReplicas int32 `json:"currentReplicas,omitempty"`
// DesiredReplicas is the desired number of pod replicas.
DesiredReplicas int32 `json:"desiredReplicas,omitempty"`
}

// ServiceHostnames defines the service hostnames in the format of <domain>:<port>, e.g. example.svc.cluster.local:80
Expand All @@ -706,6 +759,7 @@ type ServiceHostnames struct {
// +kubebuilder:resource:shortName=feast
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector
// +kubebuilder:storageversion

// FeatureStore is the Schema for the featurestores API
Expand Down
Loading
Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Loading
Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.