GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Modernize precommit hooks and optimize test performance (#5929) · Marcus-Rosti/feast@ea7d4fa

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
  • .github/workflows
    • linter.yml
    • registry-rest-api-tests.yml
    • smoke_tests.yml
    • unit_tests.yml
  • .pre-commit-config.yaml
  • Makefile
  • infra/feast-operator
    • Makefile
  • scripts
    • check-init-py.sh
    • mypy-daemon.sh
    • uv-run.sh
  • sdk/python
    • feast
      • infra
        • compute_engines
          • aws_lambda
            • __init__.py
          • dag
            • __init__.py
        • online_stores/singlestore_online_store
          • __init__.py
      • templates/snowflake
        • __init__.py
        • bootstrap.py
        • feature_repo
          • __init__.py
    • pyproject.toml
    • pytest.ini
    • tests
      • doctest
        • test_all.py
      • integration/feature_repos
        • repo_configuration.py
        • universal/online_store
          • milvus.py
      • unit
        • cli
          • test_cli.py
          • test_cli_chdir.py
        • local_feast_tests
          • test_e2e_local.py
      • utils
        • cli_repo_creator.py
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎.github/workflows/linter.yml‎

Copy file name to clipboardExpand all lines: .github/workflows/linter.yml
+8Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,14 @@ jobs:
2222
- name: Install dependencies
2323
run: |
2424
make install-python-dependencies-ci
25+
- name: Cache MyPy
26+
uses: actions/cache@v4
27+
with:
28+
path: sdk/python/.mypy_cache
29+
key: mypy-${{ runner.os }}-py${{ env.PYTHON }}-${{ hashFiles('pyproject.toml', 'setup.py', 'sdk/python/pyproject.toml', 'sdk/python/requirements/*.txt') }}
30+
restore-keys: |
31+
mypy-${{ runner.os }}-py${{ env.PYTHON }}-
32+
mypy-${{ runner.os }}-
2533
- name: Lint python
2634
run: make lint-python
2735
- name: Minimize uv cache
Collapse file

‎.github/workflows/registry-rest-api-tests.yml‎

Copy file name to clipboardExpand all lines: .github/workflows/registry-rest-api-tests.yml
+1-2Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,8 +145,7 @@ jobs:
145145
- name: Setup and Run Registry Rest API tests
146146
run: |
147147
echo "Running Registry REST API tests..."
148-
cd sdk/python
149-
pytest tests/integration/registration/rest_api/test_registry_rest_api.py --integration -s
148+
uv run pytest sdk/python/tests/integration/registration/rest_api/test_registry_rest_api.py --integration -s
150149
151150
- name: Clean up docker images
152151
if: always()
Collapse file

‎.github/workflows/smoke_tests.yml‎

Copy file name to clipboardExpand all lines: .github/workflows/smoke_tests.yml
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,6 @@ jobs:
2929
- name: Install dependencies
3030
run: make install-python-dependencies-ci
3131
- name: Test Imports
32-
run: python -c "from feast.cli import cli"
32+
run: uv run python -c "from feast.cli import cli"
3333
- name: Minimize uv cache
3434
run: uv cache prune --ci
Collapse file

‎.github/workflows/unit_tests.yml‎

Copy file name to clipboardExpand all lines: .github/workflows/unit_tests.yml
+31-1Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,40 @@ jobs:
3333
uses: astral-sh/setup-uv@v5
3434
with:
3535
enable-cache: true
36+
- name: Install system dependencies
37+
run: |
38+
if [ "$RUNNER_OS" = "Linux" ]; then
39+
sudo apt-get update
40+
sudo apt-get install -y make
41+
elif [ "$RUNNER_OS" = "macOS" ]; then
42+
# make is already installed on macOS runners
43+
which make || brew install make
44+
fi
3645
- name: Install dependencies
3746
run: make install-python-dependencies-ci
3847
- name: Test Python
39-
run: make test-python-unit
48+
run: |
49+
# Set up environment for Ray workers to access packages
50+
export PATH="${{ github.workspace }}/.venv/bin:$PATH"
51+
52+
# Dynamically detect site-packages for uv env
53+
SITE_PACKAGES=$(uv run python -c "import site; print(site.getsitepackages()[0])")
54+
55+
# Preserve any existing PYTHONPATH and add repo + site-packages
56+
export PYTHONPATH="${{ github.workspace }}/sdk/python:${PYTHONPATH}:$SITE_PACKAGES"
57+
58+
echo "Using PYTHONPATH: $PYTHONPATH"
59+
echo "Using PATH: $PATH"
60+
61+
# Ray macOS workarounds
62+
if [[ "$RUNNER_OS" == "macOS" ]]; then
63+
echo "=== Applying macOS Ray compatibility workarounds ==="
64+
export RAY_DISABLE_RUNTIME_ENV_HOOK=1
65+
export PYTHONDONTWRITEBYTECODE=1
66+
echo "Applied macOS workarounds for Python ${{ matrix.python-version }}"
67+
fi
68+
69+
make test-python-unit
4070
- name: Minimize uv cache
4171
run: uv cache prune --ci
4272

Collapse file

‎.pre-commit-config.yaml‎

Copy file name to clipboard
+6-6Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,23 @@
1-
default_stages:
2-
- push
1+
default_stages: [commit]
2+
33
repos:
44
- repo: local
55
hooks:
66
- id: format
77
name: Format
8-
stages: [ push ]
8+
stages: [commit]
99
language: system
1010
entry: make format-python
1111
pass_filenames: false
1212
- id: lint
1313
name: Lint
14-
stages: [ push ]
14+
stages: [commit]
1515
language: system
1616
entry: make lint-python
1717
pass_filenames: false
1818
- id: template
1919
name: Build Templates
20-
stages: [ commit ]
20+
stages: [commit]
2121
language: system
2222
entry: make build-templates
23-
pass_filenames: false
23+
pass_filenames: false
Collapse file

‎Makefile‎

Copy file name to clipboardExpand all lines: Makefile
+69-27Lines changed: 69 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -55,14 +55,36 @@ protos: compile-protos-python compile-protos-docs ## Compile protobufs for Pytho
5555
build: protos build-docker ## Build protobufs and Docker images
5656

5757
format-python: ## Format Python code
58-
cd ${ROOT_DIR}/sdk/python; python -m ruff check --fix feast/ tests/
59-
cd ${ROOT_DIR}/sdk/python; python -m ruff format feast/ tests/
58+
uv run ruff check --fix sdk/python/feast/ sdk/python/tests/
59+
uv run ruff format sdk/python/feast/ sdk/python/tests/
6060

6161
lint-python: ## Lint Python code
62-
cd ${ROOT_DIR}/sdk/python; python -m mypy feast
63-
cd ${ROOT_DIR}/sdk/python; python -m ruff check feast/ tests/
64-
cd ${ROOT_DIR}/sdk/python; python -m ruff format --check feast/ tests
65-
62+
uv run ruff check sdk/python/feast/ sdk/python/tests/
63+
uv run ruff format --check sdk/python/feast/ sdk/python/tests/
64+
uv run bash -c "cd sdk/python && mypy feast"
65+
66+
# New combined target
67+
precommit-check: format-python lint-python ## Run all precommit checks
68+
@echo "✅ All precommit checks passed"
69+
70+
# Install precommit hooks with correct stages
71+
install-precommit: ## Install precommit hooks (runs on commit, not push)
72+
pip install pre-commit
73+
pre-commit install --hook-type pre-commit
74+
@echo "✅ Precommit hooks installed (will run on commit, not push)"
75+
76+
# Manual full type check
77+
mypy-full: ## Full MyPy type checking with all files
78+
uv run bash -c "cd sdk/python && mypy feast tests"
79+
80+
# Run precommit on all files
81+
precommit-all: ## Run all precommit hooks on all files
82+
pre-commit run --all-files
83+
84+
# Make scripts executable
85+
setup-scripts: ## Make helper scripts executable
86+
chmod +x scripts/uv-run.sh scripts/check-init-py.sh scripts/mypy-daemon.sh
87+
6688
##@ Python SDK - local
6789
# formerly install-python-ci-dependencies-uv-venv
6890
# editable install
@@ -74,23 +96,22 @@ install-python-dependencies-minimal: ## Install minimal Python dependencies usin
7496
uv pip sync --require-hashes sdk/python/requirements/py$(PYTHON_VERSION)-minimal-requirements.txt
7597
uv pip install --no-deps -e .[minimal]
7698

77-
##@ Python SDK - system
78-
# the --system flag installs dependencies in the global python context
79-
# instead of a venv which is useful when working in a docker container or ci.
99+
##@ Python SDK - CI (uses uv with virtualenv)
100+
# Uses uv pip sync with virtualenv for CI environments
80101

81102
# Used in github actions/ci
82-
# formerly install-python-ci-dependencies-uv
83-
install-python-dependencies-ci: ## Install Python CI dependencies in system environment using uv
84-
# Install CPU-only torch first to prevent CUDA dependency issues
85-
pip uninstall torch torchvision -y || true
103+
install-python-dependencies-ci: ## Install Python CI dependencies using uv pip sync
104+
# Create virtualenv if it doesn't exist
105+
uv venv .venv
106+
# Install CPU-only torch first to prevent CUDA dependency issues (Linux only)
86107
@if [ "$$(uname -s)" = "Linux" ]; then \
87108
echo "Installing dependencies with torch CPU index for Linux..."; \
88-
uv pip sync --system --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match sdk/python/requirements/py$(PYTHON_VERSION)-ci-requirements.txt; \
109+
uv pip sync --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match sdk/python/requirements/py$(PYTHON_VERSION)-ci-requirements.txt; \
89110
else \
90111
echo "Installing dependencies from PyPI for macOS..."; \
91-
uv pip sync --system sdk/python/requirements/py$(PYTHON_VERSION)-ci-requirements.txt; \
112+
uv pip sync sdk/python/requirements/py$(PYTHON_VERSION)-ci-requirements.txt; \
92113
fi
93-
uv pip install --system --no-deps -e .
114+
uv pip install --no-deps -e .
94115

95116
# Used in github actions/ci
96117
install-hadoop-dependencies-ci: ## Install Hadoop dependencies
@@ -151,22 +172,44 @@ benchmark-python-local: ## Run integration + benchmark tests for Python (local d
151172
##@ Tests
152173

153174
test-python-unit: ## Run Python unit tests (use pattern=<pattern> to filter tests, e.g., pattern=milvus, pattern=test_online_retrieval.py, pattern=test_online_retrieval.py::test_get_online_features_milvus)
154-
python -m pytest -n 8 --color=yes $(if $(pattern),-k "$(pattern)") sdk/python/tests
175+
uv run python -m pytest -n 8 --color=yes $(if $(pattern),-k "$(pattern)") sdk/python/tests
176+
177+
# Fast unit tests only
178+
test-python-unit-fast: ## Run fast unit tests only (no external dependencies)
179+
uv run python -m pytest sdk/python/tests/unit -n auto -x --tb=short
180+
181+
# Changed files only (requires pytest-testmon)
182+
test-python-changed: ## Run tests for changed files only
183+
uv run python -m pytest --testmon -n 8 --tb=short sdk/python/tests
184+
185+
# Quick smoke test for PRs
186+
test-python-smoke: ## Quick smoke test for development
187+
uv run python -m pytest \
188+
sdk/python/tests/unit/test_unit_feature_store.py \
189+
sdk/python/tests/unit/test_repo_operations_validate_feast_project_name.py \
190+
-n 4 --tb=short
155191

156192
test-python-integration: ## Run Python integration tests (CI)
157-
python -m pytest --tb=short -v -n 8 --integration --color=yes --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
193+
uv run python -m pytest --tb=short -v -n 8 --integration --color=yes --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
158194
-k "(not snowflake or not test_historical_features_main)" \
159195
-m "not rbac_remote_integration_test" \
160196
--log-cli-level=INFO -s \
161197
sdk/python/tests
162198

199+
# Integration tests with better parallelization
200+
test-python-integration-parallel: ## Run integration tests with enhanced parallelization
201+
uv run python -m pytest sdk/python/tests/integration \
202+
-n auto --dist loadscope \
203+
--timeout=300 --tb=short -v \
204+
--integration --color=yes --durations=20
205+
163206
test-python-integration-local: ## Run Python integration tests (local dev mode)
164207
FEAST_IS_LOCAL_TEST=True \
165208
FEAST_LOCAL_ONLINE_CONTAINER=True \
166209
HADOOP_HOME=$$HOME/hadoop \
167210
CLASSPATH="$$( $$HADOOP_HOME/bin/hadoop classpath --glob ):$$CLASSPATH" \
168211
HADOOP_USER_NAME=root \
169-
python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
212+
uv run python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
170213
-k "not test_lambda_materialization and not test_snowflake_materialization" \
171214
-m "not rbac_remote_integration_test" \
172215
--log-cli-level=INFO -s \
@@ -175,7 +218,7 @@ test-python-integration-local: ## Run Python integration tests (local dev mode)
175218
test-python-integration-rbac-remote: ## Run Python remote RBAC integration tests
176219
FEAST_IS_LOCAL_TEST=True \
177220
FEAST_LOCAL_ONLINE_CONTAINER=True \
178-
python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
221+
uv run python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
179222
-k "not test_lambda_materialization and not test_snowflake_materialization" \
180223
-m "rbac_remote_integration_test" \
181224
--log-cli-level=INFO -s \
@@ -184,7 +227,7 @@ test-python-integration-rbac-remote: ## Run Python remote RBAC integration tests
184227
test-python-integration-container: ## Run Python integration tests using Docker
185228
@(docker info > /dev/null 2>&1 && \
186229
FEAST_LOCAL_ONLINE_CONTAINER=True \
187-
python -m pytest -n 8 --integration sdk/python/tests \
230+
uv run python -m pytest -n 8 --integration sdk/python/tests \
188231
) || echo "This script uses Docker, and it isn't running - please start the Docker Daemon and try again!";
189232

190233
test-python-integration-dbt: ## Run dbt integration tests
@@ -242,7 +285,7 @@ test-python-historical-retrieval:
242285
test_historical_features_persisting or \
243286
test_historical_retrieval_fails_on_validation" \
244287
sdk/python/tests
245-
288+
246289
test-python-universal-trino: ## Run Python Trino integration tests
247290
PYTHONPATH='.' \
248291
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.trino_repo_configuration \
@@ -578,7 +621,7 @@ test-python-universal-couchbase-online: ## Run Python Couchbase online store int
578621
sdk/python/tests
579622

580623
test-python-universal: ## Run all Python integration tests
581-
python -m pytest -n 8 --integration sdk/python/tests
624+
uv run python -m pytest -n 8 --integration sdk/python/tests
582625

583626
##@ Java
584627

@@ -644,7 +687,7 @@ build-feature-transformation-server-docker: ## Build Feature Transformation Serv
644687
push-feature-server-java-docker: ## Push Feature Server Java Docker image
645688
docker push $(REGISTRY)/feature-server-java:$(VERSION)
646689

647-
build-feature-server-java-docker: ## Build Feature Server Java Docker image
690+
build-feature-server-java-docker: ## Build Feature Server Java Docker image
648691
docker buildx build --build-arg VERSION=$(VERSION) \
649692
-t $(REGISTRY)/feature-server-java:$(VERSION) \
650693
-f java/infra/docker/feature-server/Dockerfile --load .
@@ -749,12 +792,12 @@ build-ui-local: ## Build Feast UI locally
749792
cd $(ROOT_DIR)/ui && yarn install && npm run build --omit=dev
750793
rm -rf $(ROOT_DIR)/sdk/python/feast/ui/build
751794
cp -r $(ROOT_DIR)/ui/build $(ROOT_DIR)/sdk/python/feast/ui/
752-
795+
753796
format-ui: ## Format Feast UI
754797
cd $(ROOT_DIR)/ui && NPM_TOKEN= yarn install && NPM_TOKEN= yarn format
755798

756799

757-
##@ Go SDK
800+
##@ Go SDK
758801
PB_REL = https://github.com/protocolbuffers/protobuf/releases
759802
PB_VERSION = 30.2
760803
PB_ARCH := $(shell uname -m)
@@ -820,4 +863,3 @@ build-go-docker-dev: ## Build Go Docker image for development
820863
docker buildx build --build-arg VERSION=dev \
821864
-t feastdev/feature-server-go:dev \
822865
-f go/infra/docker/feature-server/Dockerfile --load .
823-
Collapse file

‎infra/feast-operator/Makefile‎

Copy file name to clipboardExpand all lines: infra/feast-operator/Makefile
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -131,10 +131,10 @@ test-upgrade:
131131
test-previous-version:
132132
go test -timeout 60m ./test/previous-version/ -v -ginkgo.v
133133

134-
# Requires python3
134+
# Requires python3 with feast installed (use uv run in CI)
135135
.PHONY: test-datasources
136136
test-datasources:
137-
python3 test/data-source-types/data-source-types.py
137+
cd ../.. && uv run python infra/feast-operator/test/data-source-types/data-source-types.py
138138
go test ./test/data-source-types/
139139

140140
.PHONY: lint
Collapse file

‎scripts/check-init-py.sh‎

Copy file name to clipboard
+31Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/bin/bash
2+
# Check for missing __init__.py files in Python packages
3+
4+
set -euo pipefail
5+
6+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
7+
ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
8+
9+
# Find Python package directories missing __init__.py
10+
missing_init_files=()
11+
12+
while IFS= read -r -d '' dir; do
13+
# Skip .ipynb_checkpoints directories and other unwanted directories
14+
if [[ "${dir}" == *".ipynb_checkpoints"* ]] || [[ "${dir}" == *"__pycache__"* ]]; then
15+
continue
16+
fi
17+
18+
if [[ ! -f "${dir}/__init__.py" ]] && [[ -n "$(find "${dir}" -maxdepth 1 -name "*.py" -print -quit)" ]]; then
19+
missing_init_files+=("${dir}")
20+
fi
21+
done < <(find "${ROOT_DIR}/sdk/python/feast" -type d -print0)
22+
23+
if [[ ${#missing_init_files[@]} -gt 0 ]]; then
24+
echo "❌ Missing __init__.py files in:"
25+
printf " %s\n" "${missing_init_files[@]}"
26+
echo ""
27+
echo "Run: touch ${missing_init_files[*]/%//__init__.py}"
28+
exit 1
29+
fi
30+
31+
echo "✅ All Python packages have __init__.py files"
Collapse file

‎scripts/mypy-daemon.sh‎

Copy file name to clipboard
+55Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
#!/bin/bash
2+
# MyPy daemon for sub-second type checking
3+
4+
set -euo pipefail
5+
6+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
7+
ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
8+
9+
MYPY_CACHE_DIR="${ROOT_DIR}/sdk/python/.mypy_cache"
10+
PID_FILE="$MYPY_CACHE_DIR/dmypy.pid"
11+
12+
case "${1:-}" in
13+
start)
14+
echo "🚀 Starting MyPy daemon..."
15+
cd ${ROOT_DIR}/sdk/python
16+
uv run dmypy start -- --config-file=pyproject.toml
17+
echo "✅ MyPy daemon started"
18+
;;
19+
check)
20+
echo "🔍 Running MyPy daemon check..."
21+
cd ${ROOT_DIR}/sdk/python
22+
time uv run dmypy check feast tests
23+
;;
24+
stop)
25+
echo "🛑 Stopping MyPy daemon..."
26+
cd ${ROOT_DIR}/sdk/python
27+
uv run dmypy stop
28+
echo "✅ MyPy daemon stopped"
29+
;;
30+
restart)
31+
echo "🔄 Restarting MyPy daemon..."
32+
$0 stop
33+
$0 start
34+
;;
35+
status)
36+
echo "📊 MyPy daemon status:"
37+
cd ${ROOT_DIR}/sdk/python
38+
if uv run dmypy status; then
39+
echo "✅ MyPy daemon is running"
40+
else
41+
echo "❌ MyPy daemon is not running"
42+
fi
43+
;;
44+
*)
45+
echo "Usage: $0 {start|check|stop|restart|status}"
46+
echo ""
47+
echo "Commands:"
48+
echo " start - Start the MyPy daemon"
49+
echo " check - Run type checking with the daemon"
50+
echo " stop - Stop the MyPy daemon"
51+
echo " restart - Restart the daemon"
52+
echo " status - Check daemon status"
53+
exit 1
54+
;;
55+
esac

0 commit comments

Comments
0 (0)
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.