GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat: Lint with ruff by dmartinol · Pull Request #4043 · feast-dev/feast

Conversation

dmartinol commented Mar 26, 2024

Copy link

Copy Markdown

Contributor

What this PR does / why we need it:
Replace the current linting and formatting stack with ruff>=0.3.3 (latest version).

Collateral changes:

Update a few source code files to adapt to the known differences between isort and ruff, see here.
Remove sdk/python/setup.cfg because not supported by ruff, and replaced by the equivalent pyproject.toml.
- Move here also the configuration for mypy .
Remove lint category B' as it seems to be never raised by flake8` (https://flake8.pycqa.org/en/latest/user/error-codes.html)
- Instead, this category raises numerous errors with ruff (flake8-bugbear)
Remove lint category C, because the only error raised by that rule was C901 (here, which was also added to the ignore list.
- Vice versa, ruff has other errors in the category C4 (see here that would be raised instead.
- Note about code complexity: flake8 and ruff compute different complexities for our code base, so it would be difficult to map the max-complexity number to a value that matches the new linting tool.
Added ignore rule for E721, a new error code introduced by flake8>=6.1.0, the reference version for ruff
- ruff raises a few violations in our codebase for this error, which could be fixed in a dedicated PR
- Final note: the list of files and lines of code affected by this error is different in flake8 version 7.0.0 and ruff, meaning that ruff re-implemented this rule with a different behavior
Remove lint category T4 because never raised by flake8 and unknown to ruff (Unknown rule selector: T4).
Remove ignore rule for W503 because never raised by flake8 and unknown to ruff (Unknown rule selector: W503).
Add rule I representing the isort category (see here)
Update requirements files using pip-compile.

Which issue(s) this PR fixes:
Fixes #3956

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

dmartinol requested a review from whoahbot as a code owner

March 26, 2024 16:17

dmartinol mentioned this pull request

Mar 26, 2024

feat: faster linting with ruff #3956

Closed

sudohainguyen commented Mar 26, 2024

Copy link

Copy Markdown

Collaborator

ruff check is not enough isn't it?
there's still ruff format --check

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          feat: Lin with ruff

af97594

Signed-off-by: Daniele Martinoli <[email protected]>

dmartinol force-pushed the ruff_linter branch from 82e91cd to af97594 Compare

March 26, 2024 16:43

dmartinol commented Mar 26, 2024

Copy link

Copy Markdown

Contributor Author

ruff check is not enough isn't it?

You mean instead of python -m ruff check feast tests?
1- I just wanted to preserve the existing convention used in the Makefile to run modules with python -m
2- If agreed, we can change to ruff check feast tests (ruff check alone scans also undesired folders like docs)

there's still ruff format --check
Isn't it better to separate the concerns and use check for linting and format only for formatting, as in python -m ruff format feast/ tests/?

Note: I see that the latter command tries to format more files than the ruff check --fix, so I think I need to investigate a bit more the formatting options that were previously implemented by black --target-version py38 feast tests

sudohainguyen reacted with eyes emoji

All reactions

👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

dmartinol added 2 commits

March 26, 2024 22:22


          fea: replace black with ruff format options

26db7ca

Signed-off-by: Daniele Martinoli <[email protected]>


          fea: files reformatted due to ruff deviations from black

8bd6ca6

Signed-off-by: Daniele Martinoli <[email protected]>

dmartinol requested review from kevjumba, niklasvm, sfc-gh-madkins, sudohainguyen and younggyu-oh as code owners

March 26, 2024 21:25

dmartinol commented Mar 26, 2024

Copy link

Copy Markdown

Contributor Author

Adding ruff formatting to replace black and the 52 files updated due to this change.
All changes seems due to the Known Deviations from Black.

jeremyary reacted with thumbs up emoji

sudohainguyen reacted with rocket emoji

All reactions

👍 1 reaction
🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sudohainguyen reviewed

Mar 27, 2024

View reviewed changes

Comment thread

Makefile Resolved

Uh oh!

There was an error while loading. Please reload this page.


          fix: restored mypy in lint-python target

7027d50

Signed-off-by: Daniele Martinoli <[email protected]>

dmartinol commented

Mar 27, 2024

View reviewed changes

Comment thread

sdk/python/feast/my.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

dmartinol commented

Mar 27, 2024

View reviewed changes

Comment thread

sdk/python/feast/feature_store.py Resolved

Uh oh!

There was an error while loading. Please reload this page.


          fix: fixed formatting error

14525b0

Signed-off-by: Daniele Martinoli <[email protected]>

jeremyary added the ok-to-test label

Mar 27, 2024

shuchu commented Mar 28, 2024

Copy link

Copy Markdown

Collaborator

by any chance, will you update the Document regarding the code style & lint? or that will be a separate PR. https://docs.feast.dev/project/development-guide#code-style--linting

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

dmartinol commented Mar 28, 2024

Copy link

Copy Markdown

Contributor Author

by any chance, will you update the Document regarding the code style & lint? or that will be a separate PR. https://docs.feast.dev/project/development-guide#code-style--linting

Yes it's here

I have some doubts about the sentence "Conforms to Black code style" because even if it's implemented by ruff format, the ruff formatter page states:

The Ruff formatter is an extremely fast Python code formatter designed as a drop-in replacement for Black, available as part of the ruff CLI via ruff format.

shuchu and sudohainguyen reacted with thumbs up emoji

All reactions

👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jeremyary commented Mar 29, 2024

Copy link

Copy Markdown

Collaborator

@sudohainguyen @shuchu @tokoko any further thoughts to share on the topic here? Just hoping to see if there's more for us to tackle next week. Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          adding ruff format --check

a6c70b7

Signed-off-by: Daniele Martinoli <[email protected]>

sudohainguyen changed the title ~~feat: Lin with ruff~~ feat: Lint with ruff

Mar 29, 2024

sudohainguyen reviewed

Mar 29, 2024

View reviewed changes

Comment thread

pyproject.toml Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

pyproject.toml Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

setup.cfg Resolved

Uh oh!

There was an error while loading. Please reload this page.

dmartinol added 3 commits

April 2, 2024 08:39


          Merge remote-tracking branch 'upstream/master' into ruff_linter

b622a08


          Formatting updated files

b478cc2

Signed-off-by: Daniele Martinoli <[email protected]>


          Adding section separator

16c0a9b

Signed-off-by: Daniele Martinoli <[email protected]>

sudohainguyen added the lgtm label

Apr 2, 2024

sudohainguyen approved these changes

Apr 2, 2024

View reviewed changes

sudohainguyen left a comment

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ltgm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

tokoko approved these changes

Apr 2, 2024

View reviewed changes

tokoko left a comment

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          rebased to main

9f4dd9a

Signed-off-by: Daniele Martinoli <[email protected]>

jeremyary merged commit 7f1557b into feast-dev:master

Apr 3, 2024

franciscojavierarceo pushed a commit that referenced this pull request

Apr 16, 2024


          chore(release): release 0.36.0

474eb62

# [0.36.0](v0.35.0...v0.36.0) (2024-04-16)

### Bug Fixes

* Add __eq__, __hash__ to SparkSource for correct comparison ([#4028](#4028)) ([e703b40](e703b40))
* Add conn.commit() to Postgresonline_write_batch.online_write_batch ([#3904](#3904)) ([7d75fc5](7d75fc5))
* Add missing __init__.py to embedded_go ([#4051](#4051)) ([6bb4c73](6bb4c73))
* Add missing init files in infra utils ([#4067](#4067)) ([54910a1](54910a1))
* Added registryPath parameter documentation in WebUI reference ([#3983](#3983)) ([5e0af8f](5e0af8f)), closes [#3974](#3974) [#3974](#3974)
* Adding missing init files in materialization modules ([#4052](#4052)) ([df05253](df05253))
* Allow trancated timestamps when converting ([#3861](#3861)) ([bdd7dfb](bdd7dfb))
* Azure blob storage support in Java feature server ([#2319](#2319)) ([#4014](#4014)) ([b9aabbd](b9aabbd))
* Bugfix for grabbing historical data from Snowflake with array type features. ([#3964](#3964)) ([1cc94f2](1cc94f2))
* Bytewax materialization engine fails when loading feature_store.yaml ([#3912](#3912)) ([987f0fd](987f0fd))
* CI unittest warnings ([#4006](#4006)) ([0441b8b](0441b8b))
* Correct the returning class proto type of StreamFeatureView to StreamFeatureViewProto instead of FeatureViewProto. ([#3843](#3843)) ([86d6221](86d6221))
* Create index only if not exists during MySQL online store update ([#3905](#3905)) ([2f99a61](2f99a61))
* Disable minio tests in workflows on master and nightly ([#4072](#4072)) ([c06dda8](c06dda8))
* Disable the Feast Usage feature by default. ([#4090](#4090)) ([b5a7013](b5a7013))
* Dump repo_config by alias ([#4063](#4063)) ([e4bef67](e4bef67))
* Extend SQL registry config with a sqlalchemy_config_kwargs key ([#3997](#3997)) ([21931d5](21931d5))
* Feature Server image startup in OpenShift clusters ([#4096](#4096)) ([9efb243](9efb243))
* Fix copy method for StreamFeatureView ([#3951](#3951)) ([cf06704](cf06704))
* Fix for materializing entityless feature views in Snowflake ([#3961](#3961)) ([1e64c77](1e64c77))
* Fix type mapping spark ([#4071](#4071)) ([3afa78e](3afa78e))
* Fix typo as the cli does not support shortcut-f option. ([#3954](#3954)) ([dd79dbb](dd79dbb))
* Get container host addresses from testcontainers ([#3946](#3946)) ([2cf1a0f](2cf1a0f))
* Handle ComplexFeastType to None comparison ([#3876](#3876)) ([fa8492d](fa8492d))
* Hashlib md5 errors in FIPS for python 3.9+ ([#4019](#4019)) ([6d9156b](6d9156b))
* Making the query_timeout variable as optional int because upstream is considered to be optional ([#4092](#4092)) ([fd5b620](fd5b620))
* Move gRPC dependencies to an extra ([#3900](#3900)) ([f93c5fd](f93c5fd))
* Prevent spamming pull busybox from dockerhub ([#3923](#3923)) ([7153cad](7153cad))
* Quickstart notebook example ([#3976](#3976)) ([b023aa5](b023aa5))
* Raise error when not able read of file source spark source ([#4005](#4005)) ([34cabfb](34cabfb))
* remove not use input parameter in spark source ([#3980](#3980)) ([7c90882](7c90882))
* Remove parentheses in pull_latest_from_table_or_query ([#4026](#4026)) ([dc4671e](dc4671e))
* Remove proto-plus imports ([#4044](#4044)) ([ad8f572](ad8f572))
* Remove unnecessary dependency on mysqlclient ([#3925](#3925)) ([f494f02](f494f02))
* Restore label check for all actions using pull_request_target ([#3978](#3978)) ([591ba4e](591ba4e))
* Revert mypy config ([#3952](#3952)) ([6b8e96c](6b8e96c))
* Rewrite Spark materialization engine to use mapInPandas ([#3936](#3936)) ([dbb59ba](dbb59ba))
* Run feature server w/o gunicorn on windows ([#4024](#4024)) ([584e9b1](584e9b1))
* SqlRegistry _apply_object update statement ([#4042](#4042)) ([ef62def](ef62def))
* Substrait ODFVs for online ([#4064](#4064)) ([26391b0](26391b0))
* Swap security label check on the PR title validation job to explicit permissions instead ([#3987](#3987)) ([f604af9](f604af9))
* Transformation server doesn't generate files from proto ([#3902](#3902)) ([d3a2a45](d3a2a45))
* Trino as an OfflineStore Access Denied when BasicAuthenticaion ([#3898](#3898)) ([49d2988](49d2988))
* Trying to import pyspark lazily to avoid the dependency on the library ([#4091](#4091)) ([a05cdbc](a05cdbc))
* Typo Correction in Feast UI Readme ([#3939](#3939)) ([c16e5af](c16e5af))
* Update actions/setup-python from v3 to v4 ([#4003](#4003)) ([ee4c4f1](ee4c4f1))
* Update typeguard version to >=4.0.0 ([#3837](#3837)) ([dd96150](dd96150))
* Upgrade sqlalchemy from 1.x to 2.x regarding PVE-2022-51668. ([#4065](#4065)) ([ec4c15c](ec4c15c))
* Use CopyFrom() instead of __deepycopy__() for creating a copy of protobuf object. ([#3999](#3999)) ([5561b30](5561b30))
* Using version args to install the correct feast version ([#3953](#3953)) ([b83a702](b83a702))
* Verify the existence of Registry tables in snowflake before calling CREATE sql command. Allow read-only user to call feast apply. ([#3851](#3851)) ([9a3590e](9a3590e))

### Features

* Add duckdb offline store ([#3981](#3981)) ([161547b](161547b))
* Add Entity df in format of a Spark Dataframe instead of just pd.DataFrame or string for SparkOfflineStore ([#3988](#3988)) ([43b2c28](43b2c28))
* Add gRPC Registry Server ([#3924](#3924)) ([373e624](373e624))
* Add local tests for s3 registry using minio ([#4029](#4029)) ([d82d1ec](d82d1ec))
* Add python bytes to array type conversion support proto ([#3874](#3874)) ([8688acd](8688acd))
* Add python client for remote registry server ([#3941](#3941)) ([42a7b81](42a7b81))
* Add Substrait-based ODFV transformation ([#3969](#3969)) ([9e58bd4](9e58bd4))
* Add support for arrays in snowflake ([#3769](#3769)) ([8d6bec8](8d6bec8))
* Added delete_table to redis online store ([#3857](#3857)) ([03dae13](03dae13))
* Adding support for Native Python feature transformations for ODFVs ([#4045](#4045)) ([73bc853](73bc853))
* Bumping requirements ([#4079](#4079)) ([1943056](1943056))
* Decouple transformation types from ODFVs ([#3949](#3949)) ([0a9fae8](0a9fae8))
* Dropping Python 3.8 from local integration tests and integration tests ([#3994](#3994)) ([817995c](817995c))
* Dropping python 3.8 requirements files from the project. ([#4021](#4021)) ([f09c612](f09c612))
* Dropping the support for python 3.8 version from feast ([#4010](#4010)) ([a0f7472](a0f7472))
* Dropping unit tests for Python 3.8 ([#3989](#3989)) ([60f24f9](60f24f9))
* Enable Arrow-based columnar data transfers  ([#3996](#3996)) ([d8d7567](d8d7567))
* Enable Vector database and retrieve_online_documents API ([#4061](#4061)) ([ec19036](ec19036))
* Kubernetes materialization engine written based on bytewax ([#4087](#4087)) ([7617bdb](7617bdb))
* Lint with ruff ([#4043](#4043)) ([7f1557b](7f1557b))
* Make arrow primary interchange for offline ODFV execution ([#4083](#4083)) ([9ed0a09](9ed0a09))
* Pandas v2 compatibility ([#3957](#3957)) ([64459ad](64459ad))
* Pull duckdb from contribs, add to CI ([#4059](#4059)) ([318a2b8](318a2b8))
* Refactor ODFV schema inference ([#4076](#4076)) ([c50a9ff](c50a9ff))
* Refactor registry caching logic into a separate class ([#3943](#3943)) ([924f944](924f944))
* Rename OnDemandTransformations to Transformations ([#4038](#4038)) ([9b98eaf](9b98eaf))
* Revert updating dependencies so that feast can be run on 3.11. ([#3968](#3968)) ([d3c68fb](d3c68fb)), closes [#3958](#3958)
* Rewrite ibis point-in-time-join w/o feast abstractions ([#4023](#4023)) ([3980e0c](3980e0c))
* Support s3gov schema by snowflake offline store during materialization ([#3891](#3891)) ([ea8ad17](ea8ad17))
* Update odfv test ([#4054](#4054)) ([afd52b8](afd52b8))
* Update pyproject.toml to use Python 3.9 as default ([#4011](#4011)) ([277b891](277b891))
* Update the Pydantic from v1 to v2 ([#3948](#3948)) ([ec11a7c](ec11a7c))
* Updating dependencies so that feast can be run on 3.11. ([#3958](#3958)) ([59639db](59639db))
* Updating protos to separate transformation ([#4018](#4018)) ([c58ef74](c58ef74))

### Reverts

* Reverting bumping requirements ([#4081](#4081)) ([1ba65b4](1ba65b4)), closes [#4079](#4079)
* Verify the existence of Registry tables in snowflake… ([#3907](#3907)) ([c0d358a](c0d358a)), closes [#3851](#3851)

Reviewers

tokoko

tokoko approved these changes

sudohainguyen

sudohainguyen approved these changes

whoahbot

Awaiting requested review from whoahbot

sfc-gh-madkins

Awaiting requested review from sfc-gh-madkins

niklasvm

Awaiting requested review from niklasvm

kevjumba

Awaiting requested review from kevjumba

younggyu-oh

Awaiting requested review from younggyu-oh

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

lgtm ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: faster linting with ruff

Uh oh!

There was an error while loading. Please reload this page.

5 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.