GeezerVison

The registry entry for one of four feature views seems to be corrupted. Opening a new connection succeeds for the first three features. Upon calling the fourth feature, the instantiation fails with the following error. This errors occurs regardless of the location from which we run store = FeatureStore(config=repo_config).

  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/bin/feast", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py", line [140](https://git.takeaway.com/scoober/ml-platform/poc/feast-poc/-/jobs/11952145#L140)4, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/cli.py", line 618, in materialize_incremental_command
    store = FeatureStore(repo_path=str(repo), fs_yaml_file=fs_yaml_file)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/usage.py", line 362, in wrapper
    raise exc.with_traceback(traceback)
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/usage.py", line 348, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/feature_store.py", line [160](https://git.takeaway.com/scoober/ml-platform/poc/feast-poc/-/jobs/11952145#L160), in __init__
    self._registry = SqlRegistry(registry_config, self.config.project, None)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py", line 201, in __init__
    self.cached_registry_proto = self.proto()
                                 ^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py", line 842, in proto
    objs: List[Any] = lister(project)  # type: ignore
                      ^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py", line 582, in list_feature_views
    return self._list_objects(
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py", line 1006, in _list_objects
    return [
           ^
  File "/root/.cache/pypoetry/virtualenvs/feast-poc-t0rhiBNj-py3.11/lib/python3.11/site-packages/feast/infra/registry/sql.py", line 1008, in <listcomp>
    proto_class.FromString(row[proto_field_name])
google.protobuf.message.DecodeError: Error parsing message

Existing connections fail to pull any feature view from the feature store with:

Traceback (most recent call last):
  File \"/usr/local/lib/python3.10/site-packages/mlserver/parallel/worker.py\", line 136, in _process_request    return_value = await method(
  File \"/app/./customer-eta/eta/customer_eta.py\", line 90, in predict
    _input = self._preprocessor.preprocess(payload)
  File \"/app/./customer-eta/eta/steps/preprocessing.py\", line 28, in preprocess
    feature_data = self.feature_store.get_online_features(
  File \"/usr/local/lib/python3.10/site-packages/feast/usage.py\", line 288, in wrapper
    return func(*args, **kwargs)
  File \"/usr/local/lib/python3.10/site-packages/feast/feature_store.py\", line 1583, in get_online_features
    return self._get_online_features(
  File \"/usr/local/lib/python3.10/site-packages/feast/feature_store.py\", line 1605, in _get_online_features
    _feature_refs = self._get_features(features, allow_cache=True)
  File \"/usr/local/lib/python3.10/site-packages/feast/feature_store.py\", line 524, in _get_features
    feature_service_from_registry = self.get_feature_service(
  File \"/usr/local/lib/python3.10/site-packages/feast/usage.py\", line 299, in wrapper
    raise exc.with_traceback(traceback)
  File \"/usr/local/lib/python3.10/site-packages/feast/usage.py\", line 288, in wrapper
    return func(*args, **kwargs)
  File \"/usr/local/lib/python3.10/site-packages/feast/feature_store.py\", line 386, in get_feature_service
    return self._registry.get_feature_service(name, self.project, allow_cache)
  File \"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\", line 383, in get_feature_service
    self._refresh_cached_registry_if_necessary()
  File \"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\", line 259, in _refresh_cached_registry_if_necessary
    self.refresh()
  File \"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\", line 238, in refresh
    self.cached_registry_proto = self.proto()
  File \"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\", line 828, in proto
    projects = self._get_all_projects()
  File \"/usr/local/lib/python3.10/site-packages/feast/infra/registry/sql.py\", line 1074, in _get_all_projects
    rows = conn.execute(stmt).all()
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1385, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py\", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1948, in _execute_context
    self._handle_dbapi_exception(
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 2129, in _handle_dbapi_exception
    util.raise_(
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py\", line 211, in raise_
    raise exception
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 1905, in _execute_context
    self.dialect.do_execute(
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py\", line 736, in do_execute
    cursor.execute(statement, parameters)
  File \"/usr/local/lib/python3.10/site-packages/pymysql/cursors.py\", line 153, in execute
    result = self._query(query)
  File \"/usr/local/lib/python3.10/site-packages/pymysql/cursors.py\", line 322, in _query
    conn.query(q)
  File \"/usr/local/lib/python3.10/site-packages/pymysql/connections.py\", line 557, in query
    self._execute_command(COMMAND.COM_QUERY, sql)
  File \"/usr/local/lib/python3.10/site-packages/pymysql/connections.py\", line 861, in _execute_command
    self._write_bytes(packet)
  File \"/usr/local/lib/python3.10/site-packages/pymysql/connections.py\", line 806, in _write_bytes
    raise err.OperationalError(
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, \"MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))\")
[SQL: SELECT entities.entity_name, entities.project_id, entities.last_updated_timestamp, entities.entity_proto 
FROM entities]
(Background on this error at: https://sqlalche.me/e/14/e3q8)", "level": "ERROR"

Steps to reproduce

We haven't been able to reproduce the error yet. We have a CI job running feast materialize-incremental every 5 mins, pushing the same demo data. That job started failing at some point without any changes in the data, dependencies or setup in general.

Authentication and manual querying of the registry and databases works fine.

Specifications

Version: feast == 0.31.1, python == 3.10.10
Platform: AWS, registry: RDS, online store: dynamodb, offline store: redshift
Subsystem:

Possible Solution

We dumped the feature store and recreated it running feast apply. This is fine for now since we are in an experimental setup. Once we move towards using feast in production, we would like to have an understanding of what is happening here and how to resolve this without dumping the whole feature store.

Activity

LeonKolyang
added
kind/bug

priority/p2
on Jul 31, 2023
stale commented on Mar 17, 2024
staleBot
on Mar 17, 2024 – with Stale
More actions
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
stale
added
wontfixThis will not be worked onThis will not be worked on
on Mar 17, 2024
stale
closed this as completed on Apr 26, 2025
ntkathole
reopened this on Dec 30, 2025
stale
removed
wontfixThis will not be worked onThis will not be worked on
on Dec 30, 2025
ntkathole commented on Dec 30, 2025
ntkathole
on Dec 30, 2025
Member
More actions
Implement a fix that:

Uses LONGBLOB for MySQL to prevent truncation

Adds optional pruning logic to limit interval history

Reacted by Francisco Javier Arceo

Metadata

Assignees

No one assigned

Labels

kind/bugpriority/p2

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Corrupted registry breaks usage of feature store #3703

Description

Expected Behavior

Current Behavior

Steps to reproduce

Specifications

Possible Solution

Activity

stale commented on Mar 17, 2024

ntkathole commented on Dec 30, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions