GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — spark materialize engine not working on yarn · Issue #3331 · feast-dev/feast Closed Labels DescriptionExpected Behaviorspark materialize engine working on yarn(not local[*]) Current Behaviornot working... since spark on yarn cannot share files without a common file system(like hdfs, cifs, ...) Steps to reproduceset spark.master config to yarn Specifications
Possible SolutionAS-IS feast/sdk/python/feast/infra/materialization/contrib/spark/spark_materialization_engine.py Lines 195 to 228 in a59c33a
TO-BE class _SparkSerializedArtifacts: """Class to assist with serializing unpicklable artifacts to the spark workers""" feature_view_proto: str repo_config_byte: str @classmethod def serialize(cls, feature_view, repo_config): # serialize to proto feature_view_proto = feature_view.to_proto().SerializeToString() # serialize repo_config to disk. Will be used to instantiate the online store repo_config_byte = dill.dumps(repo_config) return _SparkSerializedArtifacts( feature_view_proto=feature_view_proto, repo_config_byte=repo_config_byte ) def unserialize(self): # unserialize proto = FeatureViewProto() proto.ParseFromString(self.feature_view_proto) feature_view = FeatureView.from_proto(proto) # load repo_config = dill.load(self.repo_config_byte) provider = PassthroughProvider(repo_config) online_store = provider.online_store return feature_view, online_store, repo_config use dill.dumps and dill.loads instead of dill.dump and dill.load string types can be serialized and deserialized using pyspark seializer. So you don't need to pass the repo_config path. this solution works for me. Activity
MetadataMetadataAssigneesLabelsTypeProjectsMilestoneRelationshipsDevelopmentIssue actions | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |