@sudohainguyen why are you removing bytewax? Is there any issue or RFC explaining reasons behind this decision?

@sudohainguyen why are you removing bytewax? Is there any issue or RFC explaining reasons behind this decision?

There are 2 main reasons:

• the current version of bytewax we're using is blocking us from supporting python 3.11
• we're not leveraging all features of bytewax: if you look at bytewax engine implementation, it is simply a kubernetes job with multiple of parallel pods reading files separately and writing data to the online store, which is powered by pyarrow zero-copy mechanism, so we decided to re-write the engine called k8s engine which remains the logic but without bytewax interface.

as a result, we see it is safe to remove bytewax 😄 hope this helps

@breno-costa see #4087 for a replacement that Harry pointed to. The discussions around the topic started out here #3993 and then continued on slack and community calls, so unfortunately no single place to see it all.

@tokoko any chance you can compile a quick doc with the decision just so we can add it as a note somewhere? Even a paragraph in the community notes would probably be sufficient.

@franciscojavierarceo let me handle it

@sudohainguyen thank you!

@sudohainguyen @tokoko Thanks for clarifying that point. We wanted to try a more scalable materialization engine and bytewax would be our first candidate. Let us try this new k8s implementation.