Description

potiuk

opened on Oct 26, 2022

Issue body actions

I know it is eaarly (Python 3.11 has just been released yesterday) but we are hoping in Apache Airflow to a much faster cycle of adding new Python releases - especially that Pyhon 3.11 introduces huge performance improvements (25% is the average number claimed) due to a very focused effort to increase single-threaded Python performance (Specialized interpreter being the core of it but also many other improvements) without actually changing any of the Python code.

The databricks-sql-python is one of the dependencies of Airlfow that need to support newer Python version so this issue is here to make you aware that we are eaager (also to help if needed) to make it faster, possibly by talking to dependencies of Beam as well (who are oftten also direct dependencies of Airflow) and help them as well. I perfectly understand we need to bubble up support so that your dependencies support them first.

Nice summary of Py3.11 support is here: https://pyreadiness.org/3.11/ - it's not very green obviously, but I hope it gets greener soon.

I'd appreciate if someone in the Databricks team attempted to migrate and have a PR running (and failing possibly until all prerequisite are met so that we can also track the progress and possibly help each other to solve any problems. I know PyArrow is blocking for now, but this is going to be solved soon lilely with apache/arrow#14499 and just opening a failing PR with it might be a good start.

I just opened such PR in Apache Airflow yesterday and plan to keep it open until it gets green :)

apache/airflow#27264

I think it would be fantastic if we could as the open source community migrate to the new Python much faster.

Looking forward to cooperation on that one :)

Reacted by Jesse and Jeonghyun An

Activity

1. susodapop commented on Oct 26, 2022

   

   susodapop

   on Oct 26, 2022

   Contributor

   More actions

   Thanks @potiuk! I opened a pull request that will run our unit tests against Python 3.11. Assuming everything passes we can bump a new release that can incorporate this ASAP.

   Reacted by Jarek Potiuk and David Black

   Reacted by Jarek Potiuk

   Reacted by Jarek Potiuk
2. potiuk commented on Oct 26, 2022

   

   potiuk

   on Oct 26, 2022

   Author

   More actions

   Cool!
3. susodapop commented on Oct 26, 2022

   

   susodapop

   on Oct 26, 2022

   Contributor

   More actions

   @potiuk my PR is blocked until pyarrow has wheels for 3.11. I see you're working with them on making this work. I'll be keeping an eye out. Once pyarrow has wheels for 3.11 I believe we'll be unblocked and ready to support Py3.11

   Thanks again for bringing this to our attention and please feel free to ping me if need be.

   Reacted by Jarek Potiuk and David Black
4. potiuk

   added 4 commits that reference this issue on Oct 27, 2022

   Attempt to add Python 3.11 support

   bdd765d

   Attempt to add Python 3.11 support

   71cbff2

   Attempt to add Python 3.11 support

   0463899
5. susodapop commented on Nov 2, 2022

   

   susodapop

   on Nov 2, 2022

   Contributor

   More actions

   Hey @potiuk I again attempted the Python 3.11 but it still fails for the reason that the GH action runner can't build wheels for Python 3.11 and a pre-made wheel isn't available for download. Do you think a 3.11 wheel is forthcoming for pyarrow? Or should we attempt to make the GH runner capable of building pyarrow from scratch?
6. susodapop commented on Nov 2, 2022

   

   susodapop

   on Nov 2, 2022

   Contributor

   More actions

   Actually, I can answer my own question: I think we can wait for the wheels as the process for the pre-made wheels is detailed in this thread: apache/arrow#14572
7. potiuk commented on Nov 3, 2022

   

   potiuk

   on Nov 3, 2022

   Author

   More actions

   Yep. I am subscribed to the thread - they are working on it and making progress. I think half of the data science world holds it's breath now :)

   Reacted by Jesse, Zongwei Chen, Benoit C. Sirois, Robin Landström and David Black
8. potiuk commented on Nov 23, 2022

   

   potiuk

   on Nov 23, 2022

   Author

   More actions

   Pyarrow just released 10.0.1 version with pre-compiled binaries for 3.11. I think you should be good to go @susodapop
9. susodapop

   closed this as completedin #60on Nov 28, 2022

Issue actions