GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Add Python 3.11 support by potiuk · Pull Request #27264 · apache/airflow

Conversation

potiuk commented Oct 25, 2022 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

Python 3.11 has been released as scheduled on October 25, 2022 and
this is the first attempt to see how far Airflow (mostly dependencies)
are from being ready to officially support 3.11.

The fio

Pyarrow dependency: ARROW-17487: [Python][Packaging][CI] Add support for Python 3.11 arrow#14499
Google Provider: Support for Python 3.11 for Google Provider (upgrading all dependencies) #27292
Google Provider - bigquery: Python 3.11 support googleapis/python-bigquery#1386
Databricks Provider:
Support for Python 3.11 databricks/databricks-sql-python#59
Papermill Provider: Support for Python 3.11 nteract/papermill#700 (it does not seem blocking the papermil provider from being released. We will release it wiht the hope the functionality we use and test works.
Azure Provider: Python 3.11 issues for uaqmp Azure/azure-uamqp-python#334
and Support for Python 3.11 Azure/azure-sdk-for-python#27066
Apache Beam Provider: Support for Python 3.11 beam#23848
Snowflake Provider: https://github.com/snowflakedb/snowflake-connector-python/pull/1349/files
JDBC Provider: Fixes for 3.11 jpype-project/jpype#1087
Hive Provider: Support Python 3.11 cloudera/python-sasl#30 (mitigated by excluding the provider from 3.11)
Plyvel: dependency: Support for Python 3.11 in binary wheels wbolster/plyvel#148

We might decide to release Airflow in 3.11 with those providers
disabled in case they are lagging behind eventually, but for the
moment we want to work with all the projects in concert to be
able to release all providers (Google Provider requires quite
a lot of work and likely Google Team stepping up and community helping
with migration to latest Goofle cloud libraries)

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hterik, MicaelJarniac, Jianxuan-Li, jacopofar, ndharasz, ratulotron, attilaszombati, pankajkoti, Jorricks, mis98zb, and bomwo reacted with thumbs up emoji

gmsantos, dstandish, pankajastro, VictorGob, bouldeterre, MicaelJarniac, Jianxuan-Li, ratulotron, attilaszombati, pankajkoti, and 3 more reacted with rocket emoji

All reactions

👍 11 reactions
🚀 13 reactions

boring-cyborg Bot added area:dev-tools area:production-image

Production image improvements and fixes

kind:documentation labels

Oct 25, 2022

potiuk requested review from BasPH, Fokko, XD-DENG, ashb, dstandish, eladkal, ephraimbuddy, feluelle, ferruzzi, jedcunningham, kaxil, mik-laj, o-nikolas, uranusjr and xinbinhuang

October 25, 2022 15:09

potiuk commented Oct 25, 2022

Copy link

Copy Markdown

Member Author

There you go.

gmsantos, technopahadi, ferruzzi, Taragolis, tienhung2812, hussein-awala, and pankajkoti reacted with heart emoji

All reactions

❤️ 7 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

potiuk force-pushed the python-3-11 branch from b06fb75 to c2dd81c Compare

October 25, 2022 15:34

mik-laj reviewed

Oct 25, 2022

View reviewed changes

Comment thread

CI.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

feluelle commented Oct 26, 2022

Copy link

Copy Markdown

Member

Really cool. Unfortunately pyarrow is currently not compatible with 3.11 😐

ldacey reacted with confused emoji

All reactions

😕 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

feluelle commented Oct 26, 2022

Copy link

Copy Markdown

Member

Related apache/arrow#14499

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Oct 26, 2022

Support for Python 3.11 apache/beam#23848

Closed

ARROW-17487: [Python][Packaging][CI] Add support for Python 3.11 apache/arrow#14499

Merged

Add Python 3.11 wheels to the CI builds. snowflakedb/snowflake-connector-python#1294

Closed

Support for Python 3.11 databricks/databricks-sql-python#59

Closed

Support for Python 3.11 nteract/papermill#700

Closed

Python 3.11 support googleapis/python-bigquery#1386

Closed

Support for Python 3.11 for Google Provider (upgrading all dependencies) #27292

Closed

potiuk force-pushed the python-3-11 branch 2 times, most recently from 2371ed5 to 0642e35 Compare

May 21, 2023 14:02

potiuk commented May 21, 2023

Copy link

Copy Markdown

Member Author

All is ready.

All tests are passing, everything looks good.

Now, the only thing we need is, we need to wait for the #31322 to be released. Currently it is impossible to find the right "provider" packages to install by airflow and generate constraints for "pypi" installation - that's why I had to cancel the constraint generation that run for ~ 1hr - because some of our providers (transitively) are not compatible with Python 3.11.

Once the providers get released, they will not yet have the Python 3.11 supported' trove classifier set - but they will be regardless discovered as compatible and used by pip` to generate the constraints.

So Python 3.11 support is now in the hands of those who test and vote on providers (cc: @eladkal)

millin and pankajkoti reacted with rocket emoji

All reactions

🚀 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

uranusjr reviewed

May 22, 2023

View reviewed changes

Comment thread

tests/providers/apache/hive/hooks/test_hive.py

Comment on lines +22 to +28

		from airflow import PY311

		if PY311:
		pytest.skip(
		"The tests are skipped because Apache Hive provider is not supported on Python 3.11",
		allow_module_level=True,
		)

uranusjr May 22, 2023

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this use pytestmark = pytest.mark.skipif(...) instead?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

potiuk commented

May 22, 2023

View reviewed changes

Comment thread

tests/models/test_param.py Outdated

potiuk May 22, 2023

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIce

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

potiuk force-pushed the python-3-11 branch 2 times, most recently from 41dceba to 2b75487 Compare

May 22, 2023 19:39

potiuk and others added 8 commits

May 22, 2023 21:50


          Attempt to add Python 3.11 support

e30ef21

Python 3.11 has been released as scheduled on October 25, 2022 and
this is the first attempt to see how far Airflow (mostly dependencies)
are from being ready to officially support 3.11.


          fixup! Attempt to add Python 3.11 support

41bc51c


          fixup! fixup! Attempt to add Python 3.11 support

82e45be


          fixup! fixup! fixup! Attempt to add Python 3.11 support

68018b4


          Cosmetic

a3162d2


          Better looping code in setup.py

4178cda


          Use parametrize to avoid if else in tests

37ecfe6


          fixup! Use parametrize to avoid if else in tests

potiuk force-pushed the python-3-11 branch from 2b75487 to 9473115 Compare

May 22, 2023 19:50

potiuk commented May 22, 2023

Copy link

Copy Markdown

Member Author

FINALLY GREEN... MERGING

noamcohen97, hterik, pankajkoti, shubham22, and tim-x-y-z reacted with hooray emoji

All reactions

🎉 5 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

potiuk merged commit c5597d1 into main

May 22, 2023

uranusjr deleted the python-3-11 branch

May 23, 2023 00:08

eladkal reviewed

May 23, 2023

View reviewed changes

Comment thread

README.md

		\| MSSQL \| 2017(\), 2019(\) \| 2017(\), 2019(\) \|
		\| \| Main version (dev) \| Stable version (2.6.1) \|
		\|------------\|------------------------------\|---------------------------\|
		\| Python \| 3.7, 3.8, 3.9, 3.10, 3.11 \| 3.7, 3.8, 3.9, 3.10, 3.11 \|

eladkal May 23, 2023

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we supporting Python 3.11 in 2.6.1?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pankajkoti reacted with thumbs up emoji

All reactions

👍 1 reaction

potiuk May 23, 2023

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope :). Good catch

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

potiuk May 23, 2023

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#31468

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

potiuk added this to the Airflow 2.6.2 milestone

May 23, 2023

pankajkoti mentioned this pull request

May 29, 2023

Remove Python 3.7 support #30963

Merged

eladkal added the type:improvement Changelog: Improvements label

Jun 8, 2023

eladkal pushed a commit that referenced this pull request

Jun 8, 2023


          Add Python 3.11 support (#27264)

bd7cd41

Add Python 3.11 support

Python 3.11 has been released as scheduled on October 25, 2022 and
finally, after all dependencies got upgraded - we can support it.

---------

Co-authored-by: Tzu-ping Chung <[email protected]>
(cherry picked from commit c5597d1)

eladkal mentioned this pull request

Jun 13, 2023

Status of testing of Apache Airflow 2.6.2rc2 #31867

Closed

40 of 62 tasks

eladkal mentioned this pull request

Jun 20, 2023

Status of testing Providers that were prepared on June 20, 2023 #32030

Closed

50 of 86 tasks

potiuk mentioned this pull request

Jul 10, 2023

Revert the Python 3.11 exclusion for date isoformat #32490

Merged

potiuk added a commit to potiuk/airflow that referenced this pull request

Jul 10, 2023


          Revert the Python 3.11 exclusion for date isoformat

cb8f643

The official Python image for 3.11 had a brief period when the
date `20120503` has been an acceptable date, but it is not
accepted as of 3.11.4 release on 5th of July.

This PR reverts  the changes in tests implemented in apache#27264 to
accomodate for it and brings back the date as error case.

potiuk added a commit that referenced this pull request

Jul 10, 2023


          Revert the Python 3.11 exclusion for date isoformat (#32490)

60eccf0

The official Python image for 3.11 had a brief period when the
date `20120503` has been an acceptable date, but it is not
accepted as of 3.11.4 release on 5th of July.

This PR reverts  the changes in tests implemented in #27264 to
accomodate for it and brings back the date as error case.

Reviewers

uranusjr

uranusjr left review comments

pankajkoti

pankajkoti left review comments

mik-laj

mik-laj left review comments

eladkal

eladkal approved these changes

pierrejeambrun

pierrejeambrun approved these changes

dstandish

dstandish approved these changes

ashb

Awaiting requested review from ashb

ashb is a code owner

BasPH

Awaiting requested review from BasPH

ephraimbuddy

Awaiting requested review from ephraimbuddy

ephraimbuddy is a code owner

feluelle

Awaiting requested review from feluelle

ferruzzi

Awaiting requested review from ferruzzi

Fokko

Awaiting requested review from Fokko

jedcunningham

Awaiting requested review from jedcunningham

jedcunningham is a code owner

kaxil

Awaiting requested review from kaxil

o-nikolas

Awaiting requested review from o-nikolas

XD-DENG

Awaiting requested review from XD-DENG

xinbinhuang

Awaiting requested review from xinbinhuang

mobuchowski

Awaiting requested review from mobuchowski

hussein-awala

Awaiting requested review from hussein-awala

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

area:dev-tools area:production-image

Production image improvements and fixes

full tests needed

We need to run full set of tests for this PR to merge

kind:documentation pinned

Protect from Stalebot auto closing

type:improvement

Changelog: Improvements

Projects

None yet

Milestone

Airflow 2.6.2

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

13 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

potiuk commented Oct 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Oct 25, 2022

Uh oh!

Uh oh!

feluelle commented Oct 26, 2022

Uh oh!

feluelle commented Oct 26, 2022

Uh oh!

potiuk commented May 21, 2023

Uh oh!

uranusjr May 22, 2023

Choose a reason for hiding this comment

Uh oh!

potiuk May 22, 2023

Choose a reason for hiding this comment

Uh oh!

potiuk commented May 22, 2023

Uh oh!

eladkal May 23, 2023

Choose a reason for hiding this comment

Uh oh!

potiuk May 23, 2023

Choose a reason for hiding this comment

Uh oh!

potiuk May 23, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

potiuk commented Oct 25, 2022 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.