Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R (deprecated), and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.
PySpark is the Python distribution of Spark.
Project home page: https://spark.apache.org/
Main documentation: https://spark.apache.org/docs/latest/
PySpark documentation: https://spark.apache.org/docs/latest/api/python/index.html
There are a few PySpark packages published by the Apache Spark project to PyPI:
pyspark: Classic PySpark, includes Spark assembly JARspyspark-connect: Classic PySpark with Spark Connect configured as the default, includespysparkpyspark-client: Pure Python Spark Connect client, no JARs or JRE needed
For more information, see the installation guide. If you're building PySpark from source, see Building Spark and pyproject.toml.
PySpark is not intended to be a complete distribution of Spark. It's meant for local development or for interacting with an existing cluster (be it Spark standalone, YARN, or Kubernetes). Using PySpark to set up a new standalone Spark cluster is not supported. To set up a standalone cluster please use the full distribution of Spark.
When using PySpark with an existing Spark cluster you must ensure that the major and minor version (e.g. 4.3.*) match or you may experience odd errors.