This repository captures the main Apache Spark website located at https://spark.apache.org. The programming docs under https://spark.apache.org/docs/ are in the main Spark repo, not here. They are built separately for each release of Spark and then copied to the website under the docs/ directory.

To contribute changes, build and test the site locally, then submit a pull request with your changes. Unless you are working through the release guide, you only need to commit changes to source files (typically Markdown). A GitHub Actions workflow will generate the corresponding HTML under site/ and push it for you.

Building the site requires Ruby 3, Jekyll, and Rouge.

gem install bundler -v 2.4.22
bundle install
bundle exec jekyll serve

The most reliable way to ensure a compatible environment is to use the official Docker build image from the Apache Spark repository.

If you haven't already, clone the Apache Spark repository. Navigate to the Spark root directory and run the following command to create the builder image:

docker build \
  --tag docs-builder:latest \
  --file dev/spark-test-image/docs/Dockerfile \
  dev/spark-test-image-util/docs/

Once the image is built, navigate to the spark-website root directory, run the script which processes the Markdown files in the Docker container.

.dev/build-docs.sh

The website is deployed automatically by ASF Infra. The deployment configuration is tracked by .asf.yaml and is documented here.

One deployment detail that appears to be critical is the presence of the content symlink to site/. Even though ASF Infra is aware of Jekyll, we perhaps do not have the exact setup required for them to automatically use our Jekyll config to understand where the site content lives. Without the content symlink, the website will just show a plain directory listing of the files in this repo.