A tool to import swiftc and clang generated indexes into Xcode.

The index-import tool makes indexes portable. The ability to copy indexes into Xcode has a few possible uses:

1. Using a separate build system (Bazel, Buck, CMake, SwiftPM, etc)
2. Distributing a CI built index to developer machines
3. Sharing an index between two or more local checkouts

The common goal is to reduce or eliminate the time Xcode spends indexing.

For Xcode to be able to use an index, the file paths contained in the index must match the paths Xcode uses for lookup. This is the main feature of index-import, rewriting the paths inside the index files. This path remapping requires knowing input and output paths.

Path remapping is done with regex substitution. index-import accepts one or more -remap flags which are formatted as <regex>=<substitution>. See the examples below. Path remapping is conceptually similar to sed s/regex/substitution/. In all cases, the substitution will either be a path within the project, or a path within DerivedData.

The simplest example is to consider the case of two checkouts of the same project on the same machine. If one project has a built index, it can be imported into the other. To do this, two paths need to be remapped: the project directory and the build directory (DerivedData).

#!/bin/bash

build_dir1="/Users/me/Library/Developer/Xcode/DerivedData/MyApp-abc123"
build_dir2="/Users/me/Library/Developer/Xcode/DerivedData/MyApp-xyz789"

index-import \
    -remap "/Users/me/MyApp=/Users/me/MyApp2" \
    -remap "$build_dir1=$build_dir2" \
    "$build_dir1/Index/DataStore" \
    "$build_dir2/Index/DataStore"

A more complex example is importing an index from a Bazel built project. This example would be run as an Xcode "Run Script" build phase, which provides many environment variables, including: SRCROOT, CONFIGURATION_TEMP_DIR, ARCHS.

#!/bin/bash

set -euo pipefail

# Input: /Users/me/Library/Developer/Xcode/DerivedData/PROJECT-abc/Build/Products
# Output: /Users/me/Library/Developer/Xcode/DerivedData/PROJECT-abc
derived_data_root=$(dirname "$(dirname "$BUILD_DIR")")
readonly xcode_index_root="$derived_data_root/Index.noindex/DataStore"

# Captures: 1) module name
readonly bazel_swiftmodules="^/__build_bazel_rules_swift/swiftmodules/(.+).swiftmodule"
readonly xcode_swiftmodules="$BUILT_PRODUCTS_DIR/\$1.swiftmodule/$ARCHS.swiftmodule"

# Captures: 1) target name, 2) object name
readonly bazel_objects="^\./bazel-out/.+?/bin/.*?(?:[^/]+)/([^/]+?)_objs(?:/.*)*/(.+?)\.swift\.o$"
readonly xcode_objects="$CONFIGURATION_TEMP_DIR/\$1.build/Objects-normal/$ARCHS/\$2.o"

index-import
    -remap "$bazel_swiftmodules=$xcode_swiftmodules" \
    -remap "$bazel_objects=$xcode_objects" \
    -remap "^\.=$SRCROOT" \
    -remap "DEVELOPER_DIR=$DEVELOPER_DIR" \
    -incremental \
    @"$index_stores_file" \
    "$xcode_index_root"

Since Xcode 14 / Swift 5.7, clang and swiftc support remapping paths in index data using -ffile-prefix-map=foo=bar and -file-prefix-map foo=bar respectively. Using this makes it easy to generate a reproducible index that can be transferred between machines, and then remapped to local only paths using one of the examples above.

The build requires CMake 3.28 or newer, Ninja, clang, and swiftc. CMake downloads Swift's llvm fork and builds the libraries needed by index-import.

CC=clang CXX=clang++ cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --target check-index-import

The index consists of two types of files, Unit files and Record files. Both are LLVM Bitstream, a common binary format used by LLVM/Clang/Swift. Record files contain no paths and can be simply copied. Only Unit files contain paths, so only unit files need to be rewritten. A read/write API is available in the clangIndex library. index-import uses IndexUnitReader and IndexUnitWriter.

The best information on the swiftc and clang index store comes from these two resources:

• Adding Index‐While‐Building and Refactoring to Clang, 2017 LLVM Developers Meeting, by Alex Lorenz and Nathan Hawes
• Indexing While Building whitepaper