The fe-filesystem module provides a pluggable filesystem abstraction for the Doris FE (Frontend). It decouples the query engine (fe-core) from concrete storage backends (S3, HDFS, Azure Blob, etc.) at both the Maven dependency level and the runtime classpath level.

Each storage backend is a self-contained plugin that is discovered and loaded at FE startup—no modification to fe-core is required to add a new storage backend.

fe-filesystem/                        (aggregator POM — no Java code)
│
├── fe-filesystem-api/                [API]  Core abstractions (FileSystem, Location, …)
├── fe-filesystem-spi/                [SPI]  Provider interface + object-storage contracts
│
├── fe-filesystem-s3/                 [IMPL] AWS S3 / S3-compatible (MinIO, …)
├── fe-filesystem-oss/                [IMPL] Alibaba Cloud OSS  (delegates to S3)
├── fe-filesystem-cos/                [IMPL] Tencent Cloud COS  (delegates to S3)
├── fe-filesystem-obs/                [IMPL] Huawei Cloud OBS   (delegates to S3)
├── fe-filesystem-azure/              [IMPL] Azure Blob Storage
├── fe-filesystem-hdfs/               [IMPL] HDFS / ViewFS / OFS / JFS
├── fe-filesystem-local/              [IMPL] Local filesystem   (testing only)
└── fe-filesystem-broker/             [IMPL] Doris Broker process (Thrift RPC)

• API — Pure-JDK interfaces and value types consumed by fe-core (zero third-party dependencies). Defines FileSystem, Location, FileEntry, DorisInputFile, DorisOutputFile, etc.
• SPI — The FileSystemProvider interface (extends PluginFactory from fe-extension-spi) plus the object-storage layer (ObjStorage, ObjFileSystem). Also compiled into fe-core.
• IMPL — Concrete backends. Each one depends on fe-filesystem-spi (and transitively on fe-filesystem-api). S3-delegating backends (OSS, COS, OBS) also depend on fe-filesystem-s3 to reuse S3FileSystem. They must not depend on fe-core, fe-common, or fe-catalog.

At FE startup, Env.initFileSystemPluginManager() creates a FileSystemPluginManager and loads providers in two phases:

1. ServiceLoader scan — Discovers FileSystemProvider implementations already on the classpath (built-in providers, test overrides).
2. Directory plugin scan — Uses DirectoryPluginRuntimeManager to scan the directory configured by Config.filesystem_plugin_root (default: ${DORIS_HOME}/plugins/filesystem). Each direct child directory is treated as an unpacked plugin directory. The runtime manager resolves pluginDir/*.jar (root-level jars, scanned for ServiceLoader registration) and pluginDir/lib/*.jar (dependency jars, available for class loading only).

Classpath providers have higher priority than directory-loaded providers.

When fe-core needs a filesystem, it calls FileSystemFactory.getFileSystem(properties):

Map<String, String> properties
        │
        ▼
FileSystemPluginManager.createFileSystem(properties)
        │
        │  for each registered FileSystemProvider:
        │    1. provider.supports(properties)  ← cheap, no I/O
        │    2. if true → provider.create(properties) → return FileSystem
        │
        ▼
First matching provider wins

Each provider's supports() method examines property keys to decide if it can handle the request. For example:

Each implementation module uses maven-assembly-plugin to produce a zip build artifact. The zip must be unpacked before deployment. At runtime, DirectoryPluginRuntimeManager expects each plugin to be an unpacked directory under filesystem_plugin_root:

${DORIS_HOME}/plugins/filesystem/
├── s3/                                   ← one directory per plugin
│   ├── doris-fe-filesystem-s3.jar        ← plugin jar at root (scanned for ServiceLoader)
│   └── lib/
│       ├── aws-sdk-*.jar                 ← third-party dependencies
│       └── ...
├── hdfs/
│   ├── doris-fe-filesystem-hdfs.jar
│   └── lib/...
└── ...

The Maven build produces a zip with the same layout. To deploy, unzip it into the appropriate subdirectory (e.g., unzip doris-fe-filesystem-s3.zip -d plugins/filesystem/s3/). Dropping the raw .zip file into the directory will not work — DirectoryPluginRuntimeManager resolves pluginDir/*.jar and pluginDir/lib/*.jar from unpacked directories only.

Jars that are already on the fe-core classpath (fe-filesystem-api, fe-filesystem-spi, fe-extension-spi) are excluded from the zip to avoid duplication.

At runtime, DirectoryPluginRuntimeManager creates an isolated ClassLoader per plugin. A ClassLoadingPolicy ensures that shared framework classes (org.apache.doris.filesystem.*, software.amazon.awssdk.*, org.apache.hadoop.*) are loaded parent-first to avoid cross- ClassLoader cast failures.

The credential path has three layers with one-way dependencies:

GCS plugin -> filesystem API identity description <- fe-storage-protocol -> fe-grpc / fe-thrift
                                  ^                          ^
                                  |                          |
                                fe-core ---------------------+

The GCS plugin owns user property aliases, validation and authentication. It returns an immutable GcpCredentialSpec through ObjectStorageAuthentication. The shared API contains only data types; it imports no SDK or generated protocol classes. Resolved credentials and partial ALTER updates are separate values. An absent field stays absent, while an explicitly empty impersonation account clears that setting. HMAC and anonymous modes do not contain native credentials.

The host-side fe-storage-protocol module converts those descriptions directly to Cloud protobuf and BE Thrift objects. It owns wire-schema mapping and depends on the filesystem API and generated protocol modules, never on filesystem plugin implementations. There is no intermediate serialization, reflection-based field mapping, or plugin-supplied wire codec. fe-core invokes the generic translator and attaches the result to its request envelopes without interpreting GCP fields.

For SHOW CREATE and Cloud stages, the protocol adapter decodes a description and the owning plugin's credentialToProperties() restores user property names. Unknown wire providers, mismatched description types, and missing or ambiguous property owners fail explicitly.

Adding a native wire variant requires a data description, a mapping in fe-storage-protocol, and matching BE/Cloud support. Filesystem plugins cannot independently extend the receiver's wire schema. The filesystem plugin API is 6.0 and connector plugin API is 15.0 because shared authentication types changed. Plugins must be rebuilt against the matching host APIs.

Native OAuth accepts only the Google Storage service-base HTTPS endpoints recognized by GcsEndpoint (port 443, without a bucket, path, or query). Arbitrary S3-compatible endpoints and raw fs.gs.auth.* overrides are not supported in native mode. HMAC authentication retains its existing endpoint behavior. BE error-log downloads use IAM signBlob; FE presigned uploads remain unsupported with native OAuth.

The binding registry preserves its existing provider order, explicit fs.<x>.support flags, multiple explicit bindings, and default HDFS binding. Routing uses the existing supports(), supportsExplicit() and supportsGuess() methods; there is no global validation or fallback hook.

GCS recognizes its provider names, native credential keys, gs:// locations and Google endpoints. S3 and MinIO exclude these GCS guesses in their own matching predicates. Explicit provider=S3 or fs.s3.support=true retains S3 authentication, including at Google endpoints. This does not change the established HDFS, OSS, COS, OBS or Azure selection order.

The S3-compatible plugins share pure GCS property recognition through fe-filesystem-gcs-common, which has no production dependencies and loads no Google credentials or SDKs. Credential validation runs when the selected provider binds its properties, including forced provider binding. S3, MinIO and Ozone reject native GCP keys instead of silently using their AWS/HMAC credential paths.

Filesystem API 6.0 retains the native identity/property hooks described above, but removes the two routing hooks from the earlier feature draft. Plugins must be rebuilt against the matching host API.

Several cloud providers (OSS, COS, OBS) are S3-compatible. Instead of duplicating the S3 implementation, they follow a delegation pattern:

1. Translate cloud-native property keys to S3-compatible keys.
2. Extend ObjStorage to override cloud-specific operations (pre-signed URLs, STS tokens).
3. Delegate core I/O to S3FileSystem (from fe-filesystem-s3).

CosFileSystemProvider
  └─→ creates S3FileSystem(CosObjStorage)
        │                     │
        │  FileSystem ops     │  Overrides getPresignedUrl(), getStsToken()
        └─────────────────────┘

┌──────────────────────────────────────────────────────────┐
│                        fe-core                           │
│  Uses: FileSystem, Location, FileEntry (from API)        │
│  Depends on: fe-filesystem-api, fe-filesystem-spi        │
│  Loads plugins from: ${DORIS_HOME}/plugins/filesystem    │
└────────────┬────────────────────────┬────────────────────┘
             │ compile-time           │ runtime (plugin)
             ▼                        ▼
  ┌──────────────────┐    ┌─────────────────────────┐
  │ fe-filesystem-api│    │ fe-filesystem-s3 (zip)  │
  │ fe-filesystem-spi│    │ fe-filesystem-hdfs (zip)│
  │ (compiled in)    │    │ fe-filesystem-xxx (zip) │
  └──────────────────┘    └─────────────────────────┘

• fe-core — Compile-time dependency on fe-filesystem-api and fe-filesystem-spi. Implementation modules are not Maven dependencies of fe-core; they are loaded at runtime from the plugin directory.
• fe-extension-spi — Provides the PluginFactory / Plugin interfaces and the DirectoryPluginRuntimeManager class used by the plugin loading infrastructure.
• fe-filesystem-local — Used in fe-core unit tests (test-scope dependency) so that tests can exercise FileSystem operations without cloud credentials.

Follow these steps to add support for a new storage backend (e.g., Google Cloud Storage):

Create a new directory fe-filesystem/fe-filesystem-gcs/ with this structure:

fe-filesystem-gcs/
├── pom.xml
└── src/
    ├── main/
    │   ├── assembly/
    │   │   └── plugin-zip.xml
    │   ├── java/org/apache/doris/filesystem/gcs/
    │   │   ├── GcsFileSystemProvider.java
    │   │   ├── GcsFileSystem.java
    │   │   └── GcsObjStorage.java          (if object-storage based)
    │   └── resources/META-INF/services/
    │       └── org.apache.doris.filesystem.spi.FileSystemProvider
    └── test/
        └── java/org/apache/doris/filesystem/gcs/
            └── ...Test.java

<project>
    <parent>
        <groupId>org.apache.doris</groupId>
        <artifactId>fe-filesystem</artifactId>
        <version>${revision}</version>
    </parent>

    <artifactId>fe-filesystem-gcs</artifactId>
    <packaging>jar</packaging>
    <name>Doris FE Filesystem - GCS</name>

    <dependencies>
        <!-- REQUIRED: SPI contract (transitively includes API) -->
        <dependency>
            <groupId>org.apache.doris</groupId>
            <artifactId>fe-filesystem-spi</artifactId>
            <version>${revision}</version>
        </dependency>

        <!-- Cloud SDK -->
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-storage</artifactId>
            <version>...</version>
        </dependency>

        <!-- Test -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <finalName>doris-fe-filesystem-gcs</finalName>
        <plugins>
            <!-- Assembly: produces the plugin zip -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <descriptors>
                        <descriptor>src/main/assembly/plugin-zip.xml</descriptor>
                    </descriptors>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals><goal>single</goal></goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Important: Do not add dependencies on fe-core, fe-common, or fe-catalog.

Copy plugin-zip.xml from an existing module (e.g., fe-filesystem-s3/src/main/assembly/). The key points:

• Place the plugin jar at the root of the zip (for ServiceLoader discovery).
• Place all runtime dependencies in lib/.
• Exclude fe-filesystem-api, fe-filesystem-spi, and fe-extension-spi (they are already on the fe-core classpath).

package org.apache.doris.filesystem.gcs;

import org.apache.doris.filesystem.FileSystem;
import org.apache.doris.filesystem.spi.FileSystemProvider;
import java.io.IOException;
import java.util.Map;

public class GcsFileSystemProvider implements FileSystemProvider {

    // Public no-arg constructor — required by ServiceLoader
    public GcsFileSystemProvider() {}

    @Override
    public boolean supports(Map<String, String> properties) {
        // Must be cheap (no network calls) and deterministic.
        String type = properties.get("_STORAGE_TYPE_");
        return "GCS".equalsIgnoreCase(type);
    }

    @Override
    public FileSystem create(Map<String, String> properties) throws IOException {
        return new GcsFileSystem(properties);
    }

    @Override
    public String name() {
        return "GCS";
    }
}

You have two choices:

• Object-storage backend — Extend ObjFileSystem (from SPI) and implement ObjStorage<C>. This gives you exists(), close(), and cloud-specific delegates for free.
• Custom backend — Implement FileSystem directly (like DFSFileSystem for HDFS).

At minimum, you must implement:

Create the file:

src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider

With content:

org.apache.doris.filesystem.gcs.GcsFileSystemProvider

Add your module to fe-filesystem/pom.xml:

<modules>
    ...
    <module>fe-filesystem-gcs</module>
</modules>

• Unit tests — Test your FileSystem and ObjStorage implementations with mocked cloud clients. Place in src/test/java/.
• Environment tests — Tests that require real cloud credentials should be tagged with @Tag("environment"). They are excluded by default and can be enabled with -Dtest.excludedGroups=none.

# Build the new module (must go through the reactor so sibling SNAPSHOTs resolve)
cd fe
mvn package -pl fe-filesystem/fe-filesystem-gcs --also-make -DskipTests

# The build produces a zip at:
#   fe-filesystem/fe-filesystem-gcs/target/doris-fe-filesystem-gcs.zip
# Deploy by unpacking into the plugin directory:
mkdir -p ${DORIS_HOME}/plugins/filesystem/gcs
unzip fe-filesystem/fe-filesystem-gcs/target/doris-fe-filesystem-gcs.zip \
  -d ${DORIS_HOME}/plugins/filesystem/gcs/

# The unpacked layout should be:
#   plugins/filesystem/gcs/
#   ├── doris-fe-filesystem-gcs.jar
#   └── lib/
#       └── *.jar
#
# NOTE: Do NOT drop the .zip file directly — it must be unpacked.

• Module depends only on fe-filesystem-spi (not fe-core/fe-common/fe-catalog)
• FileSystemProvider has a public no-arg constructor
• supports() is cheap — no network calls
• META-INF/services file is present and correct
• Assembly descriptor excludes fe-filesystem-api, fe-filesystem-spi, fe-extension-spi
• Module is listed in fe-filesystem/pom.xml <modules>
• Unit tests pass; environment tests are tagged @Tag("environment")