Towards Decentralized Intelligence Evolution

Train LLM agents collaboratively across decentralized clients, without sharing local data.

Abstract

Training AI agents powered by Large Language Models (LLMs) typically requires centralized access to user data, raising privacy and scalability concerns. We explore FedAgent (Federated Agent Reinforcement Learning), a decentralized reinforcement learning paradigm that collaboratively trains LLM agents across distributed clients without sharing local data. The central reliability question is: is FedAgent effective under uniform client distribution, and more importantly, is it robust to client heterogeneity? For the former, we provide the first empirical evidence that FedAgent matches Centralized Agent Training and outperforms Local Agent Training. For the latter, we first formalize Agent Heterogeneity at two structurally distinct levels: task-level (what clients ask the agent to do) and environment-level (the dynamics in which the agent acts), anchored on the Input-Dynamics Asymmetry of task-augmented Markov Decision Processes (MDPs), referring to the architectural fact that tasks enter the policy through its input channel, while environments do not. Then, we theoretically establish an Asymmetric Robustness Mechanism: FedAgent is robust to task-level heterogeneity but non-robust to environment-level heterogeneity. We further identify three sufficient conditions under which FedAgent recovers robustness despite environment-level heterogeneity, and illustrate four possible training-curve patterns. On real-world agent benchmarks WebShop and ALFWorld, we empirically verify that FedAgent remains robust under extreme task-level heterogeneities and traces a stable-degrade-collapse spectrum under environment-level heterogeneities.

Input-Dynamics Asymmetry

One architectural fact about LLM agents drives the entire story, and it cuts two ways.

What the policy can observe (the task) is absorbed; what it cannot directly see (the environment) is where federation can break.

The FedAgent Algorithm

FedAgent follows the classic FedAvg skeleton of sample, broadcast, local update, and average, but replaces supervised SGD with policy-gradient RL on LLM-induced policies. Each client optimizes on its own task-augmented MDP; only model parameters ever leave the device.

Two-Level Agent Heterogeneity

The asymmetry organizes how clients differ along two structurally distinct levels: what the policy observes (the task) and what it cannot (the environment).

Observable Task-level: what clients ask

Three operationally separable sub-types, each isolated by a single dispersion knob (D1 to D4: one knob each, other measures and all first-order means held fixed). Distributions below are simulated from the partition algorithms across 100 clients.

Implicit Environment-level: which world the agent acts in

A transition kernel couples action to next observation through four stages; perturbing any stage yields a distinct form of environment heterogeneity. Five WebShop variants span the spectrum.

Asymmetric Robustness Mechanism

Why is FedAgent robust to task heterogeneity but fragile to environment heterogeneity? The same asymmetry gives a clean theoretical answer.

Three Sufficient Conditions to Recover Robustness

Four Training-Curve Patterns

How closely (C1) to (C3) hold places a run on a continuous stable to degrade to collapse spectrum, readable straight off the curve.

Experiments

Qwen2.5 (1.5B to 7B) and Llama-3.2-3B, under both GRPO and PPO, on WebShop & ALFWorld over 3 seeds.

Citation

@article{chen2026fedagent,
  title   = {Is Decentralized LLM Agent RL Robust to Heterogeneity? An Asymmetric Tale},
  author  = {Chen, Canyu and Zhu, Kangyu and Chen, Zhaorun and Zhou, Zhanhui and
             Diao, Shizhe and Lu, Yiping and Li, Tian and Li, Manling and Song, Dawn},
  journal = {arXiv preprint},
  year    = {2026},
  url     = {https://fed-agent.github.io/}
}