Publications

2026

1. ICML

   

   Elastic Mixture of Rank-Wise Experts for Knowledge Reuse in Federated Fine-Tuning

   Yebo Wu^*, Jingguang Li^*, Zhijiang Guo, and 1 more author

   In ICML (Spotlight), 2026

   Abs HTML Code

   Federated fine-tuning presents a promising avenue for adapting Large Language Models (LLMs) to downstream tasks while preserving data privacy. However, the prohibitive computational and communication overhead of LLM adaptation inhibits its deployment on resource-constrained edge devices. In this paper, we propose SmartFed, a resource-efficient framework that circumvents expensive training from scratch by intelligently reusing knowledge embedded in existing LoRA modules. To fully exploit this potential and ensure scalability, we introduce the Mixture of Rank-Wise Experts (MoRE). MoRE decomposes LoRA modules into fine-grained rank-level experts, which are selectively activated based on input semantics and resource budgets. Furthermore, to optimize resource utilization, we propose Elastic Expert Quota Allocation (EEQA), a strategy that adaptively distributes expert capacity across parameter matrices based on their contribution to model performance. Extensive evaluations across multiple benchmarks demonstrate that SmartFed significantly outperforms state-of-the-art methods in both model performance and training efficiency.

2. ACL

   

   Beyond End-to-End: Dynamic Chain Optimization for Private LLM Adaptation on the Edge

   Yebo Wu^*, Jingguang Li^*, Chunlin Tian, and 3 more authors

   In ACL (Oral), 2026

   Abs HTML

   Federated fine-tuning enables privacy-preserving LLM adaptation but faces a critical bottleneck: the disparity between LLMs’ high memory demands and edge devices’ limited capacity. To break the memory barrier, we propose Chain Federated Fine-Tuning (ChainFed), an innovative paradigm that forgoes end-to-end updates in favor of a sequential, layer-by-layer manner. It first trains the initial adapter to convergence, freezes its weights, and then proceeds to the next. This iterative train-and-freeze process forms an optimization chain, gradually enhancing the model’s task-specific proficiency. ChainFed further integrates three core techniques: 1) Dynamic Layer Co-Tuning to bridge semantic gaps between sequentially tuned layers and facilitate information flow; 2) Globally Perceptive Optimization to endow each adapter with foresight beyond its local objective; 3) Function-Oriented Adaptive Tuning to automatically identify the optimal fine-tuning starting point. Extensive experiments on multiple benchmarks demonstrate the superiority of ChainFed over existing methods, boosting average accuracy by up to 46.46%.

3. ICLR

   

   Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stages

   Yebo Wu^*, Jingguang Li^*, Zhijiang Guo, and 1 more author

   In ICLR, 2026

   Abs HTML

   Federated fine-tuning enables Large Language Models (LLMs) to adapt to downstream tasks while preserving data privacy, but its resource-intensive nature limits deployment on edge devices. In this paper, we introduce Developmental Federated Tuning (DevFT), a resource-efficient approach inspired by cognitive development that progressively builds a powerful LLM from a compact foundation. DevFT decomposes the fine-tuning process into developmental stages, each optimizing submodels with increasing parameter capacity. Knowledge from earlier stages transfers to subsequent submodels, providing optimized initialization parameters that prevent convergence to local minima and accelerate training. This paradigm mirrors human learning, gradually constructing comprehensive knowledge structure while refining existing skills. To efficiently build stage-specific submodels, DevFT introduces deconfliction-guided layer grouping and differential-based layer fusion to distill essential information and construct representative layers. Evaluations across multiple benchmarks demonstrate that DevFT significantly outperforms state-of-the-art methods, achieving up to 4.59x faster convergence, 10.67x reduction in communication overhead, and 9.07% average performance improvement, while maintaining compatibility with existing approaches.

4. ACM SoCC

   

   Memory-Decoupled Layer-Wise Fine-Tuning for Efficient On-Device LLM Adaptation

   Yebo Wu^*, Jingguang Li^*, Chunlin Tian, and 3 more authors

   In ACM SoCC, 2026

   Abs

   Federated Fine-Tuning (FFT) is essential for customizing Large Language Models (LLMs) at the edge while preserving data privacy. However, its scalability is severely constrained by the memory bottleneck. Our analysis reveals that the massive static memory footprint of LLM parameters exceeds the physical capacity of edge devices, rendering standard end-to-end training infeasible. In this paper, we propose EdgeLLM, a Cloud-Edge memory management system that democratizes efficient edge LLM adaptation via progressive layer training. Specifically, EdgeLLM re-architects the training process into a sequential pipeline: it dynamically manages the device memory hierarchy by maintaining only a minimal granular subset of layers on-chip while aggressively offloading inactive ones. To maximize system efficiency and convergence stability, EdgeLLM integrates four key components: (1) Resource Scheduler, which continuously probes device runtime memory availability to maximize resource utilization via dynamic workload mapping; (2) Training Coordinator, which bridges the semantic gap caused by offloading via virtualized context synthesis, ensuring coherent feature learning without full-graph residence; (3) Execution Planner, which optimizes the computation path by intelligently pruning redundancy and prioritizing task-critical layers for targeted adaptation; and (4) I/O Manager, which neutralizes the layer-swapping overhead through asynchronous prefetching and eviction. Comprehensive evaluations demonstrate that EdgeLLM effectively breaks the memory wall, unlocking participation for resource-constrained devices previously excluded from the training cohort. Compared to state-of-the-art baselines, EdgeLLM delivers up to 11.86% performance improvement, accelerates model convergence by 2.08x, and reduces communication overhead by 1.78x.

5. TMLR

   

   A Survey on Federated Fine-Tuning of Large Language Models

   Yebo Wu, Chunlin Tian, Jingguang Li, and 7 more authors

   In TMLR, 2026

   Abs HTML Code

   Large Language Models (LLMs) have demonstrated impressive success across various tasks. Integrating LLMs with Federated Learning (FL), a paradigm known as FedLLM, offers a promising avenue for collaborative model adaptation while preserving data privacy. This survey provides a systematic and comprehensive review of FedLLM. We begin by tracing the historical development of both LLMs and FL, summarizing relevant prior research to set the context. Subsequently, we delve into an in-depth analysis of the fundamental challenges inherent in deploying FedLLM. Addressing these challenges often requires efficient adaptation strategies; therefore, we conduct an extensive examination of existing Parameter-Efficient Fine-tuning (PEFT) methods and explore their applicability within the FL framework. To rigorously evaluate the performance of FedLLM, we undertake a thorough review of existing fine-tuning datasets and evaluation benchmarks. Furthermore, we discuss FedLLM’s diverse real-world applications across multiple domains. Finally, we identify critical open challenges and outline promising research directions to foster future advancements in FedLLM. This survey aims to serve as a foundational resource for researchers and practitioners, offering valuable insights into the rapidly evolving landscape of federated fine-tuning for LLMs. It also establishes a roadmap for future innovations in privacy-preserving AI.

2025

1. arXiv

   

   Memory-Efficient Federated Fine-Tuning of Large Language Models via Layer Pruning

   Yebo Wu^*, Jingguang Li^*, Chunlin Tian, and 2 more authors

   In arXiv, 2025

   Abs HTML

   Federated fine-tuning enables privacy-preserving Large Language Model (LLM) adaptation, but its high memory cost limits participation from resource-constrained devices. We propose FedPruner, an innovative federated fine-tuning paradigm that tackles this via intelligent layer pruning. FedPruner flexibly prunes the global model, creating personalized submodels based on device memory constraints. It employs a macro-micro synergistic pruning framework: a macro-level functionality-driven layer orchestration mechanism groups layers, while a micro-level importance-aware layer selection strategy prunes within groups to build device-specific submodels. We further introduce a fine-grained variant that independently prunes Multi-Head Attention and Feed-Forward Network components to precisely preserve critical architectural elements. Extensive experimental results demonstrate that FedPruner significantly outperforms state-of-the-art approaches, achieving up to a 1.98% improvement in average model accuracy while reducing peak memory usage by 75%.

2024

1. arXiv

   

   Bridging Memory Gaps: Scaling Federated Learning for Heterogeneous Clients

   Yebo Wu, Jingguang Li, Chunlin Tian, and 3 more authors

   In arXiv, 2024

   Abs HTML

   Federated Learning (FL) enables multiple clients to collaboratively train a shared model while preserving data privacy. However, the high memory demand during model training severely limits the deployment of FL on resource-constrained clients. To this end, we propose ScaleFL, a scalable and inclusive FL framework designed to overcome memory limitations through sequential block-wise training. The core idea of ScaleFL is to partition the global model into blocks and train them sequentially, thereby reducing training memory requirements. To mitigate information loss during block-wise training, ScaleFL introduces a Curriculum Mentor that crafts curriculum-aware training objectives for each block to steer their learning process. Moreover, ScaleFL incorporates a Training Harmonizer that designs a parameter co-adaptation training scheme to coordinate block updates, effectively breaking inter-block information isolation. Extensive experiments on both simulation and hardware testbeds demonstrate that ScaleFL significantly improves model performance by up to 84.2%, reduces peak memory usage by up to 50.4%, and accelerates training by up to 1.9x.