The Intersection of Modular Architectures and Scalable AI Systems

1. Introduction

In recent years, the exponential growth of data and the increasing complexity of machine learning tasks have necessitated the development of models that are not only expressive and scalable but also computationally efficient. Among the architectural paradigms that have garnered substantial attention, the Mixture of Experts (MoE) framework has emerged as a powerful mechanism for conditional computation, where only a sparse subset of model parameters is activated for a given input. Originally proposed by Jacobs et al. in the early 1990s, MoE models are rooted in the ensemble learning paradigm, yet distinguished by their dynamic, input-dependent routing of computational resources.

Notation Summary

For clarity, we include a brief summary of the primary notations used throughout this survey:

Table 1. Summary of Notation.

Symbol	Meaning
$E_m(·)$	m-th expert function
$G_m\left(\mathbf{x}\right)$	Gating weight for expert m at input $\mathbf{x}$
${\mathcal{S}}_k\left(\mathbf{x}\right)$	Top-k selected experts for input $\mathbf{x}$
$f_{\mathrm{MoE}}\left(\mathbf{x}\right)$	Output of the MoE model at input $\mathbf{x}$
M	Total number of experts
k	Number of active experts per input
$n_e$	Number of parameters per expert
${\mathcal{F}}_{\mathrm{MoE}}$	Function class represented by MoE models
${\mathcal{R}}_n$	Rademacher complexity
$\mathcal{T}$	Task identifier in continual learning

Table 1. Summary of Notation.

Symbol	Meaning
$E_m(·)$	m-th expert function
$G_m\left(\mathbf{x}\right)$	Gating weight for expert m at input $\mathbf{x}$
${\mathcal{S}}_k\left(\mathbf{x}\right)$	Top-k selected experts for input $\mathbf{x}$
$f_{\mathrm{MoE}}\left(\mathbf{x}\right)$	Output of the MoE model at input $\mathbf{x}$
M	Total number of experts
k	Number of active experts per input
$n_e$	Number of parameters per expert
${\mathcal{F}}_{\mathrm{MoE}}$	Function class represented by MoE models
${\mathcal{R}}_n$	Rademacher complexity
$\mathcal{T}$	Task identifier in continual learning

Let $\mathcal{X}\subseteq {\mathbb{R}}^d$ denote the input space and $\mathcal{Y}\subseteq {\mathbb{R}}^k$ the output space. Given a dataset $\mathcal{D}={\left\{({\mathbf{x}}_i,{\mathbf{y}}_i)\right\}}_{i=1}^n$ with ${\mathbf{x}}_i\in \mathcal{X}$ and ${\mathbf{y}}_i\in \mathcal{Y}$, the goal is to learn a function $f:\mathcal{X}\to \mathcal{Y}$ that minimizes some loss function $\mathcal{L}:\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}_{\ge 0}$. In the MoE setting, the hypothesis class ${\mathcal{H}}_{\mathrm{MoE}}$ is constructed as a composition of a gating network $G:\mathcal{X}\to {\Delta }^{M-1}$ and a set of expert networks ${\left\{E_m\right\}}_{m=1}^M$, where ${\Delta }^{M-1}$ is the $(M-1)$-dimensional probability simplex. Formally, the MoE model computes the output for an input $\mathbf{x}\in \mathcal{X}$ as:

$$f_{\mathrm{MoE}}\left(\mathbf{x}\right)=\sum _{m=1}^M{G}_m\left(\mathbf{x}\right)E_m\left(\mathbf{x}\right),$$

(1)

where $G_m\left(\mathbf{x}\right)$ denotes the m-th component of the gating function $G\left(\mathbf{x}\right)$, and $E_m:\mathcal{X}\to \mathcal{Y}$ is the m-th expert [1]. The gating function assigns input-dependent weights to each expert, effectively selecting a subset of experts based on relevance. In sparse variants, such as those employed in large-scale language models (e.g., GShard, Switch Transformers), the sum in Equation (1) is replaced with a sparse sum over the top-k experts:

$$f_{\mathrm{MoE}}^{\mathrm{sparse}}\left(\mathbf{x}\right)=\sum _{m\in {\mathcal{S}}_k\left(\mathbf{x}\right)}{G}_m\left(\mathbf{x}\right)E_m\left(\mathbf{x}\right),$$

(2)

where ${\mathcal{S}}_k\left(\mathbf{x}\right)\subseteq \{1,\dots ,M\}$ is the set of indices corresponding to the top-k gating weights. The appeal of MoE lies in its theoretical and empirical promise to approximate complex, multimodal mappings while preserving computational efficiency [2]. Theoretically, MoE models relate closely to function spaces defined by piecewise-linear or piecewise-smooth functions. Under mild assumptions, MoE can approximate any measurable function to arbitrary accuracy, a consequence of the universal approximation theorem when the experts are themselves universal approximators (e.g., multilayer perceptrons). In the context of optimization, training MoE models introduces unique challenges, primarily due to the non-differentiability of top-k selection and the high variance in expert utilization. Various techniques, such as load-balancing regularization, straight-through estimators, and auxiliary loss terms, have been introduced to mitigate these issues. Let ${\theta }_G$ and ${\theta }_E={\left\{{\theta }_m\right\}}_{m=1}^M$ denote the parameters of the gating and expert networks, respectively. The optimization objective becomes:

$$\underset{{\theta }_G,{\theta }_E}{min}{\mathbb{E}}_{(\mathbf{x},\mathbf{y})\sim \mathcal{D}}\left[\mathcal{L}\left(f_{\mathrm{MoE}}\left(\mathbf{x}\right),\mathbf{y}\right)+\lambda ·\mathcal{R}({\theta }_G,{\theta }_E)\right],$$

(3)

where $\mathcal{R}(·)$ denotes a regularization term (e.g., to enforce balanced expert usage), and $\lambda$ is a tunable hyperparameter [3]. From a probabilistic perspective, MoE can be interpreted as a hierarchical latent variable model. Let $Z\in \{1,\dots ,M\}$ be a latent variable indicating the active expert [4]. Then the conditional likelihood becomes:

$$p(\mathbf{y}\mid \mathbf{x})=\sum _{m=1}^{M}p(Z=m\mid \mathbf{x})p(\mathbf{y}\mid \mathbf{x},Z=m),$$

(4)

which aligns with the mixture-of-distributions interpretation, where $p(Z=m\mid \mathbf{x})=G_m\left(\mathbf{x}\right)$ and $p(\mathbf{y}\mid \mathbf{x},Z=m)=E_m\left(\mathbf{x}\right)$ under deterministic experts [5]. The relevance of MoE in modern AI systems is underscored by its deployment in cutting-edge architectures for natural language processing, vision, and multi-modal learning [6]. As models scale to billions of parameters, the sparse activation in MoE allows for the decoupling of model capacity from inference cost [7]. Yet, this expressivity comes at the cost of architectural complexity, instability during training, and challenges in routing, interpretability, and load balancing. This survey aims to provide a comprehensive and mathematically grounded review of the Mixture of Experts paradigm, encompassing classical formulations, recent advancements in sparse MoE, training algorithms, theoretical underpinnings, and diverse applications across machine learning domains. Through rigorous exposition, we elucidate the principles and design choices that govern MoE models, shedding light on their power, limitations, and future research directions [8].

2. Background and Preliminaries

The Mixture of Experts (MoE) architecture can be viewed through multiple theoretical lenses, including ensemble learning, conditional computation, probabilistic graphical models, and function approximation theory [9]. This section lays the mathematical groundwork necessary for understanding the full breadth of MoE methodologies by formally describing key components such as expert functions, gating mechanisms, sparsity-inducing strategies, and probabilistic interpretations [10].

2.1. Expert Networks

Let us denote the expert set as $\mathcal{E}={\left\{E_m(·;{\theta }_m)\right\}}_{m=1}^M$, where each $E_m:\mathcal{X}\to \mathcal{Y}$ is a parameterized function representing the m-th expert, and ${\theta }_m$ are its parameters [11]. In practice, each expert may take the form of a neural network, such as a multilayer perceptron (MLP), convolutional network, or transformer block [12]. The choice of expert architecture significantly influences the representational power of the MoE ensemble [13]. Each expert function can be interpreted as a basis function in a functional basis expansion:

$$f\left(\mathbf{x}\right)=\sum _{m=1}^M{\alpha }_m\left(\mathbf{x}\right){\varphi }_m\left(\mathbf{x}\right),$$

(5)

where ${\varphi }_m\left(\mathbf{x}\right):=E_m\left(\mathbf{x}\right)$ and ${\alpha }_m\left(\mathbf{x}\right):=G_m\left(\mathbf{x}\right)$ are input-dependent coefficients from the gating function [14].

2.2. Gating Mechanism

The gating network $G:\mathcal{X}\to {\Delta }^{M-1}$ maps an input $\mathbf{x}$ to a discrete probability distribution over experts, where ${\Delta }^{M-1}$ is the $(M-1)$-dimensional probability simplex:

$${\Delta }^{M-1}=\left\{\mathbf{g}\in {\mathbb{R}}^M|g_i\ge 0,\sum _{i=1}^M{g}_i=1\right\}.$$

(6)

A typical instantiation of the gating function is a softmax transformation applied to a learned scoring function $s:\mathcal{X}\to {\mathbb{R}}^M$:

$$G_m\left(\mathbf{x}\right)={\displaystyle \frac{exp\left(s_m\left(\mathbf{x}\right)\right)}{{\sum }_{j=1}^{M}exp\left(s_j\left(\mathbf{x}\right)\right)}}.$$

(7)

In sparse MoE models, the gating function is modified to activate only the top-k experts [15]. Let ${\mathcal{S}}_k\left(\mathbf{x}\right)$ denote the index set of the top-k scores $s_m\left(\mathbf{x}\right)$ [16]. Then, we define a sparse softmax:

$$G_m^{\mathrm{sparse}}\left(\mathbf{x}\right)=\left\{\begin{array}{cc}{\displaystyle \frac{exp\left(s_m\left(\mathbf{x}\right)\right)}{{\sum }_{j\in {\mathcal{S}}_k\left(\mathbf{x}\right)}exp\left(s_j\left(\mathbf{x}\right)\right)}}\hfill & \mathrm{if}m\in {\mathcal{S}}_k\left(\mathbf{x}\right),\hfill \\ 0\hfill & \mathrm{otherwise}\left[17\right].\hfill \end{array}\right.$$

(8)

2.3. Probabilistic Interpretation

From a generative modeling standpoint, MoE may be interpreted as a latent variable model with a discrete latent variable $Z\sim \mathrm{Categorical}\left(G\right(\mathbf{x}\left)\right)$, which determines the responsible expert for each input $\mathbf{x}$. The joint distribution can be written as:

$$p(\mathbf{y},Z=m\mid \mathbf{x})=G_m\left(\mathbf{x}\right)·p(\mathbf{y}\mid \mathbf{x},Z=m),$$

(9)

with the marginal likelihood given by:

$$p(\mathbf{y}\mid \mathbf{x})=\sum _{m=1}^M{G}_m\left(\mathbf{x}\right)·p(\mathbf{y}\mid \mathbf{x},Z=m).$$

(10)

This probabilistic view facilitates the use of Expectation-Maximization (EM) algorithms for training, especially in scenarios where the expert outputs correspond to probabilistic densities or classifiers.

2.4. Function Approximation Perspective

From the perspective of approximation theory, MoE systems can be shown to approximate any continuous function $f\in C\left(\mathcal{X}\right)$ under mild assumptions [18]. If each expert $E_m$ is drawn from a universal function approximator class (e.g., feedforward neural networks), then for any $\epsilon >0$, there exists a gating function G and experts $\left\{E_m\right\}$ such that:

$$\underset{\mathbf{x}\in \mathcal{X}}{sup}∥f\left(\mathbf{x}\right)-\sum _{m=1}^M{G}_m\left(\mathbf{x}\right)E_m\left(\mathbf{x}\right)∥<\epsilon .$$

(11)

This result follows from a generalized Stone-Weierstrass theorem applied to MoE compositions, under the constraint that G and $E_m$ are continuous and suitably bounded [19].

2.5. Load Balancing and Expert Utilization

To avoid underutilization of certain experts, which leads to training inefficiencies and poor generalization, modern MoE models introduce regularization terms that penalize imbalance. Let ${\pi }_m={\mathbb{E}}_{\mathbf{x}\sim \mathcal{D}}\left[G_m\left(\mathbf{x}\right)\right]$ denote the average usage of expert m [20]. A commonly used load-balancing loss is the entropy regularizer or coefficient-of-variation penalty:

$${\mathcal{R}}_{\mathrm{load}}=\lambda ·{\mathrm{CV}}^2\left(\left\{{\pi }_m\right\}\right)=\lambda ·{\displaystyle \frac{\mathrm{Var}\left[{\pi }_m\right]}{{\left(\mathbb{E}\left[{\pi }_m\right]\right)}^2}},$$

(12)

which encourages uniformity in ${\left\{{\pi }_m\right\}}_{m=1}^M$.

2.6. Training Objectives and Gradient Estimation

Training MoE models requires careful gradient estimation, especially when sparsity leads to discrete selections [21]. Let ${\mathcal{L}}_{\mathrm{MoE}}\left(\theta \right)$ denote the full objective:

$${\mathcal{L}}_{\mathrm{MoE}}\left(\theta \right)={\mathbb{E}}_{(\mathbf{x},\mathbf{y})\sim \mathcal{D}}\left[\ell \left(f_{\mathrm{MoE}}\left(\mathbf{x}\right),\mathbf{y}\right)+{\mathcal{R}}_{\mathrm{load}}\left(\theta \right)\right].$$

(13)

In sparse models, straight-through estimators or soft relaxations (e.g., Gumbel-softmax) are employed to approximate gradients through non-differentiable top-k operations. This theoretical background provides the tools necessary to delve into the design, optimization, and practical deployment of MoE architectures [22]. In the subsequent sections, we explore classical and modern variations of MoE, survey optimization techniques, and analyze their empirical performance across multiple domains [23].

3. Taxonomy and Variants of Mixture of Experts

The Mixture of Experts framework admits a rich taxonomy, encompassing numerous architectural variants and operational paradigms. These models vary along multiple axes, including the gating function’s structure, expert specialization, sparsity of activation, parameter sharing, and probabilistic versus deterministic routing [24]. In this section, we provide a formal categorization of MoE models, elucidating the distinguishing mathematical properties of each class [25].

3.1. Soft Versus Hard Gating

The most fundamental distinction among MoE models lies in the nature of the gating function $G\left(\mathbf{x}\right)$.

3.1.1. Soft Gating

In soft-gated MoE models, all experts contribute to the final output with weights proportional to their gating probabilities:

$$f_{\mathrm{soft}}\left(\mathbf{x}\right)=\sum _{m=1}^M{G}_m\left(\mathbf{x}\right)E_m\left(\mathbf{x}\right),G_m\left(\mathbf{x}\right)\in [0,1],\sum _{m=1}^M{G}_m\left(\mathbf{x}\right)=1\left[26\right].$$

(14)

This formulation yields a smooth and differentiable function with respect to both the gating parameters and expert parameters. Soft MoE is closely related to attention mechanisms and kernel mixtures in probabilistic models.

3.1.2. Hard Gating

Hard-gated MoE activates a single expert per input, typically chosen as:

$$m^{*}=arg\underset{m}{max}{G}_m\left(\mathbf{x}\right),f_{\mathrm{hard}}\left(\mathbf{x}\right)=E_{m^{*}}\left(\mathbf{x}\right)\left[27\right].$$

(15)

Since the $argmax$ operator is non-differentiable, hard MoE models often employ surrogate gradient techniques (e.g., REINFORCE, Gumbel-softmax) for training [28].

3.2. Sparse Gating and Top-k Experts

A middle ground between soft and hard MoE is achieved via sparse gating, where only a subset ${\mathcal{S}}_k\left(\mathbf{x}\right)\subset \{1,\dots ,M\}$ of k experts are activated [29]. The output becomes:

$$f_{\mathrm{sparse}}\left(\mathbf{x}\right)=\sum _{m\in {\mathcal{S}}_k\left(\mathbf{x}\right)}{G}_m\left(\mathbf{x}\right)E_m\left(\mathbf{x}\right).$$

(16)

Sparse MoEs reduce computational cost from $\mathcal{O}\left(M\right)$ to $\mathcal{O}\left(k\right)$ and are amenable to scalable deployment in large neural architectures such as Switch Transformers and GShard.

3.3. Independent Versus Shared Experts

MoE models may be further classified by whether experts are parameterized independently or share parts of their architecture:

• Independent Experts: Each $E_m(·;{\theta }_m)$ is trained separately and has a distinct parameter set [30]. This offers high specialization but incurs greater memory cost [31].
• Shared Experts: Experts may share a common backbone or parameter subsets [32]. For example, each expert may be implemented as a residual transformation over a shared encoder:

  $$E_m\left(\mathbf{x}\right)=h\left(\mathbf{x}\right)+{\Delta }_m\left(\mathbf{x}\right),$$

  (17)

  where $h\left(\mathbf{x}\right)$ is a shared base and ${\Delta }_m$ is the expert-specific residual.

3.4. Static Versus Dynamic Experts

In static MoE, the expert functions $E_m$ are fixed post-training or specialized via pre-defined tasks [33]. In contrast, dynamic MoE adapts expert parameters or selection policy during training or inference [34]. Some dynamic models even incorporate meta-learning components, where experts are selected based on learned context embeddings [35].

3.5. Hierarchical Mixture of Experts

Hierarchical MoE introduces multiple levels of gating, leading to tree-structured architectures [36]. A typical two-level hierarchical MoE can be expressed as:

$$f\left(\mathbf{x}\right)=\sum _{i=1}^{M_1}{G}_i^{\left(1\right)}\left(\mathbf{x}\right)\left[\sum _{j=1}^{M_2}{G}_{i,j}^{\left(2\right)}\left(\mathbf{x}\right)E_{i,j}\left(\mathbf{x}\right)\right],$$

(18)

where $G^{\left(1\right)}$ is a coarse-grained gate selecting among $M_1$ super-experts, each of which controls a second-layer MoE with $M_2$ sub-experts. Hierarchical MoE enables scalable routing with logarithmic complexity in the number of total experts.

3.6. Probabilistic Versus Deterministic Routing

In probabilistic MoE, routing is stochastic and often accompanied by latent variable inference. In deterministic MoE, expert selection is deterministic and often implemented via thresholding or argmax selection. The probabilistic formulation is more amenable to Bayesian learning, while deterministic routing is typically more efficient for deployment.

3.7. Multi-Task and Multi-Modal Mixture of Experts

MoE is particularly well-suited to multi-task learning (MTL) and multi-modal fusion, where each expert is specialized to a task or modality. Let $\mathcal{T}=\{1,\dots ,T\}$ denote the task index space. For multi-task MoE, the gating function may be conditioned on both input $\mathbf{x}$ and task label $t\in \mathcal{T}$:

$$G_m(\mathbf{x},t)={\displaystyle \frac{exp\left(s_m(\mathbf{x},t)\right)}{{\sum }_{j=1}^{M}exp\left(s_j(\mathbf{x},t)\right)}}\left[37\right].$$

(19)

Similarly, in multi-modal MoE, experts are specialized to different input domains (e.g., vision, text, audio), and the gating function aggregates modality-specific signals.

3.8. Notable Architectures

Prominent implementations of MoE include:

• GShard MoE [38]: A large-scale MoE with sparse activation and gradient-based load balancing [39].
• Switch Transformer [1]: Simplifies MoE by activating a single expert per layer, leading to computational savings and robust scaling [40].
• BASE Layers [38]: Balance-efficient MoEs trained with auxiliary losses to encourage uniform routing [41].
• Task MoE / Multi-gate MoE: Used in multi-task learning where separate gating networks per task provide flexibility in expert sharing [42].

This taxonomy underscores the diversity of MoE formulations, each making unique trade-offs between expressivity, efficiency, interpretability, and scalability. In the next section, we delve into the optimization techniques employed for training these architectures efficiently and effectively.

4. Training and Optimization Techniques

Training Mixture of Experts (MoE) models introduces several challenges that extend beyond standard neural network optimization [43]. These include (i) gradient estimation under discrete expert selection, (ii) load balancing to ensure expert utilization, (iii) routing instability, and (iv) large-scale parallelization. In this section, we present a formal treatment of training objectives, regularization methods, sparse gating optimization, and gradient approximation techniques [44].

4.1. Optimization Objective

Let $\mathcal{D}={\left\{({\mathbf{x}}^{\left(i\right)},{\mathbf{y}}^{\left(i\right)})\right\}}_{i=1}^N$ be the dataset, and denote the MoE output as:

$$f_{\mathrm{MoE}}\left(\mathbf{x}\right)=\sum _{m\in {\mathcal{S}}_k\left(\mathbf{x}\right)}{G}_m\left(\mathbf{x}\right)E_m\left(\mathbf{x}\right).$$

(20)

The total loss consists of two components: (1) the primary prediction loss $\ell (f_{\mathrm{MoE}}\left(\mathbf{x}\right),\mathbf{y})$, and (2) a regularization term ${\mathcal{R}}_{\mathrm{MoE}}$:

$$\mathcal{L}={\mathbb{E}}_{(\mathbf{x},\mathbf{y})\sim \mathcal{D}}\left[\ell (f_{\mathrm{MoE}}\left(\mathbf{x}\right),\mathbf{y})\right]+{\mathcal{R}}_{\mathrm{MoE}}.$$

(21)

The regularization term may include load balancing losses, sparsity penalties, entropy constraints, and auxiliary objectives designed to stabilize expert routing [45].

4.2. Load Balancing and Auxiliary Losses

To promote balanced expert usage, many MoE models incorporate auxiliary losses [46]. Let ${\mathbb{E}}_{\mathbf{x}\sim \mathcal{D}}\left[G_m\left(\mathbf{x}\right)\right]$ denote the expected activation of expert m [47]. The following are common regularization schemes:

Load Balance Loss:

$${\mathcal{R}}_{\mathrm{load}}=\lambda ·\left({\displaystyle \frac{{\sum }_{m=1}^M{\pi }_m^2}{{\left({\sum }_{m=1}^M{\pi }_m\right)}^2}}\right),{\pi }_m:=\sum _{i=1}^N{G}_m\left({\mathbf{x}}^{\left(i\right)}\right).$$

(22)

This regularizer penalizes high variance in expert usage.

Entropy-Based Regularization:

$${\mathcal{R}}_{\mathrm{ent}}=-\lambda ·{\mathbb{E}}_{\mathbf{x}\sim \mathcal{D}}\left[\sum _{m=1}^M{G}_m\left(\mathbf{x}\right)log{G}_m\left(\mathbf{x}\right)\right].$$

(23)

This encourages higher entropy in expert selection, promoting exploration and utilization diversity [48].

4.3. Gradient Estimation under Sparse Routing

The sparse selection of experts introduces non-differentiability into the forward path, particularly with the top-k selection operator:

$${\mathcal{S}}_k\left(\mathbf{x}\right)=\mathrm{TopK}\left({\left\{s_m\left(\mathbf{x}\right)\right\}}_{m=1}^M\right)\left[49\right].$$

(24)

To handle this, several surrogate techniques are employed:

4.3.1. Straight-Through Estimator (STE)

The STE approximates the gradient of a discrete operator by treating it as an identity in the backward pass:

$${\displaystyle \frac{\partial \mathrm{TopK}\left(s\right)}{\partial s}}\approx {\mathbb{I}}_{m\in {\mathcal{S}}_k\left(s\right)}\left[50\right].$$

(25)

While biased, this estimator is simple and empirically effective [51].

4.3.2. Gumbel-Softmax Relaxation

Let $g_m\sim \mathrm{Gumbel}(0,1)$ be i.i.d. noise samples [52]. The top-k can be approximated via:

$${\tilde{G}}_m\left(\mathbf{x}\right)={\displaystyle \frac{exp\left((s_m\left(\mathbf{x}\right)+g_m)/\tau \right)}{{\sum }_{j\in {\mathcal{S}}_k\left(\mathbf{x}\right)}exp\left((s_j\left(\mathbf{x}\right)+g_j)/\tau \right)}},$$

(26)

where $\tau >0$ is a temperature parameter. As $\tau \to 0$, the distribution becomes increasingly sparse [53].

4.4. Backpropagation Through MoE

In the case of soft gating, the model remains fully differentiable, and standard backpropagation can be used:

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\theta }_m}}=\sum _{i=1}^N{\displaystyle \frac{\partial \ell }{\partial f}}·\left(G_m\left({\mathbf{x}}^{\left(i\right)}\right)·{\displaystyle \frac{\partial E_m\left({\mathbf{x}}^{\left(i\right)}\right)}{\partial {\theta }_m}}\right).$$

(27)

For sparse or hard gating, gradient paths are truncated or approximated using the aforementioned techniques [54].

4.5. Expert Routing Instability

Stochasticity in gating can lead to training instability, particularly in early phases. Techniques to stabilize routing include:

• Temperature annealing in softmax or Gumbel-softmax to gradually sharpen selection [55].
• Moving-average smoothing of gating scores over mini-batches [56].
• Noise regularization to encourage robustness in expert activation under input perturbation.

4.6. Parallelization and Scalability

Large-scale MoE models with thousands of experts require distributed training strategies:

Expert Parallelism:

Experts are partitioned across multiple devices. For input $\mathbf{x}$, only the top-k selected experts are activated and routed to relevant devices. This reduces communication overhead.

All-to-All Communication:

High-performance MoE implementations use custom kernels for collective communication, where each device exchanges selected tokens with the devices hosting their top experts.

Token Sharding and Grouped Routing:

For efficiency, tokens are grouped and routed collectively to minimize inter-device traffic and improve memory coalescence [57].

4.7. Convergence Analysis

While convergence guarantees for MoE remain under active research, empirical evidence suggests that sparse MoE training exhibits convergence behavior comparable to dense models when auxiliary losses are properly tuned. Let $\Theta$ denote the parameter space and assume bounded gradients:

$$\parallel {\nabla }_{\theta }\mathcal{L}\left(\theta \right)\parallel \le G,\forall \theta \in \Theta .$$

(28)

Then, under a learning rate schedule satisfying ${\sum }_t{\eta }_t=\infty$, ${\sum }_t{\eta }_t^2<\infty$, stochastic MoE training with appropriate smoothing converges to a local optimum. This concludes our exploration of training techniques for MoE models. In the next section, we will analyze the empirical performance and applications of these models across a wide range of tasks, from natural language processing to vision and multimodal learning [58].

5. Empirical Performance and Applications

Mixture of Experts (MoE) models have demonstrated remarkable empirical performance across a wide spectrum of tasks, particularly in large-scale settings [59]. Their ability to dynamically allocate model capacity conditioned on input data has enabled significant improvements in parameter efficiency, generalization, and computational scalability. In this section, we survey and analyze the empirical behavior of MoE architectures, benchmarking their performance across multiple domains, and elucidating their task-specific adaptations and benefits [60].

5.1. Evaluation Metrics

The performance of MoE models is typically evaluated via standard predictive metrics, augmented with sparsity-aware and efficiency-specific criteria. Let ${\widehat{y}}^{\left(i\right)}=f_{\mathrm{MoE}}\left({\mathbf{x}}^{\left(i\right)}\right)$ denote the model’s prediction and $y^{\left(i\right)}$ the target output. Common evaluation metrics include:

• Predictive Accuracy: Classification or regression error over validation set ${\mathcal{D}}_{\mathrm{val}}$:

  $$\mathrm{Acc}={\displaystyle \frac{1}{|{\mathcal{D}}_{\mathrm{val}}|}}\sum _{(\mathbf{x},y)\in {\mathcal{D}}_{\mathrm{val}}}\mathbb{I}\left\{argmax\widehat{y}=y\right\}.$$

  (29)
• Perplexity: For language modeling:

  $$\mathrm{PPL}=exp\left(-{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}log{p}_{\theta }\left(y^{\left(i\right)}\right|{\mathbf{x}}^{\left(i\right)})\right).$$

  (30)
• Expert Utilization Entropy: Measures uniformity of expert usage:

  $${\mathcal{H}}_{\mathrm{expert}}=-\sum _{m=1}^M{p}_{m}log{p}_m,p_m={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}\{m\in {\mathcal{S}}_k\left({\mathbf{x}}^{\left(i\right)}\right)\}\left[61\right].$$

  (31)
• Floating Point Operations (FLOPs): To compare compute-efficiency:

  $${\mathrm{FLOPs}}_{\mathrm{MoE}}=\sum _{i=1}^N\sum _{m\in {\mathcal{S}}_k\left({\mathbf{x}}^{\left(i\right)}\right)}\mathrm{FLOPs}\left(E_m\left({\mathbf{x}}^{\left(i\right)}\right)\right).$$

  (32)

5.2. Natural Language Processing (NLP)

MoE models have been most prominently applied in NLP, particularly in transformer-based architectures. Consider a Transformer block where the feedforward layer is replaced by a sparse MoE:

$${\mathrm{FFN}}_{\mathrm{MoE}}\left(\mathbf{x}\right)=\sum _{m\in {\mathcal{S}}_k\left(\mathbf{x}\right)}{G}_m\left(\mathbf{x}\right)·\mathrm{ReLU}(W_m^{\left(2\right)}·\mathrm{ReLU}(W_m^{\left(1\right)}·\mathbf{x}+b_m^{\left(1\right)})+b_m^{\left(2\right)}).$$

(33)

Empirical benchmarks demonstrate:

• Switch Transformer (2022) [1]: Achieved a 7× gain in training speed with comparable perplexity to dense models.
• GShard (2021) [38]: Trained 600B-parameter models on multilingual translation tasks with superior BLEU scores.
• Task-MoE (2023): Enabled parameter-efficient multi-task learning with dynamic expert routing [62].

5.3. Vision Applications

In computer vision, MoE has been integrated into convolutional and vision transformer architectures [63]. For example, in Vision MoE (V-MoE) [64], the MoE block is inserted between attention layers:

$$\mathrm{MoE}-\mathrm{MLP}\left(\mathbf{x}\right)=\sum _{m\in {\mathcal{S}}_k\left(\mathbf{x}\right)}{G}_m\left(\mathbf{x}\right)·{\varphi }_m\left(\mathbf{x}\right),$$

(34)

where ${\varphi }_m$ is an expert-specific multi-layer perceptron. Key findings:

• MoE ViT achieves state-of-the-art top-1 ImageNet accuracy with 4× fewer FLOPs than dense ViT [65].
• Experts tend to specialize on image patches of specific geometric or semantic characteristics [66].

5.4. Multimodal Learning

MoE has shown promise in multimodal architectures, where each expert is specialized to a modality (e.g., text, image, audio) or cross-modal interaction [67]. In a multimodal MoE model, let $\mathbf{x}=[{\mathbf{x}}^{\left(v\right)},{\mathbf{x}}^{\left(t\right)}]$ be the concatenation of visual and textual embeddings [35,68]. The gating function can be factorized:

$$G_m\left(\mathbf{x}\right)=\sigma \left({\alpha }_m^{\left(v\right)}·f_v\left({\mathbf{x}}^{\left(v\right)}\right)+{\alpha }_m^{\left(t\right)}·f_t\left({\mathbf{x}}^{\left(t\right)}\right)\right),$$

(35)

allowing adaptive expert routing based on modality salience. Applications include:

• CLIP-MoE: Specializes experts on alignment between text and vision.
• VATT-MoE: Enhances video-audio-text embeddings via dynamic expert routing [69].

5.5. Few-Shot and Transfer Learning

MoE models exhibit strong few-shot generalization [70]. Experts trained on different tasks or domains can be selectively reused:

$$\mathrm{Transfer}\left(f_{\mathrm{MoE}}\right)=\sum _{m\in {\mathcal{S}}_k(\mathbf{x};\mathrm{new})}{G}_m^{\mathrm{new}}\left(\mathbf{x}\right)E_m^{\mathrm{pretrained}}\left(\mathbf{x}\right).$$

(36)

MoE enables modular transfer by freezing experts and learning new gating functions.

5.6. Ablation and Scaling Studies

Empirical studies further explore the influence of various factors:

• Increasing M (number of experts) improves capacity but may cause routing collapse without regularization.
• Larger k increases compute but smooths gradients and improves stability [71].
• MoE achieves Pareto optimal trade-offs between FLOPs and accuracy in many settings [72].

5.7. Limitations in Empirical Use

Despite their empirical strengths, MoE models face practical issues:

• High variance in expert usage without load balancing.
• Routing collapse where a small subset of experts dominate.
• Communication overhead in distributed setups.
• Difficulty in debugging due to implicit specialization [73].

In summary, MoE models achieve superior empirical performance across domains, especially when compute cost is decoupled from model size. The next section will address the theoretical properties and open questions regarding expressivity, approximation bounds, and generalization of MoE architectures.

6. Theoretical Properties and Expressivity

Mixture of Experts (MoE) architectures exhibit unique theoretical properties arising from their conditional computation and modular structure [74]. In this section, we explore the expressive capacity, approximation bounds, and generalization behavior of MoE models. We further analyze their potential for modular composition, universality, and implicit regularization under sparse expert activation.

6.1. Universal Approximation Properties

Let ${\mathcal{F}}_{\mathrm{MoE}}$ denote the class of functions realizable by an MoE model with M experts and k-sparse gating:

$${\mathcal{F}}_{\mathrm{MoE}}=\left\{f\left(\mathbf{x}\right)=\sum _{m\in {\mathcal{S}}_k\left(\mathbf{x}\right)}{G}_m\left(\mathbf{x}\right)E_m\left(\mathbf{x}\right):G\in \mathcal{G},E_m\in \mathcal{H}\right\},$$

(37)

where $\mathcal{G}$ is the set of gating functions and $\mathcal{H}$ the hypothesis class for experts [75].

Proposition 1 (Universal Approximation):

If each expert $E_m$ is a universal approximator (e.g., a sufficiently wide feedforward neural network), and if the gating function can select any subset ${\mathcal{S}}_k$ conditioned on $\mathbf{x}$, then ${\mathcal{F}}_{\mathrm{MoE}}$ is a universal approximator. Proof Sketch: For any function $f^{*}\in C\left({\mathbb{R}}^d\right)$ and $ϵ>0$, partition the input space into M regions $\left\{{\Omega }_m\right\}$ such that $f^{*}$ is $ϵ$-close to a simple function $f_m$ on ${\Omega }_m$[76]. Set $G_m\left(\mathbf{x}\right)\approx {\mathbb{I}}_{\mathbf{x}\in {\Omega }_m}$ and $E_m\left(\mathbf{x}\right)\approx f_m\left(\mathbf{x}\right)$. Then $f_{\mathrm{MoE}}$ approximates $f^{*}$ with error at most $ϵ$ [77].

6.2. Capacity Scaling with Experts

Unlike dense networks, MoE models scale their representational power with the number of experts M without proportionally increasing compute, due to sparse activation. Denote by ${\mathcal{N}}_{\mathrm{dense}}\left(n\right)$ a dense network with n parameters, and ${\mathcal{N}}_{\mathrm{MoE}}(M,k)$ an MoE with M experts, each of size $n_e$, and k-sparse selection:

$$\mathrm{Total}\mathrm{Parameters}\text{:}\Theta (M·n_e),\mathrm{Active}\mathrm{Parameters}\mathrm{per}\mathrm{Example}\text{:}\Theta (k·n_e).$$

Theorem 1 (Exponential Gain in Capacity):

Suppose each expert $E_m$ belongs to a function class with VC-dimension $d_e$, and $k\ll M$. Then the MoE model class has effective capacity:

$$\mathrm{VC}\left({\mathcal{F}}_{\mathrm{MoE}}\right)\ge \Omega (M·d_e),\mathrm{while}\mathrm{only}O(k·d_e)\mathrm{parameters}\mathrm{are}\mathrm{used}\mathrm{per}\mathrm{input}.$$

(38)

6.3. Modular Representations and Disentanglement

MoE models encourage modular representations by allowing different experts to specialize on distinct input subspaces. Let $\mathbf{x}\in \mathcal{X}$ and assume a latent decomposition $\mathbf{x}=({\mathbf{x}}^{\left(1\right)},\dots ,{\mathbf{x}}^{\left(L\right)})$ [78]. Then MoE can learn a factorized representation:

$$f\left(\mathbf{x}\right)=\sum _{m=1}^M{G}_m\left(\mathbf{x}\right)·E_m\left({\mathbf{x}}^{\left(i_m\right)}\right),$$

(39)

where each expert $E_m$ focuses on a particular latent factor ${\mathbf{x}}^{\left(i_m\right)}$. This supports implicit disentanglement without explicit supervision [79].

6.4. Generalization Under Sparse Activation

Generalization bounds for MoE models derive from capacity control via sparsity and modularity [80]. Suppose a model activates at most k out of M experts per input, with each expert in a hypothesis class $\mathcal{H}$ of Rademacher complexity ${\mathcal{R}}_n\left(\mathcal{H}\right)$. Then the overall Rademacher complexity satisfies:

$${\mathcal{R}}_n\left({\mathcal{F}}_{\mathrm{MoE}}\right)\le k·{\mathcal{R}}_n\left(\mathcal{H}\right)+{\mathcal{R}}_n\left(\mathcal{G}\right),$$

(40)

where ${\mathcal{R}}_n\left(\mathcal{G}\right)$ accounts for the gating function complexity [81].

Implication:

MoE models can achieve high expressivity while maintaining generalization, provided that the number of active experts k is small and the gating function is regularized [82].

6.5. Expressivity via Piecewise Function Composition

MoE models define piecewise functions over the input space. Consider a top-1 MoE model:

$$f\left(\mathbf{x}\right)=E_{m^{*}\left(\mathbf{x}\right)}\left(\mathbf{x}\right),m^{*}\left(\mathbf{x}\right)=arg\underset{m}{max}{s}_m\left(\mathbf{x}\right).$$

(41)

Then f is piecewise continuous with regions defined by:

$${\mathcal{R}}_m=\left\{\mathbf{x}\in {\mathbb{R}}^d:s_m\left(\mathbf{x}\right)>s_{m^{\prime }}\left(\mathbf{x}\right),\forall m^{\prime }\ne m\right\}.$$

(42)

Within each region, the function is defined by a single expert [83]. This forms a partitioned decision surface, enabling MoE models to emulate decision trees, rule lists, and other hierarchical structures.

6.6. Theoretical Challenges and Open Questions

Despite their expressive power, several theoretical aspects of MoE remain open:

• Learnability: Under what conditions can the gating and expert functions converge jointly to a global optimum?
• Approximation Limits: What are the lower bounds on approximation error with fixed k and M?
• Overfitting Risks: Can MoE models overfit due to implicit overparameterization, despite sparsity at inference time?
• Compositionality: Can MoE be used to construct compositional programs with guaranteed semantics [84]?

These questions highlight the need for deeper theoretical investigations into MoE models [85]. In the following section, we discuss recent advances, current limitations, and promising future directions for developing more efficient, robust, and interpretable mixtures of experts.

7. Future Directions and Open Problems

Despite their empirical success and promising theoretical foundations, Mixture of Experts (MoE) models present several open challenges and opportunities for further research. These span algorithmic, theoretical, and practical dimensions [86]. In this section, we outline key future directions, conjectures, and open problems that could substantially advance the field.

7.1. Learning Optimal Gating Functions

One of the most critical components of MoE architectures is the gating function $G_m\left(\mathbf{x}\right)$, which governs the sparsity pattern and expert selection. While current designs rely on simple softmax-based or top-k heuristics, an optimal gating function remains elusive.

Open Problem 1 (Gating Optimality):

Let ${\mathcal{S}}_k\left(\mathbf{x}\right)=arg{max}_{S\subseteq \left[M\right],\left|S\right|=k}{\sum }_{m\in S}{s}_m\left(\mathbf{x}\right)$ be the current selection scheme [87]. Define an oracle selector:

$${\mathcal{S}}_k^{*}\left(\mathbf{x}\right)=arg\underset{S\subseteq \left[M\right],\left|S\right|=k}{min}{\parallel f\left(\mathbf{x}\right)-\sum _{m\in S}{E}_m\left(\mathbf{x}\right)\parallel }_2^2.$$

(43)

Can a learnable approximation to ${\mathcal{S}}_k^{*}$ be developed that balances predictive performance and computational complexity?

7.2. Expert Specialization and Diversity Metrics

Understanding and measuring the diversity of expert specialization is key to improving MoE interpretability and robustness [88]. Let ${\varphi }_m\left(\mathbf{x}\right)$ denote an embedding learned by expert $E_m$. Define a diversity metric:

$$D_{\mathrm{inter}}={\displaystyle \frac{1}{M(M-1)}}\sum _{m\ne m^{\prime }}{\mathbb{E}}_{\mathbf{x}\sim \mathcal{D}}\left[\mathrm{sim}({\varphi }_m\left(\mathbf{x}\right),{\varphi }_{m^{\prime }}\left(\mathbf{x}\right))\right],$$

(44)

where $\mathrm{sim}(·,·)$ is a similarity function (e.g., cosine).

Open Problem 2 (Specialization Entropy):

How can we regularize training to explicitly minimize $D_{\mathrm{inter}}$ while preserving model accuracy? What is the theoretical link between diversity and generalization error?

7.3. Dynamic Routing with Reinforcement Learning and Meta-Learning

Current routing schemes are static and differentiable. However, dynamic routing via reinforcement learning (RL) or meta-learning could potentially lead to more adaptive and task-specific expert allocation.

Research Direction:

Formulate the expert selection process as a Markov Decision Process (MDP):

$$\pi :\mathcal{X}\to \mathcal{P}\left({\{1,\dots ,M\}}^k\right),\mathrm{maximize}{\mathbb{E}}_{\mathbf{x}\sim \mathcal{D}}\left[R\left(\pi \left(\mathbf{x}\right)\right)\right],$$

where R is a reward function capturing prediction quality and computation cost.

7.4. MoE in Continual and Lifelong Learning

MoE models offer a natural structure for continual learning, where new experts can be added over time and old ones frozen.

Open Problem 3 (Catastrophic Forgetting Mitigation):

Can we design gating mechanisms such that new tasks automatically route to new experts, thereby minimizing interference with old tasks? Let ${\mathcal{T}}_1,{\mathcal{T}}_2$ be two tasks, and define interference as:

$${\Delta }_{\mathrm{task}}={\mathbb{E}}_{\mathbf{x}\in {\mathcal{T}}_1}\left[\parallel f_{\mathrm{MoE}}^{\mathrm{pre}}\left(\mathbf{x}\right)-f_{\mathrm{MoE}}^{\mathrm{post}}\left(\mathbf{x}\right)\parallel \right]\left[89\right].$$

How can we guarantee ${\Delta }_{\mathrm{task}}\approx 0$ as new tasks are introduced [90]?

7.5. Theoretical Limits and Expressivity Gaps

While MoE models are universal approximators, it remains unclear what classes of functions they represent more efficiently than dense models.

Open Problem 4 (MoE Efficiency Hierarchy):

For which function classes $\mathcal{F}$ does there exist a MoE $f_{\mathrm{MoE}}\in {\mathcal{F}}_{\mathrm{MoE}}$ such that:

$$\parallel f_{\mathrm{MoE}}-f^{*}\parallel \le ϵ\mathrm{with}O(k·n)\mathrm{active}\mathrm{parameters},$$

(45)

but any dense network $f_{\mathrm{dense}}$ satisfying the same approximation requires $\Omega (M·n)$ parameters [91]?

7.6. Scalability and Hardware Efficiency

Sparse expert routing poses significant challenges for hardware efficiency due to non-uniform memory access and communication overheads.

Research Challenge:

Design scheduling and expert placement strategies across devices such that:

$$\mathrm{Total}\mathrm{Latency}=\underset{i\in \mathrm{device}}{max}\sum _{m\in {\mathcal{E}}_i}\mathrm{Time}\left(E_m\left(\mathbf{x}\right)\right),$$

(46)

is minimized under memory and bandwidth constraints [92].

7.7. MoE in Structured Prediction and Probabilistic Inference

An underexplored direction is the use of MoE models in structured prediction tasks, such as parsing, alignment, or probabilistic inference in graphical models [56,93]. Experts can be aligned with specific structural components.

Open Question:

Can MoE architectures emulate tractable probabilistic inference (e.g., sum-product networks) by routing to locally conditioned experts?

7.8. Interpretable and Modular AI Systems

MoE models open the door to more interpretable, modular neural systems, where each expert has a dedicated semantic function [94].

Future Work:

Define a formal logic or grammar over expert compositions, and build verifiable MoE systems for safety-critical applications such as medical diagnostics or autonomous driving.

7.9. Towards Theoretical Foundations for Mixture Sparsity

Lastly, a foundational theory for mixture sparsity is lacking [95]. Given an expert set $\{E_1,\dots ,E_M\}$ and a sparsity constraint $k\ll M$, what is the optimal trade-off between model depth, sparsity, and generalization?

Conjecture (Sparse Mixture Efficiency):

There exists a sparsity threshold $k^{*}$ such that:

$$k<k^{*}\Rightarrow \mathrm{Underfitting},k>k^{*}\Rightarrow \mathrm{Overfitting},k=k^{*}\Rightarrow \mathrm{Generalization}\mathrm{Optimal}.$$

(47)

Deriving such a threshold theoretically or empirically remains a major open problem [96].

7.10. Summary of Research Directions

To conclude, the future of MoE research lies in solving the following grand challenges:

• Learning and regularizing optimal gating functions.
• Ensuring expert diversity and modular generalization.
• Incorporating reinforcement and meta-learning in routing [97].
• Enabling continual and lifelong learning with minimal forgetting [98].
• Developing theoretical foundations of mixture sparsity and compositionality [99].
• Aligning MoE with hardware constraints for scalable deployment.
• Building interpretable and verifiable MoE systems [100].

These directions not only promise to improve MoE performance, but also pave the way toward more adaptive, scalable, and principled artificial intelligence systems [101].

8. Conclusion

Mixture of Experts (MoE) models represent a powerful and flexible paradigm in contemporary artificial intelligence, offering a unique confluence of modularity, conditional computation, and scalable expressivity [102]. Through the strategic combination of multiple expert subnetworks activated selectively by input-dependent gating functions, MoE architectures have achieved state-of-the-art results across a range of domains, including language modeling, vision, reinforcement learning, and multimodal learning. In this survey, we have provided a comprehensive and mathematically rigorous analysis of MoE systems. We began by formalizing the core MoE framework, highlighting the essential components — experts, gating functions, and aggregation mechanisms — and elucidating their interaction [103]. We then classified existing MoE designs into hard versus soft gating, sparse versus dense activation, and hierarchical versus flat topologies [104]. This taxonomy provided a foundation for understanding the structural and algorithmic variants currently employed in practice [105]. We explored the theoretical underpinnings of MoE, demonstrating their universal approximation properties, capacity scaling laws, generalization bounds under sparsity, and the modular disentanglement of latent factors. We also analyzed the expressivity of MoE as a piecewise function approximator, revealing its connection to decision trees and conditional computation graphs [106]. These properties underscore MoE’s ability to efficiently represent high-complexity functions while maintaining parameter sparsity at inference time. A critical examination of training methodologies — including noisy gating, load balancing, auxiliary losses, routing regularization, and gradient sparsity — revealed both strengths and bottlenecks. Empirical instabilities, communication overhead, expert collapse, and sensitivity to hyperparameters remain significant challenges [107]. Nonetheless, these techniques have enabled the deployment of large-scale MoE systems like GShard, Switch Transformer, and V-MoE, which exhibit remarkable scalability and performance. Looking forward, we identified a suite of open problems and promising research directions [108]. These include designing optimal gating policies, improving expert diversity and interpretability, integrating MoE with reinforcement learning and meta-learning, and leveraging MoE architectures for continual learning, modular reasoning, and structured prediction [109]. Additionally, foundational questions about mixture sparsity, task interference, and the interplay between routing complexity and generalization capacity demand deeper investigation [110]. Mixture of Experts models, situated at the intersection of deep learning, modular design, and conditional computation, offer a fertile ground for innovation [111]. As research continues to advance, MoE architectures may serve as a cornerstone for building more adaptive, compositional, and efficient AI systems — potentially guiding the development of intelligent agents that reason over modular structures, dynamically allocate resources, and generalize across diverse tasks and environments.

Final Remarks

As we stand on the frontier of scalable and interpretable machine learning, Mixture of Experts offers not only a powerful tool for performance but also a lens into more modular, structured, and cognitively inspired learning systems. Continued theoretical development, rigorous empirical benchmarking, and responsible system design will be critical to unlocking the full potential of MoE in future AI.