Generalizable and Resilient Multimodal Temporal Learning

1. Introduction

Sleep plays a foundational role in sustaining neurological, cognitive, and emotional health. It has been closely associated with conditions like Alzheimer’s disease [1] and other aging-related neurocognitive impairments [2]. In clinical settings, sleep diagnostics are typically conducted through polysomnography, which records multiple biosignals—EEG, ECG, EOG, EMG, and respiratory activity—throughout the night.

These recordings are annotated by clinicians into discrete 30-second epochs based on the American Academy of Sleep Medicine (AASM) standards [3], which define Wake, REM, and N1 through N3 stages. While this approach is standardized, it is time-consuming and subject to inter-rater variability. Accordingly, there has been significant momentum in building automated systems to perform this task, aiming to enhance efficiency and broaden accessibility.

Early methods focused on handcrafted features derived from EEG signals [4,5,6,7]. More recent solutions employ deep learning architectures such as convolutional neural networks (CNNs) [8,9,10], recurrent neural networks (RNNs) [11,12], and attention-based models like Transformers [13,14,15,16]. Despite accuracy gains, these systems commonly rely solely on EEG input, making them susceptible to data quality issues such as artifacts or signal loss—common in real-world applications.

To overcome this limitation, multimodal input schemes have gained interest. By combining EEG, EOG, EMG, and other physiological channels, researchers aim to harness complementary information inherent in different signal types [19,20]. Although some systems perform direct signal concatenation [11,12], relatively few explore adaptive fusion strategies responsive to the quality of available modalities.

Fusion mechanisms differ in design. Early fusion techniques project raw modalities into a shared representation space [36], yet this often introduces complications due to heterogeneity in signal properties. Late fusion, which aggregates outputs from independent modality-specific models, can fail to fully exploit cross-modal dependencies [37]. A more balanced approach—mid-late fusion—allows feature interactions before classification, enabling both modularity and synergy [38].

Real-world clinical data is often incomplete. Traditional strategies address missing values through statistical imputation [22,24] or neural approximations [28,31]. Noise-prone signals, especially EEG, are commonly denoised through preprocessing pipelines [32,35]. Nonetheless, relatively little attention has been paid to whether fusion models themselves can learn to mitigate such imperfections without explicit repair steps.

Recent progress in multimodal learning introduces mechanisms for shared representation coordination—such as gating, cross-attention, and token-based mediation [39,44,45]. MedFuseSleep adopts this principle by incorporating cross-modal attention layers within a Transformer-based backbone. This design not only allows richer integration across modalities but also confers robustness by letting intact signals offset those affected by noise or absence.

In summary, we present MedFuseSleep, a multimodal temporal modeling system developed with both supervised and auxiliary self-supervised objectives. The contributions of this work include:

• A resilient architecture that remains functional under conditions of noise, signal dropout, or partial modality access;
• A coordinated representation learning approach using intra-transformer cross-attention layers;
• Demonstration of strong performance across multimodal and unimodal testing scenarios;
• Elimination of dependency on pre-imputation or signal restoration for training with incomplete data.

The subsequent sections detail the model architecture, experimental protocol, and comprehensive comparisons against both baseline systems and prior state-of-the-art models.

2. Related Work

Automated sleep stage classification has been an active area of research for decades. Early efforts focused on handcrafted features, while recent advances emphasize deep learning-based methods. This section reviews foundational contributions in four areas pertinent to our work: unimodal sleep classification, multimodal fusion techniques, robustness to incomplete or noisy inputs, and coordinated representation learning in multimodal systems.

2.1. Unimodal Approaches to Sleep Staging

The EEG signal has historically been the central modality for sleep staging due to its capacity to reveal intricate brain activity throughout sleep cycles. Traditional systems depended on expert-designed statistical and spectral features [4,5,6,7], followed by classical machine learning models such as support vector machines and decision trees for classification. Although these models were computationally efficient and interpretable, their generalization across different subjects or recording conditions was limited.

With the rise of deep neural networks, models such as CNNs [8,9,10] have shown strong capabilities in automatically learning spatial hierarchies from raw EEG signals. RNN-based architectures, including SeqSleepNet and its extensions [11,12], added the ability to model sequential dependencies across temporal windows. More recently, Transformer-based models have exhibited superior performance by capturing broader temporal context and long-range dependencies [13,14,15,16,17,18]. Nonetheless, most unimodal techniques still struggle in noisy environments or when the EEG signal is partially lost, leading to a growing interest in incorporating additional modalities to enhance robustness.

2.2. Multimodal Fusion in Sleep Analysis

Multimodal approaches aim to exploit the diversity and complementarity of physiological signals such as EOG, EMG, and respiration, alongside EEG [19,20]. Previous work that adopted simple concatenation of modalities often achieved modest improvements over EEG-only baselines [11,12]. However, these methods typically employ straightforward early fusion strategies that ignore the unique statistical profiles of each modality [36].

Late fusion, where each modality is processed separately and outputs are combined at the prediction stage [37], brings in structural modularity and better fault tolerance, but at the expense of weakened cross-modal synergy. In contrast, mid-to-late fusion models enable a degree of interaction among modalities before final decision-making [38], balancing independence and integration. Despite their potential, many of these frameworks lack dynamic fusion capabilities that adapt information flow according to modality reliability.

2.3. Handling Missing and Noisy Data

Clinical time series are frequently affected by noise and data loss. Classic statistical imputation techniques such as mean replacement [21] and multiple imputation methods [22,23] are not well-suited for high-dimensional, temporal contexts. Machine learning-based solutions, including MissForest [24] and various neural imputation frameworks [25,26,27,28,29], offer scalable alternatives but still require careful assumptions regarding data distribution.

For EEG in particular, missing channels or time segments have been handled via learnable embeddings that allow models to interpolate absent information from context [30,31]. Denoising techniques range from traditional preprocessing workflows [32,33,34] to robust end-to-end modeling strategies [35]. However, many of these solutions are tailored to specific signal types and may break down under extreme degradation or multimodal failure. In our work, we introduce an approach that removes the dependence on separate imputation or denoising procedures. By capitalizing on multimodal redundancy and incorporating adaptive attention strategies, our model can dynamically prioritize more reliable inputs during inference.

2.4. Coordinated Representations in Multimodal Models

The emergence of coordinated representation learning has enhanced the ability of multimodal models to exchange and align information effectively [39]. Rather than treating each modality in a siloed manner, these architectures encourage interaction across branches using methods such as gating mechanisms [40], feature exchange layers [41], and cross-modal attention modules [43,44,45,46,47]. Such strategies have been particularly influential in visual-language domains, where aligning visual regions with linguistic cues at fine granularity has significantly improved model performance. Drawing inspiration from these successes, our model embeds cross-modal attention mechanisms into the transformer encoder, facilitating interactive and flexible representation learning between physiological signals.

A key distinction of our method lies in integrating this interaction within a multi-task learning framework. Each modality contributes both to a shared output and is supervised individually via dedicated prediction heads and alignment objectives [45,50], encouraging consistent and informative representations at multiple levels. This design helps the model generalize better and maintain robustness under varying input configurations. In conclusion, while substantial progress has been made in the field of automated sleep stage classification, existing approaches still face persistent limitations regarding signal integrity and multimodal coordination. Our MedFuseSleep framework leverages the latest advances in transformer architectures, coordinated attention mechanisms, and mid-late fusion paradigms to deliver a flexible and fault-tolerant system tailored for complex real-world sleep data across diverse patient cohorts.

Figure 1. Overview of the MedFuseSleep framework.

Figure 1. Overview of the MedFuseSleep framework.

3. Methodology: MedFuseSleep Framework

In this section, we introduce MedFuseSleep, a robust and modular multimodal framework tailored for sleep stage classification from physiological signals. The model integrates several key innovations: hierarchical temporal modeling, modality-specific encoders, coordinated cross-modality interaction, and multi-objective learning. Below, we first describe the dataset and preprocessing pipeline. We then elaborate on the detailed architectural design, including inner-outer transformer encoders, coordinated representation modules, and the training objectives. Lastly, we outline comparative benchmark structures to contextualize the model’s contributions.

3.1. Dataset and Multistage Signal Preprocessing

We leverage the first phase of the Sleep Heart Health Study (SHHS-1) [51,52], a longitudinal dataset containing overnight polysomnographic recordings from 5,791 subjects aged between 39 and 90. Our work focuses on two modalities: EEG (C4-A1) and EOG (L-R), sampled at 125Hz and 50Hz respectively.

To ensure high-quality learning signals and consistency across subjects, we adopt a rigorous preprocessing pipeline:

• Stage Consolidation: Following established precedent [12], we merge N3 and N4 into a single deep sleep class. Movement and unscored segments are discarded to maintain label integrity.
• Subject-Level Filtering: Subjects missing at least one of the five AASM-standardized sleep stages (Wake, REM, N1, N2, N3) are excluded. This guarantees representation completeness in downstream supervised training.
• Edge Trimming: Since prolonged wakefulness often occurs at recording boundaries, we symmetrically trim the edges of recordings where Wake dominates other stages:

  $$n_{\mathrm{trim}}={\displaystyle \frac{N_{\mathrm{wake}}-N_{2\mathrm{nd}-\mathrm{dominant}}}{2}}$$
• Resampling and Filtering: Both EEG and EOG signals are resampled to 100Hz. A FIR bandpass filter is applied: [0.3–40] Hz for EEG, [0.3–23] Hz for EOG. This removes both low-frequency drift and high-frequency noise.
• Spectral Feature Extraction: We perform Short-Time Fourier Transform (STFT) using a 2-second Hamming window and 1-second stride (256-point window). This yields 128-dimensional frequency features per frame.
• Windowing: The entire signal is segmented into non-overlapping 30-second epochs. Each epoch is labeled based on a majority-vote strategy among overlapping frames.
• Data Partitioning: A stratified 70/30 split is used for training and testing. From the training set, 100 subjects are reserved for validation to ensure temporal separation and subject-independence.

3.2. Hierarchical Temporal Transformer Design

A unique architectural aspect of MedFuseSleep lies in its two-level modeling of temporal dependencies. Each modality is processed using a dual-stage transformer structure—an inner transformer that models fine-grained STFT dynamics within 30-second windows, and an outer transformer that captures transitions across adjacent windows.

Let $X_m\in {\mathbb{R}}^{B\times T\times D}$ be the STFT features for modality m, with B as batch size, $T=29$ (number of time frames per 30s window), and $D=128$ as spectral dimension. The transformer operates as follows:

$$\begin{array}{cc}\hfill {\mathrm{att}}_h\left(X\right)& =\mathrm{softmax}\left({\displaystyle \frac{W_h^{Q}X{\left(W_h^{K}X\right)}^{\top }}{\sqrt{d_k}}}\right)W_h^{V}X\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \mathrm{MHSA}\left(X\right)& =\mathrm{concat}[{\mathrm{att}}_1,...,{\mathrm{att}}_H]W^O\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill Z& =\mathrm{LayerNorm}(X+\mathrm{MHSA}(X\left)\right)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill \mathrm{FF}\left(Z\right)& =\mathrm{ReLU}(Z{W}_1^F+b_1^F)W_2^F+b_2^F\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill \mathrm{Output}& =\mathrm{LayerNorm}(Z+\mathrm{FF}(Z\left)\right)\hfill \end{array}$$

(5)

We set $d_{model}=128$, $d_k=16$, $d_{ff}=1024$, and $H=8$ heads. Each transformer layer is equipped with learnable relative positional encodings [58] to retain intra-frame ordering.

The inner transformer processes spectral sequences within each 30s window. Its output is aggregated using a learnable [CLS] token to form a window-level embedding $z_m^{\left[l\right]}\in {\mathbb{R}}^d$.

The outer transformer then encodes the sequence of window embeddings $\{z^{[l-w]},...,z^{[l+w]}\}$ for $w=10$, effectively modeling a 10.5-minute temporal context. This enables the model to contextualize micro-structure within macro-sleep patterns.

Figure 2. Details of the Transformer architecture.

Figure 2. Details of the Transformer architecture.

3.3. Coordinated Multimodal Interaction

Instead of naïve fusion, MedFuseSleep adopts a mid-late fusion scheme wherein each modality first builds its own temporal representation, followed by structured interaction via Cross-Modality Attention (CMA) modules.

Given two modality-specific embeddings $Z_1,Z_2\in {\mathbb{R}}^{L\times d}$, cross-attention from $Z_1$ to $Z_2$ is computed as:

$$\begin{array}{cc}\hfill {\mathrm{CA}}_h(Z_1\to Z_2)& =\mathrm{softmax}\left({\displaystyle \frac{W_h^Q{Z}_2{\left(W_h^K{Z}_1\right)}^{\top }}{\sqrt{d_k}}}\right)W_h^V{Z}_1\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill \mathrm{CA}(Z_1\to Z_2)& =\mathrm{concat}[{\mathrm{CA}}_1,...,{\mathrm{CA}}_H]W^O\hfill \end{array}$$

(7)

Each modality attends to latent cues in the other modality to form a modality-grounded feature vector. These representations are then summed:

$$Z_{\mathrm{fusion}}=\sum _{m=1}^M{Z}^{\left(m\right)}$$

where M is the number of modalities.

This structure ensures modular extensibility: additional modalities (e.g., respiration, EMG) can be seamlessly integrated via additional attention streams.

3.4. Multi-Loss Training Objective

To ensure robustness and generalization under both unimodal and multimodal settings, we employ a composite objective comprising three loss components:

• Cross-Entropy Loss (CE): Supervised loss on the final fusion output:

  $${\mathcal{L}}_{\mathrm{CE}}=\mathrm{CE}(f_{\mathrm{fusion}}\left(Z\right),y)$$
• Multi-View Supervision (MS): Separate heads predict sleep stages using modality-specific outputs:

  $${\mathcal{L}}_{\mathrm{MS}}=\sum _{m=1}^M\mathrm{CE}(f_m\left(Z_m\right),y)$$
• Contrastive Alignment Loss (AL): We use InfoNCE-style alignment [54] to enforce cross-modal consistency. For batch size B, and modality pairs $(Z_i^{\left(1\right)},Z_i^{\left(2\right)})$:

  $${\mathcal{L}}_{\mathrm{AL}}={\lambda }_A\sum _{i=1}^{B}log{\displaystyle \frac{exp\left(\frac{\langle Z_i^{\left(1\right)},Z_i^{\left(2\right)}\rangle }{\tau }\right)}{{\sum }_{j=1}^{B}exp\left(\frac{\langle Z_i^{\left(1\right)},Z_j^{\left(2\right)}\rangle }{\tau }\right)}}$$

  where $\tau$ is a temperature coefficient and ${\lambda }_A=0.1$.

The total loss is then:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{CE}}+{\mathcal{L}}_{\mathrm{MS}}+{\mathcal{L}}_{\mathrm{AL}}$$

3.5. Benchmark Architectures for Comparative Evaluation

We compare MedFuseSleep against two simplified fusion architectures:

• Early Fusion: Raw modality inputs are concatenated before transformer encoding. All interactions are implicitly learned via shared attention layers. However, this structure is brittle to missing modalities and lacks interpretability.
• Mid-Late Fusion (Non-Coordinated): Separate modality encoders are trained independently. Final features are fused via summation. This lacks any intermediate cross-modal interaction and serves as a strong minimalist baseline.

To further isolate the effects of our loss functions, we define MedFuseSleep-Base, which removes ${\mathcal{L}}_{\mathrm{MS}}$ and ${\mathcal{L}}_{\mathrm{AL}}$, retaining only ${\mathcal{L}}_{\mathrm{CE}}$.

4. Experiments

In this section, we rigorously evaluate the performance, robustness, and adaptability of our proposed model, MedFuseSleep, a multimodal fusion framework designed for sleep stage classification using EEG and EOG signals. We organize our experiments into five major aspects: implementation details, comprehensive multimodal performance analysis, evaluation under missing modality scenarios, noise robustness testing, and training under incomplete data conditions. Each subsection is designed to systematically uncover the benefits of MedFuseSleep under increasingly challenging real-world conditions, reflecting its practical utility.

4.1. Training Configuration and Implementation Details

We implemented all models using the PyTorch framework [61] and trained them on a single high-performance GPU. Optimization is carried out using the Adam optimizer [62], with a fixed learning rate of $1\times {10}^{-4}$ and a weight decay of the same magnitude. A cosine annealing schedule [63] is applied with a peak learning rate of 0.03 and 20,000 warm-up steps. Each training session is performed with a batch size of 16, where each outer sequence contains 21 epochs, resulting in a batch label size of 336 [13]. Validation is conducted every 400 steps, and early stopping is applied after 100,000 steps without improvement (roughly 9 epochs).

Each encoder within MedFuseSleep—both unimodal and multimodal—incorporates inner and outer Transformer blocks. Each block comprises 4 layers of post-normalization Transformer architecture with relative positional embeddings added to the attention keys [58]. The architecture uses 128-dimensional input features, 8-headed self-attention, and a 1024-dimensional feed-forward network. Prediction heads consist of 2-layer MLPs with a dropout rate of 0.3. To enhance generalization and parameter efficiency, weight sharing is employed between the self-attention and feedforward layers across unimodal and multimodal branches.

Table 1. Evaluation of multimodal fusion architectures across various modality configurations on SHHS-1. The table compares the performance of different fusion strategies with and without auxiliary learning (AL) and modality-specific supervision (MS). Metrics are reported as mean ± standard deviation over three splits.

Fusion Strategy	Variant	EEG+EOG			EEG Only			EOG Only
		Acc	$\kappa$	MF1	Acc	$\kappa$	MF1	Acc	$\kappa$	MF1
Early Fusion	Vanilla	89.1 ± 0.0	0.847 ± 0.001	81.7 ± 0.3	58.0 ± 1.4	0.403 ± 0.025	42.5 ± 1.9	43.6 ± 10.3	0.201 ± 0.111	29.2 ± 8.1
	+AL	89.2 ± 0.0	0.849 ± 0.000	81.9 ± 0.2	49.7 ± 14.4	0.298 ± 0.179	39.4 ± 13.8	34.0 ± 4.7	0.094 ± 0.066	21.6 ± 9.1
	+MS	89.4 ± 0.1	0.851 ± 0.002	82.1 ± 0.3	81.7 ± 1.1	0.742 ± 0.015	65.8 ± 1.0	77.7 ± 1.5	0.686 ± 0.018	62.4 ± 1.2
	+MS+AL	89.5 ± 0.1	0.853 ± 0.002	82.3 ± 0.2	87.1 ± 1.7	0.820 ± 0.021	79.7 ± 1.6	83.4 ± 3.0	0.770 ± 0.036	74.0 ± 2.3
Mid-Late Fusion	Vanilla	89.1 ± 0.1	0.848 ± 0.002	81.6 ± 0.2	85.4 ± 0.4	0.797 ± 0.006	78.2 ± 0.7	75.4 ± 2.3	0.639 ± 0.039	59.1 ± 2.5
	+AL	89.2 ± 0.1	0.848 ± 0.002	81.7 ± 0.3	85.7 ± 0.3	0.800 ± 0.004	78.2 ± 0.6	74.8 ± 2.3	0.627 ± 0.036	57.2 ± 3.0
	+MS	89.2 ± 0.1	0.849 ± 0.002	81.6 ± 0.1	87.7 ± 0.2	0.828 ± 0.003	80.1 ± 0.2	84.9 ± 0.2	0.787 ± 0.002	74.4 ± 0.2
	+MS+AL	89.3 ± 0.1	0.851 ± 0.002	81.9 ± 0.1	88.0 ± 0.2	0.831 ± 0.003	80.4 ± 0.2	85.2 ± 0.1	0.792 ± 0.002	75.1 ± 0.1
MedFuseSleep (Ours)	+MS+AL	89.5 ± 0.1	0.853 ± 0.002	82.3 ± 0.3	88.2 ± 0.2	0.834 ± 0.003	80.8 ± 0.4	85.3 ± 0.1	0.792 ± 0.001	75.3 ± 0.3
XSleepNet [12]	-	88.8	0.843	82.0	87.6	0.826	80.7	-	-	-
SleePyCo [14]	-	-	-	-	87.9	0.830	80.7	-	-	-
SleepTransformer [13]	-	-	-	-	87.7	0.828	80.1	-	-	-

Table 1. Evaluation of multimodal fusion architectures across various modality configurations on SHHS-1. The table compares the performance of different fusion strategies with and without auxiliary learning (AL) and modality-specific supervision (MS). Metrics are reported as mean ± standard deviation over three splits.

Fusion Strategy	Variant	EEG+EOG			EEG Only			EOG Only
		Acc	$\kappa$	MF1	Acc	$\kappa$	MF1	Acc	$\kappa$	MF1
Early Fusion	Vanilla	89.1 ± 0.0	0.847 ± 0.001	81.7 ± 0.3	58.0 ± 1.4	0.403 ± 0.025	42.5 ± 1.9	43.6 ± 10.3	0.201 ± 0.111	29.2 ± 8.1
	+AL	89.2 ± 0.0	0.849 ± 0.000	81.9 ± 0.2	49.7 ± 14.4	0.298 ± 0.179	39.4 ± 13.8	34.0 ± 4.7	0.094 ± 0.066	21.6 ± 9.1
	+MS	89.4 ± 0.1	0.851 ± 0.002	82.1 ± 0.3	81.7 ± 1.1	0.742 ± 0.015	65.8 ± 1.0	77.7 ± 1.5	0.686 ± 0.018	62.4 ± 1.2
	+MS+AL	89.5 ± 0.1	0.853 ± 0.002	82.3 ± 0.2	87.1 ± 1.7	0.820 ± 0.021	79.7 ± 1.6	83.4 ± 3.0	0.770 ± 0.036	74.0 ± 2.3
Mid-Late Fusion	Vanilla	89.1 ± 0.1	0.848 ± 0.002	81.6 ± 0.2	85.4 ± 0.4	0.797 ± 0.006	78.2 ± 0.7	75.4 ± 2.3	0.639 ± 0.039	59.1 ± 2.5
	+AL	89.2 ± 0.1	0.848 ± 0.002	81.7 ± 0.3	85.7 ± 0.3	0.800 ± 0.004	78.2 ± 0.6	74.8 ± 2.3	0.627 ± 0.036	57.2 ± 3.0
	+MS	89.2 ± 0.1	0.849 ± 0.002	81.6 ± 0.1	87.7 ± 0.2	0.828 ± 0.003	80.1 ± 0.2	84.9 ± 0.2	0.787 ± 0.002	74.4 ± 0.2
	+MS+AL	89.3 ± 0.1	0.851 ± 0.002	81.9 ± 0.1	88.0 ± 0.2	0.831 ± 0.003	80.4 ± 0.2	85.2 ± 0.1	0.792 ± 0.002	75.1 ± 0.1
MedFuseSleep (Ours)	+MS+AL	89.5 ± 0.1	0.853 ± 0.002	82.3 ± 0.3	88.2 ± 0.2	0.834 ± 0.003	80.8 ± 0.4	85.3 ± 0.1	0.792 ± 0.001	75.3 ± 0.3
XSleepNet [12]	-	88.8	0.843	82.0	87.6	0.826	80.7	-	-	-
SleePyCo [14]	-	-	-	-	87.9	0.830	80.7	-	-	-
SleepTransformer [13]	-	-	-	-	87.7	0.828	80.1	-	-	-

4.2. Evaluation Under Standard Multimodal Settings

We first compare the performance of three different fusion architectures: Early, Mid-Late, and MedFuseSleep, each evaluated in both unimodal and multimodal configurations. The addition of two auxiliary losses—Alignment Loss (AL) and Modality-Specific Loss (MS)—is investigated. As detailed in Table, we find that:

• Without any auxiliary loss, all three fusion models achieve competitive performance, with Mid-Late slightly outperforming others in the unimodal condition.
• The addition of MS loss consistently improves classification performance across all models, particularly in unimodal EEG or EOG testing scenarios.
• The addition of AL further enhances interaction-aware learning in the Early and MedFuseSleep models, particularly in multimodal conditions.
• When both AL and MS are included, MedFuseSleep achieves state-of-the-art results, outperforming strong baselines including XSleepNet [12], SleepTransformer [13], and SleePyCo [14].

This indicates the synergy created by jointly optimizing modality-specific and alignment-aware objectives, effectively bridging gaps in representation across modalities.

4.3. Robustness to Missing Modalities

To assess the real-world utility of MedFuseSleep, we evaluate its performance when one of the modalities (either EEG or EOG) is unavailable during inference. Notably, we do not retrain the model in this case but simply evaluate it by masking out one modality. This simulates common clinical conditions such as device malfunctions or sensor detachments.

Our findings demonstrate:

• Mid-Late fusion models degrade the least under missing modality conditions, likely due to their architectural separation between modalities.
• AL alone leads to instability in the Early fusion design, while its combination with MS loss alleviates this issue.
• MedFuseSleep significantly outperforms all other fusion strategies under missing modality settings, exceeding even unimodal specialist models trained on a single modality.

These results underscore the adaptive capacity of MedFuseSleep to recover useful information from partial inputs, thanks to its multi-objective training that encourages both modality alignment and independent prediction capability.

Table 2. Performance comparison of multimodal fusion variants on the subset of SHHS-1 containing corrupted or noisy EEG/EOG signals. The models are evaluated under real-world degraded conditions to assess their robustness. Metrics reported include Accuracy, Cohen’s $\kappa$, and Macro-F1, averaged over three runs.

Fusion Strategy	Variant	Acc	$\mathit{\kappa }$	MF1
Unimodal	EEG Only	56.1 ± 0.019	0.351 ± 0.029	43.7 ± 3.5
	EOG Only	80.6 ± 2.1	0.722 ± 0.030	69.5 ± 2.0
Early Fusion	Vanilla	77.9 ± 2.9	0.669 ± 0.046	68.6 ± 4.4
	+AL	81.1 ± 1.2	0.730 ± 0.017	70.4 ± 1.1
	+MS	81.1 ± 1.0	0.730 ± 0.015	70.2 ± 1.2
	+MS+AL	83.1 ± 0.5	0.758 ± 0.007	72.2 ± 0.6
Mid-Late Fusion	Vanilla	81.7 ± 0.3	0.702 ± 0.007	73.9 ± 0.4
	+AL	82.2 ± 0.7	0.746 ± 0.009	70.2 ± 0.7
	+MS	84.2 ± 0.5	0.774 ± 0.008	73.2 ± 1.0
	+MS+AL	83.8 ± 1.1	0.769 ± 0.016	73.0 ± 1.5
MedFuseSleep (Ours)	+MS+AL	84.0 ± 1.2	0.771 ± 0.017	73.0 ± 0.7
XSleepNet [12]	-	75.5 ± 2.6	0.641 ± 0.040	61.7 ± 4.3

Table 2. Performance comparison of multimodal fusion variants on the subset of SHHS-1 containing corrupted or noisy EEG/EOG signals. The models are evaluated under real-world degraded conditions to assess their robustness. Metrics reported include Accuracy, Cohen’s $\kappa$, and Macro-F1, averaged over three runs.

Fusion Strategy	Variant	Acc	$\mathit{\kappa }$	MF1
Unimodal	EEG Only	56.1 ± 0.019	0.351 ± 0.029	43.7 ± 3.5
	EOG Only	80.6 ± 2.1	0.722 ± 0.030	69.5 ± 2.0
Early Fusion	Vanilla	77.9 ± 2.9	0.669 ± 0.046	68.6 ± 4.4
	+AL	81.1 ± 1.2	0.730 ± 0.017	70.4 ± 1.1
	+MS	81.1 ± 1.0	0.730 ± 0.015	70.2 ± 1.2
	+MS+AL	83.1 ± 0.5	0.758 ± 0.007	72.2 ± 0.6
Mid-Late Fusion	Vanilla	81.7 ± 0.3	0.702 ± 0.007	73.9 ± 0.4
	+AL	82.2 ± 0.7	0.746 ± 0.009	70.2 ± 0.7
	+MS	84.2 ± 0.5	0.774 ± 0.008	73.2 ± 1.0
	+MS+AL	83.8 ± 1.1	0.769 ± 0.016	73.0 ± 1.5
MedFuseSleep (Ours)	+MS+AL	84.0 ± 1.2	0.771 ± 0.017	73.0 ± 0.7
XSleepNet [12]	-	75.5 ± 2.6	0.641 ± 0.040	61.7 ± 4.3

4.4. Handling Noisy Modalities

In clinical settings, physiological signals are frequently contaminated due to electrode detachment or patient movement. To test robustness against noisy inputs, we constructed a noisy subset of SHHS-1 by selecting patients whose EEG or EOG channels exhibit anomalously high standard deviation (STD) over long periods (>40% of total recording time).

Table 2 shows the model performance on these corrupted samples. We observe:

• Unimodal EEG models experience the most significant degradation in performance.
• MedFuseSleep and Mid-Late maintain superior stability across noisy input scenarios, confirming the benefit of modular architecture and multi-loss supervision.
• Early fusion is more sensitive to modality noise without auxiliary supervision; however, adding AL and MS mitigates this.
• MedFuseSleep achieves the best accuracy and Macro-F1 in noisy conditions, closely followed by Mid-Late.

These findings validate that MedFuseSleep effectively leverages complementary modality information to suppress noise, showcasing its reliability in real-world applications.

4.5. Training with Modality-Incomplete Data

In this final experimental paradigm, we evaluate MedFuseSleep’s ability to benefit from incomplete training datasets—where many samples contain only one modality. Using a base of 100 multimodal patients (with AL and MS computed), we incrementally add unimodal data points (EEG, EOG, or both from different patients) without computing cross-modal losses for these.

We summarize our findings as follows:

• Adding unimodal patients improves performance across all predictors, particularly when their number is comparable to or slightly exceeds the multimodal subset.
• When both unimodal streams are added simultaneously, the model generalizes better even without paired inputs, suggesting robust shared latent representations.
• Extreme imbalance—where unimodal data outnumbers multimodal data substantially—leads to a decrease in complementary modality performance, due to weak supervision in cross-modal alignment.

Thus, MedFuseSleep can effectively utilize heterogeneous and incomplete datasets—a common scenario in sleep staging and other biomedical tasks—provided that the imbalance between unimodal and multimodal samples is controlled. This positions MedFuseSleep as a highly practical solution for low-resource, real-world deployments.

In conclusion, our extensive experimental study demonstrates that MedFuseSleep achieves state-of-the-art performance on SHHS-1, is robust to missing and noisy modalities, and can effectively leverage incomplete training data. These findings collectively validate the design principles of our fusion strategy and underscore its applicability in challenging clinical environments.

5. Conclusion and Future Directions

In this study, we introduce MedFuseSleep, a robust and flexible multimodal framework designed to handle incomplete and noisy physiological data during sleep stage classification. The proposed model is architected to not only tolerate missing modalities during inference but also leverage the redundancy and complementarity between modalities during training. By incorporating a combination of Coordinate-aware Representation fusion and a carefully designed multi-objective loss function, MedFuseSleep demonstrates superior performance compared to both unimodal and other multimodal baselines.

Empirical evaluations across both standard and noise-augmented variants of the SHHS-1 dataset validate the generalizability and resilience of our method. Notably, MedFuseSleep achieves state-of-the-art results across a variety of experimental conditions, slightly outperforming existing competitive approaches in both multimodal (EEG + EOG) and unimodal (EEG-only, EOG-only) configurations. This robustness to missing or degraded inputs is particularly vital in real-world clinical applications where ideal data acquisition conditions are rarely met.

The training of MedFuseSleep with a joint loss that includes both modality-specific and shared-objective components significantly enhances its capacity to generalize under noisy conditions. Our results provide evidence that introducing data examples with missing modalities during training can improve the model’s ability to adapt during inference—an insight that challenges the conventional practice of only training models with complete modality configurations. This paradigm shift has implications not only for sleep staging but also for broader multimodal tasks where sensor reliability is an issue. Our key findings of this work include:

• Cross-modality benefit: Incorporating multiple modalities during training—even when some are absent at inference—provides a consistent performance uplift across all evaluation settings.
• Effective representation fusion: The Coordinate-aware fusion strategy employed by MedFuseSleep captures semantically aligned yet modality-specific information, enabling the model to effectively synthesize knowledge from partially available data streams.
• Robust training strategies: Training with samples containing incomplete modalities fosters resilience in downstream predictions, even under high signal noise or dropout scenarios.

To further support these insights, we plan to extend our future work in several directions. First, we aim to generalize MedFuseSleep to support additional physiological channels beyond EEG and EOG, such as EMG and respiratory signals, to better approximate full polysomnography. Secondly, the integration of self-supervised pretraining on large-scale unannotated sleep recordings may reduce dependence on costly manual labeling, enhancing the scalability of the framework. Additionally, we are exploring dynamic modality dropout schedules as a form of data augmentation, allowing the model to adaptively recalibrate its fusion weights depending on the availability and reliability of input sources.

Furthermore, we envision adapting MedFuseSleep for deployment in real-time embedded systems, such as wearable health monitoring devices. This involves lightweight model distillation and latency-aware architecture compression, ensuring that the performance gains achieved do not come at the cost of computational efficiency.

In conclusion, MedFuseSleep marks a promising step toward practical and resilient multimodal AI systems in clinical neurophysiology. The lessons drawn from its design and evaluation may inspire future frameworks in multimodal representation learning, particularly in domains where robustness to missing or corrupted input is critical.