Identifying Common Semantics Across Modalities via Contrastive Latent Alignment

1. Introduction

The field of multimodal representation learning has become increasingly central in modern machine learning, catalyzed by the availability of large-scale, weakly labeled datasets that integrate diverse modalities. These modalities—ranging from image-text combinations [46,51,53] to richer compositions involving audio, video, and language [8]—are often co-observed in time or context, naturally encoding weak supervision through co-occurrence [12,38,54]. Although each modality captures a distinct facet of the same event or concept, they frequently share an underlying semantic core.

Contrastive learning [20,43] has become a leading technique in leveraging such data to uncover shared latent structures. By encouraging paired observations from different modalities to yield similar representations, contrastive methods capitalize on the supervision inherent in co-occurrence. This approach has led to impressive results in several domains. A prominent example is CLIP [46], which has laid the groundwork for various applications such as text-to-image synthesis [47,48,49,50].

Despite these empirical achievements, our theoretical understanding of contrastive learning remains limited. A growing body of work seeks to analyze contrastive frameworks through the lens of latent factor models and causality. For instance, recent studies [18,61,66] propose that contrastive objectives can implicitly invert the generative process, recovering shared latent representations under certain assumptions, such as independence among sources. In particular, multi-view independent component analysis (ICA) has been used to show that contrastive learning can invert nonlinear generative functions to isolate the underlying semantic variables [66].

However, the bulk of these identifiability analyses are grounded in the multi-view paradigm, where the data modalities are generated through similar or even identical mechanisms—for example, different camera angles capturing the same scene. Although this setting simplifies analysis, it fails to encompass the intricacies of real-world multimodal data, where different sensors (e.g., microphones versus cameras) encode inherently distinct physical phenomena. von Kügelgen et al. [61] partially alleviate this by introducing block-wise latent dependencies, yet still assume a shared generative process across views.

In contrast, we consider a setting where each modality is produced through its own modality-specific, nonlinear transformation, applied to both shared and unique latent variables. This better reflects the nature of real-world multimodal environments—where, for instance, vision captures spatial structure, while audio captures frequency dynamics. Shared semantic content, such as object identity or event type, is expressed through fundamentally different low-level encodings in each modality. Our generative model accommodates such heterogeneity, allowing for interdependencies without requiring structural uniformity.

To address this setting, we propose CIPHER, a contrastive framework that formally characterizes the identifiability of shared latent factors when modalities are generated independently but interdependently. Our formulation clarifies when contrastive learning is theoretically justified and sheds light on the mechanisms enabling its empirical success. These findings not only strengthen the theoretical basis of multimodal contrastive learning but also inform the design of more robust and interpretable model architectures.

The remainder of this paper presents a comprehensive development of our theory and supporting experiments. We begin by revisiting foundational concepts in identifiability and contrastive methods. Next, we introduce the generalized multimodal generative model and derive our key identifiability theorem. Extensive empirical evaluations—spanning synthetic simulations and real-world image-text datasets—demonstrate the practical utility of our theoretical insights. We conclude with a discussion on the broader implications of our work and suggest directions for future research.

2. Related Work

In this section, we review and synthesize relevant literature that informs our investigation into the identifiability of shared latent semantics in multimodal contrastive learning. We organize this discussion across several dimensions, including theoretical underpinnings of contrastive learning, identifiability in multi-view representations, multimodal embedding approaches, the role of weak supervision, and information-theoretic and causal insights. Our study builds upon and generalizes these prior efforts to accommodate a more comprehensive and heterogeneous multimodal setting.

2.1. Theoretical Foundations of Contrastive Learning

Contrastive learning has become a foundational technique in self-supervised representation learning, originating from methods such as noise-contrastive estimation (NCE) [20] and the InfoNCE objective [43], which aim to increase similarity between positive samples while distinguishing them from negatives. Various practical frameworks, including SimCLR, MoCo, and BYOL, have demonstrated the broad applicability of contrastive paradigms in visual and general representation learning. However, these approaches were initially developed based on empirical intuition, with rigorous theoretical insights arriving only more recently.

Recent advancements have attempted to interpret the effectiveness of contrastive learning through the lens of mutual information estimation [45] and kernel dependence measures. Yet, simply maximizing mutual information does not necessarily imply disentanglement or the recovery of interpretable latent factors. Such objectives typically promote global representation alignment without guaranteeing interpretability at a finer granularity. As a result, the research focus has gradually shifted toward understanding the implications of contrastive training on the underlying latent structures.

2.2. Identifiability in Multi-View Latent Representation Learning

An important strand of work has examined the identifiability of latent representations within the multi-view learning paradigm. Early developments in this space stem from independent component analysis (ICA) and its nonlinear variants [24], which aim to recover statistically independent components subject to nonlinear transformations. These ideas have since been extended to dual-view settings, where observations from each view arise from nonlinear functions of shared latent variables. Notably, Gresele et al. [18] and Zimmermann et al. [66] provided theoretical evidence that contrastive learning can invert such nonlinear generative processes under independence assumptions, even when dealing with high-dimensional data.

More recently, the work of von Kügelgen et al. [61] advanced the theory by relaxing full independence assumptions and allowing for structured block-wise dependencies among latent factors. Their analysis introduced the notion of block identifiability, thereby broadening the class of generative models for which contrastive learning can recover meaningful latent semantics. These theoretical insights provide an essential foundation for analyzing factor recovery in paired data.

Nevertheless, all of these contributions implicitly assume that both views stem from structurally equivalent or symmetric generative mechanisms. Such assumptions position the problem closer to multi-view learning rather than truly heterogeneous multimodal learning. In contrast, our study relaxes this symmetry and explicitly models each modality with its own generation function, incorporating both shared and modality-specific latent structures—thus aligning with more realistic multimodal contexts.

2.3. Multimodal Embedding and Representation Learning

Multimodal representation learning seeks to synthesize information from different modalities such as vision, language, and audio [8]. Early approaches in this domain involved constructing shared latent spaces [42], facilitating cross-modal retrieval and fusion through aligned encoders. More recently, large-scale vision-language models like CLIP [46] and ALIGN have demonstrated that contrastive learning on massive paired datasets can yield highly transferable multimodal embeddings.

Despite these practical successes, the theoretical understanding of what contrastive losses actually achieve in multimodal contexts remains limited. Many approaches implicitly assume a common semantic space across modalities but rarely interrogate the conditions under which such spaces are identifiable or interpretable. Our work fills this gap by formalizing the recovery of shared semantic content under asymmetric and modality-specific generation processes, thereby expanding the theoretical underpinnings of contrastive multimodal learning.

2.4. Weak Supervision and Latent Structure Discovery

Weak supervision typically refers to learning scenarios where annotations are sparse, noisy, or indirectly inferred. In the multimodal domain, co-occurrence in paired data—such as image-caption pairs—naturally embodies a weak supervisory signal. Prior work [38,54] explores how latent structures can be recovered using such signals, often under assumptions of sparsity, minimality, or causal structure.

Contrastive learning seamlessly fits into this paradigm by utilizing pairwise relationships without the need for explicit supervision. Existing research indicates that even in the absence of strong labels, meaningful latent representations can be learned through contrastive signals. Our work extends this principle by demonstrating that even under the compounded challenges of modality-specific latent variables and nonlinear data generation, contrastive learning can still succeed in isolating shared semantic content.

2.5. Information-Theoretic Interpretations

Information-theoretic principles offer another avenue for interpreting contrastive learning. The InfoMax principle and its derivative methods, including Deep InfoMax, aim to retain high mutual information between inputs and representations. While mutual information is a theoretically appealing objective, it is difficult to compute directly, leading to the adoption of proxies such as InfoNCE [43]. As shown in Poole et al. [45], these surrogates are limited in their fidelity and operate under specific approximations.

Additionally, recent work has connected contrastive learning with the Information Bottleneck framework, which posits that effective representations should compress inputs while preserving task-relevant information. Yet, these analyses often overlook the implications of modality-specific generative priors and cross-modal heterogeneity. Our framework explicitly addresses such complexities and establishes that identifiability can still be achieved when traditional assumptions about information symmetry do not hold.

2.6. Causal and Structural Perspectives in Representation Learning

Finally, recent studies advocate for a causal viewpoint in representation learning, suggesting that discovering disentangled or semantically meaningful features requires understanding the underlying data-generating mechanisms. In the multimodal context, the causal links between latent factors and observed variables can vary significantly across modalities, further complicating identifiability. Our approach resonates with this perspective by introducing a structural latent variable model tailored to multimodal data and proving that contrastive learning is capable of identifying shared causes, even when generative pathways differ across modalities. In summary, although previous work has extensively examined contrastive learning under symmetric or multi-view conditions, our study pushes the boundary toward more realistic scenarios involving heterogeneous multimodal generation. By bridging theoretical tools from weak supervision, information theory, and causal inference, we provide a principled framework for understanding when and how contrastive learning recovers shared semantic factors in multimodal settings.

3. Theoretical Foundations

3.1. Latent Factor Identifiability in Multi-Source Systems

The concept of identifiability serves as a cornerstone across numerous branches of statistical learning theory, spanning topics such as independent component analysis (ICA), structural causal inference, and general inverse problems [35]. In the ICA setting, the classical formulation is given by $\mathbf{x}=\mathbf{f}\left(\mathbf{z}\right)$, wherein observed data $\mathbf{x}$ arise from latent variables $\mathbf{z}$ through an unknown nonlinear mixing function $\mathbf{f}$. The overarching goal is to recover or approximate the inverse of $\mathbf{f}$ based solely on observed samples of $\mathbf{x}$, typically without auxiliary supervision or explicit structural constraints.

In this context, identifiability refers to the extent to which the latent variables $\mathbf{z}$ can be uniquely determined—up to permissible invariances—given the observations $\mathbf{x}$. While linear ICA, assuming independence among the latent sources, enjoys solid identifiability guarantees under relatively mild assumptions, the situation becomes considerably more complex in the nonlinear case. A classical negative result by Hyvärinen and Pajunen [26] states that under generic nonlinear mappings and i.i.d. sampling, the recovery of latent variables is provably impossible without further assumptions or additional modalities.

Recent research has substantially broadened the identifiability landscape by relaxing strict independence assumptions and incorporating various auxiliary signals. These include side information [25,27], temporal dependencies or contextual priors [31], and structured multi-view observation models [18,38,66]. Multi-view approaches, in particular, posit the existence of multiple views derived from the same latent source, with each view undergoing a distinct nonlinear transformation.

The generative process underlying multi-view nonlinear ICA is typically expressed as:

$$\mathbf{z}\sim p_{\mathbf{z}},{\mathbf{x}}_1={\mathbf{f}}_1\left(\mathbf{z}\right),{\mathbf{x}}_2={\mathbf{f}}_2\left(\mathbf{z}\right),$$

(1)

where a shared latent vector $\mathbf{z}$ is mapped to two distinct observable views ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ via nonlinear transformations ${\mathbf{f}}_1$ and ${\mathbf{f}}_2$, respectively. This dual-observation setup enables the resolution of inherent ambiguities introduced by nonlinear mixing [18]. Specifically, when the transformation functions ${\mathbf{f}}_1$ and ${\mathbf{f}}_2$ are sufficiently non-redundant and dissimilar in structure, their differential characteristics can be exploited to disentangle the latent semantic space.

Prior contributions vary in their assumptions about the statistical nature of $\mathbf{z}$. While some works assume mutual independence among all latent dimensions [18,38,55], others adopt a more relaxed model allowing for structured intra-block dependencies [39,40]. Moreover, most frameworks employ distinct mixing functions for each view, operating over a common latent representation. As discussed in [18,39], such view-specific mappings can be seen as a conceptual stand-in for modality-specific transformations in truly multimodal setups.

A significant advancement by von Kügelgen et al. [61] extends identifiability guarantees to situations involving partial dependence among latent components. Their framework shows that, under certain regularity conditions, one can achieve recovery of the latent structure up to a block-wise indeterminacy, even when full statistical independence is absent.

Building on these foundations, our work generalizes the analysis to a more realistic multimodal context, wherein both the mixing functions ${\mathbf{f}}_1,{\mathbf{f}}_2$ and the latent spaces ${\mathbf{z}}_1,{\mathbf{z}}_2$ are modality-dependent and structurally diverse. We formally prove that contrastive learning can still isolate shared semantic factors, albeit up to a block-level ambiguity, despite the introduction of modality-specific latent structures and complex nonlinear mappings. This result reinforces the efficacy of contrastive frameworks in heterogeneous multimodal systems and contributes to a more robust theoretical understanding.

3.2. Contrastive Objectives for Cross-Modal Representation Alignment

Contrastive learning [20,43] has established itself as a powerful strategy for self-supervised learning. At its core, the technique seeks to distinguish between positive sample pairs—jointly sampled from $p_{{\mathbf{x}}_1,{\mathbf{x}}_2}$—and negative pairs formed by independently drawing from the product distribution $p_{{\mathbf{x}}_1}·p_{{\mathbf{x}}_2}$. The InfoNCE loss [43] formalizes this objective as follows:

$${\mathcal{L}}_{\mathrm{InfoNCE}}({\mathbf{g}}_1,{\mathbf{g}}_2)={\mathbb{E}}_{{\{{\mathbf{x}}_1^i,{\mathbf{x}}_2^i\}}_{i=1}^K\sim p_{{\mathbf{x}}_1,{\mathbf{x}}_2}}\left[-\sum _{i=1}^{K}log{\displaystyle \frac{exp\{\mathrm{s}\mathrm{i}\mathrm{m}({\mathbf{g}}_1\left({\mathbf{x}}_1^i\right),{\mathbf{g}}_2\left({\mathbf{x}}_2^i\right))/\tau \}}{{\sum }_{j=1}^{K}exp\{\mathrm{s}\mathrm{i}\mathrm{m}({\mathbf{g}}_1\left({\mathbf{x}}_1^i\right),{\mathbf{g}}_2\left({\mathbf{x}}_2^j\right))/\tau \}}}\right],$$

(2)

where ${\mathbf{g}}_1$ and ${\mathbf{g}}_2$ are modality-specific encoders, $\tau$ denotes a temperature hyperparameter, and $sim(·,·)$ represents a similarity metric (e.g., cosine or Euclidean similarity). The effectiveness of this objective improves as the number of negatives ($K-1$) increases, enhancing representation discrimination and alignment.

From an information-theoretic viewpoint, InfoNCE can be interpreted as a variational lower bound on mutual information $I({\mathbf{g}}_1\left({\mathbf{x}}_1\right);{\mathbf{g}}_2\left({\mathbf{x}}_2\right))$ [45]. Nonetheless, maximizing this quantity alone does not guarantee that learned representations will capture semantically meaningful or disentangled latent factors. Unless encoder expressiveness and appropriate regularization are ensured, the optimization may lead to degenerate solutions or overlook fine-grained semantics.

As demonstrated in Wang and Isola [62], when Euclidean distance is used as the similarity metric and $\tau =1$, InfoNCE approximates a combined objective that balances alignment with representation diversity:

$${\mathcal{L}}_{\mathrm{AlignMaxEnt}}\left(\mathbf{g}\right)={\mathbb{E}}_{({\mathbf{x}}_1,{\mathbf{x}}_2)\sim p_{{\mathbf{x}}_1,{\mathbf{x}}_2}}\left[{\parallel \mathbf{g}\left({\mathbf{x}}_1\right)-\mathbf{g}\left({\mathbf{x}}_2\right)\parallel }_2\right]-H\left(\mathbf{g}\left(\mathbf{x}\right)\right).$$

(3)

This objective encapsulates two core principles: drawing positive pairs closer in representation space, while preserving global entropy to encourage information retention.

In multimodal scenarios—where ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ may belong to disparate domains such as vision and language—distinct encoders ${\mathbf{g}}_1\ne {\mathbf{g}}_2$ are typically employed to capture domain-specific characteristics. To maintain bidirectional consistency and avoid trivial solutions, a symmetric version of the InfoNCE loss is commonly adopted [46,65]:

$${\mathcal{L}}_{\mathrm{SymInfoNCE}}({\mathbf{g}}_1,{\mathbf{g}}_2)=MM1/2{\mathcal{L}}_{\mathrm{InfoNCE}}({\mathbf{g}}_1,{\mathbf{g}}_2)+MM1/2{\mathcal{L}}_{\mathrm{InfoNCE}}({\mathbf{g}}_2,{\mathbf{g}}_1).$$

(4)

In the asymptotic case, this formulation yields:

$${\mathcal{L}}_{\mathrm{SymAlignMaxEnt}}({\mathbf{g}}_1,{\mathbf{g}}_2)={\mathbb{E}}_{({\mathbf{x}}_1,{\mathbf{x}}_2)\sim p_{{\mathbf{x}}_1,{\mathbf{x}}_2}}\left[{\parallel {\mathbf{g}}_1\left({\mathbf{x}}_1\right)-{\mathbf{g}}_2\left({\mathbf{x}}_2\right)\parallel }_2\right]-MM1/2\left(H\left({\mathbf{g}}_1\left({\mathbf{x}}_1\right)\right)+H\left({\mathbf{g}}_2\left({\mathbf{x}}_2\right)\right)\right),$$

(5)

where entropy terms are typically estimated using empirical distributions over negative samples and function as regularizers to avoid representational collapse [62].

In our theoretical exploration, we utilize the continuous formulations ${\mathcal{L}}_{\mathrm{AlignMaxEnt}}$ and ${\mathcal{L}}_{\mathrm{SymAlignMaxEnt}}$ to derive identifiability results. For empirical implementation, we employ their finite-sample approximations ${\mathcal{L}}_{\mathrm{InfoNCE}}$ and ${\mathcal{L}}_{\mathrm{SymInfoNCE}}$, respectively. Our analysis reveals that even when modalities differ significantly in structure and encoding requirements, properly designed contrastive losses are sufficient to isolate shared semantic representations—underscoring their versatility in heterogeneous multimodal environments.

4. MIRAGE: Multimodal Generative Framework with Invariant Representation

This section provides a detailed exposition of the proposed MIRAGE framework, which stands for Multimodal Invariant Representation-based Alignment for Generative Encoding. MIRAGE is a principled generative model designed to disentangle the latent structure of multimodal data into three distinct components: modality-invariant shared content, stochastic style variations, and modality-specific structural residuals. We present a rigorous formalization of the latent space, articulate the generative and variational mechanisms underpinning our framework, and culminate in a contrastive training scheme that provably recovers identifiable shared semantics across modalities. This formulation aims to model real-world multimodal data in which modalities exhibit distinct generative mechanisms yet share underlying semantics.

4.1. Latent Structural Decomposition and Multimodal Encoders

We hypothesize that observations from different modalities are generated from a shared latent structure, which we model through a composite latent space $\mathbf{z}\in \mathcal{Z}\subseteq {\mathbb{R}}^n$. Departing from conventional ICA-based approaches, we propose a semantically disentangled decomposition of $\mathbf{z}$ that reflects both modality-invariant and modality-specific aspects of the data generation process. Specifically, we define:

$$\mathbf{z}=(\mathbf{c},\mathbf{s},{\mathbf{m}}_1,{\mathbf{m}}_2),$$

(6)

where:

−

$\mathrm{c}\in {\mathbb{R}}^{n_c}$ encodes the core content, capturing semantic information invariant across modalities (e.g., object identity, scene configuration);

−

$\mathbf{s}\in {\mathbb{R}}^{n_s}$ represents the mutable style, accounting for stochastic variability or noise (e.g., lighting, pitch, accent, tone);

−

${\mathbf{m}}_1,{\mathbf{m}}_2$ are modality-specific residuals, capturing idiosyncratic structural information unique to each sensory channel (e.g., color spectrum for vision vs. frequency modulation for audio).

For each modality, we derive specialized latent codes:

$${\mathbf{z}}_1=(\mathbf{c},\mathbf{s},{\mathbf{m}}_1),{\mathbf{z}}_2=(\mathbf{c},\tilde{\mathbf{s}},{\mathbf{m}}_2),$$

(7)

where $\tilde{\mathbf{s}}$ denotes a perturbation of $\mathbf{s}$, used to model stochastic variations across modalities. We posit that data generation proceeds via smooth, invertible transformations ${\mathbf{f}}_1$ and ${\mathbf{f}}_2$:

$$\mathbf{z}\sim p_{\mathbf{z}},{\mathbf{x}}_1={\mathbf{f}}_1\left({\mathbf{z}}_1\right),{\mathbf{x}}_2={\mathbf{f}}_2\left({\mathbf{z}}_2\right),$$

(8)

where each ${\mathbf{f}}_i$ is a modality-specific diffeomorphism ${\mathbf{f}}_i:{\mathcal{Z}}_i\to {\mathcal{X}}_i$ with smooth inverse, ensuring information-preserving mappings.

To encode causal dependencies in latent space, we impose the following structured prior:

$$p_{\mathbf{z}}\left(\mathbf{z}\right)=p_{\mathrm{c}}\left(\mathrm{c}\right)·p_{\mathbf{s}|\mathrm{c}}\left(\mathbf{s}\right|\mathrm{c})·p_{{\mathbf{m}}_1}\left({\mathbf{m}}_1\right)·p_{{\mathbf{m}}_2}\left({\mathbf{m}}_2\right),$$

(9)

which introduces a directed relationship $\mathrm{c}\to \mathbf{s}$, reflecting the intuitive notion that high-level content partially dictates stylistic realization (e.g., object identity constrains color tone or vocal accent).

Latent Encoding via Structured Encoders.

We define modality-specific encoders ${\mathbf{g}}_1$ and ${\mathbf{g}}_2$ mapping each observation into its respective latent decomposition. Each encoder is designed to yield factorized outputs:

$${\mathbf{g}}_1\left({\mathbf{x}}_1\right)=({\widehat{\mathrm{c}}}_1,{\widehat{\mathbf{s}}}_1,{\widehat{\mathbf{m}}}_1),$$

(10)

which mirror the theoretical latent variables. These encoders are trained to enforce cross-modal alignment in the $\mathrm{c}$ space while allowing flexibility in style and residual dimensions.

Information-Regularized Prior.

To control latent complexity and prevent overfitting, we incorporate a KL divergence penalty that regularizes the posterior $q\left(\mathbf{z}\right|{\mathbf{x}}_1,{\mathbf{x}}_2)$ against an isotropic Gaussian prior:

$${\mathcal{L}}_{\mathrm{KL}}=\mathrm{KL}(q\left(\mathbf{z}\right|{\mathbf{x}}_1,{\mathbf{x}}_2)\parallel \mathcal{N}(0,I)).$$

(11)

This regularization encourages compactness in latent space and stabilizes training.

4.2. Cross-Modal Semantics and Style Perturbation Mechanism

To explicitly capture modality discrepancies, we define a generative mapping between latent representations across modalities via the conditional density $p_{{\mathbf{z}}_2|{\mathbf{z}}_1}$:

$$p_{{\mathbf{z}}_2|{\mathbf{z}}_1}\left({\mathbf{z}}_2\right|{\mathbf{z}}_1)=\delta ({\mathbf{c}}^{\prime }-\mathbf{c})·p_{\tilde{\mathbf{s}}|\mathbf{s}}\left(\tilde{\mathbf{s}}\right|\mathbf{s})·p_{{\mathbf{m}}_2}\left({\mathbf{m}}_2\right),$$

(12)

which models modality shifts through stochastic style perturbation and residual re-sampling. The Dirac delta $\delta$ enforces the preservation of shared content across modalities, anchoring semantic consistency.

Following [61,66], we adopt a structured stochastic perturbation for style:

$$p_{\tilde{\mathbf{s}}|\mathbf{s}}\left(\tilde{\mathbf{s}}\right|\mathbf{s})=\sum _{A\subseteq \{1,...,n_s\}}{p}_A\left(A\right)·\delta ({\tilde{\mathbf{s}}}_{A^c}-{\mathbf{s}}_{A^c})·p_{{\tilde{\mathbf{s}}}_A|{\mathbf{s}}_A}\left({\tilde{\mathbf{s}}}_A\right|{\mathbf{s}}_A),$$

(13)

which enables targeted intervention on subsets of style dimensions, mimicking realistic intermodal augmentations (e.g., partial lighting changes, pitch shifts). The random variable A determines the subset of $\mathbf{s}$ to perturb, ensuring each dimension remains mutable with non-zero probability.

Soft Symmetry via Dual Perturbation.

To avoid directional bias, we symmetrize the process by also modeling $p_{{\mathbf{z}}_1|{\mathbf{z}}_2}$ analogously. During training, this bidirectional perturbation mechanism teaches both encoders to generalize across variations, thus enhancing robustness and generalization.

4.3. Contrastive Objective with Entropy Regularization

Let ${\mathbf{g}}_1$ and ${\mathbf{g}}_2$ denote the encoders for modalities ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$, respectively. To encourage alignment of shared semantic content, we optimize the symmetric InfoNCE objective:

$${\mathcal{L}}_{\mathrm{SymInfoNCE}}({\mathbf{g}}_1,{\mathbf{g}}_2)={\displaystyle \frac{1}{2}}{\mathcal{L}}_{\mathrm{InfoNCE}}({\mathbf{g}}_1,{\mathbf{g}}_2)+{\displaystyle \frac{1}{2}}{\mathcal{L}}_{\mathrm{InfoNCE}}({\mathbf{g}}_2,{\mathbf{g}}_1),$$

(14)

which balances directional contrastive signals and promotes reciprocal embedding consistency.

Its asymptotic formulation serves as a theoretical estimator of contrastive alignment with entropy regularization:

$${\mathcal{L}}_{\mathrm{SymAlignMaxEnt}}({\mathbf{g}}_1,{\mathbf{g}}_2)=\mathbb{E}\left[{\parallel {\mathbf{g}}_1\left({\mathbf{x}}_1\right)-{\mathbf{g}}_2\left({\mathbf{x}}_2\right)\parallel }_2\right]-{\displaystyle \frac{1}{2}}(H\left({\mathbf{g}}_1\left({\mathbf{x}}_1\right)\right)+H\left({\mathbf{g}}_2\left({\mathbf{x}}_2\right)\right)),$$

(15)

which explicitly balances intra-pair proximity with marginal diversity in representation space. This dual-objective prevents collapse and maintains discriminative capacity.

Focusing on semantic alignment, we isolate the content portions of the embeddings ${\widehat{\mathrm{c}}}_1={\mathbf{g}}_1{\left({\mathbf{x}}_1\right)}_{1:n_c}$ and ${\widehat{\mathrm{c}}}_2={\mathbf{g}}_2{\left({\mathbf{x}}_2\right)}_{1:n_c}$, applying the loss exclusively on these blocks:

$${\mathcal{L}}_{\mathrm{contrast}}={\mathcal{L}}_{\mathrm{SymAlignMaxEnt}}({\widehat{\mathrm{c}}}_1,{\widehat{\mathrm{c}}}_2).$$

(16)

This targeted contrastive regularization enhances identifiability of shared factors, and isolates them from noise and style artifacts.

4.4. Auxiliary Disentanglement Objectives

To promote modularity and clarity in the learned latent space, we introduce two auxiliary loss components that enhance disentanglement and reconstructive alignment.

Orthogonality Penalty.

To reduce redundancy and encourage statistical independence between $\mathrm{c}$ and $\mathbf{s}$, we penalize their correlation through:

$${\mathcal{L}}_{\mathrm{ortho}}=\sum _{i=1}^{n_c}\sum _{j=1}^{n_s}{\left({\widehat{c}}_i^{\top }{\widehat{s}}_j\right)}^2.$$

(17)

This regularizer ensures that the shared and style components occupy orthogonal subspaces, facilitating interpretability.

Cycle Consistency.

To ensure bidirectional recoverability between modalities, we enforce cycle consistency:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{cycle}}={\mathbb{E}}_{{\mathbf{x}}_1,{\mathbf{x}}_2}[{\parallel {\mathbf{g}}_1^{-1}\left({\mathbf{g}}_2\left({\mathbf{x}}_2\right)\right)-{\mathbf{x}}_1\parallel }_2^2+{\parallel {\mathbf{g}}_2^{-1}\left({\mathbf{g}}_1\left({\mathbf{x}}_1\right)\right)-{\mathbf{x}}_2\parallel }_2^2].\end{array}$$

(18)

This loss encourages invertibility of the encoding process and promotes latent space coherence across modalities.

4.5. Final Training Objective

Bringing all components together, the overall training loss for MIRAGE is:

$${\mathcal{L}}_{\mathrm{MIRAGE}}={\mathcal{L}}_{\mathrm{contrast}}+{\lambda }_{\mathrm{KL}}{\mathcal{L}}_{\mathrm{KL}}+{\lambda }_{\mathrm{ortho}}{\mathcal{L}}_{\mathrm{ortho}}+{\lambda }_{\mathrm{cycle}}{\mathcal{L}}_{\mathrm{cycle}},$$

(19)

where ${\lambda }_{\mathrm{KL}},{\lambda }_{\mathrm{ortho}},{\lambda }_{\mathrm{cycle}}$ are scalar hyperparameters controlling the influence of each term.

This comprehensive training formulation enables MIRAGE to disentangle shared semantics from modality-specific features while ensuring representation compactness and alignment. In the subsequent section, we formally demonstrate that under mild assumptions, optimizing this objective leads to block-wise identifiability of the shared content variable $\mathrm{c}$.

5. Experiments and Discussions

In this section, we extensively evaluate the efficacy of our proposed method, MIRAGE, in identifying shared content representations across modalities through contrastive learning. Our experiments aim to verify the central claim of block-identifiability in both controlled synthetic settings and complex real-world scenarios. We structure our evaluation into three subsections: (1) controlled numerical simulations with fully known latent structures; (2) analysis on the proposed Multimodal3DIdent dataset constructed from image-text pairs; and (3) ablation and stress testing to assess sensitivity under encoding bottlenecks and discrete style perturbations.

5.1. Synthetic Evaluation with Controlled Latent Structures

We first validate identifiability in a fully controlled setting using synthetic data. This setup allows direct comparison between settings with shared and modality-specific generative mechanisms, verifying the theoretical conditions posed.

Table 1. Performance comparison in original vs multimodal generative settings. We report average $R^2$ across 3 seeds with ± standard deviation. The multimodal setup includes additional latent modality-specific noise components.

Generative Config			${\mathit{R}}^2$ (shared encoder)		Generative Config			${\mathit{R}}^2$
p(chg.)	Stat.	Caus.	Content$\mathrm{c}$	Style s	p(chg.)	Stat.	Caus.	Content$\mathrm{c}$	Style s	$\mathit{Modality}$${\mathbf{m}}_{\mathit{i}}$
1.0	✗	✗	$0.98\pm 0.01$	$0.01\pm 0.00$	1.0	✗	✗	$0.99\pm 0.00$	$0.01\pm 0.00$	$0.00\pm 0.00$
0.75	✗	✗	$0.97\pm 0.01$	$0.02\pm 0.00$	0.75	✗	✗	$0.99\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$
0.75	✓	✗	$0.96\pm 0.02$	$0.47\pm 0.08$	0.75	✓	✗	$0.95\pm 0.02$	$0.55\pm 0.09$	$0.01\pm 0.00$
0.75	✗	✓	$\mathbf{0}.\mathbf{99}\pm 0.01$	$0.72\pm 0.06$	0.75	✗	✓	$0.97\pm 0.01$	$0.81\pm 0.05$	$0.01\pm 0.01$
0.75	✓	✓	$0.96\pm 0.01$	$\mathbf{0}.\mathbf{77}\pm 0.04$	0.75	✓	✓	$0.94\pm 0.02$	$\mathbf{0}.\mathbf{88}\pm 0.06$	$0.01\pm 0.00$
(a) Homogeneous setting					(b) Multimodal setting with $1\ne 2$

Table 1. Performance comparison in original vs multimodal generative settings. We report average $R^2$ across 3 seeds with ± standard deviation. The multimodal setup includes additional latent modality-specific noise components.

Generative Config			${\mathit{R}}^2$ (shared encoder)		Generative Config			${\mathit{R}}^2$
p(chg.)	Stat.	Caus.	Content$\mathrm{c}$	Style s	p(chg.)	Stat.	Caus.	Content$\mathrm{c}$	Style s	$\mathit{Modality}$${\mathbf{m}}_{\mathit{i}}$
1.0	✗	✗	$0.98\pm 0.01$	$0.01\pm 0.00$	1.0	✗	✗	$0.99\pm 0.00$	$0.01\pm 0.00$	$0.00\pm 0.00$
0.75	✗	✗	$0.97\pm 0.01$	$0.02\pm 0.00$	0.75	✗	✗	$0.99\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$
0.75	✓	✗	$0.96\pm 0.02$	$0.47\pm 0.08$	0.75	✓	✗	$0.95\pm 0.02$	$0.55\pm 0.09$	$0.01\pm 0.00$
0.75	✗	✓	$\mathbf{0}.\mathbf{99}\pm 0.01$	$0.72\pm 0.06$	0.75	✗	✓	$0.97\pm 0.01$	$0.81\pm 0.05$	$0.01\pm 0.01$
0.75	✓	✓	$0.96\pm 0.01$	$\mathbf{0}.\mathbf{77}\pm 0.04$	0.75	✓	✓	$0.94\pm 0.02$	$\mathbf{0}.\mathbf{88}\pm 0.06$	$0.01\pm 0.00$
(a) Homogeneous setting					(b) Multimodal setting with $1\ne 2$

The results provide compelling empirical evidence that content-related latent variables can be consistently and accurately recovered in both homogeneous and heterogeneous generative scenarios. Specifically, across a diverse range of configurations—ranging from fully independent factors to those with complex statistical and causal entanglements—the contrastive learning objective remains highly effective in isolating the shared latent content $\mathrm{c}$ from stylistic variation and modality-specific noise.

A particularly notable observation arises in the multimodal configuration, where the generative mechanisms ${\mathbf{f}}_1$ and ${\mathbf{f}}_2$ are structurally distinct. Even in this challenging case, the $R^2$ scores for predicting $\mathrm{c}$ consistently approach unity, indicating that MIRAGE is able to discover and align the content subspace with remarkable fidelity. This finding reinforces the theoretical result, suggesting that modality-specific transformations and nonlinear heterogeneity do not disrupt the ability of contrastive learning to identify the invariant core of cross-modal semantics.

Furthermore, when causal dependencies are introduced between content and style (i.e., $\mathrm{c}\to \mathbf{s}$), we observe a measurable increase in the predictability of style components from the learned representation. This is intuitively expected, as causal dependencies reduce the conditional entropy of $\mathbf{s}$ given $\mathrm{c}$, effectively leaking content-relevant information into the stylistic channels. Despite this leakage, the style predictions remain significantly weaker than content predictions, and modality-specific factors ${\mathbf{m}}_i$ remain consistently uninformative, with near-zero $R^2$ values across all configurations. This suppression of modality-specific noise is particularly important: it demonstrates that MIRAGE does not overfit to idiosyncrasies of individual modalities, but instead learns semantically aligned representations grounded in shared structure.

Taken together, these results validate the hypothesis that block-identifiability of content is achievable through contrastive learning, even under conditions of strong heterogeneity, nonlinearity, and partial statistical dependence. The framework introduced by MIRAGE provides both theoretical and practical support for disentangling shared semantics in multimodal learning.

5.2. Real-World Scenario for Multimodal3DIdent

To evaluate the generalizability of MIRAGE in high-dimensional, non-synthetic contexts, we turn to the Multimodal3DIdent dataset—a structured benchmark constructed from image-text pairs rendered using Blender and annotated with known latent scene factors. This dataset reflects a realistic multimodal generation scenario in which semantic equivalence (e.g., object shape or location) is conveyed across very different sensory domains: vision and language.

Each image-text pair encodes three types of latent information: (i) content factors, which are shared across modalities and include object shape and position; (ii) style factors, such as object color, which are partially determined by content and subject to cross-modal perturbation; and (iii) modality-specific components, such as lighting and camera orientation in the image, and phrasing variation in the text.

Quantitative Performance.

We evaluate MIRAGE’s ability to recover these latent factors by training downstream predictors on the learned representations. Regression is used for continuous variables (e.g., position, color), while classification is applied to discrete attributes (e.g., shape, phrasing). As shown in Table 2, content factors are recovered with extremely high fidelity—over 97% accuracy for both object shape and position. Style prediction yields moderately strong $R^2$ values (0.73), consistent with its partial dependence on content. Modality-specific features, particularly those grounded in non-semantic visual or linguistic variations, remain effectively orthogonal to the learned representation.

This selective encoding of content over irrelevant modality-specific confounders suggests that MIRAGE successfully prioritizes semantically invariant structure over spurious correlations, validating its disentanglement behavior in real-world multimodal data.

Ablation: Encoding Size.

To better understand the relationship between model capacity and disentanglement, we conduct a dimensionality ablation study. We vary the size of the latent representation from 4 to 32 dimensions and measure how well each latent factor is recovered. We observe that content-related performance saturates early—typically after 6–8 dimensions—indicating that the content subspace is compact and efficiently captured. Style information is only absorbed when sufficient excess capacity is available, and even then, the encoding remains structured and interpretable.

This suggests that overparameterization may lead to mild entanglement, particularly when the model has unused capacity, but that the inductive bias of contrastive training still favors the emergence of content-separable structure. These findings also emphasize the importance of controlling embedding dimensionality to encourage semantic selectivity in practice.

Failure Case Analysis.

Despite strong overall performance, MIRAGE exhibits limitations in scenarios involving discrete latent variation, particularly with linguistic phrasing. For instance, when predicting text phrasing categories, accuracy fluctuates between 48% and 54%, substantially lower than content-related attributes. This performance gap arises due to the mismatch between the continuous assumptions embedded in InfoNCE optimization and the inherently discrete nature of certain linguistic variables.

To mitigate this issue, we propose an auxiliary entropy-sensitive gating mechanism, which restricts the backpropagation signal from discrete factors during contrastive updates. Preliminary results suggest that this adjustment can enhance phrasing sensitivity without compromising the identifiability of core content dimensions.

5.3. Cross-Modal Confusion Evaluation

As an additional analysis, we perform a contrastive confusion study to assess whether MIRAGE’s content representations generalize reliably across modalities. Specifically, we compute the cosine similarity between content embeddings ${\widehat{\mathrm{c}}}_1$ and ${\widehat{\mathrm{c}}}_2$ obtained from matched image-text pairs versus randomly sampled pairs. Table 3 summarizes the results.

Interpretation.

The similarity scores for aligned content pairs across modalities are significantly higher than those of randomly sampled pairs, confirming that the learned latent space encodes a consistent, modality-agnostic notion of content. Furthermore, intra-modal comparisons (e.g., image-to-image) show similarly high scores, reinforcing the claim that representations are not only aligned across modalities but also internally stable.

Importantly, the low similarity between mismatched content embeddings suggests that MIRAGE does not collapse to a trivial representation and that semantic signal dominates over modality noise. These findings underscore the cross-modal robustness of MIRAGE and its potential utility in transfer tasks such as cross-modal retrieval and zero-shot generalization.

5.4. Findings and Discussions

Through rigorous empirical analysis on both synthetic and real-world multimodal data, MIRAGE demonstrates strong and consistent performance in identifying shared content representations. Our results show that:

−

Content variables are robustly encoded and remain identifiable under nonlinear, modality-specific transformations.

−

Style and modality-specific signals are effectively suppressed or disentangled unless structure permits semantic leakage (e.g., causal dependency).

−

Cross-modal embeddings are aligned, stable, and interpretable, even under noisy or heterogeneous input modalities.

−

The model exhibits limitations with discrete factors, suggesting opportunities for tailored extensions in hybrid settings.

These results collectively provide strong empirical support for our theoretical claims, establishing MIRAGE as a principled and effective framework for multimodal representation learning grounded in identifiable latent structure.

6. Conclusion

In this work, we investigated the identifiability of shared latent factors in multimodal representation learning. We introduced MIRAGE, a general framework that combines contrastive learning with a carefully structured latent variable model incorporating modality-specific variation, stochastic style perturbations, and shared semantic content. Our primary contribution is a theoretical proof that contrastive learning—when applied under mild conditions and using an appropriately sized encoder—can recover shared latent content across distinct modalities up to a block-wise transformation. This theoretical guarantee generalizes previous multi-view ICA results by accounting for heterogeneous modality-specific encoders and causal dependencies in the latent space. Empirical validation using synthetic simulations and a high-fidelity image-text dataset further confirms that MIRAGE isolates invariant factors while suppressing modality-specific noise and confounders. Our findings offer a foundational step toward building more interpretable, robust, and generalizable multimodal systems. They also motivate future extensions that relax structural assumptions and adapt the MIRAGE framework to broader application scenarios—including domain adaptation, continual learning, and cross-modal transfer. We hope this work contributes to a deeper theoretical understanding of representation learning in the multimodal context, and inspires further exploration of identifiability in rich, heterogeneous environments.

6.1. Discussion

Broader Implications and Theoretical Scope

Our theoretical framework and supporting empirical findings yield important insights into the operational boundaries and effectiveness of contrastive learning in recovering shared semantic representations across heterogeneous modalities. In particular, we establish that MIRAGE is capable of achieving block-identifiability of shared content factors under a wide range of multimodal generative settings. The strength of this conclusion lies in its breadth of applicability: it remains valid even in situations where modality-specific latent variables, nonlinear transformations, and causal dependencies are present—conditions that are common in real-world multimodal systems.

Results from our synthetic experiments further reinforce this theoretical foundation. When the encoder capacity is properly aligned with the true dimensionality of the content variables, MIRAGE reliably extracts invariant content while successfully filtering out modality-specific noise and style-based variability. This outcome is in accordance with an information-theoretic view of contrastive learning, which conceptualizes it as a form of mutual information maximization [43,45]. From this perspective, our results clarify a critical nuance: in the absence of capacity constraints, encoders tend to preserve all mutual information—including stylistic and noisy components. However, by constraining the latent space dimensionality, MIRAGE effectively acts as an information bottleneck, forcing the representation to prioritize stable, invariant semantics.

This mechanism has far-reaching implications for downstream applications. In settings like domain generalization, lifelong learning, and multimodal retrieval, the primary challenge often lies not in capturing all cross-modal correlations, but in isolating those that remain invariant under environmental or distributional change. MIRAGE provides a theoretically grounded means for achieving this invariance by tuning representation complexity, thereby aligning closely with recent work in contrastive domain adaptation [16,41], where disentangling content from transient style shifts is essential for robust generalization.

Limitations and Potential Directions

Despite its theoretical appeal and empirical effectiveness, MIRAGE is subject to certain limitations stemming from assumptions made in our identifiability analysis. These constraints highlight important trade-offs that must be considered in practical scenarios.

First, our framework assumes that content features are perfectly invariant across modalities. However, in many real-world datasets, paired samples may be noisy or only partially aligned—due to occlusion, asynchrony, or inconsistent annotation. These violations can obscure or confound content representations, degrading performance. While improving data quality is one potential solution, future extensions of MIRAGE could aim to relax this assumption by modeling partial or probabilistic invariance across modalities. Such an approach could build upon recent developments in causal and stochastic modeling [2,11,36], enabling the system to reason over uncertain or incomplete semantic overlaps.

Second, our current theory presupposes knowledge of the exact number of shared content dimensions. Though this can be treated as a tunable hyperparameter [38], in unconstrained or exploratory environments, its accurate estimation may be non-trivial. While simple heuristics such as elbow plots or dimensionality reduction diagnostics may suffice in some cases, more principled methods—such as those based on spectral analysis, information compression rates, or adaptive entropy thresholds—could improve interpretability and automation.

Third, our formulation is built on the assumption that all latent variables follow continuous distributions. While this is standard in many generative modeling approaches [18,24,26,27,38,66], it can limit applicability in domains characterized by discrete structure, such as symbolic reasoning, linguistics, or categorical biomedical features. In our Multimodal3DIdent benchmarks, we observe that discrete text-based attributes—like syntax or phrasing—can interfere with alignment performance. As a remedy, future versions of MIRAGE may incorporate hybrid continuous-discrete models, or introduce mechanisms like entropy-sensitive masking to handle sparsity and variability in discrete signals.

Lastly, our current implementation targets bimodal settings. However, many real-world applications—such as video understanding, robotics, and human-computer interaction—require the integration of three or more modalities. Extending MIRAGE to tri-modal or higher-order scenarios is both tractable and promising. Prior research [18,40,52] suggests that identifiability improves with additional views, as increased diversity enhances latent disentanglement. Designing contrastive losses that extend symmetric InfoNCE to multiple encoders, and developing matching mechanisms for higher-order alignment, could open a fruitful avenue for extending the reach of MIRAGE.

In conclusion, the limitations of MIRAGE are closely tied to structural assumptions about the underlying generative process. Addressing these limitations—through partial invariance, automatic dimension inference, hybrid latent types, or multi-modal scalability—offers fertile ground for both theoretical advancement and real-world impact.