VMMCD: VMamba-BasedMulti-Scale Feature Guiding Fusion Network for Remote Sensing Change Detection

1. Introduction

With the increasing importance of remote sensing technology, the domain of change detection has progressively gained prominence. Change detection aims to identify changes between scenes at different time phases. Through the identification of such changes, efficacious scene monitoring can be achieved. Consequently, change detection finds extensive application in a diverse array of fields, encompassing forest vegetation monitoring [1], urban planning [2,3,4], land cover change analysis [5], disaster monitoring and evaluation [6], and military reconnaissance[7].

Conventional methodologies employed in change detection tasks generally encompass two sequential steps: unit analysis and change identification [8]. In light of the selected unit, pixel-based approaches [9] and object-based approaches [10] are available. Nevertheless, the progress in optical Very High Resolution (VHR) imaging has led to a continuous enhancement in image resolution, thereby presenting more detailed characteristics, including the texture and geometric structure of ground objects. This enhancement has, in turn, given rise to an increasing heterogeneity within regions of the same spatial scale, significantly constraining the robustness and effectiveness of traditional change detection approaches.

With the continuous evolution of software and hardware configurations, deep learning-based change detection methods have progressively gained wider acceptance. These methods have significantly enhanced the efficiency and accuracy in change detection. Currently, deep learning-based change detection methods employing CNN and Transformer have been extensively applied. Since Daudt et al. [11] proposed the Fully Convolutional Early Fusion (FC-EF) approach, integrating the Fully Convolutional Network (FCN) into change detection, CNN-based methods have prevailed for an extended period. During this era, a multitude of outstanding CNN-based change detection methods emerged[12,13,14,15,16,17]. Nevertheless, these methods are inevitably subject to the inherent limitations of CNNs. Specifically, the receptive field is confined by the convolution kernel size and the number of network layers, which gives rise to inadequate global modeling capabilities and restricts their competence in recognizing complex scenes.

To address this issue, researchers have devised diverse strategies for augmenting the modeling capabilities of CNNs. One prevalent approach involves stacking additional network layers to progressively expand their receptive fields [18,19,20]. However, this approach leads to a substantial increase in computational complexity. Another commonly adopted method is the incorporation of the attention mechanism [21,22], allowing the model to dynamically concentrate on crucial regions during data processing, thus emulating the global perception ability to a certain degree.

Later, Dosovitskiy et al. [23] introduced the Vision Transformer (ViT) [24] into the visual domain. ViT divides an image into multiple fixed-size patches (e.g., 16x16 pixels) and treats these patches as a sequence. Through its unique self-attention mechanism, each patch interacts with all other patches. This design allows the model to consider the entire image as a whole, rather than being limited to local regions, effectively addressing the problem of limited receptive fields. Thanks to this excellent global modeling capability, ViT has gradually become popular in the field of change detection [25,26,27,28,29]. Transformer-based models outperform CNNs in accuracy, owing to their potent global self-attention mechanism. However, this mechanism also endows the model with quadratic computational complexity. As per the research by Andrew et al. [30], when the computational cost and training time of the models are comparable, the accuracy of CNNs and ViTs is nearly identical. Consequently, it is arduous to optimize the accuracy and computational cost merely by opting for either the CNN or ViT architecture, without taking into account the model design and data resources.

As we are grappling with the challenge of balancing the modeling prowess and computational expense of the model, the advent of the Mamba model provided us with inspiration. The Mamba [31] model, an emerging sequence modeling approach predicated on the Structured State Space Model (S4) [32], is devised to address long-term dependency concerns. Mamba introduces input-dependent time-varying parameters to the state space model (SSM), and mitigates the modeling limitations of CNNs via the global receptive field and dynamic weighting, consequently enhancing the model’s context-based reasoning capacity. Simultaneously, Mamba exhibits a linear computational complexity and effectively curtails the computational expenditure. The efficiency of Mamba attests to its significant potential as a foundational model. Its remarkable success has precipitated its integration into the domain of vision [33,34]. To date, numerous visual tasks predicated on the Mamba model have attained satisfactory outcomes [35,36,37,38].

In the context of the change detection task, methods based on CNNs and Transformers have been extensively utilized. However, the application of methods based on Mamba remains relatively unexplored. It is our contention that the current implementation of change detection using Mamba is confronted with three primary challenges. Firstly, a majority of the existing methods deem the scanning approach of CSM proposed by VMamba to be suboptimal. Consequently, they resort to more intricate scanning techniques [39,40], with the aim of enhancing model performance. Nevertheless, these improvement strategies struggle to comprehensively encompass the patch dependency relationships across all diverse paths.

Secondly, some of the current methods have overly complex models with excessive computational costs and the number of parameters [39,40,41,42]. In contrast to certain generative tasks like image generation, the change detection task is relatively straightforward. This is because it represents a process in which the information (primarily interfering and irrelevant details) is substantially diminished from the input to the output. Consequently, we posit that the change detection model ought to be as lightweight as feasible to preclude model redundancy[43], which could otherwise result in the extraction or introduction of superfluous interfering and extraneous information.

Finally, it was determined that a missed detection issue exists regarding change areas in practical applications. In change detection, there are solely two types of errors, named false detection and missed detection, with the objective of the task being to identify the change areas within bi-temporal images. Evidently, in contrast to false detection, users exhibit a lower tolerance for missed detection. In some downstream tasks of change detection, such as disaster monitoring and evaluation [6] and military reconnaissance [7], the cost of missed detection is often unacceptable. Consequently, our primary focus is placed on missed detection errors.

In this paper, we developed the VMMCD model by leveraging the characteristics of the Mamba model to tackle the aforementioned challenges. Precisely, inspired by the concept of integrating CNN and Transformer, we devised a lightweight architecture akin to Transformer for VMMCD, which is capable of significantly augmenting the global modeling capacity of the Mamba model and circumventing model structure redundancy. We introduced a plug-and-play Multi-scale Feature Guiding Fusion (MFGF) module, representing an enhanced self-attention module. On the one hand, it can further boost the global modeling ability, and on the other hand, it can intensify the feature interaction across multiple scales, thereby resolving or mitigating the issue of missed detection. Generally, VMMCD achieves an excellent balance in both the speed-accuracy and missed-false detection aspects.

The main contributions of this paper are as follows:

• In light of the characteristics of the VMamba model, a simple yet effective model, VMMCD, is proposed. The architecture of this model has been subject to a lightweight design and employs Patch Merging to conduct hierarchical processing of tokens across various scales, thereby facilitating the global spatiotemporal modeling based on tokens by the VMamba backbone, which effectively guarantees both speed and accuracy.
• A proposed plug-and-play Multi-scale Feature Guiding Fusion (MFGF) module is capable of leveraging deep features to conduct layer-by-layer fusion of shallow features. This process fortifies the information exchange across each scale, which augment the utilization efficiency of feature information. It bolsters the global modeling capabilities of VMMCD, and efficiently mitigates or resolves the issue of missed detection.
• To validate the efficacy of our proposed methodology, a comprehensive set of qualitative and quantitative experiments were carried out on three datasets: SYSU-CD, WHU-CD, and S2Looking. The results of these experiments indicated that VMMCD exhibited satisfactory performance and, in certain aspects, achieved state-of-the-art results.

The remainder of this paper is organized as follows. Section 2 presents related work. Section 3 provides a detailed introduction to the proposed VMMCD architecture. Section 4 discusses the results of numerous comparative experiments. Finally, Section 5 draws conclusions.

2. Related Works

2.1. VMamba Model

Liu et al. [33] proposed VMamba, which incorporated Mamba into the domain of vision. The Mamba model is founded on the Structured State Space Sequence Model (S4 model). The S4 model stems from the prevalent Linear Time-Invariant (LTI) system within classical state space models:

$$\left\{\begin{array}{cc}\hfill h^{\prime }\left(t\right)& =\mathbf{A}h\left(t\right)+\mathbf{B}x\left(t\right)\hfill \\ \hfill y\left(t\right)& =\mathbf{C}h\left(t\right)\hfill \end{array}\right.$$

(1)

These equations express a one-dimensional linear mapping through a hidden intermediate state $h\left(t\right)$, where t denotes time, $\mathbf{A}\in {\mathbb{R}}^{N\times N}$, $\mathbf{B}\in {\mathbb{R}}^{1\times N}$ and $\mathbf{C}\in {\mathbb{R}}^{N\times 1}$.

In the realm of deep learning, the model under consideration is discretized, thereby necessitating a transformation from continuous-time to discrete-time formulations. A prevalent approach pertains to the utilization of the Zero-Order Hold (ZOH) method, which can be expounded as follows:

$$\left\{\begin{array}{cc}\hfill \overline{\mathbf{A}}& =exp\left(\Delta \mathbf{A}\right)\hfill \\ \hfill \overline{\mathbf{B}}& ={\left(\Delta \mathbf{A}\right)}^{-1}(exp\left(\Delta \mathbf{A}\right)-\mathbf{I})\left(\Delta \mathbf{B}\right)\hfill \end{array}\right.$$

(2)

The ultimate discretized form is presented as follows:

$$\left\{\begin{array}{cc}\hfill h_t^{\prime }& =\overline{\mathbf{A}}{h}_{t-1}+\overline{\mathbf{B}}{x}_t\hfill \\ \hfill y_t& =\mathbf{C}{h}_t\hfill \end{array}\right.$$

(3)

This allows for parallel computation of results through global convolution. Nevertheless, within the Mamba model, specific alterations are made to the aforementioned process. Precisely, Mamba attaches certain parameters to the input, transforming the system into a linear time-varying one, surmounting the limitations of LTI and acquiring additional learnable parameters. Furthermore, VMamba introduced the Cross-Scan Module (CSM) designed for two-dimensional image data, which serves as the crucial factor enabling VMamba to possess linear complexity. The CSM merely requires scanning all patches along four distinct paths to ascertain the correlation between any patch and the others.

VMamba-based methodologies have been applied to a diverse range of visual tasks. In the domain of medical image segmentation, P-Mamba [36] and VM-UNet [38] are notable examples. For hyperspectral image classification, the Mamba-in-Mamba approach [44] has been utilized. Additionally, attempts have been made to extend the 2D visual selective scanning mechanism to the realm of multimodal learning, giving rise to the VL-Mamba [45], which is a multimodal large language model predicated on a state space model.

2.2. Feature Fusion and Interaction

Feature fusion has been extensively employed in the realm of deep learning. According to the input sources, it can be categorized into multi-level fusion, multi-scale fusion, and heterogeneous feature fusion [46]. Straightforward feature fusion methods usually entail operations devoid of extra parameters, including addition [38], weighted sum, concatenation, pooling [47], and others. These methods are generally stable and do not substantially augment computational costs. Nevertheless, the performance of such feature fusion is frequently not outstanding, rendering it appropriate as a baseline approach for feature fusion. Numerous studies have put forward more efficacious feature fusion techniques. For instance, Huang et al. [48] utilized a feature fusion strategy predicated on coordinated attention to concentrate on the disparities between bitemporal images. Wang et al. [49] integrated pixel-level and object-level features to accentuate geographical proximity.

Feature interaction is frequently considered as a stage within the process of feature fusion [41]. Nevertheless, certain scholars define it as an independent procedure distinct from feature fusion, which pertains to the correlation or interaction of homogeneous/heterogeneous features during the feature extraction stage preceding the fusion process [46]. In this study, we subscribe to the former perspective that feature interaction constitutes a stage within the feature fusion process. Notably, our primary focus lies on the feature interaction process that occurs across multiple scales.

3. Proposed Method

3.1. Overall Architecture

The proposed VMMCD architecture is illustrated in Figure 1. The model employs a typical U-Net architecture, which comprises a patch embedding layer, an encoder and decoder constituted by VSS layers, a final classification layer, and MFGF. Our objective is to design a lightweight change detection framework so as to prevent the extraction or introduction of interfering and irrelevant information caused by overly complex model structures. Moreover, a lightweight model is capable of conserving computing resources. Motivated by the work in [16], we devised the overall architecture of the lightweight model VMMCD. In addition, given that the VMamba block is a plug-and-play module analogous to the ViT block with an identical number of input and output channels, we also incorporated the design from [50] to augment the global modeling capacity of VMamba.

Let’s denote the tensors for images at times T1 and T2 as $X\in {\mathbb{R}}^{C_0\times H\times W}$ and $Y\in {\mathbb{R}}^{C_0\times H\times W}$. Conventionally, C represents the number of channels ($C_0=3$), while H and W represent the height and width of the tensors, respectively.

Firstly, the inputs X and Y are fed into the Patch Embedding layer. Via a $4\times 4$ convolution operation with a stride of 4, the input images are segmented into multiple non-overlapping $4\times 4$ patches. This downsampling procedure can mitigate interfering and irrelevant information, which is conducive to the global modeling by VMamba and MFGF. Layer normalization follows, yielding embedded images $F_0^X,F_0^Y\in {\mathbb{R}}^{C\times {\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}}$, with C set to 96 by default. Afterwards, they are forwarded to the VMamba-based siamese encoder (details in Section 3.2) with shared weights for feature extraction. Several studies [39,40] have indicated that the default scanning path of the CSM in VMamba is imperfect. Our strategy is to maintain the original structure of VMamba and externally implement two enhancement methods. Specifically, the first one is to conduct downsampling via the Patch Merging, where the global linear mapping can effectively model global relationships. The second one is leveraging the self-attention block in MFGF(details in Section 3.3) to progressively guide the multi-scale features extracted by the encoder, which can effectively model the multi-scale global feature relationships and augment the expression of deep abstract features. Then, the features are fully fused through the decoder, and at the same time, the size of the feature map is gradually restored to the same as the input tensor of the same-level encoder using linear mapping. Finally, through the final classification layer, the encoder output is restored to the size of the original image, and then the final change detection result is generated through a convolution layer.

3.2. VMamba-Based Encoder and Decoder

We designed the encoder and decoder based on VMamba. The proposed encoder comprises three layers of VSS Layers with shared weights. Each layer incorporates a set of Visual State Space Blocks (VSS Blocks), with the default number set to 2, which is utilized for modeling global context information. Figure 2(a) presents the detailed structure of the VSS Block. Notably, the 2D Selective Scanning (SS2D) module proposed in VMamba [33] effectively addresses the issue of the suboptimal performance of 1D Selective Scanning (SS1D) [31] in the modeling of 2D image data.

The data processing within the SS2D module entails three steps: cross-scanning, selective scanning via the S6 block, and cross-merging. Herein, cross-scanning and cross-merging are jointly referred to as the cross-scanning module (CSM). Upon the input of data, cross-scanning unfolds the image patches into sequences along four distinct traversal paths. Subsequently, each block sequence is processed in parallel with four S6 blocks. Eventually, cross-merging reshapes and combines the obtained sequences to generate the output graph. We illustrate the self-attention method with square complexity and the cross-scanning method with linear complexity in Figure 2(c) and (d). Since the scanning paths chosen by different patches exhibit a significant amount of overlap, redundant computations are avoided. In contrast, the self-attention method with equivalent global modeling capabilities necessitates computing the correlation between all patches, leading to a computational complexity of $\mathcal{O}\left(n^2\right)$.

In the first two layers of the proposed model, with regard to the feature maps at each level, the "split-transform-fuse" strategy was employed for processing. Specifically, in order to enhance the global modeling capability of the model and circumvent the issue of overly complex models mentioned initially, for the channel dimension of the feature map, a Patch Merging layer was appended subsequent to the VSS Block, as shown in Figure 2(b). Given input $f_i\in {\mathbb{R}}^{2^{i}C\times (H/2^{i+2})\times (W/2^{i+2})}$, where i denotes the i-th stage, a Patch Merging layer firstly performs interval sampling, which split the adjacent blocks into four smaller blocks. Subsequently, they were concatenated along the channel axis, thereby transforming the original feature map into a feature map with the size of $2^{i+2}C\times (H/2^{i+3})\times (W/2^{i+3})$. Thereafter, a fully connected layer was utilized to compress the channels. Currently, the feature map is $2^{i+1}C\times (H/2^{i+3})\times (W/2^{i+3})$. This downsampling approach converts the bi-temporal images into image tokens without information loss, which is advantageous for the VMamba-based encoder to extract global features. Nevertheless, in the bottom layer, a Patch Merging layer is not requisite. It should be noted that, in the absence of channel compression, the Patch Merging layer will quadruple the number of channels of the output feature map relative to the input, resulting in an excessively large channel dimension at the bottom layer (reaching 1536) and leading to model redundancy. As stated in the first section, change detection is a process in which the amount of information is substantially reduced from input to output. In other words, it can also be regarded as a particular process of "denoising" or "eliminating", and thus, excessive extraction or introduction of interfering or irrelevant information should be minimized. Excessive feature channel dimensions and an overly large feature space will decelerate the model and impede its ability to identify key feature information within a vast amount of interfering or irrelevant information.

The inputs at each encoder stage, represented by ${\left\{{\mathcal{F}}_i^X\right\}}_{i=1}^3$ and ${\left\{{\mathcal{F}}_i^Y\right\}}_{i=1}^3$, are combined via MFGFs at skip connections. Moreover, the output streams at the lowest level are aggregated and then input into the decoder.

The proposed decoder exhibits a structure analogous to that of the encoder, wherein each VSS Layer comprises VSS Blocks. Notably, a key difference lies in the default configuration of a single VSS Block per decoder VSS Block, which is predicated on our presumption that the encoder is capable of extracting sufficient features. In the pursuit of computational efficiency, a decoder with a simpler structure was devised. Corresponding to the Patch Merging layers of the encoder, patch expansion layers are utilized for the upsampling process. Residual connections serve to integrate the outputs of the encoder with those of the skip connections, thereby reweighting the pixel values within the decoder’s feature map.

3.3. Multi-Scale Feature Guiding Fusion (MFGF) Module

To further augment the global modeling capacity of the VMamba backbone, we devised the MFGF module. As expounded in Section 1, several other change detection approaches predicated on VMamba endeavor to incorporate additional scanning paths with the aim of bolstering the modeling prowess of VMamba [39,40]. Nevertheless, these ameliorative methods remain incapable of encompassing all conceivable patch dependency relationships. Motivated by certain methodologies integrating CNN and ViT [28], we intend to amalgamate VMamba and self-attention to address the aforesaid issue.

The proposed MFGF block represents a plug-and-play module predicated on Scaled Dot-Product Attention [24]. Owing to its global modeling faculty, it can supplement the deficiency in the modeling aptitude engendered by the insufficient scanning paths of VMamba via the global self-attention mechanism. Among these, the Scaled Dot-Product Attention can be formulated as:

$$\begin{array}{c}\hfill Attention(Q,K,V)=Softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_{head}}}}\right)V\end{array}$$

(4)

Among them, Q, K, and V represent $Query$, $Key$, and $Value$, respectively.

Moreover, for each MFGF module, a new input stream from the subsequent-level MFGF was incorporated, and the multi-scale features were fused by constructing the residual connection of deep-shallow features. Given that the output in the deeper layers of the model contains more abstract and crucial information, it is intended to strengthen the interaction of multi-scale feature information and guide the fusion of shallow features with deep features.

The structure of MFGF is illustrated in Figure 3. Regarding the inputs ${\left\{{\mathcal{F}}_i^X\right\}}_{i=1}^3$ and ${\left\{{\mathcal{F}}_i^Y\right\}}_{i=1}^3$ for each level of the encoder, initially, the two signals are directly summed as the input of MFGF. This is performed to represent the characteristics of the two input feature maps on the same feature map with a reduced computational cost:

$$\begin{array}{c}\hfill {\mathcal{F}}_i={\mathcal{F}}_i^X+{\mathcal{F}}_i^Y,i=1,2,3\end{array}$$

(5)

Subsequently, a residual connection is constructed between the input of the i-th level MFGF ${\mathcal{F}}_i$ and the output of the deeper MFGF ${\mathcal{F}}_{i+1}^{\prime }$. Prior to the multiplication operation, Patch Expanding and a sigmoid activation function are implemented on ${\mathcal{F}}_{i+1}^{\prime }$. The Patch Expanding operation modifies the dimensions of ${\mathcal{F}}_{i+1}^{\prime }$ to be congruent with those of ${\mathcal{F}}_i$, and the sigmoid function transforms the values into weights ranging from 0 to 1:

$$\begin{array}{c}\hfill {\mathcal{F}}_i^G={\mathcal{F}}_i·Sigmoid\left(\mathrm{PE}\left({\mathcal{F}}_{i+1}^{\prime }\right)\right)+{\mathcal{F}}_i\end{array}$$

(6)

The resultant ${\mathcal{F}}_i^G$ is subsequently fed into the $QKV\mathrm{Conv}$ layer, thereby generating Q, K, and V outputs via three separate convolution layers.

$$\begin{array}{c}\hfill (Q_i,K_i,V_i)=QKV\mathrm{Conv}\left({\mathcal{F}}_i^G\right)\end{array}$$

(7)

The self-attention map is generated based on $Q_i$, $K_i$, and $V_i$, and then incorporated into ${\mathcal{F}}_i$ to yield the MFGF output ${\mathcal{F}}_i^{\prime }$:

$$\begin{array}{cc}\hfill {\mathcal{F}}_i^{\prime }& ={\mathcal{F}}_i+Trans\left(A\right)\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill A& =Softmax\left({\displaystyle \frac{Q_i{K}_i^T}{\sqrt{d_k}}}\right)\hfill \end{array}$$

(9)

Notably, we add MFGF only in the skip connections at all levels. Under this setting, the feature maps calculated by MFGF are downsampled by a factor of 4/8/16, which helps avoid too much computational cost incurred by MFGF.

3.4. Loss Function

Two loss functions are considered: Binary Cross-Entropy (BCE) Loss and Focal Loss [51]. The BCE Loss, which is prevalently utilized in binary classification tasks, is defined by Eq. 10, where $y_i$ denotes the ground truth value and ${\widehat{y}}_i$ represents the predicted value. However, when $y_i$ equals 0, $log\left(y_i\right)$ approaches negative infinity, resulting in an infinite loss and the collapse of training. To prevent this issue, we employ the Sigmoid BCE Loss to avoid values of 0 or 1.

Nevertheless, the CD task represents an extremely foreground-background class-imbalanced binary classification problem. This imbalance induces the model to be inclined towards making negative predictions, ultimately resulting in a significant missed detection issue. The conventional BCE Loss fails to adequately address this problem. Hence, we incorporate Focal Loss to diminish the model’s focus on the numerous easy samples during the training phase and enhance its attention towards the hard samples. Focal Loss can be formulated as per Eq. 11:

$$\begin{array}{c}\hfill {\mathrm{L}}_{BCE}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N(y_{i}log\left({\widehat{y}}_i\right)+(1-y_i)log(1-{\widehat{y}}_i))\end{array}$$

(10)

$$\begin{array}{c}\hfill {\mathrm{L}}_{Focal}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N(\alpha {(1-{\widehat{y}}_i)}^{\gamma }{y}_{i}log\left({\widehat{y}}_i\right)+(1-\alpha )·{\widehat{y}}_i^{\gamma }(1-y_i)log(1-{\widehat{y}}_i))\end{array}$$

(11)

the terms $\alpha$ and $\gamma$ here refer to the hyperparameters in Focal loss, which are used to balance the contributions of easy and hard samples. Typically, they are set to $\alpha =0.75$ and $\gamma =2$.

BCE loss and Focal Loss are combined through a weighted sum. The final loss function is defined as:

$$\begin{array}{c}\hfill {\mathrm{L}}_{ours}=\lambda {\mathrm{L}}_{BCE}+{\mathrm{L}}_{Focal}\end{array}$$

(12)

4. Experiments and Results

4.1. Datasets

In the experimental section of this paper, we utilize three publicly available datasets: SYSU-CD[52], WHU-CD[53], and S2Looking[54].

4.1.1. SYSU-CD

SYSU-CD is a category-independent remote sensing image change detection dataset proposed by Sun Yat-sen University. It contains 20,000 pairs of $256\times 256$ aerial images taken in Hong Kong from 2007 to 2014. The dataset is split into training, validation, and test sets with a ratio of 6:2:2. The main types of changes include urban construction, vegetation, road modifications, and offshore developments.

4.1.2. WHU-CD

WHU-CD is a remote sensing image chan-ge detection dataset developed by Wuhan University, primarily for building change detection tasks. The original dataset consists of a pair of aerial images with dimensions of $32507\times 15354$. For our experiments, we obtained non-overlapping $256\times 256$ image data from the researchers’ webpage. The split ratio for the training, validation, and test sets is 4536:504:2760.

4.1.3. S2Looking

The S2Looking dataset, released by the Chinese Academy of Sciences in 2021, focuses on building change detection. It comprises 5000 pairs of $1024\times 1024$ bi-temporal images with a resolution of 0.5 to 0.8 m/pixel. Compared to previous datasets, S2Looking features large viewing angles, significant illumination differences, and complex characteristics. We obtained a cropped, non-overlapping 256×256 version of the dataset from online resources, split into training, validation, and test sets at a ratio of 7:1:2.

4.2. Experimental Setup

4.2.1. Implementation Details

Our model was implemented using Pytorch and trained and tested on an NVIDIA RTX 4090. The AdamW optimizer [55] was employed with a weight decay of 2.5e-3, a learning rate of 2.5e-4, a batch size of 8, and a maximum epoch number of 50. Prior to training, data augmentation methods, including random noise addition, random rotation, and random cropping, were applied.

4.2.2. Evaluation Metrics

To conduct a comprehensive performance evaluation of VMMCD, we selected $Precision$, $Recall$, $F1$-score, and Intersection over Union ($IoU$) as the evaluation metrics. The formulations of these metrics are presented as follows:

$$\begin{array}{c}\hfill Precision={\displaystyle \frac{TP}{TP+FP}}\end{array}$$

(13)

$$\begin{array}{c}\hfill Recall={\displaystyle \frac{TP}{TP+FN}}\end{array}$$

(14)

$$\begin{array}{c}\hfill F1={\displaystyle \frac{2}{Recal{l}^{-1}+Precisio{n}^{-1}}}\end{array}$$

(15)

$$\begin{array}{c}\hfill IoU={\displaystyle \frac{TP}{TP+FP+FN}}\end{array}$$

(16)

where $TP$, $TN$, $FP$, and $FN$ represent true positives, true negatives, false positives, and false negatives, respectively. The $IoU$ is considered the most compelling evaluation metric as it reflects the overlap ratio between inference results and ground truth. $F1$ also possesses a considerable degree of persuasiveness, as it balances the $Precision$ and $Recall$, which are to some extent mutually restrictive.

4.3. Comparison to State-of-the-Art (SOTA)

In the comparative experiment, we selected several state-of-the-art (SOTA) methods for comparison. According to their feature extraction approaches, these methods can be classified into three categories: CNN-based methods (FC-EF[11], FC-Siam-Conc[11], FC-Siam-Diff[11], TinyCD[16], SNUNet[17], CGNet[41]), Transformer-based methods (BIT[27], ChangeFormer[42]), and VMamba-based methods (RS-Mamba[39], ChangeMamba[40]).

When comparing the aforementioned methods, if the dataset partitioning in the original works is consistent with ours, we directly adopt the reported metric values. Otherwise, we reproduce these open-source methods for experimentation, using all the hyperparameters recommended in the original papers. It is noteworthy that the performance metric values obtained from most of our reproductions are higher than those reported in the original papers.

4.3.1. Quantitative results

Table 1 present the comparative experimental results with the selected state-of-the-art methods on the three datasets. For ease of viewing, we highlight the first, second, and third places in red, blue, and black, respectively.

For the category-independent dataset SYSU-CD, the proposed method achieved the optimal result in the key metrics of $F1$-score and $IoU$, with values of 83.35% and 71.45%, respectively. In fact, as of now, this result surpass the highest $F1$ and $IoU$ scores (83.11% and 71.10%) reported in the state-of-the-art (SOTA) literature. This demonstrates the effectiveness of the proposed Mamba-based multi-scale feature guiding fusion change detection method.

However, for the building change detection dataset WHU-CD, the proposed method did not achieve the best results (92.52% and 86.08%) in the comparison, but rather ranked third. Nevertheless, as shown in Table 2, our model’s inference speed (73.05 fps) is higher than that of the top two methods (which are 58.58 fps and 16.89 fps, respectively), and the model size is also more lightweight.

For the highly imbalanced dataset S2Look-ing, observing the performance metrics of the compared methods reveals that, overall, the various metrics of each method are not high. This indicates that the S2Looking dataset is very challenging due to its severe class imbalance between positive and negative samples. The proposed method achieved the best $F1$ and $IoU$ results in the comparative experiments, highlighting its superiority on this dataset.

In addition, from our experience, researchers typically focus more on comprehensive and overall quantitative metrics like $F1$ and $IoU$, while comparatively less attention is given to other indicators that, though not as comprehensive, can still elucidate certain issues, such as $Precision$ and $Recall$. We posit that these indicators are also of value and can be employed. To further explicate the issue of the imbalance of missed detection and false detection, we conducted mathematical transformations and statistical analyses on $Precision$ and $Recall$, as depicted in Figure 4. We computed the proportions of false detection and missed detection within the total errors, respectively. In this context, we use green and red to denote missed detection and false detection.

Upon observing the statistical outcomes, it becomes evident that the compared methods exhibit a particular phenomenon: in the preponderant majority of cases, the proportion of missed detection exceeds that of false detection. Among them, although FC-Siam-Diff[11] has a lower proportion of missed detection on SYSU-CD, it goes to the other extreme. This implies that for the majority of methods, missed detection constitutes the most prominent factor constraining the model’s accuracy score. The occurrence of this phenomenon is not accidental. It primarily stems from the imbalance between positive and negative samples in the training data, wherein the quantity of negative samples is typically several-fold or even dozens-fold that of positive samples. Consequently, the model is predisposed to make negative predictions, thereby giving rise to missed detection. Nevertheless, our method mitigates this issue to a substantial extent. As can be discerned, in the incorrect predictions of VMMCD, the ratio of missed detection to false detection is more balanced compared to that of other methods. This indicates that our method can effectively address the problem of missed detection in the change area, thereby achieving a better balance between missed detection and false detection. The visualization results presented in Figure 4 also corroborate this contention.

4.3.2. Qualitative Visualization Results

In addition to the aforementioned quantitative indicator analysis, for the purpose of visually illustrating the efficacy of the proposed method in minimizing missed detection, we showcase the qualitative visualization outcomes of the binary change detection using the proposed method and other comparative methods in Figure 5, Figure 6, and Figure 7. For clarity, we use red and green to represent FP and FN, respectively, and mark challenging sample areas with red boxes.

In the visualization results of the SYSU-CD dataset, extensive red regions (representing false detection) and green regions (denoting missed detection) are observed, with the green regions being more conspicuous, suggesting a more pronounced problem of missed detection. Through qualitative examination of the bi-temporal change images, it is apparent that in the majority of cases, the extent of change within the regions is not highly pronounced (Figure 5(4), (5), and (6)), and the boundaries are hard to define (Figure 5(2)). These factors account for the challenging characteristics of the SYSU dataset. The visualization results depicted in Figure 5(1), (2), and (3) seem comparatively better on account of more prominent changes, larger areas of change, and a less complex background. Nevertheless, they still display diverse levels of missed and false detection. For example, in Figure 5(1), the red-boxed area exhibits analogous texture characteristics in the T1 and T2 images (sandy soil and cement surfaces, respectively), resulting in missed detection in numerous results. Overall, our proposed method generates the most favorable visualization results with the minimal amount of green (missed detection) regions, demonstrating its capacity to mitigate or resolve the problem of missed detection in change regions to a certain extent.

Regarding the WHU-CD dataset, the visualization outcomes indicate a relatively smaller quantity of red and green areas. Generally, in comparison with the higher-scoring ChangeMamba and ChangeFormer, our proposed method possesses both merits and demerits, which is evident from the visualization results. By taking Figure 6(3) as an instance, the observation of the red-boxed area discloses that our method mitigates missed detection in numerous scenarios. It is worthy of clarification that notwithstanding the missed detection exhibited by our method in Figure 6(1), these missed regions are diminutive, spatially scattered, and predominantly situated within the change area (i.e., internal voids), thereby exerting a negligible influence on the qualitative evaluation of the overall change region.

The S2Looking dataset presents a challenge because of its severe class imbalance, which is clearly shown in the GT images of Figure 7. In these images, the foreground regions account for a small portion and are precisely annotated. This notably augments the complexity of change detection tasks within this dataset, leading to generally low performance scores for the existing methods. The visualization results are in accordance with the quantitative analysis, indicating a high ratio of missed detection and false positive. Nevertheless, our proposed method still attains favorable results with the minimal extent of green (missed detection) regions. Meanwhile, the red (false detection) regions are comparable to those of several methods with high scores [39,40,41,42].

This demonstrates that our proposed method can alleviate or even solve the problem of missed detection.

4.3.3. Model Efficiency

To conduct a more comprehensive assessment of the performance of the proposed method, we further compared the inference speed, computational complexity, parameter quantity, and corresponding performance metrics of diverse methods on the SYSU-CD dataset, as presented in Table 2. In comparison with certain CNN-based methods, our proposed approach exhibits a moderate level of computational complexity and inference speed. Although it has a slightly larger parameter quantity, it demonstrates a distinct advantage in accuracy. CGNet [41] and SNUNet [17], both of which integrate self-attention mechanism, display a higher level of computational complexity and accuracy than other CNN-based methods. In comparison to certain Transformer-based approaches, our proposed method exhibits substantially reduced computational cost and a decreased number of parameters. Moreover, it outperforms these methods with regard to both inference speed and accuracy. In comparison with the latest VMamba-based methods, our method attains substantial advantages in computational complexity, parameter quantity, and inference time, all while preserving comparable or even higher accuracy. Overall, our proposed method achieves a favorable balance among computational complexity, parameter quantity, inference speed, and accuracy, thereby fulfilling the requirements for high efficiency in change detection tasks.

In summary, based on the obtained results, it can be concluded that the proposed VMMCD strikes a remarkable balance in two indicator aspects. Firstly, there is a balance between accuracy and speed. VMMCD incorporates the VMamba backbone and the multi-scale feature fusion module MFGF, thus acquiring a robust global modeling capability and attaining high accuracy. It outperforms the highest score in the state-of-the-art (SOTA) on the SYSU-CD dataset and exhibits competitive performance on the other two datasets. Simultaneously, it employs a lightweight overall architecture, enabling a high inference speed. Secondly, a balance exists between missed detection and false detection. A plethora of quantitative results indicate that, in the classification errors of VMMCD, the ratio of missed detection to false detection is more balanced compared to that of other methods. It should be emphasized that the missed detection rate of other methods is considerably higher than the false detection rate, which is inconsistent with the objective of change detection(i.e., to identify the changed areas). As stated in Section 1, in the change detection task, users exhibit a lower tolerance for missed detection in contrast to false detection. Consequently, our method effectively mitigates the issue of missed detection, thereby achieving a balance between missed detection and false detection.

4.4. Ablation Study

In this section, our ablation experiments are carried out from the following four aspects:

• Backbone networks.
• Model magnitude.
• The number of MFGFs.
• The coefficient $\lambda$ of the loss function.

4.4.1. Ablation on Backbone Networks

A comparison of several representative backbone networks was conducted. As presented in Table 3, the VMamba backbone achieved the highest performance score, with its computational cost and parameter count being only surpassed by EfficientNet-B4. This is attributed to the fact that the VMamba backbone is capable of conducting global modeling with linear computational complexity, thus simultaneously achieving breakthroughs in accuracy and speed. The Swin-small in the table exhibited suboptimal performance, which we ascribe to the local attention mechanism employed by the Swin Transformer. This mechanism diminishes the global modeling ability of the model while decreasing the computational cost. In contrast, VMamba does not require a similar approach to reduce computational cost, endowing it with a robust modeling ability.

4.4.2. Ablation on Model Magnitude

We elucidated the rationale behind our intention to design a lightweight model in Section 1 and Section 3. Specifically, the change detection process inherently entails a significant reduction in image information. Consequently, the model ought to refrain from excessive complexity to prevent the extraction or introduction of interference and irrelevant information to the greatest extent possible. To validate this hypothesis, we intend to augment the complexity of VMMCD along two distinct axes for comparative analysis. Firstly, with respect to the number of feature dimensions per layer, we adjusted the channel compression within the Patch Merging operation. This alteration will modify the magnitude of the feature space at each layer of the model. We expounded upon the transformation process of the feature channel dimensions prior to and subsequent to Patch Merging and channel compression in Section 3.2. Briefly, during the Patch Merging layer processing, the channel dimension of the features will be quadrupled, and subsequently, a linear layer will be employed for channel compression.

Secondly, in relation to the model depth, we extended the proposed VMMCD by augmenting its depth, transforming the three-layer model into a four-layer architecture. Given that the feature space of the model is the cumulative sum of its feature spaces at each scale, increasing the model depth is tantamount to incorporating an additional scale of feature space into the model. For instance, in the case of VMMCD-S4 herein, we appended the deepest level of the feature space. In accordance with some previously established viewpoints [60], this will render the features extracted by the model more abstract and sophisticated. Nevertheless, our experimental results demonstrate that augmenting the layer count is not always appropriate for the change detection task.

We verified the rationality and superiority of the proposed lightweight method VMMCD. As shown in Table 4. The values [$\times 1$, $\times 0.5$, $\times 0.25$] denote the compression ratios, where "$\times 1$" implies no compression, and "$\times 0.25$" indicates compressing the channel dimension to one-fourth, effectively reducing the expanded channels back to the original count.

Firstly, with regard to the various scenarios of channel compression, we presented their corresponding quantitative metrics. Notably, given that the pretrained VMamba model weights cannot be loaded following a change in the channel dimension, for the sake of a fair comparison, we consistently employ randomly initialized weights during training. By horizontally comparing the compression scenarios of different layers, with the performance of the model under 0.5 times channel compression serving as the reference. In the absence of compression, a certain redundancy exists in the channel dimension of the features. This redundancy causes a reduction in the density of crucial features within the feature space, posing difficulties for the model to extract key features amidst substantial interference. Moreover, it may potentially trigger an Out Of Memory (OOM) error. Conversely, when a compression rate of 0.25 is applied, the number of channels in the feature maps across all layers of the model becomes a fixed value of 96. This fixed number is disadvantageous for the model to extract profound abstract features. In the case of complex datasets, the model might lack the requisite capacity to learn the intricate patterns within the data, leading to underfitting and suboptimal performance on both the training and test sets.

Secondly, by comparing the cases in which the four-layer model and the three-layer model employ the same number of channels, it is evident that the performance scores of the four-layer model are considerably lower than those of the three-layer model. This is consistent with our initial intuition that for the change detection task, a lightweight model should be adopted to the greatest extent possible to prevent the introduction of interference and irrelevant information.

To further elucidate the feature space in the two distinct settings, we present the feature activation maps output by each layer of MFGF for both models, as illustrated in Figure 8. The four-layer model exhibits certain unfavorable characteristics during its operation. Firstly, the feature maps at the middle two scales of the model are in a state of weak activation. This implies that the features extracted by the middle two layers contribute relatively little to the overall classification process. Secondly, the abstract features extracted by the deepest layer of the model, namely "4-MFGF", are relatively inferior, and the features extracted by the shallowest layer, namely "1-MFGF", are not the edge features typically extracted by a general shallow network (despite the overall appearance being highly consistent with the ground truth), but rather certain texture features within the change area.

Based on these two characteristics, we can draw a conclusion that, the weakly activated features at the middle scale of the model impede the transmission of deep low-frequency feature information to the shallowest layer, compelling the shallow network to learn low-frequency features that are challenging to acquire, and simultaneously hindering the transmission of the gradient during the training process to the deepest layer. This is the reason why the shallow features appear to be superficially consistent with the ground truth, whereas the deep features yield poor results.

In contrast, our three-layer model, by eliminating redundant network structures, enables the shallow and deep networks to primarily extract high-frequency edge features and low-frequency abstract features, respectively, each achieving a more reasonable outcome. Simultaneously, the features of the middle layer are in a striped activation state, augmenting the features of the other two scales in certain areas and facilitating the normal transmission of the gradient. The model in this state exhibits reduced redundancy, permitting each layer to extract features at their respective levels in a normal manner.

4.4.3. Ablation on MFGFs

We validated the efficacy of the proposed MFGF in addressing the issue of missed detection. To this end, we compared the performance metrics of the MFGF module under varying numbers and positions, as presented in Table 5. Given that MFGF represents a flexible plug-and-play constituent, we further carried out experiments on the extended depth VMMCD-S4 model.

Through a comparison of the varying positions and quantities of the MFGF module within the three-layer VMMCD model, it is evident that the number of MFGF modules is positively correlated with the model’s performance. When a single MFGF module is incorporated, the overall performance score exceeds that of the baseline model without MFGF but is inferior to that of the model with two MFGF modules. Notably, when three MFGF modules are utilized, the overall performance score reaches its peak. It is noteworthy that in the experiments involving the four-layer VMMCD, the addition of the plug-and-play MFGF module also yields a relatively pronounced performance enhancement, which further validates the efficacy of the proposed model and the MFGF module.

Concurrently, we present the visualization results of certain experiments, as depicted in Figure 9. The challenging areas are marked with red boxes. The “000-111” notations in the figure denote whether the MFGF is added at the corresponding positions (e.g., “011” indicates the addition of MFGFs at positions 2 and 3), maintaining the same overall order as in Table 5. It is observable that the visualization results are congruent with the quantitative results in Table 5. As the number of MFGF modules increases, the missed detection phenomenon within the red-boxed areas is mitigated.

4.4.4. Ablation on the Coefficient $\lambda$ of the Loss Function

A comparison of the performances on three datasets with respect to different coefficients for the loss function is presented in Table 6. In practical experiments, the magnitude of Focal Loss is typically an order of magnitude less than that of BCE Loss. Consequently, a coefficient term $\lambda$ is solely added to BCE Loss. When $\lambda$ is relatively small (specifically, $\lambda \in [0,0.3]$), finer gradients are set. As can be seen from Table 6, the relationship between performance metrics and coefficients differs among the datasets. Regarding SYSU-CD, when the coefficient $\lambda$ is small ($\lambda <0.3$), there is no significant alteration in the performance metrics $F1$ and $IoU$. In contrast, when $\lambda$ becomes greater($\lambda >=0.3$), $F1$ and $IoU$ commence to decline. In the case of WHU-CD, a positive correlation exists between $F1$, $IoU$, and $\lambda$. For S2Looking, $F1$ and $IoU$ fluctuate within a certain range under different coefficients, without exhibiting a clear correlation. Considering the performance across various datasets, $\lambda$ is ultimately set to 0.2 for the aforementioned experiments.

5. Conclusion

In this research, a lightweight multi-scale feature-guided fusion change detection approach predicated on VMamba was devised. Certain recent deep learning-based change detection techniques have, on the one hand, contrived excessively intricate model architectures, giving rise to model redundancy and facilitating the extraction or introduction of interfering or extraneous information, thereby impeding the extraction of pivotal features. On the other hand, the utilization ratio of the feature information retrieved by the encoder is inadequate, culminating in the issue of missed detection. In light of the aforesaid issues, a lightweight VMamba-based model was designed to mitigate the introduction of interfering or irrelevant information. Additionally, the information exchange between layers is fortified via the proposed MFGF block, further augmenting the modeling prowess of the VMamba model. This, in turn, enhanced the utilization of change feature information and mitigated or resolved the problem of missed detection within the change area. In comparison with CNN- and Transformer-based methods, the proposed VMMCD attains a remarkable equilibrium in terms of speed-accuracy and missed-false detection. The performance score on SYSU-CD surpasses the extant zenith on SOTA and secures competitive scores on the other two datasets. In prospective work, the application of this lightweight change detection approach based on VMamba will be explored, such as optimization leveraging the long sequence modeling capacity of Mamba, with the anticipation of procuring a change detection method exhibiting even superior performance.