Spectral-Spatial Feature Fusion for Hyperspectral Anomaly Detection

1. Introduction

Hyperspectral image (HSI) contains hundreds of narrow spectral channels for each pixel, which delivers rich spectral and spatial information [1,2,3]. Owing to this virtue, HSI has been popularly employed in a considerable number of fields, such as object detection [4,5,6], image classification [7,8,9], and change detection [10,11,12]. For all application fields, hyperspectral anomaly detection, aiming to recognite the outliers whose spectra are significantly different from ambient scene, has drawn much more attention in the last few years [13].

In the last several decades, all kinds of hyperspectral anomaly detection techniques have been developed, which can be loosely grouped into several types: statistical models, subspace-based approaches, reconstruction-based approaches, and deep learning approaches. The statistical models are the most typical hyperspectral anomaly detection techniques. A representative statistical technique is the Reed-Xiaoli (RX) scheme [14], which presumes that the multivariate Gaussian distributions can be exploited to model the background and predict the probability density functions of background samples. The RX method contains two versions, i.e., local RX and global RX [15]. When the whole image is used to model Gaussian background distribution, it is named as global RX. When the RX method gauges the Gaussian model using local image pixels, it s regarded as the local RX. Nevertheless, real-world HSIs have extremely complex background, it is hard to characterize it with a multivariate Gaussian distribution. Thus, some improved versions have been studied, such as subspace RX, kernel RX, and linear filter-based RX. For example, Heesung et al. [16] developed a kernel RX detection technique, in which the hyperspectral cube was first projected into a nonlinear feature space, and then, the multivariate normal distribution was employed to fit the feature data. Guo et al. [17] developed a linear filter-based RX detection method for hyperspectral image by decreasing the weight of anomalous objects or noisy instances and enhancing one of background pixels.

The subspace-based methods postulate that the anomaly and background reflectances have high separability in a feature subspace [18,19,20]. The orthogonal subspace projection (OSP) is a classical subspace-based object detection technique. Foe instance, Chang et al. [21,22] applied an improved OSP technique for anomaly detection of hyperspectral data, where the automatic object detection process was implemented on the background pixels. Xiang et al. [23] combined local joint subspace process and classifier for hyperspectral anomaly detection. Chang et al. [24] developed a subspace selection-based isolation forest model for detecting anomaly objects in hyperspectral images which used subsampling technique rather than modeling the whole background. Furthermore, some improved detection methods have been also investigated, such as local 3D OSP [25], nonparametric OSP [26], multiple subspaces [27].

The reconstruction-based methods aim to recover the input data with a certain model, and the residual is considered as anomaly pixels. Representative reconstruction-based object detection methods mainly include low-rank representation [28,29], sparse representation (SR) [30,31], and collaborative representation [32,33]. For example, Zhu et al. [30] developed an adaptive weighted SR technique for detecting anomaly targets in hyperspectral data, in which a random selection strategy was employed to form the background dictionary. Li et al. [32] applied collaborative representation for anomaly detection of hyperspectral data, which assumes that the pixel reflectances belonging to the background area could be characterized by its adjacent pixels while the anomalous pixels cannot. Sun et al. [28] developed a low-rank and sparse matrix decomposition technique to detect the anomaly targets, where the background pixels were the low-rank part and the anomalies were thinly scattered in the whole data.

Recently, the deep learning networks have been also developed to identify the anomaly targets in hyperspectral data. Two typical unsupervised deep networks are utilized for hyperspectral anomaly detection, i.e., generative adversarial network [34,35,36] and autoencoder model [31,37,38]. For instance, Jiang et al. [34] applied generative adversarial network for detecting the anomaly objects, which assumed that the amount of background pixels was significantly greater than one of target pixels. Lu et al. [37] designed a manifold constrained autoencoder for anomaly detection of hyperspectral data, in which the manifold constraint was utilized to model the embedding representation and the autoencoder network was employed to pretain the latent intrinsic structures. In addition, some supervised deep learning networks have been also applied for detecting anomaly targets in hyperspectral data [39,40,41].

Although these approaches mentioned above use spectral and spatial information in a cascaded manner for hyperpsectral anomaly detection, they ignore the complementary property between spectral and spatial dimension. To remedy this issue, a spectral-spatial feature fusion method is proposed for hyperspectral anomaly detection, which can greatly diminish the false alarm rate by merging the spectral and spatial information. To be specific, first, a superpixel-level isolation forest method is constructed to obtain spectral anomaly score. Then, a local spatial saliency detection method is introduced to yield spatial anomaly score by using local spatial similarity in the background pixels. Finally, the spectral and spatial anomaly scores are integrated together followed by a edge-preserving filtering to generate the final detection result. Experiments on several real-world hyperspectral datasets reveal that the proposed SSIF can obtain ascendant detection results over other hyperspectral anomaly detection approaches. The key contributions of this work are concluded as follows:

1) A spectral-spatial feature fusion framework is proposed for hyperspectral anomaly detection, which makes full use of the object anomalies from both spectral and spatial diemnsions.

2) A superpixel-level isolation forest is designed to exploit the homogeneity of objects, which can well preserve the object structures.

3) Experiments on five hyperspectral data reveal that the proposed SSIF can significantly decrease the false alarm rate compared to other detection techniques.

The resting parts of this paper are grouped into five contents: Section 2 portrays the related work. Section 3 provides the detailed steps of the designed detection scheme. Section 4 analyzes the subjective and objective results of all studied methods. Section 5 provides the ablation experiments. Section 6 sums up the conclusions.

2. Related Work

2.1. Isolation Forest Algorithm

Isolation forest (iForest) was first designed for outlier detection [42]. Its core idea is that anomaly cases in a given data set are typically uncommon and distinct from regular instances, making them more prone to be separated in various binary tree architectures than the normal examples. Specifically, the input data is first separated by a randomly threshold, resulting in a tree construction from the root to the leaf. Every tree is further grown until every instance fits the predefined condition. Then, the path length, i.e., the isolation depth, of each instance is calculated with the amount of edges that the instance traverses a binary tree from the root node to the leaf node. As a result, the binary tree is recursively segmented, in which the anomalous instances arrive at leaf nodes quickly whereas normal instances must undergo many more splits before they do. In general, the path length of the anomalous instances is shoter than one of the normal instances. Finally, the anomaly value is counted by averaging the path length across multiple binary trees.

Figure 1 depicts the principle of isolation forest. As can be seen, the normal example $x_b$ needs more times to achieve segmentation while the anomalous example $x_a$ is more easily to be isolated. Accordingly, the path length of $x_b$ is higher than one of $x_a$. Therefore, the merit of the iForest is that the outliers can be identified according to the principle of isolation tree without introducing any measurement, which can improve the computing efficiency with respect to density or distance-based detection algorithms.

2.2. Domain Transform Recursive Filtering

Domain transform recursive filtering (DTRF) is a representative edge-preserving filtering [43], which can well detach the useless information while preserve the image edges and structures. Due to this advantage, the DTRF has been generally utilized in numerous aspects, such as image interpretation, object identification, and image visualization. Assume $\mathbf{I}$ be the input data, the transformed data $\mathbf{F}$ is computed as:

$${\mathbf{F}}_i={\mathbf{I}}_0+\sum _{j=1}^J(1+\frac{{\delta }_s}{{\delta }_r}|{\mathbf{I}}_j-{\mathbf{I}}_{j-1}|)$$

(1)

where ${\delta }_s$ and ${\delta }_r$ mean two free parameters which are used to regulate the smoothness degree. Afterwards, the input data $\mathbf{I}$ is further processed with recursive filter.

$${\mathbf{J}}_i=(1-a^b){\mathbf{I}}_i+a^b{\mathbf{J}}_{i-1}$$

(2)

where ${\mathbf{J}}_i$ indicates the filtered result of ith pixel. $a=exp(-\sqrt{2}/{\delta }_s)$ stands for a feedback value. b stands for the distance between ${\mathbf{F}}_{i-1}$ and ${\mathbf{F}}_i$. In this work, the DTRF is denoted as $\mathtt{DTRF}(\mathbf{I},\mathbf{G},{¡}_{\mathtt{s}},{¡}_{\mathtt{r}})$, where $\mathbf{I}$ refers to the input data, and $\mathbf{G}$ denotes the guidance data that is utilized to the distance b.

3. Proposed Method

Figure 2 depicts the schematic of the proposed spectral-spatial information fusion detection framework, which is composed of three stages. First, the superpixel-level isolation forest is designed for calculating spectral anomaly map. Then, the spatial similarity mesaurement is employed to calculate the spatial anomaly result. Finally, the spectral and spatial anomaly scores are merged together and the DTRF is leveraged to optimize the fused map so as to generate the ultimate detection map.

3.1. Spectral Anomaly Detection

Hyperspectral image contains abundant spectral information of ground targets. The subtle spectral information provides an unique diagnostic capability for object detection. To take full advantage of the spectral information in HSIs, an object-level isolation forest is developed for spectral anomaly detection in this work, which consists of three steps: superpixel segmentation, isolation tree construction, and anomaly score estimation.

1) Superpixel segmentation: Assume $\mathbf{I}$ be the input hyperspectral image, a principal component analysis technique is first conducted on the input data $\mathbf{I}$ to obatin the base image $\mathbf{B}$, where the fist principal components are viewed as the base image. Next, the entropy rate superpixel (ERS) segmentation scheme [44] is introduced to segment the base image $\mathbf{B}$.

$$\mathbf{P}=\mathtt{ERS}(\mathbf{B},K)$$

(3)

where $\mathbf{P}$ denotes the segmentation map. $\mathtt{ERS}$ represents the ERS algorithm. K is the amount of superpixels. Here, K is calculated as:

$$K=⌊N\times \frac{\tilde{M}}{M}⌋$$

(4)

where N is an predefined parameter. $\tilde{M}$ denotes the proportion of nonzero elements in the binary detected map that is measured by using the Sobel filter on the base image $\mathbf{B}$. M stands for the amount of all pixels presented in the whole image. According to the segmentation result of the $\mathbf{B}$, the position location of pixels for each superpixel ${\mathbf{P}}_i$ can be obtained. Accordingly, the 3D superpixels ${\mathbf{X}}_i,i=1,2,...,K$ in HSI can be easily attained.

2) Isolation forest construction: The obtained superpixel HSI is utilized to construct isolation forest. First, T pixels are initially stochastically chosen from the input data ${\mathbf{X}}_i$. Then, the chosen pixels are grouped into left node and right node according to a simple decision strategy. In more detail, if ${\mathbf{X}}_s^t$ is lower than the split value $\eta$, the t-th pixel is grouped into the left node. On the contrary, it is grouped into the right node. Here, s denotes a random number from 1 to D. The split threshold $\eta$ is randomly chosen between the minimum and maximum of $\mathbf{X}$. Next, each child node is further segmented with the same operation above iteratively, until one of the nether conditions is required: 1) the pixels in each child node are consistent; 2) the amount of pixels in each child node is 1; 3) the number of tree comes up to the maximum height $H_{max}$. In this paper, $H_{max}=lo{g}_{2}T$. At last, the procedure steps of the isolation tree are iterated q times to form the isolation forest.

3) Anomaly score estimation: The established isolation forest is exploited to gauge each pixel’s anomalyscore. Specifically, the amount of edges is considered as the path length of each pixel for every isolation tree, in which the pixel varies from the root to the terminal node. Since anomaly objects typically have relatively small areas and distinctive spectrum compared to surrounding background, they are readily segregated from external nodes. Accordingly, their path lengths are shorter while the background pixels have longer paths. Based on this observation, the path length is utilized to estimate anomalies. It is important to mention that the path length of individual pixel is dissimilar to all established trees. Thus, the eventual path length can be calculated with the averaging operation on all isolation trees.

The iForest has q isolation trees $\{Q_1,Q_2,...,Q_q\}$. For a test pixel $x\in \mathbf{X}$, $h_i(x)$ denotes the path length of x in $Q_i$, the avearge path length on the whole isolation trees is computed as:

$$A(h(x))=\frac{1}{T}\sum _i^T{h}_i(x)$$

(5)

Then, for test pixel x, we can compute its anomaly value $s\in (0,1]$ as:

$$s_1(x)=2^{-\frac{A(h(x))}{c(M)}}$$

(6)

Here, $c(m)=2H(m-1)-(2(m-1)/m)$. $H(m)$ is set as $ln(m)+0.5772156649$ (Euler’s constant). When the above operation is performed on each superpixel, the anomaly map ${\mathbf{S}}_1$ can be estimalated.

3.2. Spatial Anomaly Detection

The spatial shape of the anomaly object is different from its adjacent areas [45]. Based on this observation, a local saliency measurement scheme is exploited to estimate the spatial anomaly score, which computes the spatial difference between the center pixel and its ambient background.

$$s_2=min(tr{({\mathbf{C}}_w-{\mathbf{C}}_{w_i})}^T⬝tr({\mathbf{C}}_w-{\mathbf{C}}_{w_i})),i=1,2,..,W$$

(7)

where $s_2$ is the spatial anomaly score. ${\mathbf{C}}_w$ indicates the center window. ${\mathbf{C}}_{w_i}$ represents the surrounding background window. W is the window size. Figure 3 presents an illustration of local saliency detection. By sliding the red window around the blue window, the local difference information between the center window and its surrounding backfround windows can be calculated. The higher the value, the more likely it is to be the background.

Algorithm 1 Spectral-spatial feature fusion for hyperspectral anomaly detection
Input:
Input hyperspectral data $\mathbf{I}$;
Output:
Detection result $\tilde{\mathbf{R}}$
1:	According to (3), calculate the 2D superpixel segmentation result $\mathbf{P}$
2:	Calculate the corresponding 3D superpixels $\mathbf{X}$ in HSI according to the coordinate of pixels in each 2D superpixel ${\mathbf{P}}_i$.
3:	Construct isolation forest based on the 3D superpixel in HSI.
4:	Combine the constructed forest and Eq. (6) to compute the spectral anomaly score ${\mathbf{S}}_1$.
5:	According to Eq. (7), compute the spatial anomaly score ${\mathbf{S}}_2$.
6:	According to Eq. (8), fuse the spectral and spatial anomaly scores to generate the fused detection result $\mathbf{R}$.
7:	According to (9), integrate the spectral and spatial anomaly results to generate the final detection result $\tilde{\mathbf{R}}$
8:	Return $\tilde{\mathbf{R}}$

3.3. Decision Fusion

To fully utilize the complementary characteristics between spatial and spectral domains, the spectral and spatial scores are first merged together.

$$\mathbf{R}={\mathbf{S}}_1\times {\mathbf{S}}_2$$

(8)

Then, the fused detection map is further optimized with the DTRF so as to remove the noisy pixels.

(9)

where $\tilde{\mathbf{R}}$ represents the final detection map. $\mathtt{DTRF}$ is the DTRF. The base image $\mathbf{B}$ is considered as the guided image. ${\sigma }_s$ and ${\sigma }_r$ are fixed as 5 and 0.5, respectively.

4. Results

In this part, several experiments are designed to validate the superiority of the proposed SSIF. The recorded results are listed in the subsequent parts. All experiments are implemented on a Laptop, which is equipped with 64GB RAM and Core i9-10900K CPU.

4.1. Experimental Setup

1) Datasets: In this work, five real hyperspectral datasets from different imaging scenarios, i.e., Beach, Pavia city, San Diego-I, San Diego-II, and Gulfport, are exploited to judge the detection effect of all considered techniques, which are introduced as follows:

The Beach data was gained by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) device over Cat Island. This data embodies 224 reflectance channels varying from 0.4-2.5$\mu$m. Before experiments, several water absorption and noisy channels are removed, and 188 channels are preserved. Its size is of 150×150 and each pixel accounts for 17.2 m. Figure 4 presents the pseudocolor image and reference map.

The Pavia city data was collected by the Reflective Optics System Imaging Spectrometer (ROSIS) equipment. It is about Pavia city scene. This data is of 150×150 pixels with sptial resolution 1.3 m. The imaging range is from 0.43 to 0.86 $\mu$m with 205 spectral bands. This scene has water, bridge, bare soil, and buildings. The pseudocolor RGB and reference image are depicted in Figure 5.

San Diego-I dataset was acquired by the AVIRIS device over the airport region of San Diego, USA. This data’s spatial size is 100×100 pixels and each pixel accounts for 3.5 m. The imaging scope varies from 0.4 to 2.5$\mu$m with spectral interval of 10 nm. After discarding water absorption and noisy channels, 189 channels are employed for experiments. Figure 6 gives the pseudocolor image and reference image.

San Diego-II dataset is situated at the heart of the airport area of San Diego, which is named as San Diego-II. The spatial size of this data is 100×100. This scene is composed of exposed soil, hangers, parking aprons, and airports, in which three airports are taken as anomalies. Figure 7 gives the pseudocolor image and ground truth.

The Gulfport data was obtained by the AVIRIS device about the airport region of Gulfport, USA. This image contains 191 spectral channels varying from 0.55 to 1.85 $\mu$m. Its spatial size is of 100 × 100 pixels and each pixel accounts for 3.4 m. This data has highway, airport runway, vegetation, and airport. The airport is considered as anomalies. Figure 8 depicts the sample image and reference image.

2) Objective Indexes: To quantitatively assess the detection effect of different methods, two widely used metrics are adopted, i.e., ROC curve and AUC. The ROC curve provides the correlation between the detection probability (DP) and the false alarm rate (FAR) at different threhold settings. In more detail, when the detection result and the reference image are given, the DP and FAR are calculated as:

$$\mathrm{DP}=\frac{N_D}{N_T}\mathrm{FAR}=\frac{N_F}{N}$$

(10)

where $N_D$ means the amount of identified object pixels below a fixed threshold. $N_T$ means the total amount of true object region in the original dataset. $N_F$ indicates the sum of false alarm pixels. N indicates the sum of pixels in the input image. When the DP value is higher than one of other approaches at the same false alarm, it means that this method with higher DP yields the best detection result.

The AUC value is estimated with the whole region under ROC curve. A higher AUC represents a better anomaly detection result. The AUC is given by:

$$\mathrm{AUC}={\int }_{-\infty }^{+\infty }DP(H)FAR(H)dH$$

(11)

where $DP(H)$ denots how many corrent positive samples occur among all examples when the threhold is H. $FAR(H)$ records how many incorrect positive samples occur among all negative examples.

4.2. Detection Results

To illustrate the strength of the proposed SSIF, several hyperspectral anomaly detection techniques are selected as compared techniques, including RX [14], CRD [32], AED [46], KIFD [47], PTA [48], RGAE [49], and SSIIF [50]. These approaches are adopted since they are representative or recently developed techniques. Specifically, the RX is a classical statistical modeling technique. The CRD is a popular collaborative representation scheme. The AED is based on attribute filters. The PTA is tensor representation technique. The KIFD and SSIIF are the representative iForest methods. The RGAE is based on graph autoencoder model. The realted parameters in these detection methods follow the corresponding publication.

Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 depict the detection results of all techniques on five datasets. It can be found that the RX scheme yields unsatisfactory detection maps. Many anomalies cannot be well detected in the resulting maps. For example, the RX method fails to identify three airplanes for San Diego-I dataset. For the CRD scheme, the abnormal objects are hardly detected. The background information is regarded as the anomalies. For the AED method, although the anomalies can be effectively detected, the similar land covers in the background also are viewed as anomalies. For example, the island in the Beach dataset is mistakenly detected. The KIFD and PTA techniques can well detect the locations and shapes of different objects. However, they obtains very high false alarm rate for all datasets. The reason is that the KIFD and PTA methods fail to removes the interference of the background information. The RGAE method suffers from poor detection performance. The airplanes in the San Diego-I, San Diego-II, and Gulfport datasets cannot effectively detected. the SSIIF method slightly improves the detection performance. Nevertheless, the background information still can be found in the detection results. By constrast, the proposed SSIF yields the best detection maps with lowest false alarm rate among all considered approaches. The background information can be well suppressed and only anomalies are effectively seperated from the original image.

Furthermore, the objective detection performances of different approaches are quantitatively assessed via the AUC score. Table 1 lists the AUC scores of all studied detection approaches. The best detection score in each row is displayed in bold. It is obvious that the proposed SSIF produces the highest AUCs on all images. This also illustrates that the proposed method indeed yields the best detection effect compared to other techniques. The RGAE method cannot achieve stable detection results. For the San Diego-I and Gulfport datasets, the RGAE methpd yields the lowest AUCs. Moreover, the RX and CRD methods also obtain relatively low detection accuracy. Generally, among all studied detection techniques, the proposed SSIF achieves the highest objective accuracy and produces satisfactory detection results.

Figure 14 depicted the ROC curves of all detection approaches on all images. It can be observed that the detection probability of the proposed SSIF is always greater than other compared techniques when the FAR is varying from 0 to 1. This proclaims that the proposed SSIF yields superior detection results among all detection approaches. For SanDiego-II dataset, the proposed SSIF is slightly lower probability of detection than the RGAE method when the FAR is from 0.05 to 0.1. In general, by observing the ROC curves of Figure 14, it is obvious that the designed SSIF is higher than other approaches.

5. Discussion

5.1. The Influence of Different Parameters

In this subsection, the influence of all free parameters in the spectral anomaly detection branch, i.e., the amount of trees q and the amount of superpixels N, to the detection effect of the proposed method is discussed. Figure 15 (a) depicts the influence of different numbers of isolation tress. It can be found that when the amount of isolation trees is relatively small, the proposed SSIF produces low detection accuracy. When the amount of isolation trees is greater than 50, the detection effect is satisfactory. Furthermore, as the increase of the amount of isolation trees, the computing cost of the proposed framework has an increasing trend. In this work, the amount of isolation trees q is set as 50. Besides, Figure 15 (b) presents the AUCs of the proposed SSIF with different numbers of superpixels. It is shown that when the number of superpixels is relatively large, the detection accuracy tends to decrease. The reason is that excessive superpixels lead to image oversegmentation issue. In addition, a quantity of superpixels cause each local region containing fewer pixels, resulting in unstable detection results. In this work, the number of superpixels N is set as 9 for all experiments.

The window size W in the spatial anomaly detection branch also needs to be determined. Figure 16 depicts the detection accuracy of different sizes of window in the spatial anomaly detection branch. The window size W is varying from 1 to 13 with step size 2. When the window size W increases, the AUC of the proposed SSIF tends to decrease. This is because useless spatial information is used to measure the difference between the center window and the local window. By observing Figure 16, it is obvious that when the window size W is 3, the detection effect of the proposed method is the highest. Thus, the window size W is set as 3 for following experiments.

5.2. The Influence of Different Components

In this part, the influence of different components in the proposed framework to the detection performance is anlyzed. Table 2 lists the quantative scores of different components in the developed framework. By comparing the spatial branch and spectral branch, it is shown that the spectral branch yields lower detection accuracy. The reason is that hyperspectral image usually suffers from spectral mixture issue. By fusing the spectral and spatial branches, the detection accuracy gains great improvement. For example, the AUC value of the proposed SSIF is increased by 1.92% than one of the spectral branch. This experiment also illustrates that the spectral and spatial information fusion can well boost the detection effect.

5.3. Computing Time

In this part, the running time of different detection schemes on all datasets is discussed. Table 3 presents the computational cost of all detection schemes. It is shown that the RX detector perform the fastest. The reason is that this method only requires simple distance calculation. However, the RX method produces relatively poor detection result. The computational burden of the RGAE scheme is highest among all studied techniques since the autoencoders require a plentiful of iterations during the training process. By constrast, the running time of the proposed SSIF is quite competitive among all studied techniques. Furthermore, as the increase of the spatial size, the computational cost tends to increase. How to decrease the computing time is still an interesting topic.

6. Conclusions

In this study, a spectral-spatial feature fusion is designed for hyperspectral anomaly detection, which is comprised of three main stages. First, an object-level isolation forest is constructed to estimate the spectral anomaly score. Then, a local spatial similarity strategy is exploited to produce the spatial anomaly score. Finally, the spectral and spatial anomaly probabilitity maps are merged together followed by a edge-preserving filtering to yiled the ultimate detection map. Experiments on five real-world hyperspectral datasets verify that the proposed SSIF can obtain better detection results with respect to other hyperspectral anomaly detection approaches. In addition, the proposed SSIF can well suppress the background information and achieves a substantially lower false alarm rate. In the future, we will focus on developing real-time hyperspectral anomaly detection techniques. Furthermore, we will expand the applications of hyperspectral anomaly detection. For example, we can combine it with point target detection so as to achieve infrared dim small flying target recognition [51].