Research on Soft Error Detection Method for Self-Similar Data Streams Based on Structure Mapping

1. Introduction

Data flow, as an important process of network transmission, refers to the transfer of information from one point to another in the system, including the generation, transmission, storage, processing, and analysis of data. However, due to the influence of different network structures, high complexity, or noise, data flow is affected by the existence of data transmission errors, storage errors, etc., which are called soft errors. Soft errors can pose a serious threat to the stability and reliability of the system. In order to improve the accuracy of data stream processing, soft error detection becomes an urgent problem.

Ahmed Maged [1] and others proposed an anomaly detection method through real-time monitoring of high-dimensional event data. They extract statistical properties, distribution patterns, and time series features, and analyze and calculate the data to derive anomaly scores or labels for each data point. Appropriate thresholds are set to distinguish between normal and abnormal data, and the system issues timely alerts when abnormal data are detected. The computational complexity increases dramatically with the increase in data dimensions. Distance calculations between data points in high-dimensional space, cluster analysis, and density estimation become very time-consuming.

Zheng Junhua [2] et al. proposed an abnormal data stream detection method based on feature complementation. Considering noise interference, they use data preprocessing combined with feature complementation techniques to complement and analyze data features. Missing data is used as the feature target to find the comparison data stream to complete the detection. In order to achieve high-precision feature complementation, complex algorithms need to be applied, and the excessive consumption of computational resources may lead to slow operation of nodes and deadlock phenomena, which affect the real-time performance of the entire network.

Zhang Lei [3] et al. proposed a soft error detection algorithm based on the instruction set architecture of the computer for data streams. They use the parallelism of instruction set architecture technology to improve the efficiency of software redundancy algorithms. The original code and redundant code are converted into high-efficiency instruction set architecture code to generate a reinforcement program with error detection capability. By vectorizing the data in the data stream, the algorithm is able to detect multiple data points simultaneously. Although this approach improves the efficiency of the software redundancy algorithm by exploiting the parallelism of the instruction set architecture technique, generating a hardened program with error detection capability may require extensive modification and optimization of the original code, which increases the complexity and cost of implementation.

Since self-similar data streams have significant nonlinear and discrete properties, it is difficult to accurately determine the data points associated with soft error features. Traditional detection methods may not be able to capture these complex relationships effectively, leading to a decrease in detection accuracy. In this study, we propose a soft error detection method for self-similar data streams based on structural mapping, which takes advantage of the self-similarity property of data streams, i.e., they have similar statistical laws in certain local characteristics and local ranges.

2. Materials and Methods

Self-similar data streams exhibit the property of similarity across multiple time scales, which usually manifests itself in the form of bursty and long correlation of traffic, i.e., the traffic increases dramatically over a short period of time and maintains this trend over a longer period of time. Such characteristics make the patterns of data flows difficult to predict and model, increasing the complexity of soft error detection and making it difficult to effectively capture the dynamic characteristics of self-similar data flows. The exponential estimation algorithm is used to scale the time series, which makes the dynamic characteristics more intuitive and precise, and helps soft error detection to make the correct judgement.

The time series [5] of the self-similar data stream is set to be$X=\left\{x_i,i\ge 1\right\}$,Divide into subsequences, calculate subsequence [6] means and convert to aggregated sequences $X^m\left(k\right)$The formula is:

$$X^m\left(k\right)={\displaystyle \frac{1}{m}}{\displaystyle \sum _{i=\left(k-1\right)m+1}^{km}{X}_i},k=1,2,\cdots ,N/m$$

(1)

where $N$ is the sequence length; $k$ is the number of data streams; and $x_i$ is the data stream. Calculate the variance $Var{X}^m\left(k\right)$ of $X^m\left(k\right)$ as:

$$Var{X}^m\left(k\right)={\displaystyle \frac{1}{N/m}}{\displaystyle \sum _{k=1}^{N/m}{\left[X^m\left(k\right)\right]}^2}-{\left[{\displaystyle \frac{1}{N/m}}{\displaystyle \sum _{k=1}^{N/m}{X}^m\left(k\right)}\right]}^2$$

(2)

In the formula, there is a linear relationship between the coordinates of $Var{X}^m\left(k\right)$ and $m$ in the self-similar sequence [7]; and there is a straight line fitting relationship between the coordinates of $Var{X}^m\left(k\right)$ and $m$ [8], from which the exponential relationship between the variables can be found based on the equation of the slope of the straight line, as shown in Figure 1.

Setting the data stream base scale to 1, for $X=\left\{X_i,i\ge 1\right\}$, the standard deviation of the series based on linear correlation and straight line fitting correlation [9] $Var{X}^2$, is calculated.

$$Var{X}^2=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{k=1}^N\left(x_i-{\overline{x}}_i\right)}}$$

(3)

where ${\overline{x}}_i$ is the average value of the data stream. The data stream neighbourhood points are calculated to find the time series feature$R_{i,k}$ of the final self-similarity data stream as:

$$R_{i,k}={\displaystyle \frac{X_{2k}+X_{2k-1}}{2}}$$

(4)

where $X_{2k}$, $X_{2k-1}$ are the difference between neighbouring data points,$X_{2k}\in Var{X}^m\left(k\right),X_{2k-1}\in Var{X}^2$.

Repeat the above operation to calculate the linear correlation and straight line fitting correlation of features between average neighbouring points between self-similar data streams, and substitute the field parameters into the formula to solve iteratively, so that the data points related to soft error features can be judged to help achieve accurate detection.

Soft errors in self-similar data streams often occur during data transmission, storage or processing due to incorrect signals or data. And the time series characteristics of self-similar data streams are often complex and variable. Under the influence of the complexity and irregularity of the data, screening soft errors can reduce the detection accuracy due to the inability to accurately obtain sample point features. In order to ensure the accuracy and efficiency of soft error detection, the time series of data streams obtained by the above process is used as the initial value of structure mapping, and the local angle relationship between data streams and soft error data is continuously established, so as to achieve effective detection through comparison.

Structure mapping algorithm for data stream soft error detection, which belongs to a nonlinear dimensionality reduction algorithm [10], can go through the local angle [11] to establish the relationship between data streams, and at the same time reflect the stream shape structure. Its specific implementation process is to take the $R_{i,k}$ parameter in Eq. (4) as the initial structure mapping target, giving the objective function as:

$$Q={\displaystyle \sum _{i,j=1}^{m,n}{\left(u^{\left(i\right)}-u^{\left(j\right)}\right)}^2{w}_{ij}{R}_{i,k}}$$

(5)

where$u^{\left(i\right)}$ and $u^{\left(j\right)}$ are sample points; $T$ is the period parameter; and $w_{ij}$ is the connection weight between $u^{\left(i\right)}$ and $u^{\left(j\right)}$ [12].

First, determine whether an edge connection $\epsilon$ is set between samples $u^{\left(i\right)}$ and $u^{\left(j\right)}$. If $u^{\left(i\right)}$ exists in the nearest neighbourhood of $u^{\left(j\right)}$, set a connection $u^{\left(i\right)}-u^{\left(j\right)}\le \epsilon$ connecting the two via edge connection $\epsilon$. , calculate the size of the weights, using the core function i.e., set the connection weights as:

$$w_{ij}=e-{\displaystyle \frac{u^{\left(i\right)}-u^{\left(j\right)}}{\epsilon }}$$

(6)

In Eq. $e$ is the connection coefficient. Here we can make $\epsilon =\infty$, all neighbouring data stream weights are set to $w_{ij}=1$, and all other are connections are 0. This gives a symmetric connection matrix.

The minimisation function [13] is established to ensure that the two points can also be kept similar after the sum mapping, expressed as Eq:

$$\min Q=D_{ii}{\displaystyle \sum _{i,j=1}^{m,n}{\left(u^{\left(i\right)}-u^{\left(j\right)}\right)}^2{w}_{ij}{R}_{i,k}}-2D_{ij}$$

(7)

$D_{ii}$ is the Laplace matrix satisfying the diagonal mapping [14]; $D_{ij}$ is the Laplace matrix satisfying the linear mapping.

To ensure that the vectors corresponding to the obtained data stream feature values are the output values after the dimensionality reduction mapping, the initial samples are dimensionality reduced by $\min Q$ to obtain $u={\left(u_1,u_2\right)}^{\mathrm{T}}$, where $u_1$, $u_2$ are the results after the dimensionality reduction of $u^{\left(i\right)}$ and $u^{\left(j\right)}$. Based on the stream shape features [15], the structure mapping network is given and computed to obtain:

$$t_h=\sigma \left({\displaystyle \sum _{i=1}^2{w}_{hi}u}+w_h\right),h=1,2,3,4,5$$

(8)

$t_h$ is the corresponding neuron [16] value of the input data stream feature; $\sigma$ is the activation factor [17,18]; $w_{hi}$ and $w_h$ are the neuron bias.

The final detection output layer in the layer consists of 2 neurons $y_1$, $y_2$, when its output value is greater than 0.5, it represents the presence of soft error and the absence of soft error two output results, the formula is:

$$y_i=\sigma \left({\displaystyle \sum _{i=1}^m{u}_{jh}{t}_h}+u_j\right)$$

(9)

$u_j$ is the bias of the detection result output [19]; $u_{jh}$ is the weight difference between the corresponding input sample feature and the output feature; $y_i$ is the difference between the normal data stream and the soft error data stream. Finally, the self-similar data stream soft error detection is achieved by comparing the weights of the data streams:

$$\min {\displaystyle \frac{1}{2}}={\displaystyle \sum _{k=1}^n{\displaystyle \sum _{j=1}^2{\left(y_{kj}-y_{ki}\right)}^2}}$$

(10)

$y_{kj}$ is the soft data error label gradient [20]; $y_{ki}$ is the sample output label gradient.

In order to further ensure the accuracy of soft error detection of self-similar data streams obtained by Eq. (10), the results of accurate soft error detection are further verified by calculating the weights and gradient bias coefficients of the detected samples, and verifying the results of accurate soft error detection through coefficient comparisons. Calculate the output gradient comparison $g_i$ formula for the detection samples as:

$$g_i=y_{kj}\left(1-y_{kj}\right)y_{kj}\left(-y_{ki}\right)$$

(11)

Update the weights and bias coefficients as:

$${u^{\prime }}_{jh}=u_{jh}+w_{jh}\eta g_i$$

(12)

$${u^{\prime }}_j=t_h\left(1-t_h\right){\displaystyle {\sum }_{j=1}^n{u}_j}{g}_i$$

(13)

$\eta$ is the data stream bias factor. Comparison is made through Eq. (13), when the value of self-similar data stream samples $w_{jh}$, $u_{jh}$ is greater than 1, it represents that the self-similar data stream soft error detection is accurate.

3. Results

3.1. Experimental Environment

In order to verify the practical application performance of the self-similar data stream soft error detection method based on structure mapping proposed in the paper, the background network traffic data of a school is used as the original data. In the simulation experiments, in order to ensure the intuitiveness and comparability of the experimental data, an equal-length time series is set up, and considering the existence of multiple types and forms of data stream soft data, the sample data of a short period of time may not meet the detection conditions, and the external fault input is used to generate data stream soft errors. Meanwhile, experiments will be conducted simultaneously with the anomaly detection method through real-time monitoring of high-dimensional event data, and the network anomaly data stream detection algorithm based on feature complementation as a comparison method.

The detailed parameters related to the network are shown in Table 1, and the network experimental environment is shown in Figure 2.

During the experiment, a time series is set for the original data, in which the burstiness parameter of the data stream is set to 0.8 and the long correlation parameter is set to 0.6 to ensure the self-similarity characteristics of the data stream. When constructing the time series, linear correlation and straight line fitting methods are used to extract the features, with the linear correlation coefficient threshold set to 0.75 and the straight line fitting error not exceeding 0.05. In the structural mapping model, the eigenvalues of the data stream of the time series are used as the initial input, the Gaussian kernel is selected for the core function, the weight decay coefficient is set to 0.1 for the calculation of the feature weights, and the step size of the gradient derivation at the sample points is set to 0.01. Using the above parameters, the proposed method constructs the local angle relationship between the data stream and the soft error data, and is trained by a structure mapping network containing 100 neuron nodes, and finally determines the existence or non-existence of the soft error by the output values of the two neurons (with the threshold set to 0.5).

In order to verify the practical application effect of the proposed method, the flow amplitude of the self-similar data streams in the network is used as an index to give the results of the variation of the flow amplitude, as shown in Figure 3.

As can be seen in Figure 3, the proposed method has a high degree of agreement between the traffic amplitude of the data stream and the real results, and the traffic amplitude of this paper is around 10bits in the 3s time point position, while the actual results are around 11bits, with a small difference. The above results show that the proposed method has high accuracy in detecting the change of data flow amplitude, and it can accurately capture the position and change of high amplitude soft error flow, and the performance of peak and trough fluctuation of flow is also the same as the real situation.

4. Discussion

In practice, the data loss rate of queue transmission can also reflect the data flow soft error situation, the more serious the packet loss phenomenon represents more soft errors. The experimental results of the three methods are shown in Figure 4.

As can be seen from Figure 4, the agreement between the queue packet loss rate curves of the proposed method and the real results is the highest among the three methods. Moreover, the fluctuation of the curves at different time nodes hardly exhibits a large gap, which indicates that the accuracy of the detection results based on packet loss rate is higher. In contrast, there is a large gap between the curves of the other two methods and the actual situation.

Soft error detection imposes additional running time, i.e., time overhead, on the program. A smaller value of time overhead represents a shorter execution time for the detection algorithm, a quicker soft error detection process, and a lower number of detection errors. The results of the comparison of the three methods are shown in Figure 5.

From Figure 5, it can be seen that among the three methods, the distribution of the time overhead curve for the proposed method is the lowest, and there is no significant upward trend. In contrast, the overhead curves of the other two methods are higher. This indicates that under the same conditions, the method proposed in this paper can achieve highly accurate soft error detection with the fastest and lowest time consumption.

Detection results based on packet loss rate of the data stream queue: As can be seen from Figure 4, the agreement between the queue packet loss rate curves of the proposed method and the real results is the highest among the three methods. Moreover, the fluctuation of the curves at different time nodes hardly exhibits any large deviations.

5. Conclusions

In order to improve the stability and efficiency of network data stream transmission, a soft error detection method for self-similar data streams based on structural mapping is proposed. A feature analysis model based on time series is established, leveraging the self-similarity law of self-similar data streams to extract the linear correlation and straight-line fitting correlation features of adjacent nodes. These extracted features serve as a reference for subsequent soft error detection, thereby enhancing detection efficiency.

The features of the soft error data stream are used as structural mapping factors. By continuously iterating and comparing the data stream weights and gradients, efficient soft error detection is achieved. Experimental data demonstrate that the detection algorithms possess strong practical application capabilities. Each method has its own specific scenarios and scope of application. Future research will primarily focus on developing customized anomaly detection methods tailored to practical application scenarios to meet the needs of different domains and businesses. These directions will help to advance the field of structured data anomaly detection and improve its reliability and effectiveness in practical applications.

6. Patents

This research was funded by :

(1)

Natural Science Foundation of Fujian Province, grant number 2023J011807

(2)

Natural Science Foundation of Fujian Province, grant number 2023J011800

(3)

Educational Research Programme for Young and Middle-aged Teachers in Fujian Province,grant number:JAT210557