Application of Deep Learning to Identify Flutter Flight Testing Signals Parameters and Analysis of Real F-18 Flutter Flight Test Data

1. Introduction

The validation of aircraft airworthiness during flutter flight tests is a critical step in expanding the flight envelope of new or modified aircraft designs, as outlined in [1,2]. These tests involve different steps, one of them being identifying aeroelastic flutter, a condition where aerodynamic forces couple with structural dynamics resulting in self-excited oscillations. One of the final steps during flutter testing is the analysis of signals captured during flight (at different flight conditions of dynamic pressure) to estimate parameters, such as natural frequencies and damping, essential to determining safe flight conditions at each point of the flight envelope.

Traditional methodologies, such as frequency sweeps, remain the most common approach in flutter flight tests [3]. These methods, often analyzed using the Rational Fraction Polynomials technique [4,5,6], have been augmented with advanced modal estimation techniques such as Polymax and Stochastic Subspace Identification (SSI) [7,8]. Although effective, these techniques require extensive time near flutter conditions, particularly during transonic regime testing, where aeroelastic behavior becomes highly nonlinear and less predictable [9,10]. This prolonged exposure increases safety risks and operational complexity.

An alternative to frequency sweeps is the use of sine-dwell excitations, which involve short-duration, low-frequency signals [3]. Although sine-dwell excitations offer significant advantages in minimizing the time spent near critical flutter conditions, their use in flutter analysis has been limited. Few authors have explored this approach in depth. Notable examples include Lind [11], who applied Laplace wavelet techniques to estimate modal parameters, and Barros-Rodriguez et al. [12], who utilized Singular Value Decomposition (SVD) filtering to address noise challenges. Mehra [13] used the Prony method for parameter identification, while Chabalko et al. [14] investigated the use of continuous wavelet transforms to analyze short-duration signals. A more recent method, the Power Spectrum Short Time Optimization (PRESTO) approach, was introduced in [15]. This technique begins with a rapid initialization of parameters using basic spectral analysis, which provides a robust starting point. It then refines these parameters through a time-domain optimization process employing a trust-region-reflective algorithm, ensuring precise and reliable results.

Although all of these methods have shown promise in specific applications, they face notable limitations, such as computational inefficiency, reduced accuracy in low signal-to-noise ratio environments, and difficulties in handling short data lengths characteristic of sine-dwell signals. These challenges underscore the need for innovative approaches to analyze such data effectively and reliably.

Deep Learning (DL) has emerged in recent years as a transformative tool in fields such as signal processing, image recognition, and system identification [16]. Convolutional Neural Networks (CNNs) and Deep Neural Networks (DNNs), in particular, have shown exceptional performance in handling complex, noisy datasets. However, their application in flutter flight testing remains largely unexplored. Although studies like [17,18,19] have utilized DL to classify flutter conditions or predict airspeed thresholds, the direct estimation of aeroelastic parameters from flight test data, especially sine-dwell signals, has received limited attention. Challenges such as adapting DL architectures to the specific nature of flutter data and ensuring robust performance with noisy, short-duration signals remain open research questions.

This paper proposes a novel application of DL techniques, including CNNs and DNNs, to process sine-dwell signals for accurate parameter identification in flutter testing. By leveraging synthetic datasets for training and comparing DL methods against established techniques such as PRESTO [15] and Laplace Wavelet estimation [11], we aim to demonstrate the feasibility and advantages of DL in this domain. The proposed approach addresses key limitations of existing methods, such as computational inefficiency and reduced accuracy in noisy environments, while paving the way for real-time in-flight analysis.

The remainder of this paper is organized as follows. Section 2 provides an overview of the materials and methods, detailing the generation of synthetic datasets, DL architectures, and evaluation metrics. Section 3 presents the experimental results, comparing DL methods with traditional approaches on both synthetic and real flight test data. Finally, Section 4 discusses the implications of these findings and outlines future directions for research in this area.

2. Materials and Methods

This section describes the methodology and resources used in the study. The aim is to evaluate the use of Deep Learning techniques for identifying aeroelastic flutter parameters from sine-dwell signals. The generation of synthetic datasets, the design of neural network architectures, and the evaluation metrics are explained. Comparisons with traditional methods such as the PRESTO algorithm and Laplace Wavelet Matching Pursuit estimation are also included as a reference.

PRESTO and Laplace Wavelet Matching Pursuit are similar processing techniques that rely on models described in standards [20,21], providing very accurate solutions. However, the application of Machine Learning in this field has been barely explored. To date, no study has specifically focused on identifying flutter-related parameters from flight test data.

Regarding recent investigations, [17] employed a method based on DNNs to predict flutter airspeed during the analysis phase, prior to flight testing. While their work focuses on pre-flight analysis, it does not address the direct identification of flutter parameters during flight.

In [18] and [19] the authors presented approaches that align more closely with the objectives of this paper by utilizing CNN models. Their methods were applied to data extracted from flight tests, although their validation was primarily conducted using wind tunnel test data. Instead of identifying system parameters, these studies classified conditions as either flutter or no flutter during flight.

Similarly, [22] applied a DNN to analyze flight test data. However, their study focused on modeling the global behavior of the structure rather than on identifying specific flutter parameters.

Perhaps the most comparable work is that of [23], who explored various techniques, including CNNs and Multi-Layer Perceptrons (MLPs), in the context of helicopter rotor blade applications. While their problem shares similarities with the present study, it does not involve the analysis of sine-dwell data.

The method proposed by the authors focuses on accurately identifying the flutter parameters of the aeroelastic equations of motion related to aeroelastic flutter. The main advantages of this method are that once trained, MLPs, DNNs, and CNNs provide almost immediate results using a low-end personal computer, which could enable effective real-time analysis. Additionally, if the training set includes a broad enough range of training parameters, it might be possible to identify accurate flutter parameters for a given number of modes, not just two.

The main limitation of this method is the need for a good and accurate model to provide reliable data. It is challenging to obtain reliable real flight data with accurate output parameters, for example, from Ground Vibration Tests. There is no public source of reliable data to match estimated mode values with the signals themselves. Instead, the signal model from [20] was used to generate synthetic training sets (and will be the model employed to fit the data and identify parameters) using Equation (1):

$$x\left(t\right)=\sum _{j=1}^N{a}_j·e^{-{\zeta }_j{\omega }_{nj}t}sin({\omega }_{dj}t+{\phi }_j)+\nu \left(t\right),$$

(1)

where N is the number of modes, $a_j$ is a constant representing amplitude, ${\zeta }_j$ is the damping factor, ${\omega }_{nj}$ represents the natural angular frequency of the structure, t is the time variable, ${\omega }_{dj}={\omega }_{nj}\sqrt{1-{\zeta }_j^2}$ is the damped angular frequency, ${\phi }_j$ represents the phase angle, and $\nu \left(t\right)$ accounts for structural and aerodynamic noise.

The parameters that need to be identified for the characterization of the aeroelastic behavior of the j-th mode are the amplitude $a_j$, the damping factor ${\zeta }_j$, the natural angular frequency ${\omega }_{nj}$ and the phase angle ${\phi }_j$.

Even though this model may seem simplistic by assuming a linear relationship between various factors (aerodynamic and other linear and non-linear interactions, such as gaps, hysteresis from structural damping, etc.) that are not properly modeled in Equation (1), [20] ultimately calls for monitoring two main parameters: modal frequency and damping for the critical flutter modes. Assuming that the mechanism responsible for flutter can be modeled by a second-order linear Ordinary Differential Equation (ODE), all interactions are reduced to Equation (1).

It is important to note that the authors are not dismissing the significance of other non-linear interactions with the natural modes by assuming that the second-order linear ODE will govern the flutter mechanism. These interactions will influence the approach to the flutter phenomenon, either accelerating or delaying flutter onset, which is crucial given the severe consequences of reaching a flutter point. This limitation must be considered when applying any analysis technique. Experienced engineers shall develop a sensible test plan based on all available information, noting that the methodology described in this paper will not provide information on flutter onset.

Considering all caveats and limitations, the potential of this technique is substantial, and it is worth exploring to develop accurate trained networks with the described advantages.

2.1. Input layer

We will differentiate two processes for adapting the input data, depending on the network type: MLP-based (including MLPs and DNNs) and CNNs.

2.1.1. MLP-based Networks input layer

The adaptation process for MLP-based networks is illustrated in Figure 1. The initial dataset consists of a time signal extracted from one of the aircraft sensors, typically a $5s$ signal sampled at $80Hz$. However, the time length may vary depending on the expected modal frequencies of the critical flutter modes.

The following algorithm is followed:

• A Discrete Fourier Transform (DFT) is applied to the temporal signal, and the real and imaginary parts are separated.
• From each subset, 70 frequency samples are extracted (corresponding to a maximum frequency of $14Hz$ if $5s$ signals are used).
• The positive sections of both the real and imaginary components of the DFT are then concatenated.

This process results in a new signal, 140 samples long, comprised of the concatenated real and imaginary parts of the DFT, which will be used as input for the MLP-based network.

2.1.2. CNN Networks Input Layer

In the case of CNNs we need to follow a different approach. A $2D$ matrix of input data is required instead of a vector.

The starting dataset is the same as for MLP-based networks, and in order to convert it into a 2D matrix we need to perform the following preprocessing operations:

• Define a list of unitary vectors on the complex plane, which will constitute the basis for a vector space $V\in {\mathbb{R}}^k$, defined by a set of vectors ${\mathbf{v}}_k$. Each vector is calculated as:

  $${\mathbf{v}}_k=\Re \left\{e^{i·{\displaystyle \frac{2\pi k}{K}}}·X_n\right\}$$

  where k is the index of the vector, K is the total number of vectors (in this application, $K=16$, which includes 14 unitary vectors, the real part, and the imaginary part), and $X_n$ represents the frequency datapoints. These vectors are equally distributed around the $2\pi$ circumference. See Figure 2.
• Limit the original time series signal to $5s$ samples, or zero-pad until $5s$ of signal are reached.
• Calculate the DFT of the time series. The result is a vector $\mathbf{V}\in \mathbb{C}$ with 200 elements defined on the base B.
• Select the first 70 datapoints, which correspond to a maximum frequency of $14Hz$.
• Project each frequency vector on the vector space defined in step 1 above. The projection values will be used as redundant information, and the 16 projection values will constitute the rows of the input matrix. See Figure 2.

At last, the input will be a $16x70$ matrix including redundant information.

2.2. Output Layer

In all cases the output layer will consist of 4 neurons, which correspond to the natural frequencies and damping factors of the two modes analyzed (2 natural frequencies and 2 damping factors).

Note that the original parameter list to be identified also includes an amplitude and phase angle for each mode. These amplitudes and phase angles have been estimated using a classical matching pursuit process [24].

2.3. Networks Design

We will utilize three different types of neural networks: MLPs, DNNs and CNNs. While detailed descriptions of each network are beyond the scope of this paper, readers can refer to [16] for foundational information.

2.3.1. Multi-Layer Perceptron Design

The MLPs consist of an input layer with 140 neurons, one hidden layer with a different number of neurons for each network, and an output layer with 4 neurons. The hidden layer employs the sigmoid activation function, enabling non-linear transformations and capturing complex patterns within the data. A diagram of these networks can be seen in Figure 3.

In this paper, the number following each MLP designation specifies the number of neurons in the hidden layer. For example, MLP 20 indicates 20 neurons in the hidden layer.

2.3.2. Deep Neural Networks Design

DNNs are an extension of the MLP networks. In this case, the network comprises more than one hidden layer, but follows exactly the same approach as MLP networks in Section 2.3.1 above. An example of the DNN design can be found in Figure 4. As in the case of the MLPs, our application will also consider 140 input samples and 4 outputs, with a different number of hidden layers. Again, the activation function selected for the hidden layers is the sigmoid function.

In the case of DNNs, the DNN designation will indicate the number of hidden layers and the number of neurons in each hidden layer. For example, DNN 2x20 indicates a DNN with 2 hidden layers and 20 neurons in each hidden layer.

2.3.3. Convolutional Neural Networks Design

CNNs are similar to DNNs in that they both consist of multiple hidden layers. However, the primary difference lies in how they compute the output parameters. CNNs use a convolution operation between the input matrices and the different layer parameter matrices, unlike DNNs that rely on compositions of linear transformations. The complete block diagram of a CNN with two hidden layers is depicted in Figure 5. Although there are various CNN architectures, the one used in this paper is detailed below, in one example with two hidden layers. Each layer involves different steps. The process applied to our system is as follows:

• Input Layer. The inputs are matrices for each dataset, which need to be in a format that represents a regular pattern, typically an image. In our case, the input is a data matrix as described in Section 2.1.2. So, the input layer will consist of $70\times 16$ neurons.
• Convolutional process. Each convolutional process, depicted in Figure 5 as an elliptic box, includes a convolutional layer followed by several processes to enhance the operation quality:

  (a)

  Convolutional Layer. This layer performs the convolution operation on the input matrix using the weights matrix.

  (b)

  Normalization Process. This step normalizes the outputs to control their range and prevent saturation. A bias is added, and a multiplier is applied. These parameters are adjusted during training.

  (c)

  Rectification Process. This step converts negative values to zero.

  (d)

  Pooling Process. The convolutional matrices are downsampled using smaller windows.

• Connection Layer. This layer connects and reduces all outputs from the final convolutional block to the four output parameters. It functions as a classical perceptron layer, collecting outputs from the convolutional layers and producing an output to solve the regression problem.
• Regression Layer. Before returning the final parameters, a regression layer is applied to refine the outputs.

This example illustrates a CNN with two hidden convolutional blocks. However, more or fewer blocks can be used. In this paper, the trained CNNs employ 5 and 6 hidden layers for different networks.

In the case of CNNs, the designation includes the number of convolutional blocks and the number of neurons in each convolutional layer within the convolutional blocks. For example, CNN 5x100 indicates a Convolutional Neural Network with 5 convolutional processes and 100 neurons in each convolutional layer of each convolutional process.

2.4. Training Process

The training signals were synthetically generated using Equation (1), with parameters randomly sampled from Table 1. The training process was carried out using backpropagation on a computing cluster provided by the Universidad de Alcalá, utilizing MATLAB for implementation.

The noise in the training dataset was modeled as white noise with a signal-to-noise ratio (SNR) of 5 dB, using random seeds to generate unique noise profiles for each signal. Additionally, a separate synthetic dataset, independent of the training dataset, was created to evaluate the testing process. A comprehensive summary of the training dataset distribution is presented in Table 2, while details of the DNN techniques and architectures explored are outlined in Table 3.

During the training process, we also took into account several additional considerations:

• Signal Ranges: The limited range of frequencies is due to the preliminary nature of this study and the long times required for the networks to be trained. A broader range of frequencies, representative of the frequencies found in the real data available, will be employed in future papers, possibly including a different network design.
• Dampings Upscaling: The dampings have been upscaled by a factor of 15 to reach a range of values in the same order of magnitude as the rest of the parameters. The formula employed is as follows:

  $$\mathrm{Declared}\mathrm{damping}=\mathrm{Real}\mathrm{Damping}\times 15$$

  (3)
  After the estimation is finished, it will be necessary to adjust the damping results inversely.

2.5. Analysis Methodology

To study the results we carried out two sets of experiments, one with synthetic data, and another with real data:

• Synthetic data processing: A set of $10,000$ synthetic signals dataset will be created with known parameters and no noise addition. Taking this dataset as a baseline, and following the same approach as with the training dataset, 5dB SNR white noise was added. The synthetic process will measure the standard deviations of errors in parameters (frequency and damping), skewness, kurtosis and histograms of errors.
• Real data processing: A set of real data from F-18 flutter tests will be employed. The sets will be analyzed by all the techniques, the data will be reconstructed using equation 1 with the estimated parameters and plotted the results of the original vs. the reconstructed data. From there, the Mean Square Error (MSE), the Mean Absolute Error (MAE) and the Maximum Absolute Error (MaxAE), along with the regression coefficients ($R^2$, slope and y-intercept) will be compared.

3. Results

This section presents the experimental results, DL methods with traditional approaches on both synthetic and real flight test data. After processing the data, the following key results are obtained:

3.1. Synthetic Data Processing

As described in Section 2.4, several processing techniques were employed. All the DNNs were compared to Laplace Wavelet Matching Pursuit processing [11] and PRESTO processing [15] as reference. Table 4 shows the results for a SNR of 5dB, considering the case closest to the real conditions. The analysis of this table shows several interesting facts.

PRESTO and Laplace Wavelet Matching Pursuit were added to the analysis for comparison. PRESTO returns the highest standard deviation compared to the other neural networks average for frequency ($50\%$ higher) and damping ($200\%$ higher). However is strongly leptokurtic, noticeably above the rest of the methods also (4 points above for frequency and 30 points above for damping). Regarding skewness coefficient, even though is the only method that returns a negative skewness in frequency is still within limits of symmetry, like the rest of the methods. However in damping the skewness is also the highest of the techniques. Even though most are skewed to the left given the negative coefficient, PRESTO shows the highest skewness of all. The conclusion is that PRESTO returns more accurate results than the neural networks techniques, at the price of the most extreme outliers and a long processing time ($\approx 3$ orders of magnitude).

On the other hand, Laplace Wavelet Matching Pursuit returns similar values to the rest of the neural networks methods. Even though in frequency the standard deviation is higher than the rest of the neural networks ($20\%$ higher) in damping is $20\%$ below the average, specifically around the same values as the rest of the CNNs (several of the DNNs increase the average substantially). In kurtosis, in frequency and damping is more leptokurtic than the average ($1.5$ points above in frequency, including CNNs, and 2 points above in damping, although below than the CNNs). At last, the frequency skewness is substantially low, one order of magnitude below the average, indicating a strongly symmetric distribution as compared to the rest of the methods, and in damping is also skewed to the left, although in the average as compared to the rest of the methods. The conclusion is that is a method similar to the rest of the neural networks processed, although the nominal processing time is also $\approx 3$ orders of magnitude higher than the neural network techniques, a caveat needs to be expressed. To achieve a similar accuracy as the rest of the techniques, it was necessary to fine tune the dictionaries to the data searched, limited to be the smallest as possible given that the computers used ran out of memory with higher dictionaries. If the ranges f the dictionaries were the same as for PRESTO, the processing time was estimated to increase in at least 3 more orders of magnitude (meaning 6 orders of magnitude above the neural networks techniques). This fact may also account for the relatively low values for standard deviation, although cannot be confirmed.

Figure 6 shows a comparison between the error histograms of PRESTO, Laplace Wavelet Matching Pursuit, a CNN of 100 x 6 and a DNN of 100 x 100 x 100 as a samples for comparison, taking into account that the rest of the neural network processing techniques show a similar behavior. The plot shows a very similar distribution for the CNN and Laplace, sensibly better for the CNN damping and frequency, and slightly worse for the DNN frequency.

3.2. Real Data Processing

A set of F-18 flutter flight tests data, from the Spanish Air Force CLAEX, was processed and analyzed by the aforementioned techniques. Table 5 shows the results of the analysis process.

The dataset was comprised of 640 signals, extracted from 10 extensometers (6 torsion and 4 bending), 8 flight envelope test points and also 8 excitation frequency runs for each test point (totaling 640 signals). Note that in real life rarely pure bending-torsion flutter modes appear. Usually the nodes lines are not parallel and perpendicular to the cord line, but hold particular shapes, and therefore it is common to see energy from both modes in each sensor. However, approximately only $1/8$ to $3/8$ of the 640 signals carried information related to the natural modes of the structure, since during the sine-dwell runs the frequencies are changed with every excitation. Depending on the natural modes frequencies and the frequency of that test run, it may be possible to catch the energy from one, two or three excitation runs. Rarely more than that.

Considering the aforementioned, even though the 640 signals were analyzed only the 150 best signals participated of the statistics in Table 5. The original data was processed by the different techniques and the signals reconstructed from the parameters estimated. Then, the original signal values were assigned to the x axis while the reconstructed signals values were assigned to the y axis, generating a scatter plot of original vs. reconstructed data (Figure 7) and allowing to calculate regression metrics.

The analysis of the data allows to draw several conclusions. In general, the behavior of the DL techniques on real data is similar to the results on synthetic data, as is globally slightly worse than PRESTO but similar or slightly better than Laplace. It does not constitute a validation of the synthetic results, but is significant. Another conclusion is that all the DNN techniques show a similar behavior. The comparison between different deep learning techniques will be focused on the relationship between m (slope of the regression curve), $R^2$ and the different error metrics. Note that even though all the confidence intervals overlap, and therefore these differences cannot be assumed significant, there is a consistent trend in the averages of all the metrics for ones above the others. In general the error metrics show similar results for DNNs and MLPs, and slightly higher for CNNs, while the regression metrics show the opposite trend.

Comparing the DL results with Laplace, the aforementioned conclusions can be extrapolated also. Assuming only the average values and ignoring the confidence intervals, Laplace could be positioned between CNNs and MLPs, very similar to DNNs. However, the results for PRESTO are sensibly higher than the rest of the techniques, out of confidence intervals. In this case, given that the average values of the CNN techniques are located consistently in the upper bound of the rest of the techniques, the distance with PRESTO is lower than the rest, although not specially significant. This behavior is consistent among all the error metrics and regression coefficients. The slope is closer to 1 in PRESTO, meaning that the large values of amplitude are better fit, the error metrics for PRESTO are lower also, meaning that the lower values in amplitude don’t represent a biased fit (for example, if one method tends to fit higher values but ignores lower, then those errors end up accumulating in the error metrics). There is also no significant difference between the MSE and MAE in between methods, meaning that no significant outliers impacted the processing in one method before another, which is confirmed by a similar trend with MaxAE.

4. Discussion

The results presented in this study indicate that, although the PRESTO technique currently outperforms Deep Learning methods in terms of accuracy, the latter show significant potential for analyzing sine-dwell excitations during flutter tests. This represents a promising advance, in particular for transonic regime envelope expansion. The main advantage of Deep Learning techniques lies in their extraordinary short processing times compared to PRESTO and Laplace Wavelet Matching Pursuit. This makes them highly suitable for real-time analysis during flight tests, where rapid decision-making is critical.

When comparing the Deep Learning techniques, no single method demonstrated a clearly superior performance. While the increased complexity of CNNs led to slightly better results in some cases, the necessity to artificially construct a $2D$ input matrix with redundant data probably impacted the benefits of this architecture. Consequently, the improvements in accuracy were not proportional to the additional architectural complexity, as seen when comparing the performance of the simplest MLP configuration (20 neurons) with the most sophisticated CNN, which did not yield the expected gains in performance.

This observation suggests that further refinement of the architecture is necessary. Modifications may include redesigning the input and output layers to better capture the relationships between parameters or adopting a multi-network approach. For example, individual networks could be dedicated to identifying specific parameters, and their outputs could be integrated into a higher-level network along with the input data. This hierarchical approach would resemble the iterative structure of the PRESTO technique, potentially leading to more accurate and efficient parameter identification.

5. Conclusions

This study explored the application of Deep Learning techniques for identifying aeroelastic flutter parameters from sine-dwell excitations, presenting a potential alternative to traditional methods such as PRESTO and Laplace Wavelet Matching Pursuit. The findings highlight several advantages and challenges associated with these modern approaches.

Deep Learning techniques, particularly MLPs, DNNs, and CNNs, demonstrate significant potential due to their extraordinarily short processing times, which entitles them as a promising solution for real-time analysis during flight tests, in particular during the transonic regime, where the numerical models cannot provide a reliable estimation of the aeroelastic behavior. However, in terms of accuracy, currently the traditional PRESTO method currently outperforms the evaluated Deep Learning models (although the performance is similar, or slightly better, than Laplace Wavelet Matching Pursuit). This highlights the necessity of further refinement before Deep Learning can fully replace established methods.

The analysis revealed no substantial differences in accuracy among the tested Deep Learning architectures, despite the varying levels of complexity. For exampla, while CNNs showed slightly better results in some cases, their higher computational demands and the need for redundant input data limited their overall effectiveness. This lack of a clear performance advantage among architectures suggests that the current designs may not be fully optimized for the specific challenges posed by flutter parameter identification.

To address these limitations, future work should focus on redesigning the input and output layers of the networks, potentially adopting a multi-network architecture. Specialized networks dedicated to identifying specific parameters, followed by a higher-level network to integrate these results, could improve accuracy and efficiency.