The First- and Second-Order Features Adjoint Sensitivity Analysis Methodologies for Fredholm-Type Neural Integro-Differential Equations: Mathematical Framework and Illustrative Application to a Heat Transfer Model

1. Introduction

In practice, the system under consideration is modeled by learning the operator that can reproduce the system by using data sampled from the respective system. Typical operator learning problems are formulated on finite grids, using finite-difference methods that approximate the domain of the operator under investigation. Recovering the continuous limit is a challenging undertaking, particularly since irregularly sampled data may alter the evaluation of the learned operator. The use of differential equation solvers to learn dynamics through continuous deep learning models of neural networks, called “Neural Ordinary Differential Equations” (NODE), has been introduced by Chen et al. [1]. As demonstrated by various applications [1,2,3,4,5,6,7,8,9], NODE models provide an explicit connection between deep feed-forward neural networks and dynamical systems, offering flexible trade-offs between efficiency, memory costs and accuracy while bridging modern deep learning and traditional numerical modelling. However, NODE models are limited to describing systems that are instantaneous, since each time-step is determined locally in time, without contributions from the state of the system at other times.

In contradistinction to differential equations, integral equations (IE) model global spatio-temporal relations, which are learned through an IE-solver [see, e.g., 10] which samples the domain of integration continuously. Due to their non-local behavior, IE-solvers are suitable for modeling complex dynamics. The problem of learning dynamics from data through integral equations has been addressed by Zappala et al [11], who have introduced the Neural Integral Equation (NIE) and the Attentional Neural Integral Equation (ANIE). The NIE and the ANIE can be used to generate dynamics and can also be used to infer the spatio-temporal relations that generated the data, thus enabling the continuous learning of non-local dynamics with arbitrary time resolution [11,12]. Often, ordinary and/or partial differential equations can be recast in integral-equation forms that can be solved more efficiently using IE-solvers, as exemplified in scattering theory [13], fluid flow [14], and integral neutron and photon transport [15].

Zappala et al [16] have also developed a deep learning method called Neural Integro-Differential Equation (NIDE), which “learns” an integro-differential equation (IDE) whose solution approximates data sampled from given non-local dynamics. The motivation for using NIDE stems from the need to model systems that present spatio-temporal relations which transcend local modeling, as illustrated by the pioneering works of Volterra on population dynamics [17]. Combining the properties of differential and integral equations, IDEs also present properties that are unique to their non-local behavior [18,19,20], with applications in computational biology, physics, engineering and applied sciences [18,19,20,21,22,23].

All neural nets are trained by minimizing a “loss functional” which aims at representing the discrepancy between a “reference solution” and the output produced by the respective net’s decoder. The neural-net is optimized to reproduce the underlying physical system as closely as possible. However, the physical system modeled by a neural-net comprises parameters that stem from measurements and/or computations which are subject to uncertainties. Therefore, even though the neural net would ideally model perfectly the system’s parameters, the uncertainties inherent in these parameters would propagate to the subsequent results of interest, which are various functionals of the net’s decoder output rather than some “loss functional.” Hence, it is important to quantify the uncertainties induced in the decoder’s output by the uncertainties that afflict the parameters/weights underlying the physical system modeled by the respective neural-net. The quantification of the uncertainties in the net’s decoder and derived results (called ”responses”) of interest require the computation of the sensitivities of the decoder’s response with respect to the optimized weights/parameters comprised within the neural net.

Neural nets comprise not only scalar-valued weights/parameters but also scalar-valued functions (e.g., correlations, material properties, etc.) of the model’s scalar parameters. It is convenient to refer to such scalar-valued functions as “features of primary model parameters.” Cacuci [24] has recently introduced the “n^th-Order Features Adjoint Sensitivity Analysis Methodology for Nonlinear Systems (n^th-FASAM-N),” which enables the most efficient computation of the exact expressions of arbitrarily high-order sensitivities of model responses with respect to the model’s “features.” Subsequently, the sensitivities of the responses with respect to the primary model parameters are determined, analytically and trivially, by applying the “chain-rule” to the expressions obtained for the response sensitivities with respect to the model’s features/functions of parameters.

Based on the general framework of the n^th-FASAM-N methodology [24], Cacuci has developed specific sensitivity analysis methodologies for NODE-nets, as follows: the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (1st-FASAM-NODE)” [25] and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (2nd-FASAM-NODE)” [26]. The 1st-FASAM-NODE and the 2nd-FASAM-NODE are pioneering sensitivity analysis methodologies which enable the computation, with unparalleled efficiency, of exactly-determined first-order and, respectively, second-order sensitivities of decoder response with respect to the optimized/trained weights involved in the NODE’s decoder, hidden layers, and encoder.

Two important families of IDEs are the Volterra and the Fredholm equations. In a Volterra IDE, the interval of integration grows linearly during the system’s dynamics, while in a Fredholm IDE the interval of integration is fixed during the dynamic-history of the system, but at any given time instance within this interval, the system depends on the past, present and future states of the system. By applying the general concepts underlying the n^th-FASAM-N methodology [24], Cacuci [27,28] has also developed the general methodologies underlying the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Fredholm-Type (2nd-FASAM-NIE-F)” and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type (2nd-FASAM-NIE-V).” The 2nd-FASAM-NIE-F encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Fredholm-Type (1st-FASAM-NIE-F), while the 2nd-FASAM-NIE-V encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type (1st-FASAM-NIE-V).” The 1st-FASAM-NIE-F and 1st-FASAM-NIE-V methodologies, respectively, enable the computation, with unparalleled efficiency, of exactly-determined first-order sensitivities of decoder response with respect to the NIE-parameters, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIE-net. The 2nd-FASAM-NIE-F and 2nd-FASAM-NIE-F methodologies, respectively, enable the computation (with unparalleled efficiency) of exactly-determined second-order sensitivities of decoder response with respect to the NIE-parameters, requiring only as many “large-scale” computations as there are first-order sensitivities with respect to the feature functions.

This work presents the “First- and Second Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type” abbreviated as “1st-FASAM-NIDE-F” and “2nd-FASAM-NIDE-F,” respectively. These methodologies are also based on the general framework of the n^th-FASAM-N methodology [24]. The 1st-FASAM-NIDE-F is presented in Section 2, while the 2nd-FASAM-NIDE-F is presented in Section 3, in the sequel. Section 4 presents an illustrative application of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies to a heat transfer model. This illustrative model has been chosen because it can be formulated either as a first-order differential-integral equation of Fredholm type or as a conventional second-order “neural ordinary differential equation (NODE)”, while admitting exact closed-form solutions/expressions for all quantities of interest, including state functions, first-order and second-order sensitivities. The availability of these alternative formulations, either as a NIDE-F or a NODE, of the illustrative paradigm heat conduction model makes it possible to compare the detailed, step-by-step, applications of the 1st-FASAM-NIDE-F versus the 1st-FASAM-NODE methodologies (for computing most efficiently the exact expressions of the first-order sensitivities of decoder response with respect to the model parameters) and, subsequently, to compare the applications of the 2nd-FASAM-NIDE-F versus the 2nd-FASAM-NODE methodologies (for computing most efficiently the exact expressions of the second-order sensitivities of decoder response with respect to the model parameters).

The discussion offered in Section 5 concludes this work by highlighting the unparalleled efficiency of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies, respectively, for computing exact first- and second-order sensitivities, respectively, of decoder responses to model parameters in optimized NIE-F networks. Ongoing work aims at developing the “First- and Second-Order Features Adjoint Sensitivity Analysis Methodologies for Neural Integro-Differential Equations of Volterra-Type” (1st- FASAM-NIDE-V and 2nd-FASAM-NIDE-V, respectively), which will enable, in premiere, the most efficient computation of the exact expressions of the first- and second-order sensitivities of decoder-responses with respect to the optimized network’s weights/parameters for NIDE-V neural nets.

2. First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type (1st-FASAM-NIDE-F)

The mathematical expression of the network of nonlinear Fredholm-type Neural Integro-Differential Equations (NIDE-F) considered in this work generalizes the NIDE-net model introduced in [16] and is represented in component form by the following system of N^th-order integro-differential equations:

$$\begin{array}{l}{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h\left(t\right);f\left(\theta \right);t\right]\frac{d^n{h}_i\left(t\right)}{d{t}^n}}=g_i\left[h\left(t\right);f\left(\theta \right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t\in \left[t_0,t_f\right];\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.\end{array}$$

(1)

The boundary conditions, imposed at the “initial time” $t=t_0$ and/or “final time” $t=t_f$ on the functions $h_i\left(t\right)$ and their time-derivatives associated with the encoder of the NIDE-F net represented by Equation (1) are represented in operator form as follows:

$$B_j\left[h\left(t\right);f\left(\theta \right);t\right]=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(2)

The quantities appearing in Equations (1) and (2) are defined as follows:

(i)

The real-valued scalar quantities $t$ and $\tau$, $t_0\le t,\tau \le t_f$, are time-like independent variables which parameterize the dynamics of the hidden/latent neuron units. Customarily, the variable $t$ is called the “global time” while the variable $\tau$ is called the “local time. The initial time-value is denoted as $t_0$ while the stopping time-value is denoted as $t_f$. Thus, the dynamics modeled by Eq.(1) depends both on non-local effects, as well as on instantaneous information.

(ii)

The components of the $TH$-dimensional vector-valued function $h\left(t\right)\triangleq {\left[h_1\left(t\right),\dots ,h_{TH}\left(t\right)\right]}^{†}$ represents the hidden/latent neural networks; $TH$ denotes the total number of components of $h\left(t\right)$. In this work, the symbol “$\triangleq$” will be used to denote “is defined as” or, equivalently, “is by definition equal to.” The various vectors will be considered to be column vectors. Typically, vectors will be denoted using bold lower-case letters. The dagger “$†$” symbol will be used to denote “transposition.”

(iii)

The components of the column-vector $\theta \triangleq {\left[{\theta }_1,\dots ,{\theta }_{TW}\right]}^{†}$ represent the “primary” network parameters, namely scalar learnable adjustable parameters/weights, in all of the latent neural nets, including the encoders(s) and decoder(s), where $TW$ denotes the total number of adjustable parameters/weights.

(iv)

The scalar-valued components $f_i\left(\theta \right)$, $i=1,\dots ,TF$, of the vector-valued function $f\left(\theta \right)\triangleq {\left[f_1\left(\theta \right),\dots ,f_{TF}\left(\theta \right)\right]}^{†}$ represent the ”feature/functions of the primary model parameters.” The quantity $TF$ denotes the total number of such feature functions comprised in the NIDE-F. In particular, all of the model parameters that might appear solely in the boundary and/or initial conditions are considered to be included among the components of the vector $\theta$. In general, $f\left(\theta \right)$ is a nonlinear vector-valued function of $\theta$. The total number of feature functions must necessarily be smaller than the total number of primary parameters (weights), i.e., $TF<TW$. When the NIDE-F comprises only primary parameters, it is considered that $f_i\left(\theta \right)\equiv {\theta }_i$ for all $i=1,\dots ,TW\equiv TF$.

(v)

The functions ${\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]$ model the dynamics of the neurons in a latent space where the local time integration occurs, while the functions${\phi }_{i,j}\left[f\left(\theta \right);t\right]$ map the local space back to the original data space. The functions $g_i\left[h\left(t\right);f\left(\theta \right)\right]$ model additional dynamics in the original data space. In general, these functions are nonlinear in their arguments.

(vi)

The functions $c_{i,n}\left[h\left(t\right);f\left(\theta \right);t\right]$ are coefficient-functions, which may depend nonlinearly on the functions $h\left(t\right)$and $f\left(\theta \right)$, associated with the order,$n=1,\dots ,N$, of the time-derivatives $d^n{h}_i\left(t\right)/d{t}^n$ of the functions $h_i\left(t\right)$.

(vii)

The operators $B_j\left[h\left(t\right);f\left(\theta \right);t\right]$, $j=1,\dots ,BC$, represent boundary conditions associated with the encoder and/or decoder, imposed at $t=t_0$ and/or at $t=t_f$ on the functions $h_i\left(t\right)$ and on their time-derivatives; the quantity “BC” denotes the “total number of boundary conditions.”

Customarily, the NIDE-F net is “trained” by minimizing a user-chosen loss functional representing the discrepancy between a reference solution (”target data”) and the output produced by the NIDE-F decoder. The “training” process produces “optimal” values for the primary parameters $\theta \triangleq {\left[{\theta }_1,\dots ,{\theta }_{TW}\right]}^{†}$, which will be denoted in this work by using the superscript “zero,” as follows: ${\theta }^0\triangleq {\left[{\theta }_1^0,\dots ,{\theta }_{TW}^0\right]}^{†}$. Using these optimal/nominal parameter values to evaluate the NIDE-F net yields the optimal/nominal solution $h^0\left(t,x\right)\triangleq {\left[h_1^0\left(t\right),\dots ,h_{TH}^0\left(t\right)\right]}^{†}$ which will satisfy the following form of Equation (1):

$$\begin{array}{l}{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]\frac{d^n{h}_i^0\left(t\right)}{d{t}^n}}=g_i\left[h^0\left(t\right);f\left({\theta }^0\right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left({\theta }^0\right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h^0\left(\tau \right);f\left({\theta }^0\right);\tau \right]}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(3)

subject to the following optimized/trained boundary conditions:

$$B_j\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(4)

After the NIDE-F net is optimized to reproduce the underlying physical system as closely as possible, the subsequent responses of interest are no longer “loss functionals” but become specific functionals of the NIDE-F’s “decoder” output, which can be generally represented by the functional $R\left[h;f\left(\theta \right)\right]$ defined below:

$$R\left[h;f\left(\theta \right)\right]={\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h\left(t\right);f\left(\theta \right);t\right]dt}.$$

(5)

The function $D\left[h\left(t\right);f\left(\theta \right);t\right]$ models the decoder. The scalar-valued quantity $R\left[h;f\left(\theta \right)\right]$ is a functional of $h\left(t,x\right)$ and $f\left(\theta \right)$, and represents the NIDE-F’s decoder-response. At the optimal/nominal parameter values, i.e., at $\theta ={\theta }^0$, the decoder response takes on the following formal form:

$$R\left[h^0;f\left({\theta }^0\right)\right]={\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]dt}.$$

(6)

The physical system modeled by the NIDE-F net comprises parameters that stem from measurements and/or computations. Consequently, even if the NIDE-F net models perfectly the underlying physical system, the NIDE-F’s optimal weights/parameters are unavoidably afflicted by uncertainties stemming from the parameters underlying the physical system. Hence, it is important to quantify the uncertainties induced in the decoder output, $R\left[h;f\left(\theta \right)\right]$, by the uncertainties that afflict the parameters/weights underlying the physical system modeled by the NIDE-F net. The relative contributions of the uncertainties afflicting the optimal parameters to the total uncertainty in the decoder response are quantified by the sensitivities of the NIDE-F decoder-response with respect to the optimized NIDE-F parameters. The general methodology for computing the first-order sensitivities of the decoder output, $R\left[h;f\left(\theta \right)\right]$, with respect to the components of the feature function $f\left(\theta \right)$, and with respect to the primary model parameters ${\theta }_1,\dots ,{\theta }_{TW}$, will be presented in this Section.

The known nominal values ${\theta }^0$ of the primary model parameters (“weights”) characterizing the NIDE-V net will differ from the true but unknown values $\theta$ of the respective weights by variations denoted as $\delta \theta \triangleq \theta -{\theta }^0$. The variations $\delta \theta \triangleq \theta -{\theta }^0$ will induce corresponding variations $\delta f\triangleq f\left(\theta \right)-f^0$, $f^0\triangleq f\left({\theta }^0\right)$, in the feature functions. The variations$\delta \theta$ and $\delta f$ will induce, through Equation (1), variations $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\triangleq {\left[\delta h_1\left(t\right),\dots ,\delta h_{TH}\left(t\right)\right]}^{†}$ around the nominal/optimal functions $h^0\left(t\right)$. In turn, the variations $\delta f\triangleq f\left(\theta \right)-f^0$ and $v^{\left(1\right)}\left(t;x\right)$ will induce variations $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f;t\right)$ in the NIE decoder’s response.

The “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type (1st-FASAM-NIDE-F)” aims at obtaining the exact expressions of the first-order sensitivities (i.e., functional derivatives) of the decoder’s response with respect to the feature function and the primary model parameters, followed by the most efficient computation of these sensitivities. The 1st-FASAM-NIDE-F will be established by applying the same principles as those underlying the 1st-FASAM-N [24] methodology. The fundamental concept for defining the sensitivity of an operator-valued quantity $R\left(x\right)$ with respect to variations $\delta x$ in a neighborhood around the nominal values $x^0$, has been shown in 1981 by Cacuci [29] to be provided by the 1st-order Gateaux- (G-) variation $\delta R\left(x^0;\delta x\right)$ of $R\left(x\right)$, which is defined as follows:

$$\delta R\left(x^0;\delta x\right)\triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}\left[R\left(x^0+\epsilon \delta x\right)\right]\right\}}_{\epsilon =0}\triangleq \underset{\epsilon \to 0}{\lim }{\displaystyle \frac{R\left(x^0+\epsilon \delta x\right)-R\left(x^0\right)}{\epsilon }},$$

(7)

for a scalar $\epsilon$ and for arbitrary vectors $\delta x$ in a neighborhood $\left(x^0+\epsilon \delta x\right)$ around $x^0$. When the G-variation $\delta R\left(x^0;\delta x\right)$ is linear in the variation $\delta x$, it can be written in the form $\delta R\left(x^0;\delta x\right)={\left\{\partial R/\partial x\right\}}_{x^0}\delta x$, where ${\left\{\partial R/\partial x\right\}}_{x^0}$ denotes the first-order G-derivative of $R\left(x\right)$ with respect to $x$, evaluated at $x^0$.

Applying the definition provided in Equation (7) to Equation (5) yields the following expression for the first-order G-variation $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)$ of the response $R\left[h;f\left(\theta \right)\right]$:

$$\begin{array}{l}\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)={\left\{{\displaystyle \frac{d}{d\epsilon }}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]dt}\right\}}_{\epsilon =0}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\hspace{0.17em}{\left\{\delta R\left(h^0;f^0;\delta f\right)\right\}}_{dir}+{\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}\hspace{0.17em},\end{array}$$

(8)

where the “direct effect term” arises directly from variations $\delta f$ and is defined as follows:

$${\left\{\delta R\left(h^0;f^0;\delta f\right)\right\}}_{dir}\triangleq {\displaystyle \sum _{i=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt{\left\{\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_i}\delta f_i\right\}}_{{\theta }^0}}},$$

(9)

and where the “indirect effect term” arises indirectly, through the variations $v^{\left(1\right)}\left(t\right)$ in the hidden state functions $h\left(t\right)$, is defined as follows:

$${\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}\triangleq {\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\left\{\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}{v}_i^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}}.$$

(10)

The direct-effect term can be quantified using the nominal values $\left(h^0;f^0\right)$ but the indirect-effect term can be quantified only after determining the variations $v^{\left(1\right)}\left(t\right)$, which are caused by the variations $\delta f$ through the NIDE-F net defined in Equation (1).

The first-order relationship between the variations $v^{\left(1\right)}\left(t\right)$ and $\delta f$ is obtained from the first-order G-variations of Equations (1) and (2). The first-order G-variations of Equations (1) and (2), respectively, are obtained, by definition, as follows:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f;t\right]\frac{d^n\left[h_i^0\left(t\right)+\epsilon \delta h_i\left(t\right)\right]}{d{t}^n}}\right\}}_{\epsilon =0}\\ ={\left\{{\displaystyle \frac{d{g}_i\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f\right]}{d\epsilon }}\right\}}_{\epsilon =0}\\ +{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left({\theta }^0\right)+\epsilon \delta f;t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h^0\left(\tau \right)+\epsilon v^{\left(1\right)}\left(\tau \right);f\left({\theta }^0\right)+\epsilon \delta f;\tau \right]}\right\}}_{\epsilon =0}.\end{array}$$

(11)

$${\left\{{\displaystyle \frac{d}{d\epsilon }}{B}_j\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f;t\right]\right\}}_{\epsilon =0}=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0;\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(12)

Carrying out the operations indicated in Equations (11) and (12) yields the following NIDE-F net of Fredholm-type for the function $v^{\left(1\right)}\left(t\right)$:

$$\begin{array}{l}{\left\{{\displaystyle \sum _{k=1}^{TH}{\displaystyle \sum _{n=1}^N\frac{d^n{h}_i\left(t\right)}{d{t}^n}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,n}\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)+{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h;f\left(\theta \right);t\right]\frac{d^n{v}_i^{\left(1\right)}\left(t\right)}{d{t}^n}}}}}\right\}}_{{\theta }^0}\\ -{\left\{{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}\right\}}_{{\theta }^0}\\ -{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}={\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k\right\}}_{{\theta }^0}},\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(13)

$$\begin{array}{l}{\left\{{\displaystyle \sum _{k=1}^{TH}\frac{\partial B_j\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}}_{{\theta }^0}=-{\left\{{\displaystyle \sum _{k=1}^{TF}\frac{\partial B_j\left[h;f\left(\theta \right);t\right]}{\partial f_k}}\delta f_k\right\}}_{{\theta }^0},\\ at\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}j=1,\dots ,BC;\end{array}$$

(14)

where:

$$\begin{array}{l}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\triangleq -{\displaystyle \sum _{n=1}^N\frac{\partial c_{i,n}\left[f\left(\theta \right);t\right]}{\partial f_k}\frac{d^n{h}_i\left(t\right)}{d{t}^n}}+{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\phi }_{i,j}\left(f;t\right)}{\partial f_k}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\\ +{\displaystyle \frac{\partial g_i\left(h;f;t\right)}{\partial f_k}}+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_k}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k=1,\dots ,TF.\end{array}$$

(15)

The NIDE-F net represented by Equations (13) and (14) is called [24] the “1st-Level Variational Sensitivity system (1st-LVSS) and its solution, $v^{\left(1\right)}\left(t\right)$ is called [24] the “1st-level variational function.” All of the quantities in Equations (13) and (14) are to be computed at the nominal parameter values, but the respective indication has not been explicitly shown in order to simplify the notation.

It is important to note that the 1st-LVSS is linear in the variational function $v^{\left(1\right)}\left(t\right)$. Therefore, the 1st-LVSS represented by Equation (13) can be written in matrix-vector form as follows:

$$L\left[h;f\left(\theta \right);t\right]v^{\left(1\right)}\left(t\right)=\hspace{0.17em}{Q}^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\delta f\left(\theta \right),$$

(16)

where the $TH\times TF$-dimensional rectangular matrix $Q^{\left(1\right)}\left[h;f\left(\theta \right);t\right]$ comprises as components the quantities $q_{i,k}^{\left(1\right)}\left(h;f\right)$ defined in Equation (15), while the components of the $TH\times TH$ square matrix $L\left[h;f\left(\theta \right);t\right]\triangleq {\left[L_{ik}\right]}_{TH\times TH}$ are operators (algebraic, differential, integral) defined below, for $i,k=1,\dots ,TH$:

$$\begin{array}{l}{L}_{ii}{v}_i^{\left(1\right)}\left(t\right)\triangleq {\displaystyle \sum _{n=1}^N\frac{d^n{h}_i\left(t\right)}{d{t}^n}\frac{\partial c_{i,n}\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}{v}_i^{\left(1\right)}\left(t\right)}+{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h;f\left(\theta \right);t\right]\frac{d^n{v}_i^{\left(1\right)}\left(t\right)}{d{t}^n}}\\ -{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_i\left(\tau \right)}{v}_i^{\left(1\right)}\left(\tau \right)d\tau }-{\displaystyle \frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{v}_i^{\left(1\right)}\left(t\right)\hspace{0.17em};\end{array}$$

(17)

$$\begin{array}{l}{L}_{ik}{v}_k^{\left(1\right)}\left(t\right)\triangleq {\displaystyle \sum _{n=1}^N\frac{d^n{h}_i\left(t\right)}{d{t}^n}\frac{\partial c_{i,n}\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}-{\displaystyle \frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}}{v}_k^{\left(1\right)}\left(t\right)\\ -{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}}{v}_k^{\left(1\right)}\left(\tau \right);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i,k=1,\dots ,TH.\end{array}$$

(18)

Note that the 1st-LVSS would need to be solved anew for each variation $\delta F_j$, $j=1,\dots ,TF$, in order to determine the corresponding function $v^{\left(1\right)}\left(t\right)$, which is prohibitively expensive computationally if $TF$ is a large number. The need for repeatedly solving the 1st-LVSS can be avoided if the variational function $v^{\left(1\right)}\left(t\right)$ could be eliminated from appearing in the expression of the indirect-effect term defined in Equation (10). This goal can be achieved [24] by expressing the right-side of Equation (10) in terms of the solutions of the “1st-Level Adjoint Sensitivity System (1st-LASS)” to be constructed next. The construction of this 1st-LASS will be performed in a Hilbert space comprising elements of the same form as $v^{\left(1\right)}\left(t\right)\in H_1\left({\Omega }_t\right)$, defined on the domain ${\Omega }_t\triangleq t\in \left[t_0,t_f\right]$. This Hilbert space is endowed with an inner product of two elements ${\chi }^{(1)}\left(t\right)\triangleq {\left[{\chi }_1^{\left(1\right)}\left(t\right),\dots ,{\chi }_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in H_1\left({\Omega }_t\right)$ and ${\eta }^{(1)}\left(t\right)\triangleq {\left[{\eta }_1^{\left(1\right)}\left(t\right),\dots ,{\eta }_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in H_1\left({\Omega }_t\right)$, denoted as ${〈{\chi }^{(1)}\left(t\right),{\eta }^{(1)}\left(t\right)〉}_1$ and defined as follows:

$${〈{\chi }^{(1)}\left(t\right),{\eta }^{(1)}\left(t\right)〉}_1\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\left[{\chi }^{(1)}\left(t\right)\right]}^{†}{\eta }^{(1)}\left(t\right)dt}=\hspace{0.17em}{\displaystyle \sum _{j=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\chi }_i^{\left(1\right)}\left(t\right){\eta }_i^{\left(1\right)}\left(t\right)dt}}.$$

(19)

The next step is to construct the inner product of Equation (13) with a vector $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in H_1\left({\Omega }_t\right)$, where the superscript “(1)” indicates “1st-Level”, to obtain the following relationship:

$${〈a^{\left(1\right)}\left(t\right),L\left[h;f\left(\theta \right);t\right]v^{\left(1\right)}\left(t\right)〉}_1=\hspace{0.17em}{〈a^{\left(1\right)}\left(t\right),\hspace{0.17em}{Q}^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\delta f\left(\theta \right)〉}_1$$

(20)

The terms appearing in Equation (20) are to be computed at the nominal values $\left(h^0;F^0\right)$ but the respective notation has been omitted for simplicity.

Using the definition of the adjoint operator in $H_1\left({\Omega }_t\right)$, the term on the left-side of Equation (20) is integrated by parts and the order of summations is reversed to obtain the following relation:

$${〈a^{\left(1\right)}\left(t\right),L\left[h;f\left(\theta \right);t\right]v^{\left(1\right)}\left(t\right)〉}_1=\hspace{0.17em}{〈v^{\left(1\right)}\left(t\right),A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]a^{\left(1\right)}\left(t\right)〉}_1+P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right),$$

(21)

where the operator $A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\triangleq L^{*}\left[h;f\left(\theta \right);t\right]$ denotes the formal adjoint of the operator $L\left[h;f\left(\theta \right);t\right]$ and where $P\left(h;F;v^{\left(1\right)};a^{\left(1\right)}\right)$ represents the scalar-valued bilinear concomitant evaluated on the boundary $t=t_0$ and/or $t=t_f$. Note that the $TH\times TH$ matrix valued operator $A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\triangleq {\left\{A_{ij}^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\right\}}_{TH\times TH}$ acts linearly on the vector $a^{\left(1\right)}\left(t\right)$. The “star” superscript ^* will be used in this work to denote “formal adjoint operator.”

It follows from Equations (20)and (21) that the following relation holds:

$${〈v^{\left(1\right)}\left(t\right),A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]〉}_1={〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\delta f〉}_1-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right).$$

(22)

The term on the left-side of Equation (22) is now required to represent the indirect effect term defined in Equation (10) by imposing the following relation:

$$A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]a^{\left(1\right)}\left(t\right)={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h\left(t\right)}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{t}_0<t<t_f.$$

(23)

Using Equations (22) and (23) in Equation (10) yields the following expression for the indirect effect term:

$${\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}={\left\{{〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\delta f〉}_1-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\right\}}_{{\theta }^0}.$$

(24)

The boundary conditions accompanying Equation (23) for the function $a^{\left(1\right)}\left(t\right)$ are now chosen at the time values $t=t_f$ and/or $t=t_0$ so as to eliminate all unknown values of the 1st-level variational function $v^{\left(1\right)}\left(t\right)$ from the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ which remain after implementing the initial conditions provided in Equation (2). These boundary conditions for the function $a^{\left(1\right)}\left(t\right)$ can be represented in operator form as follows:

$$B_j^{*}\left[a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(25)

The Fredholm-like NIDE net represented by Equations (23) and (25) will be called the “1st-Level Adjoint Sensitivity System” and the solution, $a^{(1)}\left(t\right)$, will be called the “1st-level adjoint sensitivity function.” The 1st-LASS is solved using the nominal/optimal values for the parameters and for the function $h\left(t\right)$ but this fact has not been explicitly indicated in order to simplify the notation. Notably, the 1st-LASS is independent of any parameter variations so it needs to be solved just once to obtain the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$ The 1st-LASS is linear in $a^{(1)}\left(t\right)$ but is, in general, nonlinear in $h\left(t;x\right)$.

Adding the result obtained in Equation (24) for the indirect-effect term ${\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}$ to the result obtained in Equation (9) for the direct-effect term yields the following expression for the first-order G-differential of the response $R\left[h;f\left(\theta \right)\right]$:

$$\begin{array}{l}\delta R\left(h^0;f^0;v^{\left(1\right)};\delta F\right)={\left\{{〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\delta f〉}_1-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\right\}}_{{\theta }^0}\\ +{\displaystyle \sum _{i=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt{\left\{\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_i}\delta f_i\right\}}_{{\theta }^0}}}\triangleq {\left\{{\displaystyle \sum _{i=1}^{TF}{R}^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\delta f_i}\right\}}_{{\theta }^0},\end{array}$$

(26)

where $R^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq \partial R\left[u\left(x\right);f\left(\theta \right)\right]/\partial f_i$ denotes the first-order sensitivity of the response $R\left[u\left(x\right);f\left(\theta \right)\right]$ with respect to the components $f_i$ of the “feature”. Each sensitivity $R^{\left(1\right)}\left[i;u\left(x\right);a^{\left(1\right)}\left(x\right);f\left(\alpha \right)\right]$ is obtained by identifying the expression that multiplies the corresponding variation $\delta f_i$ and can be represented formally in the following integral form:

$$R^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{S}^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\hspace{0.17em}dt};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TF.$$

(27)

The functions $S^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ will be subsequently used for determining the exact expressions of the second-order sensitivities of the response with respect to the components of the feature function $f\left(\theta \right)$ of model parameters.

In the following subsections, the detailed forms of the 1st-LASS will be provided for first-order (n=1) and, respectively, second-order (n=2) Fredholm-like NIDE.

2.1. First-Order Neural Integral Equations of Fredholm-Type (1st-NIDE-F)

The representation of the first-order $\left(n=1\right)$ neural integral equations of Fredholm-type (1st-NIDE-F) is provided below, for$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH$:

$$c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}=g_i\left[h\left(t\right);f\left(\theta \right)\right]+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\hspace{0.17em}.$$

(28)

The typical boundary conditions provided at $t=t_0$ (“encoder”) are as follows:

$$h_i\left(t_0\right)=e_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em},$$

(29)

where the scalar values $e_i$ are known, albeit imprecisely, since they are considered to stem from experiments and/or computations. Equations (28) and (29) are customarily considered an “initial value (NIDE-F) problem” although the independent variable t could represent some other physical entity (e.g., space, energy, etc.) rather than time.

The 1st-LVSS for the function $v^{\left(1\right)}\left(t\right)$ is obtained by G-differentiating Equations (28) and (29), and has the following particular forms of Equations (13) and (14) for $n=1$:

$$\begin{array}{l}{\left\{c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]\right\}}_{{\theta }^0}{\displaystyle \frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}}+{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}\\ -{\left\{{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}\right\}}_{{\theta }^0}\\ -{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}={\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f\right)\right\}}_{{\theta }^0}\delta f_k},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(30)

$$v_i^{\left(1\right)}\left(t_0\right)=\delta e_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em};$$

(31)

where:

$$\begin{array}{l}{q}_{i,k}^{\left(1\right)}\left(h;f\right)\triangleq -{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_k}}+{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\phi }_{i,j}\left(f;t\right)}{\partial f_k}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }\\ +{\displaystyle \frac{\partial g_i\left(h;f;t\right)}{\partial f_k}}+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_k}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k=1,\dots ,TF.\end{array}$$

(32)

The 1st-LASS is constructed by using Equation (19) to form the inner product of Equation (30) with a vector $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in H_1\left({\Omega }_t\right)$ to obtain the following relationship:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_1\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}}\hspace{0.17em}.\end{array}$$

(33)

Examining the structure of the left-side of Equation (33) reveals that the bilinear concomitant will arise from the integration by parts of the first term the on the left-side of Equation (33) to obtain the following relation:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}dt}}=P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}}{dt}dt}},\end{array}$$

(34)

where the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ has the following expression, by definition:

$$P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\triangleq {\displaystyle \sum _{i=1}^{TH}\left\{a_i^{\left(1\right)}\left(t_f\right)c_{i,1}\left[h\left(t_f\right);f\right]v_i^{\left(1\right)}\left(t_f\right)-a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]v_i^{\left(1\right)}\left(t_0\right)\right\}}.$$

(35)

The second term on the left-side of Equation (33) will be recast in its “adjoint form” by reversing the order of summations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ to an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$${\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}dt}}\hspace{0.17em}.$$

(36)

The third term on the left-side of Equation (33) is now recast in its “adjoint form” by reversing the order of summations and integrations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ into an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\hspace{0.17em}{a}_i^{\left(1\right)}\left(t\right)}}{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}\hspace{0.17em}}{v}_k^{\left(1\right)}\left(\tau \right)d\tau }\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \hspace{0.17em}{v}_i^{\left(1\right)}\left(\tau \right){\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_i\left(\tau \right)}}}}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right){\phi }_{k,j}\left[f\left(\theta \right);t\right]\hspace{0.17em}}dt}.\end{array}$$

(37)

The fourth term on the left-side of Equation (33) will be recast in its “adjoint form” by reversing the order of summations and integrations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ into an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$${\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}.$$

(38)

Using the results obtained in Equations (34)‒(38) in the left-side of Equation (33) yields the following relation:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_1\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ =P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)dt\left\{-\frac{d}{dt}\right.}}\left[a_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\right]\\ +{\displaystyle \frac{\partial c_1\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}-{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }\\ \left.-{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\right\}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}}\hspace{0.17em}.\end{array}$$

(39)

The relation in Equation (39) is rearranged as follows:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta F_k}}-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)dt\left\{-\frac{d}{dt}\right.}}\left[a_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\right]+{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}\\ -{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }\left.-{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\right\}.\end{array}$$

(40)

The term on the right-side of Equation (40) is now required to represent the “indirect-effect” term defined in Equation (10), which is achieved by requiring the components of the function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ to satisfy the following system of first-order NIDE-F equations:

$$\begin{array}{l}-{\displaystyle \frac{d}{dt}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}+{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}\\ -{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }-{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\\ ={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.\end{array}$$

(41)

The relation obtained in Equation (41) is the explicit form of the relation provided in Equation (23) for the particular case when $n=1$, i.e., when considering first-order neural integral equations of Fredholm-type (1st-NIDE-F).

The unknown values $v_i\left(t_f\right)$ in the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ in Equation (40) are eliminated by imposing the following final-time conditions:

$$a_i^{\left(1\right)}\left(t_f\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.$$

(42)

It follows from Equations (33)‒(42) and (31) that the indirect-effect term defined in Equation (10) has the following expression in terms of the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$:

$${\left\{\delta R\left(h;f;a^{(1)}\right)\right\}}_{ind}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}+{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]\delta e_i\hspace{0.17em}}.$$

(43)

The first-order NIDE-F obtained in Equations (41) and (42) represents the explicit form for the particular case n=1 of the 1st-LASS represented, in general, by Equations (23) and (25). To obtain the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$, the 1st-LASS is solved backwards in time (globally) using the nominal/optimal values for the parameters and for the function $h\left(t\right)$ but this fact has not been explicitly indicated in order to simplify the notation. Notably, the 1st-LASS is independent of any parameter variations so it needs to be solved just once to obtain the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$ The 1st-LASS is linear in $a^{(1)}\left(t\right)$ but is, in general, nonlinear in $h\left(t;x\right)$.

Using the results obtained in Equations (43) and (9) in Equation (8) yields the following expression for the G-variation $\delta R\left(h^0;F^0;v^{\left(1\right)};\delta f\right)$, which is seen to be linear in the variations $\delta f_j$, $j=1,\dots ,TF$, in the model’s feature functions (induced by variations in the model’s primary parameters) and the variations $\delta e_i$, $i=1,\dots ,TH$ in the decoder’s initial conditions:

$$\begin{array}{l}\delta R\left[h\left(t\right);f;a^{(1)}\left(t\right);\delta f\right]\triangleq {\displaystyle \sum _{j=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\frac{\partial D\left[h\left(t\right);f;t\right]}{\partial F_j}\delta F_j}}+{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]\delta e_i}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{j=1}^{TF}{q}_{jk}^{\left(1\right)}\left(h;f;t\right)\delta f_j}}\triangleq {\displaystyle \sum _{j=1}^{TF}\frac{\partial R}{\partial f_j}}\delta f_j+{\displaystyle \sum _{i=1}^{TH}\frac{\partial R}{\partial e_i}}\delta e_i\hspace{0.17em}.\end{array}$$

(44)

The expression in Equation (44) is to be satisfied at the nominal/optimal values for the respective model parameters, but this fact has not been indicated explicitly in order to simplify the notation.

Identifying in Equation (44) the expressions that multiply the variations $\delta e_i\hspace{0.17em}$ yields the following expressions for the decoder response sensitivities with respect to the encoder’s initial conditions:

$${\displaystyle \frac{\partial R}{\partial e_i}}=a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\delta \left(t-t_0\right)dt};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH.$$

(45)

It is apparent from Equation (45) that the sensitivities $\partial R/\partial e_i$ are functionals of the form predicted in Equation (27). It is also apparent from Equation (45) that the sensitivities $\partial R/\partial e_i$are proportional to the values of the respective component $a_i^{\left(1\right)}\left(t_0\right)$ of the 1st-level adjoint function evaluated at the initial-time $t=t_0$. This relation provides an independent mechanism for verifying the correctness of solving the 1st-LASS from $t=t_f$ to $t=t_0$ (backwards in time) since the sensitivities $\partial R/\partial e_i$ can be computed independently of the 1st-LASS by using finite differences of appropriately high-order in conjunction with known variations $\delta e_i\hspace{0.17em}$ and the correspondingly induced variations in the decoder response. Special attention needs to be devoted, however, to ensure that the respective finite-difference formula is accurate, which may need several trials with different values chosen for the variation $\delta e_i\hspace{0.17em}$.

It also follows from Equations (44) and (32) that the sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ have the following expressions, written in the form of Equation (27):

$$\partial R/\partial f_j\triangleq R^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{S}_1^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]}\hspace{0.17em}dt;\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TF;$$

(46)

where

$$\begin{array}{l}{S}_1^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}}-{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{d{h}_i\left(t\right)}{dt}\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}}\\ +{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{\partial g_i\left(f;t;t\right)}{\partial f_j}}+{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{k=1}^{TL}\frac{\partial {\phi }_{i,k}\left(f;t\right)}{\partial f_j}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }}\\ +{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{k=1}^{TL}{\phi }_{i,k}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_j}}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TF.\end{array}$$

(47)

The subscript “1” attached to the quantity $S_1^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ indicates that this quantity refers to a “first-order” NIDE-F net, while the superscript “(1)” indicates that this quantity refers to “first-order” sensitivities.

The sensitivities with respect to the primary model parameters can be obtained by using the result shown in Equation (46) together with the “chain rule” of differentiating compound functions, as follows:

$${\displaystyle \frac{\partial R}{\partial {\theta }_j}}={\displaystyle \sum _{i=1}^{TF}\frac{\partial R}{\partial f_i}\frac{\partial f_i}{\partial {\theta }_j}}\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TW.$$

(48)

When there only model parameters (i.e., there are no feature functions of model parameters), then $f_i\left(\theta \right)\equiv {\theta }_i$ for all $i=1,\dots ,TF\triangleq TW$, and the expression obtained in Equation (46) yields directly the first-order sensitivities $\partial R/\partial {\theta }_j$, for all $j=1,\dots ,TW$. In this case, all of the sensitivities $\partial R/\partial {\theta }_j$, for all $j=1,\dots ,TW$ would be obtained by computing integrals (using quadrature formulas). In contradistinction, when features of parameters can be established, only $TF$ $\left(TF<TW\right)$ integrals would need to be computed (using quadrature formulas) to obtain the $\partial R/\partial F_j$, $j=1,\dots ,TF$; the sensitivities with respect to the model parameters would subsequently be obtained analytically using the chain-rule provided in Equation (48).

Occasionally, the boundary conditions may be provided through a measurement at the boundary $t=t_f$ (“decoder”), as follows:

$$h_i\left(t_f\right)=d_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em},$$

(49)

where the scalar values $d_i$ are known, albeit imprecisely, since they are considered to stem from experiments and/or computations. In such a case, the determination of the first-order sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ follows the same steps as in Section 2.1.2, above, yielding the following results:

(i)

The 1st-LASS will become an “initial value problem” comprising Equation (41), subject not the conditions shown in Equation (42), but subject to the following “initial conditions

$$a_i^{\left(1\right)}\left(t_0\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.$$

(50)

(ii)

The sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ will have the same formal expressions as in Equation (46) but the components of the 1st-level adjoint function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ will be the solution of Equations (41) and (50).

(iii)

The sensitivities of the response $R\left[h;f\left(\theta \right)\right]$ with respect to boundary conditions at $t=t_f$ will have the following expressions:

$${\displaystyle \frac{\partial R}{\partial d_i}}=a_i^{\left(1\right)}\left(t_f\right)c_{i,1}\left[h\left(t_f\right);f\right];\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH.$$

(51)

2.2. Second-Order Neural Integral Equations of Fredholm-Type (2nd-NIDE-F)

The representation of the second-order $\left(n=2\right)$ neural integral equations of Fredholm-type (2nd-NIDE-F) is provided below, for$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH$:

$$\begin{array}{l}{c}_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}+c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}=g_i\left[h\left(t\right);f\left(\theta \right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\hspace{0.17em}d\tau \hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH.\end{array}$$

(52)

There are several combinations of boundary conditions that can be provided, either for the function $h_i\left(t\right)$ and/or for its first-derivative $d{h}_i\left(t\right)/dt$, $i=1,\dots ,TH$, at either $t=t_0$ (encoder) or at $t=t_f$ (decoder), or a combination thereof. For illustrative purposes, consider that the boundary conditions are as follows:

$$h_i\left(t_0\right)=e_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{h}_i\left(t_f\right)=d_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.$$

(53)

The 1st-LVSS is obtained by taking the G-variations of Equations (52) and (53) to obtain the following system, comprising the forms taken on for $n=2$ by Equations (13) and (14), respectively:

$$\begin{array}{l}{\left\{c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]\right\}}_{{\theta }^0}{\displaystyle \frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}}+{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}\\ +{\left\{c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]\right\}}_{{\theta }^0}{\displaystyle \frac{d^2{v}_i^{\left(1\right)}}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}\\ -{\left\{{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}\right\}}_{{\theta }^0}\\ -{{\displaystyle \sum _{k=1}^{TH}\left\{\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}}_{{\theta }^0}={\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f\right)\right\}}_{{\theta }^0}\delta f_k},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(54)

$$v_i^{\left(1\right)}\left(t_0\right)=\delta e_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{v}_i^{\left(1\right)}\left(t_f\right)=\delta d_i;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em};$$

(55)

where for $i=1,\dots ,TH;$and $k=1,\dots ,TF$:

$$\begin{array}{l}{q}_{ik}^{\left(1\right)}\left(h;f\right)\triangleq -{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_k}}-{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_k}}+{\displaystyle \frac{\partial g_i\left(h;f;t\right)}{\partial f_k}}\\ +{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\phi }_{i,j}\left(f;t\right)}{\partial f_k}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_k}}.\end{array}$$

(56)

The 1st-LASS is constructed by using Equation (19) to form the inner product of Equation (54)with a vector $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in H_1\left({\Omega }_t\right)$ to obtain the following relationship:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}}\hspace{0.17em}.\end{array}$$

(57)

Examining the structure of the left-side of Equation (57) reveals that the bilinear concomitant will arise from the integration by parts of the first and third terms the on the left-side of Equation (57), as follows:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h;f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,2}\left[h;f\right]\frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}dt}}=P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d}{dt}\left\{a_i^{\left(1\right)}\left(t\right)c_{i.1}\left[h\left(t\right);f\right]\right\}dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d^2}{d{t}^2}\left\{a_i^{\left(1\right)}\left(t\right)c_{i.2}\left[h\left(t\right);f\right]\right\}dt}},\end{array}$$

(58)

where the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ has the following expression:

$$\begin{array}{l}P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\triangleq {\displaystyle \sum _{i=1}^{TH}\left\{a_i^{\left(1\right)}\left(t_f\right)c_{i,1}\left[h\left(t_f\right);f\right]v_i^{\left(1\right)}\left(t_f\right)-a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]v_i^{\left(1\right)}\left(t_0\right)\right\}}\\ +{\displaystyle \sum _{i=1}^{TH}\left\{a_i^{\left(1\right)}\left(t_f\right)c_{i,2}\left[h\left(t_f\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t_f\right)}{dt}-a_i^{\left(1\right)}\left(t_0\right)c_{i,2}\left[h\left(t_0\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t_0\right)}{dt}\right\}}\\ -{{\displaystyle \sum _{i=1}^{TH}{v}_i^{\left(1\right)}\left(t_f\right)\left\{a_i^{\left(1\right)}\left(t\right)\frac{d{c}_{i,2}\left[h\left(t\right);f\right]}{dt}+c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_f}\\ +{{\displaystyle \sum _{i=1}^{TH}{v}_i^{\left(1\right)}\left(t_0\right)\left\{a_i^{\left(1\right)}\left(t\right)\frac{d{c}_{i,2}\left[h\left(t\right);f\right]}{dt}+c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_0}\end{array}$$

(59)

The remaining terms on the left-side of Equation (57) will be recast into their corresponding “adjoint form” by using the results obtained in Equations (34)‒(38). Using these results together with the results obtained in Equations (58) and (59) yields the following expression for the left-side Equation (57):

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ =P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)-{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d}{dt}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}dt}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d^2}{d{t}^2}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left[h\left(t\right);f\right]\right\}dt}}\hspace{0.17em}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}dt}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}}-{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right)f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \hspace{0.17em}{v}_i^{\left(1\right)}\left(\tau \right){\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_i\left(\tau \right)}}}}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right){\phi }_{k,j}\left[f\left(\theta \right);t\right]\hspace{0.17em}}dt}\end{array}$$

(60)

Using Equation (58) and rearranging the terms on the right-side of Equation (60) yields the following relation:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta F_k}}-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)dt\left\{-\frac{d}{dt}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}\right.}}+{\displaystyle \frac{d^2}{d{t}^2}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left[h\left(t\right);f\right]\right\}\\ +{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}+{\displaystyle \frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}\\ -\left.{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}-{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }\right\}.\end{array}$$

(61)

The term on the right-side of Equation (61) is now required to represent the “indirect-effect” term defined in Equation (10), which is achieved by requiring the components of the function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ to satisfy the following 1st-LASS:

$$\begin{array}{l}-{\displaystyle \frac{d}{dt}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}+{\displaystyle \frac{d^2}{d{t}^2}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left[h\left(t\right);f\right]\right\}\\ +{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}+{\displaystyle \frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}\\ -{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}-{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }\\ ={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}.\end{array}$$

(62)

The relation obtained in Equation (62) is the explicit form of the relation provided in Equation (23) for the particular case when $n=2$, i.e., when considering second-order neural integral equations of Fredholm-type (2nd-NIDE-F).

The unknown values involving the function $v_i\left(t\right)$ in the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ defined in Equation (59) are eliminated by imposing the following conditions:

$$a_i^{\left(1\right)}\left(t_0\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{a}_i^{\left(1\right)}\left(t_f\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.$$

(63)

It follows from Equations (33)‒(42) and (31) that the indirect-effect term defined in Equation (10) has the following expression in terms of the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$:

$${\left\{\delta R\left(h;f;a^{(1)}\right)\right\}}_{ind}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{ik}^{\left(1\right)}\left(h;f;t\right)\delta f_k}-\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right),$$

(64)

where the boundary quantity $\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ contains the known remaining terms after having implemented the known boundary conditions given in Equations (55) and (63), and has the following explicit expression:

$$\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\triangleq -{{\displaystyle \sum _{i=1}^{TH}\delta d_i\left\{c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_f}+{{\displaystyle \sum _{i=1}^{TH}\delta e_i\left\{c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_0}.$$

(65)

Using the results obtained in Equations (64), (65), (56) and (9) in Equation (8) yields the following expression for the G-variation $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)$, which is seen to be linear in the variations $\delta d_i$, $\delta e_i$ ($i=1,\dots ,TH$) and $\delta f_j$ ($j=1,\dots ,TF$):

$$\begin{array}{l}\delta R\left[h\left(t\right);f\left(\theta \right);a^{(1)}\left(t\right);\delta f\right]\triangleq {\displaystyle \sum _{j=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}\delta f_j}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{j=1}^{TF}{q}_{jk}^{\left(1\right)}\left(h;f;t\right)\delta f_j}}+{{\displaystyle \sum _{i=1}^{TH}\delta d_i\left\{c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_f}\\ -{{\displaystyle \sum _{i=1}^{TH}\delta e_i\left\{c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_0}\triangleq {\displaystyle \sum _{j=1}^{TF}\frac{\partial R}{\partial f_j}}\delta f_j+{\displaystyle \sum _{i=1}^{TH}\frac{\partial R}{\partial d_i}}\delta d_i+{\displaystyle \sum _{i=1}^{TH}\frac{\partial R}{\partial e_i}}\delta e_i\hspace{0.17em}\hspace{0.17em}.\end{array}$$

(66)

The expression in Equation (66) is to be satisfied at the nominal/optimal values for the respective model parameters, but this fact has not been indicated explicitly in order to simplify the notation.

It also follows from Equations (66) and (56) that the sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ have the following expressions, written in the form of Equation (27):

$$\partial R/\partial f_j\triangleq R^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{S}_2^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]}\hspace{0.17em}dt;\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TF;$$

(67)

where

$$\begin{array}{l}{S}_2^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq {\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}}-{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{d{h}_i\left(t\right)}{dt}\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}}\\ -{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{d^2{h}_i\left(t\right)}{d{t}^2}\frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}}+{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{\partial g_i\left(f;t;x\right)}{\partial f_j}}\\ +{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{k=1}^{TL}\frac{\partial {\phi }_{i,k}\left(f;t\right)}{\partial f_j}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }}\\ +{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{k=1}^{TL}{\phi }_{i,k}\left(f;t\right)}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_j}}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TF.\end{array}$$

(68)

The subscript “2” attached to the quantity $S_2^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ indicates that this quantity refers to a “second-order” NIDE-F net, while the superscript “(1)” indicates that this quantity refers to “first-order” sensitivities. As expected, the expression of $S_2^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ reduces to the expression of $S_1^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ when the “second-order NIDE-F net” reduces to the “first-order NIDE-F net” in the case when $c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]\equiv 0$.

Identifying in Equation (66) the expressions that multiply the variations $\delta e_i\hspace{0.17em}$ yields the following expressions for the decoder response sensitivities with respect to the encoder’s initial-time conditions:

$${\displaystyle \frac{\partial R}{\partial e_i}}=-{\left\{c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}}\right\}}_{t=t_0}=-{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{c}_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\delta \left(t-t_0\right)dt};\hspace{0.17em}i=1,\dots ,TH.$$

(69)

Identifying in Equation (66) the expressions that multiply the variations $\delta d_i\hspace{0.17em}$ yields the following expressions for the decoder response sensitivities with respect to the final-time conditions:

$${\displaystyle \frac{\partial R}{\partial d_i}}=-{\left\{c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}}\right\}}_{t=t_f}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{c}_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\delta \left(t-t_f\right)dt};\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH.$$

(70)

If the boundary conditions imposed on the forward functions $h_i\left(t\right)$ and/or the first-derivatives $d{h}_i\left(t\right)/dt$, $i=1,\dots ,TH$, differ from the illustrative ones selected in Equation (53), then the corresponding boundary conditions for the 1st-level adjoint function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ would also differ from the ones shown in Equation (63), as would be expected. The components of $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ would consequently have different values; therefore, all of the first-order sensitivities $\partial R/\partial f_j$ would have values different from those computed using Equation (68), even though the formal mathematical expressions of the respective sensitivities would remain unchanged. Of course, the sensitivities $\partial R/\partial e_i$ and $\partial R/\partial d_i$ would have expressions that would differ from those in Equations (69) and (70), respectively, if the boundary conditions in Equation (53), and consequently those in Equation (63), were different, since the residual bilinear concomitant $\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ would have a different expression from that shown in Equation (65).

3. Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm Type (2nd-FASAM-NIDE-F)

The second-order sensitivities of the response $R\left[h;f\left(\theta \right)\right]$ defined in Equation (5) will be computed by conceptually using their basic definitions as being the “first-order sensitivities of the first-order sensitivities.” Recall that the generic expression of the first-order sensitivities, $R^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$, $j_1=\hspace{0.17em}1,\dots ,TF$, of the response with respect to the components of the feature function $f\left(\theta \right)$ is provided in Equation (46). It follows that the second-order sensitivities of the response with respect to the components of the feature function will be provided by the first-order G-differential $\delta R^{\left(1\right)}$ of $R^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$, which is by definition obtained as follows:

$$\begin{array}{l}\delta R^{\left(1\right)}\left[j_1;h^0\left(t\right);a^{\left(1,0\right)}\left(t\right);f^0\left(\theta \right);v^{\left(1\right)}\left(x\right);\delta a^{\left(1\right)}\left(x\right);\delta f\right]\\ \triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}\delta R^{\left(1\right)}\left[j_1;h^0\left(x\right)+\epsilon v^{\left(1\right)}\left(x\right);a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right);f^0+\epsilon \delta f\right]\right\}}_{\epsilon =0}\\ ={\displaystyle \sum _{j_2=1}^{TF}{\left\{\frac{\partial R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f\right]}{\partial f_{j_2}}\right\}}_{{\theta }^0}}\delta f_{j_2}+{\left\{\delta R^{\left(1\right)}\left[j_1;h;a^{\left(1\right)};f;v^{\left(1\right)}\left(x\right);\delta a^{\left(1\right)}\left(x\right)\right]\right\}}_{ind},\end{array}$$

(71)

where the indirect-effect term ${\left\{\delta R^{\left(1\right)}\left[j_1;h;a^{\left(1\right)};f;v^{\left(1\right)}\left(x\right);\delta a^{\left(1\right)}\left(x\right)\right]\right\}}_{ind}$ comprises all dependencies on the vectors $v^{\left(1\right)}\left(x\right)$ and $\delta a^{\left(1\right)}\left(x\right)$ of variations in the state functions $h\left(t\right)$ and $a^{\left(1\right)}\left(t\right)$, around the respective nominal values denoted as $h^0\left(t\right)$ and $a^{\left(1,0\right)}\left(t\right)$, respectively, which are computed at the nominal parameter values ${\theta }^0$. This indirect-effect term is defined as follows:

$$\begin{array}{l}{\left\{\delta R^{\left(1\right)}\left[j_1;h;a^{\left(1\right)};f;v^{\left(1\right)};\delta a^{\left(1\right)}\right]\right\}}_{ind}\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt}{\left\{{\displaystyle \frac{\partial S_1^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]}{\partial u}}{v}^{\left(1\right)}\left(x\right)\right\}}_{{\theta }^0}\\ +{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt}{\left\{{\displaystyle \frac{\partial S_1^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]}{\partial a^{\left(1\right)}}}\delta a^{\left(1\right)}\left(x\right)\right\}}_{{\theta }^0};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1=1,\dots ,TF.\end{array}$$

(72)

The variational function $\delta a^{\left(1\right)}\left(x\right)$ is the solution of the system of equations obtained by G-differentiating the 1st-LASS defined in Equations (23) and (25), which is by definition obtained as follows:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \sum _{i=1}^{TH}{A}_{ij}^{\left(1\right)}\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]\left[a_i^{\left(1,0\right)}+\epsilon \delta a_i^{\left(1\right)}\right]}\right\}}_{\epsilon =0}\\ ={\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \frac{\partial D\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]}{\partial h_j\left(t\right)}}\right\}}_{\epsilon =0};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TH.\end{array}$$

(73)

$${\left\{{\displaystyle \frac{d}{d\epsilon }}{B}_j^{*}\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;a^{\left(1,0\right)}+\epsilon \delta a^{\left(1\right)};t\right]\right\}}_{\epsilon =0};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(74)

Carrying out the operations indicated in Equations (73) and (74) yields the following relations:

$$\begin{array}{l}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial }{\partial h_k\left(t\right)}\left[{\displaystyle \sum _{i=1}^{TH}{A}_{ij}^{\left(1\right)}\left(h;f;t\right)a_i^{\left(1\right)}\left(j_1;t\right)}\right]-\frac{{\partial }^{2}D\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]}{\partial h_k\left(t\right)\partial h_{j_1}\left(t\right)}\right\}}_{{\theta }^0}}{v}_k^{\left(1\right)}\left(t\right)\\ +{\left\{{\displaystyle \sum _{i=1}^{TH}{A}_{ij}^{\left(1\right)}\left(h;f;t\right)}\right\}}_{{\theta }^0}\delta a_i^{\left(1\right)}\left(j_1;t\right)={\displaystyle \sum _{j_2=1}^{TF}{q}_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)\hspace{0.17em}}\delta f_{j_2}\left(\theta \right);\hspace{0.17em}\hspace{0.17em}\\ \hspace{0.17em}j=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}{j}_1=1,\dots ,TF.\end{array}$$

(75)

$$\begin{array}{l}{q}_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)\triangleq {\left\{{\displaystyle \frac{{\partial }^{2}D\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]}{\partial f_{j_2}\left(\theta \right)\partial h_{j_1}\left(t\right)}}\right\}}_{{\theta }^0}\\ -{\left\{{\displaystyle \frac{\partial }{\partial f_{j_2}\left(\theta \right)}}{\displaystyle \sum _{i=1}^{TH}{A}_{ij}^{\left(1\right)}\left(h;f;t\right)a_i^{\left(1\right)}\left(j_1;t\right)}\right\}}_{{\theta }^0};\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1,j_2=1,\dots ,TF.\end{array}$$

(76)

$$\begin{array}{l}{\left\{{\displaystyle \frac{\partial B_j^{*}\left(h;f;a^{\left(1\right)}\right)}{\partial h\left(t\right)}}\right\}}_{{\theta }^0}{v}^{\left(1\right)}\left(t\right)+{\left\{{\displaystyle \frac{\partial B_j^{*}\left(h;f;a^{\left(1\right)}\right)}{\partial a^{\left(1\right)}\left(t\right)}}\right\}}_{{\theta }^0}\delta a^{\left(1\right)}\left(t\right)+{\left\{{\displaystyle \frac{\partial B_j^{*}\left(h;f;a^{\left(1\right)}\right)}{\partial f\left(\theta \right)}}\right\}}_{{\theta }^0}\delta f\left(\theta \right)=0;\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.\end{array}$$

(77)

For subsequent derivations, it is convenient to represent the relations in Equation (75) in matrix-vector form, as follows:

$$V_{21}^{\left(2\right)}\left(u;a^{\left(1\right)};f\right)v^{\left(1\right)}\left(t\right)+V_{22}^{\left(2\right)}\left(u;f\right)\delta a^{\left(1\right)}\left(t\right)=Q^{\left(2\right)}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right);$$

(78)

where

$$\begin{array}{l}{V}_{21}^{\left(2\right)}\left(h;a^{\left(1\right)};f\right)\triangleq {\left\{{\displaystyle \frac{\partial A^{\left(1\right)}\left(h;a^{\left(1\right)};f\right)}{\partial h\left(t\right)}}-{\displaystyle \frac{{\partial }^{2}D\left(h;a^{\left(1\right)};f\right)}{\partial h\left(t\right)\partial h\left(t\right)}}\right\}}_{{\theta }^0};\\ V_{22}^{\left(2\right)}\left(h;a^{\left(1\right)};f\right)\triangleq {\left\{{\displaystyle \frac{\partial A^{\left(1\right)}\left(h;a^{\left(1\right)};f\right)}{\partial a^{\left(1\right)}\left(t\right)}}\right\}}_{{\theta }^0};\\ Q^{\left(2\right)}\left[j_1;j_2;h;a^{\left(1\right)};f\left(\theta \right)\right]\triangleq {\left\{q_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)\right\}}_{TH\times TF};\end{array}$$

(79)

As indicated by Equation (78), the variational functions $v^{\left(1\right)}\left(x\right)$ and $\delta a^{\left(1\right)}\left(x\right)$ are the solutions of the system of matrix equations obtained by concatenating the 1st-LVSS defined by Equations (14) and (16) with Eqs (77) and (78). The concatenated system thus obtained will be called the 2nd-Level Variational Sensitivity System (2nd-LVSS) and has the block-matrix form provided below:

$${\left\{V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]V^{\left(2\right)}\left(2;t\right)\right\}}_{{\theta }^0}={\left\{Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]\hspace{0.17em}\right\}}_{{\theta }^0};\hspace{0.17em}\hspace{0.17em}{t}_0<t<t_f;$$

(80)

$${\left\{B_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);V^{\left(2\right)}\left(2;t\right);f;\delta f\right]\right\}}_{{\theta }^0}=0\left[2\right]\triangleq {\left[0,0\right]}^{†};\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em}.$$

(81)

To distinguish block-matrices from block-vectors, two bold capital-letters have been used (and will henceforth be used) to denote block-matrices, as in the case of “the second-level variational matrix” $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$. The “2nd-level” is indicated by the superscript “(2)”. The argument “$2\times 2$”, which appears in the list of arguments of $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$, indicates that this matrix is a $2\times 2$-dimensional block-matrix comprising four submatrices, each of dimensions $TD\times TD$. The structure of the block-matrix $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ is provided below:

$$V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\triangleq \left(\begin{array}{cc}L\left[h;f\left(\theta \right);t\right]& 0\\ V_{21}^{\left(2\right)}\left(h;a^{\left(1\right)};f;t\right)& V_{22}^{\left(2\right)}\left(h;a^{\left(1\right)};f;t\right)\end{array}.\right)$$

(82)

The argument “2” which appears in the list of arguments of the vector $U^{\left(2\right)}\left(2;t\right)$ and of the “variational vector” $V^{\left(2\right)}\left(2;t\right)$ in Equation (80) indicates that each of these vectors is a 2-block column vector, each block comprising a column-vector of dimension $TD$; the vectors $U^{\left(2\right)}\left(2;t\right)$ and $V^{\left(2\right)}\left(2;t\right)$ are defined as follows:

$$U^{\left(2\right)}\left(2;t\right)\triangleq \left(\begin{array}{c}h\left(t\right)\\ a^{\left(1\right)}\left(t\right)\end{array}\right);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{V}^{\left(2\right)}\left(2;t\right)\hspace{0.17em}\triangleq \left(\begin{array}{c}{v}^{\left(1\right)}\left(t\right)\\ \delta a^{\left(1\right)}\left(t\right)\end{array}\right).$$

(83)

The 2-block vector $Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]$ is defined as follows:

$$Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]\hspace{0.17em}\triangleq \left(\begin{array}{c}{Q}^{\left(1\right)}\left[h\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right)\\ Q^{\left(2\right)}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right)\end{array}\right)\hspace{0.17em};$$

(84)

The 2-block column vector $B_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);V^{\left(2\right)}\left(2;t\right);f;\delta f\right]$ in Equation (81) represents the concatenated boundary/initial conditions provided in Equations (14) and (77), evaluated at the nominal parameter values. The argument “2” in the expression $0\left[2\right]\triangleq {\left[0,0\right]}^{†}$ in Equation (81) indicates that this expression is a two-block column vector comprising two vectors, each of which has $TD$-components, all of which are zero-valued.

The need for solving the 2nd-LVSS is circumvented by deriving an alternative expression for the indirect-effect term ${\left\{\delta R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f;v^{\left(1\right)};\delta a^{\left(1\right)}\right]\right\}}_{ind}$ defined in Equation (72), in which the function $V^{\left(2\right)}\left(2;t\right)\hspace{0.17em}$ is replaced by a 2nd-level adjoint function that is independent of variations in the model parameter and state functions. This 2nd-level adjoint function will be the solution of a 2nd-Level Adjoint Sensitivity System (2nd-LASS), which will be constructed by using the same principles as employed for deriving the 1st-LASS. The 2nd-LASS is constructed in a Hilbert space $H_2\left({\Omega }_t\right)$, ${\Omega }_t\triangleq t\in \left[t_0,t_f\right]$, comprising block-vectors having the same structure as $V^{\left(2\right)}\left(2;t\right)$ that can generically be represented as follows: ${\Phi }^{(2)}\left(2;t\right)\triangleq {\left[{\phi }^{(2)}\left(1;t\right),{\phi }^{(2)}\left(2;t\right)\right]}^{†}\in H_2\left({\Omega }_t\right)$, with ${\phi }^{(2)}\left(i;t\right)\triangleq {\left[{\phi }_{i,1}^{\left(2\right)}\left(t\right),\dots {\phi }_{i,j}^{\left(2\right)}\left(t\right),\dots ,{\phi }_{i,TH}^{\left(2\right)}\left(t\right)\right]}^{†}$, for $i=1,2$. The Hilbert space $H_2\left({\Omega }_t\right)$ is endowed with the following inner product of two vectors ${\Phi }^{(2)}\left(2;t\right)\in H_2\left({\Omega }_t\right)$ and ${\Psi }^{(2)}\left(2;t\right)\in H_2\left({\Omega }_t\right)$:

$$\begin{array}{l}{〈{\Psi }^{(2)}\left(2;t\right),{\Phi }^{(2)}\left(2;t\right)〉}_2\triangleq {\displaystyle \sum _{i=1}^2{〈{\psi }^{(2)}\left(i;t\right),{\phi }^{(2)}\left(i;t\right)〉}_1}\\ ={\displaystyle \sum _{j=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\chi }_{1,j}^{\left(2\right)}\left(t\right){\eta }_{1,j}^{\left(2\right)}\left(t\right)dt}\hspace{0.17em}}+{\displaystyle \sum _{j=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\chi }_{2,j}^{\left(2\right)}\left(t\right){\eta }_{2,j}^{\left(2\right)}\left(t\right)dt}}\hspace{0.17em}.\end{array}$$

(85)

The inner product defined in Equation (85) will be used to construct the 2nd-Level Adjoint Sensitivity System (2nd-LASS) for a 2nd-level adjoint function $A^{(2)}\left(2;t\right)\triangleq {\left[a^{(2)}\left(1;t\right),a^{(2)}\left(2;t\right)\right]}^{†}\in H_2\left({\Omega }_t\right)$, $a_i^{\left(2\right)}\left(t\right)\triangleq {\left[a_{i,1}^{\left(2\right)}\left(t\right),\dots ,a_{i,TH}^{\left(2\right)}\left(t\right)\right]}^{†}$,$i=1,2$, by implementing the following sequence of steps, which are conceptually similar to those implemented in Section 2 for constructing the 1st-FASAM-NIDE-F methodology:

• Using Equation (85), construct the inner product of the yet undetermined function $A^{(2)}\left(2;t\right)\triangleq {\left[a^{(2)}\left(1;t\right),a^{(2)}\left(2;t\right)\right]}^{†}\in H_2\left({\Omega }_t\right)$ with Equation (80) to obtain the following relation:

  $$\begin{array}{l}{\left\{{〈A^{(2)}\left(2;t\right),V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]V^{\left(2\right)}\left(2;t\right)〉}_2\right\}}_{{\theta }^0}\\ ={\left\{{〈A^{(2)}\left(2;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}.\end{array}$$

  (86)
• Use the definition of the operator adjoint to $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ in the Hilbert space $H_2\left({\Omega }_t\right)$ to transform the inner product on the left-side of Equation (86) as follows:

  $$\begin{array}{l}{\left\{{〈A^{(2)}\left(2;t\right),V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]V^{\left(2\right)}\left(2;t\right)〉}_2\right\}}_{{\theta }^0}={\left\{P^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f\right)\right\}}_{{\theta }^0}\\ +{\left\{{〈V^{\left(2\right)}\left(2;t\right),A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]A^{(2)}\left(2;t\right)〉}_2\right\}}_{{\theta }^0},\end{array}$$

  (87)

  where the quantity ${\left\{P^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f\right)\right\}}_{{\theta }^0}$ denotes the corresponding bilinear concomitant on the domain’s boundary, evaluated at the nominal values for the parameters and respective state functions, and where the operator $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\triangleq {\left\{V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\right\}}^{*}$ denotes the formal adjoint of the matrix-valued operator $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$, comprising $\left(2\times 2\right)$ block-matrices, each of dimensions $T{D}^2$, having the following block-matrix structure.

  $$A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\triangleq {\left(\begin{array}{cc}L& 0\\ V_{21}^{\left(2\right)}& V_{22}^{\left(2\right)}\end{array}\right)}^{*}=\left(\begin{array}{cc}{\left\{L^{*}\right\}}^{†}& {\left[V_{21}^{\left(2\right)*}\right]}^{†}\\ 0& {\left[V_{22}^{\left(2\right)*}\right]}^{†}\end{array}\right).$$

  (88)
• Require the inner product on the right-side of Equation (87) to represent the indirect-effect term ${\left\{\delta R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f;v^{\left(1\right)};\delta a^{\left(1\right)}\right]\right\}}_{ind}$ defined in Equation (72) by imposing the following relation:

  $${\left\{A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]A^{(2)}\left(2;j_1;t\right)\right\}}_{{\theta }^0}={\left\{Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]\right\}}_{{\theta }^0},\hspace{0.17em}{j}_1=1,\dots ,TF;$$

  (89)

  where

  $$Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]\triangleq \left(\begin{array}{c}\partial S_1^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]/\partial u\\ \partial S_1^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]/\partial a^{\left(1\right)}\end{array}\right),\hspace{0.17em}\hspace{0.17em}{j}_1=1,\dots ,TF.$$

  (90)
  Since the source-term on the right-side of Equation (89) is a distinct quantity for each value of the index $j_1$, this index has been added to the list of arguments of the function $A^{(2)}\left(2;j_1;t\right)\triangleq {\left[a^{(2)}\left(1;j_1;t\right),a^{(2)}\left(2;j_1;t\right)\right]}^{†}$ in order to emphasize that a distinct function $A^{(2)}\left(2;j_1;t\right)$ will correspond to each index $j_1$. Of course, the adjoint operator $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ that acts on the function $A^{(2)}\left(2;j_1;t\right)$ is independent of the index $j_1$ and could, in principle, be inverted just once and stored for subsequent repeated applications to the $j_1$-dependent source terms $Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]$ for computing the corresponding functions $A^{(2)}\left(2;j_1;t\right)$.
• The definition of the function $A^{(2)}\left(2;j_1;t\right)$ is completed by requiring it to satisfy adjoint boundary/initial conditions represented in operator form as follows:

  $${\left\{B_A^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);A^{(2)}\left(2;j_1;t\right);f\right]\right\}}_{{\theta }^0}=0\left[2\right];\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1=1,\dots ,TF;\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0.$$

  (91)
  The boundary/initial conditions represented by Equation (91) are determined imposing the following requirements:
  (a) they must be independent of unknown values of $V^{\left(2\right)}\left(2;t\right)$;

(b) the substitution of the boundary and/or initial conditions represented by Equations (81) and (91) into the expression of the bilinear concomitant ${\left\{P^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f\right)\right\}}_{{\theta }^0}$ must cause all terms containing unknown boundary/initial values of $V^{\left(2\right)}\left(2;t\right)$ to vanish.

The NIDE-net comprising Equations (89) and (91) is called the “2nd-Level Adjoint Sensitivity System (2nd-LASS)” and its solution, $A^{(2)}\left(2;j_1;t\right)\triangleq {\left[a^{(2)}\left(1;j_1;t\right),a^{(2)}\left(2;j_1;t\right)\right]}^{†}$, $\hspace{0.17em}{j}_1=1,\dots ,TF$, is called the “2nd-level adjoint sensitivity function.” The unique properties of the 2nd-LASS will be highlighted in the sequel, below.

Using in Equation (72) the relations defining 2nd-LASS together with the 2nd-LVSS and the relation provided in Equation (87) yields the following alternative expression for the indirect-effect term, involving the 2nd-level adjoint sensitivity function $A^{(2)}\left(2;j_1;x\right)\triangleq {\left[a^{(2)}\left(1;j_1;x\right),a^{(2)}\left(2;j_1;x\right)\right]}^{†}$ instead of the 2nd-level variational function $V^{\left(2\right)}\left(2;t\right)$:

$$\begin{array}{l}{\left\{\delta R^{\left(1\right)}\left[j_1;h;a^{\left(1\right)};f;A^{(2)}\left(2;j_1;x\right)\right]\right\}}_{ind}=-{\left\{{\widehat{P}}^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f;\delta f\right)\right\}}_{{\theta }^0}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\left\{{〈A^{(2)}\left(2;j_1;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}\end{array}$$

(92)

where ${\left\{{\widehat{P}}^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f;\delta f\right)\right\}}_{{\theta }^0}$ denotes known residual (non-zero) boundary terms which may not have vanished after having used the boundary and/or initial conditions represented by Equation (81) and (91).

Replacing the expression obtained in Equation (92) into Equation (71) yields the following expression:

$$\begin{array}{l}{\left\{\delta R^{\left(1\right)}\left[j_1;U^{\left(2\right)}\left(2;t\right);A^{(2)}\left(2;j_1;t\right);f;\delta f\right]\right\}}_{{\theta }^0}=-{\left\{{\widehat{P}}^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f;\delta f\right)\right\}}_{{\theta }^0}\\ +{\left\{{〈A^{(2)}\left(2;j_1;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}+{\displaystyle \sum _{j_2=1}^{TF}{\left\{\frac{\partial R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f\right]}{\partial f_{j_2}}\right\}}_{{\theta }^0}}\delta f_{j_2}\\ \triangleq {\displaystyle \sum _{j_2=1}^{TF}{\left\{\frac{{\partial }^{2}R\left[h;f\left(\theta \right)\right]}{\partial f_{j_2}\partial f_{j_1}}\right\}}_{{\theta }^0}}\delta f_{j_2}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1=\hspace{0.17em}1,\dots ,TF\hspace{0.17em}.\end{array}$$

(93)

The expressions of the second-order sensitivities ${\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_2}\partial f_{j_1}$ of the response with respect to the components of the feature function are obtained by performing the following sequence of operations:

(i)

Use Equation (84) to recast the second term on the right-side of Equation (93) as follows:

$$\begin{array}{l}{\left\{{〈A^{(2)}\left(2;j_1;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}={\left\{{〈a^{(2)}\left(1;j_1;t\right),Q^{\left(1\right)}\left[h\left(t\right);f\left(\theta \right);t\right]\delta f〉}_1\right\}}_{{\theta }^0}\\ +{\left\{{〈a^{(2)}\left(2;j_1;t\right),Q^{\left(2\right)}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right)〉}_1\right\}}_{{\theta }^0}.\end{array}$$

(94)

(ii)

Recall that $Q^{\left(1\right)}\left[h\left(t\right);f\left(\theta \right);t\right]\delta f\triangleq {\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k\right\}}_{{\theta }^0}}$, where the quantities $q_{i,k}^{\left(1\right)}\left(h;f;t\right)$ were defined in Equation (15). Recall that $Q^{\left(2\right)}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right)\triangleq {\displaystyle \sum _{j_2=1}^{TF}{q}_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)\hspace{0.17em}}\delta f_{j_2}\left(\theta \right)$ where the quantities $q_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)$ were defined in Equation (76). Insert these expressions in Equation (94) to obtain the following relation:

$$\begin{array}{l}{\left\{{〈A^{(2)}\left(2;j_1;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}={\left\{{\displaystyle \sum _{j_2=1}^{TF}\left[{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)a_{1,i}^{\left(2\right)}\left(j_1;t\right)dt}}\right]}\hspace{0.17em}\delta f_{j_2}\right\}}_{{\theta }^0}\\ +{\left\{{\displaystyle \sum _{j_2=1}^{TF}\left[{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{q}_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)a_{2,i}^{\left(2\right)}\left(j_1;t\right)dt}}\right]}\hspace{0.17em}\delta f_{j_2}\right\}}_{{\theta }^0}.\end{array}$$

(95)

(iii)

Insert into Equation (93) the equivalent expression obtained in Equation (95), and subsequently identifying the quantities that multiply the variations $\delta f_{j_2}$, to obtain the following expression for the second-order sensitivities ${\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_2}\partial f_{j_1}$:

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R\left[h;f\left(\theta \right)\right]}{\partial f_{j_2}\partial f_{j_1}}}={\displaystyle \frac{\partial R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f\right]}{\partial f_{j_2}}}-{\displaystyle \frac{\partial {\widehat{P}}^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f;\delta f\right)}{\partial f_{j_2}}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)a_{1,i}^{\left(2\right)}\left(j_1;t\right)dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{q}_{i,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)a_{2,i}^{\left(2\right)}\left(j_1;t\right)dt}}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}{j}_1,j_2=\hspace{0.17em}1,\dots ,TF\hspace{0.17em}.\end{array}$$

(96)

It is important to note that the 2nd-LASS is independent of variations $\delta f$ and variations $V^{\left(2\right)}\left(2;x\right)$ in the respective state functions. It is also important to note that the ${\left(2\times TD\right)}^2$-dimensional matrix $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ is independent of the index $j_1$. Only the source-term $Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]$ depends on the index $j_1$. Therefore, the same solver can be used to invert the matrix $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);\hspace{0.17em}f\right]$ in order to solve numerically the 2nd-LASS for each $j_1$-dependent source $Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]$ in order to obtain the corresponding $j_1$-dependent $2\times TD$-dimensional 2nd-level adjoint function $A^{(2)}\left(2;j_1;t\right)\triangleq {\left[a^{(2)}\left(1;j_1;t\right),a^{(2)}\left(2;j_1;t\right)\right]}^{†}$. Computationally, it would be most efficient to store, if possible, the inverse matrix ${\left\{A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\right\}}^{-1}$, in order to multiply directly the inverse matrix ${\left\{A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\right\}}^{-1}$ with the corresponding source term $Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]$, for each index $j_1$, in order to obtain the corresponding $j_1$-dependent $2\times TD$-dimensional 2nd-level adjoint function $A^{(2)}\left(2;j_1;t\right)\triangleq {\left[a^{(2)}\left(1;j_1;t\right),a^{(2)}\left(2;j_1;t\right)\right]}^{†}$.

Since the adjoint matrix $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ is block-diagonal, solving the 2nd-LASS is equivalent to solving two 1st-LASS, with two different source terms. Thus, the “solvers” and the computer program used for solving the 1st-LASS can also be used for solving the 2nd-LASS. The 2nd-LASS was designated as the “second-level” rather than the “second-order” adjoint sensitivity system, since the 2nd-LASS does not involve any explicit 2nd-order G-derivatives of the operators underlying the original system but involves the inversion of the same operators that need to be inverted for solving the 1st-LASS.

If the 2nd-LASS is solved $TF$-times, the 2nd-order mixed sensitivities ${\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_2}\partial f_{j_1}$ will be computed twice, in two different ways, in terms of two distinct 2nd-level adjoint functions. Consequently, the symmetry property ${\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_2}\partial f_{j_1}={\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_1}\partial f_{j_2}$ provides an intrinsic (numerical) verification that the 1st-level adjoint function $a^{(1)}\left(x\right)$ and the components of the 2nd-level adjoint function $A^{(2)}\left(2;j_1;x\right)$ and are computed accurately.

The second-order sensitivities of the decoder-response with respect to the optimal weights/parameters ${\theta }_k,\hspace{0.17em}k=1,\dots ,TW$, are obtained analytically by using the chain rule in conjunction with the expressions obtained in Equations (46) and(96), as follows:

$${\displaystyle \frac{{\partial }^{2}R\left[h;f\left(\theta \right)\right]}{\partial {\theta }_k\partial {\theta }_j}}={\displaystyle \frac{\partial }{\partial {\theta }_k}}\left\{{\displaystyle \sum _{i=1}^{TF}\frac{\partial R\left[h;f\left(\theta \right)\right]}{\partial f_i\left(\theta \right)}\frac{\partial f_i\left(\theta \right)}{\partial {\theta }_j}}\right\}\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j,k=1,\dots ,TW.$$

(97)

4. Illustrative Application of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F Methodologies to a Heat Transfer Model

The application of the 1st-FASAM-NIDE-F Methodology will be illustrated in this Section by considering a model of linear steady-state heat conduction through a homogeneous slab of thickness $\mathcal{l}$, having a constant thermal conductivity denoted as $k$ and involving a distributed heat source that is proportional to the temperature distribution within the slab; the proportionality constant is denoted as $Q$. The slab is considered to be insulated on one side, which is held at a temperature $T_0$. The temperature distribution within the slab, denoted as $T\left(x\right)$, is thus modeled by the following linear heat conduction equation:

$${\displaystyle \frac{d^{2}T\left(x\right)}{d{x}^2}}+{\displaystyle \frac{Q}{k}}T\left(x\right)=0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<x<\mathcal{l};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}T\left(0\right)=T_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{dT\left(x\right)}{dx}}\right\}}_{x=0}=0.$$

(98)

Consider that the model response of interest, denoted as $R\left(T\right)$, is the average temperature within the slab, which is defined as follows:

$$R\left(T\right)\triangleq {\displaystyle \frac{1}{\mathcal{l}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)\hspace{0.17em}}dx.$$

(99)

The model’s primary parameters are $k,\hspace{0.17em}Q,\hspace{0.17em}{T}_0$, which can be subject to uncertainties, but their nominal/optimal values $k^0,\hspace{0.17em}{Q}^0,\hspace{0.17em}{T}_0^0$ are considered to be known. These parameters are considered to be components of the following (column) “vector of model parameters”:

$$\theta \triangleq {\left(k,\hspace{0.17em}Q,\hspace{0.17em}{T}_0\right)}^{†}.$$

(100)

The solution of Equation (98) has the following expression:

$$T\left(x\right)=T_0\cos x\gamma (\theta );\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\gamma (\theta )\triangleq \sqrt{Q/k}\hspace{0.17em}.$$

(101)

The quantity $\gamma (\theta )$ is a “feature function” of the primary model parameters. Using in Equation (99) the result obtained in Equation (101) yields the following closed form expression for the model response:

$$R\left(T\right)\triangleq {\displaystyle \frac{T_0}{\mathcal{l}\gamma (\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(102)

At the nominal parameter values, the nominal value of the temperature distribution and of the average temperature response, respectively, have the following expressions:

$$T^0\left(x\right)=T_0^0\cos x{\gamma }^0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\gamma }^0\triangleq \gamma ({\theta }^0);$$

(103)

$$R^0\left(T^0\right)\triangleq {\displaystyle \frac{T_0^0}{\mathcal{l}{\gamma }^0}}\sin \mathcal{l}{\gamma }^0\hspace{0.17em}.$$

(104)

4.1. Applying the 1st-FASAM-NIDE-F Methodology to Obtain the First-Order Response Sensitivities to the Primary Model Parameters

The heat conduction equation presented in Equation (98) can be recast into the following equivalent NIDE-F form:

$${\displaystyle \frac{dT\left(x\right)}{dx}}+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}\hspace{0.17em}dx=T_0\gamma (\theta )\left[\sin \mathcal{l}\gamma (\theta )-\sin x\gamma (\theta )\right]\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}T\left(0\right)=T_0\hspace{0.17em}.$$

(105)

The first-order sensitivities of the response $R\left(T\right)$ will be determined from the first-order Gateaux- (G-) differential, denoted as $\delta R$, of $R\left(T\right)$, which is obtained by applying the definition of the G-differential to Equation (99), as follows:

$$\delta R\triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \frac{1}{\mathcal{l}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[T\left(x\right)+\epsilon \delta T\left(x\right)\right]\hspace{0.17em}}dx\right\}}_{\epsilon =0}={\displaystyle \frac{1}{\mathcal{l}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\hspace{0.17em}dx\hspace{0.17em}}.$$

(106)

The variation $\delta T\left(x\right)$ is the solution of the 1st-Level Variational Sensitivity System (1st-LVSS) obtained by G-differentiating Equation (105), which yields the following NIDE-F for arbitrary variations $\delta T\left(x\right)$ and $\delta \gamma (\theta )$ around the nominal values $T^0\left(x\right),\hspace{0.17em}{\theta }^0$:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{d\epsilon }}\left[{\displaystyle \frac{d\left(T^0+\epsilon \delta T\right)}{dx}}+{\left({\gamma }^0+\epsilon \delta \gamma \right)}^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(T^0+\epsilon \delta T\right)}\hspace{0.17em}dx\right]\right\}}_{\epsilon =0}\\ ={\left\{{\displaystyle \frac{d}{d\epsilon }}\left(T_0^0+\epsilon \delta T_0\right)\left({\gamma }^0+\epsilon \delta \gamma \right)\left[\sin \mathcal{l}\left({\gamma }^0+\epsilon \delta \gamma \right)-\sin x\left({\gamma }^0+\epsilon \delta \gamma \right)\right]\right\}}_{\epsilon =0};\\ {\left\{{\displaystyle \frac{d}{d\epsilon }}{\left[T^0\left(x\right)+\epsilon \delta T\left(x\right)\right]}_{x=0}\right\}}_{\epsilon =0}=\delta T_0\hspace{0.17em}.\end{array}$$

(107)

Performing the operations indicated in Equation (107) yields the following form for the 1st-LVSS:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}dx\right\}}_{\theta ={\theta }^0}=q^{\left(1\right)}\left(x\right),\\ q^{\left(1\right)}\left(x\right)\triangleq {\left\{\left[\left(\delta T_0\right)\gamma (\theta )+\delta \gamma (\theta )T_0\right]\left[\sin \mathcal{l}\gamma (\theta )-\sin x\gamma (\theta )\right]\right\}}_{\theta ={\theta }^0}\\ +\delta \gamma (\theta ){\left\{T_0\gamma (\theta )\left[\mathcal{l}\cos \mathcal{l}\gamma (\theta )-x\cos x\gamma (\theta )\right]\right\}}_{\theta ={\theta }^0}-{\left\{2\delta \gamma (\theta )\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}\hspace{0.17em}dx\right\}}_{\theta ={\theta }^0};\end{array}$$

(108)

$${\left\{\delta T\left(x\right)\right\}}_{x=0}=\delta T_0\hspace{0.17em}.$$

(109)

For subsequent reference, it is noted that the solution of the above 1st-LVSS has the following expression:

$$\delta T\left(x\right)=\delta T_0\cos x\gamma (\theta )-\delta \gamma (\theta )T_{0}x\sin x\gamma (\theta )\hspace{0.17em}.$$

(110)

The 1st-LVSS would need to be solved repeatedly, using every possible parameter variation, in order to determine the corresponding value of the temperature variation $\delta T\left(x\right)$. These repeated computations can be avoided by eliminating the appearance of the variation $\delta T\left(x\right)$ in Equation (106); this aim can be achieved by deriving an alternative expression for the response variation $\delta R$ that would not involve the variation $\delta T\left(x\right)$. This alternative expression for $\delta R$ will be constructed in terms of the first-level adjoint function, which is, in turn, obtained as the solution of the 1st-Level Adjoint Sensitivity System (1st-LASS) to be constructed next by using the inner product defined in Equation (10) for the single-component function $\delta T\left(x\right)$. Forming the inner product of Equation (109) with a yet undefined function $a^{\left(1\right)}\left(x\right)$ yields the following relation:

$${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \frac{d}{dx}}\delta T\left(x\right)+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}dx={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)q^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx.$$

(111)

The relation obtained in Equation (111) is satisfied at the nominal/optimal parameter values but this fact has not been explicitly indicated in order to simplify the notation. Integrating by parts the first term on the left-side of Equation (111) and rearranging the second term on the left-side of Equation (111) yields the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \frac{d}{dx}}\delta T\left(x\right)+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}dx=a^{\left(1\right)}\left(\mathcal{l}\right)\delta T\left(\mathcal{l}\right)\\ -a^{\left(1\right)}\left(0\right)\delta T\left(0\right)+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}dx\left\{-{\displaystyle \frac{d{a}^{\left(1\right)}\left(x\right)}{dx}}+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\right\}.\end{array}$$

(112)

The function $a^{\left(1\right)}\left(x\right)$ will now be determined as follows: (i) require that the last term on the right-side of Equation (112) be identical to the G-differential $\delta R$ defined in Equation (106); and (ii) eliminate the unknown quantity $\delta T\left(\mathcal{l}\right)$ in Equation (112). These requirements lead to the following NIDE-F for the function $a^{\left(1\right)}\left(x\right)$:

$$-{\displaystyle \frac{d{a}^{\left(1\right)}\left(x\right)}{dx}}+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx={\displaystyle \frac{1}{\mathcal{l}}};$$

(113)

$$a^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em}.$$

(114)

The NIDE-F-net represented by Equations (113) and (114) constitutes the 1st-Level Adjoint Sensitivity System (1st-LASS) for the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(x\right)$. The 1st-LASS is satisfied at the nominal parameter values but this fact has not been explicitly indicated in order to simplify the notation.

Using Equations (111)‒(114) in conjunction with Equation (106) yields the following alternative expression for the G-differential $\delta R$ in terms of $a^{\left(1\right)}\left(x\right)$:

$$\delta R={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)q^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx+a^{\left(1\right)}\left(0\right)\delta T_0.$$

(115)

Using in Equation (115) the expression provided for $q^{\left(1\right)}\left(x\right)$ in Equation (108) and identifying the expressions that multiply the variations $\delta T_0$ and $\delta \gamma (\theta )$ yields the following expressions for the first-order sensitivities of the response with respect to the initial condition $T_0$ and the feature function $\gamma (\theta )$, respectively:

$${\displaystyle \frac{\partial R}{\partial T_0}}=\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left[\sin \mathcal{l}\gamma (\theta )-\sin x\gamma (\theta )\right]}\hspace{0.17em}dx+a^{\left(1\right)}\left(0\right);$$

(116)

$$\begin{array}{l}{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}=T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left[\sin \mathcal{l}\gamma (\theta )-\sin x\gamma (\theta )\right]}\hspace{0.17em}dx\\ +T_0\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left[\mathcal{l}\cos \mathcal{l}\gamma (\theta )-x\cos x\gamma (\theta )\right]}\hspace{0.17em}dx-2\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}\hspace{0.17em}dx.\end{array}$$

(117)

The expressions obtained in Equations (116) and (117) can be evaluated after having determined the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(x\right)$. Also, these expressions are to be evaluated using the nominal/optimal parameter values, but this fact has not been explicitly indicated in order to simplify the notation. Notably, the 1st-LASS is independent of parameter variations, so it needs to be solved only once to determine $a^{\left(1\right)}\left(x\right)$. The closed-form explicit expression of the solution of the 1st-LASS represented by Equations (113) and (114) is provided below:

$$a^{\left(1\right)}\left(x\right)=-{\displaystyle \frac{2\left(x-\mathcal{l}\right)}{\mathcal{l}\left[2+{\mathcal{l}}^2{\gamma }^2(\theta )\right]}}.$$

(118)

Using the expression obtained in Equation (118) into Equations (116) and (117), respectively, and performing the respective integrations yields the following closed-form expressions:

$${\displaystyle \frac{\partial R}{\partial T_0}}={\displaystyle \frac{\sin \mathcal{l}\gamma (\theta )}{\mathcal{l}\gamma (\theta )}};$$

(119)

$${\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}={\displaystyle \frac{T_0}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{T_0}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(120)

As expected, the expressions obtained in Equations (119) and (120) coincide with the expressions that would be obtained by the direct differentiation of the expression for the model response $R\left(T\right)$ obtained in Equation (102) with respect to $T_0$ and $\gamma (\theta )$, respectively. Of course, the closed-form exact expression for the model response in terms of the model’s primary parameters and/or feature functions is not available in practice.

The sensitivities of the model response with respect to the primary parameters are obtained by using the result obtained in Equation (120) in conjunction with the following “chain-rule of differentiation”:

$${\displaystyle \frac{\partial R}{\partial Q}}={\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}{\displaystyle \frac{\partial \gamma (\theta )}{\partial Q}}={\displaystyle \frac{1}{2\sqrt{kQ}}}{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}};$$

(121)

$${\displaystyle \frac{\partial R}{\partial k}}={\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}{\displaystyle \frac{\partial \gamma (\theta )}{\partial k}}=-{\displaystyle \frac{1}{2k}}\sqrt{{\displaystyle \frac{Q}{k}}}{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\hspace{0.17em}.$$

(122)

4.2. Applying the 1st-FASAM-NODE Methodology to Obtain the First-Order Response Sensitivities to the Primary Model Parameters

The traditional form of the heat conduction model provided in Equation (98) is a neural ordinary differential equation (NODE) which can be analyzed directly by using the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (1st-FASAM-NODE) introduced by Cacuci [25]. The G-differential of Equation (98) yields the following 1st-LVSS in NODE-form satisfied by the temperature variation $\delta T\left(x\right)$:

$${\left\{{\displaystyle \frac{d^2}{d{x}^2}}\delta T\left(x\right)+{\gamma }^2(\theta )\delta T\left(x\right)\right\}}_{\theta ={\theta }^0}=-{\left\{2\delta \gamma (\theta )\gamma (\theta )T\left(x\right)\right\}}_{\theta ={\theta }^0};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<x<\mathcal{l};$$

(123)

$$\delta T\left(0\right)=\delta T_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=0}=0.$$

(124)

The 1st-LASS corresponding to the above 1st-LVSS is obtained by implementing the same steps as outlined in the previous Subsection, by constructing the inner-product of a yet undetermined function $b^{\left(1\right)}\left(x\right)$ with Equation (123) to obtain the following relation:

$${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\left[{\displaystyle \frac{d^2}{d{x}^2}}\delta T\left(x\right)+{\gamma }^2(\theta )\delta T\left(x\right)\right]=-2\delta \gamma (\theta )\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}T\left(x\right)\hspace{0.17em}dx.$$

(125)

The relation obtained in Equation (125) is to be evaluated at the nominal/optimal parameter values but this fact has not been explicitly indicated in order to simplify the notation.

Integrating by parts the first term on the left-side of Equation (125) yields the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\left[{\displaystyle \frac{d^2}{d{x}^2}}\delta T\left(x\right)+{\gamma }^2(\theta )\delta T\left(x\right)\right]=b^{\left(1\right)}\left(\mathcal{l}\right){\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}-b^{\left(1\right)}\left(0\right){\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=0}\\ -{\left\{\delta T\left(x\right){\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=\mathcal{l}}+{\left\{\delta T\left(x\right){\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0}+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}\left[{\displaystyle \frac{d^2{b}^{\left(1\right)}\left(x\right)}{d{x}^2}}+{\gamma }^2(\theta )b^{\left(1\right)}\left(x\right)\right]dx.\end{array}$$

(126)

Identifying the last term on the right-side of Equation (126) with the G-differential $\delta R$ provided in Equation (106), using the conditions provided in Equation (124) and eliminating the unknown boundary values on the right-side of Equation (126) yields the following expression for the G-differential in terms of the function $b^{\left(1\right)}\left(x\right)$:

$$\delta R=-2\delta \gamma (\theta )\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}T\left(x\right)\hspace{0.17em}dx-\delta T_0{\left\{{\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0},$$

(127)

where the 1st-level adjoint sensitivity function $b^{\left(1\right)}\left(x\right)$ is the solution of the following 1st-Level Adjoint Sensitivity System (1st-LASS):

$${\displaystyle \frac{d^2{b}^{\left(1\right)}\left(x\right)}{d{x}^2}}+{\gamma }^2(\theta )b^{\left(1\right)}\left(x\right)={\displaystyle \frac{1}{\mathcal{l}}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<x<\mathcal{l};$$

(128)

$$b^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=\mathcal{l}}=0.$$

(129)

Identifying the quantities that multiply the variations $\delta T_0$ and $\delta \gamma (\theta )$ in Equation (127) yields the following expressions for the sensitivities of the model response with respect to $T_0$ and $\gamma (\theta )$:

$${\displaystyle \frac{\partial R}{\partial T_0}}=-{\left\{{\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}{\delta }^{\prime }\left(x\right)\hspace{0.17em}dx;$$

(130)

$${\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}=-2\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}T\left(x\right)\hspace{0.17em}dx.$$

(131)

The 1st-LASS can be readily solved to obtain the following expression for the 1st-level adjoint sensitivity function $b^{\left(1\right)}\left(x\right)$:

$$b^{\left(1\right)}\left(x\right)={\displaystyle \frac{1-\cos \left(x-\mathcal{l}\right)\gamma (\theta )}{\mathcal{l}{\gamma }^2(\theta )}}.$$

(132)

Using in Equations (130) and (131) the expression for $b^{\left(1\right)}\left(x\right)$obtained above yields the following expressions:

$${\displaystyle \frac{\partial R}{\partial T_0}}={\displaystyle \frac{\sin \mathcal{l}\gamma (\theta )}{\mathcal{l}\gamma (\theta )}};$$

(133)

$${\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}={\displaystyle \frac{T_0}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{T_0}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta ).$$

(134)

All of the results obtained in Equations (125)‒(134) are to be evaluated at the nominal parameter values but this fact has not been explicitly indicated in order to simplify the notation.

4.3. Comparison: Applying the 1st-FASAM-NODE Methodology Versus Applying the 1st-FASAM-NIDE-F Methodology

In cases where the model can be equivalently expressed in either NODE or in NIE-F forms, such as shown in Equation (98) or Equation (105), respectively, it is important to highlight the similarities and differences between applying the 1st-FASAM-NODE methodology versus applying the 1st-FASAM-NIDE-F methodology for determining the first-order response sensitivities to the underlying model parameters. Evidently, the final results obtained in Equations (133) and (134) by treating the heat conduction model as a NODE, cf. Equations (98), are identical with the corresponding results obtained in Equations (119) and (120) by having treated the heat conduction model as a NIDE-F, cf. Equation (105). Furthermore, even though the form of the 1st-LVSS produced by the NODE methodology, namely Equations (123) and (124), differs from the form of the 1st-LVSS produced by the NIDE-F methodology, namely Equations (108) and (109), the solutions to these 1st-LVSS are identical to each other, having the expression provided in Equation (110)

However, the 1st-LASS corresponding to the NODE heat conduction model differs from the 1st-LASS corresponding to the NIDE-F heat conduction model, so that the corresponding 1st-level adjoint sensitivity function $b^{\left(1\right)}\left(x\right)$ for the NODE-model, namely Equation (132), differs from the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(x\right)$ for the NIDE-F heat conduction model, which is provided in Equation (118). Consequently, the expressions obtained in terms of the respective 1st-level adjoint sensitivity functions of the sensitivities of the model response with respect to the primary parameters and feature function for the NODE-representation, namely Equations (130) and (131), differ from those obtained for the NIDE-F representation, namely Equations (116) and (117). The structure of the 1st-LASS and expressions for sensitivities appear to be simpler in the NODE-representation than in the NIDE-F representation, but the choice of representation/framework will be largely influenced by the neural-net software available to the individual user.

4.4. Illustrative Application of the 2nd-FASAM-NIDE Methodology Versus the 2nd-FASAM-NODE Methodology for Computing the Second-Order Response Sensitivities to Model Features and Parameters

The general principles underlying the 2nd-FASAM-NIDE-F methodology presented in Section 3 will be applied to the paradigm heat conduction model considered in this Section in order to highlight the salient issues arising when applying this methodology to determine the second-order sensitivities of model responses to model features and parameters.

4.4.1. Application of the 2nd-FASAM-NIDE-F methodology

When applying the 2nd-FASAM-NIDE-F, the second-order sensitivities arise from the first-order sensitivities obtained in Equations (116) and (117). Thus, the second-order sensitivities arising from Equation (116) are provided by its G-differential for arbitrary variations around the nominal parameter and function values (indicated by the use of the superscript “zero”). Using in Equation (116) the result obtained in Equation (118) and applying the definition of the G-differential to the resulting expression yields the relation below:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial T_0}}\right)\triangleq \delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{ind}={\left\{{\displaystyle \frac{d}{d\epsilon }}2{\left[2+{\mathcal{l}}^2{\left({\gamma }^0+\epsilon \delta \gamma \right)}^2\right]}^{-1}\right\}}_{\epsilon =0}\\ +{\left\{{\displaystyle \frac{d}{d\epsilon }}\left({\gamma }^0+\epsilon \delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]\left[\sin \mathcal{l}\left({\gamma }^0+\epsilon \delta \gamma \right)-\sin x\left({\gamma }^0+\epsilon \delta \gamma \right)\right]}\hspace{0.17em}dx\right\}}_{\epsilon =0};\end{array}$$

(135)

where the expressions for the above direct-effect and, respectively, indirect-effect terms are obtained as shown below:

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{dir}=-{\left\{4\gamma {\mathcal{l}}^2\left(\delta \gamma \right){\left(2+{\mathcal{l}}^2{\gamma }^2\right)}^{-2}\right\}}_{{\theta }^0}\\ +\left(\delta \gamma \right){\left\{{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma +\gamma \mathcal{l}\cos \mathcal{l}\gamma -x\gamma \sin x\gamma \right)a^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\right\}}_{{\theta }^0};\end{array}$$

(136)

$$\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{ind}={\left\{\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)\delta a^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\right\}}_{{\theta }^0}$$

(137)

The direct-effect term can be evaluated immediately. The indirect-effect term depends on the variational function $\delta a^{\left(1\right)}\left(x\right)$, which is the solution of the G-differentiated 1st-LASS, comprising Equations (113) and (114), obtained by definition as follows:

$${\left\{-{\displaystyle \frac{d}{dx}}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]\right\}}_{\epsilon =0}+{\left\{{\left({\gamma }^0+\epsilon \delta \gamma \right)}^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}dx\right\}}_{\epsilon =0}=0;$$

(138)

$$\delta a^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em}.$$

(139)

Performing the operations indicated in Equation (138) yields the following NIDE-F, to be evaluated at the nominal parameter values:

$$-{\displaystyle \frac{d}{dx}}\delta a^{\left(1\right)}\left(x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\hspace{0.17em}.$$

(140)

Since the indirect-effect term only depends on the variational function $\delta a^{\left(1\right)}\left(x\right)$ but does not depend on the variational function $\delta T\left(x\right)$, the relations presented in Equations (139) and (140) constitute the 2nd-LVSS for the function $\delta a^{\left(1\right)}\left(x\right)$, which is dependent on parameter variations and would need to be solved anew for each parameter variation of interest. The need for computing $\delta a^{\left(1\right)}\left(x\right)$ can be avoided by expressing the indirect-effect term defined by Equation (137) in terms of a 2nd-level adjoint sensitivity function that is independent of parameter variations. This adjoint function will be denoted as $a^{\left(2\right)}\left(1;x\right)$, where the argument “1” indicates that this adjoint function corresponds to the first-order sensitivity $\partial R/\partial T_0$, which was chosen in this case to be the “first” 1st-order sensitivity to be considered. The 2nd-LASS to be satisfied by $a^{\left(2\right)}\left(1;x\right)$ will be constructed by applying the 2nd-FASAM-NIDE-F, which commences by forming the inner product of $a^{\left(2\right)}\left(1;x\right)$ with Equation (140), to obtain the following relation:

$$\begin{array}{l}-{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)\left[\frac{d}{dx}\delta a^{\left(1\right)}\left(x\right)\right]d}x+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\\ =-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)\hspace{0.17em}dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\hspace{0.17em}.\end{array}$$

(141)

Integrating by parts the first term on the left-side of Equation (141) and reversing the order of integrations in the remaining terms yields the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(x\right)\left[\frac{d}{dx}{a}^{\left(2\right)}\left(1;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)dx}\right]d}x-a^{\left(2\right)}\left(1;\mathcal{l}\right)\delta a^{\left(1\right)}\left(\mathcal{l}\right)\\ +a^{\left(2\right)}\left(1;0\right)\delta a^{\left(1\right)}\left(0\right)=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)\hspace{0.17em}dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\hspace{0.17em}.\end{array}$$

(142)

The first term on the left-side of Equation (142) is now required to represent the indirect-effect term defined in Equation (137) to obtain the relation below:

$${\displaystyle \frac{d}{dx}}{a}^{\left(2\right)}\left(1;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)dx}=\gamma \left(\sin \mathcal{l}\gamma -\sin x\gamma \right).$$

(143)

The unknown quantity $\delta a^{\left(1\right)}\left(0\right)$ is eliminated from Equation (142) by imposing the following condition:

$$a^{\left(2\right)}\left(1;0\right)=0.$$

(144)

Replacing the results obtained in Equations (139), (143) and (144) into Equation (142) yields the following alternative expression for the indirect-effect term:

$$\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{ind}=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)\hspace{0.17em}dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\hspace{0.17em},$$

(145)

where the 2nd-level adjoint sensitivity function $a^{\left(2\right)}\left(1;x\right)$ is the solution of the 2nd-Level Adjoint Sensitivity System (2nd-LASS) comprising Equations (143) and (144). The 2nd-LASS is a NIDE-F net that does not depend on parameter variations and needs to be solved once only at the nominal parameter values; its solution, $a^{\left(2\right)}\left(1;x\right)$, is used in Equation (145).

Adding the expressions obtained in Equations (145) and (136) yields the following expression:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial T_0}}\right)=-4\gamma {\mathcal{l}}^2\left(\delta \gamma \right){\left(2+{\mathcal{l}}^2{\gamma }^2\right)}^{-2}\\ +\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left[\sin \mathcal{l}\gamma -\sin x\gamma +\gamma \mathcal{l}\cos \mathcal{l}\gamma -x\gamma \sin x\gamma \right]}\hspace{0.17em}dx\\ -2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)\hspace{0.17em}dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\hspace{0.17em}\triangleq {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}\delta T_0+{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}\delta \gamma .\end{array}$$

(146)

It follows from Equation (146) that:

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}=0;\\ {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma +\gamma \mathcal{l}\cos \mathcal{l}\gamma -x\gamma \sin x\gamma \right)a^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\\ -2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)\hspace{0.17em}dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\hspace{0.17em}-4\gamma {\mathcal{l}}^2{\left(2+{\mathcal{l}}^2{\gamma }^2\right)}^{-2}.\end{array}$$

(147)

The 2nd-LASS represented by Equations can be solved to obtain the following closed-form expression, to be evaluated at the nominal parameter values, for its solution:

$$a^{\left(2\right)}\left(1;x\right)={\displaystyle \frac{{\gamma }^2\mathcal{l}}{1+{\gamma }^2{\mathcal{l}}^2/2}}x+\cos \gamma x-1.$$

(148)

Inserting the above expression for $a^{\left(2\right)}\left(1;x\right)$ into Equation (147) and performing the respective integrations yields the following closed-form expression for the mixed second-order sensitivity:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}={\displaystyle \frac{1}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{1}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(149)

The validity of the above expression can be readily verified by taking the appropriate derivative of either of the first-order sensitivities provided in Equations (133) and (134).

The second-order sensitivities arising from Equation (117) are provided by its G-differential for arbitrary variations around the nominal parameter and function values (indicated by the use of the superscript “zero”), which is by definition obtained as follows:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)\triangleq \delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{ind}\triangleq \left\{{\displaystyle \frac{d}{d\epsilon }}\left(T_0^0+\epsilon \delta T_0\right)\right.{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}\\ {\left.\times \left[\sin \mathcal{l}\left({\gamma }^0+\epsilon \delta \gamma \right)-\sin x\left({\gamma }^0+\epsilon \delta \gamma \right)\right]dx\right\}}_{\epsilon =0}+\left\{{\displaystyle \frac{d}{d\epsilon }}\left(T_0^0+\epsilon \delta T_0\right)\right.\left({\gamma }^0+\epsilon \delta \gamma \right)\\ {\left.\times {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]\left[\mathcal{l}\cos \mathcal{l}\left({\gamma }^0+\epsilon \delta \gamma \right)-x\cos x\left({\gamma }^0+\epsilon \delta \gamma \right)\right]}\hspace{0.17em}dx\right\}}_{\epsilon =0}\\ -2{\left\{{\displaystyle \frac{d}{d\epsilon }}\left({\gamma }^0+\epsilon \delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[T^0\left(x\right)+\epsilon \delta T\left(x\right)\right]}\hspace{0.17em}dx\right\}}_{\epsilon =0},\end{array}$$

(150)

where the direct-effect and, respectively, indirect-effect terms have the following expressions:

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{dir}=\left(\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}{\gamma }^0-\sin x{\gamma }^0\right)a^{\left(1,0\right)}\left(x\right)\hspace{0.17em}}dx\\ +\left(\delta \gamma \right)T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}{\gamma }^0-x\cos x{\gamma }^0\right)a^{\left(1,0\right)}\left(x\right)\hspace{0.17em}}dx\\ +\left(T_0^0\delta \gamma +{\gamma }^0\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}{\gamma }^0-x\cos x{\gamma }^0\right)a^{\left(1,0\right)}\left(x\right)}\hspace{0.17em}dx\\ +T_0^0{\gamma }^0\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(x^2\sin x{\gamma }^0-{\mathcal{l}}^2\sin \mathcal{l}{\gamma }^0\right)}\hspace{0.17em}{a}^{\left(1,0\right)}\left(x\right)\hspace{0.17em}dx-2\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1,0\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{T}^0\left(x\right)}\hspace{0.17em}dx;\end{array}$$

(151)

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{ind}=-2{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1,0\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{T}^0\left(x\right)}\hspace{0.17em}dx-2{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1,0\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}dx\hspace{0.17em}\\ +T_0^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}{\gamma }^0-\sin x{\gamma }^0\right)\delta a^{\left(1,0\right)}\left(x\right)}dx+T_0^0{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}{\gamma }^0-x\cos x{\gamma }^0\right)\delta a^{\left(1,0\right)}\left(x\right)}\hspace{0.17em}dx.\end{array}$$

(152)

The variational function $\delta T\left(x\right)$ is the solution of Equations (108) and (109) while the variational function $\delta a^{\left(1\right)}\left(x\right)$ is the solution of Equations (139) and (140). Altogether, these four equations constitute the 2nd-LVSS for the two-component vector-valued variational function $V^{\left(2\right)}\left(2;x\right)\hspace{0.17em}\triangleq {\left[\delta T\left(x\right),\delta a^{\left(1\right)}\left(x\right)\right]}^{†}$. The need for repeatedly solving this 2nd-LVSS for all parameter variations of interest is circumvented by eliminating the appearance of $V^{\left(2\right)}\left(2;x\right)\hspace{0.17em}\triangleq {\left[\delta T\left(x\right),\delta a^{\left(1\right)}\left(x\right)\right]}^{†}$ in the expression of the indirect-effect term defined in Equation (152), by constructing an alternative expression for this term using the solution of the 2nd-LASS, to be constructed by applying the steps outlined in Section 3, as follows:

• Consider the two-component vector function $A^{(2)}\left(2;2;x\right)\triangleq {\left[a^{(2)}\left(1;2;x\right),a^{(2)}\left(2;2;x\right)\right]}^{†}$, where the first argument denotes the component number and the second argument (“2”) indicates that this function will correspond to the “second” first-order sensitivity $\partial R/\partial \gamma (\theta )$. Using the inner product defined in Equation (85), construct the inner product of $A^{(2)}\left(2;2;x\right)\triangleq {\left[a^{(2)}\left(1;2;x\right),a^{(2)}\left(2;2;x\right)\right]}^{†}$ with Equations (108) and (140), respectively, to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)\frac{d}{dx}\delta T\left(x\right)}\hspace{0.17em}dx+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}dx\\ -{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)\frac{d}{dx}\delta a^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\\ ={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)q^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\hspace{0.17em}.\end{array}$$

  (153)
• Integrate by parts the first and third terms on the left-side of Equation (153) and rearrange the terms to obtain the following relation:

  $$\begin{array}{l}{a}^{(2)}\left(1;2;\mathcal{l}\right)\delta T\left(\mathcal{l}\right)-a^{(2)}\left(1;2;0\right)\delta T\left(0\right)-a^{(2)}\left(2;2;\mathcal{l}\right)\delta a^{\left(1\right)}\left(\mathcal{l}\right)\\ +a^{(2)}\left(2;2;0\right)\delta a^{\left(1\right)}\left(0\right)+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\left[-\frac{d}{dx}{a}^{(2)}\left(1;2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)}\right]}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(x\right)\left[\frac{d}{dx}{a}^{(2)}\left(2;2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}\hspace{0.17em}dx\right]}\hspace{0.17em}dx\\ ={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)q^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\hspace{0.17em}.\end{array}$$

  (154)
• Require the third and fourth terms on the left-side of Equation (154) to represent the indirect-effect term defined in Equation (152) by imposing the following relations:

  $$-{\displaystyle \frac{d}{dx}}{a}^{(2)}\left(1;2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)}=-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\hspace{0.17em};$$

  (155)

  $$\begin{array}{l}{\displaystyle \frac{d}{dx}}{a}^{(2)}\left(2;2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}\hspace{0.17em}dx=-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}\hspace{0.17em}dx\\ +T_0\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)+T_0\gamma \left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right).\end{array}$$

  (156)
• Eliminate the unknown terms $\delta T\left(\mathcal{l}\right)$ on the left-side of Equation (154) by imposing the following boundary conditions:

  $$a^{(2)}\left(1;2;\mathcal{l}\right)=0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{a}^{(2)}\left(2;2;0\right)=0.$$

  (157)
• Insert the boundary conditions represented by Equations (109) and (139) into Equation (154) and use the relations underlying the 2nd-LASS to obtain the following expression for the indirect-effect term defined in Equation (152):

  $$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{ind}=a^{(2)}\left(1;2;0\right)\delta T_0+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)q^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\hspace{0.17em}.\end{array}$$

  (158)

Add the expression obtained in Equation (158) to the expression of the direct-effect term provided in Equation (151) to obtain the following expression:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)\triangleq {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}\delta T_0+{\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \partial \gamma }}\delta \gamma =a^{(2)}\left(1;2;0\right)\delta T_0+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)q^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\\ -2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}+\left(\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)a^{\left(1\right)}\left(x\right)\hspace{0.17em}}dx\\ +\left(\delta \gamma \right)T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)\hspace{0.17em}}dx\\ +\left(T_0\delta \gamma +\gamma \delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\\ +T_0\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(x^2\sin x\gamma -{\mathcal{l}}^2\sin \mathcal{l}\gamma \right)}\hspace{0.17em}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx-2\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}\hspace{0.17em}dx\hspace{0.17em}.\end{array}$$

(159)

Insert the expression of $q^{\left(1\right)}\left(x\right)$ into the second term on the right-side of Equation (159) and collect the terms multiplying the variations $\delta T_0$ and $\delta \gamma$, respectively, to obtain the following expressions:

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}=a^{(2)}\left(1;2;0\right)+\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)a^{(2)}\left(1;2;x\right)}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)a^{\left(1\right)}\left(x\right)\hspace{0.17em}}dx+\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx;\end{array}$$

(160)

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \partial \gamma }}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)\left[T_0\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)+T_0\gamma \left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}\hspace{0.17em}dx\right]}\hspace{0.17em}dx\\ -2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}+T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)\hspace{0.17em}}dx\\ -2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}\hspace{0.17em}dx\hspace{0.17em}+T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)}\hspace{0.17em}dx\\ +T_0\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(x^2\sin x\gamma -{\mathcal{l}}^2\sin \mathcal{l}\gamma \right)}\hspace{0.17em}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx.\end{array}$$

(161)

The algebraic manipulations involved in obtaining the closed-form expressions of the second-order sensitivities presented in Equations (160) and (161) are straightforward but involve a large amount of algebra stemming from the fact that the 2nd-LASS involves the two-component 2nd-level adjoint sensitivity function $A^{(2)}\left(2;2;x\right)\triangleq {\left[a^{(2)}\left(1;2;x\right),a^{(2)}\left(2;2;x\right)\right]}^{†}$. The reason for needing such a two-component adjoint function stems from the expression of the first-order sensitivity $\partial R/\partial \gamma (\theta )$ provided in Equation (117), which involves both the original function $T\left(x\right)$ and the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(x\right)$. A significant amount of algebraic manipulations could be avoided by eliminating the appearance of either $T\left(x\right)$ or $a^{\left(1\right)}\left(x\right)$ in the expression of $\partial R/\partial \gamma (\theta )$. If either of these functions were eliminated from appearing in the expression of $\partial R/\partial \gamma (\theta )$, then the G-differential of $\partial R/\partial \gamma (\theta )$ would depend either just on $\delta a^{\left(1\right)}$ or just on $\delta T$, which are “single-component” (as opposed to a “two-components”) variational sensitivity functions. In such a case, the corresponding 2nd-LASS would also comprise just a single-component (as opposed to a “two-component) 2nd-level adjoint sensitivity function. These considerations will be illustrated in the following by using Equation (101) to eliminate the appearance of the function $T\left(x\right)$ in the expression provided in Equation (117) for $\partial R/\partial \gamma (\theta )$, which would consequently take on the following simplified expression:

$${\displaystyle \frac{\partial R}{\partial \gamma }}=T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)\hspace{0.17em}}\hspace{0.17em}dx.$$

(162)

Applying the definition of the G-differential to Equation (162) yields the following expression:

$$\delta \left\{{\displaystyle \frac{\partial R}{\partial \gamma }}\right\}=\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{ind},$$

(163)

where the direct-effect and the indirect-effect terms are defined below:

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{dir}=\left(\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)\hspace{0.17em}}dx\\ +\left(\delta \gamma \right)T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(-{\mathcal{l}}^2\gamma \sin \mathcal{l}\gamma +\mathcal{l}\cos \mathcal{l}\gamma +x^2\gamma \cos x\gamma -x\cos x\gamma -\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)\hspace{0.17em}}dx\end{array}$$

(164)

$$\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{ind}=T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)\delta a^{\left(1\right)}\left(x\right)\hspace{0.17em}}dx\hspace{0.17em}.$$

(165)

The appearance in Equation (165) of the variational function $\delta a^{\left(1\right)}\left(x\right)\hspace{0.17em}$is eliminated by following the same procedure as followed in the foregoing for the indirect-effect term $\delta {\left(\partial R/\partial T_0\right)}_{ind}$. Ultimately, the indirect-effect term $\delta {\left(\partial R/\partial \gamma \right)}_{ind}$ will have the following expression in terms of a 2nd-level adjoint sensitivity function denoted as $a^{\left(2\right)}\left(2;x\right)$:

$$\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{ind}=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(2;x\right)\hspace{0.17em}dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\hspace{0.17em},$$

(166)

where the 2nd-level adjoint sensitivity function $a^{\left(2\right)}\left(2;x\right)$ is the solution of the following 2nd-LASS:

$${\displaystyle \frac{d}{dx}}{a}^{\left(2\right)}\left(2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(2;x\right)dx}=T_0\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right);$$

(167)

$$a^{\left(2\right)}\left(2;0\right)=0$$

(168)

Adding the expressions obtained in Equations (164) and (166) yields the following expression for the G-differential $\delta \left\{\partial R/\partial \gamma \right\}$:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)=\left(\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)\hspace{0.17em}}dx\\ +\left(\delta \gamma \right)T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(-{\mathcal{l}}^2\gamma \sin \mathcal{l}\gamma +\mathcal{l}\cos \mathcal{l}\gamma +x^2\gamma \cos x\gamma -x\cos x\gamma -\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)\hspace{0.17em}}dx\\ -2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(2;x\right)\hspace{0.17em}dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\hspace{0.17em}\triangleq {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}\delta T_0+{\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \partial \gamma }}\delta \gamma .\end{array}$$

(169)

It follows from Equation (169) that the respective second-order sensitivities have the following expressions:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)\hspace{0.17em}}dx={\displaystyle \frac{1}{\gamma }}\cos \mathcal{l}\gamma -{\displaystyle \frac{1}{\mathcal{l}{\gamma }^2}}\sin \mathcal{l}\gamma \hspace{0.17em}.$$

(170)

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \partial \gamma }}=-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(2;x\right)\hspace{0.17em}dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)\hspace{0.17em}dy}\\ +T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(-{\mathcal{l}}^2\gamma \sin \mathcal{l}\gamma +\mathcal{l}\cos \mathcal{l}\gamma +x^2\gamma \cos x\gamma -x\cos x\gamma -\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)\hspace{0.17em}}dx.\end{array}$$

(171)

The mixed second-order sensitivity ${\partial }^{2}R/\partial T_0\delta \gamma$ in Equation (170) does not depend on the 2nd-level adjoint sensitivity function $a^{\left(2\right)}\left(2;x\right)$ and was therefore evaluated immediately. Solving Equations (167) and (168) yields the following expression, to be evaluated at the nominal parameter values, for the 2nd-level adjoint sensitivity function $a^{\left(2\right)}\left(2;x\right)$:

$$a^{\left(2\right)}\left(2;x\right)=-T_{0}x\sin \gamma x.$$

(172)

Inserting the result obtained in Equation (172) into Equation (171) and performing the respective operations yields the following expression:

$${\displaystyle \frac{{\partial }^{2}R}{\partial \gamma (\theta )\partial \gamma (\theta )}}=T_0\left(2{\displaystyle \frac{\sin \mathcal{l}\gamma (\theta )}{\mathcal{l}{\gamma }^3(\theta )}}-2{\displaystyle \frac{\cos \mathcal{l}\gamma (\theta )}{{\gamma }^2(\theta )}}-{\displaystyle \frac{\mathcal{l}\sin \mathcal{l}\gamma (\theta )}{\gamma (\theta )}}\right).$$

(173)

It is evident from Equation (147) and Equation (160) or, alternatively, Equation (173) that the mixed second-order sensitivity ${\partial }^{2}R/\partial T_0\delta \gamma$ is computed twice, employing distinct expressions involving distinct 2nd-level adjoint sensitivity functions. This mechanism provides a stringent verification of the accuracy of the computation of the respective adjoint sensitivity functions.

In practice, the closed-form analytical expressions of the original functions, such as provide in Equation (101), are seldom available. Nevertheless, if such expressions are available, they can be advantageously used to reduce the amount of computations involved in determining the response sensitivities, as shown in the foregoing.

4.4.2. Alternative derivation of the second-order sensitivities by applying the 2nd-FASAM-NODE-F methodology

When applying the 2nd-FASAM-NODE methodology, the second-order sensitivities arise from the first-order sensitivities obtained in Equations (130) and (131). Thus, the second-order sensitivities arising from Equation (130) are provided by its G-differential for arbitrary variations around the nominal parameter and function values (indicated by the use of the superscript “zero”), which is by definition obtained as follows:

$$\delta \left\{{\displaystyle \frac{\partial R}{\partial T_0}}\right\}\triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{\left(1,0\right)}\left(x\right)+\epsilon \delta b^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}{\delta }^{\prime }\left(x\right)\hspace{0.17em}dx\right\}}_{\epsilon =0}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta b^{\left(1\right)}\left(x\right)}\hspace{0.17em}{\delta }^{\prime }\left(x\right)\hspace{0.17em}dx,$$

(174)

where ${\delta }^{\prime }\left(x\right)$ denotes the derivative of the Dirac-delta functional. The variational function $\delta b^{\left(1\right)}\left(x\right)$ is the solution of the following 2nd-LVSS, obtained by G-differentiating Equations (128) and (129):

$${\displaystyle \frac{d^2}{d{x}^2}}\delta b^{\left(1\right)}\left(x\right)+{\gamma }^2\delta b^{\left(1\right)}\left(x\right)=-2\gamma \left(\delta \gamma \right)b^{\left(1\right)}\left(x\right);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<x<\mathcal{l};$$

(175)

$$\delta b^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d}{dx}}\left[\delta b^{\left(1\right)}\left(x\right)\right]\right\}}_{x=\mathcal{l}}=0.$$

(176)

The above 2nd-LVSS for the function $\delta b^{\left(1\right)}\left(x\right)$ is to be satisfied at the nominal parameter values, but the superscript “zero” (which has been used to denote this fact) has been omitted to simplify the notation. The need for repeatedly solving this 2nd-LVSS for all parameter variations of interest is circumvented by eliminating the appearance of $\delta b^{\left(1\right)}\left(x\right)$ in Equation (174). This aim will be accomplished by expressing $\delta \left\{\partial R/\partial T_0\right\}$ in terms of the solution of the 2nd-LASS to be constructed by applying the steps outlined in Section 3. Thus, consider an adjoint function that will be denoted as $b^{\left(2\right)}\left(1;x\right)$, where the argument “1” indicates that this adjoint function corresponds to the first-order sensitivity$\partial R/\partial T_0$, which is chosen in this case to be the “first” 1st-order sensitivity to be considered. The 2nd-LASS to be satisfied by $b^{\left(2\right)}\left(1;x\right)$ will be constructed by applying the 2nd-FASAM-NODE, which commences by forming the inner product of $b^{\left(2\right)}\left(1;x\right)$ with Equation (175), to obtain the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)\left[\frac{d^2}{d{x}^2}\delta b^{\left(1\right)}\left(x\right)\right]d}x+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)\delta b^{\left(1\right)}\left(x\right)dx}\\ =-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)b^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\hspace{0.17em}.\end{array}$$

(177)

Integrating by parts the first term on the left-side of Equation (177) and rearranging the terms yields the following relation:

$$\begin{array}{l}{b}^{\left(2\right)}\left(1;\mathcal{l}\right){\displaystyle \frac{d}{dx}}\delta b^{\left(1\right)}\left(\mathcal{l}\right)-b^{\left(2\right)}\left(1;0\right){\displaystyle \frac{d}{dx}}\delta b^{\left(1\right)}\left(0\right)-\delta b^{\left(1\right)}\left(\mathcal{l}\right){\displaystyle \frac{d{b}^{\left(2\right)}\left(1;\mathcal{l}\right)}{dx}}+\delta b^{\left(1\right)}\left(0\right){\displaystyle \frac{d{b}^{\left(2\right)}\left(1;0\right)}{dx}}\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta b^{\left(1\right)}\left(x\right)\left[\frac{d^2}{d{x}^2}{b}^{\left(2\right)}\left(1;x\right)+{\gamma }^2{b}^{\left(2\right)}\left(1;x\right)\right]d}x=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)b^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\hspace{0.17em}.\end{array}$$

(178)

The last term on the left-side of Equation (178) is now required to represent the G-differential defined in Equation (174) to obtain the relation below:

$${\displaystyle \frac{d^2}{d{x}^2}}{b}^{\left(2\right)}\left(1;x\right)+{\gamma }^2{b}^{\left(2\right)}\left(1;x\right)={\delta }^{\prime }\left(x\right).$$

(179)

The unknown boundary terms are eliminated from Equation (178) by imposing the following conditions:

$$b^{\left(2\right)}\left(1;0\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d}{dx}}{b}^{\left(2\right)}\left(1;0\right)\right\}}_{x=0}=0.$$

(180)

The system of equations comprising Equations (179) and (180) constitute the 2nd-LASS for the 2nd-level adjoint sensitivity function $b^{\left(2\right)}\left(1;x\right)$.

Replacing the results obtained in Equations (176), (179) and (180) into Equation (178) yields the following alternative expression for the indirect-effect term:

$$\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{ind}=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)b^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\triangleq {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}\delta T_0+{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}\delta \gamma .$$

(181)

where the 2nd-level adjoint sensitivity function $b^{\left(2\right)}\left(1;x\right)$ is the solution of the 2nd-Level Adjoint Sensitivity System (2nd-LASS) comprising Equations (179) and (180). The 2nd-LASS is a NODE net that does not depend on parameter variations and needs to be solved once only at the nominal parameter values; its solution, $b^{\left(2\right)}\left(1;x\right)$, is used in Equation (181) to determine the respective second-order response sensitivities.

Identifying in Equation (181) the quantities that multiply the respective parameter variations yields the following expressions:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}=-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)b^{\left(1\right)}\left(x\right)\hspace{0.17em}dx};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}=0.$$

(182)

Solving the 2nd-LASS represented by Equations (179) and (180) yields the following expression for the 2nd-level adjoint sensitivity function $b^{\left(2\right)}\left(1;x\right)$:

$$b^{\left(2\right)}\left(1;x\right)=H\left(x\right)\cos \gamma x,$$

(183)

where $H\left(x\right)$ denotes the Heaviside-functional. Using in Equation (182) the results obtained in Equations (183) and (132) yields the following expression:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma (\theta )}}={\displaystyle \frac{1}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{1}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(184)

As expected, the expression obtained in Equation (184) is identical to the expressions obtained in Equation (170) and (149).

The second-order sensitivities arising from the first-order sensitivity represented by Equation (131) are obtained from its G-differential for arbitrary variations around the nominal parameter and function values. Thus, applying the definition of the G-differential to Equation (131) yields the following expression:

$$\delta \left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}\triangleq \delta {\left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}}_{dir}+\delta {\left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}}_{ind}$$

(185)

where the direct-effect and indirect-effect terms have the expressions below:

$$\delta {\left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}}_{dir}\triangleq -2\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}T\left(x\right)\hspace{0.17em}dx;$$

(186)

$$\delta {\left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}}_{ind}\triangleq -2{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}\delta T\left(x\right)\hspace{0.17em}dx-2{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta b^{\left(1\right)}\left(x\right)}\hspace{0.17em}T\left(x\right)\hspace{0.17em}dx.$$

(187)

The indirect-effect term $\delta {\left\{\partial R/\partial \gamma (\theta )\right\}}_{ind}$ will be recast in terms of an alternative expression that will not involve the variational functions $\delta T\left(x\right)$ and $\delta b^{\left(1\right)}\left(x\right)$ by applying the principles of the 2nd-FASAM-NODE, which are fundamentally the same as those underlying the 2nd-FASAM-NIDE-F, as follows:

• Consider the two-component vector function $B^{(2)}\left(2;2;x\right)\triangleq {\left[b^{(2)}\left(1;2;x\right),b^{(2)}\left(2;2;x\right)\right]}^{†}$, where the first argument denotes the component number and the second argument (“2”) indicates that this function will correspond to the “second” first-order sensitivity, namely $\partial R/\partial \gamma (\theta )$. Using the inner product defined in Equation (85), construct the inner product of $B^{(2)}\left(2;2;x\right)\triangleq {\left[b^{(2)}\left(1;2;x\right),b^{(2)}\left(2;2;x\right)\right]}^{†}$ with Equations (123) and (175), respectively, to obtain the following relation, to be satisfied at the nominal parameter values (although the superscript “zero” will be omitted for simplicity):

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{(2)}\left(1;2;x\right)\frac{d^2}{d{x}^2}\delta T\left(x\right)}\hspace{0.17em}dx+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{(2)}\left(1;2;x\right)\delta T\left(x\right)}\hspace{0.17em}dx+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{(2)}\left(2;2;x\right)}{\displaystyle \frac{d^2}{d{x}^2}}\delta b^{\left(1\right)}\left(x\right)dx\\ +{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{(2)}\left(2;2;x\right)}\delta b^{\left(1\right)}\left(x\right)dx=-2\gamma (\delta \gamma ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}dx.\end{array}$$

  (188)
• Integrate by parts the first and third terms on the left-side of Equation (188) and rearrange the terms to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}\left[{\displaystyle \frac{d^2{b}^{(2)}\left(1;2;x\right)}{d{x}^2}}+{\gamma }^2{b}^{(2)}\left(1;2;x\right)\right]dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta b^{\left(1\right)}\left(x\right)\left[\frac{d^2{b}^{(2)}\left(2;2;x\right)}{d{x}^2}+{\gamma }^2{b}^{(2)}\left(2;2;x\right)\right]}\hspace{0.17em}dx\\ +P\left[\delta T\left(x\right),\delta b^{\left(1\right)}\left(x\right),b^{(2)}\left(1;2;x\right);b^{(2)}\left(2;2;x\right)\right]\\ =-2\gamma (\delta \gamma ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}dx.\end{array}$$

  (189)

  where the bilinear concomitant $P\left[\delta T\left(x\right),\delta b^{\left(1\right)}\left(x\right),b^{(2)}\left(1;2;x\right);b^{(2)}\left(2;2;x\right)\right]$ is defined below:

  $$\begin{array}{l}P\left[\delta T\left(x\right),\delta b^{\left(1\right)}\left(x\right),b^{(2)}\left(1;2;x\right);b^{(2)}\left(2;2;x\right)\right]\triangleq b^{(2)}\left(1;2;\mathcal{l}\right){\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}\\ -b^{(2)}\left(1;2;0\right){\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=0}-{\left\{\delta T\left(x\right){\displaystyle \frac{d{b}^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=\mathcal{l}}+{\left\{\delta T\left(x\right){\displaystyle \frac{b^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=0}\\ +b^{(2)}\left(2;2;\mathcal{l}\right){\displaystyle \frac{d}{dx}}\delta b^{\left(1\right)}\left(\mathcal{l}\right)-b^{(2)}\left(2;2;0\right){\displaystyle \frac{d}{dx}}\delta b^{\left(1\right)}\left(0\right)\\ -\delta b^{\left(1\right)}\left(\mathcal{l}\right){\displaystyle \frac{d{b}^{(2)}\left(2;2;\mathcal{l}\right)}{dx}}+\delta b^{\left(1\right)}\left(0\right){\displaystyle \frac{d{b}^{(2)}\left(2;2;0\right)}{dx}}.\end{array}$$

  (190)
• Require the first and second terms on the left-side of Equation (189) to represent the indirect-effect term defined in Equation (187) by imposing the following relations:

  $${\displaystyle \frac{d^2{b}^{(2)}\left(1;2;x\right)}{d{x}^2}}+{\gamma }^2{b}^{(2)}\left(1;2;x\right)=-2\gamma b^{\left(1\right)}\left(x\right);$$

  (191)

  $${\displaystyle \frac{d^2{b}^{(2)}\left(2;2;x\right)}{d{x}^2}}+{\gamma }^2{b}^{(2)}\left(2;2;x\right)=-2\gamma T\left(x\right).$$

  (192)
• Eliminate the unknown boundary terms in the expression of the bilinear concomitant defined in Equation (190) by imposing the following boundary conditions:

  $$b^{(2)}\left(1;2;\mathcal{l}\right)=0;\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d{b}^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=\mathcal{l}}=0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{b}^{(2)}\left(2;2;0\right)=0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d{b}^{(2)}\left(2;2;x\right)}{dx}}\right\}}_{x=0}=0\hspace{0.17em}.$$

  (193)
  The system comprising Equations (191)‒(193) constitutes the 2nd-LASS for the two-component 2nd-level adjoint sensitivity function $B^{(2)}\left(2;2;x\right)\triangleq {\left[b^{(2)}\left(1;2;x\right),b^{(2)}\left(2;2;x\right)\right]}^{†}$.
• Insert the boundary conditions represented by Equations (124) and (176) into Equation (193) and use the relations representing the 2nd-LASS to obtain the following expression for the indirect-effect term defined in Equation (152):

  $$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{ind}=-\delta T_0\hspace{0.17em}{\left\{{\displaystyle \frac{b^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=0}\\ -2\gamma (\delta \gamma ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}dx.\end{array}$$

  (194)

Adding the expression obtained in Equation (194) to the expression of the direct-effect term provided in Equation (186) yields the following expression for the G-differential $\delta \left\{\partial R/\partial \gamma \right\}$:

$$\begin{array}{l}\delta \left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}\triangleq -2\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}T\left(x\right)\hspace{0.17em}dx-\delta T_0\hspace{0.17em}{\left\{{\displaystyle \frac{b^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=0}\\ -2\gamma (\delta \gamma ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}dx.\end{array}$$

(195)

It follows from the expression obtained in Equation (195) that:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}=-{\left\{{\displaystyle \frac{b^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=0};$$

(196)

$${\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \delta \gamma }}=-2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}T\left(x\right)\hspace{0.17em}dx-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}dx.$$

(197)

Solving the 2nd-LASS represented by Equations (191)‒(193) yields the following expressions for the components of $B^{(2)}\left(2;2;x\right)\triangleq {\left[b^{(2)}\left(1;2;x\right),b^{(2)}\left(2;2;x\right)\right]}^{†}$:

$$\begin{array}{l}{b}^{(2)}\left(1;2;x\right)={\displaystyle \frac{2}{\mathcal{l}{\gamma }^2}}\left[-{\displaystyle \frac{1}{\gamma }}+{\displaystyle \frac{x}{2}}\sin \gamma \left(x-\mathcal{l}\right)-{\displaystyle \frac{1}{4\gamma }}\cos \gamma \left(x-\mathcal{l}\right)\right]\\ \left({\displaystyle \frac{5}{2\mathcal{l}{\gamma }^3}}\sin \gamma \mathcal{l}+{\displaystyle \frac{{\sin }^2\gamma \mathcal{l}}{{\gamma }^2\cos \gamma \mathcal{l}}}-{\displaystyle \frac{1}{{\gamma }^2\cos \gamma \mathcal{l}}}\right)\sin \gamma x+\left({\displaystyle \frac{5}{2\gamma {\mathcal{l}}^3}}\cos \gamma \mathcal{l}+{\displaystyle \frac{\sin \gamma \mathcal{l}}{{\gamma }^2}}\right)\cos \gamma x\hspace{0.17em};\end{array}$$

(198)

$$b^{(2)}\left(2;2;x\right)=-T_{0}x\sin \gamma x\hspace{0.17em}.$$

(199)

Using in Equation (196) the result obtained in Equation (198) yields the following expression:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma (\theta )}}={\displaystyle \frac{1}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{1}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(200)

As expected, the above expression coincides with the expression obtained, successively, in Equations (149), (170) and (184). Evidently, the expression of the mixed second-order sensitivity ${\partial }^{2}R/\partial T_0\delta \gamma$ can be determined in several distinct ways, using distinct adjoint sensitivity functions, thus providing alternatives for verifying the computational accuracy of the respective adjoint functions, when these functions are computed numerically, as is the case in practice.

Inserting the results obtained in Equations (101), (183), (198) and (199) into Equation (197) and performing the respective integrations yields the following expression:

$${\displaystyle \frac{{\partial }^{2}R}{\partial \gamma (\theta )\partial \gamma (\theta )}}=T_0\left(2{\displaystyle \frac{\sin \mathcal{l}\gamma (\theta )}{\mathcal{l}{\gamma }^3(\theta )}}-2{\displaystyle \frac{\cos \mathcal{l}\gamma (\theta )}{{\gamma }^2(\theta )}}-{\displaystyle \frac{\mathcal{l}\sin \mathcal{l}\gamma (\theta )}{\gamma (\theta )}}\right).$$

(201)

As expected, the above expression coincides with the expression obtained in Equation (173).

5. Discussion and Conclusions

This first part work has introduced the First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type (1st-FASAM-NIDE-F), which enables the most efficient computation of exactly obtained expressions of the first-order sensitivities of NIDE-F decoder-responses with respect to the optimized NIDE-F weights/parameters. After introducing the framework of the 1st-FASAM-NIDE-F for a NIDE-F involving arbitrarily-high-order derivatives of the dependent variable (representing the hidden/latent neural networks) with respect to the time-like independent variable, this work has presented the application of the 1st-FASAM-NIDE-F to first-order and, subsequently, second-order NIDE-F neural nets. Remarkably, the application of the 1st-FASAM-NIDE-F requires a single “large-scale” computation, for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), in order to compute all of the first-order sensitivities of the decoder response, regardless of the number of weights/parameters underlying the NIDE-F net.

Subsequently, this work has presented the general mathematical framework underlying the Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type (2nd-FASAM-NIDE-F), which enables the most efficient computation of the exactly obtained expressions of the second-order sensitivities of NIDE-F decoder-responses with respect to the optimized NIDE-F weights/parameters. Next, this work has presented the application of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies to an illustrative paradigm heat conduction model. This illustrative model has been chosen because it can be formulated either as a first-order differential-integral equation of Fredholm type or as a conventional second-order “neural ordinary differential equation (NODE)”, while admitting exact closed-form solutions/expressions for all quantities of interest, including state functions, first-order and second-order sensitivities. The availability of these alternative formulations, either as a NIDE-F or a NODE, of the illustrative paradigm heat conduction model makes it possible to compare the detailed, step-by-step, applications of the 1st-FASAM-NIDE-F versus the 1st-FASAM-NODE methodologies (for computing most efficiently the exact expressions of the first-order sensitivities of decoder response with respect to the model parameters) and, subsequently, to compare the applications of the 2nd-FASAM-NIDE-F versus the 2nd-FASAM-NODE methodologies (for computing most efficiently the exact expressions of the second-order sensitivities of decoder response with respect to the model parameters).

Ongoing work aims at developing the Second-Order Features Adjoint Sensitivity Analysis Methodologies for Neural Integro-Differential Equations of Volterra-Type (2nd-FASAM-NIDE-V), which will enable, in premiere, the most efficient computation of the exact expressions of the first- and second-order sensitivities of decoder-responses with respect to the optimized network’s weights/parameters for such neural nets.