Application of Long-Short Term Memory and Convolutional Neural Networks for Real-Time Bridge Scour Forecast
Tahrima Hashem Negin Yousefpour



























Department of Infrastructure Engineering, The University of Melbourne, VIC, Australia


[*]Corresponding Author

tahrima.hashem@unimelb.edu.au, negin.yousefpour@unimelb.edu.au


Scour around bridge piers is a critical challenge for infrastructures around the world. In the absence of analytical models and due to the complexity of the scour process, it is difficult for current empirical methods to achieve accurate predictions. In this paper, we exploit the power of deep learning algorithms to forecast the scour depth variations around bridge piers based on historical sensor monitoring data, including riverbed elevation, flow elevation, and flow velocity. 
 
We investigated the performance of Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) models for real-time scour forecasting using data collected from bridges in Alaska and Oregon from 2006 to 2021. The LSTM models achieved mean absolute error (MAE) ranging from 0.1m to 0.5m for predicting bed level variations a week in advance, showing a reasonable performance. The Fully Convolutional Network (FCN) variant of CNN outperformed other CNN configurations, showing a comparable performance to LSTMs with significantly lower computational costs. We explored various innovative random-search heuristics for hyperparameter tuning and model optimisation which resulted in reduced computational cost compared to grid-search method. The impact of different combinations of sensor features on scour prediction showed the significance of the historical time series of scour for predicting upcoming events. Overall, this study provides a greater understanding of the potential of Deep Learning (DL) for real-time scour forecasting and early warning in bridges with diverse scour and flow characteristics including riverine and tidal/coastal bridges. 



Scour Prediction, Deep-learning, Time Series Forecasting, Hyperparameter Tuning, Long-short Term Memory Network, Convolutional Neural Network






§ INTRODUCTION

Bridge scour has been a major challenge for transportation systems around the world due to the significant damage and operational disruptions it can cause, and more importantly the risk to public safety. In the US, for example, scour accounts for a significant number of bridge collapses, with more than 260,000 brides listed as vulnerable to scour failure <cit.>. 
It is difficult to reliably predict the maximum scour depth for a bridge pier due to the complexity of the underlying causes of soil erosion in interaction with flow and structure. Uncertainties in riverbed material, flow, and geomorphological conditions, as well as climate change impacts, are the main challenges in the assessment of scour risk. For the last couple of decades, several research studies  <cit.> have focused on developing empirical models to estimate scour depth based on laboratory experiments and field observations. Most of these models often overestimate the depth of the scour (and in some cases underestimate) due to insufficient generalization to various conditions of the riverbed, flow, and structure  <cit.>. 

Artificial Intelligence (AI) and Machine Learning (ML) for scour depth estimation have been investigated by a range of algorithmic strategies, such as evolutionary computing, fuzzy logic, artificial neural networks, support vector machines, and decision trees (see section  <ref>). These proposed techniques have shown promising results in estimating maximum scour depth, often outperforming traditional empirical equations. However, the performance of these models is still limited by the availability and quality of training data, which are often scarce and may not cover a wide range of geological, hydraulic and geomorphological conditions. Furthermore, these models are typically trained to predict a maximum scour depth for a given flow discharge and cannot be used for dynamic scour prediction and real-time forecast.

Yousefpour et al. <cit.> pioneered the application of deep learning (DL) solutions, using Long Short Term Memory (LSTM) networks <cit.> for real-time scour forecast. They leveraged the LSTM's superiority in temporal pattern recognition and capturing the underlying physics without direct feature extraction. They used historical scour monitoring data in Alaska to train DL models and predict future bed elevation variations and upcoming scour events. Fig. <ref> shows the process of local scour around a bridge pier; scour happens when sediments are eroded and washed away by strong vortices in interaction with the bridge substructure, often accelerating during floods. Stage (flow/water level) and sonar (bed elevation) sensors are commonly used sensors installed on the bridge piers to monitor scour by collecting real-time flow and bed elevation data. The variants of LSTM models developed in <cit.> showed promising performance in forecasting the scour depth a week in advance, for case study bridges in Alaska. However, there remain research questions about the capability of these algorithms to be upscaled for implementation in practice and deployment in different locations with variable meteorological, geological, geomorphological, and flow conditions.



In this study, we addressed fundamental research questions about the efficiency and generalization of the LSTM scour forecast models in diverse temporal and geographical domains. The LSTM models are compared against temporal CNN as an alternative deep learning solution, in terms of performance and computational cost. Random-Search (RS) heuristics for hyperparameter (HP) tuning are evaluated  <cit.> to explore alternate computationally efficient methods to Grid-Search (GS) used in  <cit.> for model optimisation. We also investigated the impacts of monitored features (input parameters), including stage, sonar, and velocity (discharge), on the performance of scour forecast models. To summarise, four research questions were addressed in this study:

(RQ1) How do LSTM scour forecast models perform in predicting recent years of scour (Alaska) and how well do these models perform in a totally different location (Oregon)?

(RQ2) How do other competitor models, including CNN with vanilla and dilated convolutions, compare with LSTMs for scour forecast? 






(RQ3) What computationally efficient heuristics can be adopted to find optimal model configurations to scour forecast DL models without sacrificing performance?

(RQ4) To what extent combinations of different sensor features can impact the performance of scour forecast DL models, and what are the most optimum feature combinations?

We carefully designed experiments and developed heuristics to investigate the aforementioned research questions as discussed in the following sections. 




§ BACKGROUND



 §.§ Maximum Scour Prediction

 
Since the 1940s, researchers have been using laboratory experiments and real-world observations to formulate empirical equations to estimate maximum scour depth. The famous equations HEC-18 <cit.>, FDOT <cit.>, and CIRIA <cit.> equations were derived by multivariate analysis, considering various factors that impact the scour process. These equations have a large number of input and model parameters and measurement of the inputs is complex. However, often small databases (a few dozen to a few hundred samples) are employed for model calibration <cit.>. Also, these empirical equations follow particular mathematical forms, such as linear, exponential, and hyperbolic terms, limiting their flexibility in capturing scour as a nonlinear phenomenon. As a result, the recommended equations often underperform in application to a broad range of scour conditions.

With the surge of data-driven methodologies, more sophisticated approaches from machine learning have been implemented for scour depth estimation, including Support Vector Machines (SVM) <cit.>, Genetic Algorithms <cit.>, Decision Trees <cit.>, Artificial Neural Networks (ANNs) <cit.>, and CNNs <cit.>. The results show that machine learning-based methods are usually more accurate than traditional equations in maximum scour depth prediction. However, these ML models face limitations when encountering unseen cases outside the convex hull of the training data,  which can lead to poor generalization. 



 §.§ Real-time Scour Monitoring and Forecast

Although the worst-case scenario (usually a 1-in-100-year flood) is considered for estimating maximum scour depth in bridge design, many uncertainty factors could still lead to scour-related bridge failures: Larger flood events can occur due to extreme weather conditions and climate change; complexities in the scour process, particularly sedimentation and riverbed soil behaviour in the interaction with the flow which can result in larger scour depth;  older bridges could fall behind the best recommended practices for scour estimation <cit.>.  Considering all these factors, the transport authorities in the past two decades have been resorting to regular monitoring to manage scour, including regular inspections and surveys as well as advanced remote sensing.

In recent years, real-time scour monitoring sensors have become a prominent solution installed on many critical and large-scale bridges <cit.>. With the development of monitoring approaches, the sensors can facilitate data collection in real-time concerning bed elevation and other flow-related information. Compared to traditional databases <cit.>, the sensor-collected data are richer and more granular. Therefore, the monitoring data can also be utilized for scour depth prediction and providing early-warning information.

Yousefpour et al. <cit.> pioneered the idea of incorporating real-time scour monitoring data to develop real-time scour forecast models using LSTMs. They incorporated the monitoring data in Alaska (2006 to 2017) collected by the U.S. Geological Survey (USGS) in partnership with the  Departments of Transport (DOTs) in the US. Specifically, historical time series of flow depth and bed elevation were used to train the LSTM models to predict future bed elevation and upcoming scour depth. The proposed model showed promising results in forecasting bed-elevation changes for at least seven days in advance. Compared to traditional empirical equations, these time-series forecasting models showed to yield more reliable predictions of upcoming scour trends and maximum scour depth.


Chang et al. <cit.> presented a real-time monitoring system, utilizing micro-cameras mounted on bridge piers, in conjunction with image processing and pattern recognition methodologies, to monitor variations in bed levels. This approach was validated through rigorous laboratory experiments, illustrating that the image data can be transformed to capture the progressive alterations in scour depth. Typically, monitoring sensors are installed on bridges deemed high risk. 

To leverage limited data to better estimate scour risks across a bridge network, Maroni et al. <cit.> propose a scour hazard model tailored for road and railway bridges. By using a Bayesian network, the model decreases uncertainty in scour depth assessments at unmonitored bridges. This approach was demonstrated in a case study involving several Scottish road bridges.

Lin et al. <cit.> developed an early scour warning system installed at Da-Chia Bridge in Taiwan. They used R-CNN (a residual CNN) to read the real-time water level through CCTV images, and dedicated vibration-based arrayed sensors to obtain real-time local bed elevation of the target pier. These two data sources were incorporated to simulate upcoming hydrodynamic flows in fluvial rivers with hydraulic structures (e.g., bridge piers). The general scour is derived from a numerical model, and the local scour is estimated by empirical equations.



§ SCOUR MONITORING DATA



We considered two scour monitoring databases for Alaska and Oregon provided by USGS to develop real-time scour forecast models:

    
  * Alaska dataset: This includes previously collected sensor readings data from 2006 to 2017 from bridges over Sheridan River, Knik River, and Chilkat River with IDs 230, 539, and 742, respectively. Readers are referred to <cit.> for full data description. In addition, in this study we incorporated new data collected for 2018 to 2021 (for 742 bridge, 2019 and 2020 data are not available). An example of processed versus raw data for Alaska 539 bridge is provided in Fig.  <ref>.
    
  * Oregon dataset: This includes sensor readings from 2018 to 2021 for two bridges over the Trask and Luckiamute rivers (see Fig.  <ref>). 

For both datasets, each observation corresponds to an hourly recorded reading of the available sensors. We followed the same methodology as <cit.> to denoise, synchronize, and filter the raw monitoring data provided by USGS. 



A strong negative correlation between stage and sonar can be observed for Alaska bridges. This is expected as higher flow velocity and depth result in increased scour depth, for most riverbed sedimentation.
In addition, Alaskan bridges show consistent seasonal scour and filling patterns throughout a year, which is a key characteristic of Live-Bed scouring; each year, major scour followed by filling episodes are observed due to high flows (floods) happening around July/August and September/October.
However, the scour patterns for the two bridges in Oregon are fundamentally different. Trask is a tidal river where bed material follows the tide cycles: scouring during outgoing tides and filling occurs during high tides. Also, the Luckiamute bridge shows a positive correlation trend between stage and sonar, contrary to expectations. Oregon USGS relates this to the presence of bed forms, which are structures such as sand dunes formed by the movement of bed material following the flow direction. 





§ APPROACH





 §.§ Problem Definition

Given the historical sensor readings, i.e., water level: Stage, riverbed elevation: Sonar, and flow discharge: Discharge over time, we define the scour forecast task as a multivariate time-series prediction problem. 

Following is the mathematical definition of the scour forecast task.


Assume a series of past values {x_t-w_in+1, x_t-w_in+2, …, x_t} for N sensors (features) with the historical window w_in where x_i ∈ℝ^N and 1 < w_in≤ t. The scour forecasting task is to predict a series of future values, {x_t+1, x_t+2, …, x_t+w_out} for the Sonar with the forecast window w_out where w_out≤ t.




 §.§ Feature Engineering

In this study, we explored adding Discharge (X_discharge) in addition to Stage (X_stage) and Sonar (X_sonar), incorporated in our previous study <cit.> for Alaskan bridges [for Oregon bridges, this sensor data was not available]. Discharge represents the water volume passing the river cross section per unit of time. X_discharge which is obtained from the mean flow velocity measured by velocimeter sensors multiplied by the river channel cross-section area measured at the bridge location  <cit.>.

We also engineered another feature, the equivalent velocity X_eVelocity, which is directly proportional to the flow velocity following Equations. <ref> to  <ref>. By assuming a constant channel equivalent cross-sectional width α, the channel cross section area can be estimated by α x d, where d is the flow depth at a bridge pier equal to the difference between Sonar and Stage elevations (Equation  <ref>). 



    d    = X_sonar- X_stage
    
        X_discharge   = d * α * velocity 
    
        X_eVelocity   =  α * velocity 
    
        X_eVelocity   = X_discharge/(X_sonar - X_stage)


The previous study showed that time features did not improve the performance of the LSTM models for Alaskan bridges. However, to enrich the periodicity information in Oregon's scour data, time features, namely year-sin and year-cos are incorporated in this study. These features are calculated by applying sin(·) and cos(·) transformations to the recorded timestep using Equation  <ref> and Equation  <ref>, respectively.





    year-sin   = sin(TS * (2π / year)) 
    year-cos   = cos(TS * (2π / year))


The features Sonar, Stage, Discharge and eVelocity are denoted as , ,  and , respectively.  Time features are denoted as y. The impact of various features and their combinations are discussed in Section <ref>.




 §.§ Deep Learning For Scour Forecast





  §.§.§ LSTM Models

Long Short Term Memory (LSTM)  <cit.> is a specialised recurrent neural network (RNN) that is capable of modelling very large input and output sequences and does not suffer from the gradient vanishing problem - a typical problem in RNNs  <cit.>. 
The principle analogy of LSTM is to remember the relevant short and long-standing information and discard the irrelevant information. Three different gates, i.e., Forget gate (Γ^f), Input gate (Γ^i) and Output gate (Γ^o), enable the LSTM model to achieve this goal as shown in Fig.  <ref> (a). In these gates, the degree of relevance of information is quantified using a sigmoid (σ) function with a value between 0 and 1. The larger the value, the higher the relevance. Equations (<ref> - <ref>) show how the hidden unit output (a_t) and LSTM memory cell output (c_t)  are recurrently computed using these gates. W and b represent the weight and bias matrices.


    Γ^f   = σ(W_f[a_t-1,x_t]+b_f)
    
    Γ^i   = σ(W_i[a_t-1,x_t]+b_i)
    
    Γ^o   = σ(W_o[a_t-1,x_t]+b_o)
    
    c̃_̃t̃   = tanh(W_c[a_t-1,x_t]+b_c)
    
    
    c_t    =  Γ^i*c̃_̃t̃ + Γ^f*c_t-1
    
    a_t    = Γ^o*tanh(c_t)








The following three variants introduced in <cit.> are implemented in this study:



  * Single-Shot (ss): In this model, one LSTM layer has been followed by one Dense and one Reshape layer [see Fig.  <ref> (b)]. This model outputs the predictions for all the timesteps over the forecast window (h) at once. Fig.  <ref> shows the architecture of the Single-Shot LSTM model for the multivariate time series forecast.

  * Two-Layer, Single-Shot (ss2): Two LSTM layers are stacked together to add more complexity to the LSTM network so that it captures more variability of the input features. The rest of the setting remains the same as Single-Shot.

  * Feedback (Auto-Regressive: fb): In this variation, we hypothesize that the model's prediction accuracy would get improved, if we input the predicted value (as opposed to the actual value) of each timestep within the label/forecast window for the prediction of the next timestep[Google Tensorflow: ]. Therefore, in this variant, LSTM predicts one timestep ahead across the entire forecast window. Fig.  <ref> (c) illustrates the feedback process. Only one LSTM layer is used, similar to the Single-Shot model.


Model configuration nomenclature for an LSTM model is encoded as follows:


    [model]-[w_in,w_out]-[units]-[dropout]

where, model is the LSTM variant, w_in and w_out are the length of model inputs and outputs, units is the number of LSTM units, and dropout denotes the dropout rate applied to the model.


  §.§.§ CNN Models


CNN models proposed in  <cit.> are widely used in computer vision for tasks like image classification  <cit.> and object detection  <cit.>. They offer excellent performance with lower computational requirements. CNNs also excel in time series modelling tasks  <cit.>. Here, we investigated three CNN variants for multi-variate time-series scour forecasting (see Fig.  <ref>).








In time-series forecast, a pivotal element of the CNN model is the convolution process—a filter sliding over input time-sequence for feature extraction as depicted in Fig.  <ref>. Equation  <ref> defines the standard convolution operation (C_vanilla) as a dot product between a filter (F) and a time-sequence (S) of lengths k and l respectively. 
Because of the weight-sharing property of a CNN model, the entire time sequence is repeatedly convolved by the same filter.
Fig.  <ref> (a) illustrates this vanilla convolution using a 3-step filter on a 9-step sequence with a stride of 1. Convolution results in a reduced-length sequence of 7 steps (l - k + 1 = 9 - 3 + 1 = 7), which leads to information loss. To prevent this loss, padded convolution (C_padded) operates on an extended sequence S', created by padding zeros to the edges of S [see Equation  <ref> and Fig.  <ref> (b)]. This ensures the input and convolved time sequences have the same length.




    C_vanilla   :=  F*S_t:t+k  | ∀ t ∈ [1, l-k] 
    
    C_padded   :=  F*S'_t:t+k  | ∀ t ∈ [0, l-k+1] 
    
    C_casual   :=  F*S'_t-k+1:t  | ∀ t ∈ [1, l] 
    
    C_dilated-casual   :=  F'*S'_t-(k+d-1)+1:t  | ∀ t ∈ [1, l]




Casual convolution (C_casual) is often a preferable algorithm in time-series forecasting to avoid information leakage from the future  <cit.>.

It precedes the input sequence with k-1 zeroes before convolution, as illustrated in Fig.  <ref> (c). Casual convolution would require a large filter or a deep network when it needs to examine a large number of past observations. Dilated Convolution overcomes this drawback by introducing a dilation factor d into the filter so that convolution can be applied on an input area larger than the original filter size, i.e., capturing larger contextual information. The modified filter F^' is obtained by dilating the original filter with (d-1) zeros as shown in Fig.  <ref> (d). 

Dilation is performed using d=2. The positions of zero correspond to the skipped timesteps in the input sequence. 
Both the casual (in Equation  <ref>) and dilated casual convolutions (C_dilated-casual) in Equation  <ref> output a sequence with the same length of the input. In the following, we introduce three CNN model configurations to perform the multivariate scour forecast:






1) Vanilla Convolutional Network (VCN):
The convolution block of the VCN model comprises one layer of standard convolution (from Equation <ref>) and one layer of nonlinear activation function, i.e., Rectified Liner Unit (ReLU). Two identical convolution blocks are stacked together as a feature extractor that precedes a flattened layer, a drop-out layer, and a fully connected dense layer. 

Finally, a reshape layer is added to ensure that the predicted time sequence contains the same number of sensor variables as input.


2) Fully Convolutional Network (FCN):
A FCN ensures that the length of the convolved sequence is the same as the input. Here, the zero-padded convolution from Equation  <ref> with a stride value of 1 is used. We followed the structure adopted by Shelhamer et al.  <cit.> for the FCN in this study, where the main block consists of one convolution layer, one batch normalization (BN) layer, and one ReLU layer. BN layer  <cit.> ensures fast convergence and improves generalisation performance. The rest of the setup remains the same as the VCN. 

3) Dilated Causal Network (DCN):
As shown in  Equation  <ref> and Fig. <ref> (d), dilation factors of 1 and 2 are applied in two convolution layers. This enables the CNN model to perform causal convolution at a fine-grained level on the input sequence and a coarse-grained level on the convolved sequence (from the lower layer). Each convolution layer is followed by one BN layer and one ReLU layer.
Model configuration nomenclatures for a temporal CNN are encoded as follows (with k and F being the filter size and no. of filters of the layers, respectively):

    [model]-[k_1]-[F_1]-[k_2]-[F_2]-[dropout]


where the model is different CNN variants introduced, k_1 and k_2 are the kernel size, F_1 and F_2 are the hidden size of the dense layer, and dropout is the dropout rate applied to the model.


 §.§ Model Evaluation Criteria

We used Mean Square Error (MSE) on the Sonar readings to train the scour forecast model. For S no. of samples in a batch (32) with x_i,t and x̂_i,t representing the actual and predicted values of Sonar feature X_sonar at timestep t for the i^th sample, the training loss is computed as follows. 

    MSE = 1/S*w_out∑_i=1^S∑_t=1^w_out(x_i,t-x̂_i,t)^2, x_i,t∈ X_sonar

We evaluated the scour prediction performance of a model using the metric MAE on the Sonar readings. For S number of samples in a batch with x_i,t and x̂_i,t representing the actual and predicted values of Sonar feature X_sonar at timestep t for the i^th sample, the performance metric, MAE is computed as follows. 

    MAE = 1/S*w_out∑_i=1^S∑_t=1^w_out|x_i,t-x̂_i,t|, x_i,t∈ X_sonar



We used the hold-out method <cit.> for evaluating the performance of the models. In the same unit as sensor readings, MAEs are reported in foot elevation throughout.


 §.§ Hyperparameter Tuning
 
 DL model optimisation and HP tuning using grid-search is computationally expensive <cit.>. Following the grid-search approach in  <cit.>, the performance of all possible HP combinations was analysed by running each configuration for a large number of iterations. 
To explore a more computationally efficient alternative to this approach, we investigated a random-search strategy 
based on the following three possible policies to select the best model configurations. In this approach, a sample of HP combinations with 50%, 35%, and 67% sample size(s) out of all possible combinations are randomly selected in a number of trials (t=20). 

The grid-search results were used as a benchmark for the "true" performance of each configuration and to evaluate the outcome of these random search strategies. 





  * meanMAE Policy: Model configurations are ranked based on the average MAE values across random trials. Lower average MAE values lead to higher rankings; this is irrespective of the number of times a model configuration appears across all trials (=f).



  * medianMAE Policy: Model configurations are ranked by taking the median of the MAE values across the trials. Models with smaller median MAE would receive a higher rank than the model with larger median values; similar to mean MAE policy, this is irrespective of the number of times a configuration appears across all trials. 



  * Bagging (Bootstrap Aggregation) Policy <cit.>:
In this policy, top k configurations with the lowest MAE values are selected on each trial. Considering f_topk as the number of times a particular configuration is ranked as top-k across t trials, k models (out of possible k*t models) are finally selected based on f_topk; model configurations with more frequent appearances in the top-k (i.e., larger f_topk) receive higher ranks. 




Finally, to observe how well the trained models generalize on unseen data, we generated prediction plots on the test datasets for the top-5 models. 
These top-5 models were selected after careful observations of the mean and standard deviation of MAE scores from both the random and grid-search policies. The prediction plots evaluate the performance of DL models in recognizing the scour (and filling) trends across time.



 §.§ Experiments

Table <ref> provides a summary of experiments we conducted to address the research questions outlined in Section  <ref>. For all experiments except for Feature Combination Analysis,  and  are used as input features. 









§ RESULTS












 §.§ Evaluation of Random Policies for HP Tuning (Experiment 1)


We evaluated the HP selection policies on Alaska 742 bridge, using the single-shot LSTM scour forecast models. 
Given the past 1, 2, 3 weeks (168, 336, 720 hrs) observations (w_in), we forecast hourly flow and bed elevation variation, aiming to predict scour for the upcoming week (w_out). Here, the full search space comprises 18 distinct HP configurations as per Table <ref>. 

TALOS's uniform mersenne strategy  <cit.> is used for random selection of HPs. 

Fig.  <ref> demonstrates the results of HP tunning using grid-search and random-search policies for the 18 scour forecast models. Top-3 models with the smallest average MAE from grid-search (GS) served as benchmarks when evaluating random-search (RS) policies. Since the distribution of MAE values is not heavily skewed as observed in the boxplots, the mean values are used to rank the models in Fig.  <ref> (a). It is noted that higher ranked models (top-5) would remain the same with median criteria. Note that all the reported MAE values throughout the paper are in m (of elevation).








The meanMAE and medianMAE random-search policies show similar trends in lower sample sizes (35% and 50%), but as expected, they deviate by increasing the sample size.  With a growing sample size, the mean-MAE estimations seem to converge towards grid-search estimates. 
The bagging policy also fails to find the top-ranked models when using small sampling sizes; however, with increasing sample size (67%), this method shows to be comparatively more robust and less sensitive to the poor-performing models compared to the meanMAE and medianMAE policies (e.g. models 12 to 18 show low ranking based on bagging but rank high in mean and median policies). Furthermore, the meanMAE and bagging policies seem to agree with 2 out of 3 top selected configurations, whereas the medianMAE policy fails to identify any of the top-performing models [Fig.  <ref> (e)].






 §.§ LSTM Evaluation (Experiment 2)


The three variants of scour forecast LSTM models were evaluated to explore performance on 1) the recent years of collected sensor data from Alaska case study bridges, 230, 539, and 742 and 2) Oregon case study bridges, Trask and Luckiamute, with drastically different scour behaviour. 
Alaska and Oregon LSTM models were trained using the hold-out method, by partitioning the data into training, validation, and test subsets, with a splitting ratio (%) of 70-20-10 and 60-20-20, respectively. 

The HP configurations (see Table <ref>) are optimized based on random-search method with meanMAE policy using %67 sample size. The top-5 LSTM model configurations for each bridge are presented in Table <ref>. The best-performing models achieve mean MAE values ranging from 0.078 to 0.410 ft (0.12 m) for Alaska bridges and 0.249 to 1.573 ft (0.48m) for Oregon bridges. 

The Single-Shot (ss) variant shows consistently better performance than Feedback (fb) variant for all bridges. Models with larger number of past observations, e.g., w_in=720, obtain smaller mean errors and appear as higher rank models. An input width 2 or 3 times the output/target width (forecast window) seems to yield the most accurate scour predictions. 















  §.§.§ Scour Forecast for Alaska Bridges

Fig.  <ref> and Fig.  <ref> present the predicted values of Sonar (bed elevation) and Stage (water elevation) by the top-5 models for bridge 230 and 539 across the test dataset, between July-2021 to Sep-2021. The performance of models is assessed based on bed elevation (scour depth) predictions. 

The models show reasonable prediction accuracy and are able to capture the scour and filling trends a week in advance throughout the test (unseen) period of the data.






 


For bridge 230, during the peak time of scour (August), the maximum error between the predicted and the actual value of scour is around 2ft (0.6m), whereas for the 539 bridge, it is around 1ft (0.3m). The inverse relation of the scour trend with the stage-water is well captured by the LSTM models with negotiable lags for all bridges. 








  §.§.§ Scour Forecast for Oregon Bridges

 Stage and Sonar prediction of top-5 models over the test dataset is presented in Fig.  <ref> and Fig.  <ref> for Luckiamute and Trask bridges, respectively. The test data is selected for the period between April-2021 to July-2021 and August-2021 to Nov-2021, for Luckiamute and Trask, where significant scour and fillings are observed.



 For Oregon Luckiamute, some of the large spikes in bed elevation seem to be due to errors in reading, which were not picked up by pre-processing techniques (verified with Oregon USGS). Yet, LSTM models are able to identify the realistic trend of bed elevation variation.  For Trask which is a tidal river showing frequent daily fluctuations due to low and high tides no significant scour/filling is observed with less than 2ft (0.6m) change in bed level. The mean predictions fail to capture these frequent fluctuations, resulting in an underestimation of scour depth. However, the lower-bound values seem to capture maximum scour depths with some lag. Nevertheless, the Trask bed elevation variation is less than a meter and does not show significant scouring. 
 
 Overall, the LSTM models perform better in capturing peak values of scour and fill episodes for Alaska bridges compared to Oregon in our case studies. This can be attributed to the distinct scour process for the two Oregon bridges. Both Oregon bridges are over coastal rivers with soft slit sediments and bedforms (dunes). Sonar and Stage do not show the typical reverse correlation as in Alaskan bridge with riverine flows and consistent seasonal patterns  <cit.>. In addition, the data availability in these locations is significantly less than in Alaska (3 years of historical data for Oregon versus 15 years for Alaska). 



  §.§.§ Sequential Training on Oregon Data

The hold-out training method is sufficient for the LSTM models to learn the underlying patterns of scour/flow characteristics at coastal rivers such as Trask and Luckimaute, leading to poor generalization.  Cross-validation <cit.> could be an alternative, however it is computationally expensive. Therefore, motivated by transfer learning <cit.>, we implemented a “Sequential Training" technique as shown in Fig. <ref> to improve the forecast accuracy for Oregon bridges. For this experiment, we used the top-3 LSTM model configurations from Section  <ref>.



The major part (90%) of the data is repeatedly split into 3 Folds and 5 Folds, with a small portion (10%) set aside for the final evaluation. Each of the folds is further divided into three parts: train, validation, and test. The model is trained using the folds in a sequential manner, i.e., the model trained (with its learned weights and biases) in a fold is going to be fine-tuned using the newer train data from the next fold. There is no overlap across the training data between two adjacent folds. This data division and the transfer of “learning" between folds saves significantly on training time and expected to improve the performance of DL models.

Table <ref> presents the MAE values on Sonar, and Fig.  <ref> shows the time-series forecast over the final-test data using the 3-Fold and 5-Fold sequentially-trained LSTM models for Trask bridge. It is observed that the prediction error is reduced using the Sequential Training method.









 §.§ CNN Evaluation (Experiment 3)


Three different variations of CNN models, i.e., VCN, DCN, and FCN as described in Section <ref> are trained for bridge 230 and 539 in Alaska and Luckiamute bridge in Oregon. Stage and Sonar are considered as both the input and target features in these models. 

Table <ref> provides the tuning range for the relevant hyperparameters. 
Grid-search is used to tune the hyperparameters of CNN models. The splitting ratios (train-validation-test) are considered as 60%-30%-10% and 60%-20%-20% for the Alaska and Oregon bridges. 

Table <ref> summarizes the performance of the CNN models for scour prediction (Sonar). The FCN variant yields the best performances, as evidenced by the lower MAE scores in Table <ref>, in comparison with the VCN and DCN models for both Alaska and Oregon bridges. The best-performing CNN models achieve competitive accuracy for both Alaska and Oregon bridges, compared with LSTM. 






  §.§.§ Scour Forecast for Alaska Bridges

 Fig.  <ref> and Fig.  <ref> demonstrate the time-series prediction of Sonar and Stage by the top-5 FCN models across the test dataset. The CNN models demonstrate a reasonable performance, comparable with LSTM models, capturing the trend of scour and filing a week in advance.





The CNN model with full convolution from Equation  <ref>
shows that it outperforms other convolutional strategies in generalization and captures both Sonar and Stage trends. Particularly, DCN models fail to capture the scour and filling trends as compared to the VCN and FCN. This implies that skipping the intermediate observations by the dilated convolutions from Equation  <ref> did not result in higher accuracy. One reason behind the superior performance by FCN can be the inclusion of the edging values while making batch-to-batch transitions via the 0-padded convolution. This ensures no loss of information while learning the underlying scour patterns.





  §.§.§ Scour Forecast for Oregon Results


Fig.  <ref> illustrates the performance of top-5 FCN models over the test dataset, between April 2021 and July 2021. Similar to the LSTM models, the CNN models perform less accurately for Oregon data compared with Alaska, particularly in capturing the scour and fill trends. The error of mean Sonar prediction varies between 1.5 to 2.5ft (0.5 to 0.75m). 









 §.§ Feature Combination Analysis (Experiment 4) 


Different combinations of features are considered as input to LSTM models to evaluate their impact in scour prediction (wrapper method). The best Single-Shot LSTM configurations were trained for this purpose over two Alaska and Oregon bridges as shown in Figs. <ref> and <ref>. Note that for Oregon bridges, the discharge data was not; therefore, dC dV are not included in the impact analysis.

The first term in model configuration nomenclatures indicates the feature combination, e.g. sNsTdV, indicates {Sonar, Stage, Discharge} input feature combination (refer to Section <ref> for features notations).







These plots show that the input feature sets containing sN (Sonar) consistently outperform the feature sets without sN for all bridges. The models with sN as the only input feature shows the best performance for all bridges. The groups sNdV and sNdV show the second smallest error for all bridges, showing comparable performance as sN alone. As expected, Discharge and eVelocity have an equivalent impact on prediction performance as sNdC and sNdV models show consistently similar error ranges. Also, Stage seems to have a similar impact on prediction as Discharge and eVelocity, as observed in very close error ranges among sNsT and sNdC, and sNdV. Therefore, adding velocity features to DL models in Alaska does not seem to improve significantly scour forecasting as initially expected. This further fortifies the results reported in  <cit.> about stage time series having a high correlation with discharge time series, therefore embodying velocity characteristics required for LSTMs to learn the scour patterns.


For the Oregon bridges, the scour models do not obtain higher accuracy by including the time features denoted as y, similar to Alaska case studies in  <cit.>. Input feature sets: sN and sNsT obtain similar accuracy in predicting scour. Similar to Alaska bridges, LSTM models show poor performance when the input feature set does not contain the Sonar values.

The strong dependency among the recent and past values of the sonar readings is evidenced by the lower MAE of the scour model with Sonar as the only feature and the high autocorrelation coefficient of sonar readings. However, Stage is shown to be more impactful for scour prediction in the case of the Luckimaute bridge. This can be related to a lack of consistent periodic scour and filing patterns as opposed to Alaska bridges and Trask bridges with tidal river bed fluctuations. Nevertheless, it is evident from this experiment that flow-related times series on their own, i.e., stage, discharge and velocity, are inadequate for reliable scour prediction. 



§ CONCLUSIONS

This paper investigated various prominent DL algorithms for scour forecast based on historic sensor monitoring data, including three variants of LSTM and CNN, for case-study bridges in Alaska and Oregon. The primary focus was to address the challenges of DL models for real-time scour prediction, including performance optimization of algorithms and hyperparameter tuning in terms of both accuracy and computational efficiency, generalisation to diverse geological, geomorphological and hydraulic/flow characteristics, and impact of various combinations of sensor features relevant to scour.

The top-performing LSTM models achieved mean absolute error (MAE) ranging from 0.1m (in Alaska) to 0.5m (in Oregon). The best CNN model, the FCN variant, showed similar error ranges (0.1m to 0.75m). Both Deep Learning (DL) algorithms showed promising performance in forecasting scour for bridges in Alaska, with CNN models, particularly the FCN variant, achieving slightly lower prediction errors. However, CNN was trained with significantly lower computational costs running on the same GPU clusters.  Higher errors were observed for Oregon bridges compared to Alaska, especially in capturing peaks of scour and filling episodes. This was related to the fundamentally different scour process in Oregon bridges, such as tidal flows and the presence of coastal bed forms, and the lack of substantial historical data. In addition, Oregon bridges did not show consistent periodical and seasonal flow and scour patterns as observed in Alaskan bridges. 

To improve the accuracy of DL models for Oregon, we implemented a sequential training approach integrated with transfer learning. This training approach proved effective in improving the performance, while reducing the computational time.

For LSTM models, the single-shot model consistently outperformed the two-layer single-shot and feedback variants. Regarding the CNN, the FCN achieved lower MAE values on all bridges compared to the Vanilla Convolutional Network and Dilated Causal Network. Additionally, our feature impact analysis using the wrapper method showed adding Discharge, and Velocity time series to Stage and Sonar did not improve the performance of DL models in Alaska. An interesting observation consistent across all bridges in Oregon and Alaska was that the models without Sonar (bed elevation) time series showed significantly higher errors. In fact, models with Sonar as the only input feature obtained similar (and in some cases slightly better) scour prediction accuracy compared with benchmark models with the Sonar plus Stage feature set. For Oregon bridges, the time features did not improve the performance of DL models similar to what was observed previously for Alaska.

The random-search method with mean MAE and Bagging heuristics introduced in this study proved to be efficient alternatives to grid-search for hyperparameter tunning. The top LSTM configurations identified by the random-search with higher sample sizes (%67) resulted in finding the most optimal configurations, while saving significantly on computational cost.

In conclusion, this study highlights the potential of deep learning models, particularly LSTM and CNN variants, for accurate scour forecasting in diverse bridge locations. However, challenges remain in adapting these models to fundamentally different scour processes and limited historical data, as observed in the Oregon bridges. Future research should focus on developing more robust and adaptable models that can handle varying scour characteristics and data limitations. Additionally, integrating domain knowledge and physics-based modelling approaches with data-driven techniques could further improve the interpretability and generalization of scour prediction models. Finally, the exploration of advanced hyperparameter optimization techniques and transfer learning strategies may help reduce computational costs and improve the efficiency of model development for scour forecasting applications.



§ ACKNOWLEDGEMENT

The funding for this research was provided by The University of Melbourne and Arup Global Research. We would like to express our gratitude to Greg Lind from USGS and Kira Glover-Cutter from Oregon DOT for providing the data and insights throughout this research. We also extend our thanks to Dr Bo Wang for his help in preparing this manuscript.





