QuipuNet: convolutional neural network for single-molecule nanopore sensing
Karolis Misiunas Niklas Ermann Ulrich F. Keyser
 




karolis@misiunas.com


ufk20@cam.ac.uk

Cavendish Laboratory, University of Cambridge, UK
 




 
    Nanopore sensing is a versatile technique for the analysis of molecules on the single-molecule level. However, extracting information from data with established algorithms usually requires time-consuming checks by an experienced researcher due to inherent variability of solid-state nanopores. Here, we develop a convolutional neural network (CNN) for the fully automated extraction of information from the time-series signals obtained by nanopore sensors. In our demonstration, we use a previously published dataset on multiplexed single-molecule protein sensing <cit.>. The neural network learns to classify translocation events with greater accuracy than previously possible, while also increasing the number of analysable events by a factor of five. Our results demonstrate that deep learning can achieve significant improvements in single molecule nanopore detection with potential applications in rapid diagnostics.



QuipuNet: convolutional neural network for single-molecule nanopore sensing
    Ulrich F. Keyser
    December 30, 2023
===========================================================================






 §.§  Introduction 






Nanopores have emerged as powerful sensing devices for single molecules <cit.>, with applications in DNA sequencing <cit.>, protein detection <cit.>, the study of protein folding <cit.>, 

SNP genotyping <cit.>, 
data storage <cit.>, and DNA computing <cit.>. A typical setup consists of two liquid filled reservoirs connected by a nanopore with diameters down to a few nanometres. An external electric field drives charged molecules through the nanopore, as shown in Figure <ref>a. The passage of molecules modulates the current, producing a characteristic signal that contains information about the shape of the molecule.


The readout is a time-series current trace corresponding to the shape of the molecule, usually called an event. Detection of such events can be achieved using simple current thresholds, but the subsequent analysis of features within each identified event is often made difficult by a poor signal-to-noise ratio, varying conformation of the molecule, and non-specific interactions with the nanopore surface. For example, Figure <ref>b shows two events from a multiplexed protein sensing technique published in <cit.>. 
The authors used a DNA molecule as a carrier for a protein target. Modifications along the DNA molecule and bound targets produce secondary current drops during the translocation event, as shown in the two traces. In the first half of the structure, DNA hairpin loops at defined positions and their corresponding secondary drops were used to encode a digital barcode. This barcode uniquely identified a binding site in the other half of the DNA molecule. The presence of a target at the binding site could be inferred from a single secondary drop in the second half. This approach allows the simultaneous detection of a large number of targets, only limited by the number of distinct barcodes. The information is encoded in additional current drops during the event, much like the knots on a string used in the Inca Quipu system <cit.>. 


Analysis of the event data requires accurate detection and subsequent interpretation of secondary current drops <cit.>. However, simple peak finding algorithms often fail at reliably classifying large parts of the data. Common causes of errors are a varying peak magnitude, noise <cit.>, fluctuating velocities <cit.>, overlapping peaks, DNA knots <cit.> and folded molecules. To mitigate these effects the nanopore community has developed sophisticated algorithms <cit.>. However, they frequently require manual parameter tuning for each dataset and supervision of algorithms <cit.>. In the worst case scenario, researchers have to manually interpret the data, leading to small sample sizes, possible confirmation bias, or data analysis duration exceeding measurement time. 


In this paper, we show that deep learning is ideally suited for automating the analysis of nanopore sensing data. For our study, we use the previously mentioned multiplexed protein sensing dataset from <cit.>. The dataset contains separate control measurements for each specific barcode, without other bit permutations present in the solution. This automatically provides labelled data to train the supervised learning model. At the same time, the data is sufficiently complex to require an elaborate algorithm for the classification of events. In <cit.> a twelve step approach was used to identify the bit sequence and presence or absence of a target on each DNA construct. That method relied on more than a dozen manually adjusted parameters that were carefully optimised, but still it could only use a small fraction (∼ 20%) of events, discarding up to 80% of the difficult-to-interpret events that failed some predefined set of criteria. 
Here, we show that machine learning models are able to interpret and classify data without the need for manual tuning and the development of complex algorithms while increasing the number of usable events by a factor of five. Our implementation is open-source and available online to enable the adaptation of deep learning to other nanopore sensing problems <cit.>. 






 §.§  Methods 




We chose convolutional neural networks (CNN) as the machine learning approach because of their suitability for detecting local patterns <cit.>. A recent study showed that CNNs perform well on simulated current traces from an STM tunnel junction <cit.>. For comparison,
DNA bases can be accurately determined from current levels using recurrent neural networks <cit.>. However, our goal is fundamentally different, as we are trying to identify the pattern encoded on the DNA secondary structure from a  variable nanopore system. Therefore, we chose to use the CNN architecture. 
A typical CNN consists of two parts, as shown in Figure <ref>c. First, a series of convolutions are applied to the raw input data. Then a dense neural network learns to interpret the processed signal. 
The output is a prediction about which class a particular input belongs to. In our case, the prediction is a barcode on the DNA constructs and whether a target has bound to it. 


Before feeding the data into the neural network, we perform two preparation steps. First, the raw dataset contains erroneous detections, caused by contaminations, incomplete DNA fragments, and non-specific interactions with the pore walls. We use standard filtration methods to remove these detections <cit.>: we exclude events whose area under the current trace (electronic charge deficit) lies outside two standard deviations of the mean, as well as those with current drops larger than 3.2× the unfolded event current level. Details are available in the supplementary material <cit.>. This filtration removes up to 30% of the detections recorded with the measurement setup. 
After filtration we still observe some events with errors, such as a missing bit in the barcode structure. Therefore, perfect accuracy is unattainable using realistic datasets.   


Secondly, we want the model to identify a molecule, but not the experiment. The problem arises because nanopores vary in shape and conductivity, leading to a  correlation between events measured with the same nanopore. It is possible for the neural network to over-fit to these variations, thereby learning to identify a nanopore instead of the barcode on a molecule. To reduce such over-fitting, we normalise the events from each nanopore to have the same unfolded current level (arbitrarily set to -1). In addition, we test the model using independent experiments to reduce the chances of spurious correlations. Table <ref> shows the number of events in the training and test sets. 







As mentioned above, our predictor model is based on a machine learning technique called neural networks. The architecture of such a network specifies how the network nodes are connected and what operations are applied. In order to find a suitable architecture for nanopore data, we investigated different alternatives by educated trial and error. The model presented here is inspired by the image classification network in <cit.>, which we modified to perform 1D convolutions. Figure <ref> shows the architecture. 

We optimised the (hyper-)parameters to work well for nanopore data by trial and error. A typical procedure is to pick one hyper-parameter, such as the number of convolution layers, then increase the number and measure the resulting accuracy. If the accuracy increases we stick with the new number, but if it decreases or does not change we stick with the old number. We then pick a different hyper-parameter and repeat the procedure. To avoid over-fitting to the test, we measured the accuracy gains using a development set, which is independent from the test set and 20 times smaller than the training set. The reported numbers in Figure <ref> are the result of our optimisation. 


The input for the neural network is a current trace from a measurement event. 
The data from <cit.> produced events with an average length of 402 data points. This includes short stretches of current recording before and after the event. 
As the maximum length of the event never exceeded 700 points, 
we use a 700-element vector as the input. The shorter events are padded at the end with Gaussian noise (μ=0, σ=0.072, corresponding to average noise levels).


Each box in Figure <ref> corresponds to a so-called `hidden layer' that performs a specific task and passes on the information. Here, we give a brief description of each component; we refer interested readers to the machine learning literature for more details <cit.>. 


Convolution layers extract features with local structure, such as peaks or steps. These layers perform a discrete convolution on a segment of the input by multiplying it with a small window, called kernel, and moving along to the next segment (stepping by a single vector element). The output is large if the input features match the kernel, where its weights are learned from the training data. For example, Figure <ref>c shows the output after the first convolution layer, where the orange line corresponds to a kernel that detects peaks. Other kernels detect other features in the input data, which are often difficult to interpret, as seen by two grey lines that correspond to different kernels.
After each convolution, we apply a batch normalisation (BN) layer that normalises the data to have zero mean and unit variance <cit.>. These layers improve our network training convergence. Finally, an activation function is applied – a piecewise function called rectified linear unit (ReLU), f(x) = max(0, x). 
This non-linear function is necessary for learning non-linear relationships between features <cit.>. The activation function completes one row in the diagram, its output goes into the next convolution layer.


Roughly speaking, the deeper layers capture more abstract and complex features. We follow the common practice of increasing the number of kernels for deeper layers <cit.>: from 64 to 128, then to 256. Each step doubles the amount of information passed to the next layer. For every two convolutions, we have a `Max Pool' layer to reduce the amount of information by down-sampling spacial dimensions. A Max Pool layer splits an input vector into segments of three numbers and returns only the maximum values within the segment. This arrangement is believed to improve spacial invariance for feature extraction <cit.>.


The dropout layer reduces over-fitting by randomly switching off a fraction of nodes in the layer above. This encourages the network to learn more robust features that do not depend on a single node <cit.>. Note that the dropout is only applied during training, because we want maximum accuracy while using the algorithm. 


The second half of the network is a densely connected neural network with two hidden layers and a ReLU activation function. In a dense network, the nodes between adjacent layers are fully connected, as illustrated in Figure <ref>c. The weights for these connections are learned from the training data. 


The output layer is adjusted depending on the task. In our case, we have two outputs: the barcode and sensing region. The barcode output is a vector with 8 elements and a softmax activation function.  The softmax normalises the output vector to have a sum of one such that each element is a proxy for the probability for a different barcode. We take the maximal value to be the predicted barcode. 
For the sensing region, the output is a single number with a sigmoid activation function. This number is a proxy for the probability of having a target bound to the sensing region. Note that these are two networks that are trained separately and give independent outputs. 



The model is trained for 200 epochs on a GPU (Nvidia GeForce GTX 1080 TI). The aim of the training is to find the weights that maximise accuracy, which corresponds to minimising a loss function. For barcodes, the loss function is categorical cross-entropy, while for the sensing region it is binary cross-entropy. To minimise the loss function we use the Adam optimisation algorithm <cit.> (LR=0.001; decay=0.97; batch size of 32). 
Typical training takes 200min, while evaluation is much faster at 1600events, making QuipuNet suitable for real-time classification. 






 §.§  Results 







QuipuNet correctly identifies almost all events even with highly complex shapes, as shown in Figure <ref>. For example, the first event in column one enters the nanopore with the barcode first, while the second and third examples enter with the sensing region first. QuipuNet can interpret both directions. Columns two and three show that it learns to identify folded DNA events which occur when a nanopore captures the DNA molecule somewhere along its length. These events are particularly difficult to interpret because there are many possible outcomes and peaks tend to be less pronounced. For comparison, the method from <cit.> discarded folded events so that only the events shown in blue could be identified.




 

Table <ref> presents a quantitative comparison of accuracy. The first metric for accuracy is precision, which gives the fraction of correctly identified events out of attempted guesses. Precision can be boosted by refusing to label difficult events. On the other hand, the recall metric gives the fraction of correctly identified events out of all the events (after filtration). For example, the Bell & Keyser method <cit.> and human experts achieve high accuracy but have a low recall because events with ambiguous barcode patterns are discarded.

QuipuNet achieves a precision of 0.946 for barcodes and 0.971 for the sensing regions. This is 1.0% and 3.4% higher than the Bell & Keyser method.
A much bigger difference can be seen in the recall metric because QuipuNet classifies all the data. The recall is five times larger than the original method <cit.> for both the barcode and sensing region. These results suggest that QuipuNet accurately classifies the nanopore event data, including folded events. As a result, QuipuNet outputs 5 times more data than the previous method for the same experiments. 


To measure human expert performance, one of the authors labelled 500 randomly chosen events and compared them with the true labels (it took around one hour). Only 45% of events could be labelled reliably because of the ambiguity introduced by folds or overlapping peaks. Compared with human performance, QuipuNet is 3.3% less precise at reading the barcode and 4.4% better at reading the sensing region. In both cases, the recall metric is more than twice that of a human expert. 



To optimise for accuracy, we can discard low confidence predictions to increase the precision. Practically, it makes sense to discard events where a barcode is simply missing or otherwise impossible to identify. To achieve this, we estimate the confidence using the maximal value of the softmax output vector and then discard events with the lowest confidence. We use a `data utilised' fraction to show how much data remains after discarding low confidence predictions. 


Figure <ref>a shows the accuracy as a function of data utilised for the barcode predictions (evaluated on the test set). The accuracy increases with the amount of discarded data, suggesting that the confidence estimator correctly identifies poor predictions. The accuracy curve is significantly above manual labelling and the Bell & Keyser method, suggesting that QuipuNet outperforms both. For illustrative purposes, at 80% utilised data QuipuNet precision is 0.987, which is higher than the human performance. Figure <ref>b shows an equivalent plot for the sensing region predictions. Here, QuipuNet achieves a nearly perfect precision of 0.997 for 80%  utilised data. In both cases, discarding low confidence predictions increases the accuracy of the QuipuNet algorithm.


The predictions for the sensing region have a higher accuracy than those for the barcodes. We attribute this to two effects. First, the sensing region typically has a higher signal-to-noise ratio, i.e. larger current drops. Secondly, the barcode prediction is an intrinsically harder problem, because the algorithm must distinguish between 8 different classes, instead of two. 


Figure <ref>c shows where the errors are made for the barcode predictions. The matrix suggests that QuipuNet makes more mistakes for certain barcodes. For example, the prediction for barcode `100' has a precision of only 0.86, which can be attributed to the small training set. It only has 876 events measured by two experiments while the third experiment was used for the test set. A larger training set is expected to improve the accuracy. 


The error matrix also provides insights for designing more robust barcodes. The barcodes `000', `001', `101', `110', and `111' all have a similar amount of training data, but the symmetric barcodes have a higher accuracy. Here, symmetric barcodes are `000', `101', and `111' (`010' has smaller amount of training data). This observation suggests that using only symmetric patterns for barcodes might improve the overall accuracy.


Finally, we trained QuipuNet on a reduced training set to assess the relationship between accuracy and training set size, as shown in Figure <ref>. 
For the sensing region, we randomly picked the same number of events for bound and unbound states. For the barcode, we randomly reduced the training set size of the `011' barcode to a number specified on the x̂ axis, while the other barcodes had the same number of events as specified in Table 1. 
The resulting recall metric reaches 80% at 2000 training events, 90% at 8000 and then slowly increases to >90% for more than 8000 training events. The increase in accuracy beyond 90% appears to be asymptotic and would require even 
larger training sets.
The classification of the sensing region (blue data in Figure <ref>) reaches higher accuracies for smaller training sets as it only has two classes and signal-to-noise for protein signals is higher than for barcodes.


 




 §.§  Discussion 



We have shown that convolutional neural networks can accurately classify events from nanopore data. Our network achieves better accuracy than the previous algorithm <cit.> or manual classification, while at the same time classifying events that were impossible to interpret before. As a result, five times more data can be analysed from the same experiments. Furthermore, the machine learning approach simplifies the analysis by eliminating manual parameter tuning and algorithm development. Instead, we rely on experiments to generate the labels that are used to train the neural network.


In the supplementary material, we use QuipuNet to analyse raw data from other nanopore experiments <cit.>. In <cit.> the authors detected single-nucleotide polymorphisms from the presence of a single binding target. Their designed DNA molecules contain only the sensing region with no barcode. We successfully reproduce results from their analysis using QuipuNet. 
Despite a significantly lower signal-to-noise level for this data set we obtain accuracy of up to 72% when including folded events. 
If only the unfolded events are analysed, the accuracy is 0.91. 
This shows that QuipuNet can be readily applied to other nanopore sensing datasets. When designing a nanopore experiment, others should consider the relationship between the desired accuracy and the number of training events.


Our work suggests that deep learning is particularly suitable for nanopore sensing because the experiments can generate large amounts of training data; often with predefined labels. A similar conclusion was reached for nanopore-based DNA sequencing, where a recurrent neural network improves the precision of DNA sequencing <cit.>. Future work may address other difficult problems in the nanopore field. Specifically, peak localisation in noisy datasets <cit.> can be trained using DNA with known modification positions. Also, running QuipuNet against simulated datasets (generated classically or with generative adversarial networks) could guide the design of the DNA structures in order to maximise the information density or readout accuracy. Both are critically important for information storage on DNA and hold the promise of highly multiplexed protein sensing for medical applications.




 §.§  Acknowledgements 


We are grateful to Tautvydas Misiunas and Scott Lowe for their advice on the model optimisation. We also thank Nicholas A.W. Bell and Jinglin Kong for providing  traces from their original experiments. 
K.M. and U.F.K. acknowledge funding from an ERC consolidator grant (Designerpores 647144). N.E. acknowledges funding from the EPSRC, Cambridge Trust and Trinity Hall, Cambridge. 






naturemag










