{'fraction_non_alphanumeric': 0.02130003672420125, 'fraction_numerical': 0.009303464316317787, 'mean_word_length': 5.13824192336589, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Symbiotic Hybrid Neural Network Watchdog For Outlier Detection

Neural networks are largely black boxes. A neural network trained to classify fruit may classify a picture of a giraffe as a banana. A neural network watchdog's job is to identify such inputs, allowing a classifier to disregard such data. We investigate whether the watchdog should be separate from the neural network or symbiotically attached. We present empirical evidence that the symbiotic watchdog performs better than when the neural networks are disjoint.


Introduction
The neural network watchdog is a tool used to determine whether a classification or regression has been performed on an input that is in-distribution or outof-distribution with respect to the training data [5]. The work focused on the disjoint approach to the watchdog, where the neural network and the watchdog autoencoder are trained separately on the same data. The watchdog network is used to determined the validity of the input data, allowing for the removal of out-of-distribution classification data from the output in parallel to classification.
To build upon the application of the watchdog, we propose the use of a symbiotic neural network where the autoencoder [12,20,21] is symbiotically attached to the neural network under scrutiny. This hybrid system is capable of generating and classifying input data without the need for completely separate networks. This allows the watchdog to regenerate input data using identical input weights and bias up to the neural network's inflection point. In our analysis, this allows for more precise watchdog performance since the generation is closely coupled to the initial classification layers. Since the generator and classifier share a number of layers, the symbiotic watchdog exhibits strong performance gains in training, evaluation, and prediction when compared to a disjoint watchdog.

Background
The precise definition of a hybrid neural network is open for interpretation and is used in different contexts by different researchers. Work done by McGarry et al. [17] discusses the integration of neural networks into symbolic systems, whereas Yang et al. [23] discusses a retrieval-generation models. Hybrid neural networks have been applied in various areas, such as power load forecasts [1], [2], [10], medical analysis techniques [7], and financial applications [14,15,24]. Their flexibility allows for new and novel techniques to solving modern day problems. The phrase hybrid also extends beyond network structures, to training and evaluation techniques [11,18].

Proof of Concept
We investigate the feasibility of creating a hybrid classifier-generator network to be used with the neural network watchdog where the watchdog shares a portion of the architecture of the neural network. In this sense, the watchdog is symbiotic. Our proof of concept hybrid network is based on a 2D convolutional image classifier and 2D convolutional autoencoder, as described below.

Symbiotic Hybrid Network Structure
To demonstrate the proof of concept, a symbiotic hybrid convolutional neural network is designed to classify and regenerate MNIST digit images. As seen in Figure 1, the symbiotic network is comprised of 3 subsections of layers: The Input Layers. The input layers of the symbiotic hybrid network consist of multiple 2D convolutional layers, as well as a flatten and dense layer, as shown in Figure 3. These layers comprise the encoding portion of the hybrid network and feeds in to both the generative and classification layers.

Input Layers 2. Generative Layers 3. Classification Layers
The Generative Layers. As seen in Figure 2, the generative layers represent the decoder portion of an autoencoder. These layers are responsible for decoding the representation generated by the input layers into an image reconstruction of the input which is used by the Watchdog to determine input validity. The Classification Layers. The classification layers convert the input layer encoding into an activation output, producing a distribution corresponding to classification probabilities.
The final symbiotic hybrid neural network structure can be seen in Figure 4.

Training The Symbiotic Hybrid Network
Training and Evaluation Datasets. For this proof of concept, we will be using both the MNIST digit and MNIST Fashion datasets. Our training set is comprised of 60,000 MNIST digit images. The MNIST images are considered in-distribution. In order to evaluate the functionality of the Watchdog, we must also introduce out-of-distribution data, provided by the MNIST Fashion dataset.
Examples of the evaluation data may be seen in Figures 5 and 6.
Once the network has been trained, the networks will be evaluated using a mixed-distribution dataset, consisting of 10,000 evaluation images from each of the MNIST digit and MNIST Fashion datasets.
Biased Training. One of the considerations with training a symbiotic neural network as shown above is the impact of back-propagation bias when dealing with multiple outputs. The network structure introduced in 1 will require biased training to improve the performance. Biasing the training weights allows for highly adaptive network performance, depending on the desired outcome of the network.
To further investigate the importance of the bias, five identical symbiotic neural networks are developed with different bias weights, as well as a sixth independent classifier and autoencoder as a control. The biases for the symbiotic networks are shown in Table 1 below: Evaluating the efficacy of the symbiotic hybrid network can be performed by comparing the RMSE values of the hybrid's classifier and generator results with the RMSE values of the disjointed (independent) watchdog.

Evaluating the Generator
The generator can be evaluated by calculating the root mean squared error (RMSE) between the original and the generated images. Figure 7 shows ex- amples of an original image, as well as the symbiotic hybrid and independent autoencoder generated images.
The average RMSE values for each of the networks is shown in Table 2. These values indicate minor variations in the performance of the generative components of the symbiotic and the stand-alone autoencoders. These variations are expected, as the training weights of the symbiotic networks are adjusted by both the classifier and the generator outputs.

Evaluating the Classifier
The classifier is measured by the evaluation dataset accuracy, as well as examining the ROC curves. Results for the classification accuracy Evaluation of the networks on the MNIST dataset can be seen in Table 3.

Symbiotic vs. Independent Networks
Training and Evaluation Execution Times. In addition to evaluating the classification performance of both the independent and symbiotic networks, a comparison of training and evaluation times is performed. For this evaluation, the following training parameters are used: 60,000 MNIST digit images for training, with 10 training epochs. These measurements are performed on both GPU and  Fig. 10: Normalized ROC curves for the watchdog guarded symbiotic and independent classifiers. The RMSE threshold for these curves is set to 6.5.
CPU runtime environments using Google's Colab notebooks. Table 4 displays the training time breakdown for the networks.
Similar to the improved performance with regards to training time, the evaluation times for the networks has been measured, and improved performance is found with the symbiotic watchdog. For the evaluation parameters, all six networks are evaluated using the 20,000 digit mixed-distribution dataset.

Conclusion
A symbiotic autoencoder watchdog is developed in conjunction with a symbiotic generator/classification neural network. Creating a hybrid classification neural network, as demonstrated here, results in better training and evaluation performance, as seen in Tables 3, 4, and 5. Classification performance closely matches the performance of an independent watchdog, as can be seen in Figures 9 and  10. The choice of an RMSE threshold is ultimately determined by the specific application. Results may vary based on data and the trade off between detection and false alarms.