abstract: In this work, we explore different approaches to combine modalities for the problem of automated age-suitability rating of movie trailers. First, we introduce a new dataset containing videos of movie trailers in English downloaded from IMDB and YouTube, along with their corresponding age-suitability rating labels. Secondly, we propose a multi-modal deep learning pipeline addressing the movie trailer age suitability rating problem. This is the first attempt to combine video, audio, and speech information for this problem, and our experimental results show that multi-modal approaches significantly outperform the best mono and bimodal models in this task.
author: Mahsa Shafaei\*, Christos Smailis\*, Ioannis A. Kakadiaris, Thamar Solorio
Department of Computer Science, University of Houston, Houston, TX, USA
bibliography: anthology.bib; eacl2021.bib
title:  A Case Study of Deep Learning Based Multi-Modal Methods for Predicting the Age-Suitability Rating of Movie Trailers

# Introduction

[^1] Movie trailers can be found in abundance throughout the web using services such as video streaming platforms. However, not all types of content in trailers are suitable for every audience. Specifically, movie trailers may contain explicit, aggressive, or violent content that may be harmful to the psyche of young viewers. Previous research has documented that some of the negative effects of mass media in young viewers include aggression and anxiety , as well as increasing the risk of sexual onset and alcohol and drug consumption, unwanted pregnancies, and sexually transmitted diseases .

The Advertising Administration of the Motion Picture Association of America ("MPAA") established guidelines for manually rating the age-suitability of movie trailers . The rating of movie trailers is independent of the rating of the movie itself, as a trailer includes only a short overview of the entire movie. Due to the time consuming nature, as well as the challenges to scale the MPAA rating process, automating the task is of practical value. Moreover, automating the task poses interesting challenges to multi-modal classification systems, as the source of the objectionable content can come from any, or the combination of, these sources: language (use of bad words or discussion of adult themes), images (graphic violent scenes, nudity, drug or alcohol use), and audio (loud noises and music score denoting suspenseful content). A successful rating approach should integrate evidence provided by the multiple modalities when making the predictions.

In this paper, we study the performance of different multi-modal deep learning methods, to automatically predict the MPAA age-suitability rating of movie trailers using cues from the video, audio, and text modalities. Our goal is to show the feasibility of automating the rating task, and in particular, the relevance of multimodal solutions. We explore the use of late fusion, feature concatenation fusion and Gated Multi-modal Unit (GMU) Fusion . Since the proposed pipeline does not use any type of metadata for trailers or movies, it can be easily extended to be applied to any type of online video content. The main contributions of this work are: (i) we introduce a new task in multi-modal classification; rating videos based on the MPAA rating metric for movie trailers; (ii) we introduce the Multi-modal Movie Trailer Rating (MM-Trailer) dataset that contains movie trailers and their corresponding MPAA tags, audio files, subtitles of the trailers, and the metadata of the target movie; and (iii) we demonstrate empirically that combining the different modalities yields significant improvements over the strongest monomodal model. Our results show that both, the GMU and late fusion approaches yield promising results.

# Related Work

This work is related to four different areas, namely: (i) text classification, (ii) video classification, (iii) audio classification, and (iv) movie classification Datasets.

**Text Classification:** In the authors proposed an RNN-based architecture for detecting violence in movies on a segment level as well as the full movie level, by using the movie's script. In the authors proposed an RNN-based architecture with an attention mechanism that jointly models the genre and the emotions in movie script to predict the MPAA rating of a full movie. The main difference between our work and aforementioned papers is that they only use scripts to predict the movie ratings (violence rating and MPAA ratings), while we employ various modalities (audio, video, and text) to predict if a trailer (not the entire movie) is appropriate or not for children. It should be noted that the rating schema is different for trailers compared to movies (details in Section ), and movies are not freely available on the internet.

**Video Classification:** Early approaches, such as on video classification using Deep Learning, explored the use of several temporal fusion methods for combining information from multiple consecutive video frames using features extracted from CNN architectures. The authors in introduced an end-to-end architecture based on a combination of CNNs used for feature extraction from RGB frames. The CNN features are then forwarded to an LSTM layer that models the temporal variation of frames. A different approach is followed in , namely 3D-CNN, where authors propose the use of a CNN variant that takes into account convolutions performed into both the spatial and temporal domains of a video. An expansion of the 3D-CNN approach was proposed by , where the authors propose a two-stream 3D-CNN architecture for video classification. Again the two streams used as input RGB frame data and Optical flow images.

**Audio Classification:** In past research, several types of handcrafted feature extraction techniques have been proposed for the audio modality with the ones being the most prominently used in the literature being Mel-frequency cepstral coefficients (MFCCs). However, recently several approaches have been proposed for combining audio features such as spectrogram information with deep learning architectures to perform audio classification . Audio has been explored as a modality for classifying movie content in several works such as . However, none of these methods has focused on the problem of movie trailer age-suitability rating.

**Movie Classification Datasets:** Several movie classification datasets have been proposed in the past. In , the authors introduced MediaEval 2013 Violent Scene Detection, which provided annotations for detecting violent scenes in movies. In , the authors proposed an evaluation framework, for Violent Scenes Detection in Hollywood and YouTube videos along with a dataset (VSD96). Although these datasets are relevant to our work, they only cover the violence aspect and cannot address the problem of age-suitability rating (violence is only one of many aspects of age rating). In the authors proposed a movie dataset focusing on the task of predicting the MPAA rating of the movie. However, the aforementioned dataset only includes movie scripts and corresponding metadata but does not include movie trailers or related age-suitability tags (As we mentioned earlier, the MPAA rating scheme is different for movies and trailer). In , authors introduced Moviescope, a dataset for movie genre classification. Similarly, it does not include MPAA age-suitability rating labels for movie trailers.

# Dataset

To the best of our knowledge, there is no previous trailer dataset with age-suitability rating. Thus, we assembled the multi-modal Movie Trailer dataset (MM-Trailer)[^2] by collecting the rated trailers from the IMDB website and YouTube. Typically trailers are advertising movies soon to be released and shown in theaters before a movie starts. Rating in trailers is shown by a colored band (red, yellow, green) and a message that appears at the beginning of the trailer. The rating of the trailers adheres to the rating of the movie being shown in the theater. For instance, if the movie playing at the theater is rated as NC-17 (no one under 17 is recommended to watch this movie), the green band trailer that is advertised before this movie may not be appropriate for children even if the color is green. The yellow band is designed for trailers advertised on the internet, and it indicates that the corresponding trailer is suitable for "age-appropriate internet users" as visitors to sites are mainly adults. The last group of trailers are red band trailers; red color indicates the content is only appropriate for a "mature audience" or "restricted audience".

Since our goal is to design an automated system that is able to predict which movie trailers are not recommended for children, we define only two classes of trailers for the dataset:

1.  **Green-band trailers:** this category includes (i) trailers with the message "all audiences", and (ii) green band trailers with "appropriate audience" whose associated movie is rated as G and PG.

2.  **Red-band trailers:** all red-band trailers, these include restricted and mature audiences (not appropriate for children).

```latex
\resizebox{0.99\columnwidth}{!}{
\begin{tabular}{|c|c|c|}
\hline
   \textbf{Green Band Trailers} & \textbf{Red Band Trailers} & \textbf{Total Trailers} \\ \hline
  1,040        &403         & 1,443                \\
\hline
\end{tabular}
}
```

We also extracted separate audio files and trailer subtitles. Subtitles include narrator and actor speech. Some of the YouTube trailers include the video subtitle. For these cases, we pre-process the subtitles by removing timestamps to keep only words. For trailers that do not include a subtitle file, we use a python speech recognition tool to automatically generate the subtitle from the audio. Our dataset includes 11G of audio streams. For each trailer the audio file is a combination of background music and vocals together, so the duration of audio is the same as the duration of the trailer. The number of total words in all trailer scripts is equal to 1,478,139 (on average, there are 576 words per trailer). Note that 20,783 words of the vocabulary set are unique words.

Moreover, we provide metadata of the movies themselves, and this includes IMDB-id of the movie, title, genre, name of actors and directors, and link to the poster image. Table shows the statistics of our dataset.

# Methodology

Our goal is to predict the age-suitability rating for movie trailers following the guidelines of the Advertising Administration of MPAA for trailer rating. The problem is formulated as a binary classification task where trailers are labeled as either appropriate for all audiences (green-band trailers) or restricted audiences (red-band trailers). To achieve this goal, the Multi-modal Movie Trailer Rating (MMTR) system is proposed. Within this system, the trailers are modeled as a fusion of three modalities: subtitles, audio, and video of the trailers. We train Recurrent Neural Networks (RNNs) for subtitles and audio, and a combination of Convolutional Neural Networks (CNNs) with LSTM for video, as separate streams in order to extract a representation for each respective modality. Then, we combine all stream representations using a fusion module to take advantage of the cues coming from different modalities. Figure shows the overall design for the system architecture.

Our approach is based on independently identifying the best individual modality model and then combining information from all three monomodal models (subtitle, audio, and video) through one of the following three fusion methods: (i) Gated Multi-modal Unit (GMU) , (ii) Feature Concatenation Fusion, or (iii) Late Fusion. All modules of the system are described in the following sections.

## Text Stream

The subtitles of the trailers are a rich source of information. They can help in identifying the topic of the video content. Moreover, the presence of specific words in the dialogue can be a strong indicator for some types of sensitive content, while more subtle cues can be inferred from analyzing the entire transcript. To model the information originating from the subtitles, we feed them to the following modules:

**BERT $+$ Long Short-Term Memory (LSTM) with Attention:** We use BERT to leverage the well-known power of transformer-based word representations. The word vectors are then passed to an LSTM layer to model the sequence of the words in order to extract the semantic information of the text. Afterwards, the resulting hidden representation of the LSTM is passed to an attention mechanism to find the importance of each word in the dialogue. Even though BERT has seen a series of improvements (RoBERTa , ALBERT ), our goal in this paper is to present empirical evidence that a multi-modal approach can solve this task with acceptable performance, of less relevance is the specific contextualized embeddings used.

**Emotion Vector:** We expect to observe that strong negative emotions (fear, anger, sadness) correlate more with red band trailers. Similarly, positive emotions, such as joy, are more correlated with green band trailers. We made this assumption following research by where they found promising results for using emotions in a movie rating task.

We model emotions with the use of the DeepMoji model . This model was trained using 1.2 billion tweets with emojis to understand how language is used to express emotions. Recent work in abusive language detection shows promising results from using DeepMoji , thus it seems reasonable to expect good results in this task as well. To incorporate this model into our system, the last hidden layer representation of the pretrained model was used to transfer the text to emotional feature vectors. Finally, the emotion vector was concatenated with the output of the attention and the entire vector was passed to a fully connected layer to further fine-tune the joint representation.

## Video Stream

The video modality is a rich source of visual and temporal cues that are useful for analyzing multimedia content. Specifically, in this task, video can help for modeling the objectionable content such as a depiction of nudity or bloody scenes and suggestive elements. To this end, in order to learn spatiotemporal video features, a CNN-LSTM model based on the works by and is adopted. Each video is sub-sampled to a fixed number of frames, evenly distributed across its duration to form a visual temporal sequence. The raw RGB frames are used as input to a CNN model. This CNN model produces a feature representation for spatial information within each frame. The output of the final pooling layer of the CNN is passed to an LSTM that models temporal dependencies between frames.

## Audio Stream

The audio of the trailer can help the model to learn the genre and theme of the movie, and as a result, it is a powerful tool to distinguish red-band trailers from green-band ones. For example, *horror* and *thriller* movies (that usually include suspenseful music) are less likely to be suited for children. In addition to the music score, the emotion conveyed by the speakers' tone and pitch can provide relevant cues for rating the trailer. It should be noted that the entire audio is used in our model (the music and dialogue combined). To model the audio, the Mel Frequency Cepstral Coefficients (MFCC) are extracted from the audio stream. MFCCs are one of the most common feature representations for audio classification and speech recognition tasks . The entire audio is divided to $n$ chunks, $n \in \{10,20,50,100\}$, then the MFCC feature vector is extracted for each chuck. Moreover, by performing averaging over the MFCC vector in each chunk, a fixed-length representation for the entire audio, regardless of its duration is obtained. The vector is then passed to an LSTM module to model the MFCC variations during the entire video. Lastly, by adding an attention mechanism, the model learns the importance of each audio chunk and feeds the weighted average of LSTM hidden representation to a fully connected layer that helps the model to be fine-tuned for the task.

## Fusion

The goal of the fusion module is to learn to predict the rating of the trailer by integrating evidence from the video, audio and text modalities. We evaluate three established fusion methods in order to form a unified representation for each trailer.

**Gated Multi-modal Unit (GMU):** The GMU allows the model to learn an intermediate representation by combining the different modalities, where the gate neurons learn to decide the contribution of each modality to the intermediate representation. A great advantage of the GMU model is its ability to adjust the activation from each modality depending on the specific instance. This method is inspired by control flow in recurrent architectures. In RNN models, the recurrent units decide how much the current and previous evidence engage in building the current state. In GMUs, the activation function for building the output using different modalities is measured, in order to form a unified intermediate representation for all modalities.

The original GMU was successfully applied to a movie dataset of plot synopsis and movie posters to predict genre. In the original paper, the authors implemented a bimodal system (the equation is provided in the Appendix). We follow their formulation to extend the model to include three modalities using the straightforward approach discussed in their paper. The exact formulation is shown in Equation ; where $W_{i}$, $Y_{i}$ are learnable parameters, $x_{i}$ is the feature vector for modality $i$ and $[.,.]$ stands for concatenation.

$$\begin{gathered}
\label{eq:1}
    h_{1} = tanh(W_{1}.x_{1})\\
    h_{2} = tanh(W_{2}.x_2) \\
    h_{3} = tanh(W_{3}.x_3) \\
    z1= \sigma (Y_{1}.[x_{1}, x_{2}, x_{3}]) \\
    z2= \sigma (Y_{2}.[x_{1}, x_{2}, x_{3}]) \\
    z3= \sigma (Y_{3}.[x_{1}, x_{2}, x_{3}])\\
    h = z_{1}*h_{1}+ z_{2}*h_{2}+ z_{3}*h_{3}
\end{gathered}$$ **Feature Concatenation Fusion:** One popular fusion method is generating a joint multi-modal representation through feature concatenation where the representation vectors of each modality are concatenated, and the unified representation is passed through multiple hidden layers or used directly for the prediction.

**Late Fusion:** Another vastly used fusion method is late fusion . In late fusion, different modalities are merged in the decision level using various rules (e.g., majority voting, averaging) . Here, the average of all modality outputs is calculated and used as the final output.

Before performing either feature concatenation or GMU based fusion, information from each modality is represented with a feature vector extracted from pretrained models, acting as modality streams. Then, we transform the vectors from all modalities into a single vector using the GMU or concatenation module. Finally, we pass the fused representation to a fully connected layer, creating a vector of size two (we have two classes). The sigmoid function is then applied to the two-dimensional vector to assign a label to each trailer. For late fusion, we capture the output of each single modality model before the sigmoid function (vectors of size two) and compute the average. Lastly, we pass the single representation to a sigmoid function for the classification.

# Experiments

The goal of this section is to demonstrate that a multi-modal approach is an effective way to solve the task. We, therefore, compare the prediction performance of single modality models against all multimodal variations of the system. We also compare the performance of the multimodal variations of our proposed system (MMTR system) against each other.

As mentioned in the dataset section, the MM-Trailer dataset is imbalanced. Thus, to obtain reliable results, 5 fold cross-validation was selected as an evaluation method. In each fold, we select 10% of the train sent as the validation set to obtain the best model. It should be noted that the dataset was split using the stratified approach, so as to ensure that each set has the same proportion of examples from each class. The metric used to evaluate the performance is the weighted F1 score, averaged over all 5 folds for each experiment. Implementation details of the different models are provided in the Appendix.

## Baseline Methods

#### Most Frequent Baseline:

The first baseline is a naive approach to show that the problem is not easy to solve. In this model, we assign the most frequent class to all the instances in the validation and test sets, and we measure the F1 score by considering the ground truth label.

#### Text Baseline - Traditional Machine Learning:

For the text baseline model, we provide a traditional machine learning method with hand-crafted features. We extract unigram and bigram features from subtitles and apply term frequency-inverse document frequency (TF-IDF) as the weighting scheme. Then, the feature vectors are passed to an SVM model for classification. We chose an SVM model as it performed a well on the similar task of violence detection .

#### Text Baseline - BERT + Attention + NRC:

A popular resource to extract the emotion in the text is the NRC emotion lexicon . This dictionary maps words to eight different emotions (anger, anticipation, joy, trust, disgust, sadness, surprise, and fear) and two sentiments (positive and negative). Using this dictionary, we compute the normalized count of words per emotion over the entire subtitle and create a vector of size 10 for each trailer. We use this vector as an alternative to DeepMoji vector in the model.

#### Text Baseline - DeepMoji + fully connected layer:

To show how much emotion by itself can contribute to the prediction of rating, we only use the DeepMoji vector as the input and pass it to a fully connected layer and sigmoid classifier for the prediction.

#### Video Baseline:

Our video baseline is based on the deep 3-dimensional convolutional network (3D CNN) architecture proposed by . The 3D-CNN architecture applies 3D convolution and 3D pooling operations on video volumes instead of images. Each video is sub-sampled to an 18 evenly distributed frames that are used as input to the model. The training was performed for 50 epochs, using a 0.5 dropout rate, with a learning rate of $10^{-5}$ and a batch size of eight samples.

#### Audio Baseline:

CNNs have shown promising results for audio classification . To this end, for each full video, the log-Mel spectrogram is extracted from the audio using the LibROSA python library and then used as input to a CNN architecture. For the log-Mel spectrograms 128 Mel-spaced frequency bins were used, while for the CNN model for this baseline, Inception V3 was adopted. The CNN model was trained for 100 epochs using a batch size of 64 samples and a learning rate of $10^{-5}$. An early stopping policy was used during training to avoid over-fitting.

```latex
\begin{table*}[h!]\centering
\scalebox{0.7}{
\begin{tabular}{|c|l|l|l|}
\hline
\multicolumn{1}{|l|}{}                  & \multicolumn{1}{c|}{\textbf{Model}}                      & \textbf{Val-WF} & \textbf{Test-WF} \\ \hline
\multirow{6}{*} {\makecell{Single Modality \\ Baselines}}\ & Most Frequent Baseline                                             &60.82   &60.37    \\\cline{2-4}
                                        & Text Baseline: Traditional Machine Learning                             &76.38      &75.02   \\
                                        & Text Baseline: BERT+ Attention (T-BA)    &  84.91 &81.99 \\
                                        & Text Baseline: BERT+ Attention+ NRC & 85.45   &81.67   \\
                                        & Text Baseline: DeepMoji+ fully connected layer (DeepMoji+FC) & 73.19  &68.23   \\  \cline{2-4}
                                        & Video Baseline                                  &81.82    &75.33       \\  \cline{2-4}
                                        & Audio Baseline                                  &82.25    &72.62        \\  \Xhline{2\arrayrulewidth}

\multirow{3}{*}{\makecell{Single Modality\\Models}}\ & Audio- MFCC (A-MFCC)                              & 77.56    &73.86      \\
                                        & Text- BERT+ Attention+ DeepMoji (T-BAD)                                &86.41 &82.67*  \\
                                        & Video- CNN/LSTM (V-CNN/LSTM)                          &87.06  &79.41   \\ \hline \hline

\multirow{3}{*}{Late (Fusion using two modalities)}                    & T-BAD + A-MFCC   &87.85    &82.41    \\
                                        & T-BAD + V-CNN/LSTM                                  &87.31    &84.12 \\
                                        & A-MFCC + V-CNN/LSTM                                  &84.63   &79.68    \\  \hline

\multirow{3}{*}{Concatenation (Fusion using two modalities)}                    & T-BAD + A-MFCC   &87.15   &82.17    \\
                                        & T-BAD + V-CNN/LSTM                                  &89.25    &82.80 \\
                                        & A-MFCC + V-CNN/LSTM                                  &87.41   & 78.70  \\\hline

\multirow{3}{*}{GMU (Fusion using two modalities)}                    & T-BAD + A-MFCC   & 88.08    &83.37     \\
                                        & T-BAD + V-CNN/LSTM                                  & 88.00   &83.34  \\
                                        & A-MFCC + V-CNN/LSTM                                  &85.63   &80.35    \\ \hline  \hline

\multirow{3}{*}{Fusion using all Modalities (MMTR)}        & Late (T-BAD + A-MFCC + V-CNN/LSTM)           &89.88    &85.60   \\
                                        & Mid (Concat) (T-BAD + A-MFCC + V-CNN/LSTM)   & 89.97  &82.75   \\

% \multirow{1}{*}{MMTR}
                                        & Mid (GMU) T-BAD + A-MFCC + V-CNN/LSTM                         &\textbf{91.05}  &\textbf{86.06}*         \\ \Xhline{2\arrayrulewidth}
                                        \hline
\end{tabular}}
\caption{Evaluation of the different variants of the MMTR system and other baselines using the MM-Trailer dataset. WF stands for weighted F1 score and results are averaged over 5 folds. A `*' indicates that the difference between the two classifiers' performance is shown to be statistically significant. ~\label{results}}

\end{table*}
```

# Results

Table summarizes the results of our experiments. To examine the contribution of each modality for the rating task, we report the results for all single modality models; Audio only Model (A-MFCC), Text only Model with DeepMoji (T-BAD), and Video only Model (V-CNN/LSTM ). As expected, our experimental results confirm that by leveraging all modalities we achieve a better result. As noted in Table  the highest weighted F1 score, 86.06%, is achieved by the GMU Fusion variant of the MMTR model with all modalities. This result improves the weighted F1-score of the best single modality model (T-BAD) over 3 percentage points ($P < 0.05$ based on t-test).

We also report the result for different combinations of two modalities using all fusion methods to show the effect of engaging all modalities (T-BAD + A-MFCC, T-BAD + V-CNN/LSTM , A-MFCC +V-CNN/LSTM , T-BAD + A-MFCC + V-CNN/LSTM). Based on the results, the combination of two modalities works better than every single modality, yet not as good as the combination of all modalities.

When comparing the different fusion approaches, we can see that GMU fusion outperforms the concatenation fusion systems. We speculate that the gains from GMU come from the ability of the gated unit to dynamically adapt the contribution of each modality to the intermediate representation. Statistical significance testing using t-test, demonstrated a significant difference between GMU and feature concatenation fusion ($p-value < 0.05$). However, the test does not confirm a significant difference between late fusion and GMU. Thus, we can claim that for the trailer age-suitability problem, late fusion can generalize as good as GMU fusion.

The results for T-BAD and T-BA indicate that DeepMoji is a relevant feature for the rating task, and it helps the model to better discriminate red-band trailers from green-band ones. However, the result of DeepMoji+FC shows that the DeepMoji model is not sufficient to solve the task.

To obtain a better understanding of fusion results, we also provide other evaluation metrics using the MMTR system variant with GMU Fusion (as GMU version is the winner approach based on the result table). Based on the detailed result, most of the incorrectly predicted instances are red-band trailers. The first potential explanation behind this observation is that there are fewer instances of red-band trailers in our training set compared to green-band. As a result, it is more difficult for the model to capture all patterns in this class. The second reason may be the diversity of video content in red-band trailers. Recall that this class covers any content that is not appropriate for children. It is thus reasonable to assume that this class is more heterogeneous than the green band class. We plan to explore the possibility of a fine-grained classification of objectionable content as the next steps in this work.

# Discussion

To analyze the weaknesses and strengths of the MMTR system, we first investigate the incorrectly predicted cases using the most effective version of the system (GMU Fusion) on each fold of the data. By averaging results over all folds, about 35% of incorrectly predicted cases with the MMTR system are also incorrectly predicted by each and every modality independently, fusion is thus unlikely to help in this case. We found that in about 50% of the instances where two modalities predict the wrong rating, the MMTR system justifiably trusts the single modality that is correct. In about 93% of the cases where only one modality is wrong, the MMTR GMU Fusion variant predicts the correct label, relying on the other two modalities.

```latex
\scalebox{0.8}{
\begin{tabular}{ccccc}
\hline
\textbf{Model}              & \textbf{precision} & \textbf{recall} & \textbf{F1-score}  \\
\hline
\multicolumn{1}{l}{Green} & 87.4\%                  &95.0 \%     &91.0 \%                                     \\
\multicolumn{1}{l}{Red} & 83.6\%                   &65.0  \%                &72.8  \%                                     \\ \hline
\multicolumn{1}{l}{Macro avg} & 85.6\%                   &79.8  \%                &82.0  \%                                     \\
\multicolumn{1}{l}{Weighted avg} &86.6 \%                   &86.4  \%                & 86.0 \%                                     \\

\hline
\end{tabular}
}
```

After averaging results among all folds, we notice that the MMTR GMU Fusion variant system is not able to predict about 38 out of 294 instances per fold. We watched 40 incorrectly classified trailers (selected across all folds) to analyze why the model is not able to successfully predict the label. We introduce the following hypothesis for each of the individual modalities:

1\) *Text Modality:* One main source of errors in text modality comes from the output of the speech recognition tool. First, the free version of the tool only works on short audio files. As a result, we split the whole audio to 10-second chunks. Thus, it is possible that we miss some words if the audio is cut off in the middle of the word. Also, low-quality audio impacts the word recognition rate of the automatic speech recognition system, which in turn cause the model not to recognize the specific bad words present in the video or the other way around, generate bad words by mistake (detect "please" as "pussy"). However, in some cases, the trailer either has very little speech (less than 10 words) or there is really no sensitive content in the language used. Not surprisingly, the text modality cannot work properly. Finally, in some green band videos, we observed that the trailer subtitles have the words "gun" and "shot", thus they are predicted incorrectly by the text modality. It seems that the text model is biased against the occurrence of these words that are presumably strongly correlated with violent content.

2\) *Video Modality:* One main reason that the video modality model misses the sensitive content may relate to the video sampling rate. The inappropriate/violent scenes in these trailers disappear fast, or they may appear with a low frequency. As a result, we may miss them during sampling the frames in our model. The second potential reason is the quality of the trailers. We recognized that some of the trailers are old or are available in small files, so the frames are blurry, and even in some cases, not very clear to the human viewer. Lastly, we found out, there are some green-band trailers that still include brief sensitive content like the depiction of guns and blood, and our video modality model predicts them as red-band. These instances are mostly the R-rated movies that are sanitized for the trailer. However, the theme of the movie reflects itself in some frames. We can conclude that sometimes a single rating is not sufficient for expressing the type of the content, and as future work, we can predict a list of sensitive material in the video instead of a single label.

3\) *Audio Modality:* In some cases, the music of the trailer is not compatible with the content. For example, we encountered musical movies with a high level of violence, but with smooth jazz music. Thus, it is difficult for the audio modality to distinguish between appropriate and inappropriate content. Moreover, in audio modality (similarly to the video modality), we capture samples from the continuous stream. Hence, if the intense audio (such as a scream or a gunshot) happens in a short period, our model may miss that.

We also investigated the genre of incorrectly predicted trailers in one of the data folds. The interesting point is that, for incorrectly classified red-band trailers, 55% are categorized as "Thriller" or "Horror" movies and 30% as "Comedy" (based on IMDB metadata). We do not incorporate metadata into our model to make the model suitable for any kind of online content. This observation shows that the genre of the movie can be a potential feature for the model if we have metadata available.

# Conclusion

In this paper, we examined the role of multi-modality in addressing the novel problem of trailer rating classification. We present a deep learning system named MMTR for automating the task of movie trailer age-suitability rating. MMTR fuses information from the video, audio, and text modalities. We also introduce a new data set to support research in this area. This dataset contains movie trailer videos along with their rating and metadata. Several experiments using our proposed dataset demonstrated performance gains using the multi-modal systems (GMU and late fusion) compared to other baselines.

Beyond the practical use of a binary classification system, we are interested to move to the more challenging task of detecting the type of objectionable content and introducing model explainability and interpretability elements within the MMTR System.

[^1]: \* These authors contributed equally to this work

[^2]: We will release the dataset for the public access when anonymity of authors is not a concern


# RANDOMNESS IN ARITHMETIC AND THE DECLINE AND FALL OF REDUCTIONISM IN PURE MATHEMATICS

The Decline and Fall of Reductionism in Pure Mathematics

The decline and fall of reductionism

## IBM Research Report RC-18532 November 1992

##

## G. J. Chaitin

##

*Lecture given Thursday 22 October 1992 at a Mathematics – Computer Science Colloquium at the University of New Mexico. The lecture was videotaped; this is an edited transcript.*

## 1. Hilbert on the axiomatic method

Last month I was a speaker at a symposium on reductionism at Cambridge University where Turing did his work. I'd like to repeat the talk I gave there and explain how my work continues and extends Turing's. Two previous speakers had said bad things about David Hilbert. So I started by saying that in spite of what you might have heard in some of the previous lectures, Hilbert was not a twit!

Hilbert's idea is the culmination of two thousand years of mathematical tradition going back to Euclid's axiomatic treatment of geometry, going back to Leibniz's dream of a symbolic logic and Russell and Whitehead's monumental *Principia Mathematica.* Hilbert's dream was to once and for all clarify the methods of mathematical reasoning. Hilbert wanted to formulate a formal axiomatic system which would encompass all of mathematics. $$\begin{array}{l}
   \mbox{\bf Formal Axiomatic System} \\
   \longrightarrow \\
   \longrightarrow \\
   \longrightarrow
\end{array}$$

Hilbert emphasized a number of key properties that such a formal axiomatic system should have. It's like a computer programming language. It's a precise statement about the methods of reasoning, the postulates and the methods of inference that we accept as mathematicians. Furthermore Hilbert stipulated that the formal axiomatic system encompassing all of mathematics that he wanted to construct should be "consistent" and it should be "complete." $$\begin{array}{l}
   \mbox{\bf Formal Axiomatic System } \\
   \mbox{$\longrightarrow$ consistent} \\
   \mbox{$\longrightarrow$ complete  } \\
   \mbox{$\longrightarrow$           }
\end{array}$$

Consistent means that you shouldn't be able to prove an assertion and the contrary of the assertion. $$\begin{array}{l}
   \mbox{\bf Formal Axiomatic System } \\
   \mbox{$\longrightarrow$ consistent $A$ $\neg A$} \\
   \mbox{$\longrightarrow$ complete  } \\
   \mbox{$\longrightarrow$           }
\end{array}$$ You shouldn't be able to prove $A$ and not $A$. That would be very embarrassing.

Complete means that if you make a meaningful assertion you should be able to settle it one way or the other. It means that either $A$ or not $A$ should be a theorem, should be provable from the axioms using the rules of inference in the formal axiomatic system. $$\begin{array}{l}
   \mbox{\bf Formal Axiomatic System } \\
   \mbox{$\longrightarrow$ consistent $A$ $\neg A$} \\
   \mbox{$\longrightarrow$ complete   $A$ $\neg A$} \\
   \mbox{$\longrightarrow$           }
\end{array}$$ Consider a meaningful assertion $A$ and its contrary not $A$. Exactly one of the two should be provable if the formal axiomatic system is consistent and complete.

A formal axiomatic system is like a programming language. There's an alphabet and rules of grammar, in other words, a formal syntax. It's a kind of thing that we are familiar with now. Look back at Russell and Whitehead's three enormous volumes full of symbols and you'll feel you're looking at a large computer program in some incomprehensible programming language.

Now there's a very surprising fact. Consistent and complete means only truth and all the truth. They seem like reasonable requirements. There's a funny consequence, though, having to do with something called the decision problem. In German it's the Entscheidungsproblem. $$\begin{array}{l}
   \mbox{\bf Formal Axiomatic System } \\
   \mbox{$\longrightarrow$ consistent $A$ $\neg A$} \\
   \mbox{$\longrightarrow$ complete   $A$ $\neg A$} \\
   \mbox{$\longrightarrow$ decision problem       } \\
\end{array}$$

Hilbert ascribed a great deal of importance to the decision problem. $$\begin{array}{l}
   \mbox{\bf HILBERT                 } \\
   \mbox{\bf Formal Axiomatic System } \\
   \mbox{$\longrightarrow$ consistent $A$ $\neg A$} \\
   \mbox{$\longrightarrow$ complete   $A$ $\neg A$} \\
   \mbox{$\longrightarrow$ decision problem       } \\
\end{array}$$

Solving the decision problem for a formal axiomatic system is giving an algorithm that enables you to decide whether any given meaningful assertion is a theorem or not. A solution of the decision problem is called a decision procedure. $$\begin{array}{l}
   \mbox{\bf HILBERT                 } \\
   \mbox{\bf Formal Axiomatic System } \\
   \mbox{$\longrightarrow$ consistent $A$ $\neg A$} \\
   \mbox{$\longrightarrow$ complete   $A$ $\neg A$} \\
   \mbox{$\longrightarrow$ decision procedure     } \\
\end{array}$$

This sounds weird. The formal axiomatic system that Hilbert wanted to construct would have included all of mathematics: elementary arithmetic, calculus, algebra, everything. If there's a decision procedure, then mathematicians are out of work. This algorithm, this mechanical procedure, can check whether something is a theorem or not, can check whether it's true or not. So to require that there be a decision procedure for this formal axiomatic system sounds like you're asking for a lot.

However it's very easy to see that if it's consistent and it's complete that implies that there must be a decision procedure. Here's how you do it. You have a formal language with a finite alphabet and a grammar. And Hilbert emphasized that the whole point of a formal axiomatic system is that there must be a mechanical procedure for checking whether a purported proof is correct or not, whether it obeys the rules or not. That's the notion that mathematical truth should be objective so that everyone can agree whether a proof follows the rules or not.

So if that's the case you run through all possible proofs in size order, and look at all sequences of symbols from the alphabet one character long, two, three, four, a thousand, a thousand and one… a hundred thousand characters long. You apply the mechanical procedure which is the essence of the formal axiomatic system, to check whether each proof is valid. Most of the time, of course, it'll be nonsense, it'll be ungrammatical. But you'll eventually find every possible proof. It's like a million monkeys typing away. You'll find every possible proof, though only in principle of course. The number grows exponentially and this is something that you couldn't do in practice. You'd never get to proofs that are one page long.

But in principle you could run through all possible proofs, check which ones are valid, see what they prove, and that way you can systematically find all theorems. In other words, there is an algorithm, a mechanical procedure, for generating one by one every theorem that can be demonstrated in a formal axiomatic system. So if for every meaningful assertion within the system, either the assertion is a theorem or its contrary is a theorem, only one of them, then you get a decision procedure. To see whether an assertion is a theorem or not you just run through all possible proofs until you find the assertion coming out as a theorem or you prove the contrary assertion.

So it seems that Hilbert actually believed that he was going to solve once and for all, all mathematical problems. It sounds amazing, but apparently he did. He believed that he would be able to set down a consistent and complete formal axiomatic system for all of mathematics and from it obtain a decision procedure for all of mathematics. This is just following the formal, axiomatic tradition in mathematics.

But I'm sure he didn't think that it would be a practical decision procedure. The one I've outlined would only work in principle. It's exponentially slow, it's terribly slow! Totally impractical. But the idea was that if all mathematicians could agree whether a proof is correct and be consistent and complete, in principle that would give a decision procedure for automatically solving any mathematical problem. This was Hilbert's magnificent dream, and it was to be the culmination of Euclid and Leibniz, and Boole and Peano, and Russell and Whitehead.

Of course the only problem with this inspiring project is that it turned out to be impossible!

## 2. Gödel, Turing and Cantor's diagonal argument

Hilbert is indeed inspiring. His famous lecture in the year 1900 is a call to arms to mathematicians to solve a list of twenty-three difficult problems. As a young kid becoming a mathematician you read that list of twenty-three problems and Hilbert is saying that there is no limit to what mathematicians can do. We can solve a problem if we are clever enough and work at it long enough. He didn't believe that in principle there was any limit to what mathematics could achieve.

I think this is very inspiring. So did John von Neumann. When he was a young man he tried to carry through Hilbert's ambitious program. Because Hilbert couldn't quite get it all to work, in fact he started off just with elementary number theory, 1, 2, 3, 4, 5, …, not even with real numbers at first.

And then in 1931 to everyone's great surprise (including von Neumann's), Gödel showed that it was impossible, that it couldn't be done, as I'm sure you all know. $$\mbox{{\bf G\"odel} 1931}$$

This was the opposite of what everyone had expected. Von Neumann said it never occurred to him that Hilbert's program couldn't be carried out. Von Neumann admired Gödel enormously, and helped him to get a permanent position at the Institute for Advanced Study.

What Gödel showed was the following. Suppose that you have a formal axiomatic system dealing with elementary number theory, with 1, 2, 3, 4, 5 and addition and multiplication. And we'll assume that it's consistent, which is a minimum requirement— if you can prove false results it's really pretty bad. What Gödel showed was that if you assume that it's consistent, then you can show that it's incomplete. That was Gödel's result, and the proof is very clever and involves self-reference. Gödel was able to construct an assertion about the whole numbers that says of itself that it's unprovable. This was a tremendous shock. Gödel has to be admired for his intellectual imagination; everyone else thought that Hilbert was right.

However I think that Turing's 1936 approach is better. $$\begin{array}{l}
   \mbox{{\bf G\"odel} 1931} \\
   \mbox{{\bf Turing}  1936}
\end{array}$$ Gödel's 1931 proof is very ingenious, it's a real tour de force. I have to confess that when I was a kid trying to understand it, I could read it and follow it step by step but somehow I couldn't ever really feel that I was grasping it. Now Turing had a completely different approach.

Turing's approach I think it's fair to say is in some ways more fundamental. In fact, Turing did more than Gödel. Turing not only got as a corollary Gödel's result, he showed that there could be no decision procedure.

You see, if you assume that you have a formal axiomatic system for arithmetic and it's consistent, from Gödel you know that it can't be complete, but there still might be a decision procedure. There still might be a mechanical procedure which would enable you to decide if a given assertion is true or not. That was left open by Gödel, but Turing settled it. The fact that there cannot be a decision procedure is more fundamental and you get incompleteness as a corollary.

How did Turing do it? I want to tell you how he did it because that's the springboard for my own work. The way he did it, and I'm sure all of you have heard about it, has to do with something called the halting problem. In fact if you go back to Turing's 1936 paper you will not find the words "halting problem." But the idea is certainly there.

People also forget that Turing was talking about "computable numbers." The title of his paper is "On computable numbers, with an application to the Entscheidungsproblem." Everyone remembers that the halting problem is unsolvable and that comes from that paper, but not as many people remember that Turing was talking about computable real numbers. My work deals with computable and dramatically uncomputable real numbers. So I'd like to refresh your memory how Turing's argument goes.

Turing's argument is really what destroys Hilbert's dream, and it's a simple argument. It's just Cantor's diagonal procedure (for those of you who know what that is) applied to the computable real numbers. That's it, that's the whole idea in a nutshell, and it's enough to show that Hilbert's dream, the culmination of two thousand years of what mathematicians thought mathematics was about, is wrong. So Turing's work is tremendously deep.

What is Turing's argument? A real number, you know $3.1415926\cdots$, is a length measured with arbitrary precision, with an infinite number of digits. And a computable real number said Turing is one for which there is a computer program or algorithm for calculating the digits one by one. For example, there are programs for $\pi$, and there are algorithms for solutions of algebraic equations with integer coefficients. In fact most of the numbers that you actually find in analysis are computable. However they're the exception, if you know set theory, because the computable reals are denumerable and the reals are nondenumerable (you don't have to know what that means). That's the essence of Turing's idea.

The idea is this. You list all possible computer programs. At that time there were no computer programs, and Turing had to invent the Turing machine, which was a tremendous step forward. But now you just say, imagine writing a list with every possible computer program. $$\begin{array}{llll}
\begin{array}[t]{l}
   p_1 \\
   p_2 \\
   p_3 \\
   p_4 \\
   p_5 \\
   p_6 \\
   \vdots
\end{array}
& & &
\begin{array}[t]{l}
   \mbox{{\bf G\"odel} 1931} \\
   \mbox{{\bf Turing}  1936}
\end{array}
\end{array}$$

If you consider computer programs to be in binary, then it's natural to think of a computer program as a natural number. And next to each computer program, the first one, the second one, the third one, write out the real number that it computes if it computes a real (it may not). But if it prints out an infinite number of digits, write them out. So maybe it's 3.1415926 and here you have another and another and another: $$\begin{array}{llll}
\begin{array}[t]{l}
   p_1 \;\; 3.1415926\cdots \\
   p_2 \;\; \cdots \\
   p_3 \;\; \cdots \\
   p_4 \;\; \cdots \\
   p_5 \;\; \cdots \\
   p_6 \;\; \cdots \\
   \vdots
\end{array}
& & &
\begin{array}[t]{l}
   \mbox{{\bf G\"odel} 1931} \\
   \mbox{{\bf Turing}  1936}
\end{array}
\end{array}$$

So you make this list. Maybe some of these programs don't print out an infinite number of digits, because they're programs that halt or that have an error in them and explode. But then there'll just be a blank line in the list. $$\begin{array}{llll}
\begin{array}[t]{l}
   p_1 \;\; 3.1415926\cdots \\
   p_2 \;\; \cdots \\
   p_3 \;\; \cdots \\
   p_4 \;\; \cdots \\
   p_5 \;\;        \\
   p_6 \;\; \cdots \\
   \vdots
\end{array}
& & &
\begin{array}[t]{l}
   \mbox{{\bf G\"odel} 1931} \\
   \mbox{{\bf Turing}  1936}
\end{array}
\end{array}$$ It's not really important—let's forget about this possibility.

Following Cantor, Turing says go down the diagonal and look at the first digit of the first number, the second digit of the second, the third… $$\begin{array}{ll}
\begin{array}[t]{l}
   p_1 \;\; -\!.\underline{d_{11}}d_{12}d_{13}d_{14}d_{15}d_{16}\cdots \\
   p_2 \;\; -\!.d_{21}\underline{d_{22}}d_{23}d_{24}d_{25}d_{26}\cdots \\
   p_3 \;\; -\!.d_{31}d_{32}\underline{d_{33}}d_{34}d_{35}d_{36}\cdots \\
   p_4 \;\; -\!.d_{41}d_{42}d_{43}\underline{d_{44}}d_{45}d_{46}\cdots \\
   p_5 \;\;                                                            \\
   p_6 \;\; -\!.d_{61}d_{62}d_{63}d_{64}d_{65}\underline{d_{66}}\cdots \\
   \vdots
\end{array}
&
\begin{array}[t]{l}
   \mbox{{\bf G\"odel} 1931} \\
   \mbox{{\bf Turing}  1936}
\end{array}
\end{array}$$

Well actually it's the digits after the decimal point. So it's the first digit after the decimal point of the the first number, the second digit after the decimal point of the second, the third digit of the third number, the fourth digit of the fourth, the fifth digit of the fifth. And it doesn't matter if the fifth program doesn't put out a fifth digit, it really doesn't matter.

What you do is you change these digits. Make them different. Change every digit on the diagonal. Put these changed digits together into a new number with a decimal point in front, a new real number. That's Cantor's diagonal procedure. So you have a digit which you choose to be different from the first digit of the first number, the second digit of the second, the third of the third, and you put these together into one number. $$\begin{array}{ll}
\begin{array}[t]{l}
   p_1 \;\; -\!.\underline{d_{11}}d_{12}d_{13}d_{14}d_{15}d_{16}\cdots \\
   p_2 \;\; -\!.d_{21}\underline{d_{22}}d_{23}d_{24}d_{25}d_{26}\cdots \\
   p_3 \;\; -\!.d_{31}d_{32}\underline{d_{33}}d_{34}d_{35}d_{36}\cdots \\
   p_4 \;\; -\!.d_{41}d_{42}d_{43}\underline{d_{44}}d_{45}d_{46}\cdots \\
   p_5 \;\;                                                            \\
   p_6 \;\; -\!.d_{61}d_{62}d_{63}d_{64}d_{65}\underline{d_{66}}\cdots \\
   \vdots                                                              \\
.\neq\!d_{11}\neq\!d_{22}\neq\!d_{33}\neq\!d_{44}\neq\!d_{55}\neq\!d_{66}\cdots
\end{array}
&
\begin{array}[t]{l}
   \mbox{{\bf G\"odel} 1931} \\
   \mbox{{\bf Turing}  1936}
\end{array}
\end{array}$$

This new number cannot be in the list because of the way it was constructed. Therefore it's an uncomputable real number. How does Turing go on from here to the halting problem? Well, just ask yourself **why** can't you compute it? I've explained how to get this number and it looks like you could almost do it. To compute the $N$th digit of this number, you get the $N$th computer program (you can certainly do that) and then you start it running until it puts out an $N$th digit, and at that point you change it. Well what's the problem? That sounds easy.

The problem is, what happens if the $N$th computer program never puts out an $N$th digit, and you sit there waiting? And that's the halting problem— you cannot decide whether the $N$th computer program will ever put out an $N$th digit! This is how Turing got the unsolvability of the halting problem. Because if you could solve the halting problem, then you could decide if the $N$th computer program ever puts out an $N$th digit. And if you could do that then you could actually carry out Cantor's diagonal procedure and compute a real number which has to differ from any computable real. That's Turing's original argument.

Why does this explode Hilbert's dream? What has Turing proved? That there is no algorithm, no mechanical procedure, which will decide if the $N$th computer program ever outputs an $N$th digit. Thus there can be no algorithm which will decide if a computer program ever halts (finding the $N$ digit put out by the $N$th program is a special case). Well, what Hilbert wanted was a formal axiomatic system from which all mathematical truth should follow, only mathematical truth, and all mathematical truth. If Hilbert could do that, it would give us a mechanical procedure to decide if a computer program will ever halt. Why?

You just run through all possible proofs until you either find a proof that the program halts or you find a proof that it never halts. So if Hilbert's dream of a finite set of axioms from which all of mathematical truth should follow were possible, then by running through all possible proofs checking which ones are correct, you would be able to decide if any computer program halts. In principle you could. But you **can't** by Turing's very simple argument which is just Cantor's diagonal argument applied to the computable reals. That's how simple it is!

Gödel's proof is ingenious and difficult. Turing's argument is so fundamental, so deep, that everything seems natural and inevitable. But of course he's building on Gödel's work.

## 3. The halting probability and algorithmic randomness

The reason I talked to you about Turing and computable reals is that I'm going to use a different procedure to construct an uncomputable real, a much more uncomputable real than Turing does. $$\begin{array}{ll}
\begin{array}[t]{l}
   p_1 \;\; -\!.\underline{d_{11}}d_{12}d_{13}d_{14}d_{15}d_{16}\cdots \\
   p_2 \;\; -\!.d_{21}\underline{d_{22}}d_{23}d_{24}d_{25}d_{26}\cdots \\
   p_3 \;\; -\!.d_{31}d_{32}\underline{d_{33}}d_{34}d_{35}d_{36}\cdots \\
   p_4 \;\; -\!.d_{41}d_{42}d_{43}\underline{d_{44}}d_{45}d_{46}\cdots \\
   p_5 \;\;                                                            \\
   p_6 \;\; -\!.d_{61}d_{62}d_{63}d_{64}d_{65}\underline{d_{66}}\cdots \\
   \vdots                                                              \\
.\neq\!d_{11}\neq\!d_{22}\neq\!d_{33}\neq\!d_{44}\neq\!d_{55}\neq\!d_{66}\cdots
\end{array}
&
\begin{array}[t]{l}
   \mbox{{\bf G\"odel} 1931} \\
   \mbox{{\bf Turing}  1936} \\
   \mbox{uncomputable reals}
\end{array}
\end{array}$$ And that's how we're going to get into much worse trouble.

How do I get a much more uncomputable real? (And I'll have to tell you how uncomputable it is.) Well, not with Cantor's diagonal argument. I get this number, which I like to call $\Omega$, like this: $$\Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}$$ This is just the halting probability. It's sort of a mathematical pun. Turing's fundamental result is that the halting problem is unsolvable—there is no algorithm that'll settle the halting problem. My fundamental result is that the halting probability is algorithmically irreducible or algorithmically random.

What exactly is the halting probability? I've written down an expression for it: $$\Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}$$ Instead of looking at individual programs and asking whether they halt, you put all computer programs together in a bag. If you generate a computer program at random by tossing a coin for each bit of the program, what is the chance that the program will halt? You're thinking of programs as bit strings, and you generate each bit by an independent toss of a fair coin, so if a program is $N$ bits long, then the probability that you get that particular program is $2^{-N}$. Any program $p$ that halts contributes $2^{-|p|}$, two to the minus its size in bits, the number of bits in it, to this halting probability.

By the way there's a technical detail which is very important and didn't work in the early version of algorithmic information theory. You couldn't write this: $$\Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}$$ It would give infinity. The technical detail is that no extension of a valid program is a valid program. Then this sum $$\sum_{\mbox{$p$ halts}} 2^{-|p|}$$ turns out to be between zero and one. Otherwise it turns out to be infinity. It only took ten years until I got it right. The original 1960s version of algorithmic information theory is wrong. One of the reasons it's wrong is that you can't even define this number $$\Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}$$ In 1974 I redid algorithmic information theory with "self-delimiting" programs and then I discovered the halting probability $\Omega$.

Okay, so this is a probability between zero and one $$0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1$$ like all probabilities. The idea is you generate each bit of a program by tossing a coin and ask what is the probability that it halts. This number $\Omega$, this halting probability, is not only an uncomputable real— Turing already knew how to do that. It is uncomputable in the worst possible way. Let me give you some clues how uncomputable it is.

Well, one thing is it's algorithmically incompressible. If you want to get the first $N$ bits of $\Omega$ out of a computer program, if you want a computer program that will print out the first $N$ bits of $\Omega$ and then halt, that computer program has to be $N$ bits long. Essentially you're only printing out constants that are in the program. You cannot squeeze the first $N$ bits of $\Omega$. This $$0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1$$ is a real number, you could write it in binary. And if you want to get out the first $N$ bits from a computer program, essentially you just have to put them in. The program has to be $N$ bits long. That's irreducible algorithmic information. There is no concise description.

Now that's an abstract way of saying things. Let me give a more concrete example of how random $\Omega$ is. Émile Borel at the turn of this century was one of the founders of probability theory. $$\begin{array}{l}
   \displaystyle
   0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1
   \\
   \mbox{\bf \'Emile Borel}
\end{array}$$

> Question: Can I ask a very simple question before you get ahead?
>
> Answer: Sure.
>
> Question: I can't see why $\Omega$ should be a probability. What if the two one-bit programs both halt? I mean, what if the two one-bit programs both halt and then some other program halts. Then $\Omega$ is greater than one and not a probability.
>
> Answer: I told you no extension of a valid program is a valid program.
>
> Question: Oh right, no other programs can halt.
>
> Answer: The two one-bit programs would be all the programs there are. That's the reason this number $$0 <
>    \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
>    < 1$$ can't be defined if you think of programs in the normal way.

So here we have Émile Borel, and he talked about something he called a normal number. $$\begin{array}{l}
   \displaystyle
   0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1
   \\
   \mbox{{\bf \'Emile Borel} --- normal reals}
\end{array}$$ What is a normal real number? People have calculated $\pi$ out to a billion digits, maybe two billion. One of the reasons for doing this, besides that it's like climbing a mountain and having the world record, is the question of whether each digit occurs the same number of times. It looks like the digits 0 through 9 each occur 10% of the time in the decimal expansion of $\pi$. It looks that way, but nobody can prove it. I think the same is true for $\sqrt{2}$, although that's not as popular a number to ask this about.

Let me describe some work Borel did around the turn of the century when he was pioneering modern probability theory. Pick a real number in the unit interval, a real number with a decimal point in front, with no integer part. If you pick a real number in the unit interval, Borel showed that with probability one it's going to be "normal." Normal means that when you write it in decimal each digit will occur in the limit exactly 10% of the time, and this will also happen in any other base. For example in binary 0 and 1 will each occur in the limit exactly 50% of the time. Similarly with blocks of digits. This was called an absolutely normal real number by Borel, and he showed that with probability one if you pick a real number at random between zero and one it's going to have this property. There's only one problem. He didn't know whether $\pi$ is normal, he didn't know whether $\sqrt{2}$ is normal. In fact, he couldn't exhibit a single individual example of a normal real number.

The first example of a normal real number was discovered by a friend of Alan Turing's at Cambridge called David Champernowne, who is still alive and who's a well-known economist. Turing was impressed with him—I think he called him "Champ"—because Champ had published this in a paper as an undergraduate. This number is known as Champernowne's number. Let me show you Champernowne's number. $$\begin{array}{l}
   \displaystyle
   0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1
   \\
   \mbox{{\bf \'Emile Borel} --- normal reals}
   \\
   \mbox{\bf Champernowne}
   \\
   .01234567891011121314\cdots99100101\cdots
\end{array}$$

It goes like this. You write down a decimal point, then you write 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, then 10, 11, 12, 13, 14 until 99, then 100, 101. And you keep going in this funny way. This is called Champernowne's number and Champernowne showed that it's normal in base ten, only in base ten. Nobody knows if it's normal in other bases, I think it's still open. In base ten though, not only will the digits 0 through 9 occur exactly 10% of the time in the limit, but each possible block of two digits will occur exactly 1% of the time in the limit, each block of three digits will occur exactly .1% of the time in the limit, etc. That's called being normal in base ten. But nobody knows what happens in other bases.

The reason I'm saying all this is because it follows from the fact that the halting probability $\Omega$ is algorithmically irreducible information that this $$0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1$$ is normal in any base. That's easy to prove using ideas about coding and compressing information that go back to Shannon. So here we finally have an example of an absolutely normal number. I don't know how natural you think it is, but it is a specific real number that comes up and is normal in the most demanding sense that Borel could think of. Champernowne's number couldn't quite do that.

This number $\Omega$ is in fact random in many more senses. I would say it this way. It cannot be distinguished from the result of independent tosses of a fair coin. In fact this number $$0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1$$ shows that you have total randomness and chaos and unpredictability and lack of structure in pure mathematics! The same way that all it took for Turing to destroy Hilbert's dream was the diagonal argument, you just write down this expression $$0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1$$ and this shows that there are regions of pure mathematics where reasoning is totally useless, where you're up against an impenetrable wall. This is all it takes. It's just this halting probability.

Why do I say this? Well, let's say you want to use axioms to prove what the bits of this number $\Omega$ are. I've already told you that it's uncomputable—right?—like the number that Turing constructs using Cantor's diagonal argument. So we know there is no algorithm which will compute digit by digit or bit by bit this number $\Omega$. But let's try to prove what individual bits are using a formal axiomatic system. What happens?

The situation is very, very bad. It's like this. Suppose you have a formal axiomatic system which is $N$ bits of formal axiomatic system (I'll explain what this means more precisely later). It turns out that with a formal axiomatic system of complexity $N$, that is, $N$ bits in size, you can prove what the positions and values are of at most $N+c$ bits of $\Omega$.

Now what do I mean by formal axiomatic system $N$ bits in size? Well, remember that the essence of a formal axiomatic system is a mechanical procedure for checking whether a formal proof follows the rules or not. It's a computer program. Of course in Hilbert's days there were no computer programs, but after Turing invented Turing machines you could finally specify the notion of computer program exactly, and of course now we're very familiar with it.

So the proof-checking algorithm which is the essence of any formal axiomatic system in Hilbert's sense is a computer program, and just see how many bits long this computer program is.[^1] That's essentially how many bits it takes to specify the rules of the game, the axioms and postulates and the rules of inference. If that's $N$ bits, then you may be able to prove say that the first bit of $\Omega$ in binary is 0, that the second bit is 1, that the third bit is 0, and then there might be a gap, and you might be able to prove that the thousandth bit is 1. But you're only going to be able to settle $N$ cases if your formal axiomatic system is an $N$-bit formal axiomatic system.

Let me try to explain better what this means. It means that you can only get out as much as you put in. If you want to prove whether an individual bit in a specific place in the binary expansion of the real number $\Omega$ is a 0 or a 1, essentially the only way to prove that is to take it as a hypothesis, as an axiom, as a postulate. It's irreducible mathematical information. That's the key phrase that really gives the whole idea. $$\begin{array}{l}
   \mbox{\bf Irreducible Mathematical Information}
   \\
   \displaystyle
   0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1
   \\
   \mbox{{\bf \'Emile Borel} --- normal reals}
   \\
   \mbox{\bf Champernowne}
   \\
   .01234567891011121314\cdots99100101\cdots
\end{array}$$

Okay, so what have we got? We have a rather simple mathematical object that completely escapes us. $\Omega$'s bits have no structure. There is no pattern, there is no structure that we as mathematicians can comprehend. If you're interested in proving what individual bits of this number at specific places are, whether they're 0 or 1, reasoning is completely useless. Here mathematical reasoning is irrelevant and can get nowhere. As I said before, the only way a formal axiomatic system can get out these results is essentially just to put them in as assumptions, which means you're not using reasoning. After all, anything can be demonstrated by taking it as a postulate that you add to your set of axioms. So this is a worst possible case—this is irreducible mathematical information. Here is a case where there is no structure, there are no correlations, there is no pattern that we can perceive.

## 4. Randomness in arithmetic

Okay, what does this have to do with randomness in arithmetic? Now we're going back to Gödel—I skipped over him rather quickly, and now let's go back.

Turing says that you cannot use proofs to decide whether a program will halt. You can't always prove that a program will halt or not. That's how he destroys Hilbert's dream of a universal mathematics. I get us into more trouble by looking at a different kind of question, namely, can you prove that the fifth bit of this particular real number $$0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1$$ is a 0 or a 1, or that the eighth bit is a 0 or a 1. But these are strange-looking questions. Who had ever heard of the halting problem in 1936? These are not the kind of things that mathematicians normally worry about. We're getting into trouble, but with questions rather far removed from normal mathematics.

Even though you can't have a formal axiomatic system which can always prove whether a program halts or not, it might be good for everything else and then you could have an **amended** version of Hilbert's dream. And the same with the halting probability $\Omega$. If the halting problem looks a little bizarre, and it certainly did in 1936, well, $\Omega$ is brand new and certainly looks bizarre. Who ever heard of a halting probability? It's not the kind of thing that mathematicians normally do. So what do I care about all these incompleteness results!

Well, Gödel had already faced this problem with his assertion which is true but unprovable. It's an assertion which says of itself that it's unprovable. That kind of thing also never comes up in real mathematics. One of the key elements in Gödel's proof is that he managed to construct an **arithmetical** assertion which says of itself that it's unprovable. It was getting this self-referential assertion to be in elementary number theory which took so much cleverness.

There's been a lot of work building on Gödel's work, showing that problems involving computations are equivalent to arithmetical problems involving whole numbers. A number of names come to mind. Julia Robinson, Hilary Putnam and Martin Davis did some of the important work, and then a key result was found in 1970 by Yuri Matijasevič. He constructed a diophantine equation, which is an algebraic equation involving only whole numbers, with a lot of variables. One of the variables, $K$, is distinguished as a parameter. It's a polynomial equation with integer coefficients and all of the unknowns have to be whole numbers—that's a diophantine equation. As I said, one of the unknowns is a parameter. Matijasevič's equation has a solution for a particular value of the parameter $K$ if and only if the $K$th computer program halts.

In the year 1900 Hilbert had asked for an algorithm which will decide whether a diophantine equation, an algebraic equation involving only whole numbers, has a solution. This was Hilbert's tenth problem. It was tenth is his famous list of twenty-three problems. What Matijasevič showed in 1970 was that this is equivalent to deciding whether an arbitrary computer program halts. So Turing's halting problem is exactly as hard as Hilbert's tenth problem. It's exactly as hard to decide whether an arbitrary program will halt as to decide whether an arbitrary algebraic equation in whole numbers has a solution. Therefore there is no algorithm for doing that and Hilbert's tenth problem cannot be solved— that was Matijasevič's 1970 result.

Matijasevič has gone on working in this area. In particular there is a piece of work he did in collaboration with James Jones in 1984. I can use it to follow in Gödel's footsteps, to follow Gödel's example. You see, I've shown that there's complete randomness, no pattern, lack of structure, and that reasoning is completely useless, if you're interested in the individual bits of this number $$0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1$$

Following Gödel, let's convert this into something in elementary number theory. Because if you can get into all this trouble in elementary number theory, that's the bedrock. Elementary number theory, 1, 2, 3, 4, 5, addition and multiplication, that goes back to the ancient Greeks and it's the most solid part of all of mathematics. In set theory you're dealing with strange objects like large cardinals, but here you're not even dealing with derivatives or integrals or measure, only with whole numbers. And using the 1984 results of Jones and Matijasevič I can indeed dress up $\Omega$ arithmetically and get randomness in elementary number theory.

What I get is an exponential diophantine equation with a parameter. "Exponential diophantine equation" just means that you allow variables in the exponents. In contrast, what Matijasevič used to show that Hilbert's tenth problem is unsolvable is just a polynomial diophantine equation, which means that the exponents are always natural number constants. I have to allow $X^Y$. It's not known yet whether I actually need to do this. It might be the case that I can manage with a polynomial diophantine equation. It's an open question, I believe that it's not settled yet. But for now, what I have is an exponential diophantine equation with seventeen thousand variables. This equation is two-hundred pages long and again one variable is the parameter.

This is an equation where every constant is a whole number, a natural number, and all the variables are also natural numbers, that is, positive integers. (Actually **non-negative** integers.) One of the variables is a parameter, and you change the value of this parameter—take it to be 1, 2, 3, 4, 5. Then you ask, does the equation have a finite or infinite number of solutions? My equation is constructed so that it has a finite number of solutions if a particular individual bit of $\Omega$ is a 0, and it has an infinite number of solutions if that bit is a 1. So deciding whether my exponential diophantine equation in each individual case has a finite or infinite number of solutions is exactly the same as determining what an individual bit of this $$0 <
   \Omega = \sum_{\mbox{$p$ halts}} 2^{-|p|}
   < 1$$ halting probability is. And this is completely intractable because $\Omega$ is irreducible mathematical information.

Let me emphasize the difference between this and Matijasevič's work on Hilbert's tenth problem. Matijasevič showed that there is a polynomial diophantine equation with a parameter with the following property: You vary the parameter and ask, does the equation have a solution? That turns out to be equivalent to Turing's halting problem, and therefore escapes the power of mathematical reasoning, of formal axiomatic reasoning.

How does this differ from what I do? I use an exponential diophantine equation, which means I allow variables in the exponent. Matijasevič only allows constant exponents. The big difference is that Hilbert asked for an algorithm to decide if a diophantine equation has a solution. The question I have to ask to get randomness in elementary number theory, in the arithmetic of the natural numbers, is slightly more sophisticated. Instead of asking whether there is a solution, I ask whether there are a finite or infinite number of solutions—a more abstract question. This difference is necessary.

My two-hundred page equation is constructed so that it has a finite or infinite number of solutions depending on whether a particular bit of the halting probability is a 0 or a 1. As you vary the parameter, you get each individual bit of $\Omega$. Matijasevič's equation is constructed so that it has a solution if and only if a particular program ever halts. As you vary the parameter, you get each individual computer program.

Thus even in arithmetic you can find $\Omega$'s absolute lack of structure, $\Omega$'s randomness and irreducible mathematical information. Reasoning is completely powerless in those areas of arithmetic. My equation shows that this is so. As I said before, to get this equation I use ideas that start in Gödel's original 1931 paper. But it was Jones and Matijasevič's 1984 paper that finally gave me the tool that I needed.

So that's why I say that there is randomness in elementary number theory, in the arithmetic of the natural numbers. This is an impenetrable stone wall, it's a worst case. From Gödel we knew that we couldn't get a formal axiomatic system to be complete. We knew we were in trouble, and Turing showed us how basic it was, but $\Omega$ is an extreme case where reasoning fails completely.

I won't go into the details, but let me talk in vague information-theoretic terms. Matijasevič's equation gives you $N$ arithmetical questions with yes/no answers which turn out to be only $\log N$ bits of algorithmic information. My equation gives you $N$ arithmetical questions with yes/no answers which are irreducible, incompressible mathematical information.

## 5. Experimental mathematics

Okay, let me say a little bit in the minutes I have left about what this all means.

First of all, the connection with physics. There was a big controversy when quantum mechanics was developed, because quantum theory is nondeterministic. Einstein didn't like that. He said, "God doesn't play dice!" But as I'm sure you all know, with chaos and nonlinear dynamics we've now realized that even in classical physics we get randomness and unpredictability. My work is in the same spirit. It shows that pure mathematics, in fact even elementary number theory, the arithmetic of the natural numbers, 1, 2, 3, 4, 5, is in the same boat. We get randomness there too. So, as a newspaper headline would put it, God not only plays dice in quantum mechanics and in classical physics, but even in pure mathematics, even in elementary number theory. So if a new paradigm is emerging, randomness is at the heart of it. By the way, randomness is also at the heart of quantum field theory, as virtual particles and Feynman path integrals (sums over all histories) show very clearly. So my work fits in with a lot of work in physics, which is why I often get invited to talk at physics meetings.

However the really important question isn't physics, it's mathematics. I've heard that Gödel wrote a letter to his mother who stayed in Europe. You know, Gödel and Einstein were friends at the Institute for Advanced Study. You'd see them walking down the street together. Apparently Gödel wrote a letter to his mother saying that even though Einstein's work on physics had really had a tremendous impact on how people did physics, he was disappointed that his work had not had the same effect on mathematicians. It hadn't made a difference in how mathematicians actually carried on their everyday work. So I think that's the key question: How should you really do mathematics?

I'm claiming I have a much stronger incompleteness result. If so maybe it'll be clearer whether mathematics should be done the ordinary way. What is the ordinary way of doing mathematics? In spite of the fact that everyone knows that any finite set of axioms is incomplete, how do mathematicians actually work? Well suppose you have a conjecture that you've been thinking about for a few weeks, and you believe it because you've tested a large number of cases on a computer. Maybe it's a conjecture about the primes and for two weeks you've tried to prove it. At the end of two weeks you don't say, well obviously the reason I haven't been able to show this is because of Gödel's incompleteness theorem! Let us therefore add it as a new axiom! But if you took Gödel's incompleteness theorem very seriously this might in fact be the way to proceed. Mathematicians will laugh but physicists actually behave this way.

Look at the history of physics. You start with Newtonian physics. You cannot get Maxwell's equations from Newtonian physics. It's a new domain of experience—you need new postulates to deal with it. As for special relativity, well, special relativity is almost in Maxwell's equations. But Schrödinger's equation does not come from Newtonian physics and Maxwell's equations. It's a new domain of experience and again you need new axioms. So physicists are used to the idea that when you start experimenting at a smaller scale, or with new phenomena, you may need new principles to understand and explain what's going on.

Now in spite of incompleteness mathematicians don't behave at all like physicists do. At a subconscious level they still assume that the small number of principles, of postulates and methods of inference, that they learned early as mathematics students, are enough. In their hearts they believe that if you can't prove a result it's your own fault. That's probably a good attitude to take rather than to blame someone else, but let's look at a question like the Riemann hypothesis. A physicist would say that there is ample experimental evidence for the Riemann hypothesis and would go ahead and take it as a working assumption.

What is the Riemann hypothesis? There are many unsolved questions involving the distribution of the prime numbers that can be settled if you assume the Riemann hypothesis. Using computers people check these conjectures and they work beautifully. They're neat formulas but nobody can prove them. A lot of them follow from the Riemann hypothesis. To a physicist this would be enough: It's useful, it explains a lot of data. Of course a physicist then has to be prepared to say "Oh oh, I goofed!" because an experiment can subsequently contradict a theory. This happens very often.

In particle physics you throw up theories all the time and most of them quickly die. But mathematicians don't like to have to backpedal. But if you play it safe, the problem is that you may be losing out, and I believe you are.

I think it should be obvious where I'm leading. I believe that elementary number theory and the rest of mathematics should be pursued more in the spirit of experimental science, and that you should be willing to adopt new principles. I believe that Euclid's statement that an axiom is a self-evident truth is a big mistake. The Schrödinger equation certainly isn't a self-evident truth! And the Riemann hypothesis isn't self-evident either, but it's very useful.

So I believe that we mathematicians shouldn't ignore incompleteness. It's a safe thing to do but we're losing out on results that we could get. It would be as if physicists said, okay no Schrödinger equation, no Maxwell's equations, we stick with Newton, everything must be deduced from Newton's laws. (Maxwell even tried it. He had a mechanical model of an electromagnetic field. Fortunately they don't teach that in college!)

I proposed all this twenty years ago when I started getting these information-theoretic incompleteness results. But independently a new school on the philosophy of mathematics is emerging called the "quasi-empirical" school of thought regarding the foundations of mathematics. There's a book of Tymoczko's called *New Directions in the Philosophy of Mathematics* (Birkhäuser, Boston, 1986). It's a good collection of articles. Another place to look is *Searching for Certainty* by John Casti (Morrow, New York, 1990) which has a good chapter on mathematics. The last half of the chapter talks about this quasi-empirical view.

By the way, Lakatos, who was one of the people involved in this new movement, happened to be at Cambridge at that time. He'd left Hungary.

The main schools of mathematical philosophy at the beginning of this century were Russell and Whitehead's view that logic was the basis for everything, the formalist school of Hilbert, and an "intuitionist" constructivist school of Brouwer. Some people think that Hilbert believed that mathematics is a meaningless game played with marks of ink on paper. Not so! He just said that to be absolutely clear and precise what mathematics is all about, we have to specify the rules determining whether a proof is correct so precisely that they become mechanical. Nobody who thought that mathematics is meaningless would have been so energetic and done such important work and been such an inspiring leader.

Originally most mathematicians backed Hilbert. Even after Gödel and even more emphatically Turing showed that Hilbert's dream didn't work, in practice mathematicians carried on as before, in Hilbert's spirit. Brouwer's constructivist attitude was mostly considered a nuisance. As for Russell and Whitehead, they had a lot of problems getting all of mathematics from logic. If you get all of mathematics from set theory you discover that it's nice to define the whole numbers in terms of sets (von Neumann worked on this). But then it turns out that there's all kinds of problems with sets. You're not making the natural numbers more solid by basing them on something which is more problematical.

Now everything has gone topsy-turvy. It's gone topsy-turvy, not because of any philosophical argument, not because of Gödel's results or Turing's results or my own incompleteness results. It's gone topsy-turvy for a very simple reason—the computer!

The computer as you all know has changed the way we do everything. The computer has enormously and vastly increased mathematical experience. It's so easy to do calculations, to test many cases, to run experiments on the computer. The computer has so vastly increased mathematical experience, that in order to cope, people are forced to proceed in a more pragmatic fashion. Mathematicians are proceeding more pragmatically, more like experimental scientists do. This new tendency is often called "experimental mathematics." This phrase comes up a lot in the field of chaos, fractals and nonlinear dynamics.

It's often the case that when doing experiments on the computer, numerical experiments with equations, you see that something happens, and you conjecture a result. Of course it's nice if you can prove it. Especially if the proof is short. I'm not sure that a thousand page proof helps too much. But if it's a short proof it's certainly better than not having a proof. And if you have several proofs from different viewpoints, that's very good.

But sometimes you can't find a proof and you can't wait for someone else to find a proof, and you've got to carry on as best you can. So now mathematicians sometimes go ahead with working hypotheses on the basis of the results of computer experiments. Of course if it's physicists doing these computer experiments, then it's certainly okay; they've always relied heavily on experiments. But now even mathematicians sometimes operate in this manner. I believe that there's a new journal called the *Journal of Experimental Mathematics.* They should've put me on their editorial board, because I've been proposing this for twenty years based on my information-theoretic ideas.

So in the end it wasn't Gödel, it wasn't Turing, and it wasn't my results that are making mathematics go in an experimental mathematics direction, in a quasi-empirical direction. The reason that mathematicians are changing their working habits is the computer. I think it's an excellent joke! (It's also funny that of the three old schools of mathematical philosophy, logicist, formalist, and intuitionist, the most neglected was Brouwer, who had a constructivist attitude years before the computer gave a tremendous impulse to constructivism.)

Of course, the mere fact that everybody's doing something doesn't mean that they ought to be. The change in how people are behaving isn't because of Gödel's theorem or Turing's theorems or my theorems, it's because of the computer. But I think that the sequence of work that I've outlined does provide some theoretical justification for what everybody's doing anyway without worrying about the theoretical justification. And I think that the question of how we should actually do mathematics requires **at least** another generation of work. That's basically what I wanted to say—thank you very much!

## Bibliography

-   G. J. Chaitin, *Information-Theoretic Incompleteness,* World Scientific, 1992.

-   G. J. Chaitin, *Information, Randomness & Incompleteness,* second edition, World Scientific, 1990.

-   G. J. Chaitin, *Algorithmic Information Theory,* revised third printing, Cambridge University Press, 1990.

[^1]: Technical Note: Actually, it's best to think of the complexity of a formal axiomatic system as the size in bits of the computer program that enumerates the set of all theorems.