abstract: Student performance of virtual introductory physics class (calculus-based mechanics) is analyzed. A fully web-enhanced class was done synchronously. The analysis is done in two categories, averaging all mid-exams (or chapter exams) and cumulative final exams. It shows that students perform well with shorter timed mid-semester assessments (or chapter exams) than the longer timed assessments. Results were then compared with two survey questions to investigate the effect of the background knowledge of physics and calculus on their performance. It shows two interesting results, a) students with basic conceptual background knowledge of physics perform well, b) calculus knowledge does not correlate well with student performance. These findings can be used to rethink and redesign the assessments for calculus-based introductory physics classes. Also, the findings can be used to develop a supplemental support program to enhance student performance.
address: Department of Physical Science, Bergen Community College, Paramus, New Jersey 07652, USA
author: Neel Haldolaarachchige
date: 2024-10-03
title: Student performance analysis of virtual introductory calculus-based physics class

# Introduction

Generally, introductory level calculus-based physics classes are not offered fully online at the university level. However, due to the COVID19 effect, it was forced all the classes move to online. There are various types of online teaching formats now available, such as online synchronous (class meets virtually on scheduled date/time), online asynchronous (there are no virtual meetings), hybrid (part of the class is virtual and the other part is in-person).  All of the above-mentioned modalities follow a structured schedule for the semester. On the other hand, there are fully online classes which have a flexible schedule that follows students' phase of learning. Virtual (or online) synchronous classes, got strong attention in recent times mainly because such modality can simulate the real in-person learning experience. In synchronous classes students and the teacher meet virtually via video conference technology. Therefore, student-teacher interaction is real-time.

Introductory calculus-based physics at the university level is one of the most challenging classes for most freshman/sophomore students.  These classes are required to advance into most science and engineering majors. And also these physics classes consist of very important fundamental background knowledge about the development of scientific models to understand the physical processes of nature. Also, introductory-level physics classes are extremely important to build the analytical skills of future science/engineering students. Therefore, the design and delivery of virtual physics classes must be done carefully. The standard of such teaching modality must be at a very high level to match the students learning outcomes of the course. This can be achieved by using a learning management system (LMS), which consists of functionalities to engage students in various learning activities, and also students can track their progress in the course.

However, the concern of the virtual teaching modalities is on the assessments.  There are various functionalities (lockdown browser, plagiarism checker, online proctoring software's, *etc.*) built into learning management systems (LMS) to support assessments and minimize (or stop) possible academic integrity issues.  The problem is non of those can be used with calculus-based physics assessments because students must solve very detailed problems that include drawing diagrams, building mathematical models, and finding algebraic solutions. Therefore, it is extremely important that assessments (quizzes and exams) be proctored in real-time (or live).

Another important question is the correlation of student's background knowledge of basic physics concepts and calculus with the performance in the assessments of calculus-based physics classes.  This question becomes one of the heated discussion topics when setting the prerequisites for the introductory calculus-based physics courses at the university level. Usually, at community colleges, only pre-calculus is a prerequisite and calculus-I is a co-requisite when registering for calculus-based physics I (calculus-based mechanics). There are no prerequisites of physics at the high school level or conceptual physics at the university level and it only recommends students to complete a conceptual physics class only if one does not have a high school physics background. Most students register for calculus-based physics-I (calculus-based Mechanics) without a proper understating of the level of the class. Therefore, it is important to investigate the correlation of student's background knowledge of conceptual physics and calculus to the performance of calculus-based Mechanics (Physics I).

This study was based on the introductory calculus-based physics-I (mechanics) at the largest community college (Bergen community college) in the state of New Jersey. All the students enrolled were engineering majors. In general, calculus-based physics is considered the most challenging class for almost every engineering major student at Bergen community college. Historically, this class shows a very high dropout rate and failure rate. Here we present the details of the design and delivery of virtual calculus-based physics-I (Mechanics) and the analysis of student performance. The details of the comparison of two types of assessments are presented. Also, we investigated the correlation of class performance and the student's background knowledge of conceptual physics and calculus.

# Methods

## Use of learning management system

In-person and virtual classes are fully web-enhanced with a learning management system (LMS). There are various LMS available and we have used the most widely used two LMS (Moodle and Blackboard).  All class activities were fully done via LMS. All classwork was submitted into LMS and was graded electronically via LMS. The electronic pen was used to comment on submissions that simulate the real in-person paper submission and corrections. The details of the class design and delivery are published elsewhere and details can be found in the following references .

## Real-time (live) proctoring of assessments via video conference technology

There are few widely used video conference technologies available such as Zoom, Webex, Microsoft Teams, Google Meeting.  We used the Webex platform to deliver the virtual synchronous classes.  Most importantly class assessments (quizzes and exams) were proctored live via Webex.  Students were asked to align the webcam to show the working area. The teacher can see student behavior during the quizzes/exams. Students were asked to keep the microphone on during the quizzes/exams to make sure that they do not talk to someone else with a second electronic item. All the quizzes and exams were closed book and only equation sheet was allowed during the quizzes/exam.

## Teaching methods and learning support

All three semesters (FA19, FA20, SP21) classes were done with the same teaching methods that include short concept, theory/principle lecture followed by the detailed problem-solving. Simulations were used to visualize almost every concept/theory/principle and also simulations were used with the discussion of in-class questions. Students were trained to use simulations very in detail during the laboratory experiments of the class. New series of lab manuals were developed for simulation-based laboratory experiments and detailed step-by-step lesson video for each simulation-based experiment were provided to students.  During the fall 2020 and spring 2021 semesters students learning was supported with supplemental instructions that are similar to recitation sessions. Two sessions (each one hour) per week were done and the sessions were focused on the most challenging problems per each chapter.

## Data analysis

Microsoft Excel (MS Excel) software package was used for data analysis and all the tables and graphs were done with Excel. A survey at the beginning of the semester was used to collect student responses to understand their background knowledge of conceptual physics and calculus knowledge. The survey was given to students within the first week of the semester via LMS (Moodle or Blackboard) and responses were collated anonymously. All the students completed the survey so the response rate is 100%. Then survey results were investigated to find a correlation of student's background knowledge with class performance.

## Type of assessments and performance analysis

The investigation of student performance is done by using the results of the virtual-synchronous calculus-based introductory physics (level I, mechanics) class from two consecutive semesters (fall 2020 and spring 2021, both during COVID19). Then the results are compared to the same class done in-person (traditional) during fall 2019 (before COVID19). Analysis of assessments are done in two aspects to understand the student performance, 1) percent of students below and above 70% of the mid-semester assessments and 2) percent of students below and above 70% of cumulative final exam. Mid-semester assessments were done with a similar set of questions in all three semesters. However, the method of mid-semester assessments was changed in spring 2021. Mid-semester assessments were longer (each 2.45hrs) timed, closed-book exams and there were three mid exams (one per every 3-4 chapters) during fall 2019 (in-person) and fall 2020 (virtual-synchronous). During spring 21 shorter (each 30 min) timed, closed-book exam was done per each chapter (total of 13 chapter exams). Also, in the SP21 semester optional practice questions for each chapter exam were given via Moodle. More than 90% of students were completed weekly practice questions.

The final exam format (3.00hrs, longer timed and close-book) was the same in all three semesters. New exam questions (for both mid-exams and final exams) were written from scratch for all three semesters but the format, concepts, theory/principles were kept the same. Each exam question was searched via Google (before each assessment) to make sure that the exact question and solution does not available on the internet. Students submitted their written solutions to LMS and the submitted solutions were carefully analyzed. There were no academic integrity issues detected. All the mid-exams (or chapter exams) and final exams were proctored live via video conference. Use of a new set of questions and live proctoring are extremely important on virtual classes to minimize (or stop) any possible academic integrity issues.

First, each student's mid exams (chapter exams) total and final exam total points were normalized relative to passing mark of 70%, This makes it easier to find the total number of students who got above and below the passing mark (70%) because the normalized value is positive for students with above the passing mark (70%) and it is negative for students with below passing mark (70%). Then, the percentage of students who got above and below the passing mark (70%) were calculated.

## Research questions

This study focuses on a few different research questions as given below and the research questions are addressed in the following section,

-   Does class modality (virtual or in-person) affect student performance? To investigate this comparison of student performance is done between in-person traditional and virtual-synchronous.

-   Does the type of assessment affect student performance? To investigate this, two types of mid-semester assessments were used (FA2019 and FA2020 – 3 mid exams, each 2.45 hrs) and SP2021 – chapter exams, each 30 min).

-   Does student's background knowledge of conceptual physics affect student performance?

-   Does student's math level (completed calculus-I or not) affect student performance?

# Results and Discussion

## Analysis of student performance in mid-semester assessments and final exam

One of the most interesting and important questions is that the effect of class modality (in-person and virtual-synchronous) on student learning. We investigated this with student performance in three semesters by comparing the class averages of mid-semester assessments and the final exams between in-person vs virtual-synchronous class modalities.

Figure (*a*) shows analysis of class average percent difference (C.A.P.D) of mid exams (dark gray color) and the final exam (light gray color) between semesters. It can be observed that C.A.P.D of mid exams and final exams between FA19 (in-person) and FA20 (virtual) is negligible. On the other hand, C.A.P.D is very high between SP21 (virtual) vs FA20 (virtual) and SP21 (virtual) vs FA19 (in-person). C.A.P.D is significantly high for mid-exams and considerable for final exams. This shows that something very specific happens during SP21 and it does not correlate with the type of the learning (virtual or in-person). Because the C.A.P.D of final exams is the same between virtual two semesters (FA20 and SP21) and virtual-in-person two semesters (SP21 and FA19).

Analysis of student performance in three semesters (FA19-in-person, FA20, and SP21 - virtual synchronous) of mid-exams shows in figure *b*. Percent of students with above 70% of mid exams (the bright blue color) is about the same in FA20 (virtual) and FA19 (in-person). This confirms that the class type (virtual or in-person) does not affect student learning. However, student's percent above 70% in mid-exams is significantly higher in SP21 compared with that of FA20 and FA19. This indicates that something very specific did happen in the SP21 semester. The increase percent above the passing percentage of students in SP21 correlates with the format of mid-exams during spring 2021. In which shorter timed closed-book mid-exams (one per each chapter) were done. Both the other semesters (fall 2019 and fall 2020) mid-exams were longer timed closed book (one per each 3-4 chapters). Therefore, it is clear that the shorter mid exams are more effective to student performance.

On the other hand, the final exam analysis of three semesters (FA19-in-person, FA20, and SP21 - virtual synchronous) shows in the figure (*c*). It can be observed that the percent of students above 70% is very much similar in all three semesters. This a very good indication that the class modality (in-person or virtual synchronous) has no effect on student learning. This further confirms that the students do not perform well in longer timed closed-book exams (final exams are 3.0 hrs long and cumulative). When comparing the percent of the student with above 70% of mid-exams and final exams in the SP21 semester, significant difference can be observed. The higher percent of the student with above 70% in mid-exams during the SP21 semester confirms that students can perform well in shorter timed closed-book exams.

## Analysis of student performance with final class grades

Figure (*a* and *b*) shows the end of the semester analysis of percentage of students (P.S.) who got below 70% (including grades of W - withdrawal, F - less than 60% and D between 60% and 70%). Figure (*a*) shows total percentage of students with below 70%. SP21 (virtual-synchronous) semester shows better student performance compared to the previous two semesters (FA19 - in-person and FA20 - virtual synchronous). The percent of students difference (P.S.D) with lower than 70% between two semesters can be seen in figure (*b*). It shows that the failure percentage dropped by 20% in SP21 when compared to the previous two semesters (FA19 and FA20). There is no difference between FA19 (in-person) and FA20 (virtual-synchronous). It is clear that the student's performance enhances significantly in the SP21 semester. This shows that something specific affects student performance in the SP21 semester. As confirmed with the analysis in figure  and the reason for better student performance in SP21 is the type of mid-semester assessments.

Figure (*c* and *d*) shows the final grade analysis for all three semesters (FA19 in dark gray color, FA20 in light gray color and SP21 in orange color). There is no difference in final class grade distribution between FA19 (in-person) and FA20 (virtual) semesters. Both graphs show that the percent of students below 70% (final grades of W, F, D) is very similar to FA19 (in-person, pre-COVID) and FA20 (virtual-synchronous, during COVID). Also, it shows in FA19 and FA20 semesters it is not possible to get the bell curve (see figure *d*) for the distribution of final grades. this is due to a large percentage (about 40%) of students below 70%. However, grade distribution in SP21 (virtual-synchronous) semester shows nice bell curve in figure (*d*). Student percent below 70% (final grades of W, F, D) in SP21 drop by about 20% compared to FA20 and FA19. This is a significant improvement in student performance.

## Effect of type of assessment to student performance

The analysis of mid-semester assessments and final exams in figure  and final class grades in figure  suggests that something very specific happens in the SP21 semester. Therefore, it is very important to discuss the reasons for this development. Classes were done with the same teaching style in all three semesters. Resources used for teaching were also the same in three semesters. Final exam assessments (3.0 hrs, proctored, closed-book) were the same in three semesters. Similar student performance in final exams (see figure *c*) in three semesters confirmed that there were no issues of academic integrity in the exams in virtual classes during FA20 and SP21. The exams were live proctored via video conference technology and this confirms that the proctoring method is effective in virtual classes.

Now, it is clear that the student performance only can be impacted due to one parameter which changes during the SP21 semester. That is the mid-semester assessments. Usually, in calculus-based physics classes, there are three mid-exams (each exam 2.45 hrs, closed-book, proctored, and covers 3-4 chapters contents). That type of mid-exam was used during FA19 (in-person), and FA20 (virtual-synchronous). However, the type of mid-semester assessments was changed to chapter exams during SP21 (virtual-synchronous) which were shorter-timed (each 30min), closed-book, proctored and one short quiz per chapter. And at the end of the semester two lowest chapter quiz grades were dropped. The effect of continuous assessments through the semester can be observed in figure (*b*). The continuous mid-semester assessments affect most of the students to perform well at the end of the semester which can be observed in figure . Also, figure (*d*) shows the bell curve nature of final grade distribution in SP21 semester. Therefore, the enhancement of student performance correlates well with continuous assessment in the SP21 semester. On the other hand, When compared to all three semesters it can be confirmed that class modality (in-person or virtual) does not impact on student performance.

## Analysis of students background knowledge of conceptual physics and calculus to class performance

Another important and interesting aspect of this study is the effect of student background knowledge of conceptual physics and calculus on the performance in calculus-based physics at the university level. Figure (*a*) shows analysis of student performance in mid-exams passing percentage of students (blue color) with respect to their background knowledge of conceptual physics (green color) and calculus-I level math (gray color). This shows that student percentage with conceptual physics background knowledge from high school (students who did physics at high school) correlate well with the passing percentage of students than that of the percent of students with calculus-I level math knowledge (who started the class after completing calculus-I).

Students with conceptual physics background in SP21 (see figure *a*) are higher than the students with calculus-I level math in both FA20 and SP21 semester. This is because calculus-I is only a co-requisite for calculus-based physics-I. Therefore, most students simultaneously take both calculus-based physics I and calculus-I. It also shows that the student percent with above 70% in SP21 is higher than that of FA20 and student percent with concept physics background in SP21. The reason for this as discussed earlier is due to the effect of better student performance in shorter timed mid-semester assessments in SP21 compared with longer-timed mid-semester assessments in FA20.

It is evident that students who started with the background of conceptual physics perform well in mid-exams than of the students who started the class with background of calculus-I level math. Figure  shows that students passing percentage in mid-exams is higher than that of the final exam in the SP21 semester. It is because the mid-exam assessment method was based on the shorter chapter exams in SP21. When observing SP21 students passing percentage in mid exams it can be seen that about 8% of students did class materials without previous conceptual physics background and this is an even higher percentage compared with student percentage did well in the mid exams without calculus-I (about 25%).

Percent of students with a passing grades of the final exam in FA20 (see figure *b*) is higher than the student percent with concept physics and calculus-I. However, figure (*b*) shows that student percent with a passing grades on the final exam is very similar to the percent of students with conceptual physics backgrounds in the SP21 semester. Also, it shows that the percent of students with calculus-I is much lower than the passing percentage of the students in the final exam in the SP21 semester. Therefore, analysis in figure (*b*) confirms that student performance in the final exam correlates well with the conceptual physics background than calculus-I.

# Conclusion

Analysis of student performance in mid-semester assessments and the final exams was done and compared in three semesters. The results show that online live proctoring of assessments with video conference technology works well when considering academic integrity. The analysis also shows that students can perform well in shorter-timed assessments than of longer-timed assessments. The effect of shorter-timed assessments confirms with further analysis of final class grades. Student's background knowledge of conceptual physics and calculus-I into class performance was done. It shows that the students who started the class with a conceptual physics background did well in class assessments than the students with the calculus-I background.

The results of this study suggest that the student performance depends on the type of assessments of calculus-based physics class. It further suggests that the prerequisites of the calculus-based physics-I class must be reevaluated. Also, the results can be used to develop effective learning support mechanisms.

# Acknowledgments

NH acknowledges the Cerullo Learning Assistance Center (CLAC) that provided learning supports (supplemental instruction - SI and tutoring) to students. Supplemental instruction (SI) was financially supported by the STEMatics grant from Department of Education. NH acknowledges the academic freedom and support provided to investigate new teaching methods by the dean of STEM division Dr. Emily Vandalovsky (and former dean - Dr. P.J. Ricato) and department chair of Physical science Dr. Ara Kahyaoglu (and former chair - Dr. Lynda Box) at Bergen community college.

# Ethical Statement

This research was done according to ethical principles outlined by Institutional Review Board (IRB) at Bergen Community College. That is in accordance with the ethical policies out lined by Institute of Physics (IOP). This article does not contain any personal data, and it does not identify any individual (students and/or teachers). All students and teachers have given explicit consent to participate in a research project.

# Bibliography


abstract: We evaluated the effectiveness of using language models, that were pre-trained in one domain, as the basis for a classification model in another domain: Dutch book reviews. Pre-trained language models have opened up new possibilities for classification tasks with limited labelled data, because representation can be learned in an unsupervised fashion. In our experiments we have studied the effects of training set size (100–1600 items) on the prediction accuracy of a ULMFiT classifier, based on a language models that we pre-trained on the Dutch Wikipedia. We also compared ULMFiT to Support Vector Machines, which is traditionally considered suitable for small collections. We found that ULMFiT outperforms SVM for all training set sizes and that satisfactory results (%) can be achieved using training sets that can be manually annotated within a few hours. We deliver both our new benchmark collection of Dutch book reviews for sentiment classification as well as the pre-trained Dutch language model to the community.
author: Benjamin van der Burgh
Leiden Institute of Advanced Computer Science
Leiden University
`firstname.lastname@example.com` Suzan Verberne
Leiden Institute of Advanced Computer Science
Leiden University
`firstname.lastname@example.com`
title: The merits of Universal Language Model Fine-tuning for Small Datasets – a case with Dutch book reviews

# Introduction

Typically, results for supervised learning increase with larger training set sizes. However, many real-world text classification tasks rely on relatively small data, especially for applications in specific domains. Often, a large, *unlabelled* text collection is available to use, but *labelled* examples require human annotation. This is expensive and time-consuming. Since deep and complex neural architectures often require a large amount of labeled data, it has been difficult to significantly beat the traditional models – such as Support Vector Machines – with neural models .

In 2018, a breakthrough was reached with the use of pre-trained neural language models and transfer learning . Transfer learning no longer requires models to be trained from scratch but allows researchers and developers to reuse features from models that were trained on different, much larger text collections (e.g. Wikipedia). For this pre-training, no explicit labels are needed; instead, the models are trained to perform straightforward language modelling tasks, i.e. predicting words in the text.

Even though the models are trained on these seemingly trivial predictive tasks, transfer learning with these models is highly effective: the pre-trained language models can be fine-tuned to perform classification tasks with a relatively small amount of labelled task-specific data. Thus, pre-trained language models can alleviate the small labelled data size for domain-specific data sets.

In their 2018 paper, Howard and Ruder show the success of transfer learning with Universal Language Model Fine-tuning (ULMFiT) for six text classification tasks. They also demonstrate that the model has a relatively small loss in accuracy when reducing the number of training examples to as few as 100 .

In this paper we further address the use of ULMFiT for small training set sizes. We consider the case of data from a new domain, where we have a large amount of unlabelled data, but limited labelled data. Given the vast number of network parameters and the limited number of training instances (100 to 1600), we expect to quickly overfit on the training data if all parameters are optimized using the small labelled data, often referred to as *catastrophic forgetting*. Alternatively, we 'freeze' the parameters of the language model, which means that we fix all network parameters except for the parameters of the final layer. In doing so, we limit the ability of the model to adapt to the target domain, but in return avoid the problem of catastrophic forgetting, since the language model parameters are untouched.

In this paper, we evaluate ULMFiT with a pre-trained language model and fixed hyperparameters for the representation layers. We only tune the drop-out multiplier and learning rate for the linear layers.

For our experiments on Dutch texts, we created a new data collection consisting of Dutch-language book reviews. We fine-tune a general pre-trained Wikipedia model on the reviews collection. We then take various-sized labelled portions of the book review data to (a) investigate the effect of training set size, and (b) compare the accuracy of ULMFiT to the accuracy of Support Vector Machines (SVM).

The contributions of this paper compared to previous work are: (1) We deliver a new benchmark dataset for sentiment classification in Dutch; (2) We deliver pre-trained ULMFiT models for Dutch language; (3) We show the merit of pre-trained language models for small labeled datasets, compared to traditional classification models.

# Data

#### Data set

We released the 110k Dutch Book Reviews Dataset (110kDBRD).[^1] This dataset contains book reviews along with associated binary sentiment polarity labels. It is inspired by the Large Movie Review Dataset  and intended as a benchmark for sentiment classification in Dutch. We scraped 110 thousand book reviews from the website Hebban.[^2] These reviews each consist of a text and a score from 1 to 5, which we converted to categorical labels (1 and 2: negative; 3: neutral; 4 and 5: positive).

#### Data split

For our experiments, we split the 110k documents as follows: 20k documents are used in the classifier training and evaluation (positive and negative classes balanced). Of those 20k, we reserve 5k documents as a held-out test set that cannot be consulted during training nor language model pre-training. The remaining 15k documents are used for training the classifier.

#### Data sampling for training

For the experiments on dataset size we use the following training set sizes $m=\{ 100, 200, 400, 800, 1600\}$. Each experiment is trained 10 times to investigate model stability. These 10 subsamples are chosen randomly out of the complete 15k training set (not balanced). Note that the same test set is used for all experiments to make the results directly comparable.

# Language model training

## General-domain language model pre-training

In order to learn text representations we use the AWD-LSTM language modelling architecture originally used by ULMFiT and implemented in the Fast.ai Python library. This library also constructs a supervised dataset by randomly masking out words in a text. We use a unidirectional language model, i.e., the target word is predicted using words that precede it. Similarly to , we have chosen Wikipedia for language modelling, because it provides a large, freely available corpus of high quality. Moreover, we found that the pre-processing scripts for the English version of Wikipedia could be re-used for the Dutch.

We used a recent dump of Wikipedia and converted it to raw text, which was then split on white spaces into tokens. After that, we replaced all numbers with the same placeholder token, such that the specific value is ignored, but the fact that a number occurred can be used in the model. The 60k most frequent tokens were included in the vocabulary $V$ and the remaining out-of-vocabulary words were replaced with a special 'unknown' token. An embedding layer of size 400 was used to learn a dense token representation, followed by 3 LSTM layers with 1150 hidden units each to form the encoder.

This is followed by a classification module that maps each representation to a score $0 \leq s_t \leq 1$, for each token $t \in V$ where $\sum{s_t} = 1$ and can as such be interpreted as a probability distribution over the vocabulary. We used the reference implementation of ULMFiT in the Fast.ai Python library. The entire Wikipedia dataset was split into 92M tokens for training and 185k for both testing and validating the language model. A slanted triangular learning rate with a learning rate of $5*10^{-3}$ was used for 20 epochs.

## Target task language model fine-tuning

After training the language model on Wikipedia, we continued training on data from our target domain, i.e., the 110k Dutch Book Review Dataset. The preprocessing was done similarly to the preprocessing on Wikipedia, but the vocabulary of the previous step was reused. We used all data except for a 5k holdout set (105k reviews) to fine-tune network parameters using the same slanted triangular learning rates. However, this time we first trained the parameters of the classification module to convert the pre-trained features into predictions for the new target dataset. After that all network parameters were trained for 10 epochs.

## Target task classifier fine-tuning

The goal is to predict the sentiment polarity (positive or negative) given a review text. Therefore, the training dataset is constructed such that the dependent variable represents a sentiment polarity instead of a token from the vocabulary. The encoder of the language model is kept, such that a dense representation can be constructed given an input text, and the classification module is replaced to adjust for the new target classes.

# Experiments

## Preprocessing

We applied the default text processing implemented in the Fast.ai Python library by splitting on whitespace and padding texts within a batch to the same length. The amount of required padding characters was reduced by grouping texts of similar length together, while adding some randomness during training to avoid showing the network with the same batches in each epoch.

## Hyperparameters

We optimized hyperparameters for each training set size and for each fold using HpBandster.[^3] A one-cycle policy, as outlined in , was used, which requires a lower and upper bound for the momentum, describing its adaptive curve during a single epoch. This resulted in five optimized hyperparameters: learning rate, momentum lower and upper, dropout and batch size. In the objective function, we optimized for binary cross-entropy loss.

## Baselines

We compared our classification models to Linear Support Vector Machines (SVM) because it is a commonly used and well performing classifier for small text collections. We used the implementation of LinearSVC in scikit-learn.[^4] LinearSVC has one hyperparameter, C, which we optimized using HpBandster on the range of values from $10^{-4}$ to $10^4$ with squared hinge loss as optimization function (default for LinearSVC in scikit-learn). For feature extraction we used the CountVectorizer and TF-IDF transformer in scikit-learn. TF-IDF weights were trained on the same 105k documents on which the ULMFiT model was fine-tuned.

For comparison we also trained two models, one SVM and one ULMFiT model, with manually tuned hyperparameters on all available book reviews in the training set (15k). These models achieved 93.84% (ULMFiT) and 89.16% (SVM).

# Results

## Effect of training set size

The results of the experiments described in Section  can be found in Figure  (left). A few observations can be made from this plot. Firstly, for the ULMFiT model, the accuracy on the test set improves with each increase in the training dataset size, as can be expected.

Secondly, both models behave rather unstable for smaller training datasets, as can be seen by the large deviation from the mean and the outliers: different random subsamples give deviant results for the smaller training set sizes. Since the pre-trained model is based on data from a different domain, it can be expected that more than 100 instances are needed to accommodate for the new domain.

## Comparison to SVM

Figure  compares the prediction accuracies for ULMFiT and SVM. We had expected the SVM model to perform better for smaller training set sizes, but it is outperformed by ULMFiT for each size. Also, the ULMFiT models show smaller deviations between random subsamples than the SVM models.

We also found that the prediction accuracy of the SVM model using all 15,000 training items (89.16%) is surpassed by the ULMFiT model when using only 1600 training instances: all 10 random subsamples for ULMFiT reach an accuracy of at least 89.54% (the left purple box in Figure ). This could mean that the pre-trained model captures many of the required characteristics of Dutch such that they can be largely used without modifications.

# Conclusions

Pre-trained language models have opened up possibilities for classification tasks with limited labelled data. In our experiments we have studied the effects of training set size on the prediction accuracy of a ULMFiT classifier based on pre-trained language models for Dutch. In order to make a fair comparison, we have used state-of-the-art optimization methods to optimize the hyperparameters of each model.

Our results confirm what had been stated in , but had not been verified for Dutch or in as much detail. For this particular dataset and depending on the requirements of the model, satisfactory results might be achieved using training sets that can be manually annotated within a few hours.

Moreover, a large part of modeling effort lies in the training of a language model on an – in this case – generic corpus, which can be reused for other domains. While the prediction accuracy could be improved by optimizing all network parameters on a large dataset, we have shown that training only the weights of the final layer outperforms our SVM models by a large margin.

ULMFiT uses a relatively simple architecture that can be trained on moderately powerful GPUs. This fact, combined with the general availability of unlabeled data and the ability to share language models, suggests that these methods could be applied in domains where manual labelling has traditionally been too expensive. Further research should be conducted to compare how differences between the source and target datasets affect the prediction accuracy and whether more powerful network architectures can also be used.

[^1]: <https://benjaminvdb.github.io/110kDBRD/>, also including the scripts used to scrape the data from the review website.

[^2]: <https://www.hebban.nl>

[^3]: <https://github.com/automl/HpBandSter>

[^4]: <https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html>


abstract: Image-generating machine learning models are typically trained with loss functions based on distance in the image space. This often leads to over-smoothed results. We propose a class of loss functions, which we call deep perceptual similarity metrics (DeePSiM), that mitigate this problem. Instead of computing distances in the image space, we compute distances between image features extracted by deep neural networks. This metric better reflects perceptually similarity of images and thus leads to better results. We show three applications: autoencoder training, a modification of a variational autoencoder, and inversion of deep convolutional networks. In all cases, the generated images look sharp and resemble natural images.
bibliography: ../dosovits_new.bib

# Introduction

Recently there has been a surge of interest in training neural networks to generate images. These are being used for a wide variety of applications: unsupervised and semi-supervised learning, generative models, analysis of learned representations, analysis by synthesis, learning of 3D representations, future prediction in videos. Nevertheless, there is little work on studying loss functions which are appropriate for the image generation task. Typically used squared Euclidean distance between images often yields blurry results, see Fig.b. This is especially the case when there is inherent uncertainty in the prediction. For example, suppose we aim to reconstruct an image from its feature representation. The precise location of all details may not be preserved in the features. A loss in image space leads to averaging all likely locations of details, and hence the reconstruction looks blurry.

However, exact locations of all fine details are not important for perceptual similarity of images. But the distribution of these details plays a key role. Our main insight is that invariance to irrelevant transformations and sensitivity to local image statistics can be achieved by measuring distances in a suitable feature space. In fact, convolutional networks provide a feature representation with desirable properties. They are invariant to small smooth deformations, but sensitive to perceptually important image properties, for example sharp edges and textures.

Using a distance in feature space alone, however, does not yet yield a good loss function; see Fig. d. Since feature representations are typically contractive, many images, including non-natural ones, get mapped to the same feature vector. Hence, we must introduce a natural image prior. To this end, we build upon adversarial training as proposed by . We train a discriminator network to distinguish the output of the generator from real images. The objective of the generator is to trick the discriminator, i.e., to generate images that the discriminator cannot distinguish from real ones. This yields a natural image prior that selects from all potential generator outputs the most realistic one. A combination of similarity in an appropriate feature space with adversarial training allows to obtain the best results; see Fig. e.

We show three example applications: image compression with an autoencoder, a generative model based on a variational autoencoder, and inversion of the AlexNet convolutional network. We demonstrate that an autoencoder with DeePSiM loss can compress images while preserving information about fine structures. On the generative modeling side, we show that a version of a variational autoencoder trained with the new loss produces images with realistic image statistics. Finally, reconstructions obtained with our method from high-level activations of AlexNet are dramatically better than with existing approaches. They demonstrate that even the predicted class probabilities contain rich texture, color, and position information.

# Related work

There is a long history of neural network based models for image generation. A prominent class of probabilistic models of images are restricted Boltzmann machines  and their deep variants . Autoencoders  have been widely used for unsupervised learning and generative modeling, too. Recently, stochastic neural networks  have become popular, and deterministic networks are being used for image generation tasks . In all these models, loss is measured in the image space. By combining convolutions and un-pooling (upsampling) layers  these models can be applied to large images.

There is a large body of work on assessing the perceptual similarity of images. Some prominent examples are the visible differences predictor  , the spatio-temporal model for moving picture quality assessment  , and the perceptual distortion metric of  . The most popular perceptual image similarity metric is the structural similarity metric (SSIM) , which compares the local statistics of image patches. We are not aware of any work making use of similarity metrics for machine learning, except a recent pre-print of . They train autoencoders by directly maximizing the SSIM similarity of images. This resembles in spirit what we do, but technically is very different. While psychophysical experiments go out of scope of this paper, we believe that deep learned feature representations have better potential than shallow hand-designed SSIM.

Generative adversarial networks (GANs) have been proposed by . In theory, this training procedure can lead to a generator that perfectly models the data distribution. Practically, training GANs is difficult and often leads to oscillatory behavior, divergence, or modeling only part of the data distribution. Recently, several modifications have been proposed that make GAN training more stable. employ a multi-scale approach, gradually generating higher resolution images. make use of a convolutional-deconvolutional architecture and batch normalization.

GANs can be trained conditionally by feeding the conditioning variable to both the discriminator and the generator . Usually this conditioning variable is a one-hot encoding of the object class in the input image. Such GANs learn to generate images of objects from a given class. Recently  used GANs for predicting future frames in videos by conditioning on previous frames. Our approach looks similar to a conditional GAN. However, in a GAN there is no loss directly comparing the generated image to some ground truth. We found that the feature loss introduced in the present paper is essential to train on complicated tasks such as feature inversion.

Most related is concurrent work of . The general idea is the same — to measure the similarity not in the image space, but rather in a feature space. They also use adversarial training to improve the realism of the generated images. However, only apply this approach to a variational autoencoder trained on images of faces, and measure the similarity between features extracted from the discriminator. Our approach is much more general, we apply it to various natural images, and we demonstrate three different applications.

# Model

Suppose we are given a supervised learning task and a training set of input-target pairs $\{\mathbf{x}_i,\, \mathbf{y}_i\}$, $\mathbf{x}_i \in \mathbb{R}^I$, $\mathbf{y}_i \in \mathbb{R}^{W \times H \times C}$ . Inputs and outputs can be arbitrary vectors. In this work, we focus on targets that are images with an arbitrary number of channels.

The aim is to learn the parameters $\mathbf{\theta}$ of a differentiable generator function $G_\mathbf{\theta}(\cdot) \colon \mathbb{R}^I\to \mathbb{R}^{W \times H \times C}$ that optimally approximates the input-target dependency according to a loss function $\mathcal{L}(G_\mathbf{\theta}(\mathbf{x}), \mathbf{y})$. Typical choices are squared Euclidean (SE ) loss $\mathcal{L}_2 (G_\mathbf{\theta}(\mathbf{x}), \mathbf{y}) = || G_\mathbf{\theta}(\mathbf{x}) - \mathbf{y}||_2^2$ or $\ell_1$ loss $\mathcal{L}_1 (G_\mathbf{\theta}(\mathbf{x}),\mathbf{y}) = ||G_\mathbf{\theta}(\mathbf{x})-\mathbf{y}||_1$. As we demonstrate in this paper, these losses are suboptimal for some image generation tasks.

We propose a new class of losses, which we call DeePSiM. These go beyond simple distances in image space and can capture complex and perceptually important properties of images. These losses are weighted sums of three terms: feature loss $\mathcal{L}_{feat}$, adversarial loss $\mathcal{L}_{adv}$, and pixel space loss $\mathcal{L}_{img}$: $$\mathcal{L}= \lambda_{feat}\, \mathcal{L}_{feat}+ \lambda_{adv}\, \mathcal{L}_{adv}+ \lambda_{img}\, \mathcal{L}_{img}.$$ They correspond to a network architecture, an overview of which is shown in Fig. . The architecture consists of three convolutional networks: the generator $G$ that implements the generator function, the discriminator $D_\varphi$ that discriminates generated images from natural images, and the comparator $C$ that computes features from images.

**Loss in feature space.** Given a differentiable comparator $C\colon \mathbb{R}^{W \times H \times C}\to \mathbb{R}^F$, we define $$\mathcal{L}_{feat}= \sum\limits_{i} || C(G_\mathbf{\theta}(\mathbf{x}_i)) - C(\mathbf{y}_i) ||_2^2.$$ $C$ may be fixed or may be trained; for example, it can be a part of the generator or the discriminator.

$\mathcal{L}_{feat}$ alone does not provide a good loss for training. It is known  that optimizing just for similarity in the feature space typically leads to high-frequency artifacts. This is because for each natural image there are many non-natural images mapped to the same feature vector [^1]. Therefore, a natural image prior is necessary to constrain the generated images to the manifold of natural images.

**Adversarial loss.** Instead of manually designing a prior, as in , we learn it with an approach similar to Generative Adversarial Networks (GANs) of . Namely, we introduce a discriminator $D_\varphi$ which aims to discriminate the generated images from real ones, and which is trained concurrently with the generator $G_\mathbf{\theta}$. The generator is trained to "trick" the discriminator network into classifying the generated images as real. Formally, the parameters $\varphi$ of the discriminator are trained by minimizing $$\label{eq:discrloss_discr}
 \mathcal{L}_{discr} = - \sum\limits_{i} \log (D_\varphi(\mathbf{y}_i)) + \log (1 - D_\varphi(G_\mathbf{\theta}(\mathbf{x}_i))),$$ and the generator is trained to minimize $$\label{eq:discrloss_gen}
 \mathcal{L}_{adv}= - \sum\limits_{i} \log D_\varphi(G_\mathbf{\theta}(\mathbf{x}_i)).$$

**Loss in image space.** Adversarial training is known to be unstable and sensitive to hyperparameters. We found that adding a loss in the image space $$\mathcal{L}_{img}= \sum\limits_{i} || G_\mathbf{\theta}(\mathbf{x}_i) - \mathbf{y}_i ||_2^2.$$ stabilizes training.

## Architectures

**Generators.** We used several different generators in experiments. They are task-specific, so we describe these in corresponding sections below. All tested generators make use of up-convolutional ('deconvolutional') layers, as in . An up-convolutional layer consists of up-sampling and a subsequent convolution. In this paper we always up-sample by a factor of $2$ and a 'bed of nails' upsampling.

In all networks we use leaky ReLU nonlinearities, that is, $LReLU(x) = \max(x,0) + \alpha \min(x,0)$. We used $\alpha = 0.3$ in our experiments. All generators have linear output layers.

**Comparators.** We experimented with four comparators:

1\. AlexNet   is a network with $5$ convolutional and $2$ fully connected layers trained on image classification.

2\. The network of has the same architecture as AlexNet, but is trained using videos with triplet loss, which enforces frames of one video to be close in the feature space and frames from different videos to be far apart. We refer to this network as VideoNet.

3\. AlexNet with random weights.

4\. Exemplar-CNN   is a network with $3$ convolutional layers and $1$ fully connected layer trained on a surrogate task of discriminating between different image patches.

The exact layers used for comparison are specified in the experiments sections.

**Discriminator.** The architecture of the discriminator was nearly the same in all experiments. The version used for the autoencoder experiments is shown in Table . The discriminator must ensure the local statistics of images to be natural. Therefore after five convolutional layers with occasional stride we perform global average pooling. The result is processed by two fully connected layers, followed by a $2$-way softmax. We perform $50 \%$ dropout after the global average pooling layer and the first fully connected layer.

There are two modifications to this basic architecture. First, when dealing with large ImageNet  images we increase the stride in the first layer from $2$ to $4$. Second, when training networks to invert AlexNet, we additionally feed the features to the discriminator. We process them with two fully connected layers with $1024$ and $512$ units, respectively. Then we concatenate the result with the output of global average pooling.

| Type   | conv | conv | conv  | conv  | conv  | pool  |  fc   | fc  |
|:-------|:----:|:----:|:-----:|:-----:|:-----:|:-----:|:-----:|:---:|
| InSize | $64$ | $29$ | $25$  | $12$  | $10$  |  $4$  |  $-$  | $-$ |
| OutCh  | $32$ | $64$ | $128$ | $256$ | $256$ | $256$ | $512$ | $2$ |
| Kernel | $7$  | $5$  |  $3$  |  $3$  |  $3$  |  $4$  |  $-$  | $-$ |
| Stride | $2$  | $1$  |  $2$  |  $1$  |  $2$  |  $4$  |  $-$  | $-$ |

Discriminator architecture.

## Training details

We modified the *caffe*  framework to train the networks. For optimization we used Adam  with momentum $\beta_1=0.9$, $\beta_2=0.999$ and initial learning rate $0.0002$. To prevent the discriminator from overfitting during adversarial training we temporarily stopped updating it if the ratio of $\mathcal{L}_{discr}$ and $\mathcal{L}_{adv}$ was below a certain threshold ($0.1$ in most experiments). We used batch size $64$ in all experiments. We trained for $500,000$-$1,000,000$ mini-batch iterations.

# Experiments

We started with a simple proof-of-concept experiment showing how DeePSiM can be applied to training autoencoders. Then we used the proposed loss function within the variational autoencoder (VAE) framework. Finally, we applied the method to invert the representation learned by AlexNet and analyzed some properties of the method.

In quantitative comparisons we report normalized Euclidean error $||a-b||_2 / N$. The normalization coefficient $N$ is the average of Euclidean distances between all pairs of different samples from the test set. Therefore, the error of $100 \%$ means that the algorithm performs the same as randomly drawing a sample from the test set.

## Autoencoder

Here the target of the generator coincides with its input (that is, $\mathbf{y}= \mathbf{x}$), and the task of the generator is to encode the input to a compressed hidden representation and then decode back the image. The architecture is shown in Table . All layers are convolutional or up-convolutional. The hidden representation is an $8$-channel feature map $8$ times smaller than the input image. We trained on the STL-10  unlabeled dataset which contains $100,000$ images $96 \times 96$ pixels. To prevent overfitting we augmented the data by cropping random $64 \times 64$ patches during training.

We experimented with four loss functions: SE and $\ell_1$ in the image space, as well as DeePSiM with AlexNet conv3 or Exemplar-CNN conv3 as comparator.

Qualitative results are shown in Fig. , quantitative results in Table . While underperforming in terms of Euclidean loss, our approach can preserve more texture details, resulting in naturally looking non-blurry reconstructions. Interestingly, AlexNet as comparator tends to corrupt fine details (petals of the flower, sails of the ship), perhaps because it has stride of $4$ in the first layer. Exemplar-CNN as comparator does not preserve the exact color because it is explicitly trained to be invariant to color changes. We believe that with carefully selected or specifically trained comparators yet better results can be obtained.

We stress that lower Euclidean error does not mean better reconstruction. For example, imagine a black-and-white striped "zebra" pattern. A monotonous gray image will have twice smaller Euclidean error than the same pattern shifted by one stripe width.

|  |  |  |  |  |  |  |  |  |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| InSize | $64$ | $32$ | $32$ | $16$ | $16$ | $8$ | $8$ | $8$ |
| OutCh | $32$ | $32$ | $64$ | $64$ | $128$ | $128$ | $64$ | $8$ |
| Kernel | $5$ | $3$ | $5$ | $3$ | $3$ | $3$ | $3$ | $3$ |
| Stride | $\downarrow \!2$ | $1$ | $\downarrow \!2$ | $1$ | $\downarrow \!2$ | $1$ | $1$ | $1$ |

Autoencoder architecture. **Top:** encoder, **bottom:** decoder. All layers are convolutional or 'up-convolutional'.

|        |      |       |                |      |                |      |                |      |
|:-------|:----:|:-----:|:--------------:|:----:|:--------------:|:----:|:--------------:|:----:|
| InSize | $8$  |  $8$  |      $8$       | $16$ |      $16$      | $32$ |      $32$      | $64$ |
| OutCh  | $64$ | $128$ |      $64$      | $64$ |      $32$      | $32$ |      $16$      | $3$  |
| Kernel | $3$  |  $3$  |      $4$       | $3$  |      $4$       | $3$  |      $4$       | $3$  |
| Stride | $1$  |  $1$  | $\uparrow \!2$ | $1$  | $\uparrow \!2$ | $1$  | $\uparrow \!2$ | $1$  |

Autoencoder architecture. **Top:** encoder, **bottom:** decoder. All layers are convolutional or 'up-convolutional'.

| SE loss | $\ell_1$ loss | Our-ExCNN | Our-AlexNet |
|:-------:|:-------------:|:---------:|:-----------:|
| $15.3$  |    $15.7$     |  $19.8$   |   $21.5$    |

Normalized Euclidean reconstruction error (in %) of autoencoders trained with different loss functions.

|    SE loss     | $\ell_1$ loss  |   Our-ExCNN    |  Our-AlexNet   |
|:--------------:|:--------------:|:--------------:|:--------------:|
| $34.6 \pm 0.6$ | $35.7 \pm 0.4$ | $50.1 \pm 0.5$ | $52.3 \pm 0.6$ |

Classification accuracy (in %) on STL with autoencoder features learned with different loss functions.

**Classification.** Reconstruction-based models are commonly used for unsupervised feature learning. We checked if our loss functions lead to learning more meaningful representations than usual $\ell_1$ and SE losses. To this end, we trained linear SVMs on the $8$-channel hidden representations extracted by autoencoders trained with different losses. We are just interested in relative performance and, thus, do not compare to the state of the art. We trained on $10$ folds of the STL-10 training set and tested on the test set.

The results are shown in Table . As expected, the features learned with DeePSiM perform significantly better, indicating that they contain more semantically meaningful information. This suggests that other losses than standard $\ell_1$ and SE may be useful for unsupervised learning. Note that the Exemplar-CNN comparator is trained in an unsupervised way.

## Variational autoencoder

A standard VAE consists of an encoder $Enc$ and a decoder $Dec$. The encoder maps an input sample $x$ to a distribution over latent variables $z\sim Enc(x) = q(z|x)$. $Dec$ maps from this latent space to a distribution over images $\tilde{x}\sim Dec(z) = p(x|z)$. The loss function is $$\label{eq:vae}
\sum\limits_i -\mathbb{E}_{q(z|x_i)}\, \log p(x_i|z) + D_{KL}(q(z| x_i) || p(z)),$$ where $p(z)$ is a prior distribution of latent variables and $D_{KL}$ is the Kullback-Leibler divergence. The first term in Eq.  is a reconstruction error. If we assume that the decoder predicts a Gaussian distribution at each pixel, then it reduces to squared Euclidean error in the image space. The second term pulls the distribution of latent variables towards the prior. Both $q(z|x)$ and $p(z)$ are commonly assumed to be Gaussian, in which case the $KL$ divergence can be computed analytically. Please refer to  for details.

We use the proposed loss instead of the first term in Eq. . This is similar to , but the comparator does not have to be a part of the discriminator. Technically, there is little difference from training an autoencoder. First, instead of predicting a single latent vector $z$ we predict two vectors $\mathbf{\mu}$ and $\mathbf{\sigma}$ and sample $z = \mathbf{\mu}+ \mathbf{\sigma}\odot \mathbf{\varepsilon}$, where $\mathbf{\varepsilon}$ is standard Gaussian (zero mean, unit variance) and $\odot$ is element-wise multiplication. Second, we add the KL divergence term to the loss: $$\mathcal{L}_{KL}= \frac{1}{2}\sum\limits_i \left(||\mathbf{\mu}_i||_2^2   + ||\mathbf{\sigma}_i||^2_2 - \langle \log \mathbf{\sigma}_i^2, \, \textbf{1} \rangle \right) .$$ We manually set the weighting of the KL term relative to the rest of the loss. Proper probabilistic derivation is non-straightforward, and we leave it for future research.

We trained on $227 \times 227$ pixel crops of $256 \times 256$ pixel ILSVRC-2012 images. The encoder architecture is the same as AlexNet up to layer fc6, and the decoder architecture is shown in Table . We initialized the encoder with AlexNet weights, however, this is not necessary, as shown in the appendix. We sampled from the model by sampling the latent variables from a standard Gaussian $z = \mathbf{\varepsilon}$ and generating images from that with the decoder.

Samples generated with the usual SE loss, as well as three different comparators (AlexNet conv5, AlexNet fc6, VideoNet conv5) are shown in Fig. . While Euclidean loss leads to very blurry samples, our method yields images with realistic statistics. Interestingly, the samples trained with the VideoNet comparator look qualitatively similar to the ones with AlexNet, showing that supervised training may not be necessary to yield a good comparator. More results are shown in the appendix.

| Type   |   fc   |   fc   |   fc   | reshape |     uconv      | conv  |
|:-------|:------:|:------:|:------:|:-------:|:--------------:|:-----:|
| InSize |  $-$   |  $-$   |  $-$   |   $1$   |      $4$       |  $8$  |
| OutCh  | $4096$ | $4096$ | $4096$ |  $256$  |     $256$      | $512$ |
| Kernel |  $-$   |  $-$   |  $-$   |   $-$   |      $4$       |  $3$  |
| Stride |  $-$   |  $-$   |  $-$   |   $-$   | $\uparrow \!2$ |  $1$  |

Generator architecture for inverting layer fc6 of AlexNet.

| Type | uconv | conv | uconv | conv | uconv | uconv | uconv |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| InSize | $8$ | $16$ | $16$ | $32$ | $32$ | $64$ | $128$ |
| OutCh | $256$ | $256$ | $128$ | $128$ | $64$ | $32$ | $3$ |
| Kernel | $4$ | $3$ | $4$ | $3$ | $4$ | $4$ | $4$ |
| Stride | $\uparrow \!2$ | $1$ | $\uparrow \!2$ | $1$ | $\uparrow \!2$ | $\uparrow \!2$ | $\uparrow \!2$ |

Generator architecture for inverting layer fc6 of AlexNet.

```latex
\begin{figure*}
\begin{center}
\setlength{\tabcolsep}{0.05cm}
\renewcommand{\arraystretch}{1}
  \begin{tabular}{cc}
  Image &
  \raisebox{-.5\height}{\includegraphics[width=0.92\linewidth]{ICML_orig.jpg}} \\
  \textsc{conv}5 &
  \raisebox{-.5\height}{\includegraphics[width=0.92\linewidth]{ICML_gan_conv5.jpg}} \\
  \textsc{fc}6 &
  \raisebox{-.5\height}{\includegraphics[width=0.92\linewidth]{ICML_gan_fc6.jpg}} \\
  \textsc{fc}7 &
  \raisebox{-.5\height}{\includegraphics[width=0.92\linewidth]{ICML_gan_fc7.jpg}} \\
  \textsc{fc}8 &
  \raisebox{-.5\height}{\includegraphics[width=0.92\linewidth]{ICML_gan_fc8.jpg}} \\
   \end{tabular}
\end{center}
   \caption{Representative reconstructions from higher layers of AlexNet.
   General characteristics of images are preserved very well.
   In some cases (simple objects, landscapes) reconstructions are nearly perfect even from \textsc{fc}8.
   In the leftmost column the network generates dog images from \textsc{fc}7 and \textsc{fc}8.
   }
\label{fig:AlexNet_recons}
\end{figure*}
```

## Inverting AlexNet

Analysis of learned representations is an important but largely unsolved problem. One approach is to invert the representation. This may give insights into which information is preserved in the representation and what are its invariance properties. However, inverting a non-trivial feature representation $\Phi$, such as the one learned by a large convolutional network, is a difficult ill-posed problem.

Our proposed approach inverts the AlexNet convolutional network very successfully. Surprisingly rich information about the image is preserved in deep layers of the network and even in the predicted class probabilities. While being an interesting result in itself, this also shows how DeePSiM is an excellent loss function when dealing with very difficult image restoration tasks.

Suppose we are given a feature representation $\Phi$, which we aim to invert, and an image $\mathbf{I}$. There are two inverse mappings: $\Phi^{-1}_R$ such that $\Phi(\Phi^{-1}_R(\mathbf{\phi})) \approx \mathbf{\phi}$, and $\Phi^{-1}_L$ such that $\Phi^{-1}_L (\Phi(\mathbf{I})) \approx \mathbf{I}$. Recently two approaches to inversion have been proposed, which correspond to these two variants of the inverse.

, as well as and  , apply gradient-based optimization to find an image $\widetilde{\mathbf{I}}$ which minimizes the loss $$||\Phi(\mathbf{I}) - \Phi(\widetilde{\mathbf{I}})||_2^2 + P(\widetilde{\mathbf{I}}),$$ where $P$ is a simple natural image prior, such as total variation (TV) regularizer. This method produces images which are roughly natural and have features similar to the input features, corresponding to $\Phi^{-1}_R$. However, the prior is limited, so reconstructions from fully connected layers of AlexNet do not look much like natural images.

train up-convolutional networks on a large training set of natural images to perform the inversion task. They use SE distance in the image space as loss function, which leads to approximating $\Phi^{-1}_L$. The networks learn to reconstruct the color and rough positions of objects well, but produce over-smoothed results because they average all potential reconstructions.

Our method can be seen as combining the best of both worlds. Loss in the feature space helps preserve perceptually important image features. Adversarial training keeps reconstructions realistic. Note that similar to and unlike , our method does not require the feature representation being inverted to be differentiable.

**Technical details.** The generator in this setup takes the features extracted by AlexNet and generates an image from them, that is, $\mathbf{x}= \Phi(\mathbf{I}),\, \mathbf{y}= \mathbf{I}$. In general we followed  in designing the generators. The only modification is that we inserted more convolutional layers, giving the network more capacity. We reconstruct from outputs of layers conv5 –fc8. In each layer we also include processing steps following the layer, that is, pooling and non-linearities. So for example conv5 means pooled features (pool5), and fc6 means rectified values (relu6).

Architecture used for inverting fc6 is the same as the decoder of the VAE shown in Table . Architectures for other layers are similar, except that for reconstruction from conv5 fully connected layers are replaced by convolutional ones. The discriminator is the same as used for VAE. We trained on the ILSVRC-2012 training set and evaluated on the ILSVRC-2012 validation set.

**Ablation study.** We tested if all components of our loss are necessary. Results with some of these components removed are shown in Fig. . Clearly the full model performs best. In the following we will give some intuition why.

Training just with loss in the image space leads to averaging all potential reconstructions, resulting in over-smoothed images. One might imagine that adversarial training would allow to make images sharp. This indeed happens, but the resulting reconstructions do not correspond to actual objects originally contained in the image. The reason is that any "natural-looking" image which roughly fits the blurry prediction minimizes this loss. Without the adversarial loss predictions look very noisy. Without the image space loss the method works well, but one can notice artifact on the borders of images, and training was less stable in this case.

**Sampling pre-images.** Given a feature vector $\mathbf{\phi}$, it would be interesting to sample multiple images $\widetilde{\mathbf{I}}$ such that $\Phi(\widetilde{\mathbf{I}}) = \mathbf{\phi}$. A straightforward approach would inject noise into the generator along with the features, so that the network could randomize its outputs. This does not yield the desired result, since nothing in the loss function forces the generator to output multiple different reconstructions per feature vector. A major problem is that in the training data we only have one image per feature vector, i.e., a single sample per conditioning vector. We did not attack this problem in our paper, but we believe it is an important research direction.

**Best results.** Representative reconstructions from higher layers of AlexNet are shown in Fig. . Comparison with existing approaches is shown in Fig. . Reconstructions from conv5 are near-perfect, combining the natural colors and sharpness of details. Reconstructions from fully connected layers are still very good, preserving the main features of images, colors, and positions of large objects.

Normalized Euclidean error in image space and in feature space (that is, the distance between the features of the image and the reconstruction) are shown in Table . The method of Mahendran&Vedaldi performs well in feature space, but not in image space, the method of Dosovitskiy&Brox — vice versa. The presented approach is fairly good on both metrics.

**Iterative re-encoding.** We performed another experiment illustrating how similar are the features of reconstructions to the original image features. Given an image, we compute its features, generate an image from those, and then iteratively compute the features of the result and generate from those. Results are shown in Fig. . Interestingly, several iterations do not significantly change the reconstruction, indicating that important perceptual features are preserved in the generated images. More results are shown in the appendix.

**Interpolation.** We can morph images into each other by linearly interpolating between their features and generating the corresponding images. Fig.  shows that objects shown in the images smoothly warp into each other. More examples are shown in the appendix.

**Different comparators.** AlexNet network we used above as comparator has been trained on a huge labeled dataset. Is this supervision really necessary to learn a good comparator? We show here results with several alternatives to conv5 features of AlexNet: 1) $\textsc{fc}6$ features of AlexNet, 2) conv5 of AlexNet with random weights, 3) conv5 of the network of  which we refer to as VideoNet.

The results are shown in Fig. . While AlexNet conv5 comparator provides best reconstructions, other networks preserve key image features as well. We also ran preliminary experiments with conv5 features from the discriminator serving as a comparator, but were not able to get satisfactory results with those.

# Conclusion

We proposed a class of loss functions applicable to image generation that are based on distances in feature spaces. Applying these to three tasks — image auto-encoding, random natural image generation with a VAE and feature inversion — reveals that our loss is clearly superior to the typical loss in image space. In particular, it allows reconstruction of perceptually important details even from very low-dimensional image representations. We evaluated several feature spaces to measure distances. More research is necessary to find optimal features to be used depending on the task. To control the degree of realism in generated images, an alternative to adversarial training is an approach making use of feature statistics, similar to . We see these as interesting directions of future work.

# Acknowledgements

The authors are grateful to Jost Tobias Springenberg and Philipp Fischer for useful discussions. We acknowledge funding by the ERC Starting Grant VideoLearn (279401).

# Appendix

Here we show some additional results obtained with the proposed method.

Figure  illustrates how position and color of an object is preserved in deep layers of AlexNet .

Figure  shows results of generating images from interpolations between the features of natural images.

Figure  shows samples from variational autoencoders with different losses. Fully unsupervised VAE with VideoNet  loss and random initialization of the encoder is in the bottom right. Samples from this model are qualitatively similar to others, showing that initialization with AlexNet is not necessary.

Figures  and  show results of iteratively encoding images to a feature representation and reconstructing back to the image space. As can be seen from Figure , the network trained with loss in the image space does not preserve the features well, resulting in reconstructions quickly diverging from the original image.

```latex
\begin{figure*}
\begin{center}
\setlength{\tabcolsep}{0.1cm}
\begin{tabular}{cc}
\includegraphics[width=0.48\linewidth]{ICML_orig_interpolate_supp.jpg} &
\includegraphics[width=0.48\linewidth]{ICML_orig_interpolate_supp.jpg} \\
\includegraphics[width=0.48\linewidth]{ICML_gan_interpolate_supp_conv5.jpg} &
\includegraphics[width=0.48\linewidth]{ICML_gan_interpolate_supp_fc6.jpg} \\
\includegraphics[width=0.48\linewidth]{ICML_gan_interpolate_supp_fc7.jpg} &
\includegraphics[width=0.48\linewidth]{ICML_gan_interpolate_supp_fc8.jpg}
\end{tabular}
\end{center}
\caption{Interpolation in feature spaces at different layers of AlexNet.
\textbf{Topmost:} input images,
\textbf{Top left:} \textsc{conv}5,
\textbf{Top right:} \textsc{fc}6,
\textbf{Bottom left:} \textsc{fc}7,
\textbf{Bottom right:} \textsc{fc}8.}
\label{fig:interpol}
\end{figure*}
```
```latex
\begin{figure*}
\begin{center}
\setlength{\tabcolsep}{0.1cm}
\begin{tabular}{cc}
\includegraphics[width=0.48\linewidth]{ICML_vae_gan_supp_71_c70_vae_940000.jpg} &
\includegraphics[width=0.48\linewidth]{ICML_vae_gan_supp_72b_ft72_morenoise_990000.jpg} \\
\includegraphics[width=0.48\linewidth]{ICML_vae_gan_supp_91_c71_unsupvideocomparator_680000.jpg} &
\includegraphics[width=0.48\linewidth]{ICML_vae_gan_supp_91b_c91_randinitenc_470000.jpg}
\end{tabular}
\end{center}
\caption{Samples from VAE with our approach, with different comparators.
\textbf{Top left:} AlexNet \textsc{conv}5 comparator,
\textbf{Top right:} AlexNet \textsc{fc}6 comparator,
\textbf{Bottom left:} VideoNet \textsc{conv}5 comparator,
\textbf{Bottom right:} VideoNet \textsc{conv}5 comparator with randomly initialized encoder.}
\label{fig:vae_samples_supp}
\end{figure*}
```
```latex
\begin{figure*}
\begin{center}
\setlength{\tabcolsep}{0.1cm}
\begin{tabular}{cc}
\includegraphics[width=0.48\linewidth]{ICML_orig_iterative_supp.jpg} &
\includegraphics[width=0.48\linewidth]{ICML_orig_iterative_supp.jpg} \\
\includegraphics[width=0.48\linewidth]{ICML_gan_iterative_supp_conv5.jpg} &
\includegraphics[width=0.48\linewidth]{ICML_gan_iterative_supp_fc6.jpg} \\
\includegraphics[width=0.48\linewidth]{ICML_gan_iterative_supp_fc7.jpg} &
\includegraphics[width=0.48\linewidth]{ICML_gan_iterative_supp_fc8.jpg}
\end{tabular}
\end{center}
\caption{Iterative re-encoding and reconstructions for different layers of AlexNet.
Each row of each block corresponds to an iteration number: 1, 2, 4, 6, 8, 12, 16, 20.
\textbf{Topmost:} input images,
\textbf{Top left:} \textsc{conv}5,
\textbf{Top right:} \textsc{fc}6,
\textbf{Bottom left:} \textsc{fc}7,
\textbf{Bottom right:} \textsc{fc}8.}
\label{fig:iterative}
\end{figure*}
```
```latex
\begin{figure*}
\begin{center}
\setlength{\tabcolsep}{0.1cm}
\begin{tabular}{c}
\includegraphics[width=0.48\linewidth]{ICML_orig_iterative_supp.jpg} \\
\includegraphics[width=0.48\linewidth]{ICML_gan_iterative_supp_just_eucl.jpg}
\end{tabular}
\end{center}
\caption{Iterative re-encoding and reconstructions with network trained to reconstruct from AlexNet \textsc{fc}6 layer with squared Euclidean loss in the image space.
On top the input images are shown.
Then each row corresponds to an iteration number: 1, 2, 4, 6, 8, 12, 16, 20.}
\label{fig:iterative_eucl}
\end{figure*}
```

[^1]: This is unless the feature representation is specifically designed to map natural and non-natural images far apart, such as the one extracted from the discriminator of a GAN.