abstract: In the last decade, scenario-based serious-games have become a main tool for learning new skills and capabilities. An important factor in the development of such systems is the overhead in time, cost and human resources to manually create the content for these scenarios. We focus on how to create content for scenarios in medical, military, commerce and gaming applications where maintaining the integrity and coherence of the content is integral for the system's success. To do so, we present an automatic method for generating content about everyday activities through combining computer science techniques with the crowd. We use the crowd in three basic ways: to capture a database of scenarios of everyday activities, to generate a database of likely replacements for specific events within that scenario, and to evaluate the resulting scenarios. We found that the generated scenarios were rated as reliable and consistent by the crowd when compared to the scenarios that were originally captured. We also compared the generated scenarios to those created by traditional planning techniques. We found that both methods were equally effective in generated reliable and consistent scenarios, yet the main advantages of our approach is that the content we generate is more varied and much easier to create. We have begun integrating this approach within a scenario-based training application for novice investigators within the law enforcement departments to improve their questioning skills.
author: Sigal Sina

Sarit Kraus

Avi Rosenfeld

bibliography: VirtualTrainer.bib
title: Using the Crowd to Generate Content for Scenario-Based Serious-Games

# Introduction

Simulations and scenarios-based games, which constitute an important subset of serious-games, are an important tool for learning new skills and capabilities. Such systems are currently being used in a broad range of applications such as military, government, educational and health care . One main factor in the development of such systems is the overhead in time, cost and human resources to manually create the textual content for these scenarios. Specifically, while the discipline of Procedural Content Generation (PCG) for Games has focused on automatic generation of artificial assets such as textures, models, terrain and game rules, the automatic tools for scenario and textual content generation or adaptation for games and education are much less common . In this paper, we provide an automatic method for generating textual content about everyday activities through combining computer science techniques with the crowd– an approach that can be generally applied to education, government or health care scenarios.

This paper focuses on how one can easily generate textual content within a scenario that can provide narratives within a variety of domains. Our approach presents a novel solution which takes manually written scenarios of everyday activities and uses it to automatically create a new revised, reliable and consistent scenario. At the core of our approach is a database of daily, social scenarios and narratives. We demonstrate how this database is created from Amazon Mechanical Turk (AMT) workers, an established method for data collection from crowd . We are able to ensure the consistency of these scenarios by providing the AMT works with a semi-structured form. This facilitates the creation of a varied database of daily activities' scenarios, their descriptions (narratives) written in natural language along with key attributes and statistical information regarding possible content replacements to the everyday activities scenarios. Once this database is large enough, we can match the best possibility for a given scenario using the k-nearest neighbor algorithm given the constraint that the replacement must preserve the integrity of the modified data and then generate a personalized description for it using a 'fill and adjust" approach. Using crowdsourcing offers several advantages: first, it enables us to construct the activities' scenarios and narratives database rapidly and at a low cost. Second, it gives us a large and diversified content of daily activities from the workers. Last, we also use crowdsourcing techniques to quickly evaluate and demonstrate the efficacy of our approach in an extensive user study with another pool of workers.

While the approach that we present is general and can be used in many scenario-based games, we specifically focused on training scenarios. Specifically, our motivation for generating everyday content comes from a joint project **`VirtualSuspect`** with the law enforcement training department. The project's purpose is to train new law enforcement detectives of property felonies to efficiently extract relevant information from an interviewee. As our approach generates content with relatively low cost and maintenance, we can easily add new training cases, allowing investigators to have repeated practice sessions using different types of investigation techniques for different cases of property felonies. In the **`VirtualSuspect`** project, the generated content can also be applicable as an alibi for a specific case, for example, consider a case where a robber broke into a private house on Sunday night and stole a laptop, jewelry and some cash money. The law enforcement investigator (the human trainee) can question a suspect, focusing on what he did on Sunday night. The interviewee (our virtual agent), which can be innocent or not, needs a coherent scenario of his Sunday activities that is consistent with the facts that are known to the investigator and/or are common knowledge. While this paper focuses only on describing the interviewee's scenario generation portion of this project, it is important to stress that our proposed approach can also be useful for other applications with everyday content, especially in training scenarios such as training doctors to ask a patient the right questions or to help train candidates in a job interview application.

We found that the scenarios and their activities' details (narratives) we generated were rated as being as believable and consistent as the original scenarios and the original activities' details, and also compared them to activities' details that were generated with the more costly planning technique. In addition to the low cost of this technique, another main advantage of our approach is that the generated activities' details of the revised scenarios and narratives are much more diversified than those with a traditional planning technique. This advantage is an important key when implemented on large amounts of data, as one needs as varied a set of replacements as possible in order to keep the modified scenarios believable.

# Related Work

The development of serious games for training is a complex and time-consuming process . While early works considered a whole story creation without getting any real-time input from the user , the later works focus on interactive narratives where the user is part of the story and can affect the plotline . Some works use also hybrid methods whereby a predefined plot is created and an autonomic agent can later add to or adapt the plot in real-time . However, these studies focus on the general plot of the story but not on the story's details, which were almost exclusively manually created by content experts. A second important direction is the Procedural Content Generation (PCG) for Games discipline. However, while there has been significant in PCG generation for serious games in the past decade, to date, tools for textual content generation are still lacking . To address this need, we created a framework for generating coherent scenarios, which include a sequence of activity events (activities) and also the activities' descriptions (narratives).

The term scenario have several definitions depending on the context it is used. According to Hendrikx et al. "Game scenarios describe, often transparently to the user, the way and order in which game events unfold. Two types of game scenarios can be distinguished, abstract and concrete. The abstract game scenarios describe how other objects inter-relate. The concrete game scenarios are explicitly presented in the game, for example as part of the game narrative.". In this paper, we refer to concrete scenario types. As such, we use the term scenario to describe a sequence of activity events (activities) and their descriptions (narratives).

As is the case with several recent works , we implement a hybrid method which constitutes a "fill and adjust" semi-automatic narrative generation method. However, unique to our work, we leverage the crowd to create the textual content. The first step of our task is to identify the activities that should be modified. This problem has been studied extensively in database systems , and we built upon previous work that uses a maximal satisfiability solver (MaxSat) in order to identify these activities. Specifically, we used the off-the-shelf "akmaxsat" solver , which was one of the winners at "MaxSat 2010" for the "Weighted Partial MaxSat" category. Once the portions of the data that must be modified were identified, we found that using crowdsourcing allowed us to create a database of alternatives quickly and at a low cost. In crowdsourcing platforms, such as AMT or MicroTask, requesters typically break up a complex task into a number of simpler sub-tasks to make them easily applicable for layman workers. Previous work found that although micro-task markets have great potential for rapidly collecting user measurements at low costs, special care is needed to formulate tasks in order to properly harness the capabilities of the crowd . In order to facilitate the successful creation of the database needed for our application, we built questionnaires with precise instructions for each task and with proper compensation for the workers.

One traditional approach for activity-details generation is planning-based systems . The planning-based approach uses a causality-driven search to link a series of primitive actions in order to achieve a goal or to perform a task. For the domain of descriptive, every day activities, hierarchical scripts can capture common ways to perform the activity. Therefore, we implemented the planning-based generator using a Hierarchical Task Network (HTN), which is one of the best-known approaches for modeling expressive planning knowledge for complex environments. We used the state-of-the-art SHOP2 planner , a well known HTN planner, which has been evaluated and integrated in many real world planning applications and domains including: evacuation planning, evaluation of enemy threats and manufacturing processes. In order to validate the significance of modified scenarios with the activity-details we created, we also implemented a planning-based activity-details generator. As we later report, this approach is more costly to implement and generates less varied scenarios than the crowdsourced approach we now describe.

# System Overview

We propose and build a system (presented in Figure ) which ensures the scenario's integrity while replacing a given set of information. In the paper we will use the following running example to explain different stages within the system. For clarity reasons, we are using a simple example.

**Example 1**.  * John is a 21-year-old male who is single and has no children. He broke into a private house on Sunday night and stole a laptop, jewelry and cash money. He is now being questioning and need an alibi story for Sunday night.*

In this example, our system needs to preserve John's activity by concealing the information that John broke into a house. This is done by replacing the activity $BrokeIntoHouse$ with a common activity, such as $EatDinner$. After this activity switch has been made, a modified scenario with a new activity-details will be generated. Our system does this by basing itself on details from a collection of reported activities, which it modifies to better match the scenario main character's profile. Referring back to example , we base the revised scenario on a reported activity of a 26-year-old male with no children who goes out to lunch (Example ). Note that in this case we need to change the details about the time and location to match the required alibi (Example ).

**Example 2**.  * "I went and got lunch and a beer at a local bar "The Liffey". It was during March Madness, so I was watching some basketball. I sat at the bar and got chicken wings. I watched a few basketball games and ate. I read the newspaper a bit too. The food at "The Liffey" is always good. The team I picked won so that was also good. "*

**Example 3**. *: "On Sunday night I went out for dinner. I did not really want to spend too much so I went to "54th Street". I sat at the bar and got chicken wings. I watched a few basketball games and ate. I read the newspaper a bit too. The food at "54th Street" is always good. The team I picked won so that was also good."*

We now focus on how the system identifies what needs to be modified in the scenario, how it finds a similar record on which to base the modified scenario, and how it generates the final scenario with the appropriate activity-details. As we analyzed the coherent problem in scenarios, we identified three main questions the system needs to address. These questions are: (1) What should be removed from the scenario? (2) With what should we replace it? (3) How should we replace it? Accordingly, our system consists of three main modules, one to address each of these questions. The modules are: (1) A Maximal Satisfiability Solver (MaxSat) which ensures the scenario's integrity and identifies the places where modifications are required. Referring back to our example, this module will identify the need to replace the $BrokeIntoHouse$ activity; (2) Activities Generator which uses an algorithm based on the k-nearest neighbor algorithm in order to choose the most appropriate activity replacement for the required modification. In our example, it suggests replacing the $BrokeIntoHouse$ activity with the $EatDinner$ activity; and (3) Activity-Details Generator which expands the required activity replacement into a realistic, reliable, descriptive activity, as can be seen in examples and .

In this paper we do not focus on the MaxSat solver as this approach has been previously studied . We use an off-the-shelf MaxSat solver to generate the optimal solution with the minimum number of modifications to the scenario. We expand the traditional usage of this approach by using placeholders to protect facts which will later be replaced. Once the constraints have been identified, the system uses the activities generator and activity-details generator modules to fill in the missing details in the scenario for these placeholders. After the scenario has been finished but before the modified scenario is returned, the MaxSat solver is run again in order to confirm that no constraints have been violated in the final modified scenario.

This paper focuses on a novel approach using crowdsourcing techniques to generate replacement activities and their details. This is done through creating an activities dataset that contains a collection of daily schedule records. These records are composed of a list of activities and a collection of possible activities replacements, which can form the skeleton of a modified scenario. The details in these activities are selected from a dataset containing a collection of descriptive reported activities written in natural language. The selected replacement activity is associated with revised details that are consistent with a specific user profile. We now detail how these modules are defined and applied.

# Formal Definition of a Scenario

Before we present the system implementation, we define several concepts to be used in its description.

-   `User Profile` `(P)` - describes the user (i.e. the scenario's writer or the scenario's subject) properties and consists of gender, age, personal status and number of children.

-   `Scenario` - a sequence of activities and their descriptions represented as a list of pairs `<AI, ADR>` where each activity instance `AI` is accompanied with an activity-details record `ADR`. The description of these two fields follows.

-   `Activity Instance` `(AI)` - is a specific occurrence of activity which is part of the scenario and is composed of the activity name, a day, start and end time, location and participants. In our example: $AI(night,$ $John,$ $BrokeIntoHouse,$ $Downtown, alone)$.

-   `Activity-Details` `Record` `(ADR)` - is a tuple `<P,ADA,ADP>` where: `P` is a user profile, `ADA` is the activity-details attributes vector and `ADP` is the activity natural language presentation. A detailed description of the latter two fields follows immediately.

-   `Activity-Details` `Attributes` `(ADA)` - contains a vector of attributes which accompanies the activity-details. This vector is a superset of the activity instance `AI`, which contains the general attributes such as participants, a day and location, but it also contains information specific to the activity-details domain, such as restaurant name and type. It can contain optional values, and thus can be full or partial, for example in the eat-at-a-restaurant activity a person can eat at a restaurant alone, but can also go with a spouse. Within example above, we represent this vector as: $\langle$day (Thursday), part-of-day (noon), name (The Liffey), type (Bar and Grill), location (downtown) and participants (alone)$\rangle$.

-   `Activity` `Presentation` `(ADP)` - is the activity's detailed description written in natural language. It is composed of three parts: (1) The activity `Introduction` describes the main facts of the activity, such as who went, when, what are the main objects' names (which movie/restaurant), where and why; (2) The activity `Body` describes the activity in detail, what was the course of events and what happened during the activity; and (3) The activity `Perception` describes how good or bad the experience was from the user's perspective. Note that we intentionally split the activity presentation into these three parts. This semi-structured free text writing is very applicable when describing social, everyday situations. It also centralizes most of the activity specific details in the introduction part, which facilitates adjusting the activity to a new user profile and attributes vector. Accordingly, the presentation of example is: (1) `Introduction`: "I went and got lunch and a beer at a local bar$\ldots$" (2) `Body`: "I sat at the bar and got chicken wings$\ldots$" and (3) `Perception`: "The food at "The Liffey" is always good$\ldots$"

# System Implementation

We developed an innovative methodology to build the datasets using crowdsourcing. In all tasks, the AMT workers were first asked to provide their profiles `P` (gender, age, personal status and number of children). Then, they were presented with a semi-structured questionnaire containing a list of questions and were asked to fill it out. Examples of these forms can be found at http://aimamt.azurewebsites.net/.

One main challenge is how to best select the most appropriate record from within the entire dataset. To accomplish this task, we define a compatibility-relevant measure (as we describe in the algorithm flow) which is based on the similarity measure between attributes in order to predict which record is the best replacement. The basic component of the similarity comparison is the decision whether two values are similar. To make this comparison we associate each one of the attributes with a specific comparison function which gets as input two values and returns one of the three values: **same**, **similar** and **other**. Note that in the case one of the values is missing, it returns the **similar** value, as we assume the generators will fill this attribute with a similar value. For example, we consider the number-of-children attribute to be the **same** if the difference between the two values is 1 or less, **similar** if it is less than or equal to 3 and **other** if one person has children and the other does not or when the difference is greater than 3. For the day attribute, the comparison function returns **same** if both values equal, **similar** if both values are weekdays or weekends and **other** otherwise.

## Activities Generator

The activities generator's goal is to find the most appropriate activity replacement, such as $AI(night,John,$ $EatDinner,Downtown,alone)$, for the scenario's placeholder provided by the solver, such as $AI(night,John,PH,$ $Downtown,alone)$. To accomplish this goal we built the KAR (**K**-nearest neighbor **A**ctivity **R**eplacement) generator whose input is a user's profile, the scenario's activities list and an indication which activity instance to needs be replaced. KAR returns a revised scenario with a replaced activity which will later be associated with a natural language description and details.

**`The Dataset`** The activities dataset, denoted $DS_A$, contains a collection of daily schedule records, `SR`, and activity records, `AR`. The `SR` is a tuple `<P,Sch>` where `P` is a user profile and `Sch` is a daily schedule represented as a list of activity instances `AI`. The `AR` is a tuple `<P,Act>` where `P` is a user profile and `Act` is the activity properties and consists of the activity name (such as see-a-movie or have-a-meeting) and six attributes: a day (a weekday or weekend), part of day, duration, location, participants and frequency. We use two types of questionnaires in order to acquire the two record types. The first form is used to define the set of possible activities and the second form is used to collect additional data on each of the activities from a variety of profiles for the activities generator (described below). In the first questionnaire, we acquire weekday and weekend schedules (in an hour resolution), where we asked the workers to describe the activities as specifically as possible and limited each activity to up to a 3 hour duration. As we later describe, this data was also used to evaluate our system. For each activity in the schedule, the worker is asked to fill in the activity name (written in free text), the participants and the location of the activity. Defining the set of possible activities requires only a few schedules, and thus we collected 16 schedules from 8 subjects (4 male, 4 female, ages 23-53) which were paid 25-40 cents each for writing two schedules. We then map all of the activities in these schedules into an enumerated list and store the converted schedules `Sch` with their profile `P` at $DS_A$ as the `SR` records. The second questionnaire used to acquire additional properties, `Act`, for each of the activities in the collected schedules. The workers were presented with a form which included a list of activity record fields (Figure ), and were asked to fill it out using predefined selection lists. The `AR` records were collected from from 60 subjects (23 male, 37 female, ages 21-71) which were paid 35 cents each and were also stored at $DS_A$.

**`KAR`** Our activities generator, KAR, is implemented using the k-nearest neighbor algorithm and it uses a compatibility measure to predict which activity record is the best replacement for the placeholder. To calculate this measure, we select 10 attributes: the 4 attributes of the profile `P` and the 6 attributes of the activity `Act`. KAR first calculates the similarity measure of each of these attributes for all of the activity records `AR` within the dataset $DS_A$ compared to the given user's profile `P` and the activity placeholder `AI`, which it needs to replace. For example with a profile (Male, 21, single, no children) and a placeholder $AI(night,$$John,$$PH,$$Downtown,$$alone)$ compared to the following activity record $\langle$(Female, 31, married, 2 children), (EatAtRest, weekend, night, one hour, downtown, spouse, once a month)$\rangle$, the similarity measures are: $\langle$gender (other), age (other), number-of-children (other), personal-status (other), day (same), part-of-day (same), duration (similar), location (same), participants (other), frequency (similar)$\rangle$. The importance for any two values to be the same or at least similar depends on the specific attribute. For example, having the same gender value in the generated scenario is much more important than having the same age. To associate different importance levels for each attribute similarity measure, we developed a scoring function that gets an attribute and a similarity measure and returns a score within the range \[-15,15\]. We refine this score function using several preliminary trial and error iterations. The KAR generator then calculates the compatibility measure for each of the records, `AR`, as a summation of the scores of each of these attributes and its calculated similarity measure. Last, KAR sorts the activities records according to this measure and uses the k-nearest neighbor algorithm in order to choose the best candidate. Specifically, we implemented two variations of this algorithm, one with K=1 and the other with K=11. While the K=1 variation returns the activity with the highest measure, the K=11 variation also takes into account the number of similar records and thus returns the activity with the highest probability from the top 11 measures.

## Activity-Details Generator

The activity-details generator module is responsible for turning a given activity instance in the revised scenario into a realistic, reliable, descriptive activity. It gets as input the user's profile `P` and a partial activity details attributes vector `ADA` (which is made up of the values given in the activity instance `AI`). It returns as output a new activity-details record `ADR` which contains a reasonable, consistent and realistic activity presentation `ADP` written in natural language, which substitutes activity in the revised scenario. We implemented two types of generators: our approach, SNACS (**S**ocial **N**arrative **A**daptation using **C**rowd**S**ourcing), which uses the activity-details records we collected from the crowd, and for comparison a traditional planning-based generator, which is a common technique for content generation in many real world domains.

**`The Dataset`** We again use the crowd as the source of the activity-details dataset, denoted as $DS_D$, and build a collection of human activity-details written in natural language for a specific activity, such as see-a-movie. We used a dedicated, semi-structured questionnaire on AMT to collect the activity-details record `ADR` which includes: the profile `P`, the activity attributes vector `ADA` and the activity presentation in natural language `ADP`. Here, the workers were asked to describe daily, social activities in natural language in as much detail as possible according to the three activity presentation parts - introduction, body and perception. Then, the workers were presented with a list of specific questions used to collect the activity-details attributes vector, such as "What was the name of the movie/restaurant?", "With whom did you go?" and "on what day?". The completed records `ADR` were then stored at $DS_D$. We intentionally split the activity's detailed description into three parts. On one hand, this semi-structured free text writing is very applicable when describing social, everyday situations, and it helps us to elicit a detailed description of the activity from the workers. On the other hand, it centralizes most of the activity-specific details in the introduction, which allows us to adjust the activity-details to a new user's profile and attributes vector during the activity details generation without the need for intensive usage of NLP tools. Specifically, we collect and store 10 activity-details for 4 activities: two are entertainment activities (see-a-movie and eat-at-a-restaurant) and two are errand activities (buying-groceries and dry-cleaning). These records were collected from 20 subjects (6 male, 14 female, ages 19-55) which were paid 50 cents each for writing two activity-details.

**`SNACS`** Our activity-details algorithm, SNACS, first selects a candidate record from the activity-details dataset $DS_D$. We present 3 variations of this selection process below. Then, the algorithm completes the missing activity-details attributes. It generates attributes which are similar to the selected, original record's attributes and matches them to the new user's profile. It starts with the participant: who went and how many people participated in the activity. It then generates the objects' names (movie, restaurant, location) and time frame attributes. For example, if in the original activity someone went to see a children's movie with his son and the new user has no children, SNACS can choose to include his niece/nephew among the participants. Next, the algorithm generates the activity's natural language presentation. First, it replaces the original activity's introduction, i.e. its first part (who went, when, where, why), with a newly generated introduction according to the new profile and the new vector of attributes. This is done by using SimpleNLP , a Natural Language Generation (NLG), template-based surface realization, which creates an actual text in natural language from a syntactic representation. We created several NLG templates for each activity type, which were randomly chosen during the introduction generation. For example, one of the NLG templates used in order to build an activity introduction for the see-a-movie activity was: "Last $\left\langle time
\right\rangle$ I went to a movie with my $\left\langle with
\right\rangle$. We went to see the movie $\left\langle movie
\right\rangle$ at $\left\langle theater \right\rangle$". Each such template can generate a few variations according to the chosen attributes. For example, the first part of the above template, where the participants are a wife and son and the time is Sunday afternoon, can generate (a) Last weekend I went to a movie with my family or (b) Last Sunday afternoon I went to a movie with my wife and my son. Finally, SNACS applies some adjustments to the body and perception parts of the chosen activity's presentation (the second and third parts). This is done by replacing the references of the original attributes' vector with the new corresponding activity attributes' vector. In example , we replaced the restaurant's name.

We implemented 3 variations of the SNACS algorithm which differ in how the original candidate activity-details record is chosen: `SNACS-Any`, `SNACS-Bst` and `SNACS-Tag`. The `SNACS-Any` variation is a baseline measure that randomly chooses one activity-details record from $DS_D$. No further logic is performed to check how appropriate that choice is. In contrast, both the `SNACS-Bst` and `SNACS-Tag` variations use a compatibility measure to select which candidate from among all records in $DS_D$ will serve as the base for the generated activity description. The compatibility measure is based on 7 attributes: the 4 attributes of the profile `P` and only 3 attributes from the activity-details attributes vector `ADA` (participants, type and part-of-day). However, when assessing the compatibility of the activity-details record `ADR`, we also have to account for the activity presentation as written in natural language. Thus, we define an importance level vector `ILV`, which corresponds to these 7 attributes, for each activity-details record `ADR` within the dataset $DS_D$. Each value in `ILV` is a value `SM` and is used to represent the importance of the compatibility of a given attribute within the activity body and perception parts of the activity presentation. These values control how much importance should be given to having similarity between the original and generated activities' attributes. Accordingly, if a given attribute within `ADR` can be modified without violating any common sense implications, then the value is **other**. At the other extreme, if that attribute is critical and even small variations can make the activity implausible, then the value is **same**. SNACS considers two approaches in which the vector `ILV` can be built for every record. The first approach, denoted as `SNACS-Bst`, uses a fixed (automatic) `ILV` across all records within $DS_D$. Specifically, it contains the **same** value for the gender attribute and a **similar** value for all of the other attributes. The second approach, denoted as `SNACS-Tag`, utilizes a content expert to manually tag every record within $DS_D$. For example, the manual `ILV` for example is $\langle$gender (same), age (similar), number-of-children (other), personal-status (other), participants (other), type (similar), part-of-day (similar)$\rangle$.

During runtime, SNACS first calculates the similarity measure of each of these attributes for all of the records `ADR` within the dataset $DS_D$ compared to the given user's profile `P` and the (partial) activity instance `AI` it needs to replace in order to select the best candidate activity-details record. Recall that we used a **similar** value in case of missing values. In example , we evaluate John's profile and the activity instance $AI(night,John,EatDinner,$ $Downtown,alone)$ compared to the record `ADR` from example . Thus, the similarity measures are: $\langle$gender (same), age (similar), number-of-children (same), personal-status (same), participants (same), type (similar), part-of-day (other)$\rangle$. We again build a score function (which was refined using trial and error iterations), but this time it gets as input an attribute, a similarity measure and an importance level and returns a score within the range \[-15,15\]. SNACS then calculates the similarity compatibility measure for each of the records, `ADR`, as a summation of the scores of each of these attributes, its calculated similarity measure and its given (fixed or manual) importance level. Finally, the record with the highest measure value is chosen as the best activity-details candidate.

**`Planning-Based Generator`** In order to validate the significance of SNACS, we also implemented a HTN planning-based generator, denoted as `Planner`, using a plot graph that we built manually. A plot graph is a script-like structure, a partial ordering of basic actions that defines a space of possible action sequences that can unfold during a given situation. As we wanted to get richer activity-details which include a detailed description of the activity written in natural language, we gave the planner an option to tailor natural language descriptions in the basic actions portion of the activity. We defined a set number, 10-15, of different descriptions that were tailored to each one of the selected actions, which assured the implemented planner had a variety of descriptions with which to build activity-details. These descriptions were manually handwritten by two experts, which needed approximately one hour to write the set of descriptions for each basic action option. Part of these descriptions were also manually tagged with specific tags, such as movie or restaurant types. The tagging gave the generator an option to choose between a generic description which can be associated with any movie/restaurant type or a specific description which can be associated with the current selected type, such as action movie or Italian restaurant. Note that in SNACS, this step is not necessary as it automatically gets the activity's detailed descriptions from the original activity-details record. We also implemented dedicated actions' realizators (SimpleNLP based) that took the planner output, a semi-structured plan, and translated it into a natural language activity presentation in addition to the introduction's realizator we also used in SNACS.

Overall, the HTN-based generator has an inherently higher cost associated with it as compared to SNACS for the following reasons: Both SNACS and the planner have the steps of building the activity introduction templates and the implementation of the logical constraints. However, the planning-based algorithm implementation also required the following additional manual steps: the manual building of the plot graph; the writing, associating and tagging of several detailed descriptions for each basic action; and writing a specific realizator for each basic action. Each one of these steps requires both time and resources from a content expert or a system's designer. In fact, because of this cost overhead, we only used the HTN-planner in order to define the two entertainment activities (see-a-movie or eat-at-a-restaurant) and intentionally did not implement the HTN-based generator for the errand activities. Nonetheless, the activities' detailed descriptions produced by SNACS were as good as those developed by this costly process, as our results detail in the next section.

**`Random Baselines `**We also implemented two random methods as baselines to ensure the validity of the experiment. The first random method, denoted as `Rnd-SNACS`, uses the SNCAS infrastructure. It randomly chooses one of the activity-details records in the dataset and then randomly fills in the rest of the activity attributes. The second random method, denoted as `Rnd-Planner`, uses the planning-based generator. We removed the plan's logical built-in constraints and used random selections instead.

# Experimental Evaluation

In this section we present the evaluation of the generated, revised scenarios and their associated descriptive activities. Note that we don't include the evaluation of the MaxSat solver, as we use an off-the-shelf, previously studied solver. We evaluated separately the activities generator (KAR) and activity-details generator (SNACS) as described in the following sections. We use crowdsourcing to evaluate the efficacy of our approach in an extensive user study (200 subjects), again in AMT but with another pool of workers. We ensure that subjects answer truthfully by including open test questions and reviewing it manually before accepting the grades. We also estimate that completing a survey should take 8-15 minutes, so we filtered out forms which were filled out within less than 4 minutes. Examples of these questionnaires can be found at http://aimamt.azurewebsites.net/.

## Activities Generation Evaluation

The purpose of this experiment is to check the integrity and coherence of the scenario's activities list after the replacement of one of its activities.

**`Setup`** We chose to use the daily schedules we already collected from the crowd for $DS_A$, as the original scenario (without the activity-details that will be evaluated next). We randomly cut a section of 7-8 hours from each of the original activities list, to which we refer as `Original`. We then randomly chose one activity for the list to be replaced. We generated three revised lists by replacing the chosen activity. Two of the lists were generated using KAR algorithm, one with K=1 and the other with K=11, to which we refer as `KAR` `K=1` and `KAR` `K=11` respectively. The third list `Random` was generated using a random replacement and was implemented as a baseline to ensure the validity the experiment. We then evaluated these 4 variations of each activities list using AMT questionnaires. Each activities list was associated with a user profile and a day and the workers were asked to rate it with reference to the profile attached. The grades were valued from 1 (Least) to 6 (Most). An even number of choices was given as we didn't want the users to choose a middle value. The users were asked to grade three aspects: reasonable, matching to profile and coherent. The users were also asked to try to recognize which, if any, activity was replaced, and explain their answers in free text. Each activities list was rated by 15-20 subjects, to ensure we had enough independent grades for each generation method. A total of 80 subjects (38 male, 42 female, ages 19-61) participated in the evaluation and were paid 25-40 cents each.

<embed src="ActivitiesResults.eps" style="width:85.0%" />

**`Results`** Table presents the average grades for each aspect and also the total average grade, which was calculated as the average of these three grades. The results show that our generated method `KAR` `K=11` got the highest results, higher than the `Original`, however, there is no significant difference between the results of `Original`, `KAR` `K=11` and `KAR` `K=1`. As expected the `Random` variation got lower grades, which are significantly lower than the others (specifically, the ANOVA test of `Random` compared to `Original`, `KAR` `K=11` and `KAR` `K=1` had a much smaller than 0.05 threshold level with p=1.72E-9, 2.8E-11 and 1.13E-8 respectively). It also can be seen from the replacement identification percentage (the last row in Table ), that only 7% of the users identify the generated activity in the `KAR` `K=11` and `KAR` `K=1` methods compared to 61% in the `Random` replacement. These percentages are significantly different from an uniform random selection, which has a probability of 20% for being chosen.

## Activity-Details Generation Evaluation

The purpose of this experiment is to check the authenticity, integrity and coherence of the generated descriptive activities which accompany the revised scenario.

**`Setup`** We chose to evaluate four types of activities: see-a-movie, eat-at-a-restaurant, buying-groceries and dry-cleaning. For each activity type, we generated 4 user profiles. Then for each profile we generated activity-details for all of the generation methods[^1]. We also randomly selected 4 additional activity-details out of the original activity-details dataset for each activity type. As before, we ran the experiment using AMT questionnaires for comparison. Each activity-details was associated with its user profile and the AMT workers were asked to rate (with the same 6-value scale as before) six aspects of the activity-details: authenticity, reasonable, matching to profile, coherent, fluency and grammar. As before, subjects were also asked to explain their choice in free text. Each activity-details was rated by 8-10 workers to ensure we had enough independent grades for each activity type and generation method. A total of 120 subjects (59 male, 61 female, ages 18-69) participated in the evaluation and were paid 40-50 cents each.

<embed src="statAllTypes.eps" style="width:95.0%" />

**`Results`** Table presents the average grade for each activity type and generation method, which was calculated as the average of the six aspects' grades. The results show that our approach produced revised activity-details which were rated as being as reliable and consistent by workers when compared to the original activity-details and the planning-based technique. The main advantage of our approach is that the descriptive activities are much easier to create. Both of the random variations `Rnd-SNACS` and `Rnd-Planner` got significantly lower grades than all the others (specifically, the ANOVA test for the movie activity-details of `Rnd-Planner` compared to `Original`, `Planner` and `SNACS-Bst` had a much smaller than 0.05 threshold level with p=1.95E-4, 4.99E-3 and 9.30E-3 respectively). The results also show that all SNACS-based algorithms got very similar grades where `SNACS-Bst` and `SNACS-Tag` are slightly better than `SNACS-Any`. Overall for the entertainment activities details, although the `Original` grades are slightly higher than all of the other methods, there is no significant difference between all of the SNACS-based generator methods or the planning-based generator or the original activity-details. For the errands activity-details, there is also no significant difference between the grades of `SNACS-Tag`, `SNACS-Bst` and the `Original` activity-details, although the `SNACS-Any` variation got lower grades for the buy groceries activity-details. We found that the results for each of the six aspect grades were very similar to the average grade[^2].

# Conclusions

The paper makes the following key two contributions: (i) It is the first work to address the problem of modifying scenarios to generate personal information but yet maintains consistency even when varied scenarios are generated. (ii) It provides a methodology to use crowdsourcing in a principled way for this task. Instead of manually modifying scenarios, which makes the development process costly in both time and resources, we used the crowd as the source of our dataset, thus reducing the time and effort needed by to maintain coherence in these scenarios. To accomplish this task, we use the MaxSat logical engine in combination with a novel approach for the generation of everyday activities scenarios using the crowd. Our evaluation showed that our revised scenarios and their activities' details were rated as being as reliable and consistent as the original scenarios and the original activities' details, and also compared them to activities' details that were generated with the more costly planning technique.

# Acknowledgements

This work was supported in part by ERC grant \# 267523.

[^1]: We implemented the planning-based methods only for the see-a-movie and eat-at-a-restaurant activities because of the cost overhead.

[^2]: We omit this table due to lack of space.


abstract: The availability of language representations learned by large pretrained neural network models (such as BERT and ELECTRA) has led to improvements in many downstream Natural Language Processing tasks in recent years. Pretrained models usually differ in pretraining objectives, architectures, and datasets they are trained on which can affect downstream performance. In this contribution, we fine-tuned German BERT and German models to identify toxic (subtask 1), engaging (subtask 2), and comments (subtask 3) in Facebook data provided by the GermEval 2021 competition. We created ensembles of these models and investigated whether and how classification performance depends on the number of ensemble members and their composition. On data, our best ensemble achieved a macro-F1 score of $0.73$ (for all subtasks), and F1 scores of $0.72$, $0.70$, and $0.76$ for subtasks 1, 2, and 3, respectively.
author: Tobias Bornheim; Niklas Grieger; Stephan Bialonski
title: FHAC at GermEval 2021: Identifying German toxic, engaging, and fact-claiming comments with ensemble learning

# Introduction

Social media plays a role in the spreading of problematic content, ranging from conspiracy theories and concerted misinformation campaigns to offensive language in user comments . Moderating comments remains a challenge due to the ever-increasing amount of user-generated content created daily. One promising approach to addressing this challenge are techniques from Natural Language Processing (NLP) that support manual moderation processes by, for example, alerting human moderators to potentially problematic comments.

Among the many factors that have driven recent progress in NLP, we note in particular (i) methodological advances in language modeling and (ii) the availability of annotated data due to shared tasks. Recent methodological advances can be traced back to the invention and availability of deep neural network models. A major contribution was the invention of the transformer architecture, which harnesses self-attention mechanisms to effectively model long-range correlations in series of tokens (e.g., sentences) . Based on the transformer architecture, neural network models such as BERT were proposed and trained in a self-supervised fashion on large unannotated text corpora. Language representations learned by BERT turned out to be effective in many downstream tasks, leading to new state-of-the-art NLP systems. While *masked language modeling* and *next sentence prediction* are used as objectives in self-supervised pretraining to learn representations in BERT, other pretraining objectives such as (ELECTRA, ) have been demonstrated to yield language representations that can be better suited for various downstream tasks . Furthermore, language representations have been learned in multilingual language models (such as mBERT) and in language specific BERT models . Recent German specific language models include the BERT-based models GBERT and GottBERT as well as the ELECTRA based model GELECTRA .

The second factor driving progress in NLP has been recurring shared tasks that foster the exchange of ideas, the development and comparative assessment of methods, as well as the availability of annotated data . In addition to multilingual shared task campaigns (see, e.g., ), there exist language-specific shared task evaluations such as GermEval which focus on NLP for the German language. A series of GermEval tasks addressed the challenge of reliably identifying offensive language and distinguishing between profane, offensive, or abusive language found in Twitter tweets . The GermEval 2021 shared task on identifying toxic, engaging, and fact-claiming comments provided German comments from a Facebook page of a political talk show of a German television broadcaster.

In this contribution, we investigate the ability of ensembles of GBERT and GELECTRA models to identify toxic, engaging, and fact-claiming comments. Our work was inspired by previous studies on German BERT models and ensemble approaches . We study the dependence of classification performance on the number of ensemble members and ensemble composition. Finally, we describe the models that were evaluated in the shared tasks and report performance scores achieved on out-of-sample data. The implementation details of our experiments are available online[^1].

# Data and tasks

The dataset of the shared task consisted of 3244 annotated Facebook comments and was provided by the organizers of GermEval 2021 . The comments were drawn from a Facebook page of a political talk show of a German television broadcaster from February till July 2019 and were anonymized by replacing links to users by @USER, links to the show by @MEDIUM, and links to the moderator of the show by @MODERATOR. Four trained annotators labeled the data by three categories, indicating toxic, engaging, and comments (see figure ).

The shared task consisted of three binary classification subtasks that aimed at predicting whether a given comment belonged to a category (class) or not . Comments were considered toxic (subtask 1) when they could violate the rules of polite behavior or violated democratic discourse values. Automated identification of such comments can be particularly valuable for managers of online communities. Comments were considered engaging (subtask 2) when they were in line with deliberative principles such as rationality, reciprocity, and mutual respect. Such comments might encourage user engagement and could be made more visible in online communities. Finally, comments were considered fact-claiming (subtask 3) if they contained assertion of facts and/or provided evidence by citing external sources. Identifying such comments may constitute a preprocessing step that could assist community managers to filter out misinformation and fake news.

We did not observe any major class imbalance  as all three classes occurred with similar frequencies in the dataset (35% toxic, 27% engaging, 34% fact-claiming). However, the Venn diagram (see figure ) demonstrated significant overlap between classes where 24% of all comments were attributed to more than one class. For instance, the large overlap between the engaging and the fact-claiming classes may point towards a correlation between these two classes. Such label correlations can be exploited by approaches to improve classification performance . Thus we pursued a two-fold strategy. (i) In our first approach, we trained a multi-label classifier to predict all the possible class attributions for a given comment. Such models are called *multi-label* in the following. (ii) In our second approach, we trained separate binary classifiers that aimed at distinguishing between toxic and non-toxic, engaging and non-engaging, or fact-claiming and non-fact-claiming classes, respectively. This approach of transforming a multi-label classification task into multiple single-label classification tasks is also known as a *binary relevance transformation* . We call such models *single-label* in the following. Note that in this case, training a size 30 ensemble to classify comments means training three separate size 30 ensembles, each making predictions for one of the three binary classification tasks.

# Methods

## Preprocessing and data splits

*Preprocessing.* All data (i.e., training and test data) was preprocessed as follows. First, all duplicates in the training data were removed, reducing the 3244 training samples to 3226 unique samples. In a second step, all in-word whitespaces were removed (e.g. transforming the sequence "A K T U E L L !" into the word "AKTUELL!") . Third, emojis were buffered with additional whitespaces such that words immediately followed by an emoji were not tokenized as unknown and emojis were tokenized separately (e.g., transforming the sentence "I always start my day with a " into "I always start my day with a coffee ") . Fourth, any leading, trailing or consecutive whitespaces were removed. Last, all comments were limited to a maximum length of 200 tokens to save computational resources and speed up training. Only 49 out of the 3226 unique sentences in the training data and 21 out of 944 sentences in the test data were affected by this step.

*Data splits.* During model exploration, models were trained with a 5-fold cross validation scheme (i.e., with 5 folds, each containing 20% of the randomly shuffled training data). The final models evaluated by GermEval 2021 were trained on all training data (i.e., on all folds) to optimize model fitting. Furthermore, during model exploration as well as for the final models, 10% of the data in the training folds was randomly selected to act as an *early stopping set* (see section ) that was not used for training.

## Models

We studied two recent transformer-based German language models called GBERT, based on the BERT architecture , and GELECTRA, based on the ELECTRA architecture . Both models use a tokenizer with a vocabulary size of 31k cased words. From the different pretrained versions of these models, we chose gbert-large[^2] and gelectra-large[^3], both with a hidden states count of 1024.

A classification head was added on top of the first output vector of both pretrained transformer models. In the GBERT architecture, the mentioned output vector was generated by inserting a classification token at the beginning of every input sequence, which is used for the task during pretraining . The classification head consisted of a linear layer with the same hidden size as the transformer model, followed by a tanh activation function and another linear layer . Although the GELECTRA architecture does not use any *next sentence prediction* task during pretraining , a classification token is still prepended to the transformer input and can be used during fine-tuning. The classification head of GELECTRA had the same architecture as that of the GBERT model, except that a GELU activation was used instead of a tanh activation .

All linear layers of both classification heads were initialized randomly, except for the first layer of the GBERT classifier, which was initialized with the weights learned during the pretraining task. Depending on whether the models were single-label or multi-label classifiers, the final linear layer consisted of either two outputs followed by a softmax function or three outputs followed by a sigmoid function.

## Training

*Evaluation scores.* To evaluate the prediction performance of a model, we determined the F1 score following the definition used throughout the GermEval shared tasks . In GermEval, the F1 score of a binary classifier is determined by calculating precision and recall for the positive class (e.g., "toxic") and for the negative class (e.g. "non-toxic"). Precision and recall are then averaged over the two classes. The F1 score is calculated as harmonic mean over averaged recall and averaged precision. By taking the arithmetic mean of F1 scores of each binary classifier, we obtained the macro-F1 score $\overline{F1} = \frac{1}{3} (F1_{toxic} + F1_{engaging} + F1_{fact})$. During model exploration, $\overline{F1}$ scores were determined for all five validation folds, and their mean and standard deviation were determined. We considered a model to be superior to other models if its $\overline{F1}$ score averaged over all validation folds (of the cross validation) was larger than those of the other models.

*Training scheme.* Each model (i.e., transformer with classification head) was trained with a batch size of 24 samples for 10 epochs using the AdamW optimizer . We used a learning rate of $\eta=5\cdot 10^{-6}$ with a linear warmup on the first 30% of the training steps from $0$ to $\eta$. Every 40 updates of the gradients, the models were evaluated on the early stopping data by calculating the macro-F1 score. If the score did not increase for two consecutive evaluations the training was interrupted and the model achieving the largest F1 score on the early stopping set was used for evaluation on the validation fold or test data.

*Loss functions.* When training single-label models, we used a negative log-likelihood loss function. Multi-label models were trained by minimizing the binary cross entropy loss function averaged over the three classes for every sample in a mini-batch.

*Threshold selection.* In multi-label models, a sample (comment) from the dataset was predicted to belong to those classes for which the respective class probabilities of the model exceeded a certain threshold. Since multi-label models can attribute a sample to three classes, three different thresholds needed to be determined. We chose these thresholds by evaluating the model for threshold values between 0 and 1 (exploring this range with a step size of 0.025) on the data reserved for early stopping and accepting the values achieving the highest F1 scores for each class separately. In single-label models, we did not need to chose any thresholds since the class membership of a sample was predicted by identifying the largest output probability of the two output neurons.

## Ensembling

Training complex models such as GBERT or GELECTRA on small datasets can lead to overfitting. Following the work by , we counteracted this phenomenon by creating ensembles of models using bootstrap aggregation. Ensemble members differed in the initial weights of the classification layers and the data samples randomly selected for early stopping. The predictions of an ensemble were determined by averaging the predicted probabilities of the ensemble members (*soft majority voting*). In single-label models, a model's prediction was then determined by identifying the output neuron associated with the largest ensemble-averaged output probability. In multi-label models, a model predicted a sample to belong to certain classes if ensemble-averaged class probabilities exceeded optimal thresholds. The optimal thresholds were determined by evaluating each ensemble member for all thresholds on the early stopping data (see section ) and accepting the thresholds with the highest macro-F1 score as the optimal values.

# Results

```latex
\begin{table*}
    \begin{center}
      \begin{tabular}{c|c|c|c|c}
          & $\mathbf{\overline{F1}}$ & $\mathbf{F1_{toxic}}$ & $\mathbf{F1_{engaging}}$ & $\mathbf{F1_{fact}}$\\
          \hline
          \multicolumn{5}{c}{\textit{model exploration}}\\
          \hline

          50 GELECTRA & & & & \\
          multi-label & 0.765 (0.008) & 0.730 (0.018) & 0.782 (0.018) & 0.784 (0.019) \\
          \hline

          50 GBERT & & & & \\
          multi-label & 0.760 (0.002) & 0.720 (0.006) & 0.777 (0.015) & 0.782 (0.013) \\
          \hline

          25+25 GELECTRA/GBERT & & & & \\
          multi-label & 0.763 (0.007) & 0.726 (0.010) & 0.780 (0.015) & 0.784 (0.015) \\
          \hline

          25+25 GELECTRA/GBERT & & & & \\
          single-label & 0.768 (0.006) & 0.736 (0.011) & 0.782 (0.014) & 0.787 (0.013) \\
          \hline
          \hline
          \multicolumn{5}{c}{\textit{final submissions}}\\
          \hline

          200 GELECTRA & & & & \\
          multi-label & 0.717 & 0.713 & 0.690 & 0.748 \\
          \hline

          200+200 GELECTRA/GBERT & & & & \\
          multi-label & \textbf{0.726} & 0.716 & \textbf{0.699} & \textbf{0.763} \\
          \hline

          30+30 GELECTRA/GBERT & & & & \\
          single-label & 0.699 & \textbf{0.718} & 0.658 & 0.723 \\
          corrected scores & \textbf{0.727} & \textbf{0.717} & 0.697 & \textbf{0.768} \\
      \end{tabular}
    \end{center}
    \caption{
        F1 scores achieved by different ensembles during model exploration on the validation folds (rows 1--4; mean and standard deviation over the folds) and F1 scores achieved by the submitted models on the test data as reported by the GermEval 2021 organizers (rows 5--7; best scores are shown in bold).
        The \emph{corrected scores} shown in the last row were calculated after correcting an error identified after submission.
      }
      \label{tab:f1scores}
\end{table*}
```

*Model exploration.* We investigated whether and how classification performance (quantified by macro-F1 score) depended on (i) ensemble size, (ii) ensemble composition, and (iii) on whether ensemble members can exploit label correlations ( models) or not (single-label models). To study the effect of (ii), we compared the classification performance of different ensemble compositions. The first ensemble consisted of fine-tuned multi-label GELECTRA models, while the second ensemble consisted of fine-tuned multi-label GBERT models. In a third ensemble we used equal parts of fine-tuned multi-label GELECTRA and GBERT models. To study the effect of (iii), we compared the third ensemble with a fourth ensemble which was composed of equal parts of single-label GELECTRA and GBERT models. Finally, we investigated the effect of (i) via a bootstrap experiment following .

The bootstrap experiment was carried out using a 5-fold cross validation scheme. We trained 100 models each of multi-label GBERT and multi-label GELECTRA, and 50 models each of single-label GBERT and single-label GELECTRA on each cross-validation split. For a given ensemble size, we created 1000 ensembles by randomly sampling with replacement from the set of trained models. Each ensemble made predictions on a validation fold by *soft majority voting*. The average macro-F1 score of an ensemble was determined by averaging the macro-F1 scores obtained on each of the 5 validation folds. Thus, for a given ensemble size, we obtained 1000 average macro-F1 scores.

Figure  shows the mean of the average macro-F1 scores obtained for different ensemble sizes and ensemble compositions. We observed classification performance to increase with ensemble size, irrespective of model composition and of whether models could or could not exploit label correlations. Largest increases were found for ensemble sizes up to 15 ensemble members, which is consistent with a previous study on a different classification task . Moreover, macro-F1 scores continued to increase beyond the ensemble size of 15.

For a given ensemble size larger than 30, classification performance between ensembles of the different compositions varied only in the third decimal of their macro-F1 score, a variation that we did not consider significant. Ensembles consisting of 100% GELECTRA models, 100% GBERT models, or 50% GELECTRA and 50% GBERT models yielded comparable macro-F1 scores. Likewise, ensembles consisting of either multi-label or models showed comparable macro-F1 scores for a fixed ensemble size. These observations were confirmed by F1 scores obtained for ensembles of size 50, reported in table  (rows 1–4).

*Submitted models.* Three ensembles were submitted and evaluated on the test data of the shared tasks reflecting the lines of investigation laid out before. The evaluated ensembles were (1) an ensemble of 200 multi-label GELECTRA models, (2) an ensemble of 200 multi-label GELECTRA and 200 multi-label GBERT models, and (3) an ensemble of 30 single-label GELECTRA and 30 single-label GBERT models which were trained on all the training data (see section ). We note that time and computational constraints limited ensemble sizes.

On the test data of the shared task, ensemble (2) achieved the largest macro-F1 score of $0.73$, followed by ensemble (1) with $0.72$ and (3) with $0.70$ (see table , rows 5–7). We identified a software bug after submission deadline that affected the scores calculated for ensemble (3) which achieved a corrected macro-F1 score of $0.73$. These results supported observations made during model exploration that ensemble composition and classification type did not significantly affect classification performance for ensemble sizes larger than 30.

# Conclusion

We trained ensembles of fine-tuned German language models, namely GELECTRA and GBERT, to classify German toxic, engaging, and comments in the GermEval 2021 shared task. We investigated whether classification performance (quantified by macro-F1 scores) depended on (i) ensemble size, (ii) ensemble composition, or (iii) whether models were trained as multi-label classifiers (and thus potentially exploiting label correlations) or as single-label classifiers. We observed that ensemble size had a significant effect on classification performance, with more ensemble members leading to better macro-F1 scores, consistent with previous observations by on a different dataset. Neither ensemble composition nor model classification type (multi- or single-label) showed significant different classification performance for the studied parameters when the ensemble size was larger than 30. Two ensembles achieved the largest macro-F1 score ($0.73$) on the test data, namely the multi-label and single-label ensembles consisting of GELECTRA and GBERT models.

[^1]: <https://github.com/fhac-fb9-ds/germeval2021>

[^2]: <https://huggingface.co/deepset/gbert-large>

[^3]: <https://huggingface.co/deepset/gelectra-large>


abstract: State-of-the-art Neural Network Architectures (NNAs) are challenging to design and implement efficiently in hardware. In the past couple of years, this has led to an explosion in research and development of automatic Neural Architecture Search (NAS) tools. AutomML tools are now used to achieve state of the art NNA designs and attempt to optimize for hardware usage and design. Much of the recent research in the auto-design of NNAs has focused on convolution networks and image recognition, ignoring the fact that a significant part of the workload in data centers is general-purpose deep neural networks. In this work, we develop and test a general multilayer perceptron (MLP) flow that can take arbitrary datasets as input and automatically produce optimized NNAs and hardware designs. We test the flow on six benchmarks. Our results show we exceed the performance of currently published MLP accuracy results and are competitive with non-MLP based results. We compare general and common GPU architectures with our scalable FPGA design and show we can achieve higher efficiency and higher throughput (outputs per second) for the majority of datasets. Further insights into the design space for both accurate networks and high performing hardware shows the power of co-design by correlating accuracy versus throughput, network size versus accuracy, and scaling to high-performance devices.
author:
;
;
;
bibliography: references.bib
title: AutoML for Multilayer Perceptron and FPGA Co-design

# Introduction and Motivation

Optimizing neural network architectures (NNA) is a difficult process in part because of the vast number of hyperparameter combinations that exist. In cases where a combination is not optimal, performance will suffer. Research suggests that the parameters of a network can directly influence the accuracy, throughput, and energy consumption of that model in deployment . The difficulty in designing performant neural networks has brought a recent surge in interest in the automatic design and optimization of neural networks. The focus of the existing body of research has been on optimizing NNA for accuracy but lack network-specific hardware optimizations. Our focus is to close this gap by using evolutionary algorithms to search an entire design space, including NNA and reconfigurable hardware. This Co-design approach provides a solution that implements a custom hardware solution for a specific NNA model.

Numerous articles have focused on image classification with convolutional neural networks, in part, due to the introduction of AlexNet in 2012 and the ImageNet database . Traditional neural networks such as multilayer perceptrons (MLP) have taken a backseat from the spotlight. Yet, their application is still very relevant, large data-centric companies such as Facebook and Google have published data showing that MLP workloads are the majority of their application base. Facebook cites the use of MLP for tasks such as determining which ads to display, which stories matter to see in a news feed, and which results to present from a search. Park et al. stress the importance of these networks and the current limitations on standard hardware and the call for what this research aims to solve, i.e., software and hardware co-design in . Google recently published a performance analysis of their TPU , showing MLP consisting of 61% of the TPU workloads. Our findings are in line with the results presenting in their paper, stating that MLP workloads are memory bound. Our co-design process is capable of designing optimal hardware structures that balance the compute and memory requirements. Section provides insights on memory bandwidth considerations.

At the heart of MLP is a general matrix multiplication (GEMM), and libraries have existed for years such as Basic Linear Algebra Subprograms (BLAS) that provide highly tuned hardware-specific GEMM implementations. Standard hardware such as CPUs and GPUs provide these subroutines to make implementing performant MLPs easy. Still, two problems persist in this typical flow: 1. target hardware is normally not considered during the creation of the MLP, and 2. standard architectures are general purpose. General purpose architectures do not necessarily offer performance scaling concerning the ANN description (see section ). Our research aims to take advantage of the reconfigurable architecture of an FPGA device that is capable of molding to a specific workload and neural network structure. Leveraging evolutionary algorithms to search the entire design space of both MLP and target hardware simultaneously, we find unique solutions that achieve both top accuracy and optimal hardware performance. Section provides top results for MLP networks and hardware configurations derived from a series of tests over multiple datasets. Correlations between traditional approaches and our approach are discussed to provide insights into how we achieve top results.

# Related Work

Automating NNA search has been an ongoing effort for the past few decades but is becoming a focus of the NNA research community because of the difficulty in designing deep networks which are ever growing in complexity. Automatic Artificial Neural Network Architectures Search (NAS) can be conducted using different strategies such as random search, evolutionary algorithms, Reinforcement Learning (RL), Bayesian optimization, and gradient-based methods . Using Evolutionary Algorithms (EAs) to search for performant architectures has been investigated extensively over the years. Some recent results indicate that evolutionary algorithms offer better results than random search and reinforcement learning . Recently, there has been growing interest in NAS for deep neural networks that specialize in image recognition . A recent survey on available tool flows is available here . The body of work on NAS concentrate on accuracy as the main measure of performance, though optimizing for NAS can lead to more simplified NNA that could in turn simplify and optimize hardware designs . On the other hand, optimizing for hardware performance parameters (latency/throughput/power) is normally done on an existing NNA design and there is no attempt to modify the NNA (layers/neurons etc.).

# Evolutionary Cell Aided Design Flow

The Evolutionary Cell Aided Design (ECAD) flow, previously described in , is shown in Figure and starts with a general industrial/research problem that (a) sufficient data exists for (b) there are well defined inputs/outputs and (c) it is a problem that can benefit from software/hardware acceleration. Once such a problem is identified a dataset will be exported into a Comma Separated Value (CSV) tabular data format, in addition a configuration file will be created and will contain information on (a) the general NNA structure including input and output sizes, initial number of layers and neurons, (b) Hardware target including reconfigurable hardware device type, DSP count, memory size and number of blocks, (c) optmization targets such as accuracy, throughput, latency, and floating-point operations. Note that the configuration file can be generated automatically based on an existing template configuration file and the dataset.

## Evolutionary Process

The ECAD Evolutionary process, based on a steady-state model , generates a population of NNA/Hardware co-design candidates each with a complete set of parameters that effect both the accuracy and the hardware performance. The parameters we considered during our searches included number of layers, layer size, activation function, and bias. Each candidate in the population is evaluated according to configurable and potentially multiple criteria, for example accuracy alone or accuracy vs throughput. The *raw* evaluation measurments are done on a software artifact, dubbed a *Worker* in ECAD terminology. The Worker returns the *raw* evaluation information to a *Master* process. The *Master* process orchestrates the evaluation process by distributing the co-design population and by evaluating the results. Result evaluation is done using user defined fitness functions. For example, an accuracy fitness function can simply return the accuracy value obtain by a simulation worker(see next section for more information on worker types). But it can also scale or weight the value or specify to minimize or maximize the value. Simple evaluations functions can be specified in the configuration file and more complex ones are written in code and added by registering them with the framework.

## ECAD Hardware

The evolutionary search has three workers at its disposal to assess the fitness of various hardware platforms. The simulation worker is useful for assessing instruction-set based architectures such as CPU and GPU, whereas the physical and hardware database workers are useful for hardware that requires design and synthesis.

Simulation workers take queries from the evolutionary engine and run them on the target hardware. For example, if the target hardware is a GPU, the evolutionary engine sends the physical worker an ANN description who converts it into a readable format for the GPU. Once the GPU returns the result, the physical worker is responsible for recording all necessary metrics that are provided back to the engine to assess the fitness. Metrics include throughput, latency, and power consumption.

Hardware database workers provide a means for hardware platforms that are easily simulated or modeled. In our experiments, for example, we leveraged the hardware database worker to provide a means of accepting both an ANN description and hardware configuration that together were run through a model to obtain the metrics for fitness evaluation. Once the engine finds performant designs via tracking each configuration's fitness, the model can be realized into hardware and run through the physical worker. FPGA proved to be a suitable architecture for the hardware database worker. The reconfigurable nature of FPGAs, coupled with modeled overlay architectures, allows the worker to assess many configurations in a relatively swift manner compared to running through synthesis tools.

Physical workers can be used to synthesize and evaluate hardware designs. While the hardware database worker provides fitness of the overall application with metrics such as throughput, the physical worker aims to provide the fitness of the hardware design itself through metrics such as power, logic utilization, and operation frequency. In the case of Intel FPGAs, the physical worker responds with ALM, M20K, and DSP utilization, power estimations, and clock frequency (Fmax).

Having access to these workers in the evolutionary engine is a powerful tool that provides various performance metrics about the hardware platform as a function of the ANN. Since ANN descriptions can vary widely, the throughput and power consumption on hardware devices can also vary widely. Understanding how a hardware device performs across ANN variations aids the decision on how to deploy ANN solutions, as seen in the experiments.

Beyond these tools providing insights into architecture performance, the Pareto frontiers that result after parsing the evolutionary design space define what the optimal solution is. Every industry deploying ANN solutions may have different opinions and demands for their use case. One size fits all, or one solution is not realistic or optimal. Having the data to make decisions based on trade-offs is highly valuable. Since some ANN descriptions are approximations, i.e., they include redundancy (although we note that newer research is starting to limit the amount of redundancy) and educated guesses, it is an unreasonable expectation to hand optimize ANN descriptions for a specific hardware platform.

## FPGA Design Model and Hardware Description

Workers can provide input to any hardware capable of being modeled efficiently. Given the reconfigurable architecture of FPGAs and modern HLS tools capable of describing overlay designs that interface nicely with software programs, we chose Intel FPGAs and OpenCL as our hardware platform and development. Any flavor of FPGA can use the overlay-style architecture we developed, and all that is required to change the design search space, for example, between Arria 10 and Stratix 10, is the hardware configuration used by the hardware database worker.

Hardware database workers receive configurations that include enough information to construct the model and derive the necessary performance results. Part of this configuration is a hardware-specific configuration file that defines the target accelerator. This information includes the name of the FPGA, the relevant primitive logic details such as DSP and SRAM count, target clock frequency, the type of global memory (DRAM) to be used, and its speed and rate. All of this information is used to estimate performance. The other part of the configuration is the ANN description. Low-level hardware suitable parts are decomposed from the ANN description and define the work that must be done by the accelerator.

General matrix multiplications are commonly used to accelerate multilayer perceptrons, so the design we used is based on a 2D systolic array architecture that includes additional functionality to support activation functions and vector additions for bias operations. This "grid" architecture has various design space variables that we allow mutations to take place on. The variables are the number of rows and columns, double buffer cache sizes for each dimension, called interleaving, and the vector width of each processing element (PE). Every variable has implications in the design space. Each mutation has a unique fitness, not only as it applies to the hardware but also the ANN. This works leverages and build on the SGEMM architecture described in .

Our model returns values we deemed fundamental, including potential and effective performance, total time, outputs per second, and latency. Giga-operations per second (GOP/s) is the unit to describe the potential and effective performance. The potential performance is typically the marketed performance that defines the roofline of the configuration, whereas the effective performance defines the actual or real performance of the configuration under a workload. The delta between them provides a metric on how well the problem maps to the model. The total time for our model is defined by the difference between the timestamp when all necessary data persists in the accelerator DRAM, and we enqueue the OpenCL kernels, and the timestamp once all results persist in DRAM and the last OpenCL kernel returns. We call this a run. Outputs per second is a generalization of the data type and provides the total results we can achieve in one second. For example, if the data type is images, then this metric could be interpreted as images per second. Lastly, latency is the time it takes from the start of a run to store one result into DRAM.

Calculating these results in the model is accomplished by starting with the baseline performance of a configuration. All data is 32-bit floating-point and maps into the hardened floating-point DSP blocks of the Arria 10 device. From , we can calculate the baseline performance by determining how many DSP blocks are doing work. The utilization of DSPs is the product of the grid dimensions and vector width. This number is the potential performance, but before considering bandwidth. Using the DRAM specs from the configuration, we can determine the ratio of how much bandwidth is available to how much we need. Cycles per block of data divided into the size of a block in bytes are used to calculate bandwidth needs. This calculation is the potential performance. Next, the grid configuration is used to break the ANN up into a series of blocked matrix multiplications. Having the data now blocked, we can derive all other results.

## MLP Mapping to Hardware

GEMM nomenclature can be used to describe the three key dimensions that make up the problem size for MLP layers. Two matrices A and B are multiplied together to form a new matrix C. If matrix A has sizes *m x k* and matrix B *k x n*, then matrix C is *m x n*. *M* is the number of inputs that are processed at once and commonly referred to as the batch size. Architectures such as GPU typically batch with a larger *M* dimension to fill up compute cores and obtain higher throughput. Our design for FPGA does not need to increase batching because the PEs can be arranged in a manner that exploits parallelism in other dimensions. This results in a lower batch and lower latency accelerator. With sufficient bandwidth, hardware efficiency can be increased by keeping the *m* dimension smaller, and further vectorizing on *k* and *n* dimensions. *N* dimension is the number of neurons that also defines a subsequent layer *k*. Lastly, the size of the dataset defines the first layer *k*. Fitting an MLP into hardware is one aspect that makes the co-design approach so powerful.

# Experiments

In this section, we show the results from running a series of evolutionary searches on six different data sets: MNIST , Fashion MNIST , Credit-g , Har , Phishing , and Bioresponse . MNIST and Fashion MNIST were chosen so we could compare against wildly used research benchmarks. Credit-g/Har/Phishing/Bioresponse were chosen since they represent potential real-world datasets.

FPGA hardware results assume the architecture described in section . Search was done on two different FPGAs, an Arria 10 1150 GX device at a clock frequency of 250 MHz and a Stratix 10 2800 device at a clock frequency of 400 MHz. After many hardware compiles, 250 MHz was, on average, the frequency the OpenCL design achieved for the Arria 10. Running at 250 MHz provides a peak throughput of 759 GFLOP/s FP32 single-precision performance. The accelerator card was the development kit from Intel, which contains a single bank of DDR4 memory, providing a peak bandwidth of 19.2 GB/s. In many cases, the evolutionary algorithm requested configurations that led to bandwidth-constrained designs. We do provide a few results that included a design space with 2 and 4 DDR banks providing 38.4 and 76.8 GB/s bandwidth respectively. All Stratix 10 models were run with 4 banks of DDR.

GPU results were obtained using three different devices. An NVIDIA Quadro M5000 with 8GB DDR5 capable of 4.3 TFLOP/s of FP32 single-precision performance with 211 GB/s memory bandwidth, a Titan X capable of 12 TFLOP/s of FP32 single-precision performance and Radeon VII 16GB HBM2, 13.44 FP32 TFLOPS and 1 TB/s memory bandwidth. Profiling for GPU was done using trace files generated from Tensor Flow. The timing report considers matrix multiplication, activation, and vector addition routines, but it does not appear to take into account DRAM transfers. FPGA timing reports do consider DRAM because memory buffering is an active component in the design. Overall, conclusions are not affected by the difference between timing reports that likely skews direct comparisons of FPGA and GPU (in favor of GPU).

Power consumption for GPU was captured during runs using the built-in nvidia-smi utility. We found the power management for GPU to be efficient since, in most cases, the effective performance was rather low (see section , and so was the power. On average, the GPU power measured around 50 W for the 150 W (maximum) device. Quartus' Power Analyzer Tool was used to capture power estimates for FPGA. Across the many Arria 10 designs compiled, we found the minimum power to be 22.5 W, maximum power to be 31.89 W, and average power to be 27 W. The difference between power numbers for FPGA and GPU was that FPGA represents chip power and GPU represents total board power. This discrepancy made it challenging to compare power between architectures. Given this discrepancy and the small variance across configurations, we decide to leave the topic of power outside the conclusions.

## Overall Performance Results

We first examine the accuracy results we were able to achieve using the evolutionary flow. Table present the top results obtained from the evolutionary algorithm searching for accuracy using a 10-fold evaluation method. This method splits the data set into 10-equal train/test folds and measures performance on each of the folds. As can be seen, ECAD MLP (our method) was able to achieve better accuracy results then all MLP based classifiers. In addition, the ECAD evolutionary process managed to outperform non-MLP based methods for the credit-g and phishing datasets. The mnist and fashion-mnist datasets were obtained outside of openml use a traditional 1-fold train/test dataset and we compare them to results published in the literature. As can be seen, our mnist and fashion-mnist accuracy results outperform the top reported results. In addition our auto MLP network has the second best reported result and is 0.0047 shy of the SVC method record holder. Table shows ECAD run time statistics for the results reported in Tables and . It reports the number of different NNA/HW combinations that were automatically generated and evaluated by the ECAD system, the average time per evaluation and total evaluation time of all candidate architectures. Note that in order to optimize the search and run time of the system, potential NNA/HW candidates are first analyzed for similarities to previous evaluations and duplicates are not evaluated twice.

Table shows the results for two top Pareto frontier solutions for each data set. Note that this search was done outside of the OpenML spec, i.e., running a single fold. The solutions provide accuracy and throughput for a Stratix 10 (S10) FPGA and TitanX (TX) GPU. In the majority of cases the FPGA achieved higher performance than the GPU. Credit-g, for example, favored GPU for higher accuracy, but looking at the second row for credit-g, by sacrificing just one point of accuracy, the FPGA sees a very significant improvement in throughput.

```latex
\begin{table*}[htb]\centering
\caption{Top 10-fold Accuracy (Acc) for All Datasets Compared to Previous Works}
\label{table:topacc10fold}
\begin{tabular}{|c|c|c|c|c|c|c|c|}
\hline
\textbf{Dataset} & \textbf{Top Acc (Any)} & \textbf{Top Method } & \textbf{Top Acc (MLP)}   & \textbf{MLP Type} & \textbf{ECAD MLP }  \\ \hline
Credit-g      & 0.7860                              & mlr.classif.ranger                & 0.7470                & *MLPClassifier           & \textbf{0.7880}    \\ \hline
Har           & 0.9957                              & *DecisionTreeClassifier           & 0.1888                & *MLPClassifier           & 0.9909             \\ \hline
Phishing      & 0.9753                              & *SVC                              & 0.9733                & *MLPClassifier           & \textbf{0.9756}    \\ \hline
Bioresponse   & 0.8160                              & mlr.classif.ranger                & 0.5423                & *MLPClassifier           & 0.8038            \\ \hline
\end{tabular}
 \begin{tablenotes}
    \small
    \item \textit{Note} The OpenML datasets/results can be found at openml.org: credit-g(https://www.openml.org/t/31), har(https://www.openml.org/t/14970), Phishing(https://www.openml.org/t/34537) and Bioresponse(https://www.openml.org/t/14966). Entries with * denote models from sklearn.
 \end{tablenotes}
\end{table*}
```
```latex
\begin{table*}[htb]\centering
    \caption{Top 1-fold Accuracy (Acc) for All Datasets Compared to Previous Works}
    \label{table:topacc1fold}
    \begin{tabular}{|c|c|c|c|c|c|c|c|}
        \hline
        \textbf{Dataset} & \textbf{Top Acc (Any)} & \textbf{Top Method } & \textbf{Top Acc (MLP)}   & \textbf{MLP Type} & \textbf{ECAD MLP} \\ \hline
        MNIST         & 0.9979                              & Manual                            & 0.9840                & Manual(no distortions)   & 0.9852 \\ \hline
        Fashion MNIST & 0.8970                              & SVC                               & 0.8770                & MLPClassifier            & 0.8923 \\ \hline
    \end{tabular}
    \begin{tablenotes}
        \small
        \item \textit{Note} MNIST and Fashion MNIST are standalone pre-split(1-fold) datasets obtained from Keras dataset collection (https://www.tensorflow.org/api\_docs/python/tf/keras/datasets).
    \end{tablenotes}
\end{table*}
```

```latex
\begin{table*}[htb]\centering
    \caption{Top Accuracy Run Time Statistics}
    \label{table:runtime}
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline
        \textbf{Dataset} & \textbf{Total Models Evaluated } & \textbf{AVG Model Evaluation Time (s)}   & \textbf{Total Evaluation Time (s)} \\ \hline
        MNIST         & 553             & 71.23                     & 39388.6 \\ \hline
        Fashion MNIST & 481             & 82.55                     & 39708.7 \\ \hline
        Credit-g      & 10480           & 2.24                      & 23495.2 \\ \hline
        Har           & 3229            & 10.20                     & 33069.4  \\ \hline
        Phishing      & 3534            & 9.24                      & 32661.3  \\ \hline
        Bioresponse   & 5309            & 5.89                      & 31285.0  \\ \hline
    \end{tabular}
    \begin{tablenotes}
        \small
        \item \textit{Note} Each model generated is a fully functional combination of NNA traits and hardware traits that is evaluated for performance on any of the measured metrics. The ECAD system \textit{caches} similar configurations and avoids reevaluating them.
    \end{tablenotes}
\end{table*}
```

|  **Dataset**  | **Accuracy** | **S10 (output/s)** | **TX (output/s)** |
|:-------------:|:------------:|:------------------:|:-----------------:|
|     MNIST     |    0.9841    |       7.97E5       |      7.73E5       |
|     MNIST     |    0.9763    |       2.45E6       |      1.97E6       |
| Fashion MNIST |    0.893     |       4.8E5        |       8.1E5       |
| Fashion MNIST |    0.8850    |       1.92E6       |       2.3E6       |
|      Har      |    0.996     |       1.16E6       |      9.59E5       |
|      Har      |    0.985     |       4.74E6       |      2.46E6       |
|   Credit-g    |     0.83     |       8.19E3       |      1.59E6       |
|   Credit-g    |     0.82     |       1.40E7       |      1.23E6       |
|  Bioresponse  |    0.798     |       4.64E5       |      1.34E6       |
|  Bioresponse  |    0.7952    |       1.36E6       |      1.66E6       |
|   Phishing    |    0.9675    |       6.81E6       |      2.27E6       |
|   Phishing    |    0.9656    |       1.16E7       |      2.27E6       |

Best Pareto Frontier Results for Searching Accuracy and Throughput

## Accuracy versus Throughput

Part of the motivation for this research is to show the adaptability of reconfigurable hardware to a neural network. This flexibility and molding over a series of GEMM calls produce particular results aimed to fit an optimal network description, i.e., highest accuracy with an optimal hardware solution producing high and efficient throughput. We begin running the evolutionary search over the HAR dataset and observe how both GPU and FPGA reacted to each evolutionary step. Figure provides results for Arria 10 and Figure provides the results for the Quadro M5000.

The evolutionary process provided many high performance results in terms of accuracy with the top at 0.995 and provided many varying results for throughput. The broad swing of throughput results (especially for GPU, being a fixed architecture) shows that many MLP solutions exist that achieve the top accuracy. GPUs accelerate each solution in the same way, so the varying levels of throughput mean the MLP structure is changing. The top throughput GPU achieved at an accuracy of 0.995 is shown to be approximately 1E6 results per second, while the bottom throughput result is approximately 6E5 results per second. For the same reasoning, as we move down a point of accuracy, the GPUs performance hardly changes, this is because of the number of neurons remains roughly the same, it is the distribution between layers causing the effect on accuracy. For GPU, there is roughly no relationship between the number of neurons and the throughput. FPGA has a different correlation. Figure shows that the distribution of neurons in an MLP greatly affects the performance. Unlike GPU, there is (potentially) a different hardware configuration for each data point. While top accuracy only reaches 1.6E5 outputs per second, moving down accuracy just 0.1% results in a giant leap to 1.56E6 outputs per second, an order of magnitude higher. Further, moving down another 0.2% results in approximately another 1.5x in performance. Each dataset tested displayed this same trend.

## Performance Scaling against Bandwidth

Most designs returned by the evolutionary algorithm tended to be smaller, using only a percentage of the available DSPs. The reason is that scaling to more DSPs requires more data, which requires more memory bandwidth. We hit the memory bandwidth roofline many times due to only having a single bank of DDR. Observing this, we ran a series of tests that provided the hardware database model with 2 and 4 banks of DDR. We found mostly a linear scaling going from 1 to 4, so we show the effect bandwidth has on throughput for 1 and 4 DDR banks in Figure . Higher bandwidth did not produce greater efficiency but did result in higher throughput overall.

## Hardware Efficiency and Scaling to Larger Devices

We modified the hardware database worker to return results based on a Stratix 10 2800 device with 4 banks of DDR. This device offers up to a 10x performance scaling from the Arria 10 device we used in previous experiments. While the Stratix 10 device is capable of performing up to 10 TFLOP/s, we used the same methodology as the Arria 10 device to search Stratix 10 devices at a clock frequency of 400 MHz scaling back the roofline to 4.6 available TFLOP/s. To better match the performance of the Stratix 10 device, we used an Nvidia Titan X device capable of 12 TFLOP/s.

We found that overall, the reconfigurable architecture had a more significant potential to perform at a higher level; however, for some data sets, the Titan X raw throughput was able to achieve the highest outputs per second. Throughput is a popular metric to consider, but we also found efficiency to be of high importance. The reason the larger FPGA device handled running these datasets through an MLP is due to the allocation of resources. When the evolutionary algorithm chooses a hardware configuration, there is an allotted space on the FPGA that contains a potential performance. Then after mapping the MLP, we get an effective performance (see section for details on potential and effective performance). The ratio of effective performance over potential performance gives us hardware efficiency. Efficiency could yield fewer FPGA resources while maintaining throughput leaving logic available for other tasks such as pre or post-processing.

Figure shows the FPGA and GPU efficiency of various solutions from searching across the MNIST dataset. The top accuracy for this particular run was 0.9845, and throughput was 796,611 and 773,162 outputs per second for the FPGA and GPU respectively, almost identical. If we consider efficiency for this result, the FPGA utilized 41.5% of the allocated logic, while the GPU only utilized 0.3%. We calculated GPU efficiency as the number of operations per second obtained from a run out of the total potential operations per second of the device. The conclusion is that without target hardware in mind during MLP development, there is a good chance of losing efficiency. Running an evolutionary search on reconfigurable hardware can balance the fitness between throughput and accuracy.

# Conclusions

We address the difficulty of designing highly performant neural networks by leveraging evolutionary search algorithms capable of finding the fittest solutions for both classification accuracy and hardware throughput. This process is shown to be both highly efficient and effective compared to traditional approaches that first design a neural network to achieve a target accuracy, then run it on general-purpose hardware. Through a series of experiments, we present our results for state of the art neural network configurations that surpass current published work. We explain the power of co-design by discussing the results of experiments showing accuracy versus throughput, performance scaling with bandwidth, and scaling designs with larger devices.

# Future Directions

We believe a custom co-design framework for MLPs and reconfigurable hardware could be of use to a large audience. We plan to continue and develop this framework, streamline its use and make it available to the general public.