Question:
represent the article for finding a single-sentence tldr

------

Examples:
Provided: Humans have a natural curiosity to imagine what it feels like to exist as someone or something else. This curiosity becomes even stronger for the pets we care for. Humans cannot truly know what it is like to be our pets, but we can deepen our understanding of what it is like to perceive and explore the world like them. We investigate how wearables can offer people animal perspective-taking opportunities to experience the world through animal senses that differ from those biologically natural to us. To assess the potential of wearables in animal perspective-taking, we developed a sensory-augmenting wearable that gives wearers cat-like whiskers. We then created a maze exploration experience where blindfolded participants utilized the whiskers to navigate the maze. We draw on animal behavioral research to evaluate how the whisker activity supported authentically cat-like experiences, and discuss the implications of this work for future learning experiences. Posthumanist philosophies characterize the human body as "the original prosthesis we all learn to manipulate", and suggest the idea that augmenting or substituting aspects of this prosthesis is the normal progression for humanity. Technology allows humans to enhance senses that may be impaired, and to extend our bodies with added senses beyond what we would otherwise be biologically limited by-giving humans the ability to improve their quality of life. " In short, we are cyborgs". Scholars have investigated how immersive virtual environments can enhance social perspective-taking, and computer-augmented, embodied perspective-taking has been shown to encourage a productive "learning stance" and to enhance both conceptual learning and engagement. Some environmental education scholars and indigenous educational scholars have suggested that building relational ties to non-human actors in nature may contribute to environmental and biology education. In a few cases, educators have asked learners to take on the embodied experiences of insects such as bees and animals such as polar bears. Danish found that children enacting a computer-augmented pollination activity embodying the roles of bees helped them learn nuances of individual and aggregate bee behavior; Lyons and colleagues found that wearable polar bear paws that simulated the feeling of traversing melting polar ice enabled people to show an empathetic understanding of the impacts of climate change. For many people, the most common experience they will have with entities who have different sensory capabilities is through everyday interaction with pets or neighborhood animals. For example, in noticing that our house cat is navigating a dark space where we would likely bump into something, we may recognize the limits of our own senses and consider how our pets' experiences are both similar to and different from our own. Our work explores how embodied technology can mediate human experiences in ways that offer people opportunities to explore and relate to the animal-like behaviors of their pets. We present the design of a cat-inspired whiskers wearable, the Whisker Beard, and an embodied navigation activity that provided a firsthand perspective-taking experience for participants curious about what it might be like to have whiskers. In addition, we discuss our philosophy of what it means for an animal-imitating experience to be authentic and we present the evaluation framework we used to understand how our whiskers activity encouraged participants to behave like cats. Our study addresses two research questions: In what ways can we create technologies and environments that remediate human experiences to be like those of non-humans? RQ2: What are humans' impressions of these technologically remediated experiences? In this paper we describe the design of a sensory augmentation whiskers wearable; the creation of a maze exploration activity for testing the experience of wearing whiskers; our analysis methods for evaluating the authenticity of an animallike experience; and outline opportunities to extend this work, as well as discuss the implications of it. Our technology, experience design, and analysis are motivated by a desire to re-shape science and science education in light of feminist critiques and visions of those fields. Whereas science and science education today tend to emphasize distance, objectivity, and dispassion, we strive to create spaces for discovery and learning that also include closeness, subjectivity, and emotion and that thereby enable learners to author identities in science based on more engaged and connected ways of knowing. The overall approach of the project is to investigate how affection for and curiosity about pets can catalyze scientific investigations and engineering for young people and their families that are based on empathy and perspective-taking, both in and out of schools. We see value in using wearable and mixed-reality technologies to provide humans with the ability to experience the lives of other beings. In particular, this would allow humans to experience what their pets' senses might be like, and thereby facilitate learning experiences that encourage empathy and perspective-taking. Once we can support interspecies perspective taking, we then wish to encourage participants to conduct scientific inquiry within that intersubjective sensational realm -a realm which the German biologist Jakob von Uexküll called "Umwelt". This agenda has the potential to unify efforts to advance scientific and social-emotional education. Humans have strong emotional attachments to their pets and these human-animal bonds coincide with higher amounts of empathy. This can motivate people's curiosities about their pets' lives and experiences. One technological and philosophical challenge involved in pursuing our agenda is the intrinsic disconnect between humans' experiences and those of other species. People only know what it is like to be human, and are unable to know on a phenomenological level what it is truly like to be their pets. Though people cohabitate with their pets, on a biological level they see, hear, smell, taste, and experience the world differently from them-ranging from slight variations of senses to things incomprehensible and alien like sonar and magnetic field detection. Despite this challenge, most pet owners believe that their pets feel something, even if they cannot fully understand what they feel. People naturally evaluate animal behavior and experience through a human lens. In a thought exercise of imagining what it is like to be a bat, Nagel highlights the mind-body problem he encounters: "... I want to know what it is like for a bat to be a bat. Yet if I try to imagine this, I am restricted to the resources of my own mind". While we will likely never solve the mind-body problem of humans truly understanding their pets' experiences, we can address the "body" aspect of the problem and use it to drive thought experiments about the lives of pets. These thought exercises can give people a better sense of animals' lives by acknowledging biological differences that many people have about how their pets see, hear, and feel the world. In this study, we present the of an early step toward feminist science and science education: the design and evaluation of a wearable device meant to offer humans the experience of exploring an environment using whiskers. This first step is intended to both elucidate how new technologies can offer transspecies sensory experiences, and to show how we might assess the validity of those experiences (i.e., the extent to which they immerse humans in the reality of another species) by comparing humans' behaviors in the new technologically-mediated umwelt with those whom the umwelt natively describes. Wearables and mixed reality technologies have been applied in a wide variety of domains ranging from educational contexts, medical settings, natural environments, performances, and museums. The ing hybrids of wearables and mixed reality technologies are sensory augmentation devices. Sensory augmentation devices give humans the ability to experience phenomena that they are physically unable to process, as well as some phenomena that are simply unnatural to the human experience. The field of sensory substitution and augmentation enables humans to use existing senses to substitute for the ones that they cannot experience, and to augment senses they already have to make them more powerful. For example, sensory substitution has been applied as a method for offering hearing and/or visually impaired people additional senses to substitute for the one in which they have an impairment. These technologies range from devices that can be implanted within the body, sending direct signals to internal mechanisms in the brain, to external substitute devices that provide tactile feedback on the surface of the skin. In one example of tactile feedback, researchers developed a non-invasive "vibratory vest" for deaf and hearing impaired individuals that processes auditory information and converts it into vibrotactile feedback on the wearer's torso. Our work builds on approaches that use these technologies to provide physical, sensory feedback experiences for the wearer. Wearables have the ability to provide hands-on, interactive, and embodied learning experiences. There is research to suggest that wearable technologies in the classroom can increase engagement and improve student attitudes towards STEM activities. One of the pedagogical affordances of wearable technologies is the ability to gather contextual information from a firsthand account. Some uses of wearables in education focus on understanding and quantifying the self, such as: calculating our heart rates, monitoring how many steps we have taken, and providing feedback about our emotional states. Other approaches embed the quantifying potential of wearables in scientific inquiry and discourse. In contrast, our research focuses on how wearable technologies can help people understand experiences beyond the self, more specifically, the potential for wearable technologies to foster empathy by helping humans understand the experiences of others. Incorporating empathy and perspective-taking into scientific question-asking is deeply rooted in feminist educational theory and practice. These practices offer a more inclusive view of what it means to actively participate in science, and promote empathy as a valued part of scientific discovery. An example of a wearable device aimed to motivate empathy is a series of mushroom foraging tools designed to build more intimate relationships between humans and the environments they are probing by connecting them physically to the environment. Nobel laureate Barbara McClintock cited her ability to get "a feel for the organism" as influential in supporting her discoveries in the area of maize cytogenetics. McClintock has said, "I know my corn plants intimately, and I find it a great pleasure to know them," a sentiment we believe could inform future wearable-mediated science education. When interacting with non-human animals, it is easy to notice how their sensory capabilities differ from humans'. For example, people may notice their cat hear or see something without hearing or seeing it themselves. In addition to animal companions' sharper senses, people may also notice qualitative differences in how they experience the environment, such as when their pets deftly navigating through small spaces. People may also notice and wonder about the presence of physical characteristics that humans lack, such as whiskers or purring behaviors. Our work in this paper is motivated by human curiosity about these differences between species and explores how taking on the physical or perceptual characteristics of an animal helps people to understand it. In this study, we specifically focus on how participants' behavior compares to navigation and foraging techniques that animals exhibit in the wild and in controlled laboratory maze experiments. An animal's ability to gather environmental information and make decisions in real-time is key for its survival. Animals must constantly make decisions and consider tradeoffs during activities like foraging for food, avoiding predators, and searching for a mate. These trade-offs describe what is called the exploration-exploitation problem, a key idea in organizational learning and animal behavioral science. A common definition describes exploration as the process of randomly searching an unrefined and unexplored area, and exploitation as the process of searching a more refined area for some reward or resource. The less information an animal has about an environment, the more likely it is to be exploring rather than exploiting. Researchers argue that optimal search strategies for animals include alternating patterns of quick movement and searching. In the process of exploring and exploiting an environment, animals exhibit "egocentric" and "geocentric" navigational strategies. In an egocentric strategy, an organism uses previously acquired information, either from memory or another internal mechanism, to move along a path. In a geocentric strategy, an organism uses cues and real-time feedback from the environment to continuously reorient itself along a path. Animals rely heavily on geocentric strategies to explore and search unfamiliar areas; it is too difficult to primarily rely on internal mechanisms when moving through new territory. The term thigmotaxis describes the behavior of either moving toward or away from touch stimulus; an example of this behavior is "wall-hugging" in animals, which is the tendency to avoid open areas and stick to the perimeter of an environment during navigation. Wall-hugging behavior in animals is commonly tested in laboratory maze environments, where high levels of wall-hugging can suggest anxiety or fear in an animal. Therefore, wall-hugging is commonly seen as a strategy that animals invoke when exploring a new and unfamiliar environment. For example, mice spend, on average, 74% of their time hugging a wall in their first five minutes exploring an unfamiliar environment, and about 65% of their time doing so by thirty minutes. We created a sensory augmentation device, the Whisker Beard, that provides a physical simulacrum of what it is like to have whiskers. We wanted our whiskers to be functionally and aesthetically reminiscent of cat whiskers, including the appearance and placement of the whiskers and the physical properties of the materials. In addition, we wanted the functionality to be authentic in the sense that a person wearing it would use the whiskers to enact behaviors similar to animals'. Whiskers, formally known as vibrissae, are rigid but flexible hairs that provide tactile sensory feedback in many mammals and rodents. Ahl describes a number of characteristics and roles for whiskers; whiskers differ from hair in that their follicles are much thicker than normal hair follicles, and that whiskers are sensitive and send signals to the brain. Whiskers are commonly localized to facial regions of animals, but can also be found on other parts of the bodies of animals. They are essential to survival, monitoring, communication, and aggression. They help serve as a sensory substitute for animals with poor close-sightedness; cats have difficulty seeing objects very close to their faces, so whiskers help them sense objects close to them, as well as protecting their face from harmful objects. A project close to our vision is a helmet that uses infrared sensors to detect close-by objects and provides vibrotactile feedback to the wearer; like our project,'s design was inspired by mammals' abilities to use whiskers to detect their surroundings. Our project and theirs are similar in that they are whisker-inspired designs for sensory augmenting wearable devices, however our goals differ in that we are not trying to offer an efficient solution for humans to navigate in low-light conditions, but rather to provide perspective-taking experiences for wearers to experience what it is like to have and use whiskers. The version of the Whisker Beard used in this study has a total of four whiskers, two for each side of the face. In order to mimic the genal (cheek) whiskers of the cat, our whiskers attach to the sides of two acrylic cheek plates that rest 2-3 cm above the skin of the wearer, therefore the whiskers protrude from the sides of the wearer's cheeks close to the mouth (see Figure 1). Each whisker's total length (38.5 cm) is approximately the average human's shoulder width (39 cm). We chose this length in order to make the whiskers extend beyond the shoulders of an average sized person, thereby imitating the appearance and functional reach of cats' whiskers. Figure 1 shows a researcher wearing the Whisker Beard. The whiskers are composed of three flexible components: a flex sensor that detects the deflection that the whiskers are receiving, polystyrene strips that extend the usable length of the flex sensor significantly and returns the flex sensor to a neutral unbent position, and Sugru, which is a moldable silicone glue that holds the polystyrene strip and flex sensor together. In addition, a small connector at the end of the whisker attaches to the cheek (see Figure 2). Our device detects when the wearer brushes their whiskers against a surface. When the whiskers are bent the change in resistance is measured and converted to a voltage. The calculated voltage measurement is then used to set the pulsewidth-modulation signal to the vibration motor control. The device provides vibrotactile feedback to the wearer by varying the intensity of vibration proportional to the bend angle through an array of vibration motors on the scalp, inspired by the work from. Each motor is coupled to one whisker. We chose to place the vibration motors on the scalp so the vibrotactile sensation would be felt on the wearer's head, and so the motors could be placed far enough apart to allow for better point discrimination (making it easier for the wearer to identify which specific motors are vibrating). The whiskers sense bidirectionally in order to better mimic a cat's actual sensory perceptions when approaching and backing out of different confined locations. The current hardware design allows for bidirectional sensing, but the software does not explicitly indicate direction of the whiskers' bends through haptic feedback, however, the participant can feel the direction of tension on the wearable. In order to study how the Whisker Beard can support animallike environmental exploration, we built a human-sized cardboard maze (4m x 6m) for our participants to explore while wearing the whiskers wearable. Mazes are a common tool in behavioral research involving rats and smaller mammals. Our hypothesis was that participants would be able to use the whiskers as a workable form of sensing during maze navigation. We wanted participants to rely on the whiskers as heavily as possible and other senses as little as possible. Therefore, we chose to blindfold the participants since vision is the dominant sense for non-visually impaired humans for gathering environmental information. We designed the maze to promote enough confusion for participants so they would need to use feedback from the whiskers to guide them. We did so in order to observe a more honest range of animal behavior. Before creating the human-size maze, we created a series of small-scale prototypes. We used these to consider different possible maze routes as well as what kinds of obstacles would promote explorations of the whiskers' affordances. We prototyped obstacles that would encourage participants to enact cat-like behaviors such as rubbing one's face on a surface, and having to back out of narrow spaces. We came to the that experimenting with unfamiliar topography was the best way to create an experience that removes participants from a human sensory experience within an exploration task. Therefore, the final maze design included multiple dead ends, corridors of different widths, vertically hinged flaps, and cutouts in the walls. In addition, due to the bidirectionality of the whiskers, we designed our obstacles such that wearers would mostly need to rely on the horizontal changes in the whiskers' shapes. Figure 3 shows a photo of the finished cardboard maze, and Figure 4 shows a digital rendering of an aerial view of the maze. We placed 10 toy mice throughout the maze to induce exploration and exploitation behavior. We gave participants this task in order to give them a survival-oriented goal to collect as many resources as possible from an unfamiliar environment in a limited time frame to strengthen the worldview of being an animal. We investigated how participants used the Whisker Beard to navigate the maze, and in our analysis we focus on what ways participants did or did not exhibit animal-like behaviors during the experimental activity, and how they reacted to the experience of cat-like sensory augmentation. To do so, we evaluate how participants behaved like cats, as well as what participants' impressions were about the activity. We draw on work from animal and biological sciences, particularly key areas of research related to navigation and animal behavior. In addition, we highlight moments from participant discussion that illustrate ideas and questions they had after the experience. We recruited six undergraduate college students who lived on campus in the dormitory that we constructed the maze in. Due to the circumstances of living in a dorm room, none of the participants currently owned a pet, however three of the participants had previously owned a cat, or still had a cat at home, and the participants who had not owned a cat had owned a dog and also knew family and friends who have had cats. All six participants completed the maze activity on the same day in subsequent 20 minute sessions. As participants arrived for their sessions, we greeted them individually in a separate room. There, we assisted each participant in attaching the Whisker Beard to their face. We spent a few minutes explaining how the device works and gave participants time to learn the correlation between whisker bend and vibration location and intensity. We did this by individually flexing each whisker while the participant was wearing it (prior to blindfolding them) and asking them to describe if they could feel the changes in the vibrotactile feedback. Once participants were familiar with the wearable, we explained the maze procedure to participants, and brought them -blindfolded -into the maze room. We replaced all 10 toy mice in the same locations before the start of each participant's session (see Figure 4). We gave each participant a maximum of 10 minutes to crawl through the maze and explore, and to collect as many toy mice as they could. We asked participants to think-aloud during the maze so we could have a better sense of their thought process throughout the activity to both validate our coding of the data, as well as collect information about participants' impressions of the activity. After each participant completed the maze activity, we asked them to refrain from discussing the experience with other participants until everyone was finished. We had them each fill out a worksheet to provide responses to questions like, how did it feel to navigate using whiskers, and what moments stood out to you when using your whiskers to navigate the maze? We selected open-ended questions such as these to gather information about participants' experiences on an individual level and to get their initial impressions right after the activity. After all participants were finished with the activity, we reconvened as a full group for a facilitated discussion. We began the discussion with three prompts on the board: "How did you feel when you first put the whiskers on?" "What particular moments in the maze stood out to you?" and "Other thoughts?" We asked participants to write their thoughts about these prompts on sticky-notes and place them on the board. Once participants completed this, we used the responses to facilitate discussion among the whole group. In order to investigate how the wearable and maze activity mediated participants' experiences in an animal-like way (RQ1), we analyzed participants' interactions, behaviors, and commentary throughout the activity through coding of the data and analysis of their movement. We assessed interrater reliability by having each coder annotate a map of the maze using the process shown in Figure 7. We compared each annotation on both maps. If the coders identified an action as a different event type as shown in Figure 6, or if one coder identified an event at a particular time and the other did not identify an event, these were marked as disagreements. We then added up all the agreements and disagreements to calculate with Figure 5. A visualization of participants' usage of their whiskers and hands over time. Note that Jodie's time is much shorter than other participants due to the fact she moved through the maze very quickly and returned to the starting point after about four minutes. Cohen's Kappa. For example, to assess agreement on where "whisker interactions" occurred, we looked at where the raters identified whisker interactions on their maps, and counted all matching identifications as agreements, and all discrepancies as disagreements. We did this for all symbols in the key and totaled the . After reaching an inter-rater reliability of k=0.62, which indicates substantial agreement, the two researchers coded the rest of the data in parallel. To analyze the paths of participants and their interactions, we captured audio and video recordings of participants' movements and interactions in the maze from five cameras: we placed four cameras in different positions along the maze's perimeter, and a researcher held one camera and followed the participant as they moved through the maze. We analyzed the videos to produce content logs and time stamps of different events. In addition, we used the videos to manually generate aerial maps that depict participants' movements and interactions through the maze. Figures 6 and 7 illustrate the orientation, position, and scale of the participant, as well as depict participants' interactions with their whiskers and hands at a given time. In addition, we denoted four subsections of the maze that we thought represented the different types of obstacles and areas (see Figure 4). The red zone is where the maze starts and contains two straight hallways. The yellow zone contains a large area of open space, as well as a wide opening that narrows to a dead end. The blue zone contains a straight hallway that opens into three smaller corridors. The green zone is a long hallway with cardboard flaps hanging on either side that participants had to push through. In Shapiro and Hall's museum mapping work, they created aerial maps to illustrate museum visitor movement and engagement to understand how the space created by the gallery facilitates learning. We apply their mapping methods in our study in order to identify how participants interacted with the space with the Whisker Beard and how their interactions and movement compared to animal-like behavior (see Figure 7 for examples of the paths we generated for analysis and refer to Figure 6 for our key). Figure 6 for our key. The map on the right contains color so that we could draw overlapping paths and keep track of the order in which it occurred (such as when participants doubled back to specific areas). In our analysis, we define moments of primarily whisker interaction as exploration, and moments of hand-swiping interactions as exploitation. We encouraged participants to solely rely on their whiskers to navigate the maze, and to only use their hands when they needed to collect mice. Because of this, we find the distinction between whisker use versus hand use as a reasonable indicator of exploration-exploitation behaviors. Finally, we coded moments of geocentric and egocentric strategies using the perspectives defined earlier, and coded wall-hugging to be moments where a participant makes con- tinued contact with their body and or whiskers against the wall. We separate our into four categories of participants': patterns of exploration and exploitation and how they related to whisker use; geocentric and egocentric strategies; wall-hugging behaviors; and reflections on the experience. The first three are aimed at providing information about the authenticity of the animal-like behaviors that participants exhibited (RQ1), while the fourth addresses what impressions people had about the wearable and the experience (RQ2). Participants alternated between periods of long exploration and relatively shorter exploitation. In Figure 5 we illustrate the exploration-exploitation search behaviors of each participant (listed with pseudonyms). Blue segments denote periods of navigation where participants were primarily relying on their whiskers to move around. Red segments denote moments where participants moved their arms across the floor in order to search for and obtain mice. Yellow segments are places where we coded participants as using both exploration and exploitation, which was rare because crawling and swiping is challenging. In Table 1 we show the breakdown of the amount of time participants spent exploring and exploiting the maze. Participants spent an average of 70.3% of time exploring and 35.8% of time exploiting. Participants switched back and forth between exploration and exploration, with an average exploration interval length of 15.2 seconds and an average exploitation interval length of 7.4 seconds. On average, participants caught seven out of the ten mice. During periods of whisker exploration, participants enacted three common whisker techniques that primarily occurred in the long, straight parts of the maze (red zone in 4). In Figure 8 we illustrate what these three techniques look like. In one technique (A) the wearer constantly drags the whiskers against the wall as they move. In the second (B) the wearer alternates between motion and pausing to brush the whiskers against the wall, and in the third (C) the wearer moves between walls alternating between the sides of the whiskers they use. All six participants enacted at least one form of A and B during the maze, and two participants utilized the C technique as well. The two participants that enacted all three techniques both have pet cats at home. Participants used a variety of geocentric and egocentric search strategies as they navigated the maze. Table 2 shows the of this categorization and demonstrates the different geocentric and egocentric strategies that participants enacted, with examples and quotes to illustrate how they used particular environmental feedback. We found that all six participants relied on enacting geocentric strategies throughout the entire activity, and used egocentric strategies more sparingly. During the think-aloud all six participants made comments about using touch and sound feedback from the whiskers, as well as tactile feedback from their hands and body. Only two participants made comments about attempting to use egocentric strategies, with one participant commenting that he was able to create an internal map during the maze (which he later admitted was incorrect after seeing the maze). All six participants exhibited wall-hugging behavior (thigmotaxis) as they moved through the maze blindfolded. Sound of the whiskers touching the walls "The navigation is also noise of the whiskers touching things." Egocentric Memory of the space from an earlier point in time "I know I'm in the center of the room, because I've been in this room before with this outlet on the floor" Memory of revisiting the same part of the maze "I think I'm going in circles" Internal recall of physical orientation "[I was] thinking about my previous and next moves, as well as an internal map." Table 2. A breakdown of geocentric and egocentric navigational strategies that participants used. Five of the six participants spent over half of their time in the maze hugging the wall, with one participant hugging the wall for 97.3% of her maze time. On average, participants spent 68.6% of their time in the maze hugging the wall (min=34.6, max=97.3, standard deviation=20.87). Participants' tendencies to hug the wall varied, but interestingly, the participant who hugged the wall the least had a higher tendency than others to bump into obstacles head first (14 times). Participants frequently wall-hugged while exploring using whisker technique A from Figure 8. Participants reflected on their experiences in the whisker activity through individual worksheet responses and a full group discussion. During their reflections, participants commented on the physical experience of the whisker activity, and drew connections to phenomena they have experienced with their own pet cats or others' cats. In response to the question, "How did it feel to wear the whiskers?" participants described both the physical sensation of how it felt to have whiskers, and the functional use of the whiskers and their ability to adapt to it. Responses that described the feeling of adapting to the whiskers include: • " As I got used to them, the whiskers started to become a part of me." • "I liked having another sense. They got much easier to use as I played around with angles and pressure on the sensed surfaces." • "It was an easy way to'see' side to side. Times in empty space though stood out, with nothing to feel in front of me made me more cautious" • "The first time I went through a small space and both whiskers activated [while] having to back out...stood out to me" • "It was a bit odd at first but I quickly got used to them. It was kinda nice having an additional aid apart from the feeling in my hands and feet. " Responses that described the more physical feeling of having vibrating whiskers include: • "I had to push through the flaps and it was almost overwhelming with vibrations." • "It was an interesting feeling having the vibration kind of tickle. When I was really close to something it was also kind of shocking and made me want to back away." Most of the above responses to the question of how it "felt" focused on the overall experience of using the whiskers to navigate, as opposed to the specific physical feeling of the vibrations on the skin. In addition, we were curious whether participants would think about their own pet cats, or other cats that they know. We asked, "Did you think about your own pet cat while you were in the maze? If so, what did you think about?" and participants responded in ways that included perspective-taking commentary that were empathetic. Their responses included: • "I thought about my friend's cat and how when she was five, she cut off the cat's whiskers thinking they were long hairs. For a good week the cat had to stumble around, falling over, and running into things often." • "How cats sometimes bump their heads into things then back-up confused. I could sympathize." • "Yes I did, I thought about how the vibration was a little like how they would use their whiskers. I also thought about how hard it would be to navigate without her whiskers" Most of the empathetic responses show participants acknowledging that it would be difficult to navigate as a cat stripped of its whiskers, similarly to how it was difficult for them to navigate without their sight. Participants described moments of feeling disoriented when they entered the large open area of the maze (yellow zone in Figure 4), and during moments of technical difficulties when a whisker fell out. According to participants, the open areas were disorienting because they lost their sense of physicality and location, suggesting some understanding of, and potential for empathy with, animals' thigmotactic strategies. The of the Whisker Beard and maze activity show examples of participants exhibiting behaviors and strategies similar to those that animals perform when searching for resources in unfamiliar environments. We separate our into discussions about their physical behaviors and strategies, as well as their impressions of the experience. As depicted in Figure 5 and Table 1, as participants explored the maze, they alternated between periods of explorative and exploitative behavior as they switched between using their whiskers and using their hands. Participants spent, on average, a longer amount of time exploring and moving through than maze than they spent hand swiping to look for mice. These are in line with animal foraging behaviors. Benichou et al. says that animals searching for resources switch between periods of motion and periods of searching. In addition, their work shows that intervals of exploration tend to be longer than intervals of exploitation. This aligns with the amount of time our participants dedicated to these behaviors. While we cannot claim that participants would not have enacted similar exploration-exploitation behaviors without whiskers, we can say that the behaviors that they enacted with whiskers were in line with foraging behaviors. Interestingly, several of the participants made use of the whiskers in ways that strikingly resembled cat behavior, as depicted in Figure 8. As participants moved down long passages, some used their whiskers to gauge the width of the passage by moving back and forth brushing each side of their whiskers on the opposing walls. This demonstrates that participants used the whiskers to enhance their spatial awareness, one of the supposed evolutionary factors behind the presence of whiskers. We noticed that when participants used techniques B and C, they mimicked the behavior of cats who rub their olfactory face glands on objects to mark their scent, as well to get a sense for the physical properties of a specific object. While this behavior in cats is not necessarily used for navigation purposes, it is used for gauging the size and shape of an object. Participants did this in order to look for hidden passageways and moveable obstacles. Our observations of participants' geocentric and egocentric behaviors provided us with a fuller picture of how participants used the whiskers in tandem with other strategies during the activity. Participants relied on the vibrotactile feedback from the Whisker Beard in determining their path of movement through the maze. In addition to the vibrotactile feedback, we found that participants also relied on the sounds the whiskers made as they brushed against the maze's cardboard surfaces. We validated this observation through think-aloud commentary that participants provided throughout the maze, and through post-maze group discussion. The fact that participants relied on additional tactics beyond the vibrations is not an inauthentic outcome, but rather a reasonable one. Participants' use of different egocentric and geocentric tactics is naturally aligned with how animals navigate the world-getting the most information from their environment by whatever means are accessible to them. The blindfolded maze procedure afforded participants the ability to experience the Whisker Beard in an unfamiliar environment. As expected, due to the unfamiliarity of the environment, participants relied on more geocentric strategies. These are in line with animal navigation research which suggests that egocentric strategies are too difficult to use when exploring new terrain, and therefore animals rely more heavily on geocentric strategies to gather real-time physical feedback. In time, participants who revisited areas of the maze began to recognize their surroundings, which led them to use internal recall from their memory to identify their approximate position; however, because they were blindfolded they still had to rely on geocentric strategies as well. Unsurprisingly, participants told us that being blindfolded and losing their sense of sight was disorienting for them; sight is one of humans', and cats', dominant senses for obtaining information about an environment. Participants described the open-space areas of the maze as "disorienting" and tended to try to find a wall as quickly as they could to reorient themselves. The level of consistent wall-hugging that participants exhibited is in line with experiments where increased levels of anxiety correlated to higher levels of thigmotaxis. Usually, animals' tendency to hug the wall would decrease as an experiment went on, except in circumstances where animals do not have enough time to fully process their environment. In our experiment, blindfolding the participants made it challenging for them to produce an accurate internal map of the space, leading them to continuously avoid open areas and rely on vibrotactile and audio feedback from the walls during navigation. The participants' reflections during the maze and post-maze show promising beginnings to meaningful discussions of animal empathy, as many drew connections to prior experiences of pets who were blind, deaf, or had their whiskers cut off and discussed how disorienting and difficult it would be for them to navigate with a sense removed. Participants described the whiskers as feeling like an extra sense, one that they were able to adapt to even in a short timeframe. Although losing their sight was disorienting, they were able to utilize the whiskers as a substitute for being able to "see." The combination of the Whisker Beard and maze activity suggests that through disorienting them and having them behave like a cat, they were able to consider what it would be like to be a cat relying on its whiskers every day, and how challenging it would be for a cat who has no whiskers at all. The most severe technology design issue we encountered during the study was whisker placement; each participant noted that the lack of having a front-facing whisker on the forehead made it difficult to avoid obstacles directly in front of the wearer. Cats have a set of superciliary or suborbital whiskers on their brow for this very purpose, and the lack of these front-facing whiskers on our wearable was frustrating for participants. We did not include them in the original design because we wanted to focus on the whiskers on the sides of the faces, not realizing how important it would be to include them in other areas on the head. Although participants were able to make use of the whiskers during this study, we believe that a longer study would give participants a more adequate amount of time to adapt to the whiskers and therefore exhibit more natural behaviors, such as participants relying more on the whiskers than their hands during resource collection. It might also allow a different set of behaviors to emerge, as wearers become more comfortable in the environment and reduce thigmotaxic behavior. We continue to investigate how the whisker wearable, and technologies like it, can remediate human experience in order to support deeper intersubjectivity with animals, and how such remediations can offer an experiential framework for science and science education. To address the aforementioned technological limitations and move towards a more authentic design, we are working towards a more modular and customizable design. The next generation of the wearable will include additional sensing capabilities that support cat senses beyond whiskers, like hearing. A new custom board, Dr. Bones (Figure 10), will serve as a connection hub and accommodate the micro:bit as the primary controller for all input and output modules. Incorporating the micro:bit into our design will enable participants to program their individual modules using the Makecode programming environment. In exposing the individual sensing and output elements of this project we aim to encourage young people to create their own sensory augmentation systems. We conjecture that this customizability will offer a variety of ways for people to engage with scientific ideas relating to animals. In addition to offering customizability for sensors, we will be offering people the ability to customize the placement of sensors and attachments on the wearable. For example, participants who wished they had a front-facing whisker during the activity will be able to create one. This will be more authentic to the way cats' whiskers work because cats and other mammals are able to direct control over the direction of their whiskers, as well as other parts of their bodies like their ears. Further, we are addressing the lack of vibrotactile directional feedback by adding three motors per whisker to our design; therefore, through motor sequencing and drive intensity, we can render the direction and angle of the bend human sensation. One intriguing aspect of our work in adding whiskers to humans was the ability to create new types of empathetic experiences. While our initial work has focused on developing new technology and the analytical infrastructure needed to test it, we see several opportunities for more deeply exploring the experiential aspects of this technology. First, we may consider how to explore the transhumanist aspects of gaining a new sensory capability. Can people find uses for wearable whiskers in their daily activities or as part of their everyday lived experience? By creating a more portable, lightweight version of the hardware, we may explore opportunities to send this device into the world to see what people make of it. Second, we may consider how wearable whiskers can increase understanding of, and empathy for, the experiences of nonhuman animals. For example, by leading a person through an experience similar to the everyday behavior of a feral cat (sneaking through backyards, chasing birds, searching for edible items), can wearers of the whiskers better comprehend and empathize with the experiences of feral cats? Finally, we may explore how experiences with wearable whiskers could increase an individual's understanding of, and relationship with, their own pets or other familiar animals. For example, children often must be taught what kinds of touch are liked and disliked by their pets; until they learn this, they may be frustrated by their pets' apparent distrust or fear when they are nearby. We may explore how to design experiences that can help a human understand a particular aspect of their pets' lived experience as a way of supporting a more respectful relationship between species. Wearable technologies and embodied learning experiences free humans from the confines of their biological limitations. This enables researchers to provide low-cost opportunities that offer firsthand perspective-taking experiences for people, allowing people to experiment with new sensory interactions, including ones that non-human animals have access to. We presented the design of the Whisker Beard, which does just that-provides humans with the opportunity to experience what it would be like to have a new sense, in this case, what it would be like to have whiskers. We introduced concepts from animal behavioral science research and described how we applied it to evaluating the experiences of participants' while immersed in an animal perspective-taking activity. Our observations of participants' enactment of animal-like behaviors, as well as their impressions about the experience suggest that they were immersed in the sensory experience of being a cat with whiskers. We are actively iterating on the designs of our hardware to offer more customizability. This will enable participants to design their own sensory augmenting technologies where they can explore their own curiosities about their pets' other senses. In near-future experiments we will iterate on the design of the wearable activity to offer a more immersive experience where participants can continue to enact animal-like behaviors. Our next steps will then be to investigate how participants developing increased awareness of animals' sensory experiences can support their enactment of empathetically-oriented design activities focused on improving animals' quality of life. Match: This paper explores using wearable sensory augmenting technology to facilitate first-hand perspective-taking of what it is like to have cat-like whiskers. Hard Negative: We explored the visualization designs that can support chronic patients to present and review their health data with healthcare providers during clinical visits.

Question: Recent improvements to Generative Adversarial Networks (GANs) have made it possible to generate realistic images in high resolution based on natural language descriptions such as image captions. Furthermore, conditional GANs allow us to control the image generation process through labels or even natural language descriptions. However, fine-grained control of the image layout, i.e. where in the image specific objects should be located, is still difficult to achieve. This is especially true for images that should contain multiple distinct objects at different spatial locations. We introduce a new approach which allows us to control the location of arbitrarily many objects within an image by adding an object pathway to both the generator and the discriminator. Our approach does not need a detailed semantic layout but only bounding boxes and the respective labels of the desired objects are needed. The object pathway focuses solely on the individual objects and is iteratively applied at the locations specified by the bounding boxes. The global pathway focuses on the image and the general image layout. We perform experiments on the Multi-MNIST, CLEVR, and the more complex MS-COCO data set. Our experiments show that through the use of the object pathway we can control object locations within images and can model complex scenes with multiple objects at various locations. We further show that the object pathway focuses on the individual objects and learns features relevant for these, while the global pathway focuses on global image characteristics and the image . Understanding how to learn powerful representations from complex distributions is the intriguing goal behind adversarial training on image data. While recent advances have enabled us to generate high-resolution images with Generative Adversarial Networks (GANs), currently most GAN models still focus on modeling images that either contain only one centralized object (e.g. faces (CelebA), objects (ImageNet), birds (CUB-200), flowers (Oxford-102), etc.) or on images from one specific domain (e.g. LSUN bedrooms, LSUN churches, etc.). This means that, overall, the variance between images used for training GANs tends to be low BID14. However, many real-life images contain multiple distinct objects at different locations within the image and with different relations to each other. This is for example visible in the MS-COCO data set BID11, which consists of images of different objects at different locations within one image. In order to model images with these complex relationships, we need models that can model images containing multiple objects at distinct locations. To achieve this, we need control over what kind of objects are generated (e.g. persons, animals, objects, etc.), the location, and the size of these objects. This is a much more challenging task than generating a single object in the center of an image. Current work BID10 BID9 BID6 ) often approaches this challenge by using a semantic layout as additional conditional input. While this can be successful in controlling the image layout and object placement, it also places a high burden on the generating process since a complete scene layout must be obtained first. We propose a model that does not require a full semantic layout, but instead only requires the desired object locations and identities (see Figure 1). One part of our model, called the global pathway, is responsible for generating the general layout of the complete image, while a second path, the object pathway, is used to explicitly generate the features of different objects based on the relevant object label and location. The generator gets as input a natural language description of the scene (if existent), the locations and labels of the various objects within the scene, and a random noise vector. The global pathway uses this to create a scene layout encoding which describes high-level features and generates a global feature representation from this. The object pathway generates a feature representation of a given object at a location described by the respective bounding box and is applied iteratively over the scene at the locations specified by the individual bounding boxes. We then concatenate the feature representations of the global and the object pathway and use this to generate the final image. The discriminator, which also consists of a global and object pathway, gets as input the image, the bounding boxes and their respective object labels, and the textual description. The global pathway is then applied to the whole image and obtains a feature representation of the global image features. In parallel, the object pathway focuses only on the areas described by the bounding boxes and the respective object labels and obtains feature representations of these specific locations. Again, the outputs of both the global and the object pathway are merged and the discriminator is trained to distinguish between real and generated images. In contrast to previous work we do not generate a scene layout of the whole scene but only focus on relevant objects which are placed at the specified locations, while the global consistency of the image is the responsibility of the other part of our model. To summarize our model and contributions: 1) We propose a GAN model that enables us to control the layout of a scene without the use of a scene layout. 2) Through the use of an object pathway which is responsible for learning features of different object categories, we gain control over the identity and location of arbitrarily many objects within a scene.3) The discriminator judges not only if the image is realistic and aligned to the natural language description, but also whether the specified objects are at the given locations and of the correct object category. 4) We show that the object pathway does indeed learn relevant features for the different objects, while the global pathway focuses on general image features and the . Having more control over the general image layout can lead to a higher quality of images BID16 BID6 and is also an important requirement for semantic image manipulation BID5 ). Approaches that try to exert some control over the image layout utilize Generative Adversarial Nets BID3, Refinement Networks (e.g. BID1 ; Xu et al. (2018a) ), recurrent attention-based models (e.g. BID12), autoregressive models (e.g. Reed et al. (2016c) ), and even memory networks supplying the image generation process with previously extracted image features (b).One way to exert control over the image layout is by using natural language descriptions of the image, e.g. image captions, as shown by Reed et al. (2016b), Zhang et al. (2018a), , and Xu et al. (2018b). However, these approaches are trained only with images and their respective captions and it is not possible to specifically control the layout or placement of specific objects within the image. Several approaches suggested using a semantic layout of the image, generated from the image caption, to gain more fine-grained control over the final image. BID10, BID9 use a scene layout to generate images in which given objects are drawn within their specified segments based on the generated scene layout. BID6 use the image caption to generate bounding boxes of specific objects within the image and predict the object's shape within each bounding box. This is further extended by BID5 by making it possible to manipulate images on a semantic level. While these approaches offer a more detailed control over the image layout they heavily rely on a semantic scene layout for the image generating process, often implying complex preprocessing steps in which the scene layout is constructed. The two approaches most closely related to ours are by BID16 and BID14. BID14 introduce a model that consists of individual "blocks" which are responsible for different object characteristics (e.g. color, shape, etc.). However, their approach was only tested on the synthetic SHAPES data set BID0, which has only comparatively low variability and no image captions. Reed et al. (2016b) condition both the generator and the discriminator on either a bounding box containing the object or keypoints describing the object's shape. However, the used images are still of relatively low variability (e.g. birds ) and only contain one object, usually located in the center of the image. In contrast, we model images with several different objects at various locations and apply our object pathway multiple times at each image, both in the Figure 1: Both the generator and the discriminator of our model consist of a global and an object pathway. The global pathway focuses on global image characteristics, such as the , while the object pathway is responsible for modeling individual objects at their specified location.generator and in the discriminator. Additionally, we use the image caption and bounding box label to obtain individual labels for each bounding box, while Reed et al. (2016b) only use the image caption as conditional information. For our approach 1, the central goal is to generate objects at arbitrary locations within a scene while keeping the scene overall consistent. For this we make use of a generative adversarial network (GAN) BID3. A GAN consists of two networks, a generator and a discriminator, where the generator tries to reproduce the true data distribution and the discriminator tries to distinguish between generated data points and data points sampled from the true distribution. We use the conditional GAN framework, in which both the generator and the discriminator get additional information, such as labels, as input. The generator G (see Figure 1) gets as input a randomly sampled noise vector z, the location and size of the individual bounding boxes bbox i, a label for each of the bounding boxes encoded as a one-hot vector l onehoti, and, if existent, an image caption embedding ϕ obtained with a pretrained char-CNN-RNN network from Reed et al. (2016b). As a pre-processing step (A), the generator constructs labels label i for the individual bounding boxes from the image caption ϕ and the provided labels l onehoti of each bounding box. For this, we concatenate the image caption embedding ϕ and the one-hot vector of a given bounding box l onehoti and create a new label embedding label i by applying a matrix-multiplication followed by a non-linearity (i.e. a fully connected layer). The ing label label i contains the previous label as well as additional information from the image caption, such as color or shape, and is potentially more meaningful. In case of missing image captions, we use the one-hot embedding l onehoti only. The generator consists of two different streams which get combined later in the process. First, the global pathway (B) is responsible for creating a general layout of the global scene. It processes the previously generated local labels label i for each of the bounding boxes and replicates them spatially at the location of each bounding box. In areas where the bounding boxes overlap the label embeddings label i are summed up, while the areas with no bounding boxes remain filled with zeros. Convolutional layers are applied to this layout to obtain a high-level layout encoding which is concatenated with the noise vector z and the image caption embedding ϕ and the is used to generate a general image layout f global.Second, the object pathway (C) is responsible for generating features of the objects f locali within the given bounding boxes. This pathway creates a feature map of a predefined resolution using convolutional layers which receive the previously generated label label i as input. This feature map is further transformed with a Spatial Transformer Network (STN) BID7 to fit into the Published as a conference paper at ICLR 2019 bounding box at the given location on an empty canvas. The same convolutional layers are applied to each of the provided labels, i.e. we have one object pathway that is applied several times across different labels label i and whose output feeds onto the corresponding coordinates on the empty canvas. Again, features within overlapping bounding box areas are summed up, while areas outside of any bounding box remain zero. As a final step, the outputs of the global and object pathways f global and f locali are concatenated along the channel axis and are used to generate the image in the final resolution, using common GAN procedures. The specific changes of the generator compared to standard architectures are the object pathway that generates additional features at specific locations based on provided labels, as well as the layout encoding which is used as additional input to the global pathway. These two extensions can be added to the generator in any existing architecture with limited extra effort. The discriminator receives as input an image (either original or generated), the location and size of the bounding boxes bbox i, the labels for the bounding boxes as one-hot vectors l onehoti, and, if existent, the image caption embedding ϕ. Similarly to the generator, the discriminator also possesses both a global (D) and an object (E) pathway respectively. The global pathway takes the image and applies multiple convolutional layers to obtain a representation f global of the whole image. The object pathway first uses a STN to extract the objects from within the given bounding boxes and then concatenates these extracted features with the spatially replicated bounding box label l onehoti. Next, convolutional layers are applied and the ing features f locali are again added onto an empty canvas within the coordinates specified by the bounding box. Note, similarly to the generator we only use one object pathway that is applied to multiple image locations, where the outputs are then added onto the empty canvas, summing up overlapping parts and keeping areas outside of the bounding boxes set to zero. Finally, the outputs of both the object and global pathways f locali and f global are concatenated along the channel axis and we again apply convolutional layers to obtain a merged feature representation. At this point, the features are concatenated either with the spatially replicated image caption embedding ϕ (if existent) or the sum of all one-hot vectors l onehoti along the channel axis, one more convolutional layer is applied, and the output is classified as either generated or real. For the general training, we can utilize the same procedure that is used in the GAN architecture that is modified with our proposed approach. In our work we mostly use the StackGAN (a) and AttnGAN (b) frameworks which use a modified objective function taking into consideration the additional conditional information and provided image captions. As such, our discriminator D and our generator G optimize the following objective function: DISPLAYFORM0 where x is an image, c is the conditional information for this image (e.g. label i, bounding boxes bbox i, or an image caption ϕ), z is a randomly sampled noise vector used as input for G, and p data is the true data distribution. Zhang et al. (2018a) and others use an additional technique called conditioning augmentation for the image captions which helps improve the training process and the quality of the generated images. In the experiments in which we use image captions (MS-COCO) we also make use of this technique 2. For the evaluation, we aim to study the quality of the generated images with a particular focus on the generalization capabilities and the contribution of specific parts of our model, in both controllable and large-scale cases. Thus, in the following sections, we evaluate our approach on three different data sets: the Multi-MNIST data set, the CLEVR data set, and the MS-COCO data set. In our first experiment, we used the Multi-MNIST data set BID2 for testing the basic functionality of our proposed model. Using the implementation provided by BID2, we created 50,000 images of resolution 64 × 64 px that contain exactly three normal-sized MNIST digits in non-overlapping locations on a black .2 More detailed information about the implementation can be found in the Appendix. Published as a conference paper at ICLR 2019 1, 6, 7 2, 8, 4 6, 6, 9 9, 7, 9 6, 1, 0 9, 0, 0 8, 2, 2 4, 6, 2 7, 3, 9 0, 2, 9Figure 2: Multi-MNIST images generated by the model. Training included only images with three individual normal-sized digits. Highlighted bounding boxes and yellow ground truth for visualization. As a first step, we tested whether our model can learn to generate digits at the specified locations and whether we can control the digit identity, the generated digit's size, and the number of generated digits per image. According to the , we can control the location of individual digits, their identity, and their size, even though all training images contain exactly three digits in normal size. Figure 2 shows that we can control how many digits are generated within an image (rows A-B, for two to five digits) and various sizes of the bounding box (row C). As a second step, we created an additional Multi-MNIST data set in which all training images contain only digits 0-4 in the top half and only digits 5-9 in the bottom half of the image. For testing digits in the opposite half, we can see that the model is indeed capable of generalizing the position (row D, left), i.e. it can generate digits 0-4 in the bottom half of the image and digits 5-9 in the top half of the image. Nevertheless, we also observed that this does not always work perfectly, as the network sometimes alters digits towards the ones it has seen during training at the respective locations, e.g. producing a "4" more similar to a "9" if in bottom half of the image, or generating a "7" more similar to a "1" if in top half of the image. As a next step, we created a Multi-MNIST data set with images that only contain digits in the top half of the image, while the bottom half is always empty. We can see (Figure 2, row D, right) that the ing model is not able to generate digits in the bottom half of the image (see Figure 6 in the Appendix for more details on this). Controlling for the location still works, i.e. bounding boxes are filled with "something", but the digit identity is not clearly recognizable. Thus, the model is able to control both the object identity and the object location within an image and can generalize to novel object locations to some extent. To test the impact of our model extensions, i.e. the object pathway in both the generator and the discriminator as well as the layout encoding, we performed ablation studies on the previously created Multi-MNIST data set with three digits at random locations. We first disabled the use of the layout encoding in the generator and left the rest of the model unchanged. In the (Figure 2, row E, left), we can see that, overall, both the digit identity and the digit locations are still correct, but minor imperfections can be observed within various images. This is most likely due to the fact that the global pathway of the generator has no information about the digit identity and location until its features get merged with the object pathway. As a next test, we disabled the object pathway of the Published as a conference paper at ICLR 2019 discriminator and left the rest of the model unmodified. Again, we see (row E, right) that we can still control the digit location, although, again, minor imperfections are visible. More strikingly, we have a noticeably higher error rate in the digit identity, i.e. the wrong digit is generated at a given location, most likely due to the fact that there is not object pathway in the discriminator controlling the object identity at the various locations. In comparison, the imperfections are different when only the object pathway of the generator is disabled (row F, left). The layout encoding and the feedback of the discriminator seem to be enough to still produce the digits in the correct image location, but the digit identity is often incorrect or not recognizable at all. Finally, we tested disabling the object pathway in both the discriminator and the generator (see row F, right). This leads to a loss of control of both image location as well as identity and sometimes even in images with more or fewer than three digits per image. This shows that only the layout encoding, without any of the object pathways, is not enough to control the digit identity and location. Overall, these indicate that we do indeed need both the layout encoding, for a better integration of the global and object pathways, and the object pathways in both the discriminator and the generator, for optimal . In our second experiment we used more complex images containing multiple objects of different colors and shapes. The goal of this experiment was to evaluate the generalization ability of our object pathway across different object characteristics. For this, we performed tests similar to BID14, albeit on the more complex CLEVR data set BID8. In the CLEVR data set objects are characterized by multiple properties, in our case the shape, the color, and the size. Based on the implementation provided by BID8, we rendered 25,000 images with a resolution of 64 × 64 pixels containing 2 − 4 objects per image. The label for a given bounding box of an object is the object shape and color (both encoded as one-hot encoding and then concatenated), while the object size is specified through the height and width of the bounding box. Similar to the first experiment, we tested our model for controlling the object characteristics, size, and location. In the first row of FIG1 we present the of the trained model, where the left image of each pair shows the originally rendered one, while the right image was generated by our model. We can confirm that the model can control both the location and the objects' shape and color characteristics. The model can also generate images containing an arbitrary number of objects (forth and fifths pair), even though a maximum of four objects per image was seen during training. The CLEVR data set offers a split specifically intended to test the generalization capability of a model, in which cylinders can be either red, green, purple, or cyan and cubes can be either gray, blue, brown, or yellow during training, while spheres can have any of these colors. During testing, the colors between cylinders and cubes are reversed. Based on these restrictions, we created a second data set of 25,000 training images for testing our model. Results of the test are shown in the second row of FIG1 (again, left image of each pair shows the originally rendered one, while the right image was generated by our model). We can see that the color transfer to novel shape-color combinations takes place, but, similarly to the Multi-MNIST , we can see some artifacts, where e.g. some cubes look a bit more like cylinders and vice versa. Overall, the CLEVR experiment confirms the indication that our model can control object characteristics (provided through labels) and object locations (provided through bounding boxes) and can generalize to novel object locations, novel amounts of objects per image, and novel object characteristic combinations within reasonable boundaries. 2 When using the ground truth bounding boxes at test time (as we do) the IS increases to 11.94 ± 0.09. 3 FID score was calculated with samples generated with the pretrained model provided by the authors. 4 The authors report a "best" value of 25.89 ± 0.47, but when calculating the IS with the pretrained model provided by the authors we only obtain an IS of 23.61. Other researchers on the authors' Github website report a similar value for the pretrained model. 5 We use the updated source code (IS of 10.62) as our baseline model. Table 1: Comparison of the Inception Score (IS) and Fréchet Inception Distance (FID) on the MS-COCO data set for different models. Note: the IS and FID values of our models are not necessarily directly comparable to the other models, since our model gets at test time, in addition to the image caption, up to three bounding boxes and their respective object labels as input. For our final experiment, we used the MS-COCO data set BID11 ) to evaluate our model on natural images of complex scenes. In order to keep our evaluation comparable to previous work, we used the 2014 train/test split consisting of roughly 80,000 training and 40,000 test images and rescaled the images to a resolution of 256 × 256 px. At train-time, we used the bounding boxes and object labels of the three largest objects within an image, i.e. we used zero to three bounding boxes per image. Similarly to work by BID9 we only considered objects that cover at least 2% of the image for the bounding boxes. To evaluate our quantitatively, we computed both the Inception Score (IS, larger is better), which tries to evaluate how recognizable and diverse objects within images are , as well as the Fréchet Inception Distance (FID, smaller is better), which compares the statistics of generated images with real images BID4. As a qualitative evaluation, we generated images that contain more than one object, and checked, whether the bounding boxes can control the object placement. We tested our approach with two commonly used architectures for text-to-image synthesis, namely the StackGAN and the AttnGAN (b), and compared the images generated by these and our models. In the StackGAN, the training process is divided into two steps: first, it learns a generator for images with a resolution of 64 × 64 px based on the image captions, and second, it trains a second generator, which uses the smaller images (64 × 64 px) from the first generator and the image caption as input to generate images with a resolution of 256×256 px. Here, we added the object pathways and the layout encoding at the beginning of both the first generator and the second generator and used the object pathway in both discriminators. The other parts of StackGAN architecture and all hyperparameters remain the same as in the original training procedure for the MS-COCO data set. We trained the model three times from scratch and randomly sampled 3 times 30,000 image captions from the test set for each model. We then calculated the IS and FID values on each of the nine samples of 30,000 generated images and report the averaged values. As presented in Table 1, our StackGAN with added object pathways outperforms the original StackGAN both on the IS and the FID, increasing the IS from 10.62 to 12.12 and decreasing the FID from 74.05 to 55.30. Note, however, that this might also be due to the additional information our model is provided with as it receives up to three bounding boxes and respective bounding box labels per image in addition to the image caption. We also extended the AttnGAN by Xu et al. (2018b), the current state-of-the-art model on the MS-COCO data set (based on the Inception Score), with our object pathway to evaluate its impact on Published as a conference paper at ICLR 2019Figure 4: Examples of images generated from the given caption from the MS-COCO data set. A) shows the original images and the respective image captions, B) shows images generated by our StackGAN+OP (with the corresponding bounding boxes for visualization), and C) shows images generated by the original StackGAN 3 a different model. As opposed to the StackGAN, the AttnGAN consists of only one model which is trained end-to-end on the image captions by making use of multiple, intermediate, discriminators. Three discriminators judge the output of the generator at an image resolution of 64 × 64, 128 × 128, and 256 × 256 px. Through this, the image generation process is guided at multiple levels, which helps during the training process. Additionally, the AttnGAN implements an attention technique through which the networks focus on specific areas of the image for specific words in the image caption and adds an additional loss that checks if the image depicts the content as described by the image caption. There, in the same way as for the StackGAN, we added our object pathway at the beginning of the generator as well as to the discriminator that judges the generator outputs at a resolution of 64 × 64 px. All other discriminators, the higher layers of the generator, and all other hyperparameters and training details stay unchanged. Table 1 shows that adding the object pathway to the AttnGAN increases the IS of our baseline model (the pretrained model provided by the authors) from 23.61 to 24.76, while the FID is roughly the same as for the baseline model. To evaluate whether the StackGAN model equipped with an object pathway (StackGAN+OP) actually generates objects at the given positions we generated images that contain multiple objects and inspected them visually. Figure 4 shows some example images, more can be seen in the Appendix in FIG3. We can observe that the StackGAN+OP indeed generates images in which the objects are at appropriate locations. In order to more closely inspect our global and object pathways, we can also disable them during the image generation process. Figure 5 shows additional examples, in which we generate the same image with either the global or the object pathway disabled during the generation process. Row C of Figure 5 shows images in which the object pathway was disabled and, indeed, we observe that the images contain mostly information and objects at the location of the bounding boxes are either not present or of much less detail than when the object pathway is enabled. Conversely, row D of Figure 5 shows images which were generated when the global pathway was disabled. As expected, areas outside of the bounding boxes are empty, but we also observe that the bounding boxes indeed contain images that resemble the appropriate objects. These indicate, as in the previous experiments, that the global pathway does indeed model holistic image features, while the object pathway focuses on specific, individual objects. When we add the object pathway to the AttnGAN (AttnGAN + OP) we can observe similar 4. Again, we are able to control the location and identity of objects through the object pathway, however, we observe that the AttnGAN+OP, as well as the AttnGAN in general, tends to place objects corresponding to specific features at many locations throughout the image. For example, if the caption contains the word "traffic light" the AttnGAN tends to place objects similar to traffic lights throughout the whole image. Since our model only focuses on generating objects at given locations, while not enforcing that these objects only occur at these locations, this behavior leads to the that the AttnGAN+OP generates desired objects at the desired locations, but might also place the same object at other locations within the image. Note, however, that we only added the object pathway Published as a conference paper at ICLR 2019Figure 5: Examples of images generated from the given caption from the MS-COCO data set. A) shows the original images and the respective image captions, B) shows images generated by our StackGAN+OP (with the corresponding bounding boxes for visualization) with the object pathway enabled, C) shows images generated by the our StackGAN+OP when the object pathway is disabled, and D) shows images generated by the our StackGAN+OP when the global pathway is disabled.to the lowest generator and discriminator and that we might gain even more control over the object location by introducing object pathways to the higher generators and discriminators, too. In order to further evaluate the quality of the generations, we ran an object detection test on the generated images using a pretrained YOLOv3 network BID15. Here, the goal is to measure how often an object detection framework, which was trained on MS-COSO as well, can detect a specified object at a specified location 5. The confirm the previously made observations: For both the StackGAN and the AttnGAN the object pathway seems to improve the image quality, since YOLOv3 detects a given object more often correctly when the images are generated with an object pathway as opposed to images generated with the baseline models. The StackGAN generates objects at the given bounding box, ing in an Intersection over Union (IoU) of greater than 0.3 for all tested labels and greater than 0.5 for 86.7% of the tested labels. In contrast, the AttnGAN tends to place salient object features throughout the image, which leads to an even higher detection rate by the YOLOv3 network, but a smaller average IoU (only 53.3% of the labels achieve an IoU greater than 0.3). Overall, our experiments on the MS-COCO data set indicate that it is possible to add our object pathway to pre-existing GAN models without having to change the overall model architecture or training process. Adding the object pathway provides us with more control over the image generation process and can, in some cases, increase the quality of the generated images as measured via the IS or FID. Our experiments indicate that we do indeed get additional control over the image generation process through the introduction of object pathways in GANs. This enables us to control the identity and location of multiple objects within a given image based on bounding boxes and thereby facilitates the generation of more complex scenes. We further find that the division of work on a global and object pathway seems to improve the image quality both subjectively and based on quantitative metrics such as the Inception Score and the Fréchet Inception Distance. The further indicate that the focus on global image statistics by the global pathway and the more fine-grained attention to detail of specific objects by the object pathway works well. This is visualized for example in rows C and D of Figure 5. The global pathway (row C) generates features for the general image layout and but does not provide sufficient details for individual objects. The object pathway (row D), on the other hand, focuses entirely on the individual objects and generates features specifically for a given object at a given location. While this is the desired behavior Published as a conference paper at ICLR 2019 of our model it can also lead to sub-optimal images if there are not bounding boxes for objects that should be present within the image. This can often be the case if the foreground object is too small (in our case less than 2% of the total image) and is therefore not specifically labeled. In this case, the objects are sometimes not modeled in the image at all, despite being prominent in the respective image caption, since the object pathway does not generate any features. We can observe this, for example, in images described as "many sheep are standing on the grass", where the individual sheep are too small to warrant a bounding box. In this case, our model will often only generate an image depicting grass and other details, while not containing any sheep at all. Another weakness is that bounding boxes that overlap too much (empirically an overlap of more than roughly 30%) also often lead to sub-optimal objects at that location. Especially in the overlapping section of bounding boxes we often observe local inconsistencies or failures. This might be the of our merging of the different features within the object pathway since they are simply added to each other at overlapping areas. A more sophisticated merging procedure could potentially alleviate this problem. Another approach would be to additionally enhance the bounding box layout by predicting the specific object shape within each bounding box, as done for example by BID6.Finally, currently our model does not generate the bounding boxes and labels automatically. Instead, they have to be provided at test time which somewhat limits the usability for unsupervised image generation. However, even when using ground truth bounding boxes, our models still outperform other current approaches that are tested with ground truth bounding boxes (e.g. BID6) based on the IS and FID. This is even without the additional need of learning to specify the shape within each bounding box as done by BID6. In the future, this limitation can be avoided by extracting the relevant bounding boxes and labels directly from the image caption, as it is done for example by BID6 With the goal of understanding how to gain more control over the image generation process in GANs, we introduced the concept of an additional object pathway. Such a mechanism for differentiating between a scene representation and object representations allows us to control the identity, location, and size of arbitrarily many objects within an image, as long as the objects do not overlap too strongly. In parallel, a global pathway, similar to a standard GAN, focuses on the general scene layout and generates holistic image features. The object pathway, on the other hand, gets as input an object label and uses this to generate features specifically for this object which are then placed at the location given by a bounding box The object pathway is applied iteratively for each object at each given location and as such, we obtain a representation of individual objects at individual locations and of the general image layout (, etc.) as a whole. The features generated by the object and global pathway are then concatenated and are used to generate the final image output. Our tests on synthetic and real-world data sets suggest that the object pathway is an extension that can be added to common GAN architectures without much change to the original architecture and can, along with more fine-grained control over the image layout, also lead to better image quality. Published as a conference paper at ICLR 2019 Here we provide some more details about the exact implementation of our experiments. To train our GAN approach on the Multi-MNIST (CLEVR) data set we use the Stage-I Generator and Discriminator from the StackGAN MS-COCO architecture 6. In our following description an upsample block describes the following sequence: nearest neighbor upsampling with factor 2, a convolutional layer with X filters (filter size 3 × 3, stride 1, padding 1), batch normalization, and a ReLU activation. The bounding box labels are one-hot vectors of size encoding the digit identity (CLEVR: encoding object shape and color). Please refer to Table 2 for detailed information on the individual layers described in the following. For all leaky ReLU activations alpha was set to 0.2.In the object pathway of the generator we first create a zero tensor O G which will contain the feature representations of the individual objects. We then spatially replicate each bounding box label into a 4 × 4 layout of shape (CLEVR:) and apply two upsampling blocks. The ing tensor is then added to the tensor O G at the location of the bounding box using a spatial transformer network. In the global pathway of the generator we first obtain the layout encoding. For this we create a tensor of shape (CLEVR: FIG1) that contains the one-hot labels at the location of the bounding boxes and is zero everywhere else. We then apply three convolutional layers, each followed by batch normalization and a leaky ReLU activation. We reshape the output to shape and concatenate it with the noise tensor of shape (sampled from a random normal distribution) to form a tensor of shape. This tensor is then fed into a dense layer, followed by batch normalization and a ReLU activation and the output is reshaped to (−1, 4, 4). We then apply two upsampling blocks to obtain a tensor of shape (−1, 16, 16).At this point, the outputs of the object and the global pathway are concatenated along the channel axis to form a tensor of shape (−1, 16, 16). We then apply another two upsampling blocks ing in a tensor of shape (−1, 64, 64) followed by a convolutional layer and a TanH activation to obtain the final image of shape (−1, 64, 64).In the object pathway of the discriminator we first create a zero tensor O D which will contain the feature representations of the individual objects. We then use a spatial transfomer network to extract the image features at the locations of the bounding boxes and reshape them to a tensor of shape (CLEVR: FIG1). The one-hot label of each bounding box are spatially replicated to a shape of (CLEVR: ) and concatenated with the previously extracted features to form a tensor of shape (CLEVR: ). We then apply a convolutional layer, batch normalization and a leaky ReLU activation to the concatenation of features and label and, again, use a spatial transformer network to resize the output to the shape of the respective bounding box before adding it to the tensor O D.In the global pathway of the discriminator, we apply two convolutional layers, each followed by batch normalization and a leaky ReLU activation and concatenate the ing tensor with the output of the object pathway. After this, we again apply two convolutional layers, each followed by batch normalization and a leaky ReLU activation. We concatenate the ing tensor with the conditioning information about the image content, in this case, the sum of all one-hot vectors. To this tensor we apply another convolutional layer, batch normalization, a leaky ReLU activation, and another convolutional layer, to obtain the final output of the discriminator of shape.Similarly to the procedure of StackGAN and other conditional GANs we train the discriminator to classify real images with correct labels (the sum of one-hot vectors supplied in the last step of the process) as real, while generated images with correct labels and real images with (randomly sampled) incorrect labels should be classified as fake.6 https://github.com/hanzhanggit/StackGAN-Pytorch Published as a conference paper at ICLR 2019 StackGAN-Stage-I For training the Stage-I generator and discriminator (images of size 64 × 64 pixels) we follow the same procedure and architecture outlined in the previous section about the training on the Multi-MNIST and CLEVR data sets. The only difference is that we now have image captions as an additional description of the image. As such, to obtain the bounding box labels we concatenate the image caption embedding 7 and the one-hot encoded bounding box label and apply a dense layer with 128 units, batch normalization, and a ReLU activation to it, to obtain a label of shape for each bounding box. In the final step of the discriminator when we concatenate the feature representation with the conditioning vector, we use the image encoding as conditioning vector and do not use any bounding box labels at this step. The rest of the training proceeds as described in the previous section, except that the bounding box labels now have a shape of. All other details can be found in Table 2.StackGAN-Stage-II In the second part of the training, we train a second generator and discriminator to generate images with a resolution of 256 × 256 pixels. The generator gets as input images with a resolution of 64 × 64 pixels (generated by the trained Stage-I generator) and the image caption and uses them to generate images with a 256 × 256 pixels resolution. A new discriminator is trained to distinguish between real and generated images. On the Stage-II generator we perform the following modifications we use the same procedure as in the Stage-I generator to obtain the bounding box labels. To obtain an image encoding from the generated 64 × 64 image we use three convolutional layers, each followed by batch normalization and a ReLU activation to obtain a feature representation of shape [−1, 16, 16]. Additionally, we replicate each bounding box label (obtained with the dense layer) spatially at the locations of the bounding boxes on an empty canvas of shape and then concatenate it along the channel axis with the image encoding and the spatially replicated image caption embedding. As in the standard StackGAN we then apply more convolutional layers with residual connections to obtain the final image embedding of shape [−1, 16, 16], which provides the input for both the object and the global pathway. The generator's object pathway gets as input the image encoding described in the previous step. First, we create a zero tensor O G which will contain the feature representations of the individual objects. We then use a spatial transformer network to extract the features from within the bounding box and reshapes those features to [−1, 16, 16]. After this, we apply two upsample blocks and then use a spatial transformer network to add the features to O G within the bounding box region. This is done for each of the bounding boxes within the image. The generator's global pathway gets as input the image encoding and uses the same convolutional layers and upsampling procedures as the original StackGAN Stage-II generator. The outputs of the object and global pathway are merged at the resolution of [−1, 64, 64] by concatenating the two outputs along the channel axis. After this, we continue using the standard StackGAN architecture to generate images of shape.The Stage-II discriminator's object pathway first creates a zero tensor O D which will contain the feature representations of the individual objects. It gets as input the image (resolution of 256 × 256 pixels) and we use a spatial transformer network to extract the features from the bounding box and reshape those features to a shape of. We spatially replicate the bounding box label (one-hot encoding) to a shape of [−1, 32, 32] and concatenate it with the extracted features along the channel axis. This is then given to the object pathway which consists of two convolutional layers with batch normalization and a LeakyReLU activation. The output of the object pathway is again transformed to the width and height of the bounding box with a spatial transformer network and then added to O D. This procedure is performed with each of the bounding boxes within the image (maximum of three during training).The Stage-II discriminator's global pathway consists of the standard StackGAN layers, i.e. it gets as input the image (256 × 256 pixels) and applies convolutional layers with stride 2 to it. The outputs of the object and global pathways are merged at the resolution of [−1, 32, 32] by concatenating the 7 Downloaded from https://github.com/reedscot/icml2016 14 Published as a conference paper at ICLR 2019 two outputs along the channel axis We then apply more convolutional with stride 2 to decrease the resolution. After this, we continue in the same way as the original StackGAN.AttnGAN On the AttnGAN 8 we only modify the training at the lower layers of the generator and the first discriminator (working on images of 64 × 64 pixels resolution). For this, we perform the same modifications as described in the StackGAN-Stage-I generator and discriminator. In the generator we obtain the bounding box labels in the same way as in the StackGAN, by concatenating the image caption embedding with the respective one-hot vector and applying a dense layer with 100 units, batch normalization, and a ReLU activation to obtain a bounding box label. In contrast to the previous architectures, we follow the AttnGAN implementation in use the gated linear unit function (GLU) as standard activation for our convolutional layers in the generator. In the generator's object pathway we first create a zero tensor O G of shape which will contain the feature representations of the individual objects. We then spatially replicate each bounding box label into a 4 × 4 layout of shape and apply two upsampling blocks with 768 and 384 filters (filter size=3, stride=1, padding=1). The ing tensor is then added to the tensor O G at the location of the bounding box using a spatial transformer network. In the global pathway of the generator we first obtain the layout encoding in the same way as in the StackGAN-I generator, except that the three convolutional layers of the layout encoding now have 50, 25, and 12 filters respectively (filter size=3, stride=2, padding=1). We concatenate it with the noise tensor of shape (sampled from a random normal distribution) and the image caption embedding to form a tensor of shape. This tensor is then fed into a dense layer with 24,576 units, followed by batch normalization and a ReLU activation and the output is reshaped to. We then apply two upsampling blocks with 768 and 384 filters to obtain a tensor of shape.At this point the outputs of the object and the global pathways are concatenated along the channel axis to form a tensor of shape. We then apply another two upsampling blocks with 192 and 96 filters, ing in a tensor of shape. This feature representation is then used by the following layers of the AttnGAN generator in the same way as detailed in the original paper and implementation. In the object pathway of the discriminator we first create a zero tensor O D which will contain the feature representations of the individual objects. We then use a spatial transfomer network to extract the image features at the locations of the bounding boxes and reshape them to a tensor of shape. The one-hot label of each bounding box is spatially replicated to a shape of (−1, 16, 16) and concatenated with the previously extracted features. We then apply a convolutional layer with 192 filters (filter size=4, stride=1, padding=1), batch normalization and a leaky ReLU activation to the concatenation of features and label and, again, use a spatial transformer network to resize the output to the shape of the respective bounding box before adding it to the tensor O D.In the global pathway of the discriminator we apply two convolutional layers with 96 and 192 filters (filter size=4, stride=2, padding=1), each followed by batch normalization and a leaky ReLU activation and concatenate the ing tensor with the output of the object pathway. After this, we again apply two convolutional layers with 384 and 768 filters (filter size=4, stride=2, padding=1), each followed by batch normalization and a leaky ReLU activation. We concatenate the ing tensor with the spatially replicated image caption embedding. To this tensor we apply another convolutional layer with 768 filters (filter size=3, stride=1, padding=1), batch normalization, a leaky ReLU activation, and another convolutional layer with one filter (filter size=4, stride=4, padding=0), to obtain the final output of the discriminator of shape. The rest of the training and all other hyperparameters and architectural values are left the same as in the original implementation. Published as a conference paper at ICLR 2019Multi-MNIST CLEVR MS-COCO-I MS-COCO-II Conv (fs=4, s=1, p=1) 128 96 192 192 Conv (fs=4, s=1, p=1) 192 Output Shape (192, 32, 32 1, 3, 7 6, 4, 9 9, 3, 5 5, 9, 5 3, 4, 6 6, 3, 5 6, 9, 9 1, 2, 1 8, 4, 6 9, 4, 1 5, 9, 1 8, 0, 9 4, 5, 5 4, 4, 6 7, 2, 6 9, 0, 0 6, 7, 1 3, 4, 2 0, 3, 7 2, 7, 8 9, 7, 2 1, 6, 1 3, 7, 9 4, 1, 5 5, 5, 3 9, 2, 1 7, 6, 4 4, 2, 1 4, 8, 3 2, 2, 4 6, 5, 4 5, 9, 6 9,4, 5 3, 0, 1 1, 0, 7 3, 8, 2 7, 3, 0 4, 3, 9 4, 8, 1 9, 6, 6 Figure 6: Systematic test of digits over vertically different regions. Training set included three normal-sized digits only in the top half of the image. Highlighted bounding boxes and yellow ground truth for visualization. We can see that the model fails to generate recognizable digits once their location is too far in the bottom half of the image, as this location was never observed during training. show typical failure cases of our model, where there is no bounding box for the foreground object present. As a our model only generates the , without the appropriate foreground object, even though the foreground object is very clearly described in the image caption. Figure 9 provides similar but for random bounding box positions. The first six examples show images generated by our StackGAN where we changed the location and size of the respective bounding boxes. The last three examples show failure cases in which we changed the location of the bounding boxes to "unusual" locations. For the image with the child on the bike, we put the bounding box of the bike somewhere in the top half of the image and the bounding box for the child somewhere in the bottom part. Similarly, for the man sitting on a bench, we put the bench in the top and the man in the bottom half of the image. Finally, for the image depicting a pizza on a plate, we put the plate location in the top half of the image and the pizza in the bottom half. Figure 8 shows of text-to-image synthesis on the MS-COCO data set with the AttnGAN architecture. Rows A show the original image and image caption, rows B show the images generated by our AttnGAN + Object Pathway and the given bounding boxes for visualization, and rows C show images generated by the original AttnGAN (pretrained model obtained from https: //github.com/taoxugit/AttnGAN). The last block of examples (last row) show typical failure cases, in which the model does generate the appropriate object within the bounding box, but also places the same object at multiple other locations within the image. Similarly as for StackGAN, FIG4 shows images generated by our AttnGAN where we randomly change the location of the various bounding boxes. Again, the last three examples show failure cases where we put the locations of the bounding boxes at "uncommon" positions. In the image depicting the sandwiches we put the location of the plate in the top half of the image, in the image with the dogs we put the dogs' location in the top half, and in the image with the motorbike we put the human in the left half and the motorbike in the right half of the image. To further inspect the quality of the location and recognizability of the generated objects within an image, we ran a test on object detection using a YOLOv3 network BID15 that was also pretrained on the MS-COCO data set 9. We use the Pytorch implementation from https://github.com/ayooshkathuria/pytorch-yolo-v3 to get the bounding box and label predictions for our images. We follow the standard guidelines and keep all hyperparameters for the YOLOv3 network as in the implementation. We picked the 30 most common training labels (based on how many captions contain these labels) and evaluate the models on these labels, see TAB7.In the following, we evaluate how often the pretrained YOLOv3 network recognizes a specific object within a generated image that should contain this object based on the image caption. For example, we expect an image generated from the caption "a young woman taking a picture with her phone" to contain a person somewhere in the image and we check whether the YOLOv3 network actually recognizes a person in the generated image. Since the baseline StackGAN and AttnGAN only receive the image caption as input (no bounding boxes and no bounding box labels) we decided to only use captions that clearly imply the presence of the given label (see TAB7). We chose this strategy in order to allow for a fair comparison of the ing presence or absence of a given object. Specifically, for a given label we choose all image captions from the test set that contain one of the associated words for this label (associated words were chosen manually, see TAB7) and then generated three images for each caption with each model. Finally, we counted the number of images in which the given object was detected by the YOLOv3 network. Table 4 shows the ratio of images for each label and each model in which the given object was detected at any location within the image. Additionally, for our models that also receive the bounding boxes as input, we calculated the Intersection over Union (IoU) between the ground truth bounding box (the bounding box supplied to the model) and the bounding box predicted by the YOLOv3 network for the recognized object. Table 4 presents the average IoU (for the models that have an object pathway) for each object in the images in which YOLOv3 detected the given object. For each image in which YOLOv3 detected the given object, we calculated the IoU between the predicted bounding box and the ground truth bounding box for the given object. In the cases in which either an image contains multiple instances of the given object (i.e. multiple different bounding boxes for this object were given to the generator) or YOLOv3 detects the given object multiple times we used the maximum IoU between all predicted and ground truth bounding boxes for our statistics. Table 4 summarizes the with the 30 tested labels. We can observe that the StackGAN with object pathway outperforms the original StackGAN when comparing the recall of the YOLOv3 network, i.e. in how many images with a given label the YOLOv3 network actually detected the given object. The recall of the original StackGAN is higher than 10% for 26.7% of the labels, while our StackGAN with object pathway in a recall greater than 10% for 60% of the labels. The IoU is greater than 0.3 for every label, while 86.7% of the labels an IoU of greater than 0.5 (original images: 100%) and 30% have an IoU of greater than 0.7 (original images: 96.7%). This indicates that we can indeed control the location and identity of various objects within the generated images. Compared to the StackGAN, the AttnGAN achieves a much greater recall, with 80% and 83.3% of the labels having a recall of greater than 10% for the original AttnGAN and the AttnGAN with object pathway respectively. The difference in recall values between the original AttnGAN and the AttnGAN with object pathway is also smaller, with our AttnGAN having a higher (lower) recall than the original AttnGAN (we only count cases where the difference is at least 5%) in 26.7% (13.3%) of the labels. The average IoU, on the other hand, is a lot smaller for the AttnGAN than for the StackGAN. We only achieve an IoU greater than 0.3 (0.5, 0.7) for 53.3% (3.3%, 0%) of the labels. As mentioned in the discussion (subsection 4.4), we attribute this to the observation that the AttnGAN tends to place seemingly recognizable features of salient objects at arbitrary locations throughout the image. This might attribute to the overall higher recall but may negatively affect the IoU.Overall, these further confirm our previous experiments and highlight that the addition of the object pathway to the different models does not only enable the direct control of object location and identity but can also help to increase the image quality. The increase in image quality is supported by a higher Inception Score, lower Fréchet Inception Distance (for StackGAN) and a higher performance of the YOLOv3 network in detecting objects within generated images. Table 4: Results of YOLOv3 detections on generated and original images. Recall provides the fraction of images in which YOLOv3 detected the given object. IoU (Intersection over Union) measures the maximum IoU per image in which the given object was detected.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Extend GAN architecture to obtain control over locations and identities of multiple objects within generated images.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.