Question: Represent the scientific article to find a short TLDR

Question: Brushing techniques have a long history with the first interactive selection tools appearing in the 1990's. Since then, many additional techniques have been developed to address selection accuracy, scalability and flexibility issues. Selection is especially difficult in large datasets where many visual items tangle and create overlapping. This paper investigates a novel brushing technique which not only relies on the actual brushing location but also on the shape of the brushed area. Firstly, the user brushes the region where trajectories of interest are visible. Secondly, the shape of the brushed area is used to select similar items. Thirdly, the user can adjust the degree of similarity to filter out the requested trajectories. This technique encompasses two types of comparison metrics, the piece-wise Pearson correlation and the similarity measurement based on information geometry. We apply it to concrete scenarios with datasets from air traffic control, eye-tracking data and GPS trajectories.. This figure shows our main rationale to define our shaped based brushing technique. A) Unselected trail set where the user wishes to select the only curved one. B) Standard brushing technique, where the brushing of the curved trail also selects every other trail which touches the brushing area. C) Our method uses the brushing input to compare with the brushed trajectory and only the trajectories similar in shape are selected. Such techniques help to visually select items of interest with interactive paradigms (i.e. lasso, boxes, brush) in a view. When the user visually detects a relevant pattern (i.e. a specific curve or a trend), the brushing technique can then be applied to select it. While this selection can be seamlessly performed, the user may still face issues when the view becomes cluttered with many tangled items. In such dense visualization, existing brushing techniques also select items in the vicinity of the target and thus capture part of the clutter (see Figure 1). To address such an issue, the user can adjust the brushing parameters by changing the brush size or the selection box locations. However, this may take time and requires many iterations or trials. This paper proposes a novel brushing technique that filters trajectories taking into account the shape of the brush in addition to the brush area. This dual input is provided at the same time and opens novel opportunities for brushing techniques. The cornerstone of such a technique relies on a shape comparison algorithm. This algorithm must provide a numerical similarity measurement which is ordered (low value for unrelated shapes, and high value for correlated shapes), continuous (no steps in the computed metric) and with a semantic so that the user can partially understand the logic behind this similarity measurement. Thus, to build such a dual filtering technique, the following design requirements (DR) must be fulfilled: • DR3: The technique enables incremental selection refinement. • DR4: The technique is interactive. Taking into account the identified requirements (DR1-DR4), this paper presents a novel shape-based brushing tool. To the best of our knowledge, such a combination of brushing and shape comparison technique has not yet been explored in trajectory analysis and this paper fills this gap. In the remainder of the paper the following subjects are presented. First, previous works in the domain of brushing and shape comparison are provided. Second, the brushing pipeline is detailed with explanations of the comparison metric data processing. Next, the use of such a technique through different use-cases is demonstrated. The brushing technique is discussed in terms of usefulness, specific application and possible limitations. Finally, the paper concludes with a summary of our contribution and provides future research directions. There are various domain-specific techniques targeting trail exploration and analysis. In this section, we explore three major components of selection techniques for trail-set exploration and analysis relevant to our work: brushing, query-by-content, and similarity measurement. Trail-set exploration relies on pattern discovery where relevant trails need to be selected for further analysis. Brushing is a selection technique for information visualization, where the user interactively highlights a subset of the data by defining an area of interest. This technique has been shown to be a powerful and generic interaction technique for information retrieval. The selection can be further refined using interactive filtering techniques. The approach presented in this paper is based on dynamic queries and direct manipulation. Systems designed specifically for spatio-temporal visualization and in particular in trajectory visualizations are very complex because of their 3D and time varying nature. Due to this, several systems and frameworks have been especially designed to visualize them. Most of these systems include selection techniques based on brushing, and some of them enable further query refinement through boolean operations. These techniques do not take into account the shape of the trails, so selecting a specific one with a particular shape requires many manipulations and iterations to fine-tune the selection. While this paper attempts to suggest a shape-based brushing technique for trail sets, researchers have explored shape-based selection techniques in different contexts, both using arbitrary shapes and sketch-based queries. Sketch-based querying presents several advantages over traditional selection. It has been used for volumetric data sets, and neural pathway selection. This last work is the closest to the current study. However, the authors presented a domain-specific application and they based their algorithm on the Euclidean distance. This is not a robust metric for similarity detection since it is hard to provide a value indicating a high similarity and varies greatly according to the domain and the data considered. In addition, this metric does not support direction and orientation matching nor the combination of brushing with filtering. In addition, user-sketched pattern matching plays an important role in searching and localizing time series patterns of interest. For example, Holz and Feiner defined a relaxed selection technique in which the users draw a query to select the relevant part of a displayed time-series. Correl et al. propose a sketch-based query system to retrieve time series using dynamic time wrapping, mean square error or the Hough transform. They present all matches individually in a small multiple, arranged according to the similarity measurement. These techniques, as the one proposed here, also take advantage of sketches to manipulate data. However, they are designed for querying rather than selecting, and specifically for 2D data. Other approaches use boxes and spheres to specify the regions of interest, and the desired trails are obtained if they intersect these regions of interest. However, many parameters must be changed by the analysts in order to achieve a simple single selection. The regions of interest must be re-scaled appropriately, and then re-positioned back and forth multiple times for each operation. Additionally, many selection box modifications are required to refine the selection and thus hinder and alter the selection efficiency. Given a set of trajectories, we are interested in retrieving the most similar subset of trajectories with a user-sketched query. Common approaches include selecting the K-nearestneighbors (KNN) based on the Euclidean distance (ED) or elastic matching metrics (e.g, Dynamic Time Warping -DTWs). Affinity cues have also been used to group objects. For example, objects of identical color are given a high similarity coefficient for the color affinity. The Euclidean distance is the most simple to calculate, but, unlike mathematical similarity measurements which are usually bound between 0 and 1 or -1 and 1, ED is unbounded and task-specific. A number of works have suggested transforming the raw data into a lower-dimensional representation (e.g., SAX, PAA ). However, they require adjusting many abstract parameters which are dataset-dependant and thus reduce their flexibility. In, global and local proximity of 2D sketches are presented. The second measure is used for similarity detection where an object is contained within another one, and is not relevant to this work. While the first measure refers to the distance between two objects (mostly circles and lines), there is no guarantee that the approach could be generalized to large datasets such as eye tracking, GPS or aircraft trajectories. In contrast, Dynamic Time Warping has been considered as the best measurement in various applications to select shapes by matching their representation. It has been used in gestures recognition, eye movements and shapes. An overview of existing metrics is available. The k-Nearest Neighbor (KNN) approach has also long been studied for trail similarity detection. However, using this metric, two trails may provide a good accurate connection (i.e, a small difference measure as above) even if they have very different shapes. Other measurements to calculate trajectory segment similarity are the Minimum Bounding Rectangles (MBR) or Fréchet Distance which leverage the perpendicular distance, the parallel distance and the angular distance in order to compute the distance between two trajectories. In order to address the aforementioned issues, we propose and investigate two different approaches. The first approach is based on directly calculating the correlations on x-axis and y-axis independently between the shape of the brush and the trails (section 3.1.1). The second approach (section 3.1.2) is based on the geometrical information of the trails, i.e, the trails are transformed into a new space (using eigenvectors of the co-variance matrix) which is more suitable for similarity detection. This paper's approach leverages the potential of these two metrics to foster efficient shape-based brushing for large cluttered datasets. As such, it allows targeting detailed motif discovery performed interactively. This section presents the interactive pipeline (Figure 2) which fulfills the identified design requirements (DR1-DR4). As for any interactive system, user input plays the main role and will operate at every stage of the data processing. First, the input data (i.e. trail set) is given. Next, the user inputs a brush where the pipeline extracts the brushed items, the brush area and its shape. Then, two comparison metrics are computed between every brushed item and the shape of the brush (similarity measurement). Followed by the binning process and its filtering, the ing data is presented to the user. The user can then refine the brush area and choose another comparison metric until the desired items are selected. As previously detailed in the related word section, many comparison metrics exist. While our pipeline can use any metric that fulfill the design requirement DR2 (continuous, ordered and meaningful comparison), the presented pipeline contains only two complementary algorithms: Pearson and FPCA. The first one focuses on a shape comparison basis with correlation between their representative vertices, while the latter focuses on curvature comparison. As shown in Figure 3, each metric produces different . The user can use either of them depending on the type of filtering to be performed. During the initial development of this technique, we first considered using the Euclidean distance (ED) and DTW, but we have rapidly observed their limitations and we argue that PC and FPCA are more suitable to trajectory datasets. First, PC values are easier to threshold. A PC value > 0.8 provides a clear indication of the similarity of 2 shapes. Moreover, to accurately discriminate between complex trajectories, we need to go beyond the performance of ED. Furthermore, the direction of the trajectories, while being essential for our brushing technique, is not supported by ED and DTW similarity measures. Another disadvantage of using ED is domain-and task-specific threshold that can drastically vary depending on the context. PC, that we used in our approach, on the other hand, uses the same threshold independently of the type of datasets. The two following sections detail the two proposed algorithms. Pearson's Correlation (PC) is a statistical tool that measures the correlation between two datasets and produces a continuous measurement between ∈ [−1, 1] with 1 indicating a high degree of similarity, and −1 an anti-correlation indicating an opposite trend. This metric is well suited (DR2) for measuring dataset similarity (i.e. in trajectory points). Pearson's Correlation PC between two trails T i and T j on the x − axis can be defined as follows: Where T i x and T j x are the means, E the expectation and σ T ix, σ T jx the standard deviations. The correlation is computed on the y − axis and the x − axis for two dimensional points. This metric is invariant in point translation and trajectory scale but it does not take into account the order of points along a trajectory. Therefore, the pipeline also considers the FPCA metric that is more appropriate to trajectory shape but that does not take into account negative correlation. Functional Data Analysis is a well-known information geometry approach that captures the statistical properties of multivariate data functions, such as curves modeled as a point in an infinite-dimensional space (usually the L 2 space of square integrable functions ). The Functional Principal Component Analysis (FPCA) computes the data variability around the mean curve of a cluster while estimating the Karhunen-Loeve expansion scores. A simple analogy can be drawn with the Principal Component Analysis (PCA) algorithm where eigen vectors and their eigen values are computed, the FCPA performs the same operations with eigen functions (piece-wise splices) and their principal component scores to model the statistical properties of a considered cluster: where b j are real-valued random variables called principal component scores. φ j are the principal component functions, which obey to: φ j are the (vector-valued) eigenfunctions of the covariance operator with eigenvalues λ j. We refer the prospective reader to the work of Hurter et al. for a Discrete implementation. With this model, knowing the mean curveγ and the principal component functions φ j, a group of curves can be described and reconstructed (Inverse FPCA) with the matrix of the principal component score b j of each curve. Usually, a finite vector Figure 2. This figure shows the interaction pipeline to filtered items thanks to the user brush input. The pipeline extracts the brushed items but also the shape of the brush which will be then used as a comparison with the brushed items (metrics stage). Then, brushed items are stored in bins displayed with small multiples where the user can interactivity refine the selection (i.e. binning and filtering stage). Finally, the user can adjust the selection with additional brushing interactions. Note that both the PC and FPCA can be used for the binning process, but separately. Each small multiple comes from one metric exclusively. Figure 3. This figure shows an example of the two metrics for shape comparison usage. The user brushed around curve 3 and thus also selected curves 1 and 2. Thanks to the Pearson computation, the associated small multiples show that only curve 2 is correlated to the shape of the brush. Curve 3 is anti correlated since it in an opposite direction to the shape of the brush. The FPCA computation does not take into account the direction but rather the curvature similarity. As such, only shape 3 is considered as highly similar to the brush shape input. (with fixed dimension d) of b j scores is selected such that the explained variance is more than a defined percentile. To compute a continuous and meaningful metric (DR2), the metric computation uses the two first Principal Components (PC) to define the representative point of a considered trajectory. Then, the metric is computed by the euclidean distance between the shape of the brush and each brushed trajectory in the Cartesian scatterplot PC1/PC2 (Figure 4). Each distance is then normalized between with 1 corresponding to the largest difference in shape between the considered shape of the brush of the corresponding trajectory. Taking into account the computed comparison metrics, the pipeline stores the ing values into bins. Items can then be sorted in continuous ways from the less similar to the most similar ones. While the Pearson measurements ∈ [−1, 1] and the FPCA ∈, this binning process operates in the same way. Each bin is then used to visually show the trajectories it contains through small multiples (we use 5 small multiples which gives a good compromise between visualization compactness and trajectory visibility). The user can then interactively filter the selected items (DR4) with a range slider on top of the small multiple visualization. The user is thus able to decide whether to remove uncorrelated items or refine the correlated one with a more restricted criteria (DR3). This technique is designed to enable flexible and rapid brushing of trajectories, by both the location and the shape of the brush. The technique's interaction paradigm is now described and illustrated in a scenario where an air traffic management expert studies the flight data depicted in Figure 6. Aircraft trajectories can be visually represented as connected line segments that form a path on a map. Given the flight level (altitude) of the aircraft, the trajectories can be presented in 3D and visualized by varying their appearances or changing their representation to basic geometry types. Since the visualization considers a large number of trajectories that compete for the visual space, these visualizations often present occlusion and visual clutter issues, rendering exploration difficult. Edge bundling techniques have been used to reduce clutter and occlusion but they come at the cost of distorting the trajectory shapes which might not always be desirable. Analysts need to explore this kind of datasets in order to perform diverse tasks. Some of these tasks compare expected aircraft trajectories with the actual trajectories. Other tasks detect unexpected patterns and perform out traffic analysis in complex areas with dense traffic. To this end, various trajectory properties such as aircraft direction, flight level and shape are examined. However, most systems only support selection techniques that rely on starting and end points, or predefined regions. We argue that the interactive shape brush technique would be helpful for these kinds of tasks, as they require the visual inspection of the data, the detection of the specific patterns and then their selection for further examination. As these specific patterns might differ from the rest of the data precisely because of their shape, a technique that enables their selection through this characteristic will make their manipulation easier, as detailed in the example scenario. We consider a dataset that includes 4320 aircraft trajectories of variable lengths from one day of flight traffic over the French airspace. We define the trail T T T as a set of real-valued consecutive points T T T = [(T T T x x x 1, T T T y y y 1)), (T T T x x x 2, T T T y y y 2),..., (T T T x x x n, T T T y y y n) ] where n is the number of points and (T T T x x x i, T T T y y y i) corresponds to the i − th coordinate of the trail. The Figure 6 depicts an example of 4133 trails (aircraft in French airspace). The brush Shape consists of a set of real-valued consecutive points S S S = [(S S Sx x x 1, S S Sy y y 1)), (S S Sx x x 2, S S Sy y y 2),..., (S S Sx x x m, S S Sy y y m) ] where m is the number of points. Note that while the length n of each trail is fixed, the length m of the S S Shape depends on the length of the user brush. The S S Shape is smoothed using a 1Cfilter and then resampled to facilitate the trail comparison. The similarity metrics are then used in subsequences of the s s shape of approximately the same length as the brush S S Shape. In order to do this, each trial is first resampled so that each pair of consecutive vertices on the trail has the same distance l vertices. The user starts by exploring the data using pan and zoom operations. They are interested in the trajectories from the south-east of France to Paris. The user can choose if they are looking for a subsequence match or an exact match. A subsequence match involves the detection of trajectories having a subsequence similar to the S S Shape locally. Exact match comparison also takes into account the length of the trajectory and the S S Shape in order to select a trajectory, i.e, its length must be approximately similar to the length of the Shape (general measurements). This option is especially useful to select a trajectory by its start and end points (e.g, finding trajectories taking off from an airport A and landing at an airport B). The exact matching is supported by analyzing the length of the trail and the Shape before applying the similarity metric algorithm. The analyst in the scenario activates the subsequence match and starts brushing in the vicinity of the target trajectories following the trajectory shape with the mouse. This will define both the brush region and the brush shape. Once the brushing has been completed, the selected trajectories are highlighted in green, as depicted in Figure 5 -(b). a b Figure 5. (a) The user brushes the trajectories in order to select those from the south-east of France to Paris. (b) They select the most correlated value to take into account the direction of the trails The similarity calculation between the Shape and the brushed region will produce a similarity value for each trail contained in the region and the trails are distributed in small multiples as detailed in Section 3. Once the user has brushed, she can adjust the selection by selecting one of the bins displayed in the small multiples and using its range slider. This enables a fine control over the final selection and makes the algorithm thresholding easier to understand as the user has control over both the granularity of the exploration and the chosen similarity level. The range slider position controls the similarity level, and its size determines the number of trajectories selected at each slider position: the smallest size selects one trajectory at a time. The range slider size and position are adjusted by direct manipulation. As the bins are equally sized, the distribution of the similarity might not be linear across the small multiples. This makes navigation easier since the trajectories distribution in the small multiples is continuous. However, this also entails that not every bin will correspond to the same similarity value interval. To keep this information available to the user, a colored heatmap (from red to green) displays the actual distribution. In the current scenario, the expert, as they wish to select only the flights to Paris and not from Paris, selects the trajectories that are correlated with the original brush. These trajectories are on the right side of the small multiple, highlighted in green as depicted in Figure 5 -(b). The expert is then interested in exploring the flights that land on the north landing strip but that are not coming from the east. For this, they perform a new shape brush to identify the planes that do come from the east, and distinguishable by the "C" shape in the trajectories, as depicted in Figure 7. To be able to select the geometry precisely, the expert changes to the FPCA metric, using the keyboard shortcut. In this case the small multiple arranges the trajectories from less similar to more similar. This entails that the small multiple based on FPCA also enables the selection of all the trajectories that do not match the specified Shape but which are contained in the brushing region. As all trajectories passing through the north landing strip are contained in the brushing region, the most similar trajectories will correspond to the ones that have a "C" shape, in the same orientation as the Shape, and thus come from the east. The less similar will be the ones that interest the analyst, so they can select them by choosing the most dissimilar small multiple as depicted in Figure 7 -(b). Figure 6. One day's aircraft trajectories in French airspace, including taxiing, taking-off, cruise, final approach and landing phases. Selecting specific trajectories using the standard brushing technique will yield inaccurate due to the large number of trajectories, occlusions, and closeness in the spatial representation. We argue that there is a strong demand for targeted brushing to select motifs in datasets. In various domains, including aircraft trajectories, eye tracking, GPS trajectories or brain fiber analysis, there is a substantial need to be able to discover hidden motifs in large datasets. Undoubtedly, retrieving desired trails in such datasets would help analysts to focus on the most interesting parts of the data. The system was built using C# and OpenTK on a 64bit 1 XPS 15 Dell Laptop. Although, both PC and FPCA provide different but valuable , the running performance was 10 times faster with PC compared to FPCA. The technique was first tested informally with ML: 3? experts from aerospace domain with more than 10 years of experience in trajectories analysis. While the collected feedback was largely positive, we observed some limitations regarding the misunderstanding of our filtering parameters. Given the novelty of our interaction technique, users needed a small training period to better understand the semantic of our small multiples. Nevertheless, our technique provides initial good selection without any parameter adjustment. Because the presented technique is not designed to replace standard brushing but rather to complement it, we argue that an evaluation comparing techniques that are substantially different will not capture the actual usefulness of our technique. The goal of this technique is to facilitate trajectories selection in dense areas, where standard brushing would require multiple user actions (panning, zooming, brushing). Therefore, a use-cases approach has been chosen to showcase relevant scenarios of usage. Eye-tracking technologies are gaining popularity for analyzing human behaviour, in visualization analysis, human factors, human-computer interaction, neuroscience, psychology and training. The principle consists in finding the likely objects of interest by tracking the movements of the user's eyes. Using a camera, the pupil center position is detected and the gaze, i.e, the point in the scene the user is fixating on, is computed using a prior calibration procedure. Therefore, the gaze data consist of sampled trails representing the movements of the user's eye gaze while completing a given task. Two important types of recorded movements characterize eye behaviour: the fixations and saccades. Fixations are the eye positions the user fixates for a certain amount of time, in other words, they describe the locations that captured the attention of the user. The saccades connect the different fixations, i.e, they represent the rapid movements of the eye from one location to another. The combination of these eye movements are called the scanpath (Figure 8A). The scanpath is subject to overplotting. This challenge may be addressed through precise brushing techniques to select specific trails. Usually, fixation events are studied to create an attention map which shows the salient elements in the scene. The salient elements are located at high density fixation areas. However, the temporal connections of the different fixations provide additional information. The saccades enable the links between the fixations to be maintained and the temporal meaning of the eye movement to be held. Discovering patterns in the raw scanpath data is difficult since, in contrast to aircraft trajectories, eye movements are sparser and less regular (Figure 8). To address this, different kinds of visualizations for scanpaths have been proposed in the literature. For example, edge bundling techniques minimize visual clutter of large and occluded graphs. However, these techniques either alter trail properties such as shape and geometric information, or are otherwise computationally expensive, which makes them unsuitable for precise exploration and mining of large trail datasets. Moreover, it is possible to animate eye movements in order to have an insight of the different fixations and saccades. However, given the large datasets of eye movements retrieved from lengthy experiments containing thousands of saccades, this approach is unnecessarily time-consuming and expensive. Therefore, we next describe how this study's approach supports proper and more efficient motif discovery on such eye tracking datasets. The tested dataset is adapted from Peysakhovich et al., where a continuous recording of eye movement in a cockpit was performed. The gaze data was recorded at 50 Hz. Sequential points located in a square of 20 × 20 pixels and separated by at least 200 ms were stored as a fixation event and replaced by their average in order to reduce noise coming from the microsaccades and the tracking device. In order to illustrate some examples, we could consider a domain expert who wishes to explore the movements of the pilot's eyes in a cockpit. When performing a task, the pilot scans the different instruments in the cockpit, focuses more on certain instruments or interacts with them. Especially in this context, the order of pilot attention is important since checking a parameter in one instrument may give an indication of the information displayed in another instrument. For example, the priority of the Primary Flight Display (PFD) instrument compared to Flight Control Unit (FCU) will differ for the cruise phase as compared to the final landing approach. As an example of analysis, the user wishes to explore the movement of the eye from the Primary Flight Display (PFD) to the Navigation Display (ND). Selecting these scanpaths using traditional brushing techniques would be challenging because of the clutter, selecting those scanpaths would introduce additional accidental selections. Therefore, he brushes these scanpaths using a shape that extends from the PFD to the ND, applying the Pearson metric to consider the direction. Figure 8 (a) depicts the brushed eye movements that correspond to the most correlated trails in the small multiple. There are several saccades between those two devices, and this is inline with the fact that saccadic movements between the PFD and the ND are typically caused by parameter checking routines. However, when the user changes the selection and brushes the scanpath between the ND and the FCU, it is surprising to see that there is only one saccade between them. Brushing now with a shape that goes between the PFD and the FCU (Figure 8 -(c)) reveals only one scanpath. This is difficult to visualize in the raw data or using the standard brushing technique. A final Shape searching for an eye movement from the PFD to the LAI and passing by the FCU, in only one saccade (Figure 8-(d) ). To determine the meaning of this behavior, the tool also enables the expert to exploit a continuous transition to increase the visibility and gain insight on when these saccadic movements occurred (temporal view). The user can change the visual mapping from the (x,y) gaze location to the (time,y) temporal view. This smooth transition avoids abrupt change to the visualization (figure 9). GPS trajectories consist of sequential spatial locations recorded by a measurement instrument. Subjects such as people, wheeled vehicles, transportation modes and devices may be tracked by analyzing the spatial positions provided by these instruments. Analysts may need to explore and analyze different paths followed by the users. The advances in position-acquisition and ubiquitous devices have granted extremely large location data, which indicate the mobility of different moving targets such as autonomous vehicles, pedestrians, natural phenomena, etc. The commonness of these datasets calls for novel approaches in order to discover information and mine the data. Traditionally, researchers analyse GPS logs by defining a distance function (e.g, KNN) between two trajectories and then applying expensive processing algorithms to address the similarity detection. For example, they first convert the trajectories into a set of road segments by leveraging map-matching algorithms. Afterwards, the relationship between trajectories is managed using indexing structures. Using the data provided by Zheng et al., we seek to investigate different locations followed by the users in Beijing. The data consists of GPS trajectories collected for the Geolife project by 182 users during a period of over five years (from April 2007 to August 2012). Each trajectory is represented by a 3D latitude, longitude and altitude point. A range of users' outdoor movements were recorded, including life routines such as travelling to work, sports, shopping, etc. As the quantity of GPS data is becoming increasingly large and complex, proper brushing is challenging. Using bounding boxes somewhat alleviate this difficulty by setting the key of interest on the major corners. However, many boxes must be placed carefully for one single selection. The boxes can help the analysts to select all the trajectories that pass through a specific location, but do not simplify the analysis of overlapping and directional trajectories. This study's approach intuitively supports path differentiation for both overlapping trajectories and takes direction into account. For example, we are interested in answering questions about the activities people perform and their sequential order. For this dataset, the authors were interested in finding event sequences that could inform tourists. The shape based brushing could serve as a tool to further explore their . For example, if they find an interesting classical sequence that passes through locations A and B they can further explore if this sequence corresponds to a larger sequence and what other locations are visited before or after. A first brushing and refinement using the FPCA metric and small multiples enables them to select all the trajectories that include a precise event sequence passing through a set of locations, as depicted in Figure 10. A second brushing using the Pearson metric enables further explorations that also take into account the direction of the trajectories. Switching between the correlated trajectories and the anti-correlated ones, the user can gain insight about the visitation order of the selected locations. The proposed brushing technique leverages existing methods with the novel usage of the shape of the brush as an additional filtering parameter. The interaction pipeline shows different data processing steps where the comparison algorithm between the brushed items and the shape of the brush plays a central role. While the presented pipeline contains two specific and complementary comparison metric computations, another one can be used as long as it fulfills the continuity and metric se- Figure 10. Three different trajectories containing three different event sequences from. mantic requirements (DR2). There are indeed many standard approaches (ED, DTW, Discrete FrÃl'chet distance) that are largely used by the community and could be used to extend our technique when faced with different datasets. Furthermore, the contribution of this paper is a novel shape-based brushing technique and not simply a shape similarity measure. In our work, we found two reasonable similarity measures that fulfill our shape-based brushing method: The FPCA distance comparison provides an accurate curve similarity measurement while the Pearson metric provides a complementary criteria with the direction of the trajectory. In terms of visualization, the binning process provides a valuable overview of the order of the trajectory shapes. This important step eases the filtering and adjustment of the selected items. It is important to mention that this filtering operates in a continuous manner as such trajectories are added or removed one by one when adjusting this filtering parameter. This practice helps to fine tune the selected items with accurate filtering parameters. The presented scenario shows how small multiple interaction can provide flexibility. This is especially the case when the user brushes specific trajectories to be then removed when setting the compatibility metrics to uncorrelated. This operation performs a brush removal. The proposed filtering method can also consider other types of binning and allows different possible representations (i.e. various visual mapping solutions). This paper illustrates the shape based brushing technique with three application domains (air traffic, eye tracking, gps data), but it can be extended to any moving object dataset. However, our evaluation is limited by the number of studied application domains. Furthermore, even if various users and practitioners participated in the design of the technique, and assessed the simplicity and intuitiveness of the method, we did not conduct a more formal evaluation. The shape based brush is aimed at complementing the traditional brush, and in no way do we argue that it is more efficient or effective than the original technique for all cases. The scenarios are examples of how this technique enables the selection of trails that would be otherwise difficult to manipulate, and how the usage of the brush area and its shape to perform comparison opens novel brushing perspectives. We believe they provide strong evidence of the potential of such a technique. The technique also presents limitations in its selection flexibility, as it is not yet possible to combine selections. Many extensions can be applied to the last step of the pipeline to support this. This step mainly addresses the DR4 where the selection can be refined thanks to user inputs. As such, multiple selections can be envisaged and finally be composed. Boolean operations can be considered with the standard And, Or, Not. While this composition is easy to model, it remains difficult for an end user to master the operations when there are more than 2 subset operations. As a solution, Hurter et al. proposed an implicit item composition with a simple drag and drop technique. The pipeline can be extended with the same paradigm where a place holder can store filtered items and then be composed to produce the final . The user can then refine the selection by adding, removing or merging multiple selections. In this paper, a novel sketch-based brushing technique for trail selection was proposed and investigated. This approach facilitates user selection in occluded and cluttered data visualization where the selection is performed on a standard brush basis while taking into account the shape of the brush area as a filtering tool. This brushing tool works as follows. Firstly, the user brushes the trajectory of interest trying to follow its shape as closely as possible. Then the system pre-selects every trajectory which touches the brush area. Next, the algorithm computes a distance between every brushed shape and the shape of the brushed area. Comparison scores are then sorted and the system displays visual bins presenting trajectories from the lowest scores (unrelated -or dissimilar trajectories) to the highest values/scores (highly correlated or similar trajectories). The user can then adjust a filtering parameter to refine the actual selected trajectories that touch the brushed area and which have a suitable correlation with the shape of the brushed area. The cornerstone of this shape-based technique relies on the shape comparison method. Therefore, we choose two algorithms which provide enough flexibility to adjust the set of selected trajectories. One algorithm relies on functional decomposition analysis which insures a shape curvature comparison, while the other method insures an accurate geometric based comparison (Pearson algorithm). To validate the efficiency of this method, we show three examples of usage with various types of trail datasets. This work can be extended in many directions. We can first extend it with additional application domains and other types of dataset such as car or animal movements or any type of time varying data. We can also consider other types of input to extend the mouse pointer usage. Virtual Reality data exploration with the so called immersive analytic domain gives a relevant work extension which will be investigated in the near future. Finally, we can also consider adding machine learning to help users brush relevant trajectories. For instance, in a very dense area, where the relevant trajectories or even a part of the trajectories are not visible due to the occlusion, additional visual processing may be useful to guide the user during the brushing process.

Represent the short summary for finding the corresponding scientific article (long)!

Interactive technique to improve brushing in dense trajectory datasets by taking into account the shape of the brush.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the article for finding a single-sentence tldr

Question: Generative Adversarial Networks (GANs) can produce images of surprising complexity and realism, but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene. Capturing such complex interactions between different objects in the world, including their relative scaling, spatial layout, occlusion, or viewpoint transformation is a challenging problem. In this work, we propose to model object composition in a GAN framework as a self-consistent composition-decomposition network. Our model is conditioned on the object images from their marginal distributions and can generate a realistic image from their joint distribution. We evaluate our model through qualitative experiments and user evaluations in scenarios when either paired or unpaired examples for the individual object images and the joint scenes are given during training. Our reveal that the learned model captures potential interactions between the two object domains given as input to output new instances of composed scene at test time in a reasonable fashion. Generative Adversarial Networks (GANs) have emerged as a powerful method for generating images conditioned on a given input. The input cue could be in the form of an image BID1 BID20, a text phrase BID32 BID23 a; BID10 or a class label layout BID18 BID19 BID0. The goal in most of these GAN instantiations is to learn a mapping that translates a given sample from source distribution to generate a sample from the output distribution. This primarily involves transforming either a single object of interest (apples to oranges, horses to zebras, label to image etc.), or changing the style and texture of the input image (day to night etc.). However, these direct input-centric transformations do not directly capture the fact that a natural image is a 2D projection of a composition of multiple objects interacting in a 3D visual world. In this work, we explore the role of compositionality in learning a function that maps images of different objects sampled from their marginal distributions (e.g., chair and table) into a combined sample (table-chair) that captures their joint distribution. Modeling compositionality in natural images is a challenging problem due to the complex interaction possible among different objects with respect to relative scaling, spatial layout, occlusion or viewpoint transformation. Recent work using spatial transformer networks BID9 within a GAN framework BID14 decomposes this problem by operating in a geometric warp parameter space to find a geometric modification for a foreground object. However, this approach is only limited to a fixed and does not consider more complex interactions in the real world. Another recent work on scene generation conditioned on text and a scene graph and explicitly provides reasoning about objects and their relations BID10.We develop a novel approach to model object compositionality in images. We consider the task of composing two input object images into a joint image that captures their joint interaction in natural images. For instance, given an image of a chair and a table, our formulation should be able to generate an image containing the same chair-table pair interacting naturally. For a model to be able to capture the composition correctly, it needs to have the knowledge of occlusion ordering, i.e., a table comes in front of chair, and spatial layout, i.e., a chair slides inside table. To the best of our knowledge, we are among the first to solve this problem in the image conditional space without any prior explicit information about the objects' layout. Our key insight is to reformulate the problem of composition of two objects into first composing the given object images to generate the joint combined image which models the object interaction, and then decomposing the joint image back to obtain individual ones. This reformulation enforces a selfconsistency constraint ) through a composition-decomposition network. However, in some scenarios, one does not have access to the paired examples of same object instances with their combined compositional image, for instance, to generate the joint image from the image of a given table and a chair, we might not have any example of that particular chair besides that particular table while we might have images of other chairs and other tables together. We add an inpainting network to our composition-decomposition layers to handle the unpaired case as well. Through qualitative and quantitative experiments, we evaluate our proposed Compositional-GAN approach in two training scenarios: (a) paired: when we have access to paired examples of individual object images with their corresponding composed image, (b) unpaired: when we have a dataset from the joint distribution without being paired with any of the images from the marginal distributions. Generative adversarial networks (GANs) have been used in a wide variety of settings including image generation BID4 BID31 BID11 and representation learning BID24 BID15. The loss function in GANs have been shown to be very effective in optimizing high quality images conditioned on available information. Conditional GANs BID18 generate appealing images in a variety of applications including image to image translation both in the case of paired and unpaired data, inpainting missing image regions BID20 BID30, generating photorealistic images from labels BID18 BID19, and solving for photo super-resolution BID13 BID12.Image composition is a challenging problem in computer graphics where objects from different images are to be overlayed in one single image. Appearance and geometric differences between these objects are the obstacles that can in non-realistic composed images. BID35 addressed the composition problem by training a discriminator network that could distinguish realistic composite images from synthetic ones. BID27 developed an end-to-end deep CNN for image harmonization to automatically capture the context and semantic information of the composite image. This model outperformed its precedents BID26 BID29 which transferred statistics of hand-crafted features to harmonize the foreground and the in the composite image. Recently, BID14 used spatial transformer networks as a generator by performing geometric corrections to warp a masked object to adapt to a fixed image. Moreover, BID10 computed a scene layout from given scene graphs which revealed an explicit reasoning about relationships between objects and converted the layout to an output image. Despite the success all these approaches gained in improving perceptual realism, they lack the realistic complex problem statement where no explicit prior information about the scene layout is given. In the general case which we address, each object should be rotated, scaled, and translated in addition to occluding others and/or being occluded to generate a realistic composite image. We propose a generative network for composing two objects by learning how to handle their relative scaling, spatial layout, occlusion, and viewpoint transformation. Given a set of images from the marginal distribution of the first object, X = {x 1, · · ·, x n}, and a set of images from the marginal distribution of the second object, Y = {y 1, · · ·, y n}, in addition to a set of real images from their joint distribution containing both objects, C = {c 1, · · ·, c n}, we generate realistic composite images containing objects given from the first two sets. We propose a conditional generative adversarial network for two scenarios: paired inputs-output in the training set where each image in C is correlated with an image in X and one in Y, and unpaired training data where images in C are not paired with images in X and Y. It is worth noting that our goal is not to learn a generative model of all possible compositions, but learn to output the mode of the distribution. The modular components of our proposed approach are critical in learning the mode of plausible compositions. For instance, our relative appearance flow network handles the viewpoint and the spatial transformer network handles affine transformation eventually making the generator invariant to these transformations. In the following sections, we first summarize a conditional generative adversarial network, and then will discuss our network architecture and its components for the two circumstances. Starting from a random noise vector, z, GANs generate images c of a specific distribution by adversarially training a generator, G, versus a discriminator, D. While the generator tries to produce realistic images, the discriminator opposes the generator by learning to distinguish between real and fake images. In the conditional GAN models (cGANs), an auxiliary information, x, in the form of an image or a label is fed into the model alongside the noise vector ({x, z} → c); BID18. The objective of cGANs would be therefore an adversarial loss function formulated as: z) )] where G and D minimize and maximize this loss function, respectively. DISPLAYFORM0 The convergence of the above GAN objective and consequently the quality of generated images would be improved if an L 1 loss penalizing deviation of generated images from their ground-truth is added. Thus, the generator's objective function would be summarized as: DISPLAYFORM1 In our proposed compositional GAN, model ({(x, y), z} → c) is conditioned on two input images, (x, y), concatenated channel-wise in order to generate an image from the target distribution p data (c).We have access to real samples of these three distributions during training (in two paired and unpaired scenarios). Similar to, we ignore random noise as the input to the generator, and dropout is the only source of randomness in the network. In some specific domains, the relative view point of the objects should be changed accordingly to generate a natural composite image. Irrespective of the paired or unpaired inputs-output cases, we train a relative encoder-decoder appearance flow network BID34, G RAFN, taking the two input images and synthesizing a new viewpoint of the first object, DISPLAYFORM0, given the viewpoint of the second one, y i encoded in its binary mask. The relative appearance flow network is trained on a set of images in X with arbitrary azimuth angles α i ∈ {−180•, −170 DISPLAYFORM1 •} along with their target images in an arbitrary new viewpoint with azimuth angle θ i ∈ {−180•, −170 DISPLAYFORM2 •} and a set of foreground masks of images in Y in the target viewpoints. The network architecture for our relative appearance flow network is illustrated in the appendix and its loss function is formulated as: DISPLAYFORM3 DISPLAYFORM4 As mentioned above, G RAFN is the encoder-decoder network predicting appearance flow vectors, which after a bilinear sampling generates the synthesized view. The encoder-decoder mask generating network, G M RAFN, shares weights in its encoder with G RAFN, while its decoder is designed for predicting foreground mask of the synthesized image. Moreover, x r i is the ground-truth image for x i in the new viewpoint,M fg xi is its predicted foreground mask, and M fg xi, M fg yi are the ground-truth foreground masks for x i and y i, respectively. In this section, we propose a model, G, for composing two objects when there is a corresponding composite real image for each pair of input images in the training set. In addition to the relative AFN discussed in the previous section, to relatively translate the center-oriented input objects, we train our variant of the spatial transformer network (STN) BID9 which simultaneously takes the two RGB images, x Figure 1: Compositional GAN training model both for paired and unpaired training data. The yellow box refers to the RAFN step for synthesizing a new viewpoint of the first object given the foreground mask of the second one, which will be applied only during training with paired data. The orange box represents the process of inpainting the input segmentations for training with unpaired data. Rest of the model would be similar for the paired and unpaired cases which includes the STN followed by the self-consistent composition-decomposition network. The main backbone of our proposed model consists of a self-consistent composition-decomposition network both as conditional generative adversarial networks. The composition network, G c, takes the two translated input RGB images, x T i and y i T, concatenated channel-wise in a batch, with size N × 6 × H × W, and generates their corresponding output,ĉ i, with size N × 3 × H × W composed of the two input images appropriately. This generated image will be then fed into the decomposition network, G dec, to be decomposed back into its constituent objects,x T i andŷ T i in addition to G M dec that predicts probability segmentation masks of the composed image,M xi andM yi. The two decomposition components G dec and G M dec share their weights in their encoder network but are different in the decoder. We assume the ground-truth foreground masks of the inputs and the target composite image are available, thus we remove from all images in the network for simplicity. A GAN loss with gradient penalty BID6 ) is applied on top of generated imagesĉ i,x T i,ŷ T i to make them look realistic in addition to multiple L 1 loss functions penalizing deviation of generated images from their ground-truth. An schematic of our full network, G, is represented in FIG2 and the loss function is summarized as: DISPLAYFORM0, y i ), and x c i, y c i are the ground-truth transposed full object inputs corresponding to c i. Moreover, L L1 (G dec) is the self-consistency constraint penalizing deviation of decomposed images from their corresponding transposed inputs and L CE is the cross entropy loss applied on the predicted probability segmentation masks. We also added the gradient penalty introduced by BID6 to improve convergence of the GAN loss functions. If viewpoint transformation is not needed for the objects' domain, one can replace x RAFN i with x i in the above equations. The benefit of adding decomposition networks will be clarified in Section 3.5. Here, we propose a variant of our model discussed in section 3.3 for broader object domains where paired inputs-outputs are not available or hard to collect. In this setting, there is not one-to-one mapping between images in sets X, Y and images in the composite domain, C. However, we still assume that foreground and segmentation masks are available for images in all three sets during training. Therefore, the is again removed for simplicity. Given the segmentation masks, M xi, M yi, of the joint ground-truth image, c i, we first crop and resize object segments x c i,s = c i M xi and y c i,s = c i M yi to be at the center of the image, similar to the input center-oriented objects at test time, calling them as x i,s and y i,s. For each object, we add a self-supervised inpainting network BID20, G f, as a component of our compositional GAN model to generate full objects from the given segments of image c i, reinforcing the network to learn object occlusions and spatial layouts more accurately. For this purpose, we apply a random mask on each x i ∈ X to zero out pixel values in the mask region and train a conditional GAN, G x f, to fill in the missing regions. To guide this masking process toward a similar task of generating full objects from segmentations, we can use the foreground mask of images in Y for zeroing out images in X. Another cGAN network, G y f, should be trained similarly to fill in the missing regions of masked images in Y. The loss function for each inpainting network would be as: DISPLAYFORM0 Therefore, starting from two inpainting networks trained on sets X and Y, we generate a full object from each segment of image c i both for the center oriented segments, x i,s and y i,s, and the original segments, x, we can train a spatial transformer network similar to the model for paired data discussed in section 3.3 followed by the composition-decomposition networks to generate composite image and its probability segmentation masks. Since we start from segmentations of the joint image rather than an input x i from a different viewpoint, we skip training the RAFN end-to-end in the compositional network, and use its pre-trained model discussed in section 3.2 at test time. After training the network, we study performance of the model on new images, x, y, from the marginal distributions of sets X and Y along with their foreground masks to generate a naturallooking composite image containing the two objects. However, since generative models cannot generalize very well to a new example, we continue optimizing network parameters given the two input test instances to remove artifacts and generate sharper BID1. Since the ground-truth for the composite image and the target spatial layout of the objects are not available at test time, the self-consistency cycle in our decomposition network provides the only supervision for penalizing deviation from the original objects through an L 1 loss. We freeze the weights of the relative spatial transformer and appearance flow networks, and only refine the weights of the composition-decomposition layers where the GAN loss will be applied given the real samples from our training set. We again ignore for simplicity given the foreground masks of the input instances. The ground-truth masks of the transposed full input images, M fg x, M fg y can be also obtained by applying the pre-trained RAFN and STN on the input masks. We then use the Hadamard product to multiply the predicted masksM x,M y with the objects foreground masks M fg x, M fg y, respectively to eliminate artifacts outside of the target region for each object. One should note that M fg x, M fg y are the foreground masks of the full transposed objects whileM x andM y are the predicted segmentation masks. Therefore, the loss function for this refinement would be: Test on the chair-table (A) and basket-bottle (B) composition tasks trained with either paired or unpaired data. "NN" stands for the nearest neighbor image in the paired training set, and "NoInpaint" shows the of the unpaired model without the inpainting network. In both paired and unpaired cases,ĉ before andĉ after show outputs of the generator before and after the inference refinement network, respectively. Also,ĉ after s represents summation of masked transposed inputs after the refinement step. wherex T,ŷ T are the generated decomposed images, and x T and y T are the transposed inputs. DISPLAYFORM0 DISPLAYFORM1 In the experiments, we will present: images generated directly from the composition network,ĉ, before and after this refinement step, images generated directly based on the predicted segmentation masks asĉ s =M x x T +M y y T. In this section, we study the performance of our compositional GAN model for both the paired and unpaired scenarios through multiple qualitative and quantitative experiments in different domains. First, we use the Shapenet dataset BID2 as our main source of input objects and study two composition tasks: a chair next to a table, a bottle in a basket. Second, we show our model performing equally well when one object is fixed and the other one is relatively scaled and linearly transformed to generate a composed image. We present our on the CelebA dataset BID17 composed with sunglasses downloaded from the web. In all our experiments, the values for the training hyper-parameters are set to λ 1 = 100, λ 2 = 50, λ 3 = 1, and the inference λ = 100. Composition of a chair and a table is a challenging problem since viewpoints of the two objects should be similar and one object should be partially occluded and/or partially occlude the other one depending on their viewpoint. This problem cannot be resolved by considering each object as a separate individual layer of an image. By feeding in the two objects simultaneously to our proposed network, the model learns to relatively transform each object and composes them reasonably. We manually made a collection of 1K composite images from Shapenet chairs and tables which can be used for both the paired and unpaired training models. In the paired scenario, we use the pairing information between each composite image and its constituent full chair and table besides their foreground masks. On the other hand, to show the performance of our model on the unpaired examples as well, we ignore the individual chairs and tables used in each composite image, and use a different subset of Shapenet chairs and tables as real examples of each individual set. We made sure that these two subsets do not overlap with the chairs and tables in composite images to avoid the occurrence of implicit pairing in our experiments. Chairs and tables in the input-output sets can pose in a random azimuth angle in the range [−180 DISPLAYFORM0 •] at steps of 10 •. As discussed in section 3.2, feeding in the foreground mask of an arbitrary table with a random azimuth angle in addition to the input chair to our trained relative appearance flow network synthesizes the chair in the viewpoint consistent with the table. Synthesized test chairs as X RAFN are represented in the third row of FIG3 -A.In addition, to study our network components, we visualize model outputs at different steps in FIG3 -A. To evaluate our network with paired training data on a new input chair and table represented as X and Y, respectively, we find its nearest neighbor composite example in the training set in terms of its constituent chair and table features extracted from a pre-trained VGG19 network BID25. As shown in the fourth row of FIG3 -A, nearest neighbors are different enough to be certain that network is not memorizing its training data. We also illustrate output of the network before and after the inference refinement step discussed in section 3.5 in terms of the generator's prediction,ĉ, as well as the direct summation of masked transposed inputs,ĉ s, for both paired and unpaired training models. The refinement step sharpens the synthesized image and removes artifacts generated by the model. Our from the model trained on unpaired data shown in the figure is comparable with those from paired data. Moreover, we depict the performance of the model without our inpainting network in the eighth row, where occlusions are not correct in multiple examples. FIG3 -A emphasizes that our model has successfully resolved the challenges involved in this composition task, where in some regions such as the chair handle, table is occluding the chair while in some other regions such as table legs, chair is occluding the table. More exemplar images as well as some of the failure cases for both paired and unpaired scenarios are presented in Appendix C.1.We have also conducted an Amazon Mechanical Turk evaluation BID33 to compare the performance of our algorithm in different scenarios including training with and without paired data and before and after the final inference refinement network. From a set of 90 test images of chairs and tables, we have asked 60 evaluators to select their preferred composite image generated by the model trained on paired data versus images generated by the model trained on unpaired data, both after the inference refinement step. As a , 57% of the composite images generated by our model trained on paired inputs-outputs were preferred to the ones generated through the unpaired scenario. It shows that even without paired examples during training, our proposed model performs reasonably well. We have repeated the same study to compare the quality of images generated before and after the inference refinement step, where the latter was preferred 71.3% of the time to the non-refined images revealing the benefit of the the last refinement module in generating higher-quality images. In this experiment, we address the compositional task of putting a bottle in a basket. Similar to the chair-table problem, we manually composed Shapenet bottles with baskets to prepare a training set of 100 paired examples. We trained the model both with and without the paired data, similarly to section 4.1, and represent outputs of the network before and after the inference refinement in FIG3 -B. In addition, nearest neighbor examples in the paired training set are shown for each new input instance (fourth row) as well as the model's predictions in the unpaired case without the inpainting network (eighth column). As clear from the , our inpainting network plays a critical role in the success of our unpaired model specially for handling occlusions. This problem statement is similarly interesting since the model should identify which pixels to be occluded. For instance in the first column of FIG3 -B, the region inside the basket is occluded by the blue bottle while the region outside is occluding the latter. More examples are shown in Appendix C.2.Similarly, we evaluate the performance of our model on this task through an Amazon Mechanical Turk study with 60 evaluators and a set of 45 test images. In summary, outputs from our paired training were preferred to the unpaired case 57% of the time and the inference refinement step was Test examples for the face-sunglasses composition task. Top two rows: input sunglasses and face images; 3rd and 4th rows: the output of our compositional GAN for the paired and unpaired models, respectively; Last row: images generated by the ST-GAN BID14 model.observed to be useful in 64% of examples. These confirm the benefit of the refinement module and the comparable performance of training in the unpaired scenario with the paired training case. Ablation Studies and Other Baselines: In Appendix C.3, we repeat the experiments with each component of the model removed at a time to study their effect on the final composite image. In addition, in Appendix C.4, we show the poor performance of two baseline models (CycleGAN and Pix2Pix) in a challenging composition task. In this section, we compose a pair of sunglasses with a face image, similar to BID14, where the latter should be fixed while sunglasses should be rescaled and transformed relatively. We used the CelebA dataset BID17, followed its training/test splits and cropped images to 128 × 128 pixels. We hand-crafted 180 composite images of celebrity faces from the training split with sunglasses downloaded from the web to prepare a paired training set. However, we could still use our manual composite set for the unpaired case with access to the segmentation masks separating sunglasses from faces. In the unpaired scenario, we used 6K images from the training split while not overlapping with our composite images to be used as the set of individual faces during training. In this case, since pair of glasses is always occluding the face, we report based on summation of the masked transposed inputs,ĉ s for both the paired training data and the unpaired one. We also compare our with the ST-GAN model BID14 which assumes images of faces as a fixed and warps the glasses in the geometric warp parameter space. Our both in paired and unpaired cases, shown in FIG4,look more realistic in terms of the scale, rotation angle, and location of the sunglasses with the cost of only 180 paired training images or 180 unpaired images with segmentation masks. More example images are illustrated in Appendix C.5.To confirm this observation, we have studied the by asking 60 evaluators to score our model predictions versus ST-GAN on a set of 75 test images. According to this study where we compare our model trained on paired data with ST-GAN, 84% of the users evaluated favorably our network predictions. Moreover, when comparing ST-GAN with our unpaired model, 73% of the evaluators selected the latter. These confirm the ability of our model in generalizing to the new test examples and support our claim that both our paired and unpaired models significantly outperform the recent ST-GAN model in composing a face with a pair of sunglasses. In this paper, we proposed a novel Compositional GAN model addressing the problem of object composition in conditional image generation. Our model captures the relative linear and viewpoint transformations needed to be applied on each input object (in addition to their spatial layout and occlusions) to generate a realistic joint image. To the best of our knowledge, we are among the first to solve the compositionality problem without having any explicit prior information about object's layout. We evaluated our compositional GAN through multiple qualitative experiments and user evaluations for two cases of paired versus unpaired training data. In the future, we plan to extend this work toward generating images composed of multiple (more than two) and/or non-rigid objects. Architecture of our relative appearance flow network is illustrated in FIG5 which is composed of an encoder-decoder set of convolutional layers for predicting the appearance flow vectors, which after a bilinear sampling generates the synthesized view. The second decoder (last row of layers in FIG5) is for generating foreground mask of the synthesized image following a shared encoder network BID34. All convolutional layers are followed by batch normalization BID7 and a ReLU activation layer except for the last convolutional layer in each decoder. In the flow decoder, output is fed into a Tanh layer while in the mask prediction decoder, the last convolutional layer is followed by a Sigmoid layer to be in the range. Diagram of our relative spatial transformer network is represented in FIG6. The two input images (e.g., chair and table) are concatenated channel-wise and fed into the localization network to generate two set of parameters, θ 1, θ 2 for the affine transformations. This single network is simultaneously trained on the two images learning their relative transformations required to getting close to the given target images. In this figure, orange feature maps are the output of a conv2d layer (represented along with their corresponding number of channels and dimensions) and yellow maps are the output of max-pool2d followed by ReLU. The blue layers also represent fully connected layers. Figure 6 and a few failure test examples in FIG8 for both paired and unpaired training models. Here, viewpoint and linear transformations in addition to occluding object regions should be performed properly to generate a realistic image. Figure 6: Test on the chair-table composition task trained with either paired or unpaired data. "NN" stands for the nearest neighbor image in the paired training set, and "NoInpaint" shows the of the unpaired model without the inpainting network. In both paired and unpaired cases, c before andĉ after show outputs of the generator before and after the inference refinement network, respectively. Also,ĉ after s represents summation of masked transposed inputs after the refinement step. In the bottle-basket composition, the main challenging problem is the relative scale of the objects besides their partial occlusions. In Figure 8, we visualize more test examples and study the performance of our model before and after the inference refinement step for both paired and unpaired scenarios. The third column of this figure represents the nearest neighbor training example found for each new input pair, (X, Y), through their features extracted from the last layer of the pre-trained VGG19 network BID25. Moreover, the seventh column shows outputs of the trained unpaired network without including the inpainting component during training revealing the necessity of the inpainting network while training with unpaired data. We repeat the experiments on composing a bottle with a basket, with each component of the model removed at a time, to study their effect on the final composite image. Qualitative are illustrated in FIG9. First and second columns show bottle and basket images which are concatenated channelwise as the input to the network. Following columns are: -3rd column: no reconstruction loss on the composite image in wrong color and faulty occlusion, -4th column: no cross-entropy mask loss in training in faded bottles, -5th column: no GAN loss in training and inference generates outputs with a different color and lower quality than the input image, -6th column: no decomposition generator (G dec) and self-consistent cycle in partially missed bottles, -7, 8th columns represent full model in paired and unpaired scenarios. Figure 8: More test on the basket-bottle composition task trained with either paired or unpaired data. "NN" stands for the nearest neighbor image in the paired training set, and "NoInpaint" shows the of the unpaired model without the inpainting network. In both paired and unpaired cases, c before andĉ after show outputs of the generator before and after the inference refinement network, respectively. Also,ĉ after s represents summation of masked transposed inputs after the refinement step. The purpose of our model is to capture object interactions in the 3D space projected onto a 2D image by handling their spatial layout, relative scaling, occlusion, and viewpoint for generating a realistic image. These factors distinguish our model from CycleGAN and Pix2Pix ) models whose goal is only changing the appearance of the given image. In this section, we compare to these two models. To be able to compare, we use the mean scaling and translating parameters of our training set to place each input bottle and basket together and have an input with 3 RGB channels (9th column in FIG9). We then train a ResNet generator on our paired training data with an adversarial loss added with a L 1 regularizer. Since the structure of the input image might be different from its corresponding ground-truth image (due to different object scalings and layouts), ResNet model works better than a U-Net but still generating unrealistic images (10th column in FIG9). We follow the same approach for the unpaired data through the CycleGAN model (11th column in FIG9). As apparent from the qualitative , it is not easy for either Pix2Pix or CycleGAN networks to learn the transformation between samples from the input distribution and that of the occluded outputs. Adding a pair of sunglasses to an arbitrary face image requires a proper linear transformation of the sunglasses to align well with the face. We illustrate test examples of this composition problem in FIG2 including of both the paired and unpaired training scenarios in the third and fourth columns. In addition, the last column of each composition example case represents the outputs of the ST-GAN model BID14.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We develop a novel approach to model object compositionality in images in a GAN framework.

Represent the TLDR for finding the corresponding scientific article

Conditional VAE on top of latent spaces of pre-trained generative models that enables  transfer between drastically different domains while preserving locality and semantic alignment.