Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Knowledge Graphs (KG), composed of entities and relations, provide a structured representation of knowledge. For easy access to statistical approaches on relational data, multiple methods to embed a KG as components of R^d have been introduced. We propose TransINT, a novel and interpretable KG embedding method that isomorphically preserves the implication ordering among relations in the embedding space. TransINT maps set of entities (tied by a relation) to continuous sets of vectors that are inclusion-ordered isomorphically to relation implications. With a novel parameter sharing scheme, TransINT enables automatic training on missing but implied facts without rule grounding. We achieve new state-of-the-art performances with signficant margins in Link Prediction and Triple Classification on FB122 dataset, with boosted performance even on test instances that cannot be inferred by logical rules. The angles between the continuous sets embedded by TransINT provide an interpretable way to mine semantic relatedness and implication rules among relations. Recently, learning distributed vector representations of multi-relational knowledge has become an active area of research (Bordes et al.; Nickel et al.; Kazemi & Poole; Wang et al.; Bordes et al.). These methods map components of a KG (entities and relations) to elements of R d and capture statistical patterns, regarding vectors close in distance as representing similar concepts. However, they lack common sense knowledge which are essential for reasoning (Wang et al.; Guo et al.; Nickel & Kiela). For example, "parent" and "father" would be deemed similar by KG embeddings, but by common sense, "parent ⇒ father" yet not the other way around. Thus, one focus of current research is to bring common sense rules to KG embeddings (Guo et al.; Wang et al.; Wei et al.( . Some methods impose hard geometric constraints and embed asymmetric orderings of knowledge (Nickel & Kiela; Vendrov et al.; Vilnis et al.( . However, they only embed hierarchy (unary Is_a relations), and cannot embed n-ary relations in KG's. Moreover, their hierarchy learning is largely incompatible with conventional relational learning, because they put hard constraints on distance to represent partial ordering, which is a common metric of similarity/ relatedness in relational learning. We propose TransINT, a new KG embedding method that isomorphically preserves the implication ordering among relations in the embedding space. TransINT restrict entities tied by a relation to be embedded to vectors in a particular region of R d included isomorphically to the order of relation implication. For example, we map any entities tied by is_father_of to vectors in a region that is part of the region for is_parent_of; thus, we can automatically know that if John is a father of Tom, he is also his parent even if such a fact is missing in the KG. Such embeddings are constructed by sharing and rank-ordering the basis of the linear subspaces where the vectors are required to belong. Mathematically, a relation can be viewed as sets of entities tied by a constraint (Stoll). We take such a view on KG's, since it gives consistancy and interpretability to model behavior. Furthermore, for the first time in KG embedding, we map sets of entitites under relation constraint to a continuous set of points (whose elements are entity vectors) -which learns relationships among not only individual entity vectors but also sets of entities. We show that angles between embedded relation sets can identify semantic patterns and implication rules -an extension of the line of thought as in word/ image embedding methods such as Mikolov et al., Frome et al. to relational embedding. Such mining is both limited and less interpretable if embedded sets are discrete (Vilnis et al.; Vendrov et al.) or each entitity itself is embedded to a region, not a member vector of it (Vilnis et al.). 1 TransINT's such interpretable meta-learning opens up possibilities for explainable reasoning in applications such as recommender systems (Ma et al.) and question answering (Hamilton et al. In this section, we describe the intuition and justification of our method. We first define relation as sets, and revisit TransH as mapping relations to sets in R d . Finally, we propose TransINT, which connects the ordering of the two aforementioned sets. We put * next to definitions and theorems we propose/ introduce. Otherwise, we use existing definitions and cite them. We define relations as sets and implication as inclusion of sets, as in set-theoretic logic. Definition (Relation Set): Let r i be a binary relation x, y entities. Then, r i (x, y) iff there exists some set R i such that the pair (x, y) ∈ R i. R i is called the relation set of r i. (Stoll) For example, consider the distinct relations in Figure 1a, and their corresponding sets in Figure 1b; Is_Father_Of(Tom, Harry) is equivalent to (Tom, Harry) ∈ R Is_Father_Of. Definition (Logical Implication): For two relations, r 1 implies r 2 (or r 1 ⇒ r 2) iff ∀x, y,; the orange dot is the origin, to emphasize that a vector is really a point from the origin but can be translated and considered equivalently. (a): first projecting #» h and #» t onto H is_f amily_of, and then requiring: first substracting #» t from #» h, and then projecting the distance (# » t − h) to H is_f amily_of and requiring (# » t − h) ⊥ ≈ rj. The red line is unique because it is when # » r is_f amily_of is translated to the origin. 2.2 : TRANSE AND TRANSH Given a fact triple (h, r, t) in a given KG (i.e. (Harry, is_father_of, Tom)), TransE wants #» h + #» r ≈ #» t where #» h, #» r, #» t are embeddings of h, r, t. In other words, the distance between two entity vectors is equal to a fixed relation vector. TransE applies well to 1-to-1 relations but has issues for N-to-1, 1-to-N and N-to-N relations, because the distance between two vectors are unique and thus two entities can only be tied with one relation. To address this, TransH constraints the distance of entities in a multi-relational way, by decomposing distance with projection (Figure 2a). TransH first projects an entity vector into a hyperplane unique to each relation, and then requires their difference is some constant value. Like TransE, it embeds an entity to a vector. However, for each relation r j, it assigns two components: a relation-specific hyperplane H j and a fixed vector #» r j on H j. For each fact triple (h, r j, t), TransH wants (Figure 2) where (Figure 2a). Revisiting TransH We interpret TransH in a novel perspective. An equivalent way to put Eq.1 is to change the order of subtraction and projection: This means that all entity vectors (#» h, #» t) such that their distance # » t − h belongs to the red line are considered to be tied by relation r j (Figure 2b) i.e. R j ≈ the red line. For example, The red line is the set of all vectors whose projection onto H j is the fixed vector #» r j. Thus, upon a deeper look, TransH actually embeds a relation set in KG (figure 1b) to a particular set in R d. We call such sets relation space for now; in other words, a relation space of some relation r i is the space where each (h, r i, t)'s # » t − h can exist. We formally visit it later in Section 3.1.. (Figure 2b) perspective. The blue line, red line, and the green plane is respectively is_father_of, is_mother_of and is_parent_of's relation space -where # » t − h's of h, t tied by these relations can exist. The blue and the red line lie on the green plane -is_parent_of's relation space includes the other two's. 2.3 TRANSINT Like TransH, TransINT embeds a relation r j to a (subspace, vector) pair (H j, #» r j). However, TransINT modifies the relation embeddings (H j, #» r j) so that the relation spaces (i.e. red line of Figure 2b) are ordered by implication; we do so by intersecting the H j's and projecting the #» r j's (Figure 3a). We explain with familial relations as a running example. Intersecting the H j's TransINT assigns distinct hyperplanes H is_f ather_of and H is_mother_of to is_father_of and is_mother_of. However, because is_parent_of is implied by the aforementioned relations, we assign H is_parent_of = H is_f ather_of ∩ H is_mother_of. TrainsINT's H is_parent_of is not a hyperplane but a line (Figure 3a), unlike in TransH where all H j's are hyperplanes. Projecting the #» r j's TransH constrains the #» r j's with projections (Figure 3a 's dotted orange lines). First, # » r is_f ather_of and # » r is_mother_of are required to have the same projection onto H is_parent_of. Second, # » r is_parent_of is that same projection onto H is_parent_of. We connect the two above constraints to ordering relation spaces. Figure 3b graphically illustrates that is_parent_of's relation space (green hyperplane) includes those of is_father_of (blue line) and is_mother_of (red line). More generally, TransINT requires two hard geometric constraints on (H j, #» r j)'s that For distinct relations r i, r j, require the following if and only if r i ⇒ r j: Intersection Constraint: We prove that these two constraints guarantee that an ordering isomorphic to implication holds in the embedding space: (r i ⇒ r j) iff (r i 's rel. space ⊂ r j 's rel. space) or equivalently, The orderings are isomorphic, because for example, if is_parent_of subsumes is_father_of, the first relation space also subsumes the latter's (Figure 3). At first sight, it may look paradoxical that the H j's and the relation spaces are inversely ordered; however, it is a natural consequence of the rank-based geometry in R d. In this section, we formally state TransINT's isomorphic guarantee and its grounds. We also discuss the intuitive meaning of our method. We denote all d × d matrices with capital letters (ex) A) and vectors with arrows on top (ex) #» b ). In R d, points are projected to linear subspaces by projection matrices; each linear subspace H i has a projection matrix Strang). For example, in Figure 4, a random point #» a ∈ R d is projected onto H 1 when multiplied by P 1; i.e. P 1 a = #» b ∈ H 1. In the rest of the paper, denote P i as the projection matrix onto subspace H i. Now, we algebraically define a general concept that subsumes relation space(Figure 3b). Let H be a linear subspace and P its projection matrix. Then, given #» k on H, the set of vectors that become #» k when projected on to H, or the solution space of With this definition, relation space (Figure 3b) is (Sol(P i, #» r i)), where P i is the projection matrix of H i (subspace for relation r i); it is the set of points Main Theorem 1 (Isomorphism): Let {(H i, #» r i)} n be the (subspace, vector) embeddings assigned to relations {R i} n by the Intersection Constraint and the Projection Constraint; P i the projection matrix of In actual implementation and training, TransINT requires something less strict than for some non-negative and small. This bounds # » t − h − #» r i to regions with thickness 2, centered around Sol(P i, #» r i) (Figure 5). We prove that isomorphism still holds with this weaker requirement. Main Theorem 2 (Margin-aware Isomorphism): For all non-negative scalar, ({Sol (P i, #» r i)} n, ⊂) is isomorphic to ({R i} n, ⊂). At a first glance, it may look paradoxical that a relation whose H i is the intersection of other relations' H j's (i.e. is_parent_of of Figure 3a) actually subsumes all the relations that were intersected (i.e. is_fahter_of, is_mohter_of). This inverse ordering of H j's and Sol(P j, #» r j) arise from the fact that the two are orthocomplements (Strang). Geometrically, projection is decomposition into independent directions; #» x = P #» x + (I − P) #» x holds for all #» x. In Fig. 4a, one can see that P and I − P are orthogonal. Algebraically, a vector #» x ∈ R d bound by P #» x = b, composed of k independent constraints (rank k), #» x is free in all other d − k directions of I − P (Fig. 4b). Thus, the lesser constraint the space to be projected onto, the more freedom a vector is given; which is isomorphic to that, for example, is_f amily_of puts more freedom on who can be tied by it than is_f ather_of. (Fig. 1b). Thus, the intuitive meaning of the above proof is that we can map degree of freedom in the logical space to that in R d. The intersection and projection constraints can be imposed with parameter sharing. We describe how shared parameters are initialized and trained. From initialization, we bind parameters so that they satisfy the two constraints. For each entity e j, we assign a d-dimensional vector #» e j. To each R i, we assign (H i, #» r i) (or (A i, #» r i)) with parameter sharing. We first construct the H's. Intersection constraint We define the H's top-down, first defining the intersections and then the subspaces that go through it. To the head R h, assign a h linearly independent rows for the basis of H h. Then, to each R i that is not a head, additionally assign a i rows linearly independent to the bases of all of its parents, and construct H i with its bases and the bases of all of its parents. Projection matrices can be uniquely constructed given the bases (Strang). Now, we initlialize the #» r i's. Projection Constraint To the head R h, pick any random x h ∈ R d and assign #» r h = P h x. To each non-head R i whose parent is R p, assign Parameters to be trained Such initialization leaves the following parameters given a KG with entities e j's and relations r i's: A h for the head relation, c i for each non-head relation, #» x i for each head and non-head relation, #» e j for each entity e j. 4.1.1 TRAINING We construct negative examples (wrong fact triplets) and train with a margin-based loss, following the same protocols as in TransE and TransH. Training Objective We adopt the same loss function as in TransH. For each fact triplet (h, r i, t), we define the score function f (h, r i, t) = ||P i (# » t − h) − #» r i || 2 and train a margin-based loss L which is aggregates f's and discriminates between correct and negative examples. where G is the set of all triples in the KG and (h, r i, t) is a negative triple made from corrupting (h, r i, t). We minimize this objective with stochastic gradient descent. Automatic Grounding of Positive Triples The parameter sharing scheme guarantees two advantages during all steps of training. First, the intersection and projection constraint are met not only at initialization but always. Second, traversing through a particular (h, r i, t) also automatically executes training with (h, r p, t) for any r i ⇒ r p. For example, by traversing (Tom, is_father_of, Harry) in the KG, the model automatically also traverses (Tom, is_parent_of, Harry), (Tom, is_family_of, Harry), even if the two triples are missing in the KG. This is because P p P i = P p with the given initialization (section 4.1.1) and thus, = f (h, r i, t) In other words, training f (h, r i, t) towards less than automatically guarantees, or has the effect of training f (h, r p, t) towards less than. This enables the model to be automatically trained with what exists in the KG, eliminating the need to manually create missing triples that are true by implication rule. We evaluate TransINT on Freebase 122 (respectively created by Vendrov et al. and Guo et al.) against the current state-of-the-art method. The task is to predict the gold entity given a fact triple with missing head or tail -if (h, r, t) is a fact triple in the test set, predict h given (r, t) or predict t given (h, r). We follow TransE and KALE's protocol. For each test triple (h, r, t), we rank the similarity score (f (e, r, t) when h is replaced with e for every entity e in the KG, and identify the rank of the gold head entity h; we do the same for the tail entity t. Aggregated over all test triples, we report: (i) the mean reciprocal rank (MRR), (ii) the median of the ranks (MED), and (iii) the proportion of ranks no larger than n (HITS@N), which are the same metrics reported by KALE. A lower MED and higher MRR and Hits HITS@N are better. TransH and KALE adopt a "filtered" setting that addresses when entities that are correct, albeit not gold, are ranked before the gold entity. For example, if the gold entity is (Tom, is_parent_of, John) and we rank every entity e for being the head of (?, is_parent_of, John), it is possible that Sue, John's mother, gets ranked before Tom. To avoid this, the "filtered setting" ignore corrupted triplets that exist in the KG when counting the rank of the gold entity. (The setting without this is called the "raw setting"). We compare our performance with that of KALE and previous methods (TransE, TransH, TransR) that were compared against it, using the same dataset (FB122). FB122 is a subset of FB15K (Bordes et al.) accompanied by 47 implication and transitive rules; it consists of 122 Freebase relations on "people", "location", and "sports" topics. Since we use the same train/ test/ validation sets, we directly copy from Guo et al. for reporting on these baselines. 5.1.1 DETAILS OF TRAINING TransINT's hyperparameters are: learning rate (η), margin (γ), embedding dimension (d), and learning rate decay (α), applied every 10 epochs to the learning rate. We find optimal configurations among the following candidates: η ∈ {0.003, 0.005, 0.01}, γ ∈ {1, 2, 5, 10}, d ∈ {50, 100}, α, ∈ {1.0, 0.98, 0.95}. We create 100 mini-batches of the training set and train for maximum of 1000 epochs with early stopping based on the best median rank. Furthermore, we try training with and without normalizing each of entity vectors, relation vectors, and relation subspace bases after every batch of training. 5.1.2 EXPERIMENT SETTINGS Out of the 47 rules in FB122, 9 are transitive rules (such as person/nationality(x,y) ∧ country/official_language(y,z) ⇒ person/languages(x,z)) to be used for KALE. However, since TransINT only deals with implication rules, we do not take advantage of them, unlike KALE. We also put us on some intentional disadvantages against KALE to assess TransINT's robustness to absence of negative example grounding. In constructing negative examples for the margin-based loss L, KALE both uses rules (by grounding) and their own scoring scheme to avoid false negatives. While grounding with FB122 is not a burdensome task, it known to be very inefficient and difficult for extremely large datasets (Ding et al.). Thus, it is a great advantage for a KG model to perform well without grounding of training/ test data. We evaluate TransINT on two settings for avoiding false negative examples; using rule grounding and only avoiding ones that exist in the KG. We call them respectively TransINT G (grounding), TransINT N G (no grounding). Table 1. While the filtered setting gives better performance (as expected), the trend is generally similar between raw and filtered. TransINT outperforms all other models by large margins in all metrics, even without grounding; especially in the filtered setting, the Hits@N gap between TransINT G and KALE is around 4∼6 times that between KALE and the best performing Trans Baseline (TransR). Also, while TransINT G performs higher than TransINT N G in all settings/metrics, the gap between them is much smaller than the that between TransINT N G and KALE, showing that TransINT robustly brings state-of-the-art performance even without grounding. The suggest two possibilities in a more general sense. First, the emphasis of true positives could be as important as/ more important than avoiding false negatives. Even without manual grounding, TransINT N G has automatic grounding of positive training instances enabled (Section 4.1.1.) due to model properties, and this could be one of its success factors. Second, hard constraint on parameter structures can bring performance boost uncomparable to that by regularization or joint learning, which are softer constraints. We also note that norm regularization of any parameter did not help in training TransINT, unlike stated in TransE, TransH, and KALE. Instead, it was important to use a large margin (either γ = 5 or γ = 10). The task is to classify whether an unobserved instance (h, r, t) is correct or not, where the test set consists of positive and negative instances. We use the same protocol and test set provided by KALE; for each test instance, we evaluate its similarity score f (h, r, t) and classify it as "correct" if f (h, r, t) is below a certain threshold (σ), a hyperparameter to be additionally tuned for this task. We report on mean average precision (MAP), the mean of classification precision over all distinct relations (r's) of the test instances. We use the same experiment settings/ training details as in Link Prediction other than additionally finding optimal σ. 5.2.1 Triple Classification are shown in Table 2. Again, TransINT G and TransINT N G both significantly outperform all other baselines. We also separately analyze MAP for relations that are/ are not affected by the implication rules (those that appear/ do not appear in the rules), shown in parentheses of Table 2 with the order of (influenced relations/ uninfluenced relations). We can see that both TransINT's have MAP higher than the overall MAP of KALE, even when the TransINT's have the penalty of being evaluated only on uninfluenced relations; this shows that TransINT generates better embeddings even for those not affected by rules. Furthermore, we comment on the role of negative example grounding; we can see that grounding does not help performance on unaffected relations (i.e. 0.752 vs 0.761), but greatly boosts performance on those affected by rules (0.839 vs 0.709). While TransINT does not necessitate negative example grounding, it does improve the quality of embeddings for those affected by rules. Traditional embedding methods that map an object (i.e. words, images) to a singleton vector learn soft tendencies between embedded vectors, such as semantic similarity (Mikolov et al., Frome et al.). 83.5 n/a 7 A common metric for such tendency is cosine similarity, or angle between two embddings. TransINT extends such line of thought to semantic relatedness between groups of objects, with angles between relation spaces. In Fig. 5b, one can observe that the closer the angle between two embedded regions, the larger the overlap in area. For entities h and t to be tied by both relations r 1, r 2, t − h has to belong to the intersection of their relation spaces. Thus, we hypothesize the following over any two relations r 1, r 2 that are not explicitly tied by the pre-determined rules: Let V 1 be the set of # » t − h's in r 1's relation space (denoted as Rel 1) and V 2 that of r 2' s. Then, Angle between Rel 1 and Rel 2 represents semantic "disjointness" of r 1, r 2; the more disjoint two relations, the closer their angle to 90 •. When the angle between Rel 1 and Rel 2 is small, if majority of V 1 belongs to the overlap of V 1 and V 2 but not vice versa, r 1 implies r 2. if majority of V 1 and V 2 both belong to their overlap, r 1 and r 2 are semantically related. Hypotheses and consider the imbalance of membership in overlapped regions. Exact calculation of this involves specifying an appropriate (Fig. 3). As a proxy for deciding whether an element of V 1 (denote v 1) belongs in the overlapped region, we can consider the distance between v 1 to and its projection to Rel 2; the further away v 1 is from the overlapped region, the larger the projected distance (visualization available in our code repository). We call the mean of such distances from V 1 to For hypothesis, we verified that the vast majority of relation pairs have angles near to 90 •, with the mean and median respectively 83.0 • and 85.4 •; only 1% of all relation pairs had angles less than 50 •. We observed that relation pairs with angle less than 20 • were those that can be inferred by transitively applying the pre-determined implication rules. Relation pairs with angles within the range of [20 •] had strong tendencies of semantic relatedness or implication; such tendency drastically weakened past 70 •. Table 3 shows the angle and imb of relations with respect to /people/person/place_of_birth, whose trend agrees with our hypotheses. While we only show a subset of the complete list, we note that almost all relation pairs generally follow such a tendency; the complete list can be accessed in our code repository. Finally, we note that such analysis could be possible with TransH as well, since their method too maps # » t − h's to lines (Fig. 2b). Throughout target tasks (Link Prediction, Triple Classification) and semantics mining, TransINT's theme of optimal regions to bound entity sets is unified and consistent. Furthermore, the integration of rules into embedding space geometrically coherent with KG embeddings alone. These two qualities were missing in existing works such as TransE or KALE, and TransINT opens up new possibilities for applying KG embeddings to explainable reasoning in applications such as recommender systems (Ma et al.) and question answering (Hamilton et al.). Our work is related to three strands of work. The first strand is Order Embeddings (Vendrov et al.) and their extensions (Vilnis et al.; Athiwaratkun & Wilson), whose limitation we discussed in the introduction. While Nickel & Kiela also approximately embed unary partial ordering, their focus is on achieving reasonably competent with unsupervised learning of rules in low dimensions, while ours is achieving state-of-the-art in a supervised setting. The second strand is those that enforce the satisfaction of common sense logical rules in the embedded KG. Wang et al. explicitly constraints the ing embedding to satisfy logical implications and type constraints via linear programming, but it only requires to do so during inference, not learning. On the other hand,Guo et al. induces that embeddings follow a set of logical rules during learning, but their approach is soft induction not hardly constrain. Our work combines the advantages of both works. We presented TransINT, a new KG embedding method that embed sets of entities (tied by relations) to continuous sets in R d that are inclusion-ordered isomorphically to relation implications. Our method achieved new state-of-the-art performances with signficant margins in Link Prediction and Triple Classification on the FB122 dataset, with boosted performance even on test instances that are not affected by rules. We further propose and interpretable criterion for mining semantic similairty among sets of entities with TransINT. Here, we provide the proofs for Main Theorems 1 and 2. We also explain some concepts necessary in explaining the proofs. We put * next to definitions and theorems we propose/ introduce. Otherwise, we use existing definitions and cite them. We explain in detail elements of R d that were intuitively discussed. In this and later sections, we mark all lemmas and definitions that we newly introduce with *; those not marked with * are accompanied by reference for proof. We denote all d × d matrices with capital letters (ex) A) and vectors with arrows on top (ex) #» b ). The linear subspace given by that are solutions to the equation; its rank is the number of constraints A(x − #» b) = 0 imposes. For example, in R 3, a hyperplane is a set of #» x = [x 1, x 2, x 3] ∈ R 3 such that ax 1 + bx 2 + cx 3 − d = 0 for some scalars a, b, c, d; because vectors are bound by one equation (or its "A" only really contains one effective equation), a hyperplane's rank is 1 (equivalently rank(A) = 1). On the other hand, a line in R 3 imposes to 2 constraints, and its rank is 2 (equivalently rank(A) = 2). Consider two linear subspaces H 1, H 2, each given by A 1 (#» x − #» b 1) = 0, A 2 (#» x − #» b 2) = 0. Then, by definition. In the rest of the paper, denote H i as the linear subspace given by some A i (#» x − #» b i) = 0. Invariance For all #» x on H, projecting #» x onto H is still #» x; the converse is also true. Lemma 1 P #» x = #» x ⇔ #» x ∈ H (Strang). Orthogonality Projection decomposes any vector #» x to two orthogonal components -P #» x and (I − P) #» x (Figure 4). Thus, for any projection matrix P, I − P is also a projection matrix that is orthogonal to P (i.e. P (I − P) = 0) (Strang). Lemma 2 Let P be a projection matrix. Then I −P is also a projection matrix such that P (I −P) = 0 (Strang). The following lemma also follows. Lemma 3 ||P #» x || ≤ ||P #» x + (I − P) #» x || = || #» x || (Strang). Projection onto an included space If one subspace H 1 includes H 2, the order of projecting a point onto them does not matter. For example, in Figure 3, a random point #» a in R 3 can be first projected onto H 1 at #» b, and then onto H 3 at #» d. On the other hand, it can be first projected onto H 3 at #» d, and then onto H 1 at still #» d. Thus, the order of applying projections onto spaces that includes one another does not matter. If we generalize, we obtain the following two lemmas (Figure 6): Lemma 4 * Every two subspaces H 1 ⊂ H 2 if and only if P 1 P 2 = P 2 P 1 = P 1. proof) By Lemma 1, if H 1 ⊂ H 2, then P 2 #» x = #» x ∀ #» x ∈ H 1. On the other hand, if H 1 ⊂ H 2, then there is some #» x ∈ H 1, #» x ∈ H 2 such that P 2 #» x = #» x. Thus, ⇔ ∀ #» y, P 2 (P 1 #» y) = P 1 #» y ⇔ P 2 P 1 = P 1. Because projection matrices are symmetric (Strang), P 2 P 1 = P 1 = P 1 T = P 1 T P 2 T = P 1 P 2. Lemma 5 * For two subspaces H 1, H 2 and vector #» k ∈ H 2,

Represent the short summary for finding the corresponding scientific article (long).

We propose TransINT, a novel and interpretable KG embedding method that isomorphically preserves the implication ordering among relations in the embedding space in an explainable, robust, and geometrically coherent way.

Represent this text to find a much much longer article about it with say hundreds-thousands of words


Question: represent the article for finding a single-sentence tldr

Question: Unsupervised domain adaptive object detection aims to learn a robust detector on the domain shift circumstance, where the training (source) domain is label-rich with bounding box annotations, while the testing (target) domain is label-agnostic and the feature distributions between training and testing domains are dissimilar or even totally different. In this paper, we propose a gradient detach based Stacked Complementary Losses (SCL) method that uses detection objective (cross entropy and smooth l1 regression) as the primary objective, and cuts in several auxiliary losses in different network stages to utilize information from the complement data (target images) that can be effective in adapting model parameters to both source and target domains. A gradient detach operation is applied between detection and context sub-networks during training to force networks to learn discriminative representations. We argue that the conventional training with primary objective mainly leverages the information from the source-domain for maximizing likelihood and ignores the complement data in shallow layers of networks, which leads to an insufficient integration within different domains. Thus, our proposed method is a more syncretic adaptation learning process. We conduct comprehensive experiments on seven datasets, the demonstrate that our method performs favorably better than the state-of-the-art methods by a large margin. For instance, from Cityscapes to FoggyCityscapes, we achieve 37.9% mAP, outperforming the previous art Strong-Weak by 3.6%. In real world scenarios, generic object detection always faces severe challenges from variations in viewpoint, , object appearance, illumination, occlusion conditions, scene change, etc. These unavoidable factors make object detection in domain-shift circumstance becoming a challenging and new rising research topic in the recent years. Also, domain change is a widely-recognized, intractable problem that urgently needs to break through in reality of detection tasks, like video surveillance, autonomous driving, etc. (see Figure 2). Revisiting Domain-Shift Object Detection. Common approaches for tackling domain-shift object detection are mainly in two directions: (i) training supervised model then fine-tuning on the target domain; or (ii) unsupervised cross-domain representation learning. The former requires additional instance-level annotations on target data, which is fairly laborious, expensive and time-consuming. So most approaches focus on the latter one but still have some challenges. The first challenge is that the representations of source and target domain data should be embedded into a common space for matching the object, such as the hidden feature space , input space ) or both of them (b). The second is that a feature alignment/matching operation or mechanism for source/target domains should be further defined, such as subspace alignment , H-divergence and adversarial learning , MRL (b), Strong-Weak alignment , etc. In general, our SCL is also a learning-based alignment method across domains with an end-to-end framework. (a) Non-adapted (b) CVPR'18 (c) CVPR'19 (d) SCL (Ours) (e) Non-adapted (f) CVPR'18 (g) CVPR'19 (h) SCL (Ours) Figure 1: Visualization of features from PASCAL to Clipart (first row) and from Cityscapes to FoggyCityscapes (second row) by t-SNE . Red indicates the source examples and blue is the target one. If source and target features locate in the same position, it is shown as light blue. All models are re-trained with a unified setting to ensure fair comparisons. It can be observed that our feature embedding are consistently much better than previous approaches on either dissimilar domains (PASCAL and Clipart) or similar domains (Cityscapes and FoggyCityscapes). Our Key Ideas. The goal of this paper is to introduce a simple design that is specific to convolutional neural network optimization and improves its training on tasks that adapt on discrepant domains. Unsupervised domain adaptation for recognition has been widely studied by a large body of previous literature (; ; ; ; ; ; ;), our method more or less draws merits from them, like aligning source and target distributions with adversarial learning (domain-invariant alignment). However, object detection is a technically different problem from classification, since we would like to focus more on the object of interests (local regions). Figure 2: Illustration of domain-shift object detection in autonomous driving scenario. Images are from INIT dataset . Some recent work has proposed to conduct alignment only on local regions so that to improve the efficiency of model learning. While this operation may cause a deficiency of critical information from context. Inspired by multi-feature/strong-weak alignment (; ;) which proposed to align corresponding local-region on shallow layers with small respective field (RF) and align imagelevel features on deep layers with large RF, we extend this idea by studying the stacked complementary objectives and their potential combinations for domain adaptive circumstance. We observe that domain adaptive object detection is supported dramatically by the deep supervision, however, the diverse supervisions should be applied in a controlled manner, including the cut-in locations, loss types, orders, updating strategy, etc., which is one of the contributions of this paper. Furthermore, our experiments show that even with the existing objectives, after elaborating the different combinations and training strategy, our method can obtain competitive . By pluging-in a new sub-network that learns the context features independently with gradient detach updating strategy in a hierarchical manner, we obtain the best on several domain adaptive object detection benchmarks. The Relation to Complement Objective Training and Deep Supervision . COL proposed to involve additional function that complements the primary objective, and updated the parameters alternately with primary and complement objectives. Specifically, cross entropy is used as the primary objective H p: where y i ∈ {0, 1} D is the label of the i-th sample in one-hot representation andŷ i ∈ D is the predicted probabilities. Th complement entropy H c is defined in COT as the average of sample-wise entropies over complement classes in a mini-batch: where H is the entropy function.ŷ c is the predicted probabilities of complement classes c. The training process is that: for each iteration of training, 1) update parameters by H p first; then 2) update parameters by H c. In contrast, we don't use the alternate strategy but update the parameters simultaneously using gradient detach strategy with primary and complement objectives. Since we aim to let the network enable to adapt on both source and target domain data and meanwhile enabling to distinguish objects from them, thus our complement objective design is quite different from COT. We will describe with details in Section 2. In essence, our method is more likely to be the deeply supervised formulation that backpropagation of error now proceeds not only from the final layer but also simultaneously from our intermediate complementary outputs. While DSN is basically proposed to alleviate "vanishing" gradient problem, here we focus on how to adopt these auxiliary losses to promote to mix two different domains through domain classifiers for detection. Interestingly, we observe that diverse objectives can lead to better generalization for network adaptation. Motivated by this, we propose Stacked Complementary Losses (SCL), a simple yet effective approach for domain-shift object detection. Our SCL is fairly easy and straight-forward to implement, but can achieve remarkable performance. We conjecture that previous approaches that focus on conducting domain alignment on high-level layers only cannot fully adapt shallow layer parameters to both source and target domains (even local alignment is applied ) which restricts the ability of model learning. Also, gradient detach is a critical part of learning with our complementary losses. We further visualize the features obtained by non-adapted model, DA , Strong-Weak and ours, features are from the last layer of backbone before feeding into the Region Proposal Network (RPN). As shown in Figure 1, it is obvious that the target features obtained by our model are more compactly matched with the source domain than any other models. Contributions. Our contributions in this paper are three-fold. • We propose an end-to-end learnable framework that adopts complementary losses for domain adaptive object detection. We study the deep supervisions in this task with a controlled manner. Our method allows information from source and target domains to be integrated seamlessly. • We propose a gradient detach learning strategy to enable complementary losses to learn a better representation and boost the performance. We also provide extensive ablation studies to empirically verify the effectiveness of each component in our framework design. • To the best of our knowledge, this is a pioneer work to investigate the influence of diverse loss functions and gradient detach for domain adaptive object detection. Thus, this work gives very good intuition and practical guidance with multi-objective learning for domain adaptive object detection. More remarkably, our method achieves the highest accuracy on several domain adaptive or cross-domain object detection benchmarks, which are new records on this task. Following the common formulation of domain adaptive object detection, we define a source domain S where annotated bound-box is available, and a target domain T where only the image can be used in training process without any labels. Our purpose is to train a robust detector that can adapt well to both source and target domain data, i.e., we aim to learn a domain-invariant feature representation that works well for detection across two different domains. As shown in Figure 3, we focus on the complement objective learning and let S = {(x i is the corresponding bounding box and category labels for sample x i}. We define a recursive function for layers k = 1, 2,..., K where we cut in complementary losses: whereΘ k is the feature map produced at layer k, F is the function to generate features at layer k and Z k is input at layer k. We formulate the complement loss of domain classifier k as follows: where k denote feature maps from source and target domains respectively. Following , we also adopt gradient reverse layer (GRL) to enable adversarial training where a GRL layer is placed between the domain classifier and the detection backbone network. During backpropagation, GRL will reverse the gradient that passes through from domain classifier to detection network. For our instance-context alignment loss L ILoss, we take the instance-level representation and context vector as inputs. The instance-level vectors are from RoI layer that each vector focuses on the representation of local object only. The context vector is from our proposed sub-network that combine hierarchical global features. We concatenate instance features with same context vector. Since context information is fairly different from objects, joint training detection and context networks will mix the critical information from each part, here we proposed a better solution that uses detach strategy to update the gradients. We will introduce it with details in the next section. Aligning instance and context representation simultaneously can help to alleviate the variances of object appearance, part deformation, object size, etc. in instance vector and illumination, scene, etc. in context vector. We define d i as the domain label of i-th training image where d i = 1 for the source and d i = 0 for the target, so the instance-context alignment loss can be further formulated as: where N s and N t denote the numbers of source and target examples. P (i,j) is the output probabilities of the instance-context domain classifier for the j-th region proposal in the i-th image. So our total SCL objective L SCL can be written as: In this section, we introduce a simple detach strategy which prevents the flow of gradients from context sub-network through the detection backbone path. We find this can help to obtain more discriminative context and we show empirical evidence (see Figure 6) that this path carries information with diversity and hence gradients from this path getting suppressed is superior for such task. As aforementioned, we define a sub-network to generate the context information from early layers of detection backbone. Intuitively, instance and context will focus on perceptually different parts of an image, so the representations from either of them should also be discrepant. However, if we train with the conventional process, the companion sub-network will be updated jointly with the detection backbone, which may lead to an indistinguishable behavior from these two parts. To this end, in this paper we propose to suppress gradients during backpropagation and force the representation of context sub-network to be dissimilar to the detection network, as shown in Algorithm 1. To our best knowledge, this may be the first work to show the effectiveness of gradient detach that can help to learn better context representation for domain adaptive object detection. Although the detach-based method has been adopted in a few work for better optimization on sequential tasks, our design and motivation are quite different from it. The details of our context sub-network architecture are illustrated in Appendix A. 3. Update detection net by detection and complementary objectives: L det +L SCL Our framework is based on the Faster RCNN , including the Region Proposal Network (RPN) and other modules. The objective of the detection loss is summarized as: where L cls is the classification loss and L reg is the bounding-box regression loss. To train the whole model using SGD, the overall objective function in the model is: where λ is the trade-off coefficient between detection loss and our complementary loss. R denotes the RPN and other modules in Faster RCNN. Following , we feed one labeled source image and one unlabeled target one in each mini-batch during training. Datasets. We evaluate our approach in three different domain shift scenarios: Similar Domains; Discrepant Domains; and From Synthetic to Real Images. All experiments are conducted on seven domain shift datasets: Cityscapes to FoggyCityscapes, Cityscapes to KITTI , KITTI to Cityscapes, INIT Dataset , PASCAL to Clipart , PASCAL to Watercolor , GTA (Sim 10K) to Cityscapes. Implementation Details. In all experiments, we resize the shorter side of the image to 600 following with ROI-align . We train the model with SGD optimizer and the initial learning rate is set to 10 −3, then divided by 10 after every 50,000 iterations. Unless otherwise stated, we set λ as 1.0 and γ as 5.0, and we use K = 3 in our experiments (the analysis of hyper-parameter K is shown in Table 7). We report mean average precision (mAP) with an IoU threshold of 0.5 for evaluation. Since there are few pioneer works for exploring the combination of different losses for domain adaptive object detection, here we conduct extensive ablation study for this part to find the best collocation of our SCL method. We follow some objective design from DA and Weak-Strong which provides guidance for us to utilize these losses. Cross-entropy (CE) Loss. CE loss measures the performance of a classification model whose output is a probability value. It increases as the predicted probability diverges from the actual label: where p c ∈ is the predicted probability observation of c class. y c is the c class label. Least-squares (LS) Loss. Following , we adopt LS loss to stabilize the training of the domain classifier for aligning low-level features. The loss is designed to align each receptive field of features with the other domain. The least-squares loss is formulated as: where D Θ (s) wh denotes the output of the domain classifier in each location (w, h). Focal Loss (FL). Focal loss L FL is adopted to ignore easy-to-classify examples and focus on those hard-to-classify ones during training: -adapted)) 30.2 53.5 DA 38.5 64.1 DA (Our impl.) 35.6 70.8 SW (Our impl.) 37.9 71.0 Ours 41.9 72.7 The are summarized in Table 1. We present several combinations of four complementary objectives with their loss names and performance. We observe that "LS-CE-F L-F L" obtains the best accuracy with Context and Detach. It indicates that LS can only be placed on the low-level features (rich spatial information and poor semantic information) and F L should be in the high-level locations (weak spatial information and strong semantic information). For the middle location, CE will be a good choice. If you use LS for the middle/high-level features or use F L on the low-level features, it will confuse the network to learn hierarchical semantic outputs, so that ILoss+detach will lose effectiveness under that circumstance. This verifies that domain adaptive object detection is supported by deep supervision, however, the diverse supervisions should be applied in a controlled manner. Furthermore, our proposed method performed much better than baseline Strong-Weak (37.9% vs.34.3%) and other state-of-the-arts. Between Cityspaces and KITTI. In this part, we focus on studying adaptation between two real and similar domains, as we take KITTI and Cityscapes as our training and testing data. Following , we use KITTI training set which contains 7,481 images. We conduct experiments on both adaptation directions K → C and C → K and evaluate our method using AP of car as in DA. As shown in Table 2, our proposed method performed much better than the baseline and other stateof-the-art methods. Since Strong-Weak didn't provide the on this dataset, we re-implement it and obtain 37.9% AP on K→C and 71.0% AP on C→K. Our method is 4% higher than the former and 1.7% higher than latter. If comparing to the non-adapted (source only), our method outperforms it with a huge margin (about 10% and 20% higher, respectively). INIT Dataset. INIT Dataset contains 132,201 images for training and 23,328 images for testing. There are four domains: sunny, night, rainy and cloudy, and three instance categories, including: car, person, speed limited sign. This dataset is first proposed for the instance-level image-to-image translation task, here we use it for the domain adaptive object detection purpose. Our are shown in Table 3. Following , we conduct experiments on three domain pairs: sunny→night (s2n), sunny→rainy (s2r) and sunny→cloudy (s2c). Since the training images in rainy domain are much fewer than sunny, for s2r experiment we randomly sample the training data in sunny set with the same number of rainy set and then train the detector. It can be observed that our method is consistently better than the baseline method. We don't provide the of s2c (faster) because we found that cloudy images are too similar to sunny in this dataset (nearly the same), thus the non-adapted is very close to the adapted methods. In this section, we focus on the dissimilar domains, i.e., adaptation from real images to cartoon/artistic. Following , we use PASCAL VOC dataset (2007+2012 training and validation combination for training) as the source data and the Clipart or Watercolor as the target data. The backbone network is ImageNet pre-trained ResNet-101. PASCAL to Clipart. Clipart dataset contains 1,000 images in total, with the same 20 categories as in PASCAL VOC. As shown in Table 4, our proposed SCL outperforms all baselines. In addition, we observe that replacing F L with CE loss on instance-context classifier can further improve the performance from 40.6% to 41.5%. More ablation are shown in our Appendix B.2 (Table 10). PASCAL to WaterColor. Watercolor dataset contains 6 categories in common with PASCAL VOC and has totally 2,000 images (1,000 images are used for training and 1,000 test images for evaluation). Results are summarized in Table 5, our SCL consistently outperforms other state-of-the-arts. Sim10K to Cityscapes. Sim 10k dataset contains 10,000 images for training which are generated by the gaming engine Grand Theft Auto (GTA). Following , we use Cityscapes as target domain and evaluate our models on Car class. Our is shown in Table 6, which consistently outperforms the baselines. Hyper-parameter K. Table 7 shows the for sensitivity of hyper-parameter K in Figure 3. This parameter controls the number of SCL losses and context branches. It can be observed that the proposed method performs best when K = 3 on all three datasets. Parameter Sensitivity on λ and γ. Figure 4 shows the for parameter sensitivity of λ and γ in Eq. 8 and Eq. 11. λ is the trade-off parameter between SCL and detection objectives and γ controls the strength of hard samples in Focal Loss. We conduct experiments on two adaptations: Cityscapes → FoggyCityscapes (blue) and Sim10K → Cityscapes (red). On Cityscapes → FoggyCityscapes, we achieve the best performance when λ = 1.0 and γ = 5.0 and the best accuracy is 37.9%. On Sim10K → Cityscapes, the best is obtained when λ = 0.1, γ = 2.0. indicate values from high to low. It can be observed that w/ detach training, our models can learn more discriminative representation between object areas and (context). Analysis of IoU Threshold. The IoU threshold is an important indicator to reflect the quality of detection, and a higher threshold means better coverage with ground-truth. In our previous experiments, we use 0.5 as a threshold suggested by many literature . In order to explore the influence of IoU threshold with performance, we plot the performance vs. IoU on three datasets. As shown in Figure 5, our method is consistently better than the baselines on different threshold by a large margin (in most cases). Why Gradient Detach Can Help Our Model? To further explore why gradient detach can help to improve performance vastly and what our model really learned, we visualize the heatmaps on both source and target images from our models w/o and w/ detach training. As shown in Figure 6, the visualization is plotted with feature maps after Conv B3 in Figure 3. We can observe that the object areas and context from detach-trained models have stronger contrast than w/o detach model (red and blue areas). This indicates that detach-based model can learn more discriminative features from the target object and context. More visualizations are shown in Appendix C (Figure 8). Detection Visualization. Figure 10 shows several qualitative comparisons of detection examples on three test sets with DA , Strong-Weak and our SCL models. Our method detects more small and blurry objects in dense scene (FoggyCityscapes) and suppresses more false positives (Clipart and Watercolor) than the other two baselines. , Strong-Weak and our proposed SCL on three datasets. For each group, the first row is the of DA, the second row is from Strong-Weak and the last row is ours. We show detections with the scores higher than a threshold (0.3 for FoggyCityscapes and 0.5 for other two). In this paper, we have addressed unsupervised domain adaptive object detection through stacked complementary losses. One of our key contributions is gradient detach training, enabled by suppressing gradients flowing back to the detection backbone. In addition, we proposed to use multiple complementary losses for better optimization. We conduct extensive experiments and ablation studies to verify the effectiveness of each component that we proposed. Our experimental outperform the state-of-the-art approaches by a large margin on a variety of benchmarks. Our future work will focus on exploring the domain-shift detection from scratch, i.e., without the pre-trained models like DSOD , to avoid involving bias from the pre-trained dataset. A CONTEXT NETWORK Our context networks are shown in Table 8. We use three branches (forward networks) to deliver the context information and each branch generates a 128-dimension feature vector from the corresponding backbone layers of SCL. Then we naively concatenate them and obtain the final context feature with a 384-dimension vector. In this section, we show the adaptation on source domains in Table 11, 12, 13 and 14. Surprisingly, we observe that the best-trained models (on target domains) are not performing best on the source data, e.g., from PASCAL VOC to WaterColor, DA obtained the highest on source domain images (although the gaps with Strong-Weak and ours are marginal). We conjecture that the adaptation process for target domains will affect the learning and performing on source domains, even we have used the bounding box ground-truth on source data for training. We will investigate it more thoroughly in our future work and we think the community may also need to rethink whether evaluating on source domain should be a metric for domain adaptive object detection, since it can help to understand the behavior of models on both source and target images. We provide the detailed of parameter sensitivity on λ and γ in Table 15 and 16 with the adaptation of from Cityscapes to FoggyCityscapes and from Sim10K to Cityscapes. Figure 9, the gradient detach-based models can adapt source and target images to a similar distribution better than w/o detach models.

Represent the TLDR for finding the corresponding scientific article

We introduce a new gradient detach based complementary objective training strategy for domain adaptive object detection.

Represent the short summary for finding the corresponding scientific article (long)

Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Convolutional neural networks (CNN) have become the most successful and popular approach in many vision-related domains. While CNNs are particularly well-suited for capturing a proper hierarchy of concepts from real-world images, they are limited to domains where data is abundant. Recent attempts have looked into mitigating this data scarcity problem by casting their original single-task problem into a new multi-task learning (MTL) problem. The main goal of this inductive transfer mechanism is to leverage domain-specific information from related tasks, in order to improve generalization on the main task. While recent in the deep learning (DL) community have shown the promising potential of training task-specific CNNs in a soft parameter sharing framework, integrating the recent DL advances for improving knowledge sharing is still an open problem. In this paper, we propose the Deep Collaboration Network (DCNet), a novel approach for connecting task-specific CNNs in a MTL framework. We define connectivity in terms of two distinct non-linear transformation blocks. One aggregates task-specific features into global features, while the other merges back the global features with each task-specific network. Based on the observation that task relevance depends on depth, our transformation blocks use skip connections as suggested by residual network approaches, to more easily deactivate unrelated task-dependent features. To validate our approach, we employed facial landmark detection (FLD) datasets as they are readily amenable to MTL, given the number of tasks they include. Experimental show that we can achieve up to 24.31% relative improvement in landmark failure rate over other state-of-the-art MTL approaches. We finally perform an ablation study showing that our approach effectively allows knowledge sharing, by leveraging domain-specific features at particular depths from tasks that we know are related. Over the past few years, convolutional neural networks (CNNs) have become the leading approach in many vision-related tasks BID12. By creating a hierarchy of increasingly abstract concepts, they can transform complex high-dimensional input images into simple low-dimensional output features. Although CNNs are particularly well-suited for capturing a proper hierarchy of concepts from real-world images, successively training them requires large amount of data. Optimizing deep networks is tricky, not only because of problems like vanishing / exploding gradients BID8 or internal covariate shift BID9, but also because they typically have many parameters to be learned (which can go up to 137 billions BID21). While previous works have looked at networks pre-trained on a large image-based dataset as a starting point for their gradient descent optimization, others have considered improving generalization by casting their original single-task problem into a new multi-task learning (MTL) problem (see BID31 for a review). As BID2 explained in his seminal work: "MTL improves generalization by leveraging the domain-specific information contained in the training signals of related tasks". Exploring new ways to efficiently gather more information from related tasks -the core contribution of our approach -can thus help a network to further improve upon its main task. The use of MTL goes back several years, but has recently proven its value in several domains. As a consequence, it has become a dominant field of machine learning BID30. Although many early and influential works contributed to this field BID5 ), recent major advances in neural networks opened up opportunities for novel contributions in MTL. Works on grasping BID17, pedestrian detection BID24, natural language processing BID14, face recognition BID26 BID27 and object detection BID16 have all shown that MTL has been finally adopted by the deep learning (DL) community as a way to mitigate the lack of data, and is thus growing in popularity. MTL strategies can be divided into two major categories: hard and soft parameter sharing. Hard parameter sharing is the earliest and most common strategy for performing MTL, which dates back to the original work of BID2. Approaches in this category generally share the hidden layers between all tasks, while keeping separate outputs. Recent in the DL community have shown that a central CNN with separate task-specific fully connected (FC) layers can successfully leverage domain-specific information BID18 BID17 BID27. Although hard parameter sharing reduces the risk of over-fitting BID1, shared layers are prone to be overwhelmed by features or contaminated by noise coming from particular noxious related tasks.Soft parameter sharing has been proposed as an alternative to alleviate this drawback, and has been growing in popularity as a potential successor. Approaches in this category separate all hidden layers into task-specific models, while providing a knowledge sharing mechanism. Each model can then learn task-specific features without interfering with others, while still sharing their knowledge. Recent works using one network per task have looked at regularizing the distance between taskspecific parameters with a 2 norm BID4 or a trace norm BID25, training shared and private LSTM submodules, partitioning the hidden layers into subspaces BID19 and regularizing the FC layers with tensor normal priors BID15. In the domain of continual learning, progressive network BID20 has also shown promising for cross-domain sequential transfer learning, by employing lateral connections to previously learned networks. Although all these soft parameter approaches have shown promising potential, improving the knowledge sharing mechanism is still an open problem. In this paper, we thus present the deep collaboration network (DCNet), a novel approach for connecting task-specific networks in a soft parameter sharing MTL framework. We contribute with a novel knowledge sharing mechanism, dubbed the collaborative block, which implements connectivity in terms of two distinct non-linear transformations. One aggregates task-specific features into global features, and the other merges back the global features into each task-specific network. We demonstrate that our collaborative block can be dropped in any existing architectures as a whole, and can easily enable MTL for any approaches. We evaluated our method on the problem of facial landmark detection in a MTL framework and obtained better in comparison to other approaches of the literature. We further assess the objectivity of our training framework by randomly varying the contribution of each related tasks, and finally give insights on how our collaborative block enables knowledge sharing with an ablation study on our DCNet. The content of our paper is organized as follows. We first describe in Section 2 works on MTL closely related to our approach. We also describe Facial landmark detection, our targeted application. Architectural details of our proposed Multi-Task approach and its motivation are spelled out in Section 3. We then present in Section 4 a number of comparative on this Facial landmark detection problem for two CNN architectures, AlexNet and ResNet18, that have been adapted with various MTL frameworks including ours. It also contains discussions on an ablation study showing at which depth feature maps from other tasks are borrowed to improve the main task. We conclude our paper in Section 5.2 RELATED WORK 2.1 MULTI-TASK LEARNING Our proposed deep collaboration network (DCNet) is related to other existing approaches. The first one is the cross-stitch (CS) BID16 ) network, which connects task-specific networks through linear combinations of the spatial feature maps at specific layers. One drawback of CS is that they are limited to capturing linear dependencies only, something we address in our proposed approach by employing non-linearities when sharing feature maps. Indeed, non-linear combinations are usually able to learn richer relationships, as demonstrated in deep networks. Another related approach is tasks-constrained deep convolutional network (TCDCN) for facial landmarks detection. In it, the authors proposed an early-stopping criterion for removing auxiliary tasks before the network starts to over-fit to the detriment of the main task. One drawback of their approach is that their criterion has several hyper-parameters, which must all be selected manually. For instance, they define an hyper-parameter controlling the period length of the local window and a threshold that stops the task when the criterion exceeds it, all of which can be specified for each task independently. Unlike TCDCN, our approach has no hyper-parameters that depend on the tasks at hand, which greatly simplifies the training process. Our two transformation blocks consist of a series of batch normalization, ReLU, and convolutional layers shaped in a standard setting based on recent advances in residual network (see Sec. 3). This is particularly useful for computationally expensive deep networks, since integrating our proposed approach requires no additional hyper-parameter tuning experiments. Our proposed approach is also related to HyperFace BID18. In this work, the authors proposed to fuse the intermediate layers of AlexNet and exploit the hierarchical nature of the features. Their goal was to allow low-level features containing better localization properties to help tasks such as landmark localization and pose detection, and allow more class-specific high-level features to help tasks like face detection and gender recognition. Although HyperFace uses a single shared CNN instead of task-specific CNNs and is not entirely related to our approach, the idea of feature fusion is also central in our work. Instead of fusing the features at intermediate layers of a single CNN, our approach aggregates same-level features of multiple CNNs, at different depth independently. Also, one drawback of HyperFace is that the proposed feature fusion is specific to AlexNet, while our method is not specific to any network. In fact, our approach takes into account the vast diversity of existing network architectures, since it can be added to any architecture without modification. Facial landmark detection (FLD) is an essential component in many face-related tasks BID22 BID10 BID0. This problem can be described as follows: given the image of an individual's face, the goal is to predict the (x, y)-position on the image of specific landmarks associated with key features of the visage. Applications such as face recognition BID3, face validation BID23, facial feature detection and tacking rely on the ability to correctly find the location of these distinct facial landmarks in order to succeed. Localizing facial key points like the center of the eyes, the corners of the mouth, the tip of the nose and the earlobes is however a challenging problem when many lighting conditions, head poses, facial expressions and occlusions increase diversity of the face images. In addition to integrating this variability into the estimation process, a FLD model must also take into account a number of correlated factors. For instance, although both an angry person and a sad person have frowned eyebrows, an angry person will have pinched lips while a sad person will have sunken mouth corners BID6. A particularity of datasets geared towards FLD is that, on top of containing the position of these various facial landmarks, they also contain a number of other labels that can be seen as tasks on their own, such as gender recognition, smile recognition, glasses recognition or face orientation. For this reason, FLD datasets are particularly well-suited to evaluate MTL frameworks. Given T task-specific convolutional neural networks (CNNs), our goal is to share domain-specific information by connecting task-specific networks together using their respective feature maps. Unlike CS, our proposed approach will define this feature sharing in terms of two distinct non-linear transformations. Linear transformations like those used in CS can limit feature sharing ability, unlike ours that can learn complex transformations and properly connect each network. For the sake of simplicity, we suppose that the networks have the same structure, which we refer to as the underlying network. Note that our approach also works with different architectures. Let us Figure 1: Example of our collaborative block applied on the feature maps of five task-specific networks. The input feature maps (shown in part 1) are first concatenated depth-wise and transformed into a global feature map (part 2). The global feature map is then concatenate with each input feature map individually and transformed into task-specific feature maps (part 3). Each ing feature map is then added back to the input feature map using a skip connection (part 4), which gives the final outputs of the block (part 5).further decompose the underlying network as a series of blocks, where each block can be as small as a single layer, as large as the whole network itself, or based on simple rules such as grouping all layers with matching spatial dimensions or grouping every n subsequent layers. The arrangement of the layers into blocks does not change the nature of the network, but instead facilitates the understanding of applying our method. In particular, it makes explicit the depth at which we connect the feature maps via our framework. Since our approach is independent on depth, we will drop the depth index on the feature maps to further simplify the equations. As such, we will define the feature map output of a block at a certain depth as x t, where t ∈ {1 . . . T}, for each task t. Our approach takes as input all x t task-specific feature maps and processes them into new feature maps y t as follows: DISPLAYFORM0 where H and F t represent the central and the task-specific aggregations respectively, and [·] denotes depth-wise concatenation. We refer to as our collaborative block. The goal of H is to combine all task-specific feature maps x t into a global feature map z representing unified knowledge, while the goal of F is to merge back the global feature map z with each input feature map x t individually. As shown in Fig. 1, H and F have the following structure: where BN stands for batch normalization, δ for the ReLU activation function and Conv (h×w) for a standard convolutional layer with filters of size (h × w). The first Conv (1×1) layer in H divides the number of feature maps by a factor of 4, while the first Conv (1×1) layer in F divides it to match the size of x t. DISPLAYFORM1 DISPLAYFORM2 As seen by the presence of a skip connection in, the recent advances in residual network inspired the structure of our collaborative block. Based on the argument by BID7 that a network may more easily learn the proper underlying mapping by using an identity skip connection, we also argue that it may help our task-specific networks to more easily integrate the domain-information of each related task. One of the advantages of identity skip connections is that learning identity mappings can be done by simply pushing all parameters of the residual mapping towards zero. We integrated this observation in our collaborative block by allowing each task-specific network to easily remove the contribution of the global feature map z with an identity skip connection, in order to account for cases where it does not help. As we see in, pushing all parameters of F t towards zero would remove its contribution, and the output of the block would then simply revert to the original input x t.Our motivation for using an identity skip connection around the global feature map z comes from the fact that the depth at which we insert our collaborative block influences the relevance of each task towards another. Considering that a network learns a hierarchy of increasingly abstract features, some task-specific networks may benefit more by sharing their low-level features than by sharing their high-level features. For instance, tasks such as landmark localization and pose detection may profit from their low-level features containing better localization properties, while tasks such as face detection and gender recognition may take advantage of their more class-specific high-level features. Our collaborative block can take into account task relevance by deactivating a different set of residual mappings F t based on the level at which it appears in the network. An example of such specialization will be shown in our ablative study in Section 4.5. FIG0 presents an example of our Deep Collaboration Network (DCNet) using ResNet18 as the underlying network. As we can see in the top part of the figure, this comes down to interleaving the underlying network block structure with our collaborative block. Each collaborative block receives as input the output of each task-specific block, processes them as detailed in Fig. 1, and sends the back to each task-specific network. Adding our approach to any underlying network can be done by simply following the same pattern of interleaving the network block structure with our collaborative block. In each instance, the left column (blue) is for un-pretrained networks, while the right column (green) is for pre-trained networks. Our proposed approach obtains the lowest failure rates overall. In this section, we detail our multi-task learning (MTL) training framework and present our experiments in facial landmark detection (FLD) tasks. We also analyze our approach by performing an ablation study and by experimenting with task importance. As described previously, the problem of facial landmark detection is to predict the (x, y)-position on the image of specific landmarks associated with key features of the visage. While the number and type of landmarks are specific to each dataset, examples of standard landmarks to be predicted are the corners of the mouth, the tip of the nose and the center of the eyes. In addition to the facial landmarks, each dataset further defines a number of related tasks. These related tasks also vary from one dataset to another, and are typically gender recognition, smile recognition, glasses recognition or face orientation. One particularity of our optimization framework is that we treat each task as a classification problem. While this is straightforward for gender, smile and glasses recognition as they are already classification tasks, it is a bit more tricky for face orientation and FLD. For face orientation, instead of predicting the roll, yaw and pitch real value as in a regression problem, we divide each component into 30 degrees wide bins and predict the label of the bin corresponding to the value. Similarly for FLD, rather than predicting the real (x, y)-position of each landmark, we divide the image into 1 pixel wide bins and predict the label of the bin corresponding to the value. Note that we still use the original real values when comparing our prediction with the ground truth, such that we incorporate our approximation errors in the final score. By doing this, do not artificially boost our performance. We report our using the landmark failure rate metric, which is defined as follows: we first compute the mean distance between the predicted landmarks and the ground truth landmarks, then normalize it by the inter-ocular distance from the center of the eyes. A normalized mean distance greater than 10% is reported as a failure. As a first experiment, we performed facial landmark detection on the Multi-Task Facial Landmark (MTFL) dataset. The dataset contains 12,995 face images annotated with five facial landmarks and four related attributes of gender, smiling, wearing glasses and face profile (five profiles in total). The training set has 10,000 images, while the test set has 2,995 images. We perform four sets of experiments using an ImageNet pre-trained AlexNet, an ImageNet pre-trained ResNet18, an un-pretrained AlexNet and an un-pretrained ResNet18 as underlying networks. For AlexNet, we apply our collaborative block after each max pooling layer, while we do as shown in FIG0 for ResNet18. Figure 4: Example predictions of our DCNet with pre-trained ResNet18 as underlying network on the MTFL task. The first two contains failure cases, while last two contains successes. Elements in green correspond to ground truth, while those in blue correspond to our prediction. In addition to providing the facial landmarks (the small dots), we also include the labels of the related tasks: gender, smiling, wearing glasses and face profile. As shown in the first example, over-exposition can have a large impact on the prediction quality. We compare our approach to several other approaches of the literature. We include single-task learning (AN-S when using AlexNet as underlying network, RN-S when using ResNet18), MTL with a single central network (AN and RN), MTL with a single central network that is widen to match the number of parameters of our approach (ANx and RNx), the HyperFace network (HF), the Tasks-Constrained Deep Convolutional Network (TCDCN) and the Cross-Stitch approach (CS). We train each network three times for 300 epochs and report landmark failure rates averaged over the last five epochs, further averaged over the three tries. One we can observe from FIG1 is that our proposed approach obtained the lowest failure rates in each case. Indeed, our DCNet with un-pretrained and pre-trained AlexNet as underlying network obtained 19.99% and 19.93% failure rates respectively, while we obtained 15.32% and 14.34% with ResNet18. This is significantly lower than the other approaches to which we compare ourselves. For instance, with AlexNet, HF had 27.75% and 27.32%, XS had 26.41% and 25.65%, and TCDCN had 25.00% 1, while with ResNet18, XS had 18.43% and 15.52% respectively. We obtained the highest improvements when using AlexNet as the underlying network. With un-pretrained and pre-trained AlexNet, we obtained improvements of 6.42% and 5.07%, while we obtained 1.43% and 1.18% with ResNet18. Performing MTL with our approach can thus improve performance over using other approaches of the literature. FIG1 is that although increasing the number of parameters of multi-task AlexNet (AN) and ResNet18 (RN) can significantly improve performance, connecting task-specific networks with our approach is more efficient. For instance, while AlexNet (ANx) and ResNet18 (RNx) with widened layers that match the number of parameters of our approach lowers the failure rates from 20.05% (RN) and 28.02% (AN) to 16.75% (RNx) and 26.88% (ANx) respectively, our approach with AlexNet and ResNet18 as underlying networks further reduces the failure rates to 15.32% and 19.99%. These show that while simply increasing the number of parameters is an effortless avenue for improving performance, developing novel architectures enhancing network connectivity may open more rewarding research directions for further leveraging the domaininformation of related tasks. FORMULA0's approach and XS for Cross-Stitch. In each instance, the left column (blue) is for un-pretrained networks, while the right column (green) is for pre-trained networks. Our proposed approach (last column) obtains the lowest failure rates overall. As a second experiment, we performed domain adaptation on the Annotated Face in the Wild (AFW) dataset BID33. The dataset has 205 Flickr images, where each image can contain more than one face. Instead of using the images as provided, we process them using the available facial bounding boxes. We extract all faces with visible landmarks, which gives a total of 377 face images. We then take each network of Section 4.2 trained on the MTFL dataset and evaluate them without fine-tuning on these images. FIG4 presents the of this experiment. As it was the case in Section 4.2, our approach obtained the best overall. Indeed, our DCNet with un-pretrained and pre-trained AlexNet as underlying network obtained 43.77% and 45.62% failure rates respectively, while it obtained 37.75% and 37.84% with ResNet18. This is significantly lower then the other approaches. For instance, with AlexNet, HF had 48.54% and 51.81%, and XS had 48.98% and 49.43%, while with ResNet18, XS had 43.24% and 40.58% respectively. Unlike what we observed in Section 4.2, our approach obtained the highest improvement with AlexNet when using an un-pretrained underlying network, while it obtained the highest improvement with ResNet18 when using a pre-trained underlying network. Indeed, DCNet with un-pretrained and pretrained AlexNet obtained 4.24% and 2.39% improvements, while it obtained 3.10% and 2.74% with ResNet18 respectively. An interesting we can observe from FIG4 is that approaches with pre-trained AlexNet did not perform as well as those with pre-trained ResNet18, but rather increased the failure rates in comparison to the ones with un-pretrained AlexNet. For instance, single-task un-pretrained AlexNet obtained 48.72%, while single-task pre-trained AlexNet obtained a higher 50.84%. We also see a similar failure rate degradation for HF, XS and our approach. This is not the case when using ResNet18. Indeed, single-task un-pretrained ResNet18 obtained 44.92%, while single-task pretrained ResNet18 obtained a lower 41.20%. Although the dataset is small and more experiments would help to better understand why this is happening, these suggests that ResNet18 is more capable of adapting its pre-trained features for domain adaptation. As third experiment, we evaluate the influence of the number of training examples on MTL, using the Annotated Facial Landmarks in the Wild (AFLW) dataset BID11. The dataset has 21,123 Flickr images, where each image can contain more than one face. Instead of using the images as provided, we process them using the available facial bounding boxes. We extract all faces with visible landmarks, which gives a total of 2,111 images. This dataset defines 21 facial landmarks and has 3 related tasks (gender, wearing glasses and face orientation). For face orientation, we divide the roll, yaw and pitch into 30 degrees wide bins (14 bins in total), and predict the label corresponding to each real value. Figure 6: Landmark failure rate progression (in %) on the AFLW dataset with varying train / test ratio using a pre-trained ResNet18 as base network. Each curve is the average over three tries. Even though our approach has the slowest convergence rate, it outperforms the others in four of the five cases. Our experiment works as follows. With a pre-trained ResNet18 as underlying network, we compare our approach to single-task ResNet18 (RN-S), multi-task ResNet18 (RN) and Cross-Stitch network (XS) by training on a varying number of images. We use five different train / test ratios, starting with 0.1 / 0.9 up to 0.9 / 0.1 by 0.2 increment. In other words, we train each approaches on the first 10% of the available images and test on the other 90%, then repeat for all the other train / test ratios. We use the same training framework as in section 4.2. We train each network three times for 300 epochs, and report the landmark failure rate averaged over the last five epochs, further averaged over the three tries. As we can see in TAB2, our approach obtained the best performance in all cases except the first one. Indeed, we observe between 1.46% and 3.05% improvements with train / test ratios from 0.3 / 0.7 to 0.9 / 0.1, while we obtain a negative relative change of 4.90% with train / test ratio of 0.1 / 0.9. In fact, since all multi-task approaches obtained higher failure rates than the single-task approach, this suggests that the networks are over-fitting the small training set. Nonetheless, these show that we can obtain better performance using our approach. Figure 6 presents the landmark failure rate progression on all train / test ratios for each network. One interesting we can see from these figures is that our approach has the slowest convergence rate during the first stage of the training process. For instance, with a train / test ratio of 0.9 / 0.1, our Figure 7: Landmark failure rate improvement (in %) of our approach compared to XS when sampling random task weights. We used a pre-trained ResNet18 as underlying network. The histogram at the left and the plot at the top right represents performance improvement achieved by our proposed approach (positive value means lower failure rates), while the plot at the bottom right corresponds to the log of the task weights. Our approach outperformed XS in 86 out of the 100 tries, thus empirically demonstrating that our learning framework was not unfavorable towards XS and that our approach is less sensitive to the task weights λ.approach converges at about epoch 100, while the others start converging at about epoch 50. In fact, while the other methods have similar convergence rate, the epoch at which our approach converges increases as the number of training images decrease. Indeed, our approach converges at about epoch 130 with a train / test ratio of 0.5 / 0.5, while it converges at about epoch 160 with a train / test ratio of 0.1 / 0.9. Even though the convergence rate is slower, our approach still converges to a similar train failure rate. This gives a smaller train-to-test gap, which indicates that our approach has better generalization abilities. One particularity that we observe in TAB2 is that the XS network has relatively high failure rates. In the previous experiments of sections 4.2 and 4.3, XS had either similar or better performance than the other approaches (except ours). This could be due to our current multi-task learning framework that is unfavorable towards XS, which may prevent it from leveraging the domain-information of the related tasks. In order to investigate whether this is the case, we perform the following additional experiment. Using a pre-trained ResNet18 as underlying network, we compare our approach to XS by training each network 100 times using task weights randomly sampled from a log-uniform distribution. Specifically, we first sample from a uniform distribution γ ∼ U(log(1e−4), log), then use λ = exp(γ) as the weight. We trained both XS and our approach for 300 epochs with the same task weights using a train / test ratio of 0.5 / 0.5. Figure 7 presents the of this experiment. The plot at the top right of the figure represents the landmark failure rate improvement (in %) of our approach compared to XS, while the plot at the bottom right corresponds to the log of the task weights for each try. In 86 out of the 100 tries, our approach had a positive failure rate improvement, that is, obtained lower failure rates than XS. Moreover, as we can see in the histogram at the left of Fig. 7, in addition being normally distributed around a mean of 2.78%, our approach has a median failure rate improvement of 3.14% and a maximum improvement of 8.45%. These show that even though we sampled at random different weights for the related tasks, our approach outperforms XS in the majority of the cases. Our learning framework is therefore not unfavorable toward XS. As a final experiment, we perform an ablation study on our approach using the MTFL dataset with an un-pretrained ResNet18 as base network. The goal of this experiment is to test whether the observed performance improvements is due to the increased ability of each task-specific network to share their respective features, rather than only due to the intrinsic added representative ability of using additional non-linear layers. Our experiment works as follows: we evaluate the impact of removing the contribution of each task-specific network by masking their respective feature maps at the input of the central aggregation transformation. By referring to Fig. 1, this corresponds to Figure 8: Results of our ablation study on the MTFL dataset with an un-pretrained ResNet18 as underlying network. We remove each task-specific features from each respective central aggregation layer and evaluate the effect on landmark failure rate. The columns represent the task-specific network, while the rows correspond to the network block structure. Blocks with a high saturated color were found to have a large impact on performance. For instance, this ablative study shows that the influence of high-level face profile features is large within our proposed architecture, which corroborates with the well-known fact that facial landmark locations are highly correlated to profile orientation. This thus constitutes an empirical evidence of domain-specific information sharing via our approach.zeroing out the designated feature maps during concatenation in part 2 of the collaborative block (at the bottom of the figure). Note that the collaborative ResNet18 is trained on the MTFL dataset using the same framework as explained in Sec. 4.1, and the ablation study is performed on the test set. Figure 8 presents the of our ablation study. The columns represent each task-specific network, while the rows correspond to the network block structure. The blocks are ordered from bottom (input) to top (prediction), while the task-specific networks are ordered from left (main task) to right (related tasks). The color saturation indicates the influence of removing the task-specific feature maps from the corresponding central aggregation. A high saturation reflects high influence, while a low saturation reflects low influence. A first interesting that we can see from Fig. 8 is that removing features from the facial landmark detection network significantly increases landmark failure rate. For instance, we observe a negative (worse) relative change of 29.72% and 47.00% in failure rate by removing features from B3 and B2 respectively. This is interesting, as it shows that the main-task network contributes to and feed from the global features computed by the central aggregation transformation. Note that due to using a skip connection between the input and the global features, the network can remove the contribution of the global features by simply zeroing out its task-specific aggregation weights. These show that the opposite is instead happening, where the task-specific features from the facial landmark network influence the quality of the computed global features, which in turn influence the quality of the subsequent task-specific features. Another interesting is that B5 from the face profile-related task has the highest influence on failure rate. Indeed, we observe a negative relative change of 83.87% by removing the corresponding features maps from the central aggregation. Knowing that face orientation has a direct impact on the location of the facial landmarks, it makes sense that features from the head related task would be useful for improving landmark predictions. What is interesting in this case is that B5 has the highest-level features of all blocks, due to being at the top of the hierarchy of increasingly abstract features. Since we expect the highest-level features of the head network to resemble a standard rotation matrix, it is evident that the landmark network would use these rich features to better rotate the predicted facial landmarks. This is what we observe in Fig. 8. These constitute an empirical evidence that our approach not only allows leveraging domain-specific information from related tasks, but also improves knowledge sharing with better network connectivity. In this paper, we proposed the deep collaboration network (DCNet), a novel approach for connecting task-specific networks in a multi-task learning setting. It implements feature connectivity and sharing through two distinct non-linear transformations inside a collaborative block, which also incorporates skip connection and residual mapping that are known for their good training behavior. The first transformation aggregates the task-specific feature maps into a global feature map representing unified knowledge, and the second one merges it back into each task-specific network. One key characteristic of our collaborative blocks is that they can be dropped in virtually any existing architectures, making them universal adapters to endow deep networks with multi-task learning capabilities. Our on the MTFL, AFW and AFLW datasets showed that our DCNet outperformed several state-of-the-art approaches, including cross-stitch networks. Our additional ablation study, using ResNet18 as underlying network, confirmed our intuition that the task-specific networks exploited the added flexibility provided by our approach. Additionally, these task-specific networks successfully incorporated features having varying levels of abstraction. Evaluating our proposed approach on other MTL problems could be an interesting avenue for future works. For instance, the recurrent networks used to solve natural language processing problems could benefit from incorporating our novel method leveraging domain-information of related tasks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose a novel approach for connecting task-specific networks in a multi-task learning setting based on recent residual network advances.

Represent the TLDR for finding the corresponding scientific article

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.