Question: Represent the scientific article to find a short TLDR

Question: Hierarchical Task Networks (HTN) generate plans using a decomposition process guided by extra domain knowledge to guide search towards a planning task. While many HTN planners can make calls to external processes (e.g. to a simulator interface) during the decomposition process, this is a computationally expensive process, so planner implementations often use such calls in an ad-hoc way using very specialized domain knowledge to limit the number of calls. Conversely, the few classical planners that are capable of using external calls (often called semantic attachments) during planning do so in much more limited ways by generating a fixed number of ground operators at problem grounding time. In this paper we develop the notion of semantic attachments for HTN planning using semi co-routines, allowing such procedurally defined predicates to link the planning process to custom unifications outside of the planner. The ing planner can then use such co-routines as part of its backtracking mechanism to search through parallel dimensions of the state-space (e.g. through numeric variables). We show empirically that our planner outperforms the state-of-the-art numeric planners in a number of domains using minimal extra domain knowledge. Planning in domains that require numerical variables, for example, to drive robots in the physical world, must represent and search through a space defined by real-valued functions with a potentially infinite domain, range, or both. This type of numeric planning problem poses challenges in two ways. First, the description formalisms BID6 might not make it easy to express the numeric functions and its variables, ing in a description process that is time consuming and error-prone for real-world domains BID17. Second, the planners that try to solve such numeric problems must find efficient strategies to find solutions through this type of state-space. Previous work on formalisms for domains with numeric values developed the Semantic Attachment (SA) construct BID3 ) in classical planning. Semantic attachments were coined by (Weyhrauch 1981) to describe the attachment of an interpretation to a predicate symbol using an external procedure. Such construct allows the planner to reason about fluents where numeric values come from externally defined functions. In this paper, we extend the basic notion of semantic attachment for HTN planning by defining the semantics of the functions used as semantic attachments in a way that allows the HTN search and backtracking mechanism to be substantially more efficient. Our current approach focused on depth-first search HTN implementation without heuristic guidance, with free variables expected to be fullyground before task decomposition continues. Most planners are limited to purely symbolic operations, lacking structures to optimize usage of continuous resources involving numeric values BID9. Floating point numeric values, unlike discrete logical symbols, have an infinite domain and are harder to compare as one must consider rounding errors. One could overcome such errors with delta comparisons, but this solution becomes cumbersome as objects are represented by several numeric values which must be handled and compared as one, such as points or polygons. Planning descriptions usually simplify such complex objects to symbolic values (e.g. p25 or poly2) that are easier to compare. Detailed numeric values are ignored during planning or left to be decided later, which may force replanning BID17. Instead of simplifying the description or doing multiple comparisons in the description itself, our goal is to exploit external formalisms orthogonal to the symbolic description. To achieve that we build a mapping from symbols to objects generated as we query semantic attachments. Semantic attachments have already been used in classical planning BID3 ) to unify values just like predicates, and their main advantage is that new users do not need to discern between them and common predicates. Thus, we extend classical HTN planning algorithms and their formalism to support semantic attachment queries. While external function calls map to functions defined outside the HTN description, we implement SAs as semi co-routines BID1, subroutines that suspend and resume their state, to iterate across zero or more values provided one at a time by an external implementation, mitigating the potentially infinite range of the external function. Our contributions are threefold. First, we introduce SAs for HTN planning as a mechanism to describe and evaluate external predicates at execution time. Second, we introduce a symbol-object table to improve the readability of symbolic descriptions and the plans generated, while making it easier to handle external objects and structures. Finally, we empirically compare the ing HTN planner with a modern classical planner BID10 in a number of mixed symbolic/numeric domains showing substantial gains in speed with minimal domain knowledge. Classical planning algorithms must find plans that transform properties of the world from an initial configuration to a goal configuration. Each property is a logical predicate, a tuple with a name and terms that relate to objects of the world. A world configuration is a set of such tuples, which is called a state. To modify a state one must apply an operator, which must fulfill certain predicates at the current state, as preconditions, to add and remove predicates, the effects. Each operator applied creates a new intermediate state. The set of predicates and operators represent the domain, while each group of objects, initial and goal states represent a problem within this domain. In order to achieve the goal state the operators are used as rules to determine in which order they can be applied based on their preconditions and effects. To generalize the operators and simplify description one can use free variables to be replaced by objects available, a process called grounding. Once a state that satisfies the goal is reached, the sequence of ground operators is the plan BID14. A plan is optimal, iff it achieves the best possible quality in some criteria, such as number of operators, time or effort to execute; or satisficing if it reaches the goal without optimizing any metrics. PDDL BID11 ) is the standard description language to describe domains and problems, with features added through requirements that must be supported by the planner. Among such features are numeric-valued fluents to express numeric assignments and updates to the domain, as well as events and processes to express effects that occur in parallel with the operators in a single instant or during a time interval. Hierarchical planning shifts the focus from goal states to tasks to exploit human knowledge about problem decomposition using a hierarchy of domain knowledge recipes as part of the domain description BID13 ). This hierarchy is composed of primitive tasks that map to operators and non-primitive tasks, which are further refined into sub-tasks using methods. The decomposition process is repeated until only primitive-tasks mapping to operators remain, which in the plan itself. The goal is implicitly achieved by the plan obtained from the decomposition process. If no decomposition is possible, the task fails and a new expansion is considered one level up in the hierarchy, until there are no more possible expansions for the root task, only then a task decomposition is considered unachievable. Unlike classical planning, hierarchical planning only considers tasks obtained from the decomposition process to solve the problem, which both limits the ability to solve problems and improves execution time by evaluating a smaller number of operators. The HTN planning description is more complex than equivalent classical planning descriptions, since it includes domain knowledge with potentially recursive tasks, being able to solve more problems than classical planning. Classical planners with heuristic functions can solve problems mixing symbolic and numeric values efficiently using a process of discretization. A discretization process converts continuous values into sets of discrete symbols at often predefined granularity levels that vary between different domains. However, if the discretization process is not possible, one must use a planner that also supports numeric features, which requires another heuristic function, description language and usually more computing power due to the number of states generated by numeric features. Numeric features are especially important in domains where one cannot discretize the representation, they usually appear in geometric or physics subproblems of a domain and cannot be avoided during planning. Unlike symbolic approaches where literals are compared for equality during precondition evaluation, numeric value comparison is non-trivial. To avoid doing such comparison for every numeric value the user is left responsible for explicitly defining when one must consider rounding errors, which impacts description time and complexity. For complex object instances (in the object-oriented programming sense), such as polygons that are made of point instances, comparison details in the description are error-prone. Details such as the order of polygon points and floating point errors in their coordinates are usually irrelevant for the planner and the domain designer and should not be part of the domain description as they are part of a lowlevel specification. Such low-level specifications can be implemented by external function calls to improve what can be expressed and computed by a HTN planner. Such functions come with disadvantages, as they are not expected to keep an external state, returning a single value solely based on the provided parameters. While HTN planners can abstract away the numeric details via external function calls, there are limitations to this approach if a particular function is used in a decomposition tree where it is expected to backtrack and try new values from the function call (i.e. if the function is meant to be used to generate multiple terms as part of the search strategy). An external function must return a list of values to account for all possible decompositions so the planner tries one at a time until one succeeds. Generating a complete list is too costly when compared to computing a single value, as the first value could be enough to find a feasible plan. A semantic attachment, on the other hand, acts as an external predicate that unifies with one possible set of values at a time, rather than storing a complete list of possible sets of values to be stored in the state structure. This implementation saves time and memory during planning, as only backtracking causes the external co-routine to resume generating new unifications until a plan (or a certain amount of plans) is found. Each SA acts as a black box that simulates part of the environment encoding the in state variables that are often orthogonal to other predicates BID8. While common predicates are stored in a state structure, SAs are computed at execution time by co-routines. With a state that is not only declarative, with parts being procedurally computed, it is possible to minimize memory usage and delegate complex state-based operations to external methods otherwise incompatible or too complex for current planning description languages and planners that require grounding. We abstract away the numeric parts of the planning process encoded through SAs in a layer between the symbolic planner and external libraries. We leverage the abstract architecture of FIG0 with three layers inspired by the work of de BID2. In the symbolic layer we manipulate an anchor symbol as a term, such as polygon1, while in the external layer we manipulate a Polygon instance with N points as a geometric object based on what the selected external library specifies. With this approach we avoid complex representations in the symbolic layer. Instances created by the external layer that must be exposed to the symbolic layer are compared with stored object instances to reuse a previously defined symbol or create a new one, i.e. always represent position 2,5 with p1. This process makes symbol comparison work in the planning layer even for symbols related to complex external objects. The symbol-object table is also used to transform symbols into usable object instances by external function calls and SAs. Such table is global and consistent during the planning process, as each unique symbol will map the same internal object, even if such symbol is discarded in one decomposition branch. Once operations are finished in the external layer the process happens in reverse order, objects are transformed back into symbols that are exposed by free variables. The intermediate layer acts as the foreign function interface between the two layers, and can be modified to accommodate another external library without modifications to the symbolic description. SAs can work as interpreted predicates BID12, evaluating the truth value of a predicate procedurally, and also grounding free variables. SAs are currently limited to be used as method preconditions, which must not contain disjunctions. As only conjunctions and negations are allowed, one can reorder the preconditions during the compilation phase to improve execution time, removing the burden of the domain designer to optimize a mostly declarative description by hand, based on how free variables are used as SA Consider the abstract method example of Listing 1, with two SAs among preconditions, sa1 and sa2. The compiled output shown in Algorithm 1 has both SAs evaluated after common predicates, while function calls happen before or after each SA, based on which variables are ground at that point. In Line 4 the free variables fv1 and fv3 have a ground value that can only be read and not modified by other predicates or SAs. In Line 7 every variable is ground and the second function call can be evaluated. Algorithm 1 Compilation phase may reorder preconditions to optimize execution time. DISPLAYFORM0 for each fv1, fv3; state ⊂ {pre1,t1,t2, pre2,fv3,fv1} do 4:for each sa1(t1, fv1) do 5: free variable fv2 6:for each sa2(fv1, fv2) do 7: DISPLAYFORM1 The other limitation of current SA co-routines is that they must unify with a valid value within their internal iterations or have a stop condition, otherwise the HTN process will keep backtracking and evaluating the SA seeking new values and never returning failure. Due to the implementation support of arbitrary-precision arithmetic and accessing data from real-world streams of data/events (which are always new and potentially infinite) a valid value may never be found, and we expect the domain designer to implement mechanisms to limit the maximum number of times a SA might try to evaluate a call (i.e. to have finite stop conditions). This maximum number of tries can be implemented as a counter in the internal state of a SA, which is mostly used to mark values provided to the HTN to avoid repetition, but may achieve side-effects in external structures. The amount of side-effects in both external functions calls and SAs increase the complexity of correctness proofs and the ability to inspect and debug domain descriptions. A common problem when moving in dynamic and continuous environments is to check for object collisions, as agents and objects do not move across tiles in a grid. One solution is to calculate the distance between both objects centroid positions and verify if this value is in a safe margin before considering which action to take. To avoid the many geometric elements involved in this process we can map centroid position symbols to coordinate instances and only check the symbol returned from the symbol-object table, ignoring specific numeric details and comparing a symbol to verify if objects are near enough to collide. This process is illustrated in Figure 2, in which p 0 and p 1 are centroid position symbols that match symbols S 0 and S 1 in the symbol-object table, which maps their value to point objects O 0 and O 1. Such internal objects are used to compute distance and return a symbolic distance in situations where the actual numeric value is unnecessary. Figure 2: The symbol to object table maps symbols to object-oriented programming instances to hide procedural logic from the symbolic layer. In order to find a correct number to match a spatial or temporal constraint one may want to describe the relevant interval and precision to limit the amount of possibilities without having to discretely add each value to the state. Planning descriptions usually do not contain information about numeric intervals and precision, and if there is a way to add such information it is through the planner itself, as global definitions applied to all numeric functions, i.e. timestep, mantissa and exponent digits of DiNo BID15 ). The STEP SA described in Algorithm 2 addresses this problem, unifying t with one number at time inside the given interval with an step. To avoid having complex effects in the move operators one must not update adjacencies between planning objects during the planning process. Instead one must update only the Algorithm 2 The STEP SA replaces the pointer of t with a numeric symbol before resuming control to the HTN.1: function STEP(t, min = 0, max = ∞, = 1) 2: for i ← min to max step do 3: t ← symbol(i) 4:yield Resume HTN object position, deleting from the old position and adding the new position. Such positions come from a partitioned space, previously defined by the user. The positions and their adjacencies are either used to generate and store ground operators or stored as part of the state. To avoid both one could implement adjacency as a co-routine while hiding numeric properties of objects, such as position. Algorithm 3 shows the main two cases that appear in planning descriptions. In the first case both symbols are ground, and the co-routine resumes when both objects are adjacent, doing nothing otherwise, failing the precondition. In the second case s2, the second symbol, is free to be unified using s1, the first symbol, and a set of directions D to yield new positions to replace s2 pointer with a valid position, one at a time. In other terms, this co-routine either checks whether s2 is adjacent to s1 or tries to find a value adjacent to s1 binding it to s2 if such value exists. Algorithm 3 This ADJACENT SA implementation can either check if two symbols map to adjacent positions or generate new positions and their symbols to unify s2. DISPLAYFORM0 yield 8: else if s2 is free 9:for each (x, y) ∈ D do 10:nx ← x + x(s1); ny ← y + y(s1) 11:if 0 ≤ nx < WIDTH ∧ 0 ≤ ny < HEIGHT 12:s2 ← symbol(nx, ny) 13: yield We conducted emprirical tests in a machine with Dual 6-core Xeon CPUs @2GHz / 48GB memory, repeating experiments three times to obtain an average. The show a substantial speedup over the original classical description from ENHSP BID16 ) with more complex descriptions. Our HTN implementation is available at github.com/Maumagnaguagno/HyperTensioN U. In the Plant Watering domain BID7 one or more agents move in a 2D grid-based scenario to reach taps to obtain certain amounts of water and pour water in plants spread across the grid. Each agent can carry up to a certain amount of water and each plant requires a certain amount of water to be poured. Many state variables can be represented as numeric fluents, such as the coordinates of each agent, tap and plant, the amount of water to be poured and being carried by each agent, and the limits of how much water can be carried and the size of the grid. There are two common problems in this scenario, the first is to travel to either a tap or a plant, the second is the top level strategy. To avoid considering multiple paths in the decomposition process one can try to move straight to the goal first, and only to the goal in scenarios without obstacles, which simplifies the travel method. To achieve this straightforward movement we modify the ADJACENT SA to consider the goal position also, using an implementation of Algorithm 4. The top level strategy may consider which plant is closer to a tap or closer to an agent, how much water an agent can carry and so on. The simpler top level strategy is to verify how much water must be poured to a plant, travel to a tap, load water, travel to the previously selected plant and pour all the water loaded. Repeating this process until every plant has enough water poured. The travel method description using our modified JSHOP input language is shown in Listing 2 and compiled to Algorithm 5. We compare with the fastest satisficing configurations of ENHSP (sat and c sat) in FIG1, which shows that our approach is faster with execution times constantly below 0.01s, with both planners obtaining nonstep-optimal plans. Algorithm 4 In this goal-driven ADJACENT SA the positions are coordinate pairs, and two variables must be unified to a closer to the goal position in an obstacle-free scenario.1: function ADJACENT(x, y, nx, ny, gx, gy) 2: x ← numeric(x); y ← numeric(y) 3: gx ← numeric(gx); gy ← numeric(gy) 4:compare returns -1, 0, 1 for <, =, >, respectively 5: nx ← symbol(x + compare(gx, x)) 6: ny ← symbol(y + compare(gy, y)) 7: yield In the Car Linear domain BID0 the goal is to control the acceleration of a car, which has a minimum and maximum speed, without external forces applied, only moving through one axis to reach its destination, and requiring a small speed to safely stop. The idea is to propagate process effects to state functions, in this case acceleration to speed and speed to position, while being constrained to an acceptable speed and acceleration. The planner must decide when and for how long to increase or decrease acceleration, therefore becoming a temporal planning problem. We use a STEP SA to iterate over the time variable and propagate temporal effects and constraints, i.e. speed at time t. We compare the execution time of our approach with ENHSP with aibr, ENHSP main configuration for planning with autonomous processes, in TAB4. There is no comparison with a native HTN approach, as one would have to add a discrete finite set of time predicates (e.g. time 0) to the initial state description to be selected as time points during planning. For an agent to move in a continuous space it is common practice to simplify the environment to simpler geometric shapes for faster collision evaluation. One possible simplification is to find a circle or sphere that contains each obstacle and use this new shape to evaluate paths. In this context the best path is the one with the shortest lines between initial position and goal, considering bitangent lines between each simplified obstacle plus the amount of arc traversed on their borders, also know as Dubins path BID5. One possible approach for a satisficing plan is to move straight to the goal or to the closest obstacle to the goal and repeat the process. A precondition to such movement is to have a visible target, without any other obstacle between the current and target positions. A second consideration is the entrance direction, as clock or counterclockwise, to avoid cusped edges. Cusped edges are not part of optimal realistic paths, as the moving agent would have to turn around over a single point instead of changing its direction a few degrees to either side. For the problem defined in FIG3 Two possible approaches can be taken to solve the search over circular obstacles using bitangents. One is to rely on an external solver to compute the entire path, a motion planner, which could happen during or after HTN decomposition has taken place. When done during HTN decomposition, as seen in Listing 3, one must call the SEARCH-CIRCULAR function and consume the ing steps of the plan stored in the intermediate layer, not knowing about how close to the goal it could reach in case of failure. When done after HTN decomposition, one must replace certain dummy operators of the HTN plan and replan in case of failure. The second approach is to rely on parts of the external search, namely the VIS-IBLE function and CLOSEST SA, to describe continuous search to the HTN planner. The VISIBLE function returns true if from a point on a circle one can see the goal, false otherwise. The CLOSEST SA generates unifications from a circle with an entrance direction to a point in another circle with an exit direction, new points closer to the goal are generated first. Differently from external search, one can deal with failure at any moment, while being able to modify behavior with the same external parts, such as the initial direction the search starts with. Another advantage over the original solution is the ability to ask for N plans, which forces the HTN to backtrack after each plan is found and explore a different path until the amount of plans found equals N or the HTN planner fails to backtrack. A description of such approach is show in Listing 4. The execution time variance between the solutions is not as important as their different approaches to obtain a , from an external greedy best-first search to a HTN depth-first search. The external search also computes bitangents on demand, as bitangent precomputation takes a significant amount of time for many obstacles. We developed a notion of semantic attachments for HTN planners that not only allows a domain expert to easily define external numerical functions for real-world domains, but also provides substantial improvements on planning speed over comparable classical planning approaches. The use of semantic attachments improves the planning speed as one can express a potentially infinite state representation with procedures that can be exploited by a strategy described as HTN tasks. As only semantic attachments present in the path decomposed during planning are evaluated, a smaller amount of time is required when compared with approaches that precompute every possible value during operator grounding. Our description language is arguably more readable than the commonly used strategy of developing a domain specific planner with customized heuristics. Specifically, we allow designers to easily define external functions in a way that is readable within the domain knowledge encoded in HTN methods at design time, and also dynamically generate symbolic representations of external values at planning time, which makes generated plans easier to understand. Our work is the first attempt at defining the syntax and operation of semantic attachments for HTNs, allowing further research on search in SA-enabled domains within HTN planners. Future work includes implementing a cache to reuse previous values from external procedures applied to similar previous states BID4 ) and a generic construction to access such values in the symbolic layer, to obtain data from explored branches outside the state structure, i.e. to hold mutually exclusive predicate information. We plan to develop more domains, with varying levels of domain knowledge and SA usage, to obtain better comparison with other planners and their ing plan quality. The advantage of being able to exploit external implementations conflicts with the ability to incorporate such domain knowledge into heuristic functions, as such knowledge is outside the description. Further work is required to expose possible metrics from a SA to heuristic functions.

Represent the short summary for finding the corresponding scientific article (long).

An approach to perform HTN planning using external procedures to evaluate predicates at runtime (semantic attachments).

We represent a computer program using a set of simpler programs and use this representation to improve program synthesis techniques.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recent literature suggests that averaged word vectors followed by simple post-processing outperform many deep learning methods on semantic textual similarity tasks. Furthermore, when averaged word vectors are trained supervised on large corpora of paraphrases, they achieve state-of-the-art on standard STS benchmarks. Inspired by these insights, we push the limits of word embeddings even further. We propose a novel fuzzy bag-of-words (FBoW) representation for text that contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors. We show that max-pooled word vectors are only a special case of fuzzy BoW and should be compared via fuzzy Jaccard index rather than cosine similarity. Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair. This method is both efficient and easy to implement, yet outperforms current baselines on STS tasks by a large margin and is even competitive with supervised word vectors trained to directly optimise cosine similarity. Natural languages are able to encode sentences with similar meanings using very different vocabulary and grammatical constructs, which makes determining the semantic similarity between pieces of text a challenge. It is common to cast semantic similarity between sentences as the proximity of their vector representations. More than half a century since it was first proposed, the Bag-of-Words (BoW) representation (; BID47 BID37 remains a popular baseline across machine learning (ML), natural language processing (NLP), and information retrieval (IR) communities. In recent years, however, BoW was largely eclipsed by representations learned through neural networks, ranging from shallow BID36 BID21 to recurrent BID12 BID53, recursive BID51 BID55, convolutional BID30 BID32, self-attentive BID57 BID9 and hybrid architectures BID19 BID56 BID66.Interestingly, BID5 showed that averaged word vectors BID38 BID44 BID6 BID29 weighted with the Smooth Inverse Frequency (SIF) scheme and followed by a Principal Component Analysis (PCA) post-processing procedure were a formidable baseline for Semantic Textual Similarity (STS) tasks, outperforming deep representations. Furthermore, BID59 and BID58 showed that averaged word vectors trained supervised on large corpora of paraphrases achieve state-of-the-art , outperforming even the supervised systems trained directly on STS.Inspired by these insights, we push the boundaries of word vectors even further. We propose a novel fuzzy bag-of-words (FBoW) representation for text. Unlike classical BoW, fuzzy BoW contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors. Next, we show that max-pooled word vectors are a special case of fuzzy BoW. Max-pooling significantly outperforms averaging on standard benchmarks when word vectors are trained unsupervised. Since max-pooled vectors are just a special case of fuzzy BoW, we show that the fuzzy Jaccard index is a more suitable alternative to cosine similarity for comparing these representations. By contrast, the fuzzy Jaccard index completely fails for averaged word vectors as there is no connection between the two. The max-pooling operation is commonplace throughout NLP and has been successfully used to extract features in supervised systems BID10 BID32 BID31 BID13 BID12 BID15; however, to the best of our knowledge, the present work is the first to study max-pooling of pre-trained word embeddings in isolation and to suggest theoretical underpinnings behind this operation. Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair. DynaMax outperforms averaged word vector with cosine similarity on every benchmark STS task when word vectors are trained unsupervised. It even performs comparably to BID58's vectors under cosine similarity, which is a striking as the latter are in fact trained supervised to directly optimise cosine similarity between paraphrases, while our approach is completely unrelated to that objective. We believe this makes DynaMax a strong baseline that future algorithms should aim to beat in order to justify more complicated approaches to semantic similarity. As an additional contribution, we conduct significance analysis of our . We found that recent literature on STS tends to apply unspecified or inappropriate parametric tests, or leave out significance analysis altogether in the majority of cases. By contrast, we rely on nonparametric approaches with much milder assumptions on the test statistic; specifically, we construct bias-corrected and accelerated (BCa) bootstrap confidence intervals BID17 for the delta in performance between two systems. We are not aware of any prior works that apply such methodology to STS benchmarks and hope the community finds our analysis to be a good starting point for conducting thorough significance testing on these types of experiments. The bag-of-words (BoW) model of representing text remains a popular baseline across ML, NLP, and IR communities. BoW, in fact, is an extension of a simpler set-of-words (SoW) model. SoW treats sentences as sets, whereas BoW treats them as multisets (bags) and so additionally captures how many times a word occurs in a sentence. Just like with any set, we can immediately compare SoW or BoW using set similarity measures (SSMs), such as DISPLAYFORM0 These coefficients usually follow the pattern #{shared elements} #{total elements}. From this definition, it is clear that sets with no shared elements have a similarity of 0, which is undesirable in NLP as sentences with completely different words can still share the same meaning. But can we do better?For concreteness, let's say we want to compare two sentences corresponding to the sets A = {'he', 'has', 'a', 'cat'} and B = {'she', 'had', 'one', 'dog'}. The situation here is that A ∩ B = ∅ and so their similarity according to any SSM is 0. Yet, both A and B describe pet ownership and should be at least somewhat similar. If a set contains the word'cat', it should also contain a bit of'pet', a bit of'animal', also a little bit of'tiger' but perhaps not too much of an'airplane'. If both A and B contained'pet','animal', etc. to some degree, they would have a non-zero similarity. This intuition is the main idea behind fuzzy sets: a fuzzy set includes all words in the vocabulary simultaneously, just with different degrees of membership. This generalises classical sets where a word either belongs to a set or it doesn't. We can easily convert a singleton set such as {'cat'} into a fuzzy set using a similarity function sim(w i, w j) between words. We simply compute the similarities between'cat' and all the words w j in the vocabulary and treat those values as membership degrees. As an example, the set {'cat'} really becomes {'cat' : 1, 'pet' : 0.9, 'animal' : 0.85, . . ., 'airplane' : 0.05, . . .} Fuzzifying singleton sets is straightforward, but how do we go about fuzzifying the entire sentence {'he', 'has', 'a', 'cat'}? Just as we use the classical union operation ∪ to build bigger sets from smaller ones, we use the fuzzy union to do the same but for fuzzy sets. The membership degree of a word in the fuzzy union is determined as the maximum membership degree of that word among each of the fuzzy sets we want to unite. This might sound somewhat arbitrary: after all, why max and not, say, sum or average? We explain the rationale in Section 2.1; and in fact, we use the max for the classical union all the time without ever noticing it. Indeed, {'cat'} ∪ {'cat'} = {'cat'} and not {'cat' : 2}. This is simply because we computed max = 1 and not sum = 2. Similarly {'cat'} ∪ ∅ = {'cat'} since max = 1 and not avg = 1/2.The key insight here is the following. An object that assigns the degrees of membership to words in a fuzzy set is called the membership function. Each word defines a membership function, and even though'cat' and'dog' are different, they are semantically similar (in terms of cosine similarity between their word vectors, for example) and as such give rise to very similar membership functions. This functional proximity will propagate into the SSMs, thus rendering them a much more realistic model for capturing semantic similarity between sentences. To actually compute the fuzzy SSMs, we need just a few basic tools from fuzzy set theory, all of which we briefly cover in the next section. Fuzzy set theory BID63 is a well-established formalism that extends classical set theory by incorporating the idea that elements can have degrees of membership in a set. Constrained by space, we define the bare minimum needed to compute the fuzzy set similarity measures and refer the reader to BID34 for a much richer introduction. Definition: A set of all possible terms V = {w 1, w 2, . . ., w N} that occur in a certain domain is called a universe. DISPLAYFORM0 Notice how the above definition covers all the set-like objects we discussed so far. If L = {0, 1}, then A is simply a classical set and µ is its indicator (characteristic) function. If L = N ≥0 (non-negative integers), then A is a multiset (a bag) and µ is called a count (multiplicity) function. In literature, A is called a fuzzy set when L =. However, we make no restrictions on the range and call A a fuzzy set even when L = R, i.e. all real numbers. Definition: Let A = (V, µ) and B = (V, ν) be two fuzzy sets. The union of A and B is a fuzzy set A ∪ B = (V, max(µ, ν)). The intersection of A and B is a fuzzy set A ∩ B = (V, min(µ, ν)).Interestingly, there are many other choices for the union and intersection operations in fuzzy set theory. However, only the max-min pair makes these operations idempotent, i.e. such that A∪A = A and A ∩ A = A, just as in the classical set theory. By contrast, it is not hard to verify that neither sum nor average satisfy the necessary axioms to qualify as a fuzzy union or intersection. Definition: Let A = (V, µ) be a fuzzy set. The number |A| = w∈V µ(w) is called the cardinality of a fuzzy set. Fuzzy set theory provides a powerful framework for reasoning about sets with uncertainty, but the specification of membership functions depends heavily on the domain. In practice these can be designed by experts or learned from data; below we describe a way of generating membership functions for text from word embeddings. From the algorithmic point of view any bag-of-words is just a row vector. The i-th term in the vocabulary has a corresponding N -dimensional one-hot encoding e (i). The vectors e (i) are orthonormal and in totality form the standard basis of R N. The BoW vector for a sentence S is simply DISPLAYFORM0, where c i is the count of the word w i in S.The first step in creating the fuzzy BoW representation is to convert every term vector e (i) into a membership vector µ (i). It really is the same as converting a singleton set {w i} into a fuzzy set. We call this operation'word fuzzification', and in the matrix form it is simply written as Algorithm 1 DynaMax-Jaccard Input: Word embeddings for the first sentence DISPLAYFORM1 Input: Word embeddings for the second sentence DISPLAYFORM2 Input: A vector with all zeros z ∈ R Output: DISPLAYFORM0 Here W ∈ R N ×d is the word embedding matrix and U ∈ R K×d is the'universe' matrix. Let us dissect the above expression. First, we convert a one-hot vector into a word embedding w (i) = e (i) W. This is just an embedding lookup and is exactly the same as the embedding layer in neural networks. Next, we compute a vector of similarities DISPLAYFORM1 and all the K vectors in the universe. The most sensible choice for the universe matrix is the word embedding matrix itself, i.e. U = W. In that case, the membership vector µ (i) has the same dimensionality as e (i) but contains similarities between the word w i and every word in the vocabulary (including itself).The second step is to combine all µ (i) back into a sentence membership vector µ s. At this point, it's very tempting to just sum or average over all DISPLAYFORM2. But we remember: in fuzzy set theory the union of the membership vectors is realised by the element-wise max-pooling. In other words, we don't take the average but max-pool instead: DISPLAYFORM3 Here the max returns a vector where each dimension contains the maximum value along that dimension across all N input vectors. In NLP this is also known as max-over-time pooling BID10. Note that any given sentence S usually contains only a small portion of the total vocabulary and so most word counts c i will be 0. If the count c i is 0, then we have no need for µ (i) and can avoid a lot of useless computations, though we must remember to include the zero vector in the max-pooling operation. We call the sentence membership vector µ S the fuzzy bag-of-words (FBoW) and the procedure that converts classical BoW b S into fuzzy BoW µ S the'sentence fuzzification'. Suppose we have two fuzzy BoW µ A and µ B. How can we compare them? Since FBoW are just vectors, we can use the standard cosine similarity cos(µ A, µ B). On the other hand, FBoW are also fuzzy sets and as such can be compared via fuzzy SSMs. We simply copy the definitions of fuzzy union, intersection and cardinality from Section 2.1 and write down the fuzzy Jaccard index: DISPLAYFORM0.Exactly the same can be repeated for other SSMs. In practice we found their performance to be almost equivalent but always better than standard cosine similarity (see Appendix B). So far we considered the universe and the word embedding matrix to be the same, i.e. U = W. This means any FBoW µ S contains similarities to all the words in the vocabulary and has exactly the same dimensionality as the original BoW b S. Unlike BoW, however, FBoW is almost never sparse. This motivates us to choose the matrix U with fewer rows that W. For example, the top principal axes of W could work. Alternatively, we could cluster W into k clusters and keep the centroids. Of course, the rows of such U are no longer word vectors but instead some abstract entities. A more radical but completely non-parametric solution is to choose U = I, where I ∈ R d×d is just the identity matrix. Then the word fuzzifier reduces to a word embedding lookup: DISPLAYFORM0 The sentence fuzzifier then simply max-pools all the word embeddings found in the sentence: DISPLAYFORM1 From this we see that max-pooled word vectors are only a special case of fuzzy BoW. Remarkably, when word vectors are trained unsupervised, this simple representation combined with the fuzzy Jaccard index is already a stronger baseline for semantic textual similarity than the averaged word vector with cosine similarity, as we will see in Section 4.More importantly, the fuzzy Jaccard index works for max-pooled word vectors but completely fails for averaged word vectors. This empirically validates the connection between fuzzy BoW representations and the max-pooling operation described above. From the linear-algebraic point of view, fuzzy BoW is really the same as projecting word embeddings on a subspace of R d spanned by the rows of U, followed by max-pooling of the features extracted by this projection. A fair question then is the following. If we want to compare two sentences, what subspace should we project on? It turns out that if we take word embeddings for the first sentence and the second sentence and stack them into matrix U, this seems to be a sufficient space to extract all the features needed for semantic similarity. We noticed this empirically, and while some other choices of U do give better , finding a principled way to construct them remains future work. The matrix U is not static any more but instead changes dynamically depending on the sentence pair. We call this approach Dynamic Max or DynaMax and provide pseudocode in Algorithm 1. Just as SoW is a special case of BoW, we can build the fuzzy set-of-words (FSoW) where the word counts c i are binary. The performance of FSoW and FBoW is comparable, with FBoW being marginally better. For simplicity, we implement FSoW in Algorithm 1 and in all our experiments. As evident from Equation FORMULA5, we use dot product as opposed to (scaled or clipped) cosine similarity for the membership functions. This is a reasonable choice as most unsupervised and some supervised word vectors maximise dot products in their objectives. For further analysis, see Appendix A. Any method that casts semantic similarity between sentences as the proximity of their vector representations is related to our work. Among those, the ones that strengthen bag-of-words by incorporating the sense of similarity between individual words are the most relevant. The standard Vector Space Model (VSM) basis e (i) is orthonormal and so the BoW model treats all words as equally different. BID50 proposed the'soft cosine measure' to alleviate this issue. They build a non-orthogonal basis f (i) where cos(f (i), f (j) ) = sim(w i, w j), i.e. the cosine similarity between the basis vectors is given by similarity between words. Next, they rewrite BoW in comparing different combinations of fuzzy BoW representation (either averaged or max-pooled, or the DynaMax approach) and similarity measure (either cosine or Jaccard). The bolded methods are ones proposed in the present work. Note that averaged vectors with Jaccard similarity are not included in these plots, as they consistently perform 20-50 points worse than other methods; this is predicted by our analysis as averaging is not an appropriate union operation in fuzzy set theory. In virtually every case, max-pooled with cosine outperforms averaged with cosine, which is in turn outperformed by max-pooled and DynaMax with Jaccard. An exception to the trend is STS13, for which the SMT subtask dataset is no longer publicly available; this may have impacted the performance when averaged over different types of subtasks.terms of f (i) and compute cosine similarity between transformed representations. However, when cos(DISPLAYFORM0, where w i, w j are word embeddings, their approach is equivalent to cosine similarity between averaged word embeddings, i.e. the standard baseline. BID35 consider L1-normalised bags-of-words (nBoW) and view them as a probability distributions over words. They propose the Word Mover's Distance (WMD) as a special case of the Earth Mover's Distance (EMD) between nBoW with the cost matrix given by pairwise Euclidean distances between word embeddings. As such, WMD does not build any new representations but puts a lot of structure into the distance between BoW. BID65 proposed an alternative version of fuzzy BoW that is conceptually similar to ours but executed very differently. They use clipped cosine similarity between word embeddings to compute the membership values in the word fuzzification step. We use dot product not only because it is theoretically more general but also because dot product leads to significant improvements on the benchmarks. More importantly, however, their sentence fuzzification step uses sum to aggregate word membership vectors into a sentence membership vector. We argue that max-pooling is a better choice because it corresponds to the fuzzy union. Had we used the sum, the representation would have really reduced to a (projected) summed word vector. Lastly, they use FBoW as features for a supervised model but stop short of considering any fuzzy similarity measures, such as fuzzy Jaccard index. BID24 BID32 BID0 proposed and developed soft cardinality as a generalisation to the classical set cardinality. In their framework set membership is crisp, just as in classical set theory. However, once the words are in a set, their contribution to the overall cardinality depends on how similar they are to each other. The intuition is that the set A = {'lion', 'tiger', 'leopard'} should have cardinality much less than 3, because A contains very similar elements. Likewise, the set B = {'lion', 'airplane', 'carrot'} deserves a cardinality closer to 3. We see that the soft cardinality framework is very different from our approach, as it'does not consider uncertainty in the membership of a particular element; only uncertainty as to the contribution of an element to the cardinality of the set' BID24. To evaluate the proposed similarity measures we set up a series of experiments on the established STS tasks, part of the SemEval shared task series 2012-2016 BID1 BID32 BID0 BID4 BID7. The idea behind the STS benchmarks is to measure comparing other BoW-based methods to ones using fuzzy Jaccard similarity. The bolded methods are ones proposed in the present work. We observe that even classical crisp Jaccard is a fairly reasonable baseline, but it is greatly improved by the fuzzy set treatment. Both max-pooled word vectors with Jaccard and DynaMax outperform the other methods by a comfortable margin, and the max-pooled version in particular performs astonishingly well given its great simplicity.how well the semantic similarity scores computed by a system (algorithm) correlate with human judgements. Each year's STS task itself consists of several subtasks. By convention, we report the mean Pearson correlation between system and human scores, where the mean is taken across all the subtasks in a given year. Our implementation wraps the SentEval toolkit BID11 and is available on GitHub 1. We also rely on the following publicly available word embeddings: GloVe BID44 trained on Common Crawl (840B tokens); fastText BID6 ) trained on Common Crawl (600B tokens); word2vec BID39;c) trained on Google News, CoNLL BID64, and Book Corpus; and several types of supervised paraphrastic vectors -PSL BID59, PP-XXL BID60, and PNMT BID58.We estimated word frequencies on an English Wikipedia dump dated July 1 st 2017 and calculated word weights using the same approach and parameters as in BID5. Note that these weights can in fact be derived from word vectors and frequencies alone rather than being inferred from the validation set BID18, making our techniques fully unsupervised. Finally, as the STS'13 SMT dataset is no longer publicly available, the mean Pearson correlations reported in our experiments involving this task have been re-calculated accordingly. We first ran a set of experiments validating the insights and derivations described in Section 2. These are presented in FIG0. The main takeaways are the following:• Max-pooled word vectors outperform averaged word vectors in most tasks.• Max-pooled vectors with cosine similarity perform worse than max-pooled vectors with fuzzy Jaccard similarity. This supports our derivation of max-pooled vectors as a special case of fuzzy BoW, which thus should be compared via fuzzy set similarity measures and not cosine similarity (which would be an arbitrary choice).• Averaged vectors with fuzzy Jaccard similarity completely fail. This is because fuzzy set theory tells us that the average is not a valid fuzzy union operation, so a fuzzy set similarity is not appropriate for this representation.• DynaMax shows the best performance across all tasks, possibly thanks to its superior ability to extract and max-pool good features from word vectors. Next we ran experiments against some of the related methods described in Section 3, namely WMD BID35 and soft cardinality BID28 with clipped cosine similarity as an affinity function and the softness parameter p = 1. From FIG1, we see that even classical Jaccard index is a reasonable baseline, but fuzzy Jaccard especially in the DynaMax formulation handily outperforms comparable methods. For context and completeness, we also compare against other popular sentence representations from the literature in TAB0. We include the following methods: BoW with ELMo embeddings BID54. Note that avg-cos refers to taking the average word vector and comparing by cosine similarity, and word2vec refers to the Google News version. Clearly more sophisticated methods of computing sentence representations do not shine on the unsupervised STS tasks when compared to these simple BoW methods with high-quality word vectors and the appropriate similarity metric. † indicates the only STS13 (to our knowledge) that includes the SMT subtask. BID46, Skip-Thought, InferSent BID12, Universal Sentence Encoder with DAN and Transformer BID8, and STN multitask embeddings BID54. These experiments lead to an interesting observation: • PNMT embeddings are the current state-of-the-art on STS tasks. PP-XXL and PNMT were trained supervised to directly optimise cosine similarity between average word vectors on very large paraphrastic datasets. By contrast, DynaMax is completely unrelated to the training objective of these vectors, yet has an equivalent performance. Finally, another well-known and high-performing simple baseline was proposed by BID5. However, as also noted by BID41, this method is still offline because it computes the sentence embeddings for the entire dataset, then performs PCA and removes the top principal component. While their method makes more assumptions than ours, nonetheless we make a head-to-head comparison with them in TAB2 using the same word vectors as in BID5, showing that DynaMax is still quite competitive. To strengthen our empirical findings, we provide ablation studies for DynaMax in Appendix C, showing that the different components of the algorithm each contribute to its strong performance. We also conduct significance testing in Appendix D by constructing bias-corrected and accelerated (BCa) bootstrap confidence intervals BID17 for the delta in performance between two algorithms. This constitutes, to the best of our knowledge, the first attempt to study statistical significance on the STS benchmarks with this type of non-parametric analysis that respects the statistical peculiarities of these datasets. In this work we combine word embeddings with classic BoW representations using fuzzy set theory. We show that max-pooled word vectors are a special case of FBoW, which implies that they should be compared via the fuzzy Jaccard index rather than the more standard cosine similarity. We also present a simple and novel algorithm, DynaMax, which corresponds to projecting word vectors onto a subspace dynamically generated by the given sentences before max-pooling over the features. DynaMax outperforms averaged word vectors compared with cosine similarity on every benchmark STS task when word vectors are trained unsupervised. It even performs comparably to supervised vectors that directly optimise cosine similarity between paraphrases, despite being completely unrelated to that objective. Both max-pooled vectors and DynaMax constitute strong baselines for further studies in the area of sentence representations. Yet, these methods are not limited to NLP and word embeddings, but can in fact be used in any setting where one needs to compute similarity between sets of elements that have rich vector representations. We hope to have demonstrated the benefits of experimenting more with similarity metrics based on the building blocks of meaning such as words, rather than complex representations of the final objects such as sentences. In the word fuzzification step the membership values for a word w are obtained through a similarity function sim (w, u (j) ) between the word embedding w and the rows of the universe matrix U, i.e. DISPLAYFORM0 In Section 2.2, sim(w, u (j) ) was the dot product w · u (j) and we could simply write µ = wU T. There are several reasons why we chose a similarity function that takes values in R as opposed to DISPLAYFORM1 First, we can always map the membership values from R to and vice versa using, e.g. the logistic function σ(x) = 1 1+e −ax with an appropriate scaling factor a > 0. Intuitively, large negative membership values would imply the element is really not in the set and large positive values mean it is really in the set. Of course, here both'large' and'really' depend on the scaling factor a. In any case, we see that the choice of R vs. is not very important mathematically. Interestingly, since we always max-pool with a zero vector, fuzzy BoW will not contain any negative membership values. This was not our intention, just a by-product of the model. For completeness, let us insist on the range and choose sim (w, u (j) ) to be the clipped cosine similarity max (0, cos(w, u (j) )). This is in fact equivalent to simply normalising the word vectors. Indeed, the dot product and cosine similarity become the same after normalisation, and max-pooling with the zero vector removes all the negative values, so the ing representation is guaranteed to be a-fuzzy set. Our for normalised word vectors are presented in TAB3.After comparing TAB0 we can draw two . Namely, DynaMax still outperforms avg-cos by a large margin even when word vectors are normalised. However, normalisation hurts both approaches and should generally be avoided. This is not surprising since the length of word vectors is correlated with word importance, so normalisation essentially makes all words equally important BID48. In Section 2 we mentioned several set similarity measures such as Jaccard BID23, OtsukaOchiai (; BID42 and Sørensen-Dice (; BID52 coefficients. Here in TAB4, we show that fuzzy versions of the above coefficients have almost identical performance, thus confirming that our are in no way specific to the Jaccard index. Table 5 : Mean Pearson correlation on STS tasks for the ablation studies. As described in Appendix C, it is clear that the three components of the algorithm -the dynamic universe, the max-pooling operation, and the fuzzy Jaccard index -all contribute to the strong performance of DynaMax-Jaccard. The DynaMax-Jaccard similarity (Algorithm 1) consists of three components: the dynamic universe, the max-pooling operation, and the fuzzy Jaccard index. As with any algorithm, it is very important to track the sources of improvements. Consequently, we perform a series of ablation studies in order to isolate the contribution of each component. For brevity, we focus on fastText because it produced the strongest for both the DynaMax and the baseline FIG0 ).The of the ablation study are presented in Table 5. First, we show that the dynamic universe is superior to other sensible choices, such as the identity and random 300 × 300 projection with components drawn from N. Next, we show that the fuzzy Jaccard index beats the standard cosine similarity on 4 out 5 benchmarks. Finally, we find that max considerably outperforms other pooling operations such as averaging, sum and min. We conclude that all three components of DynaMax are very important. It is clear that max-pooling is the top contributing factor, followed by the dynamic universe and the fuzzy Jaccard index, whose contributions are roughly equal. As discussed in Section 4, the core idea behind the STS benchmarks is to measure how well the semantic similarity scores computed by a system (algorithm) correlate with human judgements. In this section we provide detailed and significance analysis for all 24 STS subtasks. Our approach can be formally summarised as follows. We assume that the human scores H, the system scores A and the baseline system scores B jointly come from some trivariate distribution P (H, A, B), which is specific to each subtask. To compare the performance of two systems, we compute the sample Pearson correlation coefficients r AH and r BH. Since these correlations share the variable H, they are themselves dependent. There are several parametric tests for the difference between dependent correlations; however, their appropriateness beyond the assumptions of normality remains an active area of research BID22 BID62 BID61. The distributions of the human scores in the STS tasks are generally not normal; what's more, they vary greatly depending on the subtask (some are multimodal, others are skewed, etc.).Fortunately, nonparametric resampling-based approaches, such as bootstrap BID16, present an attractive alternative to parametric tests when the distribution of the test statistic is unknown. In our case, the statistic is simply the difference between two correlations∆ = r AH − r BH. The main idea behind bootstrap is intuitive and elegant: just like a sample is drawn from the population, a large number of'bootstrap' samples can be drawn from the actual sample. In our case, the dataset consists of triplets DISPLAYFORM0. Each bootstrap sample is a of drawing M data points from D with replacement. Finally, we approximate the distribution of ∆ by evaluating it on a large number of bootstrap samples, in our case ten thousand. We use this information to construct bias-corrected and accelerated (BCa) 95% confidence intervals for ∆. BCa BID17 ) is a fairly advanced second-order method that accounts for bias and skewness in the bootstrapped distributions, effects we did observe to a small degree in certain subtasks. Once we have the confidence interval for ∆, the decision rule is then simple: if zero is inside the interval, then the difference between correlations is not significant. Inversely, if zero is outside, we may conclude that the two approaches lead to statistically different . The location of the interval further tells us which one performs better. The are presented in TAB6. In summary, out of 72 experiments we significantly outperform the baseline in 56 (77.8%) and underperform in only one (1.39%), while in the remaining 15 (20.8%) the differences are nonsignificant. We hope our analysis is useful to the community and will serve as a good starting point for conducting thorough significance testing on the current as well as future STS benchmarks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Max-pooled word vectors with fuzzy Jaccard set similarity are an extremely competitive baseline for semantic similarity; we propose a simple dynamic variant that performs even better.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.