Question: Represent the article for finding a single-sentence TLDR!

Question: Coalition operations are essential for responding to the increasing number of world-wide incidents that require large-scale humanitarian assistance. Many nations and non-governmental organizations regularly coordinate to address such problems but their cooperation is often impeded by limits on what information they are able to share. In this paper, we consider the use of an advanced cryptographic technique called secure multi-party computation to enable coalition members to achieve joint objectives while still meeting privacy requirements. Our particular focus is on a multi-nation aid delivery scheduling task that involves coordinating when and where various aid provider nations will deliver relief materials after the occurrence of a natural disaster. Even with the use of secure multi-party computation technology, information about private data can leak. We describe how the emerging field of quantitative information flow can be used to help data owners understand the extent to which private data might become vulnerable as the of possible or actual scheduling operations, and to enable automated adjustments of the scheduling process to ensure privacy requirements Coalition operations are becoming an increasing focus for many nations. The need for collaboration derives from increasing awareness of mutual interests among both allies and nations that traditionally have not worked closely together. For example, coalition operations for Humanitarian Assistance and Disaster Relief (HADR) have increased substantially in recent years in both numbers and scope. With the impact of global warming, it is anticipated that there will be even more large-scale humanitarian crises ing from adverse weather-related events and sea-level rises. These coalitions often involve not just government and military organizations but also non-governmental organizations (NGOs) and commercial entities. A key challenge facing coalitions is how to collaborate without releasing information that could jeopardize national (or organizational) interests. Information security mechanisms for the military to date have focused on safeguarding information by limiting its access and use. This approach has lead to a significant undersharing problem, which impedes Distribution Statement A: Approved for Public Release; Distribution is Unlimited.effective joint operations. Recent work on defining access control mechanisms is useful for enabling selective sharing of information that is safe to release BID4. However, many coalition tasks require information that participants do not wish to make available to others. For example, consider the problem of scheduling the delivery of aid (food, water, medicine, fuel) to impacted nations after a natural disaster. Historically, international response has involved dozens of countries and NGOs, each with the desire to contribute in interconnected and potentially conflicting ways. Ideally coordination would be achieved by directly sharing information about the amount of aid each contributor has available or can produce, where that aid is situated, the position and storage capacity of ships for delivering the aid, harbor facilities where aid ships could dock, etc. But the owners of this data may be unwilling to directly share it with coalition partners, for fear of revealing information that impacts national security (e.g., ship locations) or competitive advantages (e.g., a company's backlog of inventory).To address this problem of coalition collaboration without revealing private information, we exploit a cryptographic technology called secure multi-party computation (MPC) BID6. MPC protocols enable mutually distrusting parties to perform joint computations on private information while it remains encrypted. In our work, we use MPC to enable privacy-preserving computations over several types of coordination tasks related to scheduling aid delivery. While participants cannot directly learn others' private inputs from the process/protocol of a secure multi-party computation, they may be able to infer something about private data based on the of the computation. For this reason, our privacy-aware scheduling solution makes use of a complementary technology called quantitative information flow (QIF) to measure the degree to which private data used in the scheduling task might leak to other participants. Insights gained from this analysis are folded back into the scheduling process via an adaptive workflow capability, to ensure that the vulnerability of private data stays within acceptable thresholds. The paper is organized as follows. We begin by summarizing our aid distribution task, including a description of the core scheduling problem and the data (private and non-private) belonging to the various coalition members. We then provide a short overview of secure multi-party computation followed by a description of how we employ it within a broader adaptive workflow framework to address the aid delivery scheduling task. Next, we describe our use of quantitative information flow to assess the vulnerability of private information as the scheduling progresses and to adapt the scheduling workflow in light of those assessments to ensure adherence to predefined privacy requirements. We conclude by identifying directions for future work and summarizing our contributions. The aid delivery problem that we consider involves two categories of participants:• N Aid Provider nations, each of which has some number S n of ships with aid (e.g., food, medicine) to be delivered• a single Aid Recipient nation that has P ports to which aid can be deliveredCollectively, these participants need to formulate a schedule for delivering aid on board the Aid Provider ships to ports belonging to the Aid Recipient, ensuring delivery prior to a specified deadline. As summarized in FIG0, a solution involves assigning to each Aid Provider ship: a port to which its aid should be delivered, a berth at the port, and a docking time. These assignments are subject to various constraints on ship and port characteristics to ensure physical compatibility between the ship and the port/berth, schedule availability of the berth, and the ability for the ship to completing the docking before the assigned deadline. We further seek an assignment that optimizes according to the following load-balancing criteria. Optimization Criteria: Load-balancing across ports. Let Assigned(Port i) designate the number of aid provider ships assigned to aid recipient port Port i. An optimal solution is a set of assignments that minimizes DISPLAYFORM0 Generating a solution to this scheduling problem requires information from the various parties about their assets (ships, ports), some of which they would prefer not to share with other coalition members. FIG0 summarizes this private data. In both FIG0 and FIG0 the problem data is color-coded, with blue used for private data belonging to an Aid Provider nation and green for private data belonging to the Aid Recipient nation. As the coloring clearly shows, determining a solution requires combining private information from multiple parties, which motivates our use of secure multi-party within our scheduling algorithm. To address privacy concerns in our aid delivery use case, we leverage MPC. MPC protocols compute a function and reveal the output of the computation without revealing private inputs to any other participant. Early MPC work was based on two-party problems BID6, and subsequent work produced general approaches for any number of participants to participate in shared computation without learning private information BID0.Most MPC approaches involve modeling the desired algorithm as a boolean circuit (fixed-time algorithms expressed as a combination of AND, OR, and NOT logic gates). However, circuit-based approaches may require unrolling loops, introducing potential performance implications. Alternate approaches based on Oblivious RAM (Goldreich and Ostrovsky 1996) do not require this full unrolling though have other performance trade-offs. We use a circuit-based MPC approach in our scenario, though our privacy analysis and adaptive scheduling do not depend on the specifics of the underlying MPC approach. MPC has proven to be a powerful tool to enable a wide range of use cases. For example, private keys can be split among several hosts in order to reduce the number of hosts that must be compromised in order to obtain the key. A computation that requires the decryption operation can then be done under MPC between the hosts, ensuring that the whole key is never revealed in the clear BID0. MPC can also be used between companies in collaborative supplychain management, where the reluctance to share sensitive data (such as costing and capacities) can lead to sub-optimal production plans. The use of MPC allows for collaborative supply-chain planning without compromising sensitive data BID0.In our setting, each data owner provides their private input to a MPC circuit designed such that their inputs are not revealed to other participants. Then participants follow the protocol for the multi-party computation, which allows them to collectively compute the solution to the planning and scheduling optimization problem. In the end, only the final (schedule) is revealed to the relevant parties. Creating a single MPC circuit to solve the full optimized scheduling problem in a general way is not practical at this point in time. For this reason, our solution approach consists of a workflow that decomposes the scheduling task into the following sequence of steps. Step A. Collect relevant inputs:• Aid Provider: determine ports that can be reached by each ship before the deadline D • Aid Recipient: select ports to which aid will be acceptedStep B. Determine ports that satisfy joint ship/harbor physical compatibility constraints:• Aid Provider ship draft is compatible with the harbor depth in the port • Aid Recipient port has sufficient offload capacityStep C. Determine all berths within each feasible port fromStep B that satisfy joint ship/berth compatibility and berth availability constraints:• Aid Provider ship fits within the berth For a Ship to be scheduled to dock at a given Docking Time at a Berth belonging to a Port, the following must hold: DISPLAYFORM0 • ship-earliest-arrival ≤ deadline Berth in Port• ship-length ≤ berth-length Docking Time at Berth• ship-earliest-arrival ≤ docking-time ≤ deadline • berth-availability (pairs of berth-occupied-start and berth-occupied-end) for berth at docking-time Step D. Schedule the aid ships across possible ship-portberth-arrival-time options (from Step C) in accordance with the optimization goal of load-balancing across ports. Step A is a simple information gathering/filtering task performed locally by individual participants. Step B requires computation over private inputs from both the Aid Provider and the Aid Recipient. Steps C and D combine private inputs with intermediate from earlier steps. For these reasons, we use three secure multi-party circuits within our workflow:• A two-party circuit for the physical compatibility test inStep B (Aid Provider and Aid Recipient) • A two-party circuit for the viability test in Step C (Aid Provider and Aid Recipient) • An N-party circuit for optimizing berth allocation in Step D (all N Aid Providers)The circuits for Steps B and C are straightforward but must be run for each possible ship/port (Step B) and ship/berth (Step C) combination. The circuit for Step D is more complex. Because it is not possible to implement a truly optimized solution in the MPC circuit model, we instead opted for the following greedy approach to load balancing across ports.1. Initialization: set the list of ship-port-berth solutions to be empty and the working set of options to be the set of shipport-berth-unload time entries from Step C 2. Select a port P with the fewest number of assignments and for which there remain options 3. Select earliest ship-port-berth-unload time entry for P and add to the solutions list 4. Remove from the set of options all entries for the ship and berth selected in Step 3 5. Repeat steps 2-4 until no more ship-port-berth solutions remain 6. Output the solution listWe use the Lumen agent technology (described in (Myers et al. 2011)) to provide an adaptive workflow capability for executing this scheduling process. Although we depict only one workflow here, more generally our adaptive workflow capability draws from a library of alternative approaches to a range of aid distribution scheduling problems, enabling solution approaches to be matched to the specifics of a given situation. For example, we have defined alternative workflows that embed different scheduling and optimization strategies and that make use of secure multi-party computation in different ways as a means of investigating tradeoffs between efficiency and privacy. In this section we describe what is involved in using QIF to reason about questions of how private data can be inferred from the of a computation. The "Millionaires' Problem" BID6 ) is an early usecase for multi-party computation in which two people, Alice and Bob, use MPC to securely determine which of them is richer without revealing their actual wealth to each other. But while they don't learn the other's precise values, they do learn something about each other. The nature of the relationship between the revealed output and the private inputs determines how much one may infer about the inputs from the . For example, if Alice and Bob use MPC to instead compute the arithmetic mean of their net worths, Alice could use the input she provided and the ing mean from the computation to solve for Bob's exact net worth. Alice may not be able to extract Bob's input via examination of the MPC circuit itself, but MPC cannot change the relationship between inputs and output that exists outside the circuit. In our Aid Delivery scenario, private "inputs" may involve sensitive capability and operational details. As the scenarios increase in scope and complexity, it becomes difficult to reason informally about what an adversary may be able to infer.1 We use Quantitative Information Flow (QIF) to address these concerns by characterizing an adversary's ability to make these inferences based on information-theoretic bounds on the relationship between and private inputs. A QIF analysis begins by transforming a program (such as our aid distribution workflow) into a model that represents the relationship between the (private) inputs and outputs. Specifically, these models are based on informationtheoretic channels, which we use to construct a mapping from prior 2 distributions on private data to distributions on posterior distributions BID0 Smith 2009 ).We use these models to support "predictive-mode" adaptive workflows in which we reason about what an adversary could learn from any possible set of private inputs, as well as "posterior-mode" in which we determine how much an adversary may learn from some specific concrete . We can construct games in which we quantify the adversary's inference capability by measuring how their chances of "winning" (i.e., guessing a piece of private data) improve given additional information. We call the probability that an adversary can with one chance correctly guess a piece of private data the vulnerability of that variable. In the "prior game," the defender picks a private input value by sampling from the prior distribution of possible input values. The adversary makes their best guess of the defender's input based only on their knowledge of the prior distribution of possible input values. The prior vulnerability 1 In this work, an'adversary' is a so-called honest-but-curious actor who attempts to infer the values of private data by observing the public of each step in the workflow. Adversaries could be other coalition partners or an unrelated third party that has access to the computational framework.2 A'prior' can be thought of as the initial set of beliefs that an adversary has about a system. An adversary may have a prior over the lengths of naval ships (e.g., between 10 and 1,500 feet long), or the possible locations of the ships. If an adversary has no reason to believe one value is more likely than any other then the prior is a uniform distribution over the space of secrets, hence it is known as a uniform prior.is the probability the adversary will correctly guess the defender's private input (without any additional information).In the "posterior game," the defender again picks a private input, but then performs the computation using that input and shares (only) the with the adversary. We also assume the adversary has full knowledge of how the inputs relate to outputs in the computation (e.g., could construct an identical channel-based model of the computation). Here the chance the adversary correctly guesses the defender's private input is called the posterior vulnerability. To summarize, the prior game is how likely an adversary is to guess the private input before seeing the of the computation, the posterior game is how likely the adversary is to guess the private input after seeing the specific output of the computation. Vulnerability as a metric of risk is appealing because of its relatively intuitive nature. The caveat to vulnerability as a metric is that it is extremely sensitive to the chosen prior belief. Therefore, care must be taken in deciding upon a prior in order to assure that the metric reflects reality as much as possible. Our predictive-mode adaptation strategy is based on QIF predictive leakage, which attempts to predict how much will be leaked about the private data before the computation is run on the concrete private values. We can also approximate predictive leakage using Monte Carlo simulation, permitting the use of these metrics even in scenarios where operational requirements make it infeasible to complete the precise predictive leakage within a desired time frame. Incorporating predictive leakage metrics into our workflows enables identification of potentially higher-risk situations where it may be preferable to adapt or halt the workflow (based on some policy) rather than participating in a computation that may reveal too much. Our posterior-mode adaptation strategy is based on QIF dynamic leakage, which takes into account the actual of a computation. As opposed to the predictive leakage's assessments, that average all possible private input values, dynamic leakage enables our workflows to incorporate more accurate assessments of what was actually revealed in the specific ongoing workflow. As discussed previously, the HADR aid delivery problem provides a strong foundation for analyzing adaptive workflows in a privacy-aware setting. In this section we describe how the notion of vulnerability can be used to inform the cooperating parties of risk to their private data. To begin our discussion we first describe a simplified version of the HADR aid delivery problem involving a single ship, S, and a single port, P. In this simplified scenario, the only private data is S's location (ship_loc). We assume that other relevant data (such as ship maximum speed) is known publicly.def reachable(deadline): dist = distance(ship_loc, port_loc) return (dist <= (max_speed * deadline))Figure 2: The pseudo-code for a simple queryThe channel is "is S able to reach port P within d hours?". As pseudo-code we write this channel as a function of d, over some global set of private (ship_loc) and public (port_loc, max_speed) variables. Prior Belief In our simplified scenario an adversary who wants to determine S's private data will have some prior belief about that data. In this case, the prior belief will be some area of the ocean where S could be located. This may or may not be informed by other information available to the adversary. While QIF can be calculated over non-uniform priors, in this exposition we assume uniform priors for the sake of simplicity. Figure 3a shows a graphical representation of a possible prior for the simplified scenario. Because our prior is uniform, the adversary's chance of guessing the ship's position (the prior vulnerability, V prior), is simply V prior = 1 number of possible locations in the prior This makes intuitive sense: under a uniform prior, the adversary's likelihood of winning the prior game is equivalent to choosing a point out of the prior at random. Posterior Belief When ship S, cooperating in the simplified scenario, responds to the reachable query in Figure 2, the adversary is able to observe the output of the channel. This observation allows the adversary to rule out entire sections of the space of possible locations. If the of the query is True, then the adversary can infer that the location of S is within a circle of radius r, where r is the distance that the ship can travel at max_speed with deadline amount of time. Inversely, if the of the query is False, then the adversary could infer that the ship must be outside of the same circle. Observing False is the circumstance illustrated in Figure 3b.The important point is to see that regardless of the of the query, the adversary may be able to infer rule out subsets of possible values for the private data. The probability the adversary has of guessing the ship's position after seeing the channel output (the posterior vulnerability, V post) is simply V post = 1 number of possible locations in the posterior Further Queries Because cooperation is the goal, it is likely that the coordinator of our simple scenario will need to query S multiple times. This is particularly true if the initial query's was False. The coordinator may query S with reachable(d2), where d2 is a new, longer, deadline. If the is then True then the adversary is able to infer that S resides within the ring formed by two overlapping circles: an inner circle with radius r, described above, and a wider circle with radius r2, where r2 is the distance that S is able to travel at max speed in time d2. This circumstance is illustrated in Figure 3c.Worst-Case Simple Scenario If we add another port, P 2, to the simple scenario above, but keep all other details the same, it may be possible for S's location to be determined within a very small tolerance. If the of reachable is True for both P and P 2, then (using the same process as above) the adversary would intersect the appropriate circles. The smaller the intersection, the more the adversary knows about the ship's position. The simplified scenario above only has one piece of private data: the ship's position. As a reminder, FIG0 shows the private data in the full HADR scenario. In this section we present some from an analysis over a model of the full scenario, using a single ship (Ship #9) as a running example. Prior Vulnerability As discussed above, an adversary has some notion of the possible values that a secret can take on even before running any computations over that secret data. This is referred to as the Prior. When analyzing a system, the analyst must choose appropriate ranges for the private data in question. For some variables this may be straightforward (e.g., the ocean sector in question when deciding on a prior for a ship's location), for others it may depend on domain knowledge (e.g., the appropriate range for naval ship drafts).We can visualize the prior vulnerability over location easily. Figure 5 shows several aspects of the HADR scenario. The boundaries of the map in the image are the boundaries of the prior belief over ship position. It is important to remind ourselves that the choice of prior (part of which is determined by the geographic area under consideration) affects the vulnerability metric significantly. Increasing the area decreases the prior vulnerability, while decreasing the area increases the prior vulnerability. Because the prior models the adversary's view of the world, it should be constructed carefully. Predictive Vulnerability for Steps A+B In resourceconstrained systems, it is useful to be able to predict how much of a given resource would be used if a certain action were to be executed. Private data can similarly be viewed as a form of resource for which predictive analysis can be applied to assess the impact of planned or possible actions. The ability to predict the future vulnerability can be implemented in several ways, the practicality and efficiency of different methods depends heavily on the constraints of the system, imposed by the state-space and the adversarial model.(a) The initial prior belief: no reason to believe the ship is in one location over any other.(b) Posterior belief after observing a False: The ship must be outside the distance of the reachable query.(c) If a subsequent query for a further distance returns True the ship must be within the newly formed ring. Figure 3: The effects of observing query on a prior belief over the simplified scenario. In our system we have chosen an approximate method that enables an analyst to calculate a histogram over the possible vulnerability outcomes without any access to the private data. This method uses Monte Carlo simulation to run our analysis over randomly sampled points from the space of possible private data values. FIG1 shows the of a Monte Carlo simulation of the predictive vulnerability of running Steps A+B (using 13788 samples 4). The vertical dotted line shows the median value (6.847603e−6%) of all the samples. This method of approximation provides analysts with various options. In a scenario where the preservation of privacy is paramount, the analyst may focus on the right-hand side of the figure, where the potential vulnerability is higher, 0.03039514%, though still unlikely. This method also adapts well to use cases where an analyst may have access to some of the private data. For example, an analyst for one of the coalition partner nations might have access to that nation's private data, but not the private data of the ports. Using this same method of sampling from the space of (unknown) private data, such an analyst would be able to approximate the future vulnerability of their data if they were to respond to a query. Posterior Vulnerability for Steps A+B Once a step is taken, we can calculate the posterior vulnerability of the private data. Unlike the predictive analysis in the previous section, this analysis is'exact' in that the real vulnerability cannot be more than what the analysis reports. Ship #9 Position 6.847603e−6%In this case, the posterior vulnerability coincides with the median of the predictive vulnerability. This is not too surprising as the median was also the most likely value by a significant margin. Predictive Vulnerability for Step C Figure 6 shows the Monte Carlo simulation for vulnerability of a ship after Step C is completed. 5 The median predicted value in this instance, 6.644298e−4%, is two orders of magnitude higher than the vulnerability after Steps A+B alone. This makes intuitive sense as much more information is revealed after Step C that can be used to infer a ship's private data. Unsurprisingly, the maximum sampled predictive vulnerability is also substantially higher: 0.2012072%.Posterior Vulnerability for Steps C As with the posterior vulnerability for Steps A+B, the posterior vulnerability forStep C is based on a sound analysis using the real of the workflow step, and are not simulated as in the predictive vulnerability. Ship #9 Position 2.049806e−4%In the case of Step C, the posterior vulnerability for Ship #9 is lower than the median of the predictive (6.644298e−4%). From an analyst's perspective, this could mean that Ship #9 has revealed even less about its private data than the'average' ship would in this scenario. Step D has no meaningful consequences on the vulnerability of the private data for any stakeholder in the HADR scenario if the of Step C has been observed by the adversary. The reason for this is that Step D's algorithm can be computed completely from the of previous steps in the workflow, i.e. it does not require the values of the private data directly. Interestingly, this point reinforces an important aspect of QIF analysis: even though Step C's was computed with private data, the vulnerability metrics from the QIF analysis of Step C take The vulnerability assessment computed by the QIF capability provides insights to data owners as to the security of private information that they wish to protect. We exploit these insights within our workflow manager to adapt the scheduling process in order to ensure adherence to privacy objectives. More specifically, we use the QIF capability in two ways: in a predictive mode to estimate the amount of leakage associated with potentially performing a particular query or task and in a posterior mode to track actual leakage based on the specific values that a query or task returns. When executing a particular task our workflow manager invokes the predictive mode to estimate leakage. If the estimate of aggregate leakage for designated private data does not exceed set thresholds, then the workflow proceeds and the posterior leakage analysis is invoked to determine actual leakage values. If the estimate does exceed the threshold then the workflow is either terminated or (if possible) modified via a remediation strategy to keep leakage below the threshold. The idea behind a remediation strategy is to modify the problem or state in ways that will likely reduce impacts on private data. For example, our aid delivery problem requires computing the reachability of a port by a given deadline. Knowing that a given ship can reach the port by that deadline reveals information about the combination of ship position and maximum speed. One simple remediation strategy is to postpone the deadline, which will reveal less information about the position and speed values (e.g., the fact that a ship can reach a port by a given deadline reveals something about the lower bound for its max-speed; a later deadline introduces greater uncertainty as to what that speed might be by decreasing that lower bound). Our Lumen-based adaptive workflow engine includes such remediation strategies to enable adaptivity based on QIF predictive analyses. Privacy thresholds are implemented using an existing policy framework within Lumen that was developed previously to enable users to impose boundaries on the behaviors of autonomous agents BID2. The privacy policies have the general form:Keep below <percentage> the probability of knowing <private-data> within <tolerance> Below we show two examples used in the system, one for the Aid Provider nation and one for the Aid Recipient nation. Aid Provider Sample Policy:"Keep below 10 % the probability of knowing the location of my ships within 50 NMs"Aid Recipient Sample Policy: "Keep below 20 % the probability of knowing the port harbor depth within 40 feet" FIG3 shows sample vulnerability assessments for two types of private data (max-speed, location) for a select set of ships belonging to an individual Aid Provider nation. The top image shows the initial vulnerabilities of the data, prior to performing any computations; the bottom image shows the vunlerabilities after workflow completion. The display shows the QIF-derived vulnerability level as a colored bar representing the adversary's likelihood of guessing the private data within the specified tolerance. The vertical line bisecting the display for each piece of private data marks the policy-prescribed threshold of acceptability for the vulnerability. We note that the initial vulnerabilities for the ship locations are non-zero but so small as to not be perceptible in the image. Here, we consider two avenues for future work. Analogs can be drawn between our use of the QIF analysis to predict and track vunlerabilities of private data within the scheduling workflow and prior work on estimating resource usage in workflows (Morley, Myers, and YorkeSmith 2006). Although we currently consider individual actions incrementally as the workflow executes, we envision performing predictive vulnerability assessments of entire workflows prior to execution, to enable informed choices about alternative approaches before any information usage has occurred. Generating useful assessments will require predictive QIF techniques that consider cases beyond worstcase leakage, whose inherent pessimism can make them of limited value for certain vulnerability assessment tasks. Longer term, such analyses could also potentially open the door to using first-principles planning techniques to synthesize privacy-aware workflows on an as needed basis that are tailored to the specifics of a given task and privacy requirements. The scalability of QIF techniques can be an issue for systems where there are complex relationships between sets of variables. Some work has been done on attaining scalability by enhancing static analysis techniques with approximations that speed up analysis with probabilistic bounds on certainty BID5. However, there is still further work required before the QIF analysis of arbitrary channels could scale to use cases such as the end-to-end HADR scenario considered in this paper. In particular, the methods described in this paper utilize bespoke analyses for the channels under consideration, providing a more scalable approach at the cost of generality. One future direction may be to design Domain Specific Languages that enable description of a scenario and its analysis in tandem. A key challenge facing coalitions is how to collaborate without releasing information that could jeopardize national (or organizational) interests. In this paper, we consider this challenge for a realistic scheduling problem tied to aid delivery. Our work makes several contributions. First, we show how state-of-the-art secure multi-party computation can be used to safeguard private information with an overall distributed scheduling solution to the aid delivery problem. A second contribution relates to the use of quantitative information flow (QIF): even with secure multi-party computation, scheduling outputs can reveal information about coalition members' private data. We show how QIF can be applied to assess the vulnerability of providate data for both prospective (i.e., where are not known) and actual (i.e., where are known) computations. As a third contribution, these assessments can be used to adapt the scheduling algorithm to ensure it remains within accepted vulnerability thresholds established by data owners.

Privacy can be thought about in the same way as other resources in planning

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems


Question: Represent the article for finding a single-sentence TLDR

Question: The challenge of learning disentangled representation has recently attracted much attention and boils down to a competition. Various methods based on variational auto-encoder have been proposed to solve this problem, by enforcing the independence between the representation and modifying the regularization term in the variational lower bound. However recent work by has demonstrated that the proposed methods are heavily influenced by randomness and the choice of the hyper-parameter. This work is built upon the same framework in Stage 1 , but with different settings; to make it self-contained, we provide this manuscript, which is unavoidably very similar to the report for Stage 1. In detail, in this work, instead of designing a new regularization term, we adopt the FactorVAE but improve the reconstruction performance and increase the capacity of network and the training step. The strategy turns out to be very effective in achieving disentanglement. The great success of unsupervised learning heavily depends on the representation of the feature in the real-world. It is widely believed that the real-world data is generated by a few explanatory factors which are distributed, invariant, and disentangled . The challenge of learning disentangled representation boils down into a competition 1 to build the best disentangled model. The key idea in disentangled representation is that the perfect representation should be a one-to-one mapping to the ground truth disentangled factor. Thus, if one factor changed and other factors fixed, then the representation of the fixed factor should be fixed accordingly, while others' representation changed. As a , it is essential to find representations that (i) are independent of each other, and (ii) align to the ground truth factor. Recent line of works in disentanglement representation learning are commonly focused on enforcing the independence of the representation by modifying the regulation term in the and FactorVAE . See Appendix A for more details of these model. To evaluate the performance of disentanglement, several metrics have been proposed, including the FactorVAE metric , Mutual Information Gap (MIG) , DCI metric , IRS metric , and SAP score . However, one of our findings is that these methods are heavily influenced by randomness and the choice of the hyper-parameter. This phenomenon was also discovered by. Therefore, rather than designing a new regularization term, we simply use FactorVAE but at the same time improve the reconstruction performance. We believe that, the better the reconstruction, the better the alignment of the ground-truth factors. Therefore, the more capacity of the encoder and decoder network, the better the would be. Furthermore, after increasing the capacity, we also try to increase the training step which also shows a significant improvement of evaluation metrics. The final architecture of FactorVAE is given in Figure 1. Note that, this work is built upon the same framework in stage 1 , but with different settings; to make it self-contained, we provide this manuscript, which is unavoidably very similar to the report for Stage 1. Overall, our contribution can be summarized as follow: we found that the performance of the reconstruction is also essential for learning disentangled representation, and we achieve state-of-the-art performance in the competition. In this section, we explore the effectiveness of different disentanglement learning models and the performance of the reconstruction for disentangle learning. We first employ different kinds of variational autoencoder including BottleneckVAE, AnneledVAE, DIPVAE, BetaTCVAE, and BetaVAE with 30000 training step. Second, we want to know whether the capacity plays an important role in disentanglement. The hypothesis is that the larger the capacity, the better reconstruction can be obtained, which further reinforces the disentanglement. In detail, we control the number of latent variables. In this section, we present our experiment in stage 1 and stage 2 of the competition. We first present the performance of different kinds of VAEs in stage 1, which is given in Table 1. It shows that FactorVAE achieves the best when the training step is 30000. In the following experiment, we choose FactorVAE as the base model. Then, as shown in Table 2, we increase the step size and we find that the best was achieved at 1000k training steps. The experiment in this part may not be sufficient, but it still suggests the fact that the larger the capacity is, the better the disentanglement performance. Since we increase the capacity of the model, it is reasonable to also increase the training steps at the same time. Furthermore, as shown in Table 3, using sufficient large training step (≥ 800k), we investigate the effectiveness of the number of latent variables. This experiment is performed in stage 2 and suggests that the FactorVAE and the DCI metric are positive as the latent variables increase, while the other metrics decrease. The best in the ranking is marked as bold, which suggests that we should choose an appropriate number of latent variables. In this work, we conducted an empirical study on disentangled learning. We first conduct several experiments with different disentangle learning methods and select the FactorVAE as the base model; and second we improve the performance of the reconstruction, by increasing the capacity of the model and the training step. Finally, our appear to be competitive. (VAE) , a generative model that maximize the following evidence lower bound to approximate the intractable distribution p θ (x|z) using q φ (z|x), where q φ (z|x) denote Encoder with parameter φ and p θ (x|z) denote Decoder with parameter θ. As shown in Table 4, all the lower bound of variant VAEs can be described as Reconstruction Loss+ Regularization where all the Regularization term and the hyper-parameters are given in this table.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the scientific article to find a short TLDR

Question: We propose a generative adversarial training approach for the problem of clarification question generation. Our approach generates clarification questions with the goal of eliciting new information that would make the given context more complete. We develop a Generative Adversarial Network (GAN) where the generator is a sequence-to-sequence model and the discriminator is a utility function that models the value of updating the context with the answer to the clarification question. We evaluate on two datasets, using both automatic metrics and human judgments of usefulness, specificity and relevance, showing that our approach outperforms both a retrieval-based model and ablations that exclude the utility model and the adversarial training. A goal of natural language processing is to develop techniques that enable machines to process naturally occurring language. However, not all language is clear and, as humans, we may not always understand each other BID10; in cases of gaps or mismatches in knowledge, we tend to ask questions BID9. In this work, we focus on the task of automatically generating clarification questions: questions that ask for information that is missing from a given linguistic context. Our clarification question generation model builds on the sequence-to-sequence approach that has proven effective for several language generation tasks BID37 BID39 BID5. Unfortunately, training a sequence-to-sequence model directly on context/question pairs yields generated questions that are highly generic 1, corroborating a common finding in dialog systems BID17. Our goal is to be able to generate questions that are useful and specific. To achieve this, we begin with a recent observation of BID30, who considered the task of question reranking: the system should learn to generate clarification questions whose answers have high utility, which they defined as the likelihood that this question would lead to an answer that will make the context more complete (§2.3). Inspired by this, we construct a question generation model that first generates a question given a context, and then generates a hypothetical answer to that question. Given this (context, question, answer) tuple, we train a utility calculator to estimate the usefulness of this question. We then show that this utility calculator can be generalized using ideas for generative adversarial networks BID8 for text BID40, wherein the utility predictor plays the role of the "discriminator" and the question generator is the "generator" (§2.2), which we train using the MIXER algorithm BID29.We evaluate our approach on two question generation datasets: for posts on Stack Exchange and for Amazon product descriptions (Figure 1). Using both automatic metrics and human evaluation, we demonstrate that our adversarially trained model generates a more diverse set of questions than all the baseline models. Furthermore, we find that although all models generate questions that are relevant to the context at hand, our adversarially-trained model generates questions that are more specific to the context. Our goal is to build a model that, given a context, can generate an appropriate clarification question. As a running example, we will use the Amazon setting: where the dataset consists of (context, question, answer) triples where the context is the product description, question is clarification question about that product that (preferably) is not already answered in the description and answer is the seller's (or other users') reply to the question. Representationally, our question generator is a standard sequence-to-sequence model with attention (§2.1). The learning problem is: how to train the sequence-to-sequence model to produce good question. An overview of our training setup is shown in FIG1. Given a context, our question generator outputs a question. In order to evaluate the usefulness of this question, we then have a second sequence-to-sequence model called the "answer generator" that generates a hypothetical answer based on the context and the question (§2.5). This (context, question and answer) triple is fed into a UTILITY calculator, whose initial goal is to estimate the probability that this question/answer pair is useful in this context (§2.3). This UTILITY is treated as a reward, which is used to update the question generator using the MIXER BID29 algorithm (§2.2). Finally, we reinterpret the answer-generator-plus-utility-calculator component as a discriminator for differentiating between true (context, question, answer) triples and synthetic triples (§ 2.4), and optimize this adversarial objective using MIXER. We use a standard attention based sequence-to-sequence model BID21 for our question generator. Given an input sequence (context) c = (c 1, c 2, ..., c N), this model generates an output sequence (question) q = (q 1, q 2, ..., q T). The architecture of this model is an encoder-decoder with attention. The encoder is a recurrent neural network (RNN) operating over the input word embeddings to compute a source context representationc. The decoder uses this source representation to generate the target sequence one word at a time: DISPLAYFORM0 In Eq 1,h t is the attentional hidden state of the RNN at time t and W s and W c are parameters of the model (details in Appendix A). The predicted token q t is the token in the vocabulary that is assigned the highest probability using the softmax function. The standard training objective for sequence-tosequence model is to maximize the log-likelihood of all (c, q) pairs in the training data D which is equivalent to minimizing the loss, DISPLAYFORM1 2.2 TRAINING THE GENERATOR TO OPTIMIZE QUESTION UTILITYTraining sequence-to-sequence models for the task of clarification question generation (with context as input and question as output) using maximum likelihood objective unfortunately leads to the generation of highly generic questions, such as "What are the dimensions?" when asking questions about home appliances. This issue has been observed in dialog generation as well BID17. Recently BID30 observed that usefulness of a question can be better measured as the utility that would be obtained if the context were updated with the answer to the proposed question. We use this observation to define a UTILITY based reward function and train the question generator to optimize this reward. We train the UTILITY reward to predict the likelihood that a question would generate an answer that would increase the utility of the context by adding useful information to it (see §2.3 for details).Similar to optimizing metrics like BLEU and ROUGE, this UTILITY function also operates on discrete text outputs, which makes optimization difficult due to non-differentiability. A successful recent approach dealing with the non-differentiability while also retaining some advantages of maximum likelihood training is the Mixed Incremental Cross-Entropy Reinforce BID29 algorithm (MIXER). In MIXER, the overall loss L is differentiated as in REINFORCE BID38: DISPLAYFORM2 where y s is a random output sample according to the model p θ, where θ are the parameters of the network. We then approximate the expected gradient using a single sample q s = (q DISPLAYFORM3 In REINFORCE, the policy is initialized random, which can cause long convergence times. To solve this, MIXER starts by optimizing maximum likelihood and slowly shifts to optimizing the expected reward from Eq 3. For the initial ∆ time steps, MIXER optimizes L mle and for the remaining (T − ∆) time steps, it optimizes the external reward. In our model, we minimize the UTILITY-based loss L max-utility defined as: DISPLAYFORM4 where r(q p) is the UTILITY based reward on the predicted question and r(q b) is a baseline reward introduced to reduce the high variance otherwise observed when using REINFORCE.In MIXER, the baseline is estimated using a linear regressor that takes in the current hidden states of the model as input and is trained to minimize the mean squared error (||r( DISPLAYFORM5 Instead we use a self-critical training approach BID32 where the baseline is estimated using the reward obtained by the current model under greedy decoding during test time. Given a (context, question, answer) triple, BID30 introduce a utility function UTILITY(c, q, a) to calculate the value of updating a context c with the answer a to a clarification question q. The inspiration for thier utility function is to estimate the probability that an answer would be a meaningful addition to a context, and treat this as a binary classification problem where the positive instances are the true (context, question, answer) triples in the dataset whereas the negative instances are contexts paired with a random (question, answer) from the dataset. The model we use is to first embed of the words in the context c, then use an LSTM (long-short term memory) BID12 to generate a neural representationc of the context by averaging the output of each of the hidden states. Similarly, we obtain a neural representationq and a of q and a respectively using question and answer LSTM models. Finally, a feed forward neural network F UTILITY (c,q,ā) predicts the usefulness of the question. The UTILITY function trained on true vs random samples from real data (as described in the previous section) can be a weak reward signal for questions generated by a model due to the large discrepancy between the true data and the model's outputs. In order to strengthen the reward signal, we reinterpret the UTILITY function (coupled with the answer generator) as a discriminator in an adversarial learning setting. That is, instead of taking the UTILITY calculator to be a fixed model that outputs the expected quality of a question/answer pair, we additionally optimize it to distinguish between true question/answer pairs and model-generated ones. This reinterpretation turns our model into a form of a generative adversarial network (GAN) BID8.A GAN is a training procedure for "generative" models that can be interpreted as a game between a generator and a discriminator. The generator is an arbitrary model g ∈ G that produces outputs (in our case, questions). The discriminator is another model d ∈ D that attempts to classify between true outputs and model-generated outputs. The goal of the generator is to generate data such that it can fool the discriminator; the goal of the discriminator is to be able to successfully distinguish between real and generated data. In the process of trying to fool the discriminator, the generator produces data that is as close as possible to the real data distribution. Generically, the GAN objective is: DISPLAYFORM0 where x is sampled from the true data distributionp, and z is sampled from a prior defined on input noise variables p z.Although GANs have been successfully used for image tasks, training GANs for text generation is challenging due to the discrete nature of outputs in text. The discrete outputs from the generator make it difficult to pass the gradient update from the discriminator to the generator. Recently, BID40 proposed a sequence GAN model for text generation to overcome this issue. They treat their generator as an agent and use the discriminator as a reward function to update the generative model using reinforcement learning techniques. Our GAN-based approach is inspired by this sequence GAN model with two main modifications: a) We use the MIXER algorithm as our generator (§2.2) instead of policy gradient approach; and b) We use the UTILITY function (§2.3) as our discriminator instead of a convolutional neural network (CNN).In our model, the answer is an latent variable: we do not actually use it anywhere except to train the discriminator. Because of this, we train our discriminator using (context, true question, generated answer) triples as positive instances and (context, generated question, generated answer) triples as the negative instances. Formally, our objective function is:LGAN DISPLAYFORM1 where U is the UTILITY discriminator, M is the MIXER generator,p is our data of (context, question, answer) triples and A is our answer generator. Question Generator. We pretrain our question generator using the sequence-to-sequence model §2.1 where we define the input sequence as the context and the output sequence as the question. This answer generator is trained to maximize the log-likelihood of all ([context+question], answer) pairs in the training data. Parameters of this model are updated during adversarial training. Answer Generator. We pretrain our answer generator using the sequence-to-sequence model §2.1 where we define the input sequence as the concatenation of the context and the question and the output sequence as the answer. This answer generator is trained to maximize the log-likelihood of all (context, question) pairs in the training data. Unlike the question generator, the parameters of the answer generator are kept fixed during the adversarial training. Discriminator. We pretrain the discriminator using (context, question, answer) triples from the training data. For positive instances, we use a context and its true question, answer and for negative instances, we use the same context but randomly sample a question from the training data (and use the answer paired with that random question). We base our experimental design on the following research questions:1. Do generation models outperform simpler retrieval baselines? 2. Does optimizing the UTILITY reward improve over maximum likelihood training? 3. Does using adversarial training improve over optimizing the pretrained UTILITY? 4. How do the models perform when evaluated for nuances such as specificity and usefulness? We evaluate our model on two datasets. The first is from StackExchange and was curated by BID30; the second is from Amazon, curated by BID22, and has not previously been used for the task of question generation. StackExchange. This dataset consists of posts, questions asked to that post on stackexchange.com (and answers) collected from three related subdomains on stackexchage.com (askubuntu, unix and superuser). Additionally, for 500 instances each from the tune and the test set, the dataset includes 1 to 5 other questions identified as valid questions by expert human annotators from a pool of candidate questions. This dataset consists of 61, 681 training, 7710 validation and 7709 test examples. Amazon. Each instance consists of a question asked about a product on amazon.com combined with other information (product ID, question type "Yes/No", answer type, answer and answer time).To obtain the description of the product, we use the metadata information contained in the amazon reviews dataset BID23. We consider at most 10 questions for each product. This dataset includes several different product categories. We choose the Home and Kitchen category since it contains a high number of questions and is relatively easy category for human based evaluation. This dataset consists of 19, 119 training, 2435 validation and 2305 test examples, and each product description contains between 3 and 10 questions (average: 7). We compare three variants (ablations) of our proposed approach, together with an information retrieval baseline: GAN-Utility is our full model which is a UTILITY function based GAN training (§ 2.4) including the UTILITY discriminator, a MIXER question generator and a sequence-tosequence based answer generator. Max-Utility is our reinforcement learning baseline with a pretrained question generator described model (§ 2.2) without the adversarial training. MLE is the question generator model pretrained on context, question pairs using maximum likelihood objective (§2.1). Lucene 3 is a TF-IDF (term frequency-inverse document frequency) based document ranking system which given a document, retrieves N other documents that are most similar to the given document. Given a context, we use Lucene to retrieve top 10 contexts that are most similar to the given context. We randomly choose a question from the 10 questions paired with these contexts to construct our Lucene baseline BID45. Experimental details of all our models are described in Appendix B. We evaluate initially with several automated evaluation metrics, and then more substantially based on crowdsourced human judgments. Automatic metrics include: Diversity, which calculates the proportion of unique trigrams 5 in the output to measure the diversity as commonly used to evaluate dialogue generation BID17; BLEU BID27, which evaluate n-gram precision between a predicted sentence and reference sentences; and METEOR BID1, which is similar to BLEU but includes stemmed and synonym matches when measuring the similarity between the predicted sequence and the reference sequences. Table 1: DIVERSITY as measured by the proportion of unique trigrams in model outputs. BLEU and METEOR scores using up to 10 references for the Amazon dataset and up to six references for the StackExchange dataset. Numbers in bold are the highest among the models. All for Amazon are on the entire test set whereas for StackExchange they are on the 500 instances of the test set that have multiple references. Human judgments involve showing contexts and generated questions to crowdworkers 6 and asking them to evaluate the questions along several axes. Roughly, we ask for the following five judgments for each question (exact wordings in Appendix C): Is it relevant (yes/no); Is it grammatical (yes/comprehensible/incomprehensible); How specific is it to this product (four options from "specific to only this product" to "generic to any product"); Does this question ask for new information not contained in the discription (completely/somewhat/no); and How useful is this question to a potential buyer (four options from "should be included in the description" to "useful only to the person asking"). For the last three questions, we also allowed a "not applicable" response in the case that the question was either ungrammatical or irrelevant. Table 1 shows the on the two datasets when evaluated according to automatic metrics. In the Amazon dataset, GAN-Utility outperforms all ablations on DIVERSITY, suggesting that it produces more diverse outputs. Lucene, on the other hand, has the highest DIVERSITY since it consists of human generated questions, which tend to be more diverse because they are much longer compared to model generated questions. This comes at the cost of lower match with the reference as visible in the BLEU and METEOR scores. In terms of BLEU and METEOR, there is inconsistency. Although GAN-Utility outperforms all baselines according to METEOR, the fully ablated MLE model has a higher BLEU score. This is because BLEU score looks for exact n-gram matches and since MLE produces more generic outputs, it is much more likely that it will match one of 10 references compared to the specific/diverse outputs of GAN-Utility, since one of those ten is highly likely to itself be generic. In the StackExchange dataset GAN-Utility outperforms all ablations on both BLEU and METEOR. Unlike in the Amazon dataset, MLE does not outperform GAN-Utility in BLEU. This is because the MLE outputs in this dataset are not as generic as in the amazon dataset due to the highly technical nature of contexts in StackExchange. As in the Amazon dataset, GAN-Utility outperforms MLE on DIVERSITY. Interestingly, the Max-Utility ablation achieves a higher DIVERSITY score than GAN-Utility. On manual analysis we find that Max-Utility produces longer outputs compared to GAN-Utility but at the cost of being less grammatical. Table 2 shows the numeric of human-based evaluation performed on the reference and the system outputs on 500 random samples from the test set of the Amazon dataset. 7 These overall show that the GAN-Utility model successfully generates the most specific questions, while being equally good at seeking new information and being useful to potential buyers. All approaches produce relevant, grammatical questions. All our models are all equally good at seeking new information, but are weaker than Lucene, which performs better according to new information but at Table 2: Results of human judgments on model generated questions on 500 sample Home & Kitchen product descriptions. The options described in §3.3 are converted to corresponding numeric range (as described in Appendix C). The difference between the bold and the non-bold numbers is statistically insignificant with p <0.001. Reference is excluded in the significance calculation. the cost of much lower specificity and slightly lower relevance. Our models are all equally good also at generating useful questions: their usefulness score is significantly better than both Lucene and Reference, largely because Lucene and Reference tend to ask questions that are more often useful only to the person asking the question, making them less useful for potential other buyers (see FIG3). Our full model, GAN-Utility, performs significantly better when measured by specificity to the product, which aligns with the higher DIVERSITY score obtained by GAN-Utility under automatic metric evaluation. Question Generation. Most previous work on question generation has been on generating reading comprehension style questions i.e. questions that ask about information present in a given text BID11 BID33 BID11 BID6. Outside reading comprehension questions, BID15 use crowdsourcing to generate question templates, BID19 use templated questions to help authors write better related work sections, BID24 introduced visual question answer tasking that focuses on generating natural and engaging questions about an image. BID25 introduced an extension of this task called the Image Grounded Conversation task where they use both the image and some initial textual context to generate a natural follow-up question and a response to that question. BID3 propose an active question answering model where they build an agent that learns to reformulate the question to be asked to a question-answering system so as to elicit the best possible answers. BID6 extract large number of question-answer pairs from community question answering forums and use them to train a model that can generate a natural question given a passage. Neural Models and Adversarial Training for Text Generation. Neural network based models have had significant success at a variety of text generation tasks, including machine translation BID0 BID21, summarization BID26 ), dialog (BID2 BID16 BID36, textual style transfer BID13 BID14 BID31 and question answering BID39 .Our task is most similar to dialog, in which a wide variety of possible outputs are acceptable, and where lack of specificity in generated outputs is common. We addresses Table 3 : Example outputs from each of the systems for a single product description this challenge using an adversarial network approach BID8, a training procedure that can generate natural-looking outputs, which have been effective for natural image generation BID4 . Due to the challenges in optimizing over discrete output spaces like text, BID40 introduced a Seq(uence)GAN approach where they overcome this issue by using RE-INFORCE to optimize. BID18 train an adversarial model similar to SeqGAN for generating next utterance in a dialog given a context. However, unlike our work, their discriminator is a binary classifier trained to distinguish between human and machine generated utterances. Finally, BID7 introduce an actor-critic conditional GAN for filling in missing text conditioned on the surrounding context. In this work, we describe a novel approach to the problem of clarification question generation. Given a context, we use the observation of BID30 that the usefulness of a clarification question can be measured by the value of updating the context with an answer to the question. We use a sequence-to-sequence model to generate a question given a context and a second sequenceto-sequence model to generate an answer given the context and the question. Given the (context, predicted question, predicted answer) triple we calculator the utility of this triple and use it as a reward to retrain the question generator using reinforcement learning based MIXER model. Further, to improve upon the utility function, we reinterpret it as a discriminator in an adversarial setting and train both the utility function and the MIXER model in a minimax fashion. We find that our adversarial training approach produces more diverse questions compared to both a model trained using maximum likelihood objective and a model trained using utility reward based reinforcement learning. There are several avenues of future work in this area. Following BID24, we could combine text input with image input to generate more relevant questions. Because some questions can be answered by looking at the product image in the Amazon dataset BID22, this could help generate more relevant and useful questions. As in most One significant research challenge in the space of free text generation problems when the set of possible outputs is large, is that of automatic evaluation BID20: in our we saw some correlation between human judgments and automatic metrics, but not enough to trust the automatic metrics completely. Lastly, integrating such a question generation model into a real world platform like StackExchange or Amazon to understand the real utility of such models and to unearth additional research questions. A DETAILS OF SEQUENCE-TO-SEQUENCE MODELIn this section, we describe the attention based sequence-to-sequence model introduced in §2.1 of the main paper. In Eq 1,h t is the attentional hidden state of the RNN at time t obtained by concatenating the target hidden state h t and the source-side context vectorc t, and W s is a linear transformation that maps h t to an output vocabulary-sized vector. The predicted token q t is the token in the vocabulary that is assigned the highest probability using the softmax function. Each attentional hidden stateh t depends on a distinct input context vectorc t computed using a global attention mechanism over the input hidden states as: DISPLAYFORM0 a nt h n DISPLAYFORM1 The attention weights a nt is calculated based on the alignment score between the source hidden state h n and the current target hidden state h t. In this section, we describe the details of our experimental setup. We preprocess all inputs (context, question and answers) using tokenization and lowercasing. We set the max length of context to be 100, question to be 20 and answer to be 20. Our sequence-to-sequence model (§ 2.1) operates on word embeddings which are pretrained on in domain data using Glove BID28. We use embeddings of size 200 and a vocabulary with cut off frequency set to 10. During train time, we use teacher forcing. During test time, we use beam search decoding with beam size 5. We use a hidden layer of size two for both the encoder and decoder recurrent neural network models with size of hidden unit set to 100. We use a dropout of 0.5 and learning ratio of 0.0001 In the MIXER model, we start with ∆ = T and decrease it by 2 for every epoch (we found decreasing ∆ to 0 is ineffective for our task, hence we stop at 2). In this section, we describe in detail the human based evaluation methodology introduced in §3.3 of the main paper.

represent the short summary for finding the corresponding scientific article (long)!

We propose an adversarial training approach to the problem of clarification question generation which uses the answer to the question to model the reward.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR:

Question: Pattern databases are the foundation of some of the strongest admissible heuristics for optimal classical planning. Experiments showed that the most informative way of combining information from multiple pattern databases is to use saturated cost partitioning. Previous work selected patterns and computed saturated cost partitionings over the ing pattern database heuristics in two separate steps. We introduce a new method that uses saturated cost partitioning to select patterns and show that it outperforms all existing pattern selection algorithms. A * search BID10 with an admissible heuristic BID23 ) is one of the most successful methods for solving classical planning tasks optimally. An important building block of some of the strongest admissible heuristics are pattern database (PDB) heuristics. A PDB heuristic precomputes all goal distances in a simplified state space obtained by projecting the task to a subset of state variables, the pattern, and uses these distances as lower bounds on the true goal distances. PDB heuristics were originally introduced for solving the 15-puzzle BID2 and have later been generalized to many other combinatorial search tasks (e.g., BID21 BID7 and to the setting of domainindependent planning BID3 .Using a single PDB heuristic of reasonable size is usually not enough to cover sufficiently many aspects of challenging planning tasks. It is therefore often beneficial to compute multiple PDB heuristics and to combine their estimates admissibly BID15 . The simplest approach for this is to choose the PDB with the highest estimate in each state. Instead of this maximization scheme, we would like to sum estimates, but this renders the ing heuristic inadmissible in general. However, if two PDBs are affected by disjoint sets of operators, they are independent and we can admissibly add their estimates BID19 BID7 . BID11 later generalized this idea by introducing the canonical heuristic for PDBs, which computes all maximal subsets of pairwise independent PDBs and then uses the maximum over the sums of independent PDBs as the heuristic value. Cost partitioning BID17 BID40) is a generalization of the independence-based methods above. It makes the sum of heuristic estimates admissible by distributing the costs of each operator among the heuristics. The literature contains many different cost partitioning algorithms such as zero-one cost partitioning BID4 BID11 ), uniform cost partitioning BID17, optimal cost partitioning BID17 BID16 BID18 BID25, posthoc optimization BID26 and delta cost partitioning BID6.In previous work BID34, we showed experimentally for the benchmark tasks from previous International Planning Competitions (IPC) that saturated cost partitioning (SCP) BID30 BID37 is the cost partitioning algorithm of choice for PDB heuristics. Saturated cost partitioning considers an ordered sequence of heuristics. Iteratively, it gives each heuristic the minimum amount of costs that the heuristic needs to justify all its estimates and then uses the remaining costs for subsequent heuristics until all heuristics have been served this way. Before we can compute a saturated cost partitioning over pattern database heuristics, we need to select a collection of patterns. The first domain-independent automated pattern selection algorithm is due to BID3. It partitions the state variables into patterns via best-fit bin packing. BID5 later used a genetic algorithm to search for a pattern collection that maximizes the average heuristic value of a zero-one cost partitioning over the PDB heuristics. BID11 proposed an algorithm that performs a hill-climbing search in the space of pattern collections (HC). HC evaluates a collection C by estimating the search effort of the canonical heuristic over C based on a model of IDA * runtime BID20. BID8 presented the Complementary PDBs Creation (CPC) method, that combines bin packing and genetic algorithms to create a pattern collection minimizing the estimated search effort of an A * search BID22. BID28 repeatedly compute patterns using counterexample-guided abstraction refinement (CEGAR): starting from a random goal variable, their CEGAR algorithm iteratively finds solutions in the corresponding projection and executes them in the original state space. Whenever a solution cannot be executed due to a violated precondition, it adds the missing precondition variable to the pattern. Finally, BID26 systematically generate all interesting patterns up to a given size X (SYS-X). Experiments showed that cost-partitioned heuristics over SYS-2 and SYS-3 yield accurate estimates BID26 BID34, but using all interesting patterns of larger sizes is usually infeasible. We introduce SYS-SCP, a new pattern selection algorithm based on saturated cost partitioning that potentially considers all interesting patterns, but only selects useful ones. SYS-SCP builds multiple pattern sequences that together form the ing pattern collection. For each sequence σ, it considers the interesting patterns in increasing order by size and adds a pattern P to σ if P is not part of an earlier sequence and the saturated cost partitioning heuristic over σ plus P is more informative than the one over σ alone. We consider optimal classical planning tasks in a SAS + -like notation BID1 and represent a planning task Π as a tuple V, O, s 0, s. Each variable v in the finite set of variables V has a finite domain dom(v). A partial state s is defined over a subset of variables vars(s) ⊆ V and maps each v ∈ vars(s) to a value in dom(v), written as s [v]. We call the pair v, s [v] an atom and interchangeably treat partial states as mappings from variables to values or as sets of atoms. If vars(s) = V, we call s a state. We write S(Π) for the set of all states in Π.Each operator o in the finite set of operators O has a precondition pre(o) and an effect eff(o), both of which are partial states, and a cost cost(o) ∈ R Transition systems assign semantics to planning tasks. Definition 1 (Transition Systems). A transition system T is a labeled digraph defined by a finite set of states S(T), a finite set of labels L(T), a set T (T) of labeled transitions s − → s with s, s ∈ S(T) and ∈ L(T), an initial state s 0 (T), and a set S (T) of goal states. A planning task Π = V, O, s 0, s induces a transition system T with states S(Π), labels O, transitions {s DISPLAYFORM0 Separating transition systems from cost functions allows us to evaluate the same transition system under different cost functions, which is important for cost partitioning. A cost function for transition system T is a function cost : L(T) → R ∪ {−∞, ∞}. It is finite if −∞ < cost < ∞ for all labels. It is nonnegative if cost ≥ 0 for all labels. We write C(T) for the set of all cost functions for T.Note that we assume that the cost function of the planning task is non-negative and finite, but as in previous work we allow negative BID25 and infinite costs BID32 in cost partitionings. The generalization to infinite costs is necessary to cleanly state some of our definitions. Definition 3 (Weighted Transition Systems). A weighted transition system is a pair T, cost where T is a transition system and cost ∈ C(T) is a cost function for T.The cost of a path π = s DISPLAYFORM0 It is ∞ if the sum contains both +∞ and −∞. If s n is a goal state, π is called a goal path for s 0.Definition 4 (Goal Distances and Optimal Paths). The goal distance of a state s ∈ S(T) in a weighted transition system T, cost is defined as inf π∈Π (T,s) cost(π), where Π (T, s) is the set of goal paths from s in T. (The infimum of the empty set is ∞.) We write h * DISPLAYFORM1 Optimal classical planning is the problem of finding an optimal goal path from s 0 or showing that s 0 is unsolvable. We use heuristics to estimate goal distances BID23. DISPLAYFORM2 Cost partitioning makes adding heuristics admissible by distributing the costs of each operator among the heuristics. Definition 6 (Cost Partitioning). Let T be a transition system. A cost partitioning for a cost function cost ∈ C(T) is a tuple cost 1,..., cost n ∈ C(T) n whose sum is bounded by cost: DISPLAYFORM3 n over the heuristics h 1,..., h n for T induces the cost-partitioned heuristic DISPLAYFORM4. If the sum contains +∞ and −∞, it evaluates to the leftmost infinite value. One of the cost partitioning algorithms from the literature is saturated cost partitioning BID31. It is based on the insight that we can often reduce the amount of costs given to a heuristic without changing any heuristic estimates. Saturated cost functions formalize this idea. Definition 7 (Saturated Cost Function). Consider a transition system T, a heuristic h for T and a cost function cost ∈ C(T). A cost function scf ∈ C(T) is saturated for h and cost if 1. scf ≤ cost for all labels ∈ L(T) and 2. h(scf, s) = h(cost, s) for all states s ∈ S(T).A saturated cost function scf is minimal if there is no other saturated cost function scf for h and cost with scf ≤ scf for all labels ∈ L(T).Whether we can efficiently compute a minimal saturated cost function depends on the type of heuristic. In earlier work BID31, we showed that this is possible for explicitly-represented abstraction heuristics BID12, which include PDB heuristics. Definition 8 (Minimum Saturated Cost Function for Abstraction Heuristics). Let T, cost be a weighted transition system and h an abstraction heuristic for T with abstract transition system T. The minimum saturated cost function mscf for h and cost is DISPLAYFORM5 Given a sequence of abstraction heuristics, the saturated cost partitioning algorithm iteratively assigns to each heuristic only the costs that the heuristic needs to preserve its estimates and uses the remaining costs for subsequent heuristics. Definition 9 (Saturated Cost Partitioning). Consider a transition system T and a sequence of abstraction heuristics DISPLAYFORM6 receives a cost function rem and returns the minimum saturated cost function for h i and rem. The saturated cost partitioning cost 1,..., cost n of a function cost ∈ C(T) over H is defined as: DISPLAYFORM7 where the auxiliary cost functions rem i represent the remaining costs after processing the first i heuristics in H. We write h SCP H for the saturated cost partitioning heuristic over the sequence of heuristics H. In this work, we compute saturated cost partitionings over pattern database heuristics. A pattern for task Π with variables V is a subset P ⊆ V. By syntactically removing all variables from Π that are not in P, we obtain the projected task Π| P inducing the abstract transition system T P. The PDB heuristic h P for a pattern P is defined as h P (cost, s) = h * T P (cost, s| P), where s| P is the abstract state that s is projected to in Π| P. For the pattern sequence P 1,..., P n we define h DISPLAYFORM8 One of the simplest pattern selection algorithms is to generate all patterns up to a given size X (Felner, Korf, and Hanan 2004) and we call this approach SYS-NAIVE-X. It is easy to see that for tasks with n variables, SYS-NAIVE-X generates X i=1 n i patterns. Usually, many of these patterns do not add much information to a cost-partitioned heuristic over the patterns. Unfortunately, there is no efficiently computable test that allows us to discard such uninformative patterns. Even patterns without any goal variables can increase heuristic estimates in a cost partitioning BID27.However, in the setting where only non-negative cost functions are allowed in cost partitionings, there are efficiently computable criteria for deciding whether a pattern Algorithm 1 SYS-SCP: Given a planning task with states S(T), cost function cost and interesting patterns SYS, select a subset C ⊆ SYS.1: function SYS-SCP(Π) 2: DISPLAYFORM9 repeat for at most T x seconds 4: DISPLAYFORM10 for P ∈ ORDER(SYS) and at most T y seconds do 6:if P / ∈ C and PATTERNUSEFUL(σ, P) then 7: DISPLAYFORM11 until σ = 10: DISPLAYFORM12 is interesting, i.e., whether it cannot be replaced by a set of smaller patterns that together yield the same heuristic estimates BID26. The criteria are based on the causal graph CG(Π) of a task Π BID13. CG(Π) is a directed graph with a node for each variable in Π. If there is an operator with a precondition on u and an effect on v = u, CG(Π) contains a precondition arc from u to v. If an operator affects both u and v, CG(Π) contains co-effect arcs from u to v and from v to u. Definition 10 (Interesting Patterns). A pattern P is interesting if 1. CG(Π| P) is weakly connected, and 2. CG(Π| P) contains a directed path via precondition arcs from each node to some goal variable node. The systematic pattern generation method SYS-X generates all interesting patterns up to size X. We let SYS denote the set of all interesting patterns for a given task. On IPC benchmark tasks, SYS-X often generates much fewer patterns than SYS-NAIVE-X for the same size limit X. Still, it is usually infeasible to compute all SYS-X patterns and the corresponding projections for X > 3 within reasonable amounts of time and memory. Also, we hypothesize that even when considering only interesting patterns, usually only a small percentage of the systematic patterns up to size 3 contribute much information to the ing heuristic. For these two reasons we propose a new pattern selection algorithm that potentially considers all interesting patterns, but only selects the ones that it deems useful. Our new pattern selection algorithm repeatedly creates a new empty pattern sequence σ and only appends those interesting patterns to σ that increase any finite heuristic values of a saturated cost partitioning heuristic computed over σ. Algorithm 1 shows pseudo-code for the procedure, which we call SYS-SCP. It starts with an empty pattern collection C. In each iteration of the outer loop, SYS-SCP creates a new empty pattern sequence σ and then loops over the interesting patterns P ∈ SYS in the order chosen by ORDER (see Section 3.2) for at most T y seconds. SYS-SCP appends a pattern P to σ and includes it in C if there is a state s for which the saturated cost partitioning over σ extended by P has a higher finite heuristic value than the one over σ alone. Once an iteration selects no new patterns or SYS-SCP hits the time limit T x, the algorithm stops and returns C.We impose a time limit T x on the outer loop of the algorithm since the number of interesting patterns is exponential in the number of variables and therefore SYS-SCP usually cannot evaluate them all in a reasonable amount of time. By imposing a time limit T y on the inner loop, we allow SYS-SCP to periodically start over with a new empty pattern sequence. The most important component of the SYS-SCP algorithm is the PATTERNUSEFUL function that decides whether to select a pattern P. The function enumerates all states s ∈ S(Π), which is obviously infeasible for all but the smallest tasks Π. Fortunately, we can efficiently compute an equivalent test in the projection to P. Lemma 1. Consider a planning task Π with non-negative cost function cost and induced transition system T. Let s ∈ S(T) be a state, P be a pattern for Π and σ be a (possibly empty) sequence of patterns P 1,..., P n for Π. Finally, let rem be the remaining cost function after computing h DISPLAYFORM0 ⇔ 0 < h * T P (rem, s| P) < ∞ Step 1 substitutes P 1,..., P n for σ and Step 2 uses the definition of saturated cost partitioning heuristics. For Step 3 we need to show that x = n i=1 h Pi (cost i, s) is finite. The inequality states x < ∞. We now show x ≥ 0, which implies x > −∞. Using requirement 1 for saturated cost functions from Definition 7 and the fact that rem 0 = cost is non-negative, it is easy to see that all remaining cost functions are non-negative. Consequently, h Pi (cost i, s) = h Pi (rem i−1, s) ≥ 0 for all s ∈ S(T), which uses requirement 2 from Definition 7 and the fact that goal distances are non-negative in transition systems with non-negative weights. Step 4 uses the definition of PDB heuristics. Consider a planning task Π with non-negative cost function cost and induced transition system T. Let P be a single pattern and σ be a (possibly empty) sequence of patterns. Finally, let rem be the remaining cost function after computing DISPLAYFORM0 Follows directly from Lemma 1 and the fact that projections are induced abstractions: for each abstract state s in an induced abstraction there is at least one concrete state s which is projected to s.We use Theorem 1 in our SYS-SCP implementation by keeping track of the cost function rem, i.e., the costs that remain after computing h SCP σ. We select a pattern P if there are any goal distances d with 0 < d < ∞ in T P under rem. Theorem 1 also removes the need to compute h SCP σ⊕P from scratch for every pattern P. This is important since we want to decide whether or not to add P quickly and this operation should not become slower when σ contains more patterns. To obtain high finite heuristic values for solvable states it is important to choose good cost partitionings. In contrast, cost functions are irrelevant for detecting unsolvable states. This is the underlying reason why Lemma 1 only holds for finite values and therefore why SYS-SCP ignores unsolvable states. However, we can still use the information about unsolvable states contained in projections. It is easy to see that each abstract state in a projection corresponds to a partial state in the original task. If an abstract state is unsolvable in a projection, we call the corresponding partial state a dead end. Since projections preserve all paths, any state in the original task subsuming a dead end is unsolvable. We can extract all dead ends from the projections that SYS-SCP evaluates and use this information to prune unsolvable states during the A * search BID24. We showed in earlier work that the order in which saturated cost partitioning considers the component heuristics has a strong influence on the quality of the ing heuristic BID35. Choosing a good order is even more important for SYS-SCP, since it usually only sees a subset of interesting patterns within the allotted time. To ensure that this subset of interesting patterns covers different aspects of the planning task, we let the ORDER function generate the interesting patterns in increasing order by size. This leaves the question how to sort patterns of the same size. We propose four methods for making this decision. The first one (random) simply orders patterns of the same size randomly. The remaining three assign a key to each pattern, allowing us to sort by key in increasing or decreasing order. Causal Graph. The first ordering method is based on the insight that it is often more important to have accurate heuristic estimates near the goal states rather than elsewhere in the state space (e.g., BID15 BID39 . We therefore want to focus on patterns containing goal variables or variables that are closely connected to goal variables. To quantify "goalconnectedness" we use an approximate topological ordering ≺ of the causal graph CG(Π). We let the function cg: V → N + 0 assign each variable v ∈ V to its index in ≺. For a given pattern P, the cg ordering method returns the key cg(v 1),..., cg(v n), where v i ∈ P and cg(v i) < cg(v j) for all 1 ≤ i < j ≤ n. Since the keys are unique, they define a total order. Sorting the patterns by cg in decreasing order (cg-down), yields the desired order which starts with "goal-connected" patterns. States in Projection. Given a pattern P, the ordering method states returns the key |S(Π| P)|, i.e., the number of states in the projection to P. We use cg-down to break ties. Active Operators. Given a pattern P, the ops ordering method returns the number of operators that affect a variable in P. We break ties with cg-down. We implemented the SYS-SCP pattern selection algorithm in the Fast Downward planning system BID14 and conducted experiments with the Downward Lab toolkit on Intel Xeon Silver 4114 processors. Our benchmark set consists of all 1827 tasks without conditional effects from the optimization tracks of the 1998-2018 IPCs. The tasks belong to 48 different domains. We limit time by 30 minutes and memory by 3.5 GiB. All benchmarks 1, code 2 and experimental data 3 have been published online. To fairly compare the quality of different pattern collections, we use the same cost partitioning algorithm for all collections. Saturated cost partitioning is the obvious choice for the evaluation since experiments showed that it is preferable to all other cost partitioning algorithms for HC, SYS-2 and CPC patterns in almost all evaluated benchmark domains BID34 BID28.Diverse Saturated Cost Partitioning Heuristics. For a given pattern collection C, we compute diverse saturated cost partitioning heuristics using the diversification procedure by BID35: we start with an empty family of saturated cost partitioning heuristics F and a setŜ of 1000 sample states obtained with random walks ). Then we iteratively sample a new state s and compute a greedy order ω of C that works well for s BID36. If h SCP ω has a higher heuristic estimate for any state s ∈Ŝ than all heuristics in F, we add h SCP ω to F. We stop this diversification procedure after 200 seconds and then perform an A * search using the maximum over the heuristics in F. TAB1: Number of tasks solved by SYS-SCP using different time limits T x and T y for the outer loop (x axis) and inner loop (y axis).cg-up states-up random ops-down states-down ops-up cg-down Coverage cg-up -5 6 5 4 3 3 1140.0 states-up 6 -6 8 5 2 2 1153.0 random 10 10 -8 7 6 3 1148.2 ops-down 7 8 9 -4 7 3 1141.0 states-down 9 8 9 7 -4 2 1152.0 ops-up 11 12 12 11 11 -6 1166.0 cg-down 12 10 12 10 9 6 -1168.0 TAB2: Per-domain coverage comparison of different orders for patterns of the same size. The entry in row r and column c shows the number of domains in which order r solves more tasks than order c. For each order pair we highlight the maximum of the entries (r, c) and (c, r) in bold. Right: Total number of solved tasks. The for random are averaged over 10 runs (standard deviation: 3.36).Before we compare SYS-SCP to other pattern selection algorithms, we evaluate the effects of changing its parameters in four ablation studies. We use at most 2M states per PDB and 20M states in the PDB collection for all SYS-SCP runs. TAB1 shows that a time limit for the outer loop is more important than one for the inner loop, but for maximum coverage we need both limits. The combination that solves the highest number of tasks is 10s for the inner and 100s for the outer loop. We use these values in all other experiments. All configurations from Instead of discarding the computed pattern sequences when SYS-SCP finishes, we can turn each pattern sequence σ into a full pattern order by randomly appending all SYS-SCP patterns missing from σ to σ and pass the ing order to the diversification procedure. Feeding the diversification exclusively with such orders leads to solving 1130 tasks, while using only greedy orders for sample states BID36 ) solves 1156 tasks. We obtain the best by diversifying both types of orders, solving 1168 tasks, and we use this variant in all other experiments. In the next experiment, we evaluate the obvious baseline for SYS-SCP: selecting all (interesting) patterns up to a fixed size. TAB3 holds coverage of SYS-NAIVE-X and SYS-X for 1 ≤ X ≤ 5. We also include variants (*-LIM) that use at most 100 seconds, no more than 2M states in each projection and at most 20M states per collection. For the *-LIM variants, we sort the patterns in the cg-down order. The show that interesting patterns are always preferable to naive patterns, both with and without limits, which is why we only consider interesting patterns in SYS-SCP. Imposing limits is not important for SYS-1 and SYS-2, but leads to solving many more tasks for X ≥ 3. Overall, SYS-3-LIM has the highest total coverage (1088 tasks). In Table 4 we compare SYS-SCP to the strongest pattern selection algorithms from the literature: HC, SYS-3-LIM, CPC and CEGAR. (See Table 6 for per-domain coverage .) We run each algorithm with its preferred parameter values, which implies using at most 900s for HC and CPC and 100s for the other algorithms. HC is outperformed by all other algorithms. Interestingly, already the simple SYS-3-LIM approach is competitive with Table 4: Per-domain coverage comparison of pattern selection algorithms. For an explanation of the data see the caption of TAB2.CPC and CEGAR. However, we obtain the best with SYS-SCP. It is preferable to all other pattern selection algorithms in per-domain comparisons: no algorithm has higher coverage than SYS-SCP in more than three domains, while SYS-SCP solves more tasks than each of the other algorithms in at least 21 domains. SYS-SCP also has the highest total coverage of 1168 tasks, solving 70 more tasks than the strongest contender. This is a considerable improvement in the setting of optimal classical planning, where task difficulty tends to scale exponentially. In our final experiment, we evaluate whether Scorpion BID37, one of the strongest optimal planners in IPC 2018, benefits from using SYS-SCP patterns. Scorpion computes diverse saturated cost partitioning heuristics over HC and SYS-2 PDB heuristics and Cartesian abstraction heuristics (CART) BID31. We abbreviate this combination with COMB=HC+SYS-2+CART. In TAB6 we compare the original Scorpion planner, three Scorpion variants that use different sets of heuristics and the top three optimal planners from IPC 2018, Delfi 1 (Sievers et al. 2019), Complementary 1 BID9 and Complementary 2 BID8 ). (Table 6 holds perdomain coverage .) In contrast to the configurations we evaluated above, all planners in TAB6 prune irrelevant operators in a preprocessing step BID0.The show that all Scorpion variants outperform the top three IPC 2018 planners in per-domain comparisons. We also see that Scorpion benefits from using SYS-SCP PDBs instead of the COMB heuristics in many domains. Using the union of both sets is clearly preferable to using either COMB or SYS-SCP alone, since it raises the total coverage to 1261 by 56 and 44 tasks, respectively. For maximum coverage (1265 tasks), Scorpion only needs SYS-SCP PDBs and Cartesian abstraction heuristics. We introduced a new pattern selection algorithm based on saturated cost partitioning and showed that it outperforms Table 6: Number of tasks solved by different planners.

Using saturated cost partitioning to select patterns is preferable to all existing pattern selection algorithms.

Represent the TLDR for finding the corresponding scientific article