Question: Represent the article for finding a single-sentence TLDR:

Question: Program verification offers a framework for ensuring program correctness and therefore systematically eliminating different classes of bugs. Inferring loop invariants is one of the main challenges behind automated verification of real-world programs which often contain many loops. In this paper, we present Continuous Logic Network (CLN), a novel neural architecture for automatically learning loop invariants directly from program execution traces. Unlike existing neural networks, CLNs can learn precise and explicit representations of formulas in Satisfiability Modulo Theories (SMT) for loop invariants from program execution traces. We develop a new sound and complete semantic mapping for assigning SMT formulas to continuous truth values that allows CLNs to be trained efficiently. We use CLNs to implement a new inference system for loop invariants, CLN2INV, that significantly outperforms existing approaches on the popular Code2Inv dataset. CLN2INV is the first tool to solve all 124 theoretically solvable problems in the Code2Inv dataset. Moreover, CLN2INV takes only 1.1 second on average for each problem, which is 40 times faster than existing approaches. We further demonstrate that CLN2INV can even learn 12 significantly more complex loop invariants than the ones required for the Code2Inv dataset. Program verification offers a principled approach for systematically eliminating different classes of bugs and proving the correctness of programs. However, as programs have become increasingly complex, real-world program verification often requires prohibitively expensive manual effort (; ;). Recent efforts have focused on automating the program verification process, but automated verification of general programs with unbounded loops remains an open problem (; . Verifying programs with loops requires determining loop invariants, which captures the effect of the loop on the program state irrespective of the actual number of loop iterations. Automatically inferring correct loop invariants is a challenging problem that is undecidable in general and difficult to solve in practice . Existing approaches use stochastic search , heurstics-based search , PAC learning based on counter examples , or reinforcement learning . However, these approaches often struggle to learn complex, real-world loop invariants. In this paper, we introduce a new approach to learning loop invariants by modeling the loop behavior from program execution traces using a new type of neural architecture. We note that inferring loop invariants can be posed as learning formulas in Satisfiability Modulo Theories (SMT) over program variables collected from program execution traces . In principle, Neural networks seem well suited to this task because they can act as universal function approximators and have been successfully applied in various domains that require modeling of arbitrary functions . However, loop invariants must be represented as explicit SMT formulas to be usable for program verification. Unfortunately, existing methods for extracting logical rules from general neural architectures lack sufficient precision , while inductive logic learning lacks sufficient expressiveness for use in verification . We address this issue by developing a novel neural architecture, Continuous Logic Network (CLN), which is able to efficiently learn explicit and precise representations of SMT formulas by using continuous truth values. Unlike existing neural architectures, CLNs can represent a learned SMT formula explicitly in its structure and thus allow us to precisely extract the exact formula from a trained model. In order to train CLNs, we introduce a new semantic mapping for SMT formulas to continuous truth values. Our semantic mapping builds on BL, or basic fuzzy logic (Hájek, 2013), to support general SMT formulas in a continuous logic setting. We further prove that our semantic model is sound (i.e., truth assignments for the formulas are consistent with their discrete counterparts) and complete (i.e., all formulas can be represented) with regard to the discrete SMT formula space. These properties allow CLNs to represent any quantifier-free SMT formula operating on mixed integer-real arithmetic as an end-to-end differentiable series of operations. We use CLNs to implement a new inference system for loop invariants, CLN2INV, that significantly outperforms state-of-the-art tools on the Code2Inv dataset by solving all 124 theoretically solvable problems in the dataset. This is 20 problems more than LoopInvGen, the winner of the SyGus 2018 competition loop invariant track . Moreover, CLN2INV finds invariants for each program in 1.1 second on average, more than 40 times faster than LoopInvGen. We also demonstrate that CLN2INV is able to learn complex, real-world loop invariants with combinations of conjunctions and disjunctions of multivariable constraints. Our main contributions are: • We introduce a new semantic mapping for assigning continuous truth values to SMT formulas that is theoretically grounded and enables learning formulas through backpropagation. We further prove that our semantic model is sound and complete. • We develop a novel neural architecture, Continuous Logic Networks (CLNs), that to the best of our knowledge is the first to efficiently learn precise and explicit SMT formulas by construction. • We use CLNs to implement a new loop invariant inference system, CLN2INV, that is the first to solve all 124 theoretically solvable problems in the Code2Inv dataset, 20 more than the existing methods. CLN2INV is able to find invariants for each problem in 1.1 second on average, 40× faster than existing systems. • We further show CLN2INV is able to learn 12 more complex loop invariants than the ones present in the Code2Inv dataset with combinations of multivariable constraints. Related Work. Traditionally, loop invariant learning relies on stochastic or heuristics-guided search . Other approaches like NumInv analyze traces and discover conjunctions of equalities by solving a system of linear equations . LoopInvGen uses PAC learning of CNF using counter-examples . By contrast, Code2Inv learns to guess loop invariants using reinforcement learning with recurrent and graph neural networks . However, these approaches struggle to learn complex invariants. Unlike these works, CLN2INV efficiently learns complex invariants directly from execution traces. There is a extensive work on PAC learning of boolean formulas, but learning precise formulas require a prohibitively large number of samples . Several recent works use differentiable logic to learn boolean logic formulas from noisy data (; ;) or improving adversarial robustness by applying logical rules to training . By contrast, our work learns precise SMT formulas directly by construction, allowing us to learn richer predicates with compact representation in a noiseless setting. A variety of numerical relaxations have been applied to SAT and SMT solving. Application-specific approximations using methods such as interval overapproximation and slack variables have been developed for different classes of SMT . More recent work has applied recurrent and graph neural networks to Circuit SAT problems and unsat core detection (; ; Selsam & Bjørner, 2019). FastSMT uses embeddings from natural language processing like skip-gram and bag-of-words to represent formulas for search strategy optimization . Unlike these approaches, we relax the SMT semantics directly to generate a differentiable representation of SMT. In this section, we introduce the problem of inferring loop invariants and provide a brief overview of Satisfiability Modulo Theories (SMT), which are used to represent loop invariants. We provide into fuzzy logic, which we extend with our new continuous semantic mapping for SMT. Loop invariants capture loop behavior irrespective of number of iterations, which is crucial for verifying programs with loops. Given a loop, while(LC){C}, a precondition P, and a postcondition Q, the verification task involves finding a loop invariant I that can be concluded from the pre-condition and implies the post-condition . Formally, it must satisfy the following three conditions, in which the second is a Hoare triple describing the loop: Example of Loop Invariant. Consider the example loop in Fig.1. For a loop invariant to be usable, it must be valid for the precondition (t = 10 ∧ u = 0), the recursion step when t = 0, and the post condition (u = 20) when the loop condition is no longer satisfied, i.e., t = 0. The correct and precise invariant I for the program is (2t + u = 20). //pre: t=10 /\ u=0 while (t != 0){t = t -1; u = u + 2;} //post: u=20 The desired loop invariant I for the left program is a boolean function over program variables t, u such that: The desired and precise loop invariant I is (2t + u = 20). Satisfiability Modulo Theories (SMT) are an extension of Boolean Satisfiability that allow solvers to reason about complex problems efficiently. Loop invariants and other formulas in program verification are usually encoded with quantifier-free SMT. A formula F in quantifier-free SMT can be inductively defined as below: ∈ {=, =, <, >, ≤, ≥} where E 1 and E 2 are expressions of terms. The loop invariant (2t + u = 20) in Fig. 1 is an SMT formula. Nonlinear arithmetic theories admit higher-order terms such as t 2 and t * u, allowing them to express more complex constraints. For example, (¬(2 ≥ t 2)) is an SMT formula that is true when the value of the high-order term t 2 is larger than 2. Basic fuzzy logic (BL) is a class of logic that uses continuous truth values in the range and is differentiable almost everywhere 1 (Hájek, 2013). BL defines logical conjunction with functions called t-norms, which must satisfy specific conditions to ensure that the behavior of the logic is consistent with boolean First Order Logic. Formally, a t-norm (denoted ⊗) in BL is a binary operator over truth values in the interval satisfying the following conditions: consistency (1 ⊗ x = x and 0 ⊗ x = 0), commutativity (x ⊗ y = y ⊗ x), associativity (x ⊗ (y ⊗ z) = (x ⊗ y) ⊗ z), and monotonicity (x 1 ≤ x 2 =⇒ x 1 ⊗ y ≤ x 2 ⊗ y). Besides these conditions, BL also requires that t-norms be continuous. Given a t-norm ⊗, its associated t-conorm (denoted ⊕) is defined with DeMorgan's law: t ⊕ u ¬(¬t ⊗ ¬u), which can be considered as logical disjunction. A common t-norm is the product t-norm x ⊗ y = x · y with its associated t-conorm x ⊕ y = x + y − x · y. We introduce a continuous semantic mapping, S, for SMT on BL that is end-to-end differentiable. The mapping S associates SMT formulas with continuous truth values while preserving each formula's semantics. In this paper, we only consider quantifier-free formulas. This process is analogous to constructing t-norms for BL, where a t-norm operates on continuous logical inputs. We define three desirable properties for continuous semantic mapping S that will preserve formula semantics while facilitating parameter training with gradient descent: 1. S(F) should be consistent with BL. For any two formulas F and F, where F (x) is satisfied and F (x) is unsatisfied with an assignment x of formula terms, we should have S(F)(x) < S(F)(x). This will ensure the semantics of SMT formulas are preserved. 2. S(F) should be differentiable almost everywhere. This will facilitate training with gradient descent through backpropogation. 3. S(F) should be increasing everywhere as the terms in the formula approach constraint satisfaction, and decreasing everywhere as the terms in the formula approach constraint violation. This ensures there is always a nonzero gradient for training. Continuous semantic mapping. We first define the mapping for ">" (greater-than) and "≥" (greater-than-or-equal-to) as well as adopting definitions for "¬", "∧", and "∨" from BL. All other operators can be derived from these. For example, "≤" (less-than-or-equal-to) is derived using "≥" and "¬", while "=" (equality) is then defined as the conjunction of formulas using "≤" and "≥." Given constants B > 0 and > 0, we first define the the mapping S on ">" and "≥" using shifted and scaled sigmoid functions: Illustrations of these functions are given in Appendix A. The validity of our semantic mapping lie in the following facts, which can be proven with basic algebra. When goes to zero and B * goes to infinity, our continuous mapping of ">" and "≥" will preserve their original semantics. Under these conditions, our mapping satisfies all three desirable properties. In practice, for small and large B, the properties are also satisfied if |t − u| >. Next we define the mapping S for boolean operators "∧", "∨" and "¬" using BL. Given a specific t-norm ⊗ and its corresponding t-conorm ⊕, it is straightforward to define mappings of "∧", "∨" and "¬": on the above definitions, the mapping for other operators can be derived as follows: The mapping S on "=" is valid since the following limit holds (see Appendix B for the proof). The mapping for other operators shares similar behavior in the limit, and also fulfill our desired properties under the same conditions. Using our semantic mapping S, most of the standard operations of integer and real arithmetic, including addition, subtraction, multiplication, division, and exponentiation, can be used normally and mapped to continuous truth values while keeping the entire formula differentiable. Moreover, any expression in SMT that has an integer or real-valued can be mapped to continuous logical values via these formulas, although end-to-end differentiability may not be maintained in cases where specific operations are nondifferentiable. In this section, we describe the construction of Continuous Logic Networks (CLNs) based on our continuous semantic mapping for SMT on BL. CLN Construction. CLNs use our semantic mapping to provide a general neural architecture for learning SMT formulas. In a CLN, the learnable coefficients and smoothing parameters correspond to the learnable parameters in a standard feedforward network, and the continuous predicates, tnorms, and t-conorms operate as activation functions like ReLUs in a standard network. In this paper, we focus on shallow networks to address the loop invariant inference problem, but we envision deeper general purpose CLNs that can learn arbitrary SMT formulas. When constructing a CLN, we work from an SMT Formula Template, in which every value is marked as either an input term, a constant, or a learnable parameter. Given an SMT Formula Template, we dynamically construct a CLN as a computational graph. Figure 2 shows a simple formula template and the constructed CLN. We denote the CLN model constructed from the formula template S(F) as M F. CLN Training. Once the CLN has been constructed based on a formula template, it is trained with the following optimization. Given a CLN model M constructed from an SMT template with learnable parameters W, and a set X of valid assignments for the terms in the SMT template, the expected value of the CLN is maximized by minimizing a loss function L that penalizes model outputs that are less than one. A minimum scaling factor β is selected, and a hinge loss is applied to the scaling factors (B) to force the differentiable predicates to approach sharp cutoffs. The offset is also regularized to ensure precision. The overall optimization is formulated as: where λ and γ are hyperparameters respectively governing the weight assigned to the scaling factor and offset regularization., and L is any loss function strictly decreasing in domain. Given a CLN that has been trained to a loss approaching 0 on a given set of valid assignments, we show that the ing continuous SMT formula learned by the CLN is consistent with an equivalent discrete SMT formula. In particular, we prove that such a formula is sound, (i.e., a CLN will learn a correct SMT formula with respect to the training data), and that our continuous mapping is complete, (i.e., CLNs can represent any SMT formula that can be represented in discrete logic). We further prove that CLNs are guaranteed to converge to a globally optimal solution for formulas, which can be expressed as the conjunction of linear equalities. We provide formal definitions and proofs for soundness and completeness in Appendix C and optimality in Appendix D. We use CLNs to implement a new inference system for loop invariants, CLN2INV, which learns invariants directly from execution traces. CLN2INV follows the same overall process as other loop invariant inference systems such as LoopInvGen and Code2Inv -it iterates through likely candidate invariants and checks its validity with an SMT solver. The key difference between our method and other systems is that it learns a loop invariant formula directly from trace data. Figure 2 provides an overview of the architecture. Preprocessing. We first perform static analysis and instrument the program to prepare for training data generation. In addition to collecting the given precondition and post-condition, the static analysis extracts all constants in the program, along with the loop termination condition. We then instrument the program to record all program variables before each loop execution and after the loop termination. We also restrict the loop to terminate after a set number of iterations to prevent loops running indefinitely (for experiments in this paper, we set the max loop iterations to 50). We also strengthen the precondition to ensure loop execution (see Appendix E). Training Data Generation. We generate training data by running the program repeatedly on a set of randomly initialized inputs that satisfy the preconditions. Unconstrained variables are initialized from a uniform distribution centered on 0 with width r, where r is a hyper-parameter of the sampling process. Variables with either upper or lower bound precondition constraints are initialized from a uniform distribution adjacent to their constraints with width r, while variables with both upper and lower bounds in the precondition are sampled uniformly within their bounds. For all of our experiments in this paper, we set r to 10. When the number of uninitialized variables is small (i.e., less than 3), we perform this sampling exhaustively. An example of training data generation is provided in Appendix F. Template Generation. We generate templates in three stages with increasing expressiveness: 1. We first generate templates directly from the pre-condition and post-condition. 2. We next extract the individual clauses from the pre-and post-condition, as well as the loop condition, and generate templates from conjunctions and disjunctions of each possible pair of clauses. 3. We finally generate more generic templates of increasing complexity with a combination of one or more equality constraints on all variables combined with conjunctions of inequality constraints, which are based on the loop condition and individual variables. We describe the template generation in detail in Appendix F. To detect when higher order terms may be present in the invariant, we perform a log-log linear regression on each variable relative to the loop iteration, similarly to. If the loop contains one or more variables that grow superlinearly relative to the loop iteration, we add higher order polynomial terms to the equality constraints in the template, up to the highest degree detected among the loop variables. CLN Construction and Training. Once a template formula has been generated, a CLN is constructed from the template using the formulation in §4. As an optimization, we represent equality constraints as Gaussian-like functions that retain a global maximum when the constraint is satisfied as discussed in Appendix G. We then train the model using the collected execution traces. Invariant Checking. Invariant checking is performed using SMT solvers such as Z3 (De Moura & Bjørner, 2008). After the CLN for a formula template has been trained, the SMT formula for the loop invariant is recovered by normalizing the learned parameters. The invariant is checked against the pre, post, and recursion conditions as described in §2.1. If the correct invariant is not found, we return to the template generation phase to continue the search with a more expressive template. We compare the performance of CLN2INV with two existing methods and demonstrate the efficacy of the method on several more difficult problems. Finally, we conduct two ablation studies to justify our design choices. Performance Comparison. We compare CLN2INV to two state-of-the-art methods: Code2Inv (based on neural code representation and reinforcement learning) and LoopInvGen (PAC learning over synthesized CNF formulas) . We limit each method to one hour per problem in the same format as the SyGuS Competition . CLN2INV is able to solve all 124 problems in the benchmark. LoopInvGen solves 104 problems while Code2inv solves 90. 2 Figure 3a shows the measured runtime on each evaluated system. CLN2INV solves problems in 1.1 second on average, which is over 40× faster than LoopInvGen, the second fastest system in the evaluation. It spends the most time on solver calls (0.6s avg.) and CLN training (0.5s avg.), with negligible time spent on preprocessing, data generation, and template generation on each problem (<20ms ea.) 3. We provide a breakdown of runtimes in Appendix I. In general, CLN2INV has similar performance to LoopInvGen on simple problems but is able to scale efficiently to complex problems. Figure 3b shows the number of Z3 calls made by each method. For almost all problems, CLN2INV requires fewer Z3 calls than the other systems, although for some difficult problems it uses more Z3 calls than Code2Inv. Table 1 summarizes of the performance evaluation. Code2Inv require much more time on average per problem, but minimizes the number of calls made to an SMT solver. In contrast, LoopInvGen is efficient at generating a large volume of guessed candidate invariants, but is much less accurate for each individual invariant. CLN2INV can be seen as balance between the two approaches: it searches over candidate invariants more quickly than Code2Inv, but generates more accurate invariants than LoopInvGen, ing in lower overall runtime. We consider two classes of more difficult loop invariant inference problems that are not present in the Code2Inv dataset. The first require conjunctions and disjunctions of multivariable constraints, and the second require polynomials with many higher order terms. Both of these classes of problems are significantly more challenging because they are more complex and cause the space of possible invariants to grow much more quickly. To evaluate on problems that require invariants with conjunctions and disjunctions of multivariable constraints, we construct 12 additional problems. For these problems, we only consider loops with invariants that cannot be easily inferred with pre-and post-condition based heuristics. Appendix J describes these problems in more detail and provides examples. CLN2INV is able to find correct invariants for all 12 problems in less than 20 seconds, while Code2Inv and LoopInvGen time out after an hour. To evaluate on problems with higher order polynomial invariants, we test CLN2INV on the power summation problems in the form u = k t=0 t d for a given degree d, which have been used in evaluation for polyonomial loop invariant inference . We discuss these problems in more detail in Appendix J. CLN2INV can correctly learn the invariant for 1st and 2nd order power summations, but cannot learn correct invariants for 3rd, 4th or 5th order summations, which have many more higher order terms. We do not evaluate the other methods on these problems because they are not configured for nonlinear arithmetic by default. Effect of CLN Training on Performance. CLN2INV relies on a combination of heuristics using static analysis and learning formulas from execution traces to correctly infer loop invariants. In this ablation we disable model training and limit CLN2INV to static models with no learnable parameters. Static CLN2INV solves 91 problems in the dataset. Figure 4 shows a comparison of full CLN2INV with one limited to static models. CLN2INV's performance with training disabled shows that a large number of problems in the dataset are relatively simple and can be inferred from basic heuristics. However, for more difficult problems, CLN learning is key to inferring correct invariants. We develop a novel neural architecture that explicitly and precisely learns SMT formulas by construction. We achieve this by introducing a new sound and complete semantic mapping for SMT that enables learning formulas through backpropagation. We use CLNs to implement a loop invariant inference system, CLN2INV, that is the first to solve all theoretically solvable problems in the Code2Inv benchmark and takes only 1.1 second on average. We believe that the CLN architecture will also be beneficial for other domains that require learning SMT formulas. A CONTINUOUS PREDICATES Figure 5 shows examples of shifted sigmoids for S(>), S(≥), and S(=). Combing these , we have For any t-norm, we have 0 ⊗ 1 = 0, 1 ⊗ 1 = 1, and 1 ⊗ 0 = 0. Put it altogether, we have (f (t, u; B,) ⊗ g(t, u; B,)) = 1 t = u 0 t = u which concludes the proof. Soundness. Given the SMT formula F, the CLN model M F constructed from S(F) always preserves the truth value of F. It indicates that given a valid assignment to the terms x in F, Completeness. For any SMT formula F, a CLN model M can be constructed representing that formula. In other words, CLNs can express all SMT formulas on integers and reals. We formally state these properties in Theorem 1. Before that we need to define a property for t-norms. The product t-norm and Godel t-norm have this property, while the Lukasiewicz t-norm does not. Theorem 1. For any quantifier-free linear SMT formula F, there exists CLN model M, such that as long as the t-norm used in building M satisfies Property 1. Proof. For convenience of the proof, we first remove all <, ≤, = and = in F, by transforming. Now the only operators that F may contain are >, ≥, ∧, ∨, ¬. We prove Theorem 1 by induction on the constructor of formula F. In the following proof, we construct model M given F and show that it satisfied Eq.. We leave the proof for why M also satisfied Eq. to readers. Atomic Case. When F is an atomic clause, then F will be in the form of x * W + b > 0 or x * W + b ≥ 0. For the first case, we construct a linear layer with weight W and bias b followed by a sigmoid function scaled with factor B and right-shifted with distance. For the second case, we construct the same linear layer followed by a sigmoid function scaled with factor B and left-shifted with distance. Simply evaluating the limits for each we arrive at And from the definition of sigmoid function we know 0 ≤ M (x; B,) ≤ 1. Negation Case. If F = ¬F, from the induction hypothesis, F can be represented by models M satisfying Eq. From the induction hypothesis we know that F (x) = F alse. So F (x) = ¬F (x) = T rue. Conjunction Case. If F = F 1 ∧ F 2, from the induction hypothesis, F 1 and F 2 can be represented by models M 1 and M 2, such that both (F 1, M 1) and (F 2, M 2) satisfy Eq.. Let p 1 and p 2 be the output nodes of M 1 and M 2. We add a final output node p = p 1 ⊗ p 2. So M (x; B,) = M 1 (x; B,) ⊗ M 2 (x; B,). Since (⊗) is continuous and so are M 1 (x; B,) and M 2 (x; B,), we know their composition M (x; B,) is also continuous. (Readers may wonder why M 1(x; B,) is continuous. Actually the continuity of M (x; B,) should be proved inductively like this proof itself, and we omit it for brevity.) From the definition of (⊗), we have Eq. 0 ≤ M (x; B,) ≤ 1. Now we prove the =⇒ side of Eq.. For any x, if F (x) = T rue which means both F 1 (x) = T rue and F 2 (x) = T rue, from the induction hypothesis we know that lim →0 Then we prove the ⇐= side. From the induction hypothesis we know that M 1 (x; B,) ≤ 1 and M 2 (x; B,) ≤ 1. From the non-decreasing property of t-norms (see §2.3), we have Then from the consistency property and the commutative property, we have Put them altogether we get Because we know lim →0 M (x; B,) = 1, according to the squeeze theorem in calculus, we get From the induction hypothesis, we know that F 1 (x) = T rue. We can prove F 2 (x) = T rue in the same manner. Finally we have Disjunction Case. For the case F = F 1 ∨ F 2, we construct M from M 1 and M 2 as we did in the conjunctive case. This time we let the final output node be p = p 1 ⊕ p 2. From the continuity of (⊗) and the definition of (⊕) (t ⊕ u = 1 − (1 − t) ⊗ (1 − u)), (⊕) is also continuous. We conclude M (x; B,) is also continuous and 0 ≤ M (x; B,) ≤ 1 by the same argument as F = F 1 ∧ F 2. Now we prove the " =⇒ " side of Eq.. For any assignment x, if F (x) = T rue which means F 1 (x) = T rue or F 2 (x) = T rue. Without loss of generality, we assume F 1 (x) = T rue. From the induction hypothesis, we know lim →0 For any (⊕) and any 0 ≤ t, t ≤ 1, if t ≤ t, then Using this property and the induction hypothesis M 2 (x; B,) ≥ 0, we have From the induction hypothesis we also have M 1 (x; B,) ≤ 1. Using the definition of (⊕) and the consistency of (⊗) (0 ⊗ x = 0), we get M 1 (x; B,) ⊕ 0 = M 1 (x; B,). Put them altogether we get Because we know lim →0 + B· →∞ M 1 (x; B,) = 1, according to the squeeze theorem in calculus, we Then we prove the " ⇐= " side. Here we need to use the existence of limit: This property can be proved by induction like this proof itself, thus omitted for brevity. Let Since we have lim →0 M (x; B,) = 1, we get Using Property 1 of (⊗) (defined in §4), we have c 1 = 1 ∨ c 2 = 1. Without loss of generality, we assume c 1 = 1. From the induction hypothesis, we know that Careful readers may have found that if we use the continuous mapping function S in §3, then we have another perspective of the proof above, which can be viewed as two interwoven parts. The first part is that we proved the following lemma. Corollary 1. For any quantifier-free linear SMT formula F, Corollary 1 indicates the soundness of S. The second part is that we construct a CLN model given S(F). In other words, we translate S(F) into vertices in a computational graph composed of differentiable operations on continuous truth values. Optimality. For a subset of SMT formulas (conjunctions of multiple linear equalities), CLNs are guaranteed to converge at the global minumum. We formally state this in Theorem 2. We first define another property similar to strict monotonicity. Property 2. ∀t 1 t 2 t 3, (t 1 < t 2) and (t 3 > 0) implies (t 1 ⊗ t 3 < t 2 ⊗ t 3). Theorem 2. For any CLN model M F constructed from a formula, F, by the procedure shown in the proof of Theorem 1, if F is the conjunction of multiple linear equalities then any local minimum of M F is the global minimum, as long as the t-norm used in building M F satisfies Property 2. Proof. Since F is the conjunction of linear equalities, it has the form Here W = {w ij} are the learnable weights, and {t ij} are terms (variables). We omit the bias b i in the linear equalities, as the bias can always be transformed into a weight by adding a constant of 1 as a term. For convenience, we define f (x) = S(x = 0) =. Given an assignment x of the terms {t ij}, if we construct our CLN model M F following the procedure shown in the proof of Theorem 1, the output of the model will be When we train our CLN model, we have a collection of m data points {t ij1}, {t ij2},..., {t ijm}, which satisfy formula F. If B and are fixed (unlearnable), then the loss function will be Suppose W * = {w * ij} is a local minima of L(W). We need to prove W * is also the global minima. To prove this, we use the definition of a local minima. That is, For convenience, we denote u ik = li j=1 w ij t ijk. Then we rewrite Eq. as If we can prove at Then because (i) f reaches its global maximum at 0, (ii) the t-norm (⊗) is monotonically increasing, (iii) L is monotonically decreasing, we can conclude that W * is the global minima. Here we just show the case i = 1. The proof for i > 1 can be directly derived using the associativity of (⊗). Since f (x) > 0 for all x ∈ R, using Property 2 of our t-norm (⊗), we know that α k > 0. Now the loss function becomes Because (i) f (x) is an even function decreasing on x > 0 (which can be easily proved), (ii) (⊗) is monotonically increasing, (iii) L is monotonically decreasing, for −δ < γ < 0, we have Combing Eq. and Eq., we have Now we look back on Eq.. Since (i) L is strictly decreasing, (ii) the t-norm we used here has Property 2 (see §4 for definition), (iii) α k > 0, the only case when (=) holds is that for all 1 ≤ k ≤ m, we have f (|u ik (1 + γ)|) = f (|u ik |). Since f (x) is strictly decreasing for x ≥ 0, we have |u ik (1 + γ)| = |u ik |. Finally because −1 < −δ < γ < 0, we have u ik = 0. Theorem 3. Given a program C: assume(P); while (LC) {C} assert(Q); If we can find a loop invariant I for program C: assume(P ∧ LC); while (LC) {C} assert(Q); and P ∧ ¬LC =⇒ Q, then I ∨ (P ∧ ¬LC) is a correct loop invariant for program C. Proof. Since I is a loop invariant of C, we have We want to prove I ∨ (P ∧ ¬LC) is a valid loop invariant of C, which means We prove the three propositions separately. To prove P ∧ LC =⇒ I ∨ (P ∧ ¬LC), we transform it into a stronger proposition P ∧ LC =⇒ I, which directly comes from (a). For {(I ∨ (P ∧ ¬LC)) ∧ LC}C{I ∨ (P ∧ ¬LC)}, after simplification it becomes {I ∧ LC}C{I ∨ (P ∧ ¬LC)}, which is a direct corollary of (b). For (I ∨ (P ∧ ¬LC)) ∧ ¬LC =⇒ Q, after simplification it will become two separate propositions, I ∧ ¬LC =⇒ Q and P ∧ ¬LC =⇒ Q. The former is exactly (c), and the latter is a known condition in the theorem. Training Data Generation Example. Figure 6 provides an example of our training data generation procedure. The uninitialized variable k is sampled according to the precondition k ≤ 8 within the predefined width r = 10. So we end up enumerating k = 8, 7,..., −2. For each k, the loop is executed repeatedly until termination, thus generating a small set of samples. The final training set is the union of these small sets. Figure 6: Illustration of how training data is generated. After the sampling procedure in (b) we have a collection of 88 samples which will later be fed to the CLN model. Template Generation. Templates are first generated from the pre-and post-conditions, followed by every pair of clauses extracted from the pre-condition, post-condition, and loop-condition. Generic templates are then constructed consisting of one or more general equality constraints containing all variables conjoined with inequality constraints. Three iterations of the generic template generation are shown here: Algorithm 1 summarizes the template generation process. In it the following functions are defined: Construct a template given an smt formula. extract clauses: Extract individual clauses from smt formulas. estimate degrees: Performs log-log linear regression to estimate degree of each variable. polynomial kernel: Executes polynomial kernel on variables and data for a given degree. is single constraint: Checks if condition is a single inequality constraint. extract loop constraint: Converts the loop condition to learnable smt template. Note that templates are generated on demand, so each template is used to infer a possible invariant before the next is generated. for eq clause in eq clauses do template len ← template len + 1 31: end while We use a Gaussian-like function S(t = u) = exp(− (t−u) 2 2σ2 ) to represent equalities in our experiments. It has the following two properties. First, it preserves the original semantic of = when σ → 0, similar to the mapping S(t = u) = Second, if we view S(t = u) as a function over t − u, then it reaches its only local maximum at t − u = 0, which means the equality is satisfied. Here we provide an example which is Problem 106 in the dataset. The post condition a ≥ m is wrong if we start from a = 0, m = 1, and k = 0. int k = 0; int a, m; assume(a <= m); while (k < 1) {if (m < a) m = a; k = k + 1; } assert(a >= m); Executing the program with these inputs in a = 0, m = 1, and k = 1 as the if condition is never satisfied. But clearly, the post condition a ≥ m is violated. Below we tabulate the counterexamples invalidating the nine removed problems from the dataset: I RUNTIME BREAKDOWN In Table 3 we provide a more detailed analysis of how much time is spent on each stage of the invariant inference pipeline in CLN2INV. Measurements are averaged over 5 runs. The system spends most time on solver calls (0.6s avg.) and CLN training (0.5s avg.) with negligible time spent on preprocessing, sampling, and template generation. For most problems in the Code2Inv benchmark, CLN2INV completes CLN training quickly (less than 0.2s) and spends most of its time performing solver checks, but it requires more epochs to train on some complex problems with many variables. Multivariable Conjunction/Disjunction Invariants. In this subsection, we discuss in detail two of the 12 more difficult loop invariant problems we have created. The first problem is shown in Figure 7. The correct loop invariant for the program is ((t + u = 0) ∨ (t − u = 0)) ∧ (t ≤ 0). The plot of the trace in Figure 7b shows that the points lie on one of the two lines expressible as linear equality constraints. These constraints along with the inequality can be learned from the execution trace using CLN2INV in under 20 seconds. Bernoulli distribution with success probability 0.5. Although the branching behavior is not deterministic, we know (t + u = 0) ∧ (v + w = 0) ∧ (u + w ≥ 0) is a correct invariant as it holds regardless of which branch is taken. Our CLN2INV can learn this invariant within 20 seconds. For both problems in Figure 7 and 8, both Code2inv and LoopInvGen time out after one hour without finding a solution. Polynomial Invariants. Here we provide and an example of the higher order polynomial problems; more precisely, the power summation problems in the form u = k t=0 t d for a given degree d. We found that CLN2INV was able to learn invariants for programs with 10 terms within 2nd degree, but struggled with problems with 20 or more terms of 3rd degree or higher. Table 4 summarizes these . Consider the example loop in Figure 9 which computes the sum of the first k cubes. We know this sum has a closed form solution: For this problem, we would hope to extract the invariant: However, by naively using the polynomial kernel just as methods like NumInv suggest , we will have 35 monomials of degree at most four over three variables as candidate terms (t 3 u, t 2 k 2, tu 2 k, ...), and the model must learn to ignore all the terms except u, t 4, t 3, and t 2. Additionally, by nature of polynomials the highest order term is a good approximation for the whole polynomial. Thus, u = t 4 is a good approximation based on the data. We observe our model will find the correct coefficient for t 4 but the accuracy degrades on the lower ordered terms. The difficulty of learning polynomial invariants using CLN is an interesting direction for future studies. //pre: t = u = 0 /\ k >= 0 while (t < k) {t++; u += t * t * t;} //post: 4u == k ** 2 * (k + 1) ** 2 Figure 9: Pseudocode for Polynomial Invariant Problem

represent the short summary for finding the corresponding scientific article (long)

Fewshot example: "We show how neural encoding models can be trained to capture both the signal and spiking variability of neural population data using GANs." == "Neural population responses to sensory stimuli can exhibit both nonlinear stimulus- dependence and richly structured shared variability. Here, we show how adversarial training can be used to optimize neural encoding models to capture both the deterministic and stochastic components of neural population data. To account for the discrete nature of neural spike trains, we use the REBAR method to estimate unbiased gradients for adversarial optimization of neural encoding models. We illustrate our approach on population recordings from primary visual cortex. We show that adding latent noise-sources to a convolutional neural network yields a model which captures both the stimulus-dependence and noise correlations of the population activity. Neural population activity contains both nonlinear stimulus-dependence and richly structured neural variability. An important challenge for neural encoding models is to generate spike trains that match the statistics of experimentally measured neural population spike trains. Such synthetic spike trains can be used to explore limitations of a model, or as realistic inputs for simulation or stimulation experiments. Most encoding models either focus on modelling the relationship between stimuli and mean-firing rates e.g., or on the statistics of correlated variability ('noise correlations'), e.g.. They are typically fit with likelihood-based approaches (e.g. maximum likelihood estimation MLE, or variational methods for latent variable models). While this approach is very flexible and powerful, it has mostly been applied to simple models of variability (e.g. Gaussian inputs). Furthermore, MLE-based models are not guaranteed to yield synthetic data that matches the statistics of empirical data, particularly in the presence of latent variables. Generative adversarial networks (GANs) are an alternative to fitting the parameters of probabilistic models. In adversarial training, the objective is to find parameters which match the statistics of empirical data, using a pair of competing neural networks -a generator and discriminator. The generator maps the distribution of some input random variable onto the empirical data distribution to try and fool the discriminator. The discriminator attempts to classify input data as samples from the true data distribution or from the generator. This approach has been used extensively to produce realistic images and for text generation. Recently, Molano-Mazon et al. trained a generative model of spike trains, and Arakaki et al., rate models of neural populations, using GANs. However, to the best of our knowledge, adversarial training has not yet been used to train spiking models which produce discrete outputs and which aim to capture both the stimulusdependence of firing rates and shared variability. We propose to use conditional GANs for training neural encoding models, as an alternative to likelihood-based approaches. A key difficulty in using GANs for neural population data is the discrete nature of neural spike trains: Adversarial training requires calculation of gradients through the generative model, which is not possible for models with a discrete sampling step, and hence, requires the application of gradient estimators. While many applications of discrete GANs use biased gradient estimators based on the concrete relaxation technique, we find that unbiased gradient estimators REINFORCE and REBAR lead to better fitting performance. We demonstrate our approach by fitting a convolutional neural network model with shared noise sources to multi-electrode recordings from V1. We want to train a GAN, conditioned on the visual input, to generate multivariate binary spike counts y which match the statistics of empirical data. We model binary spike trains (i.e. each bin t, neuron n and trial i corresponds to an independent draw from a Bernoulli distribution) conditioned on the stimulus x and latent variable z which induces shared variability. We use a convolutional neural network (CNN) f with parameters θ to capture the mapping from x and z to the firing rate λ (Fig. 1), with a sigmoid nonlinearity σ in the last layer, Generator architecture The discriminator D (Fig. 1) receives both the stimulus and the corresponding (simulated or experimental) spike trains. It uses a CNN (similar in architecture to the generator) to embed the stimulus, and combines it with the spike train via fully connected layers. For each timebin and trial, D outputs the probability of the input spike train being real. The objective is to find the Nash equilibrium of a minmax game between the generator G and the discriminator D, Due to this objective, GANs are notoriously challenging to train as the training algorithm is sensitive to the gradients with respect to the discriminator parameters. We used the cross-entropy objective as in equation 3, but constrained the discriminator gradients using spectral normalisation, and employed gradient norm clipping for the generator gradients. Dealing with discrete data Obtaining the gradients for the generator requires backpropagation through both generator and discriminator networks. Most applications of GANs have been on continuous data. However, spikes are discrete and thus, the generator has a discrete sampling step which blocks backpropagation. Previous attempts to overcome this problem include using concrete relaxation, REINFORCE or REBAR to estimate gradients. Concrete relaxation approximates the binary variables as continuous values which are close to 0 and 1. This allows backpropagation through the sampling step, but leads to biased gradients. The REINFORCE gradient estimator provides unbiased but high-variance gradients using the log-derivative trick. The REBAR gradient estimator combines concrete relaxation and REINFORCE by using the relaxed outputs as a control variate to the REINFORCE gradient. For the applications and hyper-parameter settings we used, we found that both REINFORCE and REBAR performed better than concrete relaxation, and that REBAR did not have much improvement over REINFORCE (Fig. 2). Architecture and dataset We fit the GANs to a publicly available dataset, consisting of 69 cells recorded from macaque V1, while animals watched a 30s movie repeated 120 times. The movie consisted of 750 frames of size 320 x 320 pixels, which we downsampled to 27 x 27 pixels. We binned the spikes at a 40ms resolution, and binarized the ing spike trains. Since only 5% of the spike counts in each bin were non-binary after re-binning, we assumed that the binarization would not significantly alter the . For the generator, which receives 10 consecutive movie frames of size 27 x 27 pixels as input, we used a 3-layer CNN architecture similar to Kindel et al. (Fig. 1A) -layers 1+2: convolutional with 16 and 32 filters, size 7 by 7, each followed by a MaxPool layer with kernel size 3 and stride 2, followed by LeakyRELUs with slope 0.2. The final layer of the CNN was a fully connected layer with units equal to the number of neurons in the dataset. To capture the stimulus-independent variability shared between the neurons, we added Gaussian white noise to the units of the convolutional layers. The noise was shared between the units of these layers, multiplied by a separate weight for each unit. The discriminator network consisted of a CNN embedding for the stimulus, similar in structure to the generator, but without the added shared noise (Fig. 1B), and 5 fully connected ReLU layers. Training We trained the two networks in parallel for 15k epochs, each consisting of 2 discriminator and 1 generator update. With batch size 50, we used ADAM with learning rate 0.0001, β 1 = 0.9 and β 2 = 0.999 to optimise the network parameters. The first 650 timebins were used for training the networks and the last 100 timebins for validation. All hyper-parameters were set by hand. We fit a 3-layer CNN generative model of binary spike counts to neural population data recorded in the V1 area of macaque visual cortex, using adversarial training as described above. For comparison, we fit a CNN with a similar architecture to the GAN generator -but without the shared noise layersto the same dataset, using supervised learning, i.e. by optimizing the cross-entropy between predicted firing probabilities and experimentally observed spike trains. We also fit a Dichotomised Gaussian (DG) model, which explicitly represents shared variability via the covariance matrix of a multi-variate Gaussian distribution. On the training data, all approaches reproduced the gross structure in the spike train rasters (Fig. 3A) and accurately captured the firing rates (here: spike-probabilities per bin, Fig. 3B left). However, the supervised model did not accurately reproduce total pairwise correlations between the neurons (Fig. 3B center), since its noise-correlations 3C) are constrained to be 0. Thus, the histogram of population spike counts for data generated from the supervised model is also substantially different from that of (Fig. 3B right). The DG model accurately captured the total correlations and the pairwise noise-correlation matrix since it designed to match the peri-stimulus time histogram (PSTH) and the noise-correlations of the spike trains. However, it did not perfectly capture the population spike count histogram, as it only models second order correlation between neurons, while the population spike count histogram also depends on higher order correlations. In contrast, the GAN generator, with the addition of just a few shared noise parameters to the supervised model, was able to accurately capture the total correlation, the population spike histogram and pairwise noise-correlation matrix. However, we found that neither model generalised well to the held-out test-dataset, possibly because of the short training-set and high variability of this dataset. When we fit a CNN with the exact same architecture to simulated data with higher SNR and the same dimensions as the V1 dataset, we found that the CNN was able to capture the PSTH and SNR in the test data (Fig. 4). On the V1 dataset, the adversarially trained CNN and the DG model were similarly good in reproducing correlations and spike count histograms. This might occur if the spike trains of the neural population are homoscedastic (i.e. the variability does not depend on time or stimuli), as assumed by the DG model. Adversarial training is limited only by the flexibility of the network, and can also capture heteroscedastic, i.e. stimulus-dependent noise. Hence, we simulated heteroscedastic data with the same dimensionality as the V1 dataset. We added latent noise to the second layer of the CNN, with variance proportional to the squared input from the previous layer. We fit both a CNN and a DG model to this simulated dataset. We found that CNN trained with the GAN framework was able to accurately capture the covariances estimated from'low signal' and'high signal' timebins separately (Fig. 5), unlike the DG model. We here showed how adversarial training of conditional generative models that produce discrete outputs (i.e. neural spike trains) can be used to generate data that matches the distribution of spike trains recorded in-vivo, and in particular, its firing rates and correlations. We used unbiased gradient estimators to train conditional GANs on discrete spike trains and spectral normalisation to stabilise training. However, training of discrete GANs remains sensitive to the architecture of the discriminator, as well as hyper-parameter settings. We showed that we are able to successfully train adversarial models in cases where supervised and Dichotomised Gaussian models fail. In future, adversarial training could be used to capture higher-order structure in neural data, and could be combined with discriminators that target certain statistics of the data that might be of particular interest, in a spirit similar to maximum entropy models. Similarly, this approach could also be extended to capture temporal features in neural population data such as spike-history dependence or adaptation effects. Since we condition the discriminator on the input stimulus, adversarial training could be used for transfer learning across multiple datasets. Generative models trained this way to produce realistic spike trains to various input stimuli, may be used to probe the range of spiking behaviour in a neural population under different kinds of stimulus or noise perturbations."

We introduce the Continuous Logic Network (CLN), a novel neural architecture for automatically learning loop invariants and general SMT formulas.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: Single cell RNA sequencing (scRNAseq) technology enables quantifying gene expression profiles by individual cells within cancer. Dimension reduction methods have been commonly used for cell clustering analysis and visualization of the data. Current dimension reduction methods tend overly eliminate the expression variations correspond to less dominating characteristics, such we fail to find the homogenious properties of cancer development. In this paper, we proposed a new and clustering analysis method for scRNAseq data, namely BBSC, via implementing a binarization of the gene expression profile into on/off frequency changes with a Boolean matrix factorization. The low rank representation of expression matrix recovered by BBSC increase the resolution in identifying distinct cell types or functions. Application of BBSC on two cancer scRNAseq data successfully discovered both homogeneous and heterogeneous cancer cell clusters. Further finding showed potential in preventing cancer progression. Cancer the biggest deadly threat to human has been a huge puzzle since its determination in 1775. From once considered as contagious to nowadays cancer immunotherapy, the modern medication continues to evolve in tackling this problem . And yet, not enough to make a huge difference, 1,762,450 people have been diagnosed with cancer and 606,880 has died in 2018 . The development of single cell RNA sequencing (scRNA-seq), which measures each single cell in cancer tissue with over 20,000 dimension of genes (features), picturized the hologram of cancer and its micro-environment with high resolution (; ;). As illustrated in Figure 1A, classic analysis pipeline takes a linear (PCA) or non-linear (t-SNE) dimension reduction of the high dimensional input data, by which loadings of the top bases are further used for cell clustering and visualization . Figure 1: Classic analysis pipeline for scRNA-seq data and Melanoma example Cancer cell heterogeneity hampers theraputic development. We use the melanoma dataset as an example. Cells in a scRNA-seq data are always with multiple crossed conditions, such as types of cancer, origin of patients and different cell types. By analyzing melanoma scRNA-seq data with classic pipeline, we differentiated the cell type of each cell in its cancer microenvironment (CME) (figure 1B). All cell types other than cancer cell are constituted by multiple patients (figure 1C), validated the accuracy of classic pipeline in cell type identification. While on cancer cell, each patient forms a distinct cluster (highlighted in shadow), suggesting confounding patient-wise heterogeneity. Similar phenomenon also exists in breast cancer and head and neck cancer. On the other hand, being an investment-heavy industry like medical industry, the uniqueness of each cancer patient contradicts its general principle as to In addition, f follows a beta distribution accounting for the collective effect of the probability to shift the expression from off to on (k on) and from on to off (k of f). y denotes the true expression of gene i inside cell j and x is the observation of y with Gaussian error. Recent study revealed that, regulated by enhancers, burst frequency f is the major facilitator of cell type specific gene expression landscape . Though f and k size cannot be precisely fitted from our observed data, since y follows the Poisson distribution of the pure product of k size and f, we could still capture the most significant frequency changes across different cells. That is, we could infer whether f is above or equal to zero, corresponding to expression/no-expression of the gene, from our observed data. Counting this property, we thus propose the following approximate gene expression bi-state models. where F denotes a latent binary matrix of f, which is considered as a low rank representation of k different cell types, generated by the Boolean product of two binary matrix A and B plus a Boolean flipping error E. Y denotes the true quantitative expression level generated from F, and X is considered as a measure of Y with i.i.d. Gaussian error. Here our approach takes the approximating Y by Hadamard product between X andÂ n×k ⊗B k×m, i.e. whereÂ n×k andB k×m are the estimation of A n×k and B k×m. Bi-state and Boolean matrix factorization for scRNA-seq data (BBSC). In sight of this, we developed a novel scRNA-seq pattern mining and analysis pipeline namely BBSC (Figure 2), by implementing a data binarization process for the inference of ON/OFF bi-state expression patterns. In addition, we proposed a fast binary matrix factorization (BMF) method, namely PFAST, adapting to the large scale of scRNA-seq data. BBSC can be easily implemented with classic dimension reduction based analysis procedure. Application of BBSC on scRNA-seq of the head and neck cancer and melanoma data successfully revealed the cancer homogeneity hence increased the sensitivity in identifying sub types of cells. In addition, cancer cell clusters expressing the epithelial mesenchymal transition (EMT) markers were specifically identified by BBSC in head and neck cancer study, which consist cancer cells from different patient samples, suggesting heterogeneous cancer cells may adopt a similar strategy in cancer metastasis process. We summarize our contributions as follows: • We constructed a scRNA-seq analysis pipeline, BBSC, for retrieving cancer homogeneity properties. BBSC is by far the first analysis pipeline accounting the fundamental interplay between cell type and gene expression in the analysis of scRNA-seq data. • As a major component in BBSC pipeline, we proposed a fast and efficient BMF algorithm, PFAST, in adapting to the large scale of scRNA-seq data. • In the analysis of head and neck cancer data, BBSC identified that cancer cell may adapt similar strategies in metastasis. This finding could be applied to prevent cancer progression. So far, two strategies have been used to optimize the classic pipeline for scRNA-seq data analysis: using extra information to supervise the dimension reduction, such as CITE-seq and REAP-seq data combining scRNA-seq with additional protein information or a recent work by , by maximizing the similarity with bulk RNA seq data for scRNAseq imputation; and limiting analysis to the genes known to be related with desired biological features. Both strategies require substantial prior information that is either expensive or unsuitable for studying biological characterization. In this paper, we developed a new strategy rooted from a perspective that differences in cell types and physiological states correspond to different bi-state frequency patterns, which could retrieve effectively by Boolean matrix factorization. Following the Boolean algebra, BMF decomposes a binary matrix as the Boolean product of two lower rank binary matrices and has revealed its strength in retrieving information from binary data. Due to the NP completeness of the BMF problem, several heuristic solutions have been developed, among which two series of works are most frequently utilized . One is ASSO algorithm developed by. ASSO first generates potential column basis from row-wise correlation. Then adopts a greedy searching from generated basis for the BMF fitting. The second series of work is the PANDA algorithm developed by. PANDA aims to identify the top 1-enriched submatrices in a binary matrix from noise. In each iteration, PANDA excludes the current fitting from the input matrix and retains a residual matrix for further fitting. More recently, Bayesian inference has involved in this field. retrieve patterns from factor-graph model by deriving MAP using message passing (denoted MP). proposed OrMachine, provide full probabilistic inference for binary matrices. While ASSO and PANDA being regarded as the baseline in BMF, MP and OrMachine represent state-of-the-art performance. As shown in Figure 2, we implemented a data binarization and PFAST algorithm to constrain scRNA-seq data before a regular dimension reduction based analysis, which forms a new analysis pipeline namely BBSC. BBSC first binarizes the input data via the on/off expression states of each gene. The approximated matrix, namely recover matrix, is further constructed by the Hadamard product of the original expression matrix and the BMF fitted binary matrix. Regular dimension reduction and cell clustering analysis is then conducted on the recovered matrix. Figure 3: Infer F from scRNA-seq data To determine a gene is truly expressed or not is to examine X ij on. Empirically, we assume the lowest none zero expression value of each gene approximates the distribution of. Since type I error is far damaging than type II error in biological experiments, we utilized the 95% quantile of distribution as the threshold of ON expression state, i.e., gene expression above the threshold is considered as f > 0 while expression below the threshold is considered as with an OFF state, i.e. f = 0. We applied this binarization procedure on two high quality scRNAseq cancer datasets of head and neck cancer (Figure 3A) and melanoma (Figure 3B). To justify the threshold of ON/OFF state computed in this way, we compared the representation of data in the lower dimension by the overall silhouette score, which measures the similarity of each data point to own cluster compare to others. The overall silhouette score represents the goodness of the clustering. Note that cell cluster information is retrieved directly from original paper. In both datasets, the binarization approach significantly increased the performance of cluster representation, suggesting our binarization can remove true noise and still maintains the biological information. We developed a fast and efficient BMF algorithm, namely PFAST to cope with the large scale of modern data of interest. PFAST follows the general framework of PANDA algorithm. In each iteration, PANDA has two main sub functions, core pattern discovery (Core) and extensions of a core pattern (Core ext). Core finds the most enriched square of 1s under current residual matrix. Core ext expands the generated core patterns with not included area. To find most precise patterns amid noise, PANDA calculates global loss at each step. Though PANDA only works on the residual matrix in each iteration, it still involves already generated patterns for calculating loss. This look back property and global loss calculation may play a major role in decomposing noisy binary data. However, the associated computational pressure makes PANDA inapplicable for large-scale scRNAseq data. Fortunately, during our binarization process, 95% of noise has been eliminated, which compensates an extensive binary pattern mining as PFAST. Unlike PANDA, PFAST only focus on the loss in a local scale. Moreover, PFAST abolished the look back property, only focus the loss decrease for current pattern. Taken together, PFAST is an extensive BMF algorithm. Each iteration of PFAST has a computational complexity of O(mn). Like PANDA, PFAST will only work iteratively on residual matrix that has not been covered by any identified patterns before hitting the convergence criteria. The choice of convergence criteria can be modified for different needs. The popular convergence criteria are set by identifying top k patterns or covering certain proportion of the non-zero values in the matrices. Detailed algorithms of PFAST is illustrated below: Algorithm 1: PFAST Inputs: Binary matrix F, Threshold t, and τ Outputs: A ∈ {0, 1} n×k, B ∈ {0, 1} k×m P F AST (F, t, τ): Since OrMachine has been deprecated, we compared the performance of PFAST with ASSO, PANDA,and MP on simulated datasets. We simulated binary matricesX n×m = U n×k ⊗ V k×m where each element of U and V follows an identical Bernoulli random variable. In the simulation, we set n = m = 1000, k = 5, and two signal level p = 0.2/0.4, corresponding to sparse and dense matrix. We compared the performance with three criterion: reconstructed error, sparsity, and time cost. Specifically, reconstructed error measures the overall fitting of each method, and sparsity measures the parsimonious level of the pattern matrices. Detailed definition of reconstructed error and sparsity are given below. Intuitively, a good binary matrix factorization should have small reconstructed error and proper sparsity level. To the best of our knowledge, the conditions to guarantee Algorithm 2: PFAST core Inputs: Residual matrix F r Outputs: a ∈ {0, 1} n, b ∈ {0, 1} m P F AST core(F r): s = {s 1, ..., s n} ← sorting based on row-wise sum a unique solution of the BMF problem have not been theoretically derived, thus we do not directly compare the factorized and true pattern matrices directly, i.e., U vs A *, and V vs B *, where A * and B * denote the pattern matrices decomposed by the three different algorithms. Note that ASSO and PFAST require one additional parameter as a standard input. To achieve a fair comparison, we tested different parameters for each method and used the parameter with the best performance for the comparison. The convergence criteria for all the methods were set as when 5 patterns were identified, corresponds to the true rank of simulated matrices; identified patterns already covers 95% of the non-zero values. All the experiments ran on the same laptop with i7-7600U CPU and 16 GB memory. We conducted the evaluation for 10 times, detailed are shown in Figure 4. The definitions of reconstructed error and sparsity are Comparing to ASSO, PANDA, and MP, our analysis showed that PFAST achieved superior performance in both sparse and dense matrices. The running time of PFAST is significant lower than all other methods. We also observed better convergence of PFAST. ASSO tended to find the most inclusive patterns so that they usually converged with very few dense patterns. PANDA was designed to identified significant patterns from noise. Its low tolerance to noise caused a relative slow pace in convergence. MP revealed its robustness in fitting binary data. However, it has the highest computational cost compared to others. The performance of PFAST demonstrated its balanced computational cost and fitting accuracy. With the significant improvement of speed, PFAST still manages to We applied classic tSNE-based dimension reduction and BBSC analysis on the head and neck cancer and melanoma data sets, as detailed below. For both datasets, we recovered the bi-state model of data by binarizing the expression matrix into ON/OFF expression state with 95% Gaussian noise quantile. PFAST was applied on the binary matrices with threshold setting to 0.6. The choice of convergence criteria can vary according to different needs. Here, we set convergence as 1) top 10 patterns have been identified, 2) 40% of non-zero values has been recovered. The rationale here is that scRNA-seq data is overall sparse. It usually cost extensive patterns to achieve a small reconstructed error. However, the later discovered patterns introduced more bias, where the later patterns are more likely to be related to other factors rather than cell type. Empirically, top 10 patterns and 40% cutoff achieve better cell type identification ability. In analyzing the head and neck cancer and melanoma data sets, it ed in 5 and 10 patterns respectively. In both analysis pipeline, we conducted dimension reduction using t-SNE with perplexity setting to 30 with 20000 max iterations. It is noteworthy that no cell clustering was made in this analysis. All the cell type annotation and patient information were directly retrieved from the original paper. As illustrated in Figure 5A,E, the 2D embedding achieved from the classic pipeline well separated cells by their phenotypic types. Fibroblast, T-, B-, myeloid and cancer cells et al forms distinct individual clusters. Further analysis of the association between cell groups and patient information confirmed same type of the immune and stromal cells from different patients form one cell group, while the cancer cells are grouped by specific patient over the 2D embedding (Figure 5B,F). These observations are consistent with original work. On the other hand, on the 2D embedding of the BBSC pipeline, cell of different phenotypic types form into distinct groups. Comparing to the classic pipeline, BBSC retrieved data generated more groups of subtypes of Fibroblast, T cells and cancer cells (Figure 5C,G). The split cell groups identified by BBSC show higher association with intra-cancer heterogeneity. We further investigated the association between the patient origin and cell group over the 2D embedding of the BBSC data (Figure 5D,H). Interestingly, in both datasets, we observed several cell groups, marked with yellow circles, that are constituted by cancer cells for different patients. These cancer cell groups correspond to the common sub cell populations prevalently shared by cancer tissues with different patients, which may suggest hallmark functions developed in the disease progression. To identify the functional characteristics of BBSC derived cell groups, we checked the differentially expressed genes associated with the cell groups of cancer cells. We first achieved five distinct clusters of the cancer cells over the 2D embedding of the BBSC retrieved data by using k-mean method (Figure 6A). Figure 6B illustrates the newly clustered cancer cell in the 2D embedding derived by the classic tSNE method. The cluster 1 and 2 are formed by cells from different patients while the cluster 3 to 5 were associated with specific patients. We identified significant differential expression of epithelial-mesenchymal transition (EMT) marker genes among the five clusters (Figure 6C). EMT is regarded as a hallmark event in cancer cells metastasis approach for carcinomas such as head and neck cancer. Under this process, cancer cells lose their epithelial properties and become mesenchymal-like cells with higher migratory capabilities for escaping the cancer tissue into circulating system. We identified the cluster 1 and 2 behaved distinct difference compared with cluster 3 to 5 on EMT marker genes. Cells in the cluster 1 and 2 are with overly expressed mesenchymal markers such as CDH3, TGFB1, ITGB6 and VIM. While the cluster 3 to 5 overly express epithelial markers genes such as CDH1, CLDN4, CLDN7, KRT19 and EPCAM. Our analysis clearly demonstrated the BBSC substantially removed inter-cancer heterogeneity that enables the identification of cancer cells from different patients with common functional characteristics. More importantly, the observation also suggests though cancer cell are very different in each patient, they ought to take similar strategy in the metastasis process. Targeting the progression strategy revealed in this study may have huge therapeutic impact in preventing cancer progression. Enabled by the development of single cell technology, we now can observe the complicated biological process like cancer with unprecedented resolution. However, the classic analysis pipeline fails to deliver detailed information: 1) it does not reveal common characteristic of cancer cell in different cancer patients. 2) Even it separates functional cells; it fails to reveal intra-cluster heterogeneity. To solve above problems, we have developed BBSC analysis pipeline. Rooted from casting the frequency change in gene expression, we have applied BMF in the feature selection process, which avoids adding new expensive and potentially noisy information. We have applied tailored binarizing process for each dataset. Moreover, to deal with big scale tall matrix like scRNAseq data, we have developed a fast and efficient algorithm called PFAST. Letting alone its fast speed in handling large-scale data, it shows high accuracy compared with state-of-art BMF algorithms. We have applied BBSC on two high quality cancer studies, head and neck cancer and melanoma. In both datasets, BBSC shutters the big clusters into several sub clusters, and promotes a gateway to analysis intra-cluster heterogeneity. Moreover, BBSC manages to get common cancer sub cell clusters in both datasets, and decreases the patient-wise heterogeneity that hindered cancer therapeutic development. We next have justified the biological meanings of BBSC derived sub clusters by looking into the sub cancer clusters in head and neck cancer. By analyzing their detailed expression profile, We find out that the common clusters are in the EMT transition process indicating these cancer cells play an important part in cancer metastasis. While patient specific clusters are in the early EMT process indicating that these cells are still in the original cancer micro environment. These findings have first justified the biological importance of BBSC derived sub clusters. Secondly, it brings much insightful ideas in the clinical application. We now can hypothesize that when cancer cells seek metastasis, they will transform into similar states that are common across different patients. The characteristic of the common clusters may serve as target in preventing cancer metastasis. Furthermore, we validate that the heterogeneity of cancer comes from the original cancer tissue. Also BBSC shows promising in deciphering this kind of heterogeneity. Especially in head and neck cancer study, BBSC distinctly divides cancer cells from the same patient into two sub clusters. Due to our limited expertise in cancer biology, we did not look closely in this property. However, we believe this would bring insightful ideas in the cause of cancer origin heterogeneity. Overall BBSC is an efficient and valuable analysis platform for scRNAseq or other single cell data. It is capable to bring insightful knowledge for our detailed understanding of complicated biological process.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Our finding shed lights in preventing cancer progression

Represent the TLDR for finding the corresponding scientific article

Considering neural network optimization process as a model selection problem, we introduce a biological plausible normalization method that extracts statistical regularity under MDL principle to tackle imbalanced and limited data issue.


Question: Represent the scientific article to find a short TLDR

Question: While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We display proofs of concept under 2 flow architectures: discrete autoregressive flows enable bidirectionality, allowing for example tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows (i.e., with layer structure from RealNVP) enable parallel generation such as exact nonautoregressive text modeling. There have been many recent advances in normalizing flows, a technique for constructing high-dimensional continuous distributions from invertible transformations of simple distributions BID22 BID25 BID23. Applications for high-dimensional continuous distributions are widespread: these include latent variable models with expressive posterior approximations BID22 BID20 BID12, parallel image generation BID6 BID11, parallel speech synthesis BID19, and general-purpose density estimation BID18.Normalizing flows are based on the change-of-variables formula, which derives a density given an invertible function applied to continuous events. There have not been analogous advances for discrete distributions, where flows are typically thought to not be applicable. Instead, most research for discrete data has focused on building either latent-variable models with approximate inference BID2, or increasingly sophisticated autoregressive models that assume a fixed ordering of the data BID0 BID26. In this paper, we present an alternative for flexible modeling of discrete sequences by extending continuous normalizing flows to the discrete setting. We demonstrate proofs of concept of discrete flows with two architectures:1. Discrete autoregressive flows enable multiple levels of autoregressivity. For example, one can design a bidirectional language model of text where each token depends on both left-to-right and right-to-left contexts while maintaining an exact likelihood and sampling.2. Discrete bipartite flows (i.e., with flow structure similar to RealNVP BID6) enable flexible models with parallel generation. For example, one can design nonautoregressive text models which maintain an exact likelihood for training and evaluation. Bidirectional models. Classically, bidirectional language models have been pursued but require approximate inference BID16. Unlike bidirectional models, autoregressive models must impose a specific ordering, and this has been shown to matter across natural language processing tasks BID7 BID28. Bidirectionality such as in encoders have been shown to significantly improve in neural machine translation BID3. Most recently, BERT has shown bidirectional representations can significantly improve transfer tasks BID4. In this work, discrete autoregressive flows enable bidirectionality while maintaining the benefits of a (tractable) generative model. Nonautoregressive models. There have been several advances for flexible modeling with nonautoregressive dependencies, mostly for continuous distributions BID5 BID11. For discrete distributions, BID21 and BID24 have considered retaining blockwise dependencies while factorizing the graphical model structure in order to simulate hierarchically. BID8 and BID10 apply latent variable models for fast translation, where the prior is autoregressive and the decoder is conditionally independent. BID13 adds an iterative refinement stage to initial parallel generations. In this work, discrete bipartite flows enable nonautoregressive generation while maintaining an exact density-analogous to RealNVP advances for image generation BID6. Normalizing flows transform a probability distribution using an invertible function BID25 BID22 BID23. Let x be a D-dimensional continuous random variable whose density can be computed efficiently. Given an invertible function f: R D → R D, the change-of-variables formula provides an explicit construction of the induced distribution on the function's output, y = f (x): DISPLAYFORM0 The transformation f is referred to as a flow and x is referred to as the base distribution. Composing multiple flows can induce further complex distributions. For an arbitrary invertible f, the determinant of the Jacobian incurs an O(D 3) complexity, which is as costly as modeling with a full rank covariance matrix. Thus, normalizing flows are designed so that the determinant of the flow's Jacobian can be computed efficiently. Here, we review two popular flow transformations. Autoregressive flows. Autoregressive functions such as recurrent neural networks and Transformers BID26 have been shown to successfully model data across modalities. Specifically, assume a base distribution x ∼ p(x). With µ and σ as autoregressive functions of y, i.e. DISPLAYFORM0, and σ d > 0 for all d, the flow computes a location-scale transform BID18, DISPLAYFORM1 The transformation is invertible and in fact, the inverse can be vectorized and computed in parallel: DISPLAYFORM2 In addition to a fast-to-compute inverse, the autoregressive flow's Jacobian is lower-triangular, so its determinant is the product of the diagonal elements, d=1 σ d. This enables autoregressive flow models to have efficient log-probabilities for training and evaluation. Bipartite flows. Real-valued non-volume preserving (RealNVP) flows are another transformation BID6. For some d < D, RealNVP coupling flows follow a bipartite rather than autoregressive factorization: DISPLAYFORM3 where σ and µ are functions of x 1:d with σ > 0. By changing the ordering of variables between each flow, the composition of RealNVP flows can learn highly flexible distributions. RealNVP flows have a lower-triangular Jacobian where its determinant is again the product of diagonal elements, D i=d+1 σ i. RealNVP flows are not as expressive as autoregressive flows, as a subset of variables don't undergo a transformation. However, both their forward and inverse computations are fast to compute, making them suitable for generative modeling where fast generation is desired. Normalizing flows depend on the change of variables formula (Equation 1) to compute the change in probability mass for the transformation. However, the change of variables formula applies only to continuous random variables. We extend normalizing flows to discrete events. Let x be a discrete random variable and y = f (x) where f is some function of x. The induced probability mass function of y is the sum over the pre-image of f: DISPLAYFORM0 where f −1 (y) is the set of all elements such that f (x) = y. For an invertible function f, this simplifies to FORMULA0 ). It is the same but without the log-determinant-Jacobian. Intuitively, the log-determinant-Jacobian corrects for changes to the volume of a continuous space; volume does not exist for discrete distributions so there is no need to adjust it. Computationally, Equation 4 is appealing as there are no restrictions on f such as fast Jacobian computations in the continuous case, or tradeoffs in how the log-determinantJacobian influences the output density compared to the base distribution. DISPLAYFORM1 Next we develop discrete invertible functions. To build intuition, first consider the binary case. Given a D-dimensional binary vector x, one natural function applies the XOR bitwise opera- tor, Example. Let D = 2 where p(x) is defined by the following probability table: DISPLAYFORM0 DISPLAYFORM1 0.63 0.07 DISPLAYFORM2 The data distribution cannot be captured by a factorized one p(x 1)p(x 2). However, it can with a flow: set f (x 1, x 2) = (x 1, x 1 ⊕x 2); p(x 1) with probabilities [0.7, 0.3]; and p(x 2) with probabilities [0.9, 0.1]. The flow captures correlations that cannot be captured alone with the base. More broadly, discrete flows perform a multi-dimensional relabeling of the data such that it's easier to model with the base. This is analogous to continuous flows, which whiten the data such that it's easier to model with the base (typically, a spherical Gaussian).Modulo location-scale transform. To extend XOR to the categorical setting, consider a Ddimensional vector x, each element of which takes on values in 0, 1,..., K − 1. One can perform location-scale transformations on the modulo integer space, DISPLAYFORM3 Here, µ d and σ d are autoregressive functions of y taking on values in 0, 1,..., K −1 and 1,..., K − 1 respectively. For this transformation to be invertible, σ and K must be coprime (an explicit solution for σ −1 is Euclid's algorithm). An easy way to ensure coprimality is to set K to be prime; mask noninvertible σ values for a given K; or fix σ = 1. Setting K = 2 and σ = 1, it's easy to see that the modulo location-scale transform generalizes XOR. The idea also extends to the bipartite flow setting: the functions (µ, σ) are set to for a subset of the data dimensions, and are functions of that subset otherwise. Example. FIG2 illustrates an example of using flows to model correlated categorical data. Following BID15, the data is drawn from a mixture of Gaussians with 8 means evenly spaced around a circle of radius 2. The output variance is 0.01, with samples truncated to be between −2.25 and 2.25, and we discretize at the 0.05 level. A factorized base distribution cannot capture the data correlations, while a single discrete flow can. (Note the modulo location-scale transform does not make an ordinal assumption. We display ordinal data as an example only for visualization; other experiments use non-ordinal data.) With discrete flow models, the maximum likelihood objective per datapoint is log p(y) = log p(f −1 (y)), DISPLAYFORM0 7.7 7.6 8.0 7.9 D = 5, K = 10 10.7 10.3 11.5 10.7 D = 10, K = 5 15.9 15.7 16.6 16.0 Table 1: Negative log-likelihoods for the full rank discrete distribution (lower is better). Autoregressive flows improve over its autoregressive base. Bipartite flows improve over its factorized base and achieve nats close to an autoregressive distribution while remaining parallel. Autoregressive Base Autoregressive Flow 15.05 5.81 Table 2: Negative log-likelihoods on the square-lattice Ising model (lower is better). Higher coupling strength corresponds to more spatial correlations.where the flow f has free parameters according to its autoregressive or bipartite network, and the base distribution p has free parameters as a factorized (or itself an autoregressive) distribution. Gradient descent with respect to base distribution parameters is straightforward. To perform gradient descent with respect to flow parameters, one must backpropagate through the discrete-output function µ and σ. We use the straight-through gradient estimator BID1. In particular, the (autoregressive or bipartite) network outputs two vectors of K logits θ d for each dimension d, one for the location and scale respectively. On the forward pass, we take the argmax of the logits, where for the location, DISPLAYFORM1 Because the argmax operation is not differentiable, we replace Equation 6 on the backward pass with the softmax-temperature function: DISPLAYFORM2 As the temperature τ → 0, the softmax-temperature becomes close to the argmax and the bias of the gradient estimator disappears. However, when τ is too low, the gradients vanish, inhibiting the optimization. Work with the Gumbel-softmax distribution indicates that this approximation works well when the number of classes K < 200 BID14 BID9, which aligns with our experimental settings; we also fix τ = 0.1. In addition to the experiment in FIG2, we perform three toy experiments to show the utility of discrete autoregressive flows and discrete bipartite flows. For discrete autoregressive flows, we used an autoregressive Categorical base distribution where the first flow is applied in reverse ordering.(This setup lets us compare its advantage of bidirectionality to the baseline of an autoregressive base with 0 flows.) For discrete bipartite flows, we used a factorized Categorical base distribution where the bipartite flows alternate masking of even and odd dimensions. A natural experiment is to analyze the expressivity of the flows for an arbitrary discrete distribution. In particular, we sample a true set of probabilities for all D dimensions of K classes according to a Dirichlet distribution of size K D − 1, α = 1. For the network for both the base and flows, we used a Transformer with 64 hidden units. Table 1 displays negative log-likelihoods (nats) of trained models over data simulated from this distribution. Across the data dimension D and number of classes K, autoregressive flows gain several nats over the autoregressive base distribution, which has no flow on top. Bipartite flows improve over its factorized base and in fact obtain nats competitive with the autoregressive base while remaining fully parallel for generation. Addition. , we examine an addition task: there are two input numbers with D digits (each digit takes K = 10 values), and the output is their sum with D digits (we remove the D + 1 th digit if it appears). Addition naturally follows a right-to-left ordering: computing the leftmost digit requires carrying the remainder from the rightmost computations. Given an autoregressive base which poses a left-to-right ordering, we examine whether the bidirectionality that flows offer can adjust for wrong orderings. We use an LSTM to encode both inputs, apply 0 or 1 flows on the output, and then apply an LSTM to parameterize the autoregressive base where its initial state is set to the concatenated two encodings. All LSTMs use 256 hidden units for D = 10; 512 for D = 20.For D = 10, an autoregressive base achieves 4.0 nats; an autoregressive flow achieves 0.2 nats (i.e., close to the true deterministic solution over all pair of 10-digit numbers). A bipartite model with 1, 2, and 4 flows achieves 4.0, 3.17, and 2.58 nats respectively. For D = 20, an autoregressive base achieves 12.2 nats; an autoregressive flow achieves 4.8 nats. A bipartite model with 1, 2, 4, and 8 flows achieves 12.2, 8.8, 7.6, and 5.08 nats respectively. Ising Model. We examine how bidirectional generative models can be used for learning undirected models. For the base network, we used a single layer LSTM with 8 hidden units. For the flow network, we used an embedding layer with 8 hidden units. Table 2 displays negative log-likelihoods (nats) of trained models over data simulated from Ising models with varying lattice size and coupling strength. As Ising models are undirected models, the autoregressive base posits a poor inductive bias by fixing an ordering and sharing network parameters across the individual conditional distributions. Over data dimension D and coupling, autoregressive flows perform as well as, or improve upon, autoregressive base models. We describe discrete flows, a class of invertible functions for flexible modeling of discrete data. Note our experiments are only toy to show proofs of concept. We're continuing to push these ideas to larger-scale text data. We're also applying discrete inverse autoregressive flows, which enable flexible variational approximations for discrete latent variable models. One open question remains with scaling discrete flows to large numbers of classes: in particular, the straight-through gradient estimator works well for small numbers of classes such as for character-level language modeling, but it may not work for (sub)word-level modeling where the vocabulary size is greater than 5,000.

We extend autoregressive flows and RealNVP to discrete data.