Concurrency and Redis - <AUTHOR> http://santosh-log.heroku.com/2011/07/02/concurrency-and-redis/ <comment><USER2> A lot of (especially web) applications suffer from race conditions. Luckily for most of them the effects are either temporary and/or not severe.

It does not only apply to Redis but basically all data stores unless something like transactions in RDBMSs are used and even then you can get into trouble at times.

The last example still has a problem when an exception is thrown and the 'redis.del "setting_up_auth_token"' fails for whatever reason. Furthermore it has a race condition if a process wants to read the content of 'web_service_auth_token' because it sees 'setting_up_auth_token' has been set but the other process who is setting it up has not reached the redis.set line for it.

The example itself is not very good as I'd rather do a little bit of extra work (would not hurt here) than being vulnerable to failures.

Assuming the work really needs to be done strictly once - either because it's a lot or the logic requires it - you would probably have to do something like the following pseudo code:

      while (true) {
        watch("setup_key")
        v = get("setup_key")
        if v == 2 {
           break
        }
        if v == 1 {
           sleep(1)
           continue
        }
        multi()
        setex("setup_key", 10, 1) // 10s timeout
        if !exec() {
          continue // someone else got the lock in the meantime, try again
        }
        // we now have a temporary exclusive "lock"
        ... do the real work ...
        set("setup_key", 2)
      }

What does this do? Well, setup_key acts like an exclusive lock which has 3 states: non-existing, 1 and 2.

 _non-existing_ means we are the first, lets do the work. _1_ means the lock is being held, sleep and check again. _2_ means the work has been finished.

The watch(),multi(),exec() work as a CAS (really, whytf is this not a single command?) to atomically acquire the lock. If the lock is held by someone, we just sleep 1 second and try again. I use a timeout on the key in case the client holding the lock dies to prevent a deadlock.

But this is STILL not perfect. What if your setup work takes longer than the timeout specified for the temporary key? I haven't spent enough time thinking about it right now to come up with a solution if there even is any for redis, so I'll leave it as an exercise to the reader :)

To be honest: with slightly different commands/semantics, Redis could be MUCH easier to write correct applications for. Especially the "transaction" support is mediocre imho.

Conclusion: Many web applications are inherintly and severely concurrent. Concurrency is HARD to get right. There are many patterns like locks, atomic operations, transactions etc. - all with completely different semantics. Redis is singlethreaded for a reason :)

<comment><USER3> I've used counters in Redis for mutex control. If incr returns 1, it means we have succeeded. Otherwise it decrements it and tries again after 1 second until it suceeds.

Here is the code: <https://gist.github.com/1062584>

<comment><USER2> What if the decr() fails because of for example a network issue? What if the process who is holding the lock dies?

:)

<comment><USER3> Good point, never thought about decr failing!

</comment></comment></comment></comment><comment><USER4> Can't seem to comment on the blog itself, so:

> This code assumes the existence of a UUID library, which returns a unique ID > on every call. Now, if more than one process were to run, they would each > create their own temp_list's

Wouldn't this still race in sense that a second update to the SafeIps table may be overwritten by the atomic move?

(A inserts into SafeIps and generates unique temp list, B inserts into SafeIps and generates another temp list, B moves its temp list to "safe_ips", A moves its temp list to "safe_ips" => oops, we've lost B's insert.)

<comment><USER5> You have to use Redis' INCR to generate the UUIDs.

<comment><USER2> That does not help at all with the issue at hand.

The problem is not colliding UUIDs, it is a race condition between getting the list of IPs and then writing them to redis.

<comment><USER5> Oh, well in the pseudo(?) code he seems to push the content from temp lists into the main list, so it seemed that as long as you have separate temp lists for A,B you wouldn't lose anything, but I probably misread it.

But anyway, as the other comment mentions, there is some transaction support <http://redis.io/topics/transactions>

I'm not sure how much Redis will focus on transactions but if it doesn't you're out of luck or reinvent the SQL wheel on top of it.</comment></comment></comment></comment>


How Statisticians Found AirFrance Flight Two Years After It Crashed - <AUTHOR> http://www.technologyreview.com/view/527506/how-statisticians-found-air-france-flight-447-two-years-after-it-crashed-into-atlantic/?utm_campaign=socialsync&utm_medium=social-post&utm_source=facebook <comment><USER2> It was not really "statisticians" who found it.

The initial searches based their original search area on backdrift from debris found, which is _very unreliable_

Instead, if they had focused their search area on the original route and estimated flight time before crash they would have found it sooner.

THEN if you don't find it you try different things.

Their Bayesian analysis was basically considering the originally scanned area (looking for ULBs - very unreliable as well) and where they didn't find it.

I'm not trying to rain on their parade, but I think the main mistake was (not theirs - the search team) relying more on backdrift than other information (basically figure 14 here [http://isif.org/fusion/proceedings/Fusion_2011/data/papers/1...](http://isif.org/fusion/proceedings/Fusion_2011/data/papers/140.pdf) \- the highest probability area is, obviously, near the last known position)

</comment><comment><USER3> The biggest problem with MH370 is that there is not a single "higer quality" location datapoint available. In AF447 you had dead bodies and iirc debris, which at least helped to roughly identify the area of the crash site.

With MH370, the possibilities are next to infinite. Nothing is known except radio data with dozens kilometers of accuracy.

<comment><USER2> Correct

With AF447 you had the exact position at approximately 10min before the crash (and debris).

With MH370 and the Inmarsat data you have something that is _much more uncertain_

</comment></comment><comment><USER4> This was actually sent round a couple of weeks ago where I work by one of our executives (I work at a place that has Statistics in its name).

I didn't mention it then, because I didn't want to be a debbie downer and hurt everyone's "rah rah statistics yay!" feeling. Or send something round that contradicts an executive :P But the paper seemed dodgy as hell to me.

This wasn't solved with Statistics. Nor was it solved by Bayesian statistics. It was solved when people did lots of searching, failed to find anything for a while despite searching where the plane was, and then they eventually found the plane near its last known location despite having already searched there.

If anything, the paper was just an application of the texas sharpshooter fallacy. I believe the authors made several models, and then included the one in the paper that showed the result they wanted.

See paper: [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.370...](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.370.2913&rep=rep1&type=pdf)

Indeed, see the graphical representation of the probability distribution in the paper, you'll notice that the plane was found in a section of the probability distribution that had already been searched earlier: an area that should now have a lower probability than the opposite half of the probability distribution if we're using semi-reasonable bayesian techniques. Indeed, this is what we see in figure seven.

In figure 8, we see their new probability distribution under the assumption that the beacons did not work at all, where they try to say their method pinpointed where the plane was. But since this was not known at the time, and figure 7 is the far more reasonable bayesian model given previous searches in the area (because figure 8 assumes a 100% probability of both beacons failing, something practically no bayesian would do), i posit that they either made that model after the fact, or they indeed had several models fitting several different scenarios, and after the plane was found, they chose the one that best fit the data post-facto. Odds are, if the plane was found, it would fit one of their scenarios, and they would then write a paper saying how their model was such a success. Figure 7 is the far more reasonable bayesian distribution, and it actually tells you to now search in the wrong area.

If they followed bayesian methods, there is in fact a bottom half of their distribution that should of been searched next (where the plane wasn't), and they in fact found the plane in an area that had already been passively searched: an area that should have downplayed by bayesian probabilities for future searches because of the unlikely-ness of this area being searched and yet still finding nothing.

I actually like Bayes, A LOT, but this is not a good example, except perhaps of the precept: if you want to find a lost plane, its probably a good idea to start looking near where it was last located.

<comment><USER5> More or Less covered this much better.

The company had already been called in once and had completely failed to find anything because their assumption was that the plane would be unlikely to be anywhere that had been searched. This was their second go and they changed their assumptions.

 _" It still was a minor miracle that we found it," says Keller._

[http://www.bbc.co.uk/news/magazine-26680633](http://www.bbc.co.uk/news/magazine-26680633)

</comment></comment><comment><USER6> For those interested in more technical details here is paper I found -

[http://isif.org/fusion/proceedings/Fusion_2011/data/papers/1...](http://isif.org/fusion/proceedings/Fusion_2011/data/papers/140.pdf)

[https://www.informs.org/ORMS-Today/Public-Articles/August- Vo...](https://www.informs.org/ORMS-Today/Public-Articles/August- Volume-38-Number-4/In-Search-of-Air-France-Flight-447)

</comment><comment><USER7> The thing that I find most interesting about this part of the AF447 story is this:

> But Stone and co chose to include the possibility that the _acoustic beacons > may have failed_ , a crucial decision that led directly to the discovery of > the wreckage. [Emphasis mine]

</comment><comment><USER8> The actual paper on the topic.

[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.370...](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.370.2913&rep=rep1&type=pdf)

</comment><comment><USER9> Another similar case of using stats and science to find something (in this case German U-boats) in the middle of ocean. [http://www.amazon.co.uk/Blacketts-War-Defeated-U-Boats- Broug...](http://www.amazon.co.uk/Blacketts-War-Defeated-U-Boats- Brought/dp/030759596X)

UK/America/other allies were suffering horrendous shipping losses caused by German U-boats. What really helped turn the tide was scientists using 'science' to predict where to find the German U-boats.

</comment><comment><USER10> Yeah, bayesian search theory is an interesting topic. A while back I wrote a program to search for intermittent bugs using it (a sort of bayesian version of 'git bisect') It hasn't seen real use (Intermittent bugs tend to involve real hardware, so you can't just try it out on any bug database) . But if anyone wants to try it out, it is at [https://github.com/Ealdwulf/bbchop](https://github.com/Ealdwulf/bbchop)</comment>


The Paxos Algorithm, or How to Win a Turing Award - <AUTHOR> https://lamport.azurewebsites.net/tla/paxos-algorithm.html <comment><USER2> I had the privilege of attending this lecture in person. One eerie thing about it was that Dr. Lamport didn't improvise _at all_ \- he just read from the slides and that was all he had to say!

Overall it was a good introduction to TLA+ which I think has a big advantage of being very simple to understand and use - no complicated math needed apart from set theory. What I didn't quite understand is how to make sure that your models are implementable in code (if you start with a model) and faithfully describe your actual code and the reality itself. Seems like all bugs will end up lurking in this gap between the model and what is actually executed and there is no reliable way to close this gap.

<comment><AUTHOR> This is a common question among TLA+ beginners.

There _can_ be bugs in the implementation. Those would either be bugs due to a mistranslation of the spec -- i.e., an error in the implementation -- or, because the spec also implicitly or explicitly describes the assumptions about the environment, a mismatch between the actual environment (hardware, software stack, user behavior) and the spec -- i.e. an error in the spec.

In both cases, having the spec makes finding those bugs easier, and in the first case, the bugs are usually cheap to find and fix. The costliest bugs are in the design, and that's where TLC, a model checker for TLA+, can automatically spot them.

Now, what if you want a formal -- i.e. mechanical -- relationship between the code and the spec? In principle, this is possible, and has been done; e.g. see [1] for C, and there have been other, similar projects for Java. In practice, we don't do it. Why? Well, we can't. There is no known technique that can verify anything but small programs/specs for arbitrary correctness properties. The reason is that the scalability of all formal methods has limitations. Currently, we can formally verify a few thousand lines of formal specification. That specification can be either code or a high-level spec. That means that we can either verify very small programs, or a high-level spec that represents a large program. The latter is the _only_ thing available to us to help with large program, and how TLA+ is normally used. Small programs are verified end-to-end (i.e., that the actual code, or even machine code, conforms to a high-level spec) only in very special, niche, cases, and/or at a great cost.

What else can we do? We can verify code not against a high-level spec, but against some specification of small, local properties. This way, every code unit is verified separately [2]. This could help catch errors in translation, but, while useful, the benefit is not as great as verifying a high-level spec. This approach, a high-level spec verified formally, translated informally to code, and the code is then verified formally against local properties only, is how Altran UK builds their software; they use the older Z, instead of TLA+ for the high-level spec, and SPARK for code-level verification. Local properties can be verified with languages/tools like SPARK, JML and its various tools for Java, ACSL and various tools for C, and others, including dependent type languages used in research.

There is also an affordable way to mechanically check a program's conformance to a TLA+ spec, but only approximately -- it isn't sound. It's called "model- based trace-checking," and the idea is to grab logs from the program, or an entire distributed system, and let TLC check that they conform to a possible execution of the spec. I've written about this technique here [3].

It is important to realize that all kinds of software assurance techniques, from proof assistants to unit tests, come with significant tradeoffs to confidence, effort and scale, and that _none_ can guarantee that a system will not fail. We must choose an appropriate balance for our specific system, usually by employing more than one approach (usually more than two). Personally, I think that high-level TLA+ specifications, checked with the TLC model checker (not with the TLAPS proof assistant) are at a pretty good sweet- spot for a relatively large number of mainstream software projects in terms of cost/benefit compared to other tools and approaches (but should still be combined with testing and code review).

[1]: [https://cedric.cnam.fr/fichiers/art_3439.pdf](https://cedric.cnam.fr/fichiers/art_3439.pdf)

[2]: You may think that it's possible to verify local properties of local units, and then compose them to ensure conformance to a high-level spec. Unfortunately, this way doesn't beat the scalability limitation, and also works only on very small programs.

[3]: [https://pron.github.io/files/Trace.pdf](https://pron.github.io/files/Trace.pdf)

<comment><USER3> _> In practice, we don't do it._

There are lots of very large Coq code bases that work just fine with code extraction. Scalability of extraction isn't the problem.

Is it really true that TLA scales to large programs? Why isn't there a Compcert in TLA? Why isn't there a Flyspec in TLA? And how about stuff like CertiGrad, where just the spec is pretty involved?

My impression has always been that Coq/HOL scale and TLA-style systems break down once your spec gets big enough (let alone the impl).

<comment><USER4> TLA+ does have things that make it hard for you to modularize and reuse pieces of it in the large.

The first category of problems deal with tooling. TLA+ doesn't have a great story when it comes to automation. There is no package manager. Version control is trickier than it needs to be (it's not obvious what files you might want to commit to git and which you shouldn't when you first start working with TLA+). There isn't a lot of support for automatically running things at the commandline (the community mostly uses the TLA+ toolbox, which is a GUI- based IDE). You can and people do these things, but they feel like distinctly second-class citizens. Really TLA+ feels tied to the TLA+ toolbox, warts and all.

The second problem is language support. TLA+'s main way of breaking down problems into modules and then using one module as an interface for another (its notion of refinement) suffers from problems analogous to the ones faced when dealing with inheritance. In TLA+'s case, it is easy to add additional new states into a state machine defined in another module (this is TLA+'s celebrated idea of stuttering), but is rather more annoying to do the opposite, e.g. show that in your new module a single state is meant to correspond to a two states in another module (e.g. you have a transaction primitive that allows you to assume that two states can be done simultaneously).

This is complicated by the fact that the easiest verification technique in TLA+ is model-checking, which fares the worst when dealing with transitive chains of logical implication. TLA+'s model checker does not have a native notion of logical implication. Instead the way you e.g. prove the statement A => B is by effectively proving A and B but privileging A to be the search space that the model checker operates in. This means chains of the form A => B, B => C, C => D and therefore A => D (and all the intermediate stages) are somewhat annoying. You either prove each step with separate models along the chain and just let a human infer that this means A => D, or you prove the statement A => (B and C and D), without using any of the intermediate implications.

Without robust support for transitive implication, the option of using composition in the traditional composition vs. inheritance debate becomes much less palatable as an option when it comes to inheritance.

To some extent this is a limitation inherent in model checking and goes away with the proof based verifier for TLA+ TLAPS. But there isn't a collection, that I know of, of proved theorems for TLAPS like many other theorem provers.

I suspect again this is due to both the lack of a package manager in the TLA+ ecosystem as well as some not so ergonomic parts of TLAPS. TLAPS feels very magical. Its main way of proving things is what's often referred to as sledgehammer or auto in other theorem provers; write the next step in your proof and hope the computer is smart enough to infer it from the previous step. It feels magical when it works. When the computer can't figure something out, it can be a huge pain to figure out why. It doesn't feel like I'm mechanically building up a proof so much as it feels like I'm playing a guessing game with the computer to see which leaps of logic it'll accept. This gets better with time as you develop an intuition of which leaps work and which don't, but can still be painful. This is mainly because TLAPS itself isn't a prover, but rather delegates to the "magic" methods of other provers.

Finally set theory and first order logic, the theoretical basis of TLA+, sometimes makes showing equivalence between items (which is vitally necessary when dealing with code written by multiple individuals who might use different encodings for the same idea) very difficult. In particular, trying to use the fact that A is equivalent to B, therefore some predicate P has an exactly analogous P' on B is tricky. Another way of stating this is that TLA+ as a language makes it a bit difficult to leverage isomorphisms and homomorphisms to reduce the amount of work you have to do. This is somewhere where alternative logical systems can have a big leg up.

All that being said, among the technologies commonly referred to as "formal verification," I would say that TLA+ is the one I would most commonly recommend to people in business situations. This precisely because of what many people view as a flaw, that TLA+ is a formal specification language and not a way to actually write or verify actual running code.

Because TLA+ specs are divorced from your codebase, they require far less buy- in to see dividends. You don't have to rewrite your codebase in another language. You don't even have to change your coding practices.

Indeed TLA+'s closest competitor on most teams is also precisely what most teams lack, which is high-quality documentation.

This means you can integrate TLA+ into your work with extremely little risk. Try formalizing your mental model in TLA+. See if it reveals interesting conclusions. If it does, congratulations! TLA+ has been useful to you and you can keep it around as an artifact detailing your thoughts in the future. If not, delete your specs and no harm no foul. No code depends on its existence. You don't need some lengthy transition period to migrate systems away from "using" TLA+. You lose no more time than the amount you put into writing your spec.

<comment><AUTHOR> > you to modularize and reuse pieces of it in the large.

There is no such thing as formal specifications "in the large." The largest software specification ever written in any language that could be fully verified is no more than a few thousand lines (although proofs can be very long). Having said that, I think TLA+ module system is fine; reminds me of ML.

> The second problem is language support. TLA+'s main way of breaking down > problems into modules and then using one module as an interface for another > (its notion of refinement) suffers from problems analogous to the ones faced > when dealing with inheritance.

I don't think so, but if you're modularizing your specs you may be doing something wrong.

> but is rather more annoying to do the opposite, e.g. show that in your new > module a single state is meant to correspond to a two states in another > module

Right, but the problem here is fundamental. Verifying temporal quantifiers is co-NP-hard _in the number of states_.

> This means chains of the form A => B, B => C, C => D and therefore A => D > (and all the intermediate stages) are somewhat annoying.

Yes, but still about 10x less annoying than writing proofs. :)

> Without robust support for transitive implication, the option of using > composition in the traditional composition vs. inheritance debate becomes > much less palatable as an option when it comes to inheritance.

Again, using composition is a hint that you may be doing it wrong.

> But there isn't a collection, that I know of, of proved theorems for TLAPS > like many other theorem provers.

There are some basic ones here: [https://github.com/tlaplus/v2-tlapm/tree/master/library](https://github.com/tlaplus/v2-tlapm/tree/master/library)

Deductive proofs are currently not a viable verification approach in industry (except in some small niches, and/or to tie together model-checking results), so libraries of that kind are mostly useful for researchers who research formal mathematics or deductive verification; that's not exactly TLA+'s target audience. Other proof assistants are used almost exclusively for research or by researchers so they're optimized for that.

> This is mainly because TLAPS itself isn't a prover, but rather delegates to > the "magic" methods of other provers.

That's not why (you can choose specific Isabelle tactics in your BY clause). The reason is that the TLA+ proof language was designed to be declarative. I guess some people prefer a more imperative style.

> Another way of stating this is that TLA+ as a language makes it a bit > difficult to leverage isomorphisms and homomorphisms to reduce the amount of > work you have to do. This is somewhere where alternative logical systems can > have a big leg up.

There are many things in TLA+'s design that make deductive verification harder, and many things in other language's designs that makes deductive verification harder. There is no one best way to do this.

I've written about this (a bit more generally) here: [https://old.reddit.com/r/tlaplus/comments/abi3oz/using_tla_t...](https://old.reddit.com/r/tlaplus/comments/abi3oz/using_tla_to_understand_xen_vchan/ed0imf1/)

I think the difference is that TLA+ was designed not for research but for industry use, and so its preferences take into account what's feasible. There is no point making something that few people would do easier at the expense of making something many would do harder.

> Because TLA+ specs are divorced from your codebase, they require far less > buy-in to see dividends. You don't have to rewrite your codebase in another > language. You don't even have to change your coding practices.

... And because there is no tool in existence that can verify those systems at the code-level anyway, so it's not like there's any other option. You get ~2000-5000 lines of formal spec. You can use them for code and get a tiny program or for a high-level spec formally "divorced" from your code.

<comment><USER4> There is formal specification in the large if you count formal mathematics. Mizar's library of formalized mathematics is easily millions of lines if not tens of millions of lines at this point. There are individual formalized mathematics repositories in other languages (Coq and Agda included I think? It's been a while since I looked into that) that run into the hundreds of thousands of lines of code.

So there are pointers as to how to organize formal specs for extremely large projects.

The reason as far as I can tell for why there aren't large software verification projects as opposed to pure formal math is that the cost benefit analysis is wildly out of whack. It makes essentially no business sense to try to formally verify things past several thousand lines. This doesn't seem like a bright line to me though, e.g. that there is a technological inflection point at several thousand lines that prevents further scaling. Again formal mathematics repositories are a counterexample. It's just that because of the high cost of formal verification, there is a business ceiling at the equivalent of man hours willing to be spent somewhere in the vicinity of hundreds of thousands of lines of unverified code to millions of lines of unverified code and you can rapidly exhaust that budget with full formal verification (which can easily account for orders of magnitude increases in time spent).

Ah thanks for the TLAPS repo. I forgot about the library, but even there I'm not seeing a lot of reuse in it. There's some of it, but not a lot. Maybe just because users of TLAPS are far fewer than users of TLC.

Indeed you can choose specific Isabelle tactics in your BY line. In practice I'm not sure I've ever seen it in any TLAPS proof. I've very very briefly played with that functionality and at first glance it seems much more difficult than using Isabelle directly because of the added layer of indirection. If you have one in the wild, I'd love to take a look to learn from.

I definitely agree there is no one way to do things. I've left out all the things that I think TLA+ is fantastic at. I think it's the closest thing to mathematics that can be practically used in industry. Far more so than the rhetoric you usually hear about FP languages, TLA+ is immediately familiar to anyone with a pure mathematics background, both syntactically and semantically. TLA+, unlike something like Agda, also has a fairly straightforward set of semantics and mathematical grounding in its theory. The benefit of writing each spec from scratch allows them to be independently understandable with no need to understand the idiosyncrasies of arcane libraries or frameworks.

But I do think it has problems with modularity. My state example wasn't to do with the difficulty of verification or the explosion of the state space (which honestly don't bother me, my TLA+ specs are often exponential or worse wrt model checking time in the size of my constants, but I still get a lot of mileage out of them), but rather the difficulty in even expressing that idea. When I'm confronted with wanting one state to stand in for two I often have to resort to using two separate specs, one with fewer states and then the other being the original spec and then showing my new spec fulfills both. I think the modularity problems mostly stemming from tooling and language choices, but a little bit from theory (although Mizar is also set theory based). For example I don't think I've ever used a 3rd party module in a TLA+ spec I've written. Even trying to express the equivalence between two specs that just use different strings for the same states can be tricky. Wouldn't it be cool if people could import Lamport's Paxos spec directly into their own specs of their own systems? Or any other specs someone has written specifying one part of their system? That would be pretty awesome if there was something akin to an NPM repository of all these specs for various algos that you could import into your own spec. The tools for this exist in Mizar. They do not for TLA+.

Turning my attention back to deductive verification rather than the expression part of the equation, more specifically for TLAPS, I'm actually not a fan of the imperative tactics that are found in Coq. But an error message better than what amounts to red highlighting and "no" would be nice in TLAPS. Something where you can mix YOLOing with more hands-on proof building, with tactics, functions, or even just a trace of the constraint search would be great.

I think the idea that you have some limited thousands of line to "spend" is a useful bit of intuition. One knob you're leaving out though is the richness of the properties you're trying to prove. As you decrease the richness of the properties you can prove you can drastically decrease your burden. This starts bleeding into static analyzers and then if you dial the knob even further down into mainstream type systems. But the knob can be fairly low and still be useful.

For example absence of crashing/abnormal exits is something that is feasible to prove on very large codebases. It's hard to do completely end-to-end (even CompCert's front-end isn't proven), mainly because of the difficulty of fully modeling stuff like filesystems, but can be done for modules spanning hundreds of thousands of lines of code. It is of course a weak property. That the program does something is nowhere near the same thing as doing the correct thing. But it is nonetheless a very useful one.

Absence of SQL injection attacks and CSRF attacks are other examples (can be done either by external analysis or by leveraging language level features), although those feel a bit more like cheating since you basically cleverly restrict the amount of code that "matters."

That would be the closest to an alternative to formal specifications I could suggest. I still think formal specs are lower risk because this still requires coding in certain styles amenable to this static check (or languages that enforce it).

<comment><AUTHOR> > There is formal specification in the large if you count formal mathematics.

Sure, but TLA+ is not aimed at research. The biggest problem, by far, plaguing not just proof assistants but formal logic in general, pretty much since its inception, is complexity. Which is ironic because the entire purpose of formal logic -- from Leibniz's original dream to Frege's early success -- is to be simple, clear and easy _tool_ for _practitioners_. Compared to the dismal failure of formal logic in achieving that (which was acknowledged and lamented by the likes of von Neumann and Turing, but has since been shrugged off even though the problem has gotten considerably worse), I think the fact that people who are not trained logicians use TLA+ to do real work makes it a crowning achievement not only in software verification but in the history of formal logic. To achieve that, TLA+ had to shed away everything that isn't absolutely necessary. Lamport once said after hearing a lecture about some proposed addition to TLA+: "it's taken you 6 months to add something to TLA+ that it had taken me 15 years to remove".

> It makes essentially no business sense to try to formally verify things past > several thousand lines.

Well, we don't know how to do that in a way that would be feasible; I consider that a "limit" because the software we use and need is very commonly three orders of magnitude ahead of what deductive verification can handle. And the gap has been constant or even growing for the past 40 years or so.

> Again formal mathematics repositories are a counterexample.

I disagree. Formal mathematics is about taking years to prove a proposition that could be written in one or a couple of lines. I think that the resemblance between deductive mathematical proofs and deductive software verification is like that between painting the Sistine Chapel and painting a large hospital. The similarity is superficial.

> If you have one in the wild, I'd love to take a look to learn from.

I wouldn't call it "in the wild", and it is used to bootstrap something fundamental (induction), but here: [https://github.com/tlaplus/v2-tlapm/blob/master/library/Natu...](https://github.com/tlaplus/v2-tlapm/blob/master/library/NaturalsInduction_proofs.tla#L30)

Also, search for "IsaM" here: [https://github.com/tlaplus/v2-tlapm/blob/master/library/Well...](https://github.com/tlaplus/v2-tlapm/blob/master/library/WellFoundedInduction_proofs.tla)

> But I do think it has problems with modularity. My state example wasn't to > do with the difficulty of verification or the explosion of the state space > ... but rather the difficulty in even expressing that idea.

Can you DM me on Twitter (get there from the about page of my blog: [https://pron.github.io](https://pron.github.io)) or send me an email (my name at Google's mail service)? I'm curious to see the problem you're referring to.

> Wouldn't it be cool if people could import Lamport's Paxos spec directly > into their own specs of their own systems?

Well, I don't yet understand the exact issue you're having, but no, I don't think that would be particularly helpful at all, for the following reason: If people wanted to _modify_ Paxos itself, they can get their hands on Lamport's spec and tinker with it. If they want to _use_ it as a component, then they can write an abstraction of it in a few line. All of TLA+ is about the abstraction/refinement relation ( _that_ is, IMO, what makes its theory cooler than Coq et al. for software).

> But an error message better than what amounts to red highlighting and "no" > would be nice in TLAPS.

Yep.

> One knob you're leaving out though is the richness of the properties you're > trying to prove. As you decrease the richness of the properties you can > prove you can drastically decrease your burden. This starts bleeding into > static analyzers and then if you dial the knob even further down into > mainstream type systems. But the knob can be fairly low and still be useful.

Right, but we know more than to say it's just richness. We know exactly what kind of properties scale (through abstract interpretation, of which both type systems and static analysis are examples): inductive (= compositional) ones. I.e. a property such that if it holds for _any_ state (whether or not reachable by the program), then any step in the program preserves it, or, alternatively, a property P, such that P(x) and P(y), where x and y are terms, implies that P(x ∘ y). Problem is that most functional correctness properties are not inductive/compositional for the language (i.e. that the above two description hold for any term in the language).

> That would be the closest to an alternative to formal specifications I could > suggest.

I think that's one additional tool. But a knob that's easier to turn is soundness. The difference between verifying a property to 100% certainty and to 99.999% certainty can be 10x in cost, and as -- unlike mathematics -- an engineer is not interested in the correctness of an abstract algorithm but in that of a physical system that can always fail with some nonzero probability -- absolute confidence is never strictly a requirement. Of course, this is not just _my_ opinion but it also seems to be that of many formal methods researchers, which is why concolic testing is such a hot topic right now.

I think that specification and verification must be separate -- i.e. you specify at any "depth" and then choose a verification strategy based on cost/benefit (so, no to dependent types), and in most cases, it's possible that concolic testing would be enough.

<comment><AUTHOR> P.S.

1\. The very large Coq/Agda texts are usually large because of the proofs, not the specification/propositions. There's a 5-10x ratio, if not more.

2\. I don't think that TLA+'s theory is intrinsically more "straightforward" than Agda's, even though it's certainly more familiar (I know some people who would insist that type theory is more straightforward than set theory). I think that _a lot_ of work has gone into making TLA+ relatively easy to grasp; I think it's more design than a specific choice of theory, although the choice of theory certainly affects the design. In addition, as I've already mentioned, I think that TLA itself (i.e. not the set theory in the "+") is a particularly beautiful theory for talking about computation and discrete systems, that has some obvious advantages over the lambda calculus; in particular the abstraction/refinement relation that's the core of TLA (the semantic domain of TLA is a Boolean lattice where the less-than relation is behavior refinement).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> My advisor's advisor worked with Lamport, and my advisor's thesis was in distributed consensus in file systems. My advisor worked heavily on a lot of the Paxos follow up work. He told me that Paxos is actually so complicated that only Lamport really understands it.

He said that after the original paper came out, everyone was trying to simplify it, but no one was successful. He said almost any implementation he saw (in those early years) contained serious errors.

It's a truly remarkable thing.

<comment><USER6> I earnestly believe that "Paxos is impossible to understand" is a baseless meme, a sort of learned helplessness in the CS community. I understand Paxos perfectly well. You can see it all laid out there in the TLA+ spec, and explained in many videos. It is absolutely untrue that only Lamport understands Paxos.

<comment><USER7> I'm sure many people understand it. I personally find it very hard to fully understand, and "Paxos Made Simple" didn't help. About one year ago, I went through all the steps of the proof, I implemented it. I _really_ tried to understand it. Today, without external reference, I probably wouldn't be able to implement it again. For such a short piece of code, it's surprisingly difficult.

</comment></comment></comment><comment><USER8> I have always been puzzled as to why Brian Oki & Barbara Liskov's earlier (1988) "Viewstamped Replication" has been shadowed by Lamport's work. Isn't the RAFT that rocked Paxos's boat not a variant of their work?

I would appreciate knowledgeable insights about this.

<comment><USER9> I don't know if raft "rocked" paxos's boat or not. It's a social phenomenon where Stanford people and their associates evangelize raft not because it is superior but just because it came from Stanford. People think raft is easier to implement but from my studies of available implementations the truth is it's actually just more likely that someone will Dunning-Krueger their way into believing they can implement it. There are a large number of defective raft implementations in the wild. There are almost no paxos implementations, correct or otherwise, making it harder to reason about its popularity and ergonomics. One thing is for sure: if you think you do not quite understand paxos then it is quite likely that you also do not understand raft, even though raft makes you feel like you might.

<comment><USER10> Raft also has a frequently-linked, simple visual explanation[1]. If that's at all accurate to what Raft is, that would certainly make Raft easier to understand than Paxos, although that's a function of the learning materials available, not anything inherently more understandable about the algorithm itself.

I feel that I understand Raft and not Paxos, although currently that's simply because I haven't yet tried to understand Paxos. But the reason I am still interested in Paxos is that looking at Raft, it still seems like Raft has a bottleneck at the leader. Essentially, it looks like what makes this distributed consensus look simple is that only leader elections are actually distributed: the rest of the algorithm isn't distributed at all. When I get around to reading about Paxos I'm hoping it will be different.

[1] [https://raft.github.io/](https://raft.github.io/)

<comment><USER2> I find that the core of Paxos (the Synod protocol) is very easy to understand - it is literally just two exchanges of messages! The tricks that you need to add to go from the Synod protocol and get an actual production implementation of the replicated log are another matter, the original paper only hints about some of them. There is e.g. the Paxos Made Live paper (link: [https://www.cs.utexas.edu/~lorenzo/corsi/cs380d/papers/paper...](https://www.cs.utexas.edu/~lorenzo/corsi/cs380d/papers/paper2-1.pdf)) that has more on this topic.

BTW Paxos also has a leader - this is what makes these protocols efficient. They need only the linear number of messages instead of the quadratic "everybody talks to everybody" design.

<comment><USER11> >BTW Paxos also has a leader - this is what makes these protocols efficient.

That's the funny part of Paxos. Paxos is specifically sold as a multi-proposer protocol. The most subtle and hard to understand part of the protocol is the clever 2 phase proposal election with value inheritance. But it's all pointless because in order to make the system work you need a single coordinator you elect using timeouts or reliable message protocols.

</comment></comment></comment></comment></comment><comment><USER12> I wonder if DS would have a different tone now if the leading figure would be somebody else more focused in making things intuitive and understandable. I've the feeling that many DS concepts, after you understand them, you say like: wow that was really not very well explained in the books I read.

<comment><USER13> Making a concept intuitive and easy to understand is extremely difficult to do. A person who could do this would probably have to understand Paxos _better_ than Lamport does.

I wouldn't assume that Leslie Lamport doesn't want his algorithm to be understood. It's just that explaining things is really, really, really hard.

(This is also why teaching is an incredibly difficult profession.)

<comment><USER14> Making an explanation intuitive takes time and effort, but I don't think it requires the explainer to have a better understanding of the concept than another person. They probably must have a good understanding of the concept, but once there is a good enough understanding, I think it's more a matter of finding the right way to explain it.

Lamport's original Paxos paper ([https://lamport.azurewebsites.net/pubs/lamport- paxos.pdf](https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf)) is notoriously difficult to understand because he decided to write the whole paper as an archeological survey report. Writing an explanation more intuitive than that isn't difficult even with only a basic understanding of Paxos -- just avoid the whole island and Parliament thing. His later paper ([https://lamport.azurewebsites.net/pubs/paxos- simple.pdf](https://lamport.azurewebsites.net/pubs/paxos-simple.pdf)) is much more clear.

<comment><USER15> reading the paxos-simple.pdf it seems it's very much like blockchain

</comment></comment></comment></comment><comment><USER16> Probably my favourite Lamport story is that his original Paxos paper was written in such a basic style (literally, Byzantine generals) that readers failed to realise exactly how revolutionary the content was.

<comment><USER17> Academic papers are almost encouraged to be hard to follow to make them sound more impressive.

<comment><USER18> Yes, but the Paxos paper is the opposite. It read like a silly homily. It's like "why's poignant guide to algorithms" except that the algorithm being described was complex, novel, and incredibly useful. It was so not full of itself that it was ignored for years. It seems to be proof that trying to sound impressive is actually important.

<comment><USER14> > algorithm being described was complex

I'd argue that the algorithm itself is simple, and that the difficult part is understanding why it works (i.e., why it satisfies the safety guarantees).

> It seems to be proof that trying to sound impressive is actually important.

I agree with this, and would add that making a paper sound impressive rarely conflicts with making it easy to understand. If anything, pointing out the contributions and applications of an approach help readers to understand that approach. At any rate, the same paper made easier to understand will generally be more likely to be accepted.

</comment></comment></comment></comment><comment><USER19> I've tried googling paxos algorithm a bit, but don't understand what it does and why it is useful. Could someone please explain it to me like I'm five?

<comment><USER12> It's goal in the end is very simple: you have N different state machines and you want every state machine to process the same operations in the same order. To do that you need they agree every time a new operation must be processed. (Multi) Paxos is an agreement algorithm that does that.

<comment><USER20> Does Paxos work in adversarial environment (where some nodes are compromised and try to trick the system by sending forged messages) ?

<comment><USER7> No, but there are other versions who do. Look up byzantine consensus.

</comment></comment></comment></comment><comment><USER21> The rest of the material from the summer school is available here: [https://sptdc.ru/schedule/](https://sptdc.ru/schedule/)

</comment><comment><USER11> Paxos is wrong, as Lamport explains on page 7 of "Paxos made simple", but apparently nobody reads that far.

<comment><USER22> Page 7 talks about the conditions under which a liveness condition can be satisfied. What about that is wrong? If you mean paxos is wrong because it can't guarantee progress, then sure it's wrong (and because of FLP, all consensus algorithms are)

<comment><USER11> Paxos livelocks (doesn't work). So Lamport suggests electing a coordinator via timeouts(since flp is a joke). But if you can elect via timeouts you don't need paxos at all. QED

The really annoying thing is that "Paxos Made Simple" begins by outlining the algorithm as a solution for a completely async network (unlike real networks) but then towards the end, just blows that off and falls back on timeouts. But if you have timeouts there are far simpler and more efficient solutions.

FWIW: FLP is a silly "theorem" that says "if you can't tell the difference between a system that has crashed and one that is slow - why then you can't tell the difference."

<comment><USER6> Paxos livelocks in a very unstable state which requires multiple independent nodes to sent specific messages at specific times. If you add random intervals between retries, the probability of staying in a livelocked state quickly grows astonishingly low.

<comment><USER11> I wrote a little simulation. Paxos rarely gets stuck with a small number of proposers/acceptors, but jams up solid once you get to, I forget, 10 or 20. Here it is:

[https://www.yodaiken.com/wp- content/uploads/2016/10/paxos.pt...](https://www.yodaiken.com/wp- content/uploads/2016/10/paxos.ptpx_.ppxt_.png)

Embarrassingly badly written Lua code for the simulator [https://github.com/yodaikenv/consensus](https://github.com/yodaikenv/consensus)

The key to me is that once you accept the need for leader election, you have much simpler alternatives and can toss away the rest of paxos.

<comment><USER6> This is authentically a very interesting simulation and I would be interesting to see a writeup if you have one. I will say that you only have multiple proposers if the proposers cannot talk to one another, as otherwise they would be silenced by any simple leader election protocol (heartbeats + node rank, etc.) It's true that the leader election protocol is not part of paxos per se but all implementations will have one, and the protocol livelocks in the unusual case where there's an enduring network partition and the multiple proposers are sending messages with specific timing.

Edit: I took a look at your post[0] and it is interesting. I'll take a look at CM.

[0] [https://www.yodaiken.com/papers-and-talks/distributed- consen...](https://www.yodaiken.com/papers-and-talks/distributed-consensus- paxos-raft-etc/)

</comment></comment></comment></comment></comment></comment><comment><USER15> there's a tla video course by lamport : [https://www.youtube.com/watch?v=p54W-XOIEF8&list=PLWAv2Etpa7...](https://www.youtube.com/watch?v=p54W-XOIEF8&list=PLWAv2Etpa7AOAwkreYImYt0gIpOdWQevD)

</comment><comment><USER15> paxos is used heavily in cloud spanner, also tldr; [https://youtu.be/8-Bc5Lqgx_c?t=4494](https://youtu.be/8-Bc5Lqgx_c?t=4494) , it's also interesting to watch two mathematicians go at it : [https://youtu.be/8-Bc5Lqgx_c?t=4902](https://youtu.be/8-Bc5Lqgx_c?t=4902)

</comment><comment><USER23> Are all Turing Award winners Turing Award Complete?</comment>