Removing Python's GIL: The Gilectomy [video] - <AUTHOR> https://www.youtube.com/watch?v=P3AyI_u66Bw <comment><USER2> This was a great talk, and was presented to a packed house (see below). If you don't want to watch the whole video, the issue really boils down to this:

1) Lot's and lot's of fine grained locks. One on every collection. Oooof.

2) All that locking and unlocking absolutely thrashes the cache with Larry's current prototype implementation. I was surprised at how little time was spent doing the actual locking/unlocking itself. I was blown away by the performance impact of maintaining cache coherent locks (at least in Larry's current implementation). And for reference, I was a logic designer on mainframes in the 1980's where we paid attention to making locks perform well across independent caches, so I'm no newb around this issue, but it was still striking to me.

(packed house) Larry's joke at the beginning about "practicing for getting on your plane later tonight" is a reference to the packed seating. The Portland convention center staff were taking the Portland fire marshall's directives quite seriously. We spent several minutes making sure every seat was occupied, and then the staff evicted the standee's :/

<comment><USER3> I wonder if you could completely elide the CAS needed by userspace locks (the bit which thrashes the cache I presume) while there is only one thread running. Then when a new thread is created for the first time (if it ever is), pause the program and promote and acquire all the locks.

Like how the JVM assumes classes are final and removes virtual calls until it first sees a subclass.

<comment><USER4> Sorry if I keep responding, but there's a lot of interesting points in your question.

> Like how the JVM assumes classes are final and removes virtual calls until > it first sees a subclass.

AFAIK, the python VM doesn't do any JITting at all. A runtime decision in the VM about handling locks is impossible at this point, I think.

<comment><USER2> There is a version of Python that JITs -- I forget it's name at the moment, though.

<comment><USER5> there are several. we (microsoft) are also trying to get an api into cpython so that anyone can use to build JITs for python. our sample implementation is at:

[https://github.com/Microsoft/Pyjion](https://github.com/Microsoft/Pyjion)

pycon 2016 talk:

[https://www.youtube.com/watch?v=1DAIzO3QXcA](https://www.youtube.com/watch?v=1DAIzO3QXcA)

</comment></comment></comment></comment></comment><comment><USER6> I have some thoughts:

\- Doing better than atomic inc/dec for those reference counts is going to be hard. All of those techniques are still in an "unconfirmed myth" state AFAICT: someone published a paper but nobody has confirmed that the result holds on a broader set of machines, workloads, baselines, etc.

\- Great call on userspace locking. Note that you can do this portably. You don't need special OS support. See [https://webkit.org/blog/6161/locking-in- webkit/](https://webkit.org/blog/6161/locking-in-webkit/)

\- Seems like lots of the locking use cases can indeed be made lock-free if you are willing to roll up your sleeves and get dirty. That's what I would do.

\- I still bet that the dominant cost is lock contention and he is not analyzing his data correctly. He appears to claim that it can't be locks because the total CPU time is greater than the total length of critical sections and that some analysis tools tell him that there is massive cache trashing. But that's _exactly_ what happens if you contend on locks too often. Lock contention causes context switches and thread migrations. Both of those things require cache flushes. So the code that runs under contention will report massive cache thrashing because it will have a high probability of being on a cold cache. Programs indeed will run slower under contention than without it, and while his slow-down is extreme, I've seen worse and fixed it by removing some contention. He should find every contended lock and kill it with fire.

\- The dip at 4 cores doesn't surprise me. Computers are strange and most "scalability" charts (X axis is CPUs, Y axis is some measure of perf) I've made had weirdly reproducible dips and jerks.

<comment><USER7> I asked about lock-free data structures from 2 major open-source libs, but Larry found major issues with them. He may consider some of these data structures in the future. I opened a bounty for this.

See closed github issues for gilectomy.

<comment><USER6> Not lock-free data structures. That's usually a fool's errand. You need lock- free _hacks_.

Exhibit #1: the lock to protect lazy initialization. That just needs a CAS on the slow path and a load-load fence on the fast path. Delete the object you created if you lose the race and try again.

Exhibit #2: you can probably do dirty things to make loading from dicts/lists not require locks even though storing to them does.

<comment><USER7> How maintainable are these hacks in the long run? Is there firm theoretical basis, if yes, then sources?

<comment><USER6> The lazy initialization example is a classic algorithm. Usually concurrency hackers rediscover it from first principles. It's easy to maintain because the concurrency protocol it uses is easy to understand. Here's how I'd write it:

        template<typename T, typename... Args>
        void constructSingleton(Atomic<T*>& singletonRef, Args&&... args)
        {
            for (;;) {
                T* oldValue = singletonRef.load();
                if (oldValue)
                    return oldValue;
                T* newValue = new T(std::forward<Args>(args)...);
                if (singletonRef.compareExchangeWeak(nullptr, newValue))
                    return newValue;
                delete newValue;
            }
        }

This works so long as the singletons are happy to be deleted. I'm assuming that's true here.

<comment><USER7> I think this is C++11, while CPython is using C89 and may update to use some C99 features supported by major compilers (GCC, Clang, MSVC).

<comment><USER6> Then write it in C89. It's not hard. My first lock-free algo was in C89.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> What a nice talk. It's a shame the only question we got to hear him answer from the audience (only time left for 1) was such a lame troll. If you attend conferences, please don't waste everyone's time with nonsense.

This is a python conference talking about a long-known perf issue and the guy just asked why not use another language. It's like going to WWDC and asking why everyone isn't using Android.

<comment><USER9> I have been subjected to more than a decade of Python propaganda about the GIL how it is a good thing because:

* it prevents deadlocks

* it prevents livelocks

* it means you don't have to lock when you share across threads

and how it's not a big deal because

* you're IO-bound anyway

* you can easily write performance critical parts in C

* you can use multiple processes, in fact that's better design anyway

So I would have asked more critical questions. Feel free to call that trolling if you want.

<comment><USER10> It seems to be a common mindset among python developers. You often hear the reason you can't have something in Python is because it's theoretically impossible, impractical and too difficult - but then it happens in another language. I was convinced that dynamic languages would always be slow (see [https://wiki.python.org/moin/Why%20is%20Python%20slower%20th...](https://wiki.python.org/moin/Why%20is%20Python%20slower%20than%20the%20xxx%20language)) but then Google gave us V8.

</comment></comment></comment><comment> kris-s I hope this is a fruitful endeavor but I can't help but feeling the pressure to "Not have another Py2 -> 3" situation will win out and the GIL will remain. Which, I think is completely fine.

For me Python is the ultimate glue code - it's the duck tape of my programming world. If I know multi-core performance is going to be an issue up front, I would pick another language.

<comment><USER11> Ideally Golang.

No other language really does it right like Go - unifying event based and traditional multi-threading paradigms in a way that transparently utilizes all the cores on your system, while allowing you to write plain old iterative, blocking code.

Go may be less than ideal in many other regards (i.e. the rest of the language), but it gets this right.

<comment><USER12> > _No other language really does it right like Go_

Not trying to start a war but... Erlang, Haskell, F#, and a few others abstracted Evented IO and parallelism before Go even existed.

<comment><USER11> I should have added mainstream, non-functional, compiled language.

<comment><USER13> Rust?

I would never consider Go mainstream myself though. Erlang is practically more mainstream.

<comment><USER14> Rust offers almost everything that Go does, but with a (much) nicer compiler, tooling that makes me drool, memory safety, and all with zero-cost abstractions that keep Rust as fast or faster than the naive C++ equivalent.

So even though I work in a majority Go company, and I feel like Go is quite mainstream, I firmly believe that Rust is the future!

Btw, sorry if this Rust enthusiasm comes across as a bit over the top, I'm quite tired. I just really like it is all.

<comment><USER15> Rust is a great and powerful language, probably the best alternative to C++ nowadays, but I think you're overselling it.

Rust definitely does things Go doesn't, but Go also offers things Rust doesn't: builtin concurrency and parallelism with goroutines and channels, garbage collection (which is useful when your app can tolerate its moderate overhead), very fast compilation, and great tooling.

<comment><USER16> Rust has builtin concurrency and parallelism in the standard library via threads, channels, and more, it just doesn't have green threads in the stdlib. As well, Rust also comes bundled with great tooling.

<comment><USER17> > _it just doesn 't have green threads in the stdlib_

Green Threads is one of the most important features in golang - to pretend that "Rust offers almost everything that Go does", but ignore the number 1 feature is dishonest.

<comment><USER16> I'm not the one who claimed that "Rust offers almost everything that Go does", but to say that concurrency and tooling aren't things that Rust excels at is equally dishonest. :P

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> I had a personal conversation with Larry Hastings (the presenter) at PyCon. Here are a couple of notes from that chat, phrased neutrally. Some of these points may be reïterated in the linked video:

\- We can view this work is as a revisiting of Greg Stein's GIL-removal attempt in Python 1.4:

[http://dabeaz.blogspot.com/2011/08/inside-look-at-gil- remova...](http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-removal- patch-of.html)

It seems wholly reasonable to revisit the approach in light of how the language and ecosystem have changed since 1999.

There are demands made of CPython core developers to remove or address the problem of the GIL, and these efforts demonstrate how much work is necessary to do that successfully.

\- Comparing single-threaded performance in a GIL implementation against single-threaded performance in a GIL-less implementation is considered an unfair comparison. A GIL-less will do extra book-keeping that will necessarily result in slower single-threaded performance.

<comment><USER19> > \- Comparing single-threaded performance in a GIL implementation against > single-threaded performance in a GIL-less implementation is considered an > unfair comparison. A GIL-less will do extra book-keeping that will > necessarily result in slower single-threaded performance.

Unless you can choose between having the GIL or not, I think it's perfectly reasonable to compare the performance. If you can choose, then I think it's still useful to know the kind of overhead you're adding.

</comment></comment><comment><USER20> Is there anything inherent to the language that makes this global lock thing difficult to be without in Python, but not in other languages?

<comment><USER21> > Is there anything inherent to the language that makes this global lock thing > difficult to be without in Python, but not in other languages?

Both language and interpreter design. I was wanting to give a talk or at least write about some of the internals of the interpreter and language and how to do it better next time.

In case there is interest for this I might actually get around doing that for once.

(I have played around with a different from of GIL-less execution a few years back that was based on independent interpreters and message passing thread bound objects but I ran into so many issues with the interpreter :()

<comment><USER22> I would certainly be interested to learn about better language and interpreter design.

Do you think anyone will ever use the lessons learned about interpreters and write a "better next time" implementation of _Python_ , or do you only see an improved runtime appearing alongside a significantly different language?

</comment></comment></comment><comment><USER23> It seems to me that the option of going with a tracing garbage collector is preferable. Removing the GIL will require a lot of changes anyway, and it may be better to go all they way so as to preserve performance. It would affect the C API and extensions modules would have to be reworked for this, but on the other hand you avoid all the issues with reference counting. If extension modules rely on code generation, such as Cython, a move to a radically different C API might be less painful than expected.

From what I know, most current managed languages are using GC, not reference counting, so perhaps it's an inherently better approach.

<comment><USER24> Most of the Python implementations other than CPython use GC rather than reference counting. PyPy does. Iron Python did.[1]

There is a version of PyPy without a GIL[2], but it runs much slower on ordinary code and is still under development. The developers are looking for financial support.[3] The approach is to identify large blocks of code as transactions, and run them in parallel. If they try to access the same data, one transaction fails and is backed out. It's like database rollback.

But you have to write your code like this:

        from transaction import TransactionQueue

        tr = TransactionQueue()
        for key, value in bigdict.items():
            tr.add(func, key, value)
        tr.run()

[1] [http://doc.pypy.org/en/latest/cpython_differences.html](http://doc.pypy.org/en/latest/cpython_differences.html) [2] [http://doc.pypy.org/en/latest/stm.html](http://doc.pypy.org/en/latest/stm.html) [3] [http://pypy.org/tmdonate2.html](http://pypy.org/tmdonate2.html)

</comment></comment><comment><USER25> Semi related to this, is it possible to run multiple CPython interpreters in the same process, but with restrictions on shared memory? The idea being that each interpreter would still have its own GIL, each interpreter would have restrictions on shared memory (like sharing immutable structures only through message passing). Note, I'm not a big Python user so if this already exists, has been discussed, etc I am not aware.

<comment><USER18> Sort of.

Note that Python has support for shared memory:

[https://docs.python.org/2/library/multiprocessing.html#shari...](https://docs.python.org/2/library/multiprocessing.html#sharing- state-between-processes)

In fact, `numpy` has its own mechanisms to support shared memory between processes:

[https://bitbucket.org/cleemesser/numpy- sharedmem](https://bitbucket.org/cleemesser/numpy-sharedmem)

Neither of these approaches seem to be used very commonly in practice.

Python itself has some "sub-interpreter" support. There was a long conversation about this last year:

[https://mail.python.org/pipermail/python- ideas/2015-June/034...](https://mail.python.org/pipermail/python- ideas/2015-June/034177.html)

Finally, I have a working approach using `dlmopen` to host multiple interpreters within the same process:

[https://gist.github.com/dutc/eba9b2f7980f400f6287](https://gist.github.com/dutc/eba9b2f7980f400f6287)

\- the approach is so bizarre, because it's a very naïve multiple-embedding. It was intended to prove that you could run a Python 2 and a Python 3 together in the same process as part of a dare. This was thought impossible, since there are symbols with non-unique names that the dynamic linker would be unable to distinguish (which lead me to the `RTLD_DEEPBIND` flag for `dlopen`,) and that there is global state in a Python interpreter that interacts in undesirable ways (which lead me to `dlmopen` and linker namespaces.)

\- this approach is stronger than the traditional subinterpreter approach, since I can host multiple interpreters of distinct versions. i.e., I can host a Python 1.5 inside a Python 2.7 inside a Python 3.5.

\- the approach is stronger in that I completely isolate C libraries. There's a good amount of functionality provided by C libraries that maintain global state. e.g., `locale.setlocale` is a wrapping of C stdlib locale and is globally scoped.

\- this approach is weaker in that it requires a dynamic linker that supports linker namespaces, which effectively limits its use on Windows

\- this approach is weaker in that it's not complete: there's insufficient interest in this approach for me to actually write the shims to allow communication between processes.

\- this approach is weaker in that it has some weird restrictions such as being able to spawn only 15 sub-interpreters before running out of thread- local storage space

I suppose the premise is that the GIL-removal efforts involve pessimistic coördination. A sub-interpreter approach might have a lighter touch and allow the user to handle coördination between processes (perhaps even requiring/allowing them to handle locks themselves.)

<comment><USER26> I followed the sub-interpreter thread with great interest, but after the implementation went in I haven't seen anyone build the kind of multiprocess tooling that it was designed to enable. Have you heard of anything?

</comment></comment></comment><comment><USER19> For the performance improvements, I think I'm missing something but why does coalesced reference counting have a high overhead?

Is it normally implemented along with buffered reference counting? It feels like those fit together very neatly, one thread managing the counts and receiving updates from the other threads, and each other thread tries to only send updates it needs to.

Is it simply a case of doing something basic a lot of times is faster than doing something smarter a few times because computers are just really fast at basic things? Or is there something more to this?

</comment><comment><USER27> Linux's pthread locks are already implemented on top of futex and have a fast user space path for the non-contended case.

</comment><comment><USER28> I'm just glad that Guido hasn't gone on record saying there will never be an official python without a GIL.

<comment><USER4> Guido's on record as having three "political" requirements for any GIL-less Python:

1\. Same or better performance 2\. No breaking existing extensions 3\. Not overly-complicating the cpython implementation

These are tough requirements, but obviously sensible, and Hasting's discussion of trying to meet them is interesting.</comment></comment>


Python Idioms [pdf] - <AUTHOR> http://safehammad.com/downloads/python-idioms-2014-01-16.pdf <comment><USER2> Interesting philosophical points.

To me personally, testing for 'truthy' and 'falsy' values, or relying on exceptions rather than checking values in advance, feels like sloppy and imprecise programming.

A string being empty or not, or an array having items or not, or a boolean being true or false, are all qualitatively totally different things to me -- and just because Python _can_ treat them the same, doesn't mean a programmer _should_ take advantage of that fact. Sometimes it's possible to _over_ -simplify things in a way that obfuscates instead of clarifying.

When I read:

        if name and pets and owners

I have no intuitive idea of what that means, of what's going on in the program. When I read

        if name != '' and len(pets) > 0 and owners != {}

I understand it exactly.

But by this point, I've come to understand that, for a lot of people, it seems to be the opposite. It seems to be more of a philosophical difference, not right/wrong.

<comment><USER3> I think part of the reasoning behind the truthy / falsy mechanic is that it's more robust. If, for whatever reason, we did:

      name = None

instead of

      name = ''

Then the second conditional would fail, whereas the first would still be fine.

<comment><USER4> Checking for empty strings can be done with len(mystring)==0 for this reason. In many other languages this method is standard and recommended practice.

Relying on implicit conversions is just sloppy. What if that variable was never supposed to be None in the first place. Better with an exception than continuing with corrupt data.

Remember another python motto: Explicit is better than implicit.

<comment><USER5> > Checking for empty strings can be done with len(mystring)==0 for this > reason.

`len` blows up on None, so this blows up completely instead of just failing.

> Relying on implicit conversions is just sloppy.

There is no implicit conversion. Truthiness is a protocol, it does not convert anything anywhere.

<comment><USER4> > `len` blows up on None, so this blows up completely instead of just failing

That's the whole point! As I said, Better with an exception than continuing with corrupt data. Or maybe I should say unsupported data type rather than corrupt data.

<comment><USER6> At the point where you are doing your validation (or where you are doing the actual work with the variable) you can make that decision.

It's entirely possible that upstream code is content with any type as long as it's coercible. A lot of Python code just wants an iterable, for example, or something that can be coerced to a string.

It's definitely possible to have a situation where you _need_ a string specifically (or an integer specifically), but it's generally better to have your code coerce the data whenever possible so as to allow for more logical code.

e.g. 's.send(str(object))' (if object has a relevant __str__()) makes perfect sense in code. You don't necessarily need a string, you just need something that can behave as a string in a sensible manner.

You also end up with scenarios where None is a 'valid value' in e.g. the SQL sense of 'value not specified'; for example, company_name might be '' (empty string provided) or None (no value provided); either way company_name is Falsey, and your logic should work the same.

</comment></comment></comment></comment></comment></comment><comment><USER7> I disagree with promoting try / catch. Exceptions like ValueError can really happen almost anywhere, so it is usually better to sanitize your inputs.

E.g. something like:

        try:
            something = myfunc(d['x'])
        except ValueError:
            something = None

The programmer's intent is probably to only catch errors in the key lookup d['x'], but if there is some bug in the implementation of myfunc() or any of the functions called by myfunc() which causes a ValueError to unintentionally be raised, it will be caught by the except.

For dictionary lookups specifically, get() is usually preferable:

        something = d.get('x')
        if something is not None:
            something = myfunc(something)

Or if dictionary may potentially contain None values:

        if 'x' in d:
            something = myfunc(d['x'])

<comment><USER8> That example would surely be better as:

        something = myfunc(d['x']) if 'x' in d else None

<comment><USER9> In this form you perform the lookup twice: once to test 'x' in d and then again to actually get the value d['x']. Try clauses in Python are very inexpensive if they pass (don't raise an exception), so often the try..except version would be preferable.

In any event don't optimize prematurely and use a profiler rather than guessing if performance is an issue. ;-)

</comment></comment></comment><comment><USER10> For point 10:

'_' is often aliased as gettext to ease translation of string:

        from django.utils.translation import ugettext as _

        translated_str = _('Something to translate')

so using it will overwrite the alias. Instead, you can use '__' (double underscore) as ncoghlan suggests below his answer [1]. or you can use the 'unused_' prefix as Google Python Style Guide suggests [2] or you can change your code, so you don't need to use '_' as Alex Martelli suggests in his answer [3].

[1]: [http://stackoverflow.com/a/5893946/720077](http://stackoverflow.com/a/5893946/720077)

[2]: [http://google- styleguide.googlecode.com/svn/trunk/pyguide.ht...](http://google- styleguide.googlecode.com/svn/trunk/pyguide.html?showone=Lint#Lint)

[3]: [http://stackoverflow.com/a/1739541/720077](http://stackoverflow.com/a/1739541/720077)

<comment><USER11> I have personally never seen this before. I'd be wary to use it, since it breaks the "_ is a throwaway" idiom, as well as the REPL "_ is the results of the last expression" function.

Aliasing it to "t" or "txl" seems like a saner way, if I'm honest.

<comment><USER12> This has been established practise in Zope, Plone et al for many years.

[http://developer.plone.org/i18n/internationalisation.html#ma...](http://developer.plone.org/i18n/internationalisation.html#marking- translatable-strings-in-python)

<comment><USER13> It's standard Python practice since Martin von Löwis's work in support for internationalization, which he presented at the IPC6 Python conference in 1997.

Quoting from [http://www.python.org/workshops/1997-10/proceedings/loewis.h...](http://www.python.org/workshops/1997-10/proceedings/loewis.html) :

> Here are several ways of producing the message catalogs. This is the one > suggested by the GNU gettext documentation. First, all translatable message > strings in the source code must be marked. In order to disturb readability > as little as possible, the wrapper function around each string should be > called _ (underscore). This use of the underscore usually does not interfere > with its meaning in the interactive mode. A gettextized module would then > begin with

        import intl

        _=intl.gettext

</comment></comment></comment></comment><comment><USER14> One of the handy things not mentioned:

       with open("x.txt") as f:
           data = f.read()
           # do something with data

<comment><USER15> absolutely! that one is quite high up on my python_idioms_to_import file

</comment></comment><comment><USER16> The very first one, "Make a script both importable and executable," needs some caveats. It's useful sometimes, but people often use it in places where it is not a great idea. Here's why:

1) If you are in the mindset of "I want a single file which is both a library and a program," how will you name it? Files to import must end with ".py" and follow C-style name rules, so cannot start with a number cannot contain hyphens. This leads many people to break conventions when naming their programs, because on Unix, hyphens are conventional in program names but underscores are not (count the examples of each in /usr/bin on any system). And naming your program something like "2to3" is impractical if you want it to be importable also.

2) It is unclear where to install files which are both programs and libraries. Programs usually go in some sort of "bin" directory (again, on Unix systems), but libraries do not. Where do you put a file which is both?

3) Sometimes the __name__ == '__main__' trick is used to include unit tests within a module. That's not bad, but consider using Python's excellent doctest feature instead, which often serves the same need but in a richer way.

<comment><USER17> I use the __name__ == '__main__' thing for unit testing.

I don't know if this is a generally applicable technique, but a lot of my modules interact with hardware or physical measurements, so I have to "see" the results in order to believe that the units are working. Often, what I'm looking for is problems with what's actually being measured, and the effect of changing operating conditions, not just my own copious programming bugs.

For this reason, my unit tests can be pretty elaborate, with GUI, graphs, and other stuff. The unit test also functions as a "demo" of the module.

<comment><USER16> Absolutely--that's a great example of a library module that may also be usefully executed. I had something similar today: a module that sends email. It's useful to be able to run it (by explicit "python foo.py", not chmod +x) and see a sample email in my inbox.

Unfortunately, for every good use of this trick, there seem to be two poor ones. Oh well. Most of the things in TFA are more generally applicable.

</comment></comment></comment><comment><USER18> > 9\. Create dict from keys and values using zip

In 2.7+, I'd recommend a dictionary comprehension instead.

<comment><USER19> In this case,

        {k: v for k, v in zip(keys, values)}

edit:

As I mention below, this becomes useful when you want to do e.g.

        {f(k): g(v) for k, v in zip(keys, values)}

<comment><USER20> I'd disagree, as

      dict(zip(keys, value))

is more concise, doesn't introduce extra variables, and doesn't repeat itself, and explicitly names a dict rather than using a symbol.

<comment><USER19> I like list comprehensions and to me dict comprehensions feel like a natural extension of this. It means that you can do things like

        {k.upper(): v ** 2 for k, v in zip(keys, values)}

Or generally

        {f(k): g(v) for k, v in zip(keys, values)}

I find this very extensible. In terms of conciseness, they both fit on a single line and I like the whitespace inside a dict comprehension.

</comment></comment></comment></comment><comment><USER21> As an implementor of Hy (a homoiconic lisp frontend to Python) I've found certain Python idioms to be rather infuriating of late.

In particular:

        >>> 0 == False
        True

Which makes the idiom of testing truthiness quite annoying in parsing code such as:

        def is_digit_char(s):
            """ Return a parsed integer from 's'."""
            try:
                return int(s)
            except (ValueError, TypeError):
                return None

Which is harmless enough except that as a predicate it sucks because parsing "0" will return False in a context where I'd rather know whether I parsed an integer or not... which leads to non-idiomatic code.

This is mainly because True/False are essentially aliases for 1/0 and as such won't identify so:

        >>> 0 is False
        False
        >>> 0 is 0
        True

So it's important to remember another Tim Peters-ism: _Although practicality beats purity._ As read in the Zen of Python it seems he's referring to special cases which this might be.

As a shameless aside, you should see what we're working on in Hy. There will likely come a point where we'll be able to do kibit-style suggestions of idiomatic transformations to your Python code.

 _Update_ : I ran into this while trying to write token parsers for a parser- combinator lib in Hy.

<comment><USER22> Your function is_digit_char(s) is peculiar. From the name I would expect it to return a boolean, instead it returns a number or None.

To write it like that and have a problem with the falseness of 0 it means that you use it in a way to both use it as conditional expression and an integer value, e.g.

        digit = is_digit_char('0')
        if digit: # fail
            print(digit)
         else:
            print('not a digit')

You should then check for his equality to None

         digit = is_digit_char('0')
         if digit is None: # pass
             print('not a number')
          else:
             print(digit)

But I would argue that you were in search for troubles when you wrote an is_something() function that doesn't return a boolean. That is not idiomatic.

p.s. Hy is too crazy :-)

<comment><USER21> It's a trivial example and I wouldn't focus on it too much.

It's not that peculiar -- instead of parsing the same character twice you simply return the value that you parsed or None. My inspiration was from the CLHS predicate function, DIGIT-CHAR-P [0].

The real point I was making is that Python has warts that make writing idiomatic code impractical in some situations. I suggest that practicality take precedence over purity. There are some situations that lead to non- idiomatic code and that's okay.

[0] [http://clhs.lisp.se/Body/f_digi_1.htm](http://clhs.lisp.se/Body/f_digi_1.htm)

 _Update_ : forgot the link. :)

 _Update update_ : Perhaps peculiar to Python because all values of integers are not False except for 0 whereas in another language that doesn't have this wart, anything that isn't False is True... even 0. In other words, anything that isn't False is True. :D

<comment><USER5> > In other words, anything that isn't False is True. :D

Except for `nil` (lisps, ruby), and possibly a host of other things depending on the language (empty strings in javascript).

<comment><USER21> Scheme is the only PL I know that has an explicit #f value.

CL, for all intents and purposes, treats nil as False... but some find the conflation of nil and the empty list runs into the same issue when operating on s-exprs.

        CL-USER> (if '() 1 0)
        0

vs

        scheme> (if '() 1 0)
        1

</comment></comment></comment></comment></comment><comment><USER23> Although many of them boil down to preferences and philosophical points of view, I find these kinds of idioms useful. Whenever I write code in a new language, I want to "write as a native" so that I can maximize the effect that the language has on how I think about programming.

For Python in particular, Jeff Knupp's "Writing Idiomatic Python"[1] (not free, but not expensive, either) goes into detail on a lot of the concepts in the OP's slides. (I'm not affiliated with Jeff in any way, just a satisfied customer.)

[1] [http://www.jeffknupp.com/writing-idiomatic-python- ebook](http://www.jeffknupp.com/writing-idiomatic-python-ebook)

<comment><USER24> What level would you say the idioms in the book are at? I have already been programming in python for a little while, and I wouldn't want to pay for something which I already know. It would be nice if there were more sample idioms on that site so you could have a better idea of what the rest of the book was like.

<comment><USER25> Given that the author is willing to give away copies to those that can't afford the book, I'm sure he'll consider a preview/promise refund if you send him an email? (See bottom of page).

Note: Also not affiliated in any way.

[edit: given:

[http://www.goodreads.com/book/show/17354838-writing- idiomati...](http://www.goodreads.com/book/show/17354838-writing-idiomatic- python-2-7-3)

it doesn't look like it's of all that much value to an experienced python developer. I'd love to hear some other comments, though.]

</comment></comment></comment><comment><USER26> Alex Martelli gave a nice talk called "Permission or Forgiveness" about the exception handling style recommended by OP: [http://pyvideo.org/video/1338/permission-or- forgiveness-0](http://pyvideo.org/video/1338/permission-or-forgiveness-0). He has some nice insights into this issue.

That being said, I think there are some situations where you want to check for problems up front (possibly in addition to exception handling). In particular, if you are parsing some data from outside the program, you may want to provide some context about what was wrong. KeyError is not very helpful to your users.

</comment><comment><USER27> As a huge Python fan, I'm ashamed to admit but I don't get the

        while True:
            break

What's the problem? I supose the use case is

        while True:
            # do stuff
            if some_condition:
                break

What is the alternative? 'while some_condition'? That means we must have the 'some_condition' variable outside of the loop. And if we have multiple exit points it may become a mess.

<comment><USER11> Personally, because I find infinite loops to be a real PITA, I prefer to do:

        for _ in xrange(100000):
            break
        else:
            logging.error("ran into an infinite loop")

unless I really do need an infinite loop for things like event handler loop, which is admittedly quite rare.

<comment><USER28> That seems incredibly silly, and looks like it could lead to very infrequent bugs (the worst kind). `while True` is shorter, simpler, and conveys the actual purpose better.

Maybe having a statement like that when testing is okay, but in production code that looks insane.

<comment><USER11> I'd argue that `while True` would be fine in testing, but problematic for production, particularly in the types of programs I write most frequently (long running, minimal supervision). In these daemons, stalling on infinite loops is significantly more painful than dropping out of a loop early occasionally (and with proper logging that it did fall out of the loop, to boot).

It's certainly not as idiomatic, but it's more correct in the long run. My eyes were opened to this when reading through the NASA C guidelines. Closing the door on infinite loops lets programs recover gracefully and do the correct thing for the duration of the programming, as opposed to thing that _may_ be correct for that moment of operation: i.e. looping 1-2 more times beyond the limits of the xrange counter.

</comment></comment></comment></comment><comment><USER19> I would add generator expressions:

        (f(x) for x in list_of_inputs)

Just like a list comprehension, but with (...) rather than [...] and with lazy evaluation.

These are useful when you don't need to evaluate all of the inputs at once but still want to iterate over them at some point later on.

<comment><USER29> Not to mention how awesome they become when you feed them as a set of values to a function.

        values = (f(x) for x in list)
        g(*values)

</comment></comment><comment><USER30> Thanks for this write up. I didn't know about enumerate. I never thought of swapping variables as in example 4 either.

I noticed one small mistake in section 9:

      d[keys] = values[i]

Should be:

      d[key] = values[i]

<comment><USER19> For a good few months I was doing

        for i in range(len(input_list)):
            v = input_list[i]
            ...do something with v and i...

enumerate is much nicer!

        for i, v in enumerate(input_list):
            ...

</comment></comment><comment><USER9> This is a great, if somewhat basic list. My $0.02: Python is a tool for thinking clearly. That you can run your "thoughts" as written is a very nice bonus, of course.

There are some really interesting things that Python allows:

        >>> d = {}
        >>> d[23], d[14], d['hmm'] = 'abc'
        >>> d
        {'hmm': 'c', 14: 'b', 23: 'a'}

<comment><USER9> I couldn't resist. Moar funky-cool Python:

        >>> from string import ascii_letters
        >>> ors = ['|'] * (len(ascii_letters) * 2 - 1)
        >>> ors[::2] = ascii_letters
        >>> ''.join(ors)
        'a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|A|B|C|D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z'

(Obviously, in this example '|'.join(ascii_letters) would suffice, but if the objects weren't strings...)

<comment><USER31> Not gonna lie... your snippet made me say "WTF?" Why not just do:

      from string import ascii_letters
      '|'.join( ascii_letters )

(I also like list slicing... but only when necessary.)

<comment><USER9> Yeah, I realized after posting it. (See edit.) The original code was using non-string objects. I basically come up with this to get the same behavior as str.join(). ;-)

<comment><USER32>
        '|'.join(str(x) for x in y) # or
        '|'.join(map(str, y))

<comment><USER9> The original use for that code was to build a list of parsing objects to create a single parsing object that was the OR'ing of the basic ones.

Aw, heck, code speaks louder than words:

[https://github.com/PhoenixBureau/PigeonComputer/blob/master/...](https://github.com/PhoenixBureau/PigeonComputer/blob/master/pigeon/cola/bubbles.py#L200)

<comment><USER32>
        intersperseM or $ map chartok whitespace

Oh... wait... sorry... wrong language.

</comment></comment></comment></comment></comment></comment></comment><comment><USER33> His last slide could be written more idiomatically as

        ''.join('Thanks!')

<comment><USER10> And this would be more idiomatic:

        'Thanks!'

(It is faster also! :D)

<comment><USER33> Ahh but that is a mere expression, not an idiom :-)

</comment></comment></comment><comment><USER34> Srsly, what Python programmer writes the code in the "Bad" examples therein? This list looks like it's from 2005 or something.

<comment><USER35> Lists like this aren't for experienced pythonistas, but more for new people to know what things to avoid, and what not to copy if they do come across it online (in an article from 2005...) :-)

</comment></comment><comment><USER36> On page 20, there is slight mistake - count is used in the second ("NOT SO GOOD") example but "i" is printed.

<comment><USER37> Or was it really a mistake?

<comment><USER36> Are there any anaphoric variable names in python, like cmd.exe?

</comment></comment></comment><comment><USER38> On page 20, didn't he/she mean "print(count, name)" in the BAD example?

On page 18, the Java comment: it depends. E.g. see [1]

[1] [http://stackoverflow.com/questions/299068/how-slow-are- java-...](http://stackoverflow.com/questions/299068/how-slow-are-java- exceptions)

</comment><comment><USER39> Another one that I find useful -- using `map` instead of list comprehensions when you want to apply a function to every element. So instead of:

        [str(x) for x in array]

Do this:

         map(str, array)

</comment><comment><USER40> >pets = ['Dog', 'Cat', 'Hamster']

>for pet in pets:

> print('A', pet, 'can be very cute!')

This may be nit picking but I prefer output like this:

print 'A %s can be very cute!' %(pet)

<comment><USER41> Possibly because you haven't moved to Python 3?

<comment><USER42> Python 3 doesn't dictate that style, just the parentheses

</comment></comment></comment><comment><USER43> I don't understand the truth table on slide 9 for " \- vs None " and "__nonzero__ (2.x) " __bool__ (3.x) vs __nonzero__ (2.x) " __bool__ (3.x) "

<comment><USER44> I'm pretty sure the "-" vs "None" meant that there wasn't a truthy alternative to None. More accurate would have been "N/A" vs "None". Not clear on the __nonzero__/__bool__ stuff...

<comment><USER11> __nonzero__/__bool__ are the methods called on objects when evaluating truthiness. So, if you're implementing custom objects, those are what you would use, and return either True or False, based on their truthiness value.

As an example, you might represent __bool__ on a database connection object to reflect whether there is a live connection, allowing you to:

        if not conn:
            conn = MySQLdb.Connect()

</comment></comment></comment><comment><USER45> String concatenation is pretty fast nowadays with the + operator.

<comment><USER46> Do you have a source? Also, I would argue that concatenation with + is less readable than string formatting.

<comment><USER47> I think that concatenation with + is okay when you're concatenating two strings. And CPython can optimize x = x + y or x += y calls.

[http://docs.python.org/2/library/stdtypes.html](http://docs.python.org/2/library/stdtypes.html)

</comment></comment></comment><comment><USER48> I'd love to see a writeup like this for Perl or Javascript. Has anyone come across one?

How about Java?

</comment><comment><USER49> i agree with all but #2. this seems to embrace conciseness as simplicity or understandability. it forgets the more cardinal value that explicit is better than implicit.</comment>


Cross-Platform Language Design [pdf] - <AUTHOR> http://lampwww.epfl.ch/~doeraene/thesis/doeraene-thesis-2018-cross-platform-language-design.pdf <comment><USER3> I find compilation of statically typed languages to JS, as in Scala.js, worthwhile in principle, for the reasons that Haoyi argued so well in a blog post a while back [1]. But there's one major reason I would be reluctant to use Scala.js: runtime footprint. 45 KB gzipped (the number given on scala- js.org) is pretty heavy. Is that an inevitable trade-off for the power of Scala, or is there still more work that can be done to bring that down?

[1]: [http://www.lihaoyi.com/post/FromfirstprinciplesWhyIbetonScal...](http://www.lihaoyi.com/post/FromfirstprinciplesWhyIbetonScalajs.html)

<comment><USER4> So I seem to be in the minority on this, and I'm interested in responses about why I'm wrong, but I believe that for most projects the weight in KB your framework/runtime doesn't matter _at all_ when compared to other concerns such as which language to use in your company's front-end stack.

Let's say your framework size is something wildly unrealistically ludicrous, say 10 MB after gzipping etc. Assuming daily deploys, with the average US internet connection being 18 Mbps, that's a wait time of about 4.4 seconds per user per day: Even in this nuclear unthinkable scenario of 10MB, the result to the end user is a 4 second wait time if they refresh right after your deploy, and then it's instant due to caching. This is a noticeable, minor annoyance to the user but nothing to lose sleep over. Your app can still be successful, as GMail has proven, which would probably take 10s to start even on a quantum computer.

Now let's bring those numbers down to the real world. We're choosing between a framework that's 20kb and one that's 50kb. 8 and 22 ms total user wait time _per deploy_ respectively. Even on mobile, users will never know the difference.

Back to Scala.js. I love Scala, and I want to work with and hire people who love Scala. Using this language is of critical importance to my business. Why should I consider giving it up to save 14ms/user/day?

<comment><USER5> Download time is not the only metric you should watch out for, there's also execution time.

<comment><USER6> You don't execute all the paths all the time.

The framework is basically the entire Scala runtime. You need a list - <USER2> execute that code. Not a biggy

</comment></comment></comment></comment><comment><USER7> I needed to write a little web utility a few months back and was pleasantly surprised at just how easy Scala.js makes it (as a Scala programmer): write some code, put a couple of lines in config and you're basically done. I wrote a blog post about my deployment experience at [http://m50d.github.io/2018/11/29/deploying-scalajs-with- netl...](http://m50d.github.io/2018/11/29/deploying-scalajs-with-netlify) and honestly felt like I was having to pad it out to article length.

</comment><comment><USER8> Hi, author here. Feel free to ask any question :-)

<comment><USER9> Nice work, congratulations!

What do you think will be the long-term story for Scala.js, with WebAssembly becoming mainstream, and current WASM shortcommings (e.g. GC and concurrency) out of the way? Do you expect Scala.js folding back into Scala?

Other question, does Scala's moving towards DOT-based foundations (e.g. dotty) affect Scala.js?

<comment><USER8> Wasm will not influence Scala.js' semantics. Scala.js "folding back into Scala" would be a regression as far as development for JavaScript platforms goes. Scala without Scala.js means a Scala that cannot interact with its environment, which has JavaScript APIs (e.g., the DOM). I am looking forward to Wasm with GC features as a different target for Scala.js (not Scala), with the same semantics, but different trade-offs in terms of performance and compactness.

Scala moving to DOT-based foundations is orthogonal to Scala.js. We'll have to rewrite parts of the compiler to adapt to dotty's internal, of course (and that is being done [1]), but at the language level, it is entirely orthogonal. Of course, we'll benefit from a better union type, which is very important for Scala.js, but it's not really a language interaction.

[1] [https://github.com/lampepfl/dotty/pull/5143](https://github.com/lampepfl/dotty/pull/5143)

</comment></comment></comment><comment><USER6> For those that don't know- scalatags is an amazing html-ish library that allows you to write html once and render both from server and from client. SSR and SPA in one for you cool kids out there.

Not to mention no more misspellings of attributes with strong typed html

</comment><comment><USER6> I hope lightbend will spend some marketing dollars when it reaches 1.0!

<comment><USER10> Doraene doesn't work for Lightbend though.

<comment><USER6> Then lightbend should buy Scala.js and scalatags and add a virtual Dom based reactive renderer. (currently string and Dom are available)

<comment><USER11> There are several Scala libraries providing this (the most popular probably being [https://github.com/japgolly/scalajs- react](https://github.com/japgolly/scalajs-react)). Have you tried one of them?

<comment><USER6> I want something very simple. Scala.rx + scalatags.</comment></comment></comment></comment></comment>


Wachovia Paid Trivial Fine for $400 Billion of Money Laundering - <AUTHOR> http://www.nakedcapitalism.com/2011/04/wachovia-paid-trivial-fine-for-nearly-400-billion-of-drug-related-money-laundering.html <comment><USER2> This exaggerated and misleading story was pretty well taken apart and stomped to death on Reddit: [http://www.reddit.com/r/reddit.com/comments/ghn9y/wachovia_p...](http://www.reddit.com/r/reddit.com/comments/ghn9y/wachovia_paid_trivial_fine_for_nearly_400_billion/c1nmbhi)</comment>


I love chess, but I don't trust FIDE's website with my password or passport - <AUTHOR> https://grahamcluley.com/2015/08/chess-fide-website-password/

<comment><USER2> There are way too many websites, who still do this. Some brand names you'd never expected in the list [http://plaintextoffenders.com/](http://plaintextoffenders.com/)</comment>


"Design outside-in, develop inside-out" - <AUTHOR> https://smalldata.tech/blog/2019/01/16/design-outside-in-develop-inside-out <comment><USER2> OP, I am assuming you're the author of the article. I think it could be improved by relating it to an experience when the advice given wasn't followed. At the moment I am struggling to gain anything from it.

<comment><AUTHOR> This is actually very interesting!

The article is mainly trying to answer the question - how do I begin with a new project? Should I make the UI first or should I make the backend first?

One of my projects where this advice was not followed was one where we decided to build something without having a clear idea of how it was going to be used. In this case we decided to focus on the backend and just had a very rough UI. This project went way overbudget because by the time we managed to get it to the users and got feedback, we had to refactor the entire data model and API due to new requirements.

I suppose one critical take-away from all of this is that proximity to users is a key component of a project's success :)

<comment><USER2> > I suppose one critical take-away from all of this is that proximity to users > is a key component of a project's success

Thank you. That is a great take-away actually.

</comment></comment></comment><comment><USER3> Why is the data model important? Even if your end-users are other programmers (say you're developing an http service) what's important is the API. The data model seems an implementation detail that shouldn't drive the design process.

<comment><USER4> I think this is a fair question, and I find some of the explanations here wanting.

You could implement the same UI the the same external specifications with a number of different data models. In theory, your users would never be able to tell. But in practice, a good data model is going to lend itself to performant software that can be changed predictably and without too many bugs.

The data model is internal quality.</comment></comment>


Ask HN:  If everyone could program, would you feel less self-worth? - <AUTHOR>

It's easy to imagine a world where programming is a required skill just as reading is.<p>Would you want to live in such a world? <comment><USER2> No, I would not want to live in such a world, but it has nothing to do with self-worth. I don't want programming to be a required skill for the same reason I don't want chemical engineering to be a required skill: most people don't need it. Most people need reading. Most people need math (despite the fact that it IS a required skill yet almost no one can actually do it). Most people need composition skills. Hence, the three R's (that would be reading, 'riting and 'rithmetic for those who aren't familiar with the phrase). Most people can also benefit from a basic understanding of history and science.

Programming and chemical engineering though? I see no reason for everyone to learn those. Computer usage, yes. Programming, not so much.

</comment><comment><USER3> even in a world where reading is a required skill, there are many people who can't, many people who are bad at it, and many people who are professionally good at it.

<comment><USER4> I was thinking the exact same thing, but for writing. Well said.

</comment></comment><comment><USER5> Programming is not like reading, it is more like solving puzzles.

Everybody can learn how to program at some level. But few really enjoy it and even fewer will feel satisfaction at squashing some obscure bug. It is closely related to having patience and to tinker with stuff.

I'd be more than happy to have the rest of the world program at some high level but I find it very hard to imagine a world where it is required to solve puzzles.

Just like you can't make the scientific method a requirement for life you can't do that with programming either. It works, and well, but not everybody will be able to even if you mandate it.

Compare it with mandating everybody to be able to play the violin tomorrow, and what effect that would have on the self-worth of the violinists today...

</comment><comment><USER6> I would suggest that writing:coding::reading:computer-use. Arguably, we already have an appropriate term in "computer literacy."

That is, I agree (assuming "coding" is equivalent to "customization" or, perhaps, even "confinguration") that we're rapidly approaching computer literacy as a required required skill, if we aren't there already.

All that said, I disagree that it's easy to imagine a world where programming is a required skill, nor even that programming is a skill. Rather, it is a combination of talent and skills.

Programmer:computers::novelist:books::reporter:periodical.

All that said, even if everyone could program, that wouldn't make them all equally successful, so my self-worth would not be affected. Besides, I consider myself a sysadmin, not a programmer.

</comment><comment><USER7> If you expand programming to its most basic definition, "customizing the behavior of a computer", then we are nearly there already. People can make computers do much of what they want (view TV shows, find interesting things to read, play music, write journals, buy things, etc.) without using an actual programming language.

Reading isn't as universal as you'd think. Arguably, the ability to read in one language is a different skill than reading in another, just as reading at a 1st grade level is different than being able to read the engineering manuals of the space shuttle. At some point, there are limitations to what a person can do, but within their limits they can do many great things. I think the same will remain true of computer use.

</comment><comment><USER8> I would feel enormously pleased, because it would mean that we had finally lowered the barrier of entry for programming to a point where people with little or no specialist training could create useful programs that were largely correct and bug-free, presumably with a minimal commitment of time deemed reasonable by the average person.

I think it would basically mean that we had "solved programming", which would be something of an achievement.

As for me, there would still need to be people who figured out how to keep programming theory ahead of the current state-of-the-art in terms of practice, so I reckon I'd at least have a shot at still having a job.

</comment><comment><USER9> It would probably be better. It wouldn't hurt any of our egos, because only a few people would be able to program well, just as, despite near universal literacy, only a few people can write well. But people would at least give us precise bug reports when things broke.

</comment><comment><USER10> Sure because there are always different levels of skill and desire.

My self-worth is dependent on how well I think I solved a given problem, not the fact that there are others who could solve it also.</comment>


Show HN: Hire-a-Cofounder - <AUTHOR> https://founderpanel.com <comment><AUTHOR> This was a quick project, but something I've been wanting to do for a while. It's a slight twist on finding cofounders. I'm not sure how effective it will be, but the idea of getting hired instead of immediately partnering on new projects in interesting to me.

If you're cofounder material and open to new projects, please submit a listing and let me know what you think.

</comment><comment><USER2> Google chrome says my connection is not private to your website

<comment><AUTHOR> I just got the warning on my browser too - thanks for the heads up, much appreciated. Looks like I forgot to confirm my contact email at namecheap and they temporarily changed the dns info. fixed.</comment></comment>


Perils of C++ debugging: Petrifying functions - <AUTHOR> http://www.yosefk.com/blog/the-c-sucks-series-petrifying-functions.html

<comment><USER2> The short version:

He hates C++ and C++ debuggers

Creating huge objects on the stack makes a program dump core due to a stack overflow, depending on the default stack size of the operating system. So a program that used to run OK might suddenly crash after an OS update and the error output wouldn't be very helpful.

<comment> loup-vaillant You may have missed the paragraph about books pretending C++ is high level. My understanding is that he hates _that_ , not necessarily C++ itself. I see this rant as yet another proof that C++ is as low level as C.

I think the sooner we admit this simple fact, the sooner we will be able to see C++ for what it is: a niche language, probably not worth it's huge spec.

<comment><USER3> I've written low-level C++ calling functions from Fortran and doing all sorts of magic to do out-of-core matrix calculations.

I've written high-level C++. Generic code with all my objects cleaning up after themselves.

So my experience is that C++ is a large language that allows programming in many niches.

<comment><USER4> _Generic code with all my objects cleaning up after themselves._

Are you sure they really cleaned up after themselves, and you don't have a subtle memory leak waiting to happen because of some unnoticed and rare and disastrous problem with the order in which your destructors get called? :)

<comment><USER5> With proper overloads you can easily track your memory usage or even which instances are leaked.

So yes, C++ is low-level (it does after all include C), but it has features to abstract away tedious details — this allows higher level programming but being in control of how the abstractions are implemented, as there are many performance trade-offs in choosing (implementation of) abstractions, and that is what makes C++ such nice language for software where performance is important.

</comment></comment></comment></comment></comment><comment><USER6> He may be using the wrong debugger. When I recreate and run his project in Visual Studio, I get this error:

Unhandled exception at 0x00b31747 in Bogus.exe: 0xC00000FD: Stack overflow.

The debugger points you to the offending line in a well-commented assembly routine described as ";_chkstk - check stack upon procedure entry".

It's immediately obvious this is a stack overflow, which takes you past the first half of his long article. However the second part of the article is still valid -- you still have to track down which of your objects is the big one.

</comment><comment><USER7> I wonder here if the problem is really just the small default stack size on Linux. In fact, why _is_ the stack limited to 8MB by default?

(I'm no fan of C++, but I suspect a number of languages that allow general stack allocated objects could fail in the same way).

<comment><USER8> 8 MB is also the default on Mac OS X. That's for the main thread -- for other threads (at least those created by the standard pthread mechanism), it's only 512 kB:

<http://developer.apple.com/mac/library/qa/qa2005/qa1419.html>

</comment></comment><comment><USER9> As a long time C++ programmer is very much appreciate expert C++ advice, even if it comes from someone who hates C++.

This was great, and I would like to add that any good programmer should know that it is easily possible to run out of stack space, especially with recursion.

</comment><comment><USER10> This reminds me of a guy I knew who spent all the time working on his car. All day long, and all night, he'd be outside underneath banging away at something.

I asked him once, "say, you must really like your car"

He said, "Hell no! This POS! Can't stand it. It's a wonder it even runs for one day without my having to tinker with it"

You have to really love something a lot to have an entire series of articles about how bad it is.

And as far as the specifics of this article, it's only so much noise. C++ lets me shoot myself in the foot because different operating systems have different stack sizes and nobody told me about it.

Okay -- welcome to the big leagues, kid, where we play without nets.

<comment><AUTHOR> It's not just the articles on his blog; yosefk is also the author of the C++ FQA Lite: <http://yosefk.com/c++fqa/>

He probably hates C++ more than anyone in the universe. (And probably knows more about it than anyone except veteran C++ implementers and committee members.)

<comment><USER10> Yes I thought this author was a rerun -- I remember his FQA being posted here a year or so ago.

The opposite of love is not hate; it's ambivalence. So he must really still carry a torch for the old girl.

<comment><AUTHOR> From a different post: _"...because that would offend C++, The Hate Of My Life, who does have an ANSI standard. Relax, darling. It's you, and only you, that I truly hate. The others mean nothing to me."_

[http://www.yosefk.com/blog/python-teaching-kids-and- biting-b...](http://www.yosefk.com/blog/python-teaching-kids-and-biting-bits- dont-mix.html)</comment></comment></comment></comment>


Show HN: SORS, a sci-fi doctor sim game inspired by Papers,Please and real science - <AUTHOR> http://steamcommunity.com/sharedfiles/filedetails/?id=402437511

<comment><AUTHOR> For those interested, the technology this game is inspired by is Spatially Offset Raman Spectroscopy, being trialled in a UK hospital. More info here: [http://www.stfc.ac.uk/CLF/Facilities/Ultra/Highlights/15073....](http://www.stfc.ac.uk/CLF/Facilities/Ultra/Highlights/15073.aspx)

The diseases in the game are all based on the potential uses of the technology. The graph patterns for the gameplay are all based on SORS readings from scientific papers.

I'm writing an education guide so that this game can also be used in schools. It covers certain Biology, Chemistry, Maths and Geography topics at roughly a High School Level (UK: GCSE/A-level).

Feedback most welcome!

</comment><comment><USER2> Dunno if this is an alpha version, but personally I think the visuals are a mess. Papers, Please (since you cite it as element of comparison) was very charming from this point of view - as every grapahic element of the game fell within a precise style and was presented very cleanly.

Here we have stereotypical green light terminals, OCR Font mixed with Mission Impossible strikethrough font mixed with bitmap fonts mixed with serious fonts: cartoon portraits with flash-style-shaded body illustrations with real life pictures.

<comment><AUTHOR> Sorry you don't like it! I guess it is a personal thing. We've had a lot of positive comments about the visuals though.

For what it's worth (not that it'll sway your view), the font choices were a conscious decision to make the medical technology look a bit outdated, because you know what hospitals run by administrators are like.

Also, the mix of cartoon visuals and the real-life pictures was again a deliberate decision. you may notice they never appear on the same screen together, and everything in-world (portraits, bodies) are cartoon style. Reference pictures are real-life.

But anyway, like you say, it's personal preference.</comment></comment>


The secret codes of British banknotes - <AUTHOR> http://www.bbc.com/future/story/20150624-the-secret-codes-of-british-banknotes <comment><USER2> For real lulz: print this pattern of small yellow dots on all your business correspondence with big companies, just to screw with their document management systems. Hours of endless fun for all the bureaucrats!

(There's got to be a PDF/postscript file/print filter for the pattern somewhere. Anyone? EDIT: Ah, found it here: [http://wildsparx.com/eurionize/](http://wildsparx.com/eurionize/) \-- download has vanished, but source is on github: [https://github.com/jplona/code/blob/master/eurionize.pl](https://github.com/jplona/code/blob/master/eurionize.pl) )

<comment><USER3> Or print it on a t-shirt. If anyone tried to capture a screenshot from a CCTV video feed, well, too bad...

<comment><USER4> While looking for that I stumbled across patterns and text that only shows up after photocopying due to effects such as aliasing and low-pass filtering, also pretty neat! [https://en.wikipedia.org/wiki/Void_pantograph](https://en.wikipedia.org/wiki/Void_pantograph)

And I easily found some shops carrying shirts:

[http://www.spreadshirt.de/eurion-t- shirts-C4408A23774319#/de...](http://www.spreadshirt.de/eurion-t- shirts-C4408A23774319#/detail/23774319T6A2PC120434135PA4PC120434136PA4PC120434137PA4PC120434138PA4PC120434139PA4PC120434140PA4PC120434141PA4PC120434142PA4PC120434143PA4PC120434144PA4)

[http://www.zazzle.com/eurion_constellation_t_shirt-235509764...](http://www.zazzle.com/eurion_constellation_t_shirt-235509764478000766)

if you know German, here are some people from AK Vorrat planning a campaign, I don't know whether it's still up to date:

[http://wiki.vorratsdatenspeicherung.de/EURion](http://wiki.vorratsdatenspeicherung.de/EURion)

And it has been discussed before, huh:

[https://news.ycombinator.com/item?id=8926114](https://news.ycombinator.com/item?id=8926114)

</comment></comment></comment><comment><USER5> I've done some preliminary work that measures the amount of "attention" an image processing algorithm pays to a given pixel (in terms of amount of computation done on it). Hard to tell at the moment how well it's working, but the results for the £10 note do seem to focus more on a region that includes a Eurion:

[http://laredo-13.mit.edu/~brendan/eurion_psp_gaze.png](http://laredo-13.mit.edu/~brendan/eurion_psp_gaze.png)

(This is from Paint Shop Pro, version 8.0, which refuses to edit the above image)

<comment><USER6> How are you measuring that?

<comment><USER7> I found these videos on his website:

[http://laredo-13.mit.edu/~brendan/eurion_attention.mp4](http://laredo-13.mit.edu/~brendan/eurion_attention.mp4)

[http://laredo-13.mit.edu/~brendan/eurion_tcn.mp4](http://laredo-13.mit.edu/~brendan/eurion_tcn.mp4)

<comment><USER5> Yup. These weren't as exciting as I'd hoped. The former (eurion_attention.mp4) basically shows a sliding window of what pixels had computation done on them over time; the most recent 1000 bytes (~330 pixels) get "lit up" in each frame of the video. You can see in the early part of the video how it scans over the image multiple times, but the later parts are not very interesting (and the whole thing is absurdly long; I recommend watching at 50X).

The second (eurion_tcn.mp4) one is simpler, and just tracks the mapping between pixel and compute number [1] over time.

[1] [https://news.ycombinator.com/item?id=9778871](https://news.ycombinator.com/item?id=9778871)

</comment></comment></comment></comment><comment><USER8> It's not just EURion though.

[https://news.ycombinator.com/item?id=7006848](https://news.ycombinator.com/item?id=7006848)

[http://www.cl.cam.ac.uk/~sjm217/projects/currency/](http://www.cl.cam.ac.uk/~sjm217/projects/currency/)

</comment><comment><USER6> This describes the geometry of the EURion constellation: "Recognizing Banknote Patterns for Protecting Economic Transactions" [http://paginaspersonales.deusto.es/igor.ira/publications/201...](http://paginaspersonales.deusto.es/igor.ira/publications/2010/text/igor.ira_Recognizing_Banknote_Patterns_for_Protecting_Economic_Transactions.pdf)

This patent describes how it is recognized: [https://www.google.com/patents/US20060279767](https://www.google.com/patents/US20060279767) The short version of which is "find all the circles, measure the distance between them; if you find five with the right distances you've found EURion".

<comment><USER9> So wait it doesn't account for dense circle patterns? Like if I have a bunch of circles, then it rejects the image if _any_ five of them have bad distances? I would think it would account for blank space somehow.

<comment><USER10> Suggests a way to DOS an image processing application by including a lot of circles, too

</comment></comment></comment><comment><USER11> Ah, good old Markus Kuhn. Always up for poking security features using any way possible. If you're interested in this kind of thing, you should read [https://www.lightbluetouchpaper.org/](https://www.lightbluetouchpaper.org/) which is UoC's security research blog.

</comment><comment><USER12> While I agree with the law, I take exception to my own possessions being hijacked for law enforcement. Apparently there's no need for a warrant to sneak in robotic cops in people's homes and offices.

</comment><comment><USER6> BTW If you are interested in stories about counterfeiting currency I can recommend: The Art of Making Money: The Story of a Master Counterfeiter

[http://www.amazon.com/The-Art-Making-Money- Counterfeiter/dp/...](http://www.amazon.com/The-Art-Making-Money- Counterfeiter/dp/1592405576)

</comment><comment><USER13> Meanwhile, the rest of the world is switching to plastic banknotes: [https://en.wikipedia.org/wiki/Polymer_banknote](https://en.wikipedia.org/wiki/Polymer_banknote)

<comment><USER14> …except Europe.

<comment><USER15> Northern Ireland was actually the first part of the UK to have these, they were introduced back in 2000 to celebrate the new millennium, you can still find them in circulation.

<comment><USER16> Can you? I don't think I've seen one for maybe 10 years.

<comment><USER15> I had one a couple of weeks ago, next time I get one I'll hang onto it I think.

</comment></comment></comment></comment></comment><comment><USER17> I can't find this pattern on swiss bank notes. Any idea if the swiss bank notes can be copied on xerox machines? Or do the swiss banknotes use a different pattern?

</comment><comment><USER18> Hmm, my $1 and $2 bills don't have this, though my $5, $10, and $20 have something like this with the O's (05 for $5)

</comment><comment><USER19> This is not news, is it? I remember reading about this pattern of circles 5+ years ago.

<comment><USER20> Five years? According to wikipedia it's been around since 1996.

</comment></comment><comment><USER21> You can't falsify banknotes with regular printers and regular paper, but you may need to scan and print them for some art project in which case you should just use Linux and avoid those silly protection measures.

<comment><USER2> Ahem: you can't falsify banknotes with regular printers and commodity paper _well enough to fool an alert human being_ , but the scanners in some vending machines are another matter. With 7 colour 1200+ dpi inkjets readily available for photo printing, if you could find a vaguely passable source of paper you could do wonders -- and one option forgers have used is to take low value notes and bleach them before overprinting with a higher denomination print pattern. (I think this is the rationale for high-denomination Euro banknotes -- €50 and above -- having embedded RFID chips.)

<comment><USER22> > used is to take low value notes and bleach them before overprinting with a > higher denomination print pattern. (I > think this is the rationale for > high-denomination Euro banknotes -- €50 and above -- having embedded RFID > chips.

Citation needed

You can't take a lower denomination bank-note and print a higher denomination one on an Euro note: the sizes are different (of course, maybe you can pass one note for the next in sequence, but it is slightly harder)

Also no RFID is mentioned here [https://en.wikipedia.org/wiki/Euro_banknotes#Security_featur...](https://en.wikipedia.org/wiki/Euro_banknotes#Security_features)

<comment><USER23> I always thought the coolest thing about Euro notes is their use of traces of the element Europium:

[https://en.wikipedia.org/wiki/Europium](https://en.wikipedia.org/wiki/Europium)</comment></comment></comment></comment>


Most Startups Should be Deer Hunters - <AUTHOR> http://www.bothsidesofthetable.com/2009/09/16/most-startups-should-be-deer-hunters/ <comment><USER2> Interesting points and useful analogies. But, I think it would be better framed within a specific subset of businesses rather than all startups. Also some subset of strategies.

Classical disruption for example, suggests a completely different strategy. Start with rats (ie an animal no one else even considers game). That is usually applied in a context where a rapidly improving disruptive technology is at the core of the business and it is getting ready to eat everything else. But it works in some other cases too. Adwords started as a service for fringe advertisers (rats to most ad sellers). Bullshit infoproducts, affiliate marketers & micro-businesses spending $5 on 100 clicks. They quickly moved into rabbits (ISPs, online businesses) and giant rats (that happens when your service is so useful, it actually makes customers). Deer came easily. Local dentists, ISPs, virtually every SME had an online marketing consultant at the door. Adwords is ideal for deer. Before that all they had was expensive and unreliable local media ads and trade rags. These days adwords gets a nice amount of elephant meat in its diet and they still do it with rat guns.

Its great to hear entrepreneurs argue about things in such definite terms. They are basically making the case for their way.

I think the main thing to take away though is to be extremely aware of what you are hunting. Make a conscious decision. Pick a strategy that works for your game and be aware of the risk profiles and gotchas of the strategy. They all have them.

</comment><comment><USER3> I'm not an avid hunter but I was hoping the metaphor would involve something about waiting in a tree stand or in the bushes -- for f--king hours on end, without the promise of even seeing a deer if you failed to apply deer urine and/or judge the wind properly...but it still being an enjoyable test of patience and whatnot. Deer aren't easy to kill if you're abiding by normal hunting regulations.

<comment><USER4> _I'm not an avid hunter but I was hoping the metaphor would involve something about waiting in a tree stand or in the bushes -- for f--king hours on end, without the promise of even seeing a deer if you failed to apply deer urine and/or judge the wind properly...but it still being an enjoyable test of patience and whatnot._

I also thought he was setting up analogy that was going to be something like that. Except I was thinking he was going to talk about the need to scout the land, understand your prey, use appropriate bait, choose the right weapon, etc. Those things would all work as analogies for a lesson about startups as well, I think.

</comment></comment><comment><USER4> Wow, that really resonates with me. As an enterprise play, we are definitely at risk of picking the wrong segment here. Although, our options are a little bit more limited due to the nature of our products. We're doing what you might call "Enterprise 2.0" stuff, operating at the intersection of social- networking (see: Enterprise Social Network), social-network-analysis, information retrieval and business process management... The stuff we're building really only starts to have utility _in_ the larger companies. So we're still interviewing people, and researching and exploring to try and find the right target segment.

There's a train of thought in my mind, fueled by our first big round of Customer Discovery interviews, that we are looking at (using employee count as a rough approximation for overall company size) companies with 1000 employees or more. _Maybe_ it really starts at 500 employees, but we're still working on that. Depending on where you draw the line between deer and elephant, we could wind up in either segment (or both).

Rabbits are definitely out for us though, at least given our current model. If things don't work going this route, we could pivot to a SaaS app that does less, and go down-market to the smaller companies. We did interview a few smaller companies earlier and saw some interest in that model, so it could be an option.

Our next step is to do a very focused Customer Discovery effort, focusing on manufacturing companies in the Southeast (mainly NC to start with, since that's where we are) with 1000 employees and up. But we'll mix in a few of the ones that are between 500 and 1000 as well, as we try to isolate the right focus.

The idea of landing an elephant or two _does_ seem tempting, but msuster makes some good points there. We definitely don't want to get too caught up special- casing stuff for one big customer, to the point that they become our _only_ customer, because we can't serve anybody else!

</comment><comment><USER5> Excellent article. This is relevant advice not just to startups, but to every company - you need to decide what kind of customers you're going after.

I see it a lot in our Software Consultancy. We're constantly focusing on what our goal is: small startups, medium sized companies, or large businesses. And whether the're technical companies, or non-technical. Those are the two biggest issues we have, and they affect _everything_ we do - from marketing (market technical skills? Market "solving problems without bothering you about it"?) to pricing (value-based? time based? Fixed-price Project based?) to how to grow technically (should we learn more languages and platforms to sell our skills to technical customers? Or should we lean towards learning a more specific niche if we're targeting a specific type of company).

</comment><comment><USER6> The part about avoiding rabbits really hits home. You see so many SaaS companies going for ~$19-$99/mo business models. Think about the thousands of customers you need to gather & retain to turn a few million revenue with that. Feasible if you can automate and scale the customer acquisition process but any direct sales effort is likely to be a non-starter.

And since it's Friday, here's some folks who are really nailing the recommended approach: <http://goo.gl/ue7uo> :-)

</comment><comment><USER7> >This is especially problematic in the Web 2.0 / Freemium world where too many company build their business models around trying to build massive scale of free customers and then convert a small share to low monthly payments. I guess it has worked for some companies? (Basecamp? Who else?)

Dropbox? Skype? I feel like there's a lot of freemium companies that make it.

I like this message though. Good advice for not just startups.

<comment><USER8> It's not efficient to hunt rabbits. You have to lay traps and just wait for them to come. You can eat them as soon as they are trapped (sell them an app) or keep them in the trap (sell them a service, like Dropbox, or rent them to the zoos, like all AD/social-based sites). You have a problem when your rabbit trap starts attracting deers (the elephants are too big to see it) and you're not prepared. They can break your trap and also release all the rabbits.

</comment></comment><comment><USER9> Awesome metaphor! At my company we are going after all three, and I can see how narrowing down our focus with laser like precision (on deer of course) could really help define our priorities.

<comment><USER10> 'Many start-ups (and even growth firms) lack this discipline and they therefore serve customers off all sizes. This leads to suboptimal results for all.'

</comment></comment><comment><USER11> Sound advice wrapped up in a very understandable analogy.

I have a little side-project going on which could easily cater to all three of those categories and, to be honest, in the back of my mind I have always thought about the Elephants when the realistic part of me knows I should only cater to the Deer.

Nice to be reminded to focus properly, I guess that elephant-related feature I was thinking about can be deleted

</comment><comment><USER12> Any thoughts on what decides the point at which a business is no longer an early stage startup? When does it make sense to level up from deer hunter to elephant hunter?

<comment><USER13> After you've bagged enough deer to be able to feed the team of elephant hunters for 6--18 months (the typical enterprise sales cycle time).

</comment></comment><comment><USER14> Another aspect: just have fun and shoot everything that moves.

</comment><comment><USER15> Also, in the old version, you can only carry 100 pounds of food back to the cart. The hunting subgame was fun for its time, but you're really better off putting your money into oxen and, if needed, buying food along the way.

Finally, don't ford the rivers. Pay passage if you can. Above all, start in May. You won't regret it.

<comment> T-hawk Good advice for simply making it to Oregon. Not deep enough to play for the best finish, starting as the farmer with little money but earning triple score. With those limited resources, you need to hunt as much as possible. And getting the top score means arriving with all party members in excellent health, which means leaving as early as you can in March and resting copiously along the way at the first sign of any disease.

Anyway, this article is pretty much reflective of Joel Spolsky's old article on software pricing. Selling $100,000 to an elephantine enterprise requires $50k worth of sales process. Selling $10 software to 10,000 rabbits requires $50k worth of customer support and infrastructure. Selling $1000 software to 100 users is the sweet spot.</comment></comment>


Hacking the Kindle Touch - <AUTHOR> http://wiki.mobileread.com/wiki/Kindle_Touch_Hacking

<comment><USER3> What are the benefits of hacking/jailbreaking/modifying the Kindle Touch?

<comment><USER4> Removing the ads.

<comment> excuse-me Here I would side with Amazon.

It's not like skipping ads on Tivo or being forced to watch unstoppable trailers on a Disney DVD.

You have made an honest and fair deal with Amazon. You can buy a Kindle or they will pay you $20 in return for viewing some pretty unobtrusive ads. Hacking this version to skip the ads is basically unfair.

In reality few enough people will do it to impact Amazon, and the jailbreakers probably also own other kindles AND buy more books from Amazon than the average person - but Amazon is on the side of the angels here.

<comment><USER5> How is skipping ads on Tivo any different? The implicit contract of broadcast television is that the ads pay for the content.

<comment> excuse-me I haven't been offered a deal by the advertiser to watch ads for a discount or pay more for no ads (plus I'm paying a subscription for the cable) - <USER2> watching TV ads isn't stealing whatever Mr Turner says!

It would be like going to buy a car and saying and asking for a discount because you had seen their ads on billboards along the freeway!</comment></comment></comment></comment></comment>


Muen: An X86/64 Separation Kernel for High Assurance - <AUTHOR> https://muen.sk/ <comment><USER2> How these are uses to split applications to reduce risk:

[http://os.inf.tu-dresden.de/papers_ps/nizza.pdf](http://os.inf.tu- dresden.de/papers_ps/nizza.pdf)

Survey in 2017 of most separation kernels with verification methods:

[https://arxiv.org/pdf/1701.01535.pdf](https://arxiv.org/pdf/1701.01535.pdf)

This one is a medium-assurance kernel that tries to knock out most errors or injections using SPARK Ada.

[https://en.m.wikipedia.org/wiki/SPARK_(programming_language)](https://en.m.wikipedia.org/wiki/SPARK_\(programming_language\))

In practice, a medium or high assurance evaluation would also require testing of about everything, covert-channel analysis to spot leaks, verification the object code matches the source, and independent team re-running those with a pentest on top. INTEGRITY-178B and VxWorks MILS are separation kernels that are finished with or in evaluation with all that data. seL4 has strongest formal methods on code and assembly but other stuff is still being done at R&D lab with no evaluation. Both it and Muen were integrated into Genode OS framework with all three needing more review, pentestimg, and improvements.

<comment><USER3> Regarding the arxiv link on verification methods, I'd be careful reading too much into the summary tables. The authors don't appear to have done their homework there (they overstate the C&A accomplishments of the company I work at and our competitors).

On an unrelated note, I'm pleasantly surprised to see a tech report I contributed to as an undergrad cited - How on earth did they find it?

<comment><USER2> It's worth noting. However, I can't be too harsh about it since a number of vendors were constantly bullshiting me when I was trying to get to the bottom of what they did. I had to dig deep into protection profiles, BS marketing, comments online, interviews, and so on. I also called out Green Hills through proxies about a fake, "independent" blog they set up basically to bash Lynx and talking like INTEGRITY (not INTEGRITY-178B) was EAL6+. I ended up telling people to watch out for politics with evaluators for the proprietary companies then straight up read the research reports or source + docs for the rest. It ain't quite what it was under TCSEC which had more technical stuff and less politics where technical requirements where main form of political argument back then.

Far as your paper, I'd guess they did what I often do by typing in common keywords into search engines. "Formal," "security" "policy" were three that were in about any certified project. On the policy side, the words "policy" and "model[ing]" were used interchangeably here and there. I got a lot of results with these in a few, quick Googles although not yours. I imagine your paper was in a search or a citation of something with similar terms. Just speculating, though. Cool that you got to see your work surface on HN front page, though. :)

<comment><USER3> Yeah, dealing with vendors can be frustrating, for sure.

I'm still a little sad about the I-178 eval, since a lot of the work I saw done by RC was quite good, if limited in scope. Seeing The vendor (imo) misrepresent that effort to sell other software was rather disappointing to someone who had (at the time) just spent a couple years jumping on the formal methods bandwagon.

I'm also not particularly happy with the trend towards process, rather than results based evals. (DO-178, and CSFC, I'm looking at you), but I can see how we got here after the EAL7 well was poisoned.

<comment><USER2> "I'm still a little sad about the I-178 eval, since a lot of the work I saw done by RC was quite good, if limited in scope."

I still give them credit for doing what they could for a robust, usable piece of software with some kind of evaluation. Hopefully fixing whatever problems were noted. Their page on it is also something I've shared plenty of times because some of the design elements are really good. I especially like how each process donates their own CPU and memory to accomplish kernel calls. Maybe that's bypassable but it seems like a good default. The list of certification artifacts on bottom-right also [mostly] detail the assurance activities required in secure and certified systems. I poke at FOSS projects that claim security saying, "Which of these did you not do and why?" Usually they say "a covert- _what_ analysis?" ;)

The evaluations have become too political. Under Walker's initiative, they didn't play or make toothless threats. IBM themselves were first to be shown the door when they made a C1-class product instead of C2. STOP OS (in XTS-400), GEMSOS, SNS Server, and KeyKOS were some of most secure OS's market ever produced. That was partly due to features (current focus) but mostly due to _assurance activities_ the predecessors of EAL6/7 mandated to systematically add quality or security to every aspect of the lifecycle. Although modern stuff is political, the assurance methods work in both EAL6/7 and DO-178B to measurably improve safety/security _if applied for real_. I wrote an essay on how to improve security certification to learn from prior regimes with new one focusing on assurance activities of various components (eg TOE's) with a compositional rating added and no politics allowed. Here it is:

[https://pastebin.com/dACA5gPh](https://pastebin.com/dACA5gPh)

Far as formal methods, there are recent developments that make it a lot easier for people to do on the low end. For one, TLA+ is making waves after the Amazon report on internal use. hwayne is making a nice tutorial using the PlusCal subset here:

[http://learntla.com/](http://learntla.com/)

One problem with full verification is they often try to do everything in one tool. I've long encouraged splitting it up among tools picking each best for the job. DeepSpec w/ CertiKOS is doing that with what looks like a DSL style of formal methods. COGENT let one team do an ext2 filesystem certified to low- level code. One I found interesting given all the work with the B method was E-SPARK where they did formal verification of specs in Event-B whose verification conditions were fed into SPARK to prove it at code level. We need more of that. Far as SPARK itself, Carlisle built IRONSIDES DNS in SPARK.

[https://journal.ub.tu- berlin.de/eceasst/article/view/785/782](https://journal.ub.tu- berlin.de/eceasst/article/view/785/782)

[http://ironsides.martincarlisle.com/](http://ironsides.martincarlisle.com/)

My concept is using a macro-supporting, imperative language to write the application w/ formal specs for everything. Then, this is automatically translated to Rust, C, Ada2012/SPARK2014, and SPIN/TLA+ to let the code get hit by all tooling I can find for all of them in static/dynamic analysis, SPARK's prover, and test generation. Any problems found are reinterpreted in terms of the original code for developer to check if false alarm or actual problem. Concept being high-assurance labor is expensive but push-button tools that just point to severe flaws might go a long way. They can even develop and use minor tests to build plenty of features during the day with background processes or build cluster running the verification process incrementally with some tracker notifying developers of when ready are ready to look at.

I call it Brute-Force Assurance since it basically throws cheap labor and CPU's at the problem. What do you think of this concept for medium-assurance security leaning toward high at least for stopping code injections?

<comment><USER3> Thanks for the references - I'll look into them, I'm definitely interested in multi-tool workflows given my experience (systems programmers aren't necessarily mathematicians). If only the systems programmer could be pointed at tools that don't have horrible false positive rates ... ;)

<comment><USER2> On that note, have you or anyone else you've known gotten to use Astree Analyzer? Ive been wanting to hear from field users if it's as good as they claim on bug elimination and false positives.

</comment></comment></comment></comment></comment></comment></comment><comment><USER4> On the last conference held by the Chaos Computer Club in Germany, there was a talk about a covert side channel[0] using - if I understood and remembered that correctly - cache timings.

The researchers that gave the talk claimed that they could establish a very slow but practically undetectable communication channel between two VMs running on the same host.[1]

Since the introduction to Muen explicitly mentions defense against covert channels, could somebody explain to me if this system would be able to prevent such communication? And if so, is it possible to explain the mechanism to somebody like me who knows basically nothing about these things?

[0] [https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wron...](https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wrong_with_insert_x86_instruction_here)

[1] Again, if I understood that talk more or less correctly (probably less), undetectable refers both to the OS running in the VM and to the hypervisor.

<comment><USER2> I can't speak for Muen but I can a bit for this problem area. This attack has been known for a _long_ time. It's why physical separation and CPU's with separate caches were used in high-assurance security. Anything shared is a potential, covert channel. Timing channels are the hardest to mitigate so share nothing if you're worried about that, right? Well, some sharing is needed so what to do.

1\. Since security kernels of the old days, any shared stuff is supposed to be cleared before the next thing uses it. That principle here would be register overwriting and a cache flush when a separation kernel switches between partitions. It's called "periods processing" in the literature.

2\. If no shared cache, then we can keep Security Domain A running on Core A, Security Domain B on Core B, etc. There can be multiple, concurrent VM's on each core so long as they are in same security classification (trust each other). Take the penalty of separation when it matters.

3\. Partitioning and/or real-time caches. In real-time caches, you lock the cache for determinism. That determinism might be used to aid covert, channel mitigation. The other mechanism is stronger whereby you design the cache itself to know mutually-distrusting processes are using it. A combo of the CPU, cache, and/or separation kernel might wall off parts of it, erase parts of it, replace parts of it, and so on. There's quite a few prototypes in CompSci of this. Maybe some commercial implementations but my brain is too tired to remember.

4\. Asynchronous, randomized execution to throw off timing. I came up with this one independently studying the problem probably with inspiration from others randomizing timing in asynchronous way. More importantly, another person did the same concept implemented on a prototype CPU.

5\. Run with no cache. Very painful. Barely a solution. It's been done, though.

I'm sure more of this can be combined in interesting ways with randomized storage in caches w/ randomized timing and/or that controlled by separation kernels. It's mostly a hardware problem, though, like software safety against code injection. Most in high-assurance have moved on from separation kernels to other tech, esp modified CPU's, to solve the problem bottom-up like Burroughs B5000 did for some things. Last efforts I looked into were Cambridge CHERI processor w/ CheriBSD, SAFE processor at crash-safe.org, and Watchdog[Lite]. Gotta add cache protections, IO/MMU, RAM cryptosystem, TRNG, and so on.

<comment><USER5> Wrt "run with no cache": is a shared DRAM that much better than a shared cache? The DRAM state is affected by what previously accessed it, in fact DRAM effectively has an SRAM on-chip "cache" keeping the current active row per bank. ("Cache" in quotes because it's a bit crummy if you think of it as a cache - there's just one way, #lines is 4-8 and their size is measured in kilobytes - but technically it's a cache.)

<comment><USER2> Far as DRAM state, the separation kernel keeps the state of one process separate from another. Harder to do with a cache without defeating the performance-enhancing purpose of a cache.

"in fact DRAM effectively has an SRAM on-chip "cache" keeping the current active row per bank."

Had no idea. That could be a problem. See why we systems people defaulted on physical separation during most of the Moore's Law advances? Never know what hardware issue will pop up.

</comment></comment></comment></comment><comment><USER6> > PCI device passthrough using Intel VT-d (DMAR and IR)

I wonder how they formally verify that. DMAR is incredibly complicated. It's not even clear to me that malicious firmware on a PCIe device can be contained by DMAR, even assuming a perfect chipset and kernel.

<comment><USER2> "I wonder how they formally verify that. "

They might be using a different definition of formally verify than you think. SPARK is one of the "lightweight" methods that tries to automate a lot of stuff with the rest hand-proven. They might just mean it's formally verified to not have all the common code injections since SPARK can do that with a good deal automated vs manual proof. Full correctness of that operation would be something different they might need a proof assistant for or considerable effort.

An example of one doing full, formal verification of all kinds of components is CertiKOS at DeepSpec:

[http://deepspec.org/](http://deepspec.org/)

[http://flint.cs.yale.edu/certikos/](http://flint.cs.yale.edu/certikos/)

</comment></comment><comment><USER7> any reason why 64-bit windows hosts are not supported?

<comment><USER2> They have 32-bit through Genode OS Framework and VirtualBox. That might mean they got it through what Genode project was doing with full virtualization as indicated here:

[https://genode.org/documentation/release- notes/16.08#Virtual...](https://genode.org/documentation/release- notes/16.08#VirtualBox_4_on_top)

Genode/Muen might just not be able to handle 64-bit yet. The Genode versions are also a bit behind since it's a small number of people doing the upgrades on components ranging from custom to common.</comment></comment>


Which Wearable will you be buying [Poll]? - <AUTHOR> http://www.appstorechronicle.com/2013/09/poll-smartwatchwearable-will-buying.html

<comment><USER2> I don't see space glasses on that list.</comment>