Belgium and the Netherlands Swap Land, and Remain Friends - <AUTHOR> http://www.nytimes.com/2016/11/28/world/europe/belgium-netherlands-land-dispute.html <comment><USER2> There's a really interesting bit at the end:

> _Norway was considering an unlikely birthday present for neighboring > Finland: an Arctic mountain peak on Mount Halti, the highest mountain in > Finland, whose 4,478-foot summit is in Norway._

> _To mark the 100th anniversary of Finland's declaration of independence from > Russia, a group of Norwegians was urging the government to move a point on > its border with Finland about 490 feet to the north and 650 feet to the > east._

> _But that ran afoul of an article of the Norwegian Constitution that states > unequivocally that the Kingdom of Norway is "indivisible and inalienable," > and the friendly gesture was called off, at least for now._

The constitution is available online, and has other interesting bits in it, especially as a contrast to the US Constitution: [https://www.stortinget.no/globalassets/pdf/english/constitut...](https://www.stortinget.no/globalassets/pdf/english/constitutionenglish.pdf)

> _The King shall at all times profess the Evangelical-Lutheran religion._

There's also a repealed Article 10, but the English-language Wikipedia article doesn't say anything about it.

<comment><USER3> The repealed Article 10 simply says that "The King is of age, when he turns 20 years. As soon as he enters the 21st year, he declares himself publicly to be an adult."

[https://lovdata.no/dokument/HIST/lov/1814-05-17-18140517](https://lovdata.no/dokument/HIST/lov/1814-05-17-18140517)

I don't quite see why that was repealed, though. Some Norwegian might tell.

As a Finn I sort of appreciate the gesture of proposing moving border to bring the hilltop of Halti to Finland, but I don't really think it is that significant.

The main issue I know about is an old confusion in schoolbooks. The highest point in Finland is 1324 meters above sea level, on the Norwegian border, on the slope of Halti near the top. The top is on Norwegian side and its height is 1328 meters. Someone writing a schoolbook long ago found out that Halti is 1328 meters high and put that as the elevation of highest hill in Finland, without considering that this highest point is actually in Norway.

[edit: typos]

<comment><USER4> I'm not Norwegian, but I think the confusion is because the numbering of the Articles has changed. Your "Article 10" is now Article 8. [1] The repealed Article (originally Article 12, now listed as Article 10) concerned coronations. [2]

[1]: [https://www.stortinget.no/en/Grunnlovsjubileet/In- English/Th...](https://www.stortinget.no/en/Grunnlovsjubileet/In-English/The- Constitution---Complete-text/)

[2]: [http://www.royalcourt.no/artikkel.html?tid=35248&sek=35247](http://www.royalcourt.no/artikkel.html?tid=35248&sek=35247)

<comment><USER3> Ah, right. Thanks.

</comment></comment></comment></comment><comment><USER5> The comment at the bottom of the article reminds me why secession is virtually impossible in most places in the world. Any such attempts falter because they're instantly declared unconstitutional by the parent country.

The global community encourages this kind of status quo because many countries worry their ethnic minority will be next.

The only way sadly still seems to be decades of bloodshed before the host population gets fed up enough to let them go.

I don't only mean developing countries like Sudan either. It took until 1998 for the Belfast agreement to be signed over Northern Ireland for example.

<comment><USER6> I don't think this is always true. Scotland was given its referendum. Britain voted on secession from the EU. Northern Ireland is talking about unification.

There are certainly places like Spain which have been beating the drum of 'No Catalan Independence', but there are modern states that do recognize some uncodified right to self determination.

Some resistance is natural though, and in some cases good -- tribal divisions can happen on any line. If California seceded, the Jefferson region would want to secede from that, and no doubt there would be even smaller fractures.

In the end, I think every state has its own gravity and own structural strength. Grow too big, and you suffer from your own weight, risk fractures.

<comment><USER5> But Northern Ireland _is_ an example of freedom through bloodshed. There was decades of violence before the UK conceded to give them genuine choice. Arguably Scotland benefited from this change in the UK's public opinion as a result of Ireland, without having to go through the same process.

<comment><USER7> > But Northern Ireland _is_ an example of freedom through bloodshed. There was > decades of violence before the UK conceded to give them genuine choice.

I think that's an oversimplification; the intent was there to give Ireland self-governance from the 1880s (initially as a single, unified area). Notably, the Government of Ireland Act 1914, better known as the Third Home Rule Bill (the prior two had failed to pass through Parliament) would've led to a unitary Ireland, though this led to the Home Rule Crisis (though there had been some rioting going back to the 1880s) and was ultimately never implemented due to the outbreak of war.

The real fatal errors in the case of Ireland were the handling of the Easter Rising of 1916 and the coupling of implementation of Home Rule with conscription in Ireland, along with many more errors during the guerrilla war that followed.

Had WW1 not happened when it did, who knows what would have happened in Ireland.

<comment><USER8> That's a pretty charitable assessment.

The 19th century saw England enact a policy of near genocide, mass deportation and political/economic oppression.

Ireland was little different than any other third world colony.

<comment><USER7> Oh, the UK completely fucked over Ireland, there's little doubt about that, despite _legally_ being in a far better place than the colonies (they actually had representation, most obviously) they practically weren't.

The question is whether bloodshed was inevitable in the path to independence, and I don't think it's reasonable to call the actions of the state bloodshed on that path except when it was in direct suppression of it (and I don't think there was any of that prior to the Easter Rising, unless one considers the rebellion of 1798 when Ireland was still legally a separate country, though had had the same ruler as England since the 12th century, albeit with a separate political structure).

Certainly the blood on the UK's hands led to many of the calls for independence, but if independence had been granted in, e.g., 1886 (i.e., the First Home Rule Bill) would it be freedom through bloodshed, or would it be freedom through wanting to do better yourselves? I'd definitely say the latter.

</comment></comment></comment></comment></comment></comment><comment><USER9> Surprisingly this is not an exclave swap: [http://www.amusingplanet.com/2012/11/the-curious-case-of- baa...](http://www.amusingplanet.com/2012/11/the-curious-case-of-baarle- nassau-and.html)

<comment><USER10> Well, it's un-inhabited, so it's a lot easier to swap, as opposed to exclaves who usually have inhabitants who would not be too happy to switch nationality.

<comment><USER11> You would not switch nationality, just the country where you live. If I move to another country, my nationality stays the same.

<comment><USER3> However, if another country moves to you, it makes several things awkward.

There are plenty of exclaves with country borders out there. For instance, here's a bit of Germany inside Switzerland:

[https://goo.gl/maps/bw3PwD19JhR2](https://goo.gl/maps/bw3PwD19JhR2)

The village is called Büsingen.

[https://en.wikipedia.org/wiki/B%C3%BCsingen_am_Hochrhein](https://en.wikipedia.org/wiki/B%C3%BCsingen_am_Hochrhein)

In United Arab Emirates there is an Omanian exclave called Madha, which in turn contains a UAE exclave called Nahwa:

[https://goo.gl/maps/REXtsfb3gHs](https://goo.gl/maps/REXtsfb3gHs)

[https://en.wikipedia.org/wiki/Nahwa](https://en.wikipedia.org/wiki/Nahwa)

And there's even one third-order enclave.

[https://en.wikipedia.org/wiki/Dahala_Khagrabari](https://en.wikipedia.org/wiki/Dahala_Khagrabari)

Dahala Khagrabari is a piece of India, surrounded by Bangladesh, surreounded by India, surrounded by Bangladesh.

And things stay this way because it is quite awkward to change places where people live to belong to another nation - however impractical these exclaves may be.

<comment><USER12> Dahala Khagrabari is no longer an exclave. India and Bangaldesh executed a wide-ranging land swap in 2015 to resolve a large number of these tiny exclaves.

<comment><USER3> Ah, thanks for information. So, border adjustments are possible there as well (India and Bangladesh are on friendly terms, unlike India and Pakistan).

</comment></comment></comment></comment></comment></comment><comment><USER3> It's interesting that so close to the heart of European Union, the police and judicial authorities needs special permits to enter through the area of neighbouring country, even if they won't do any official business there, they just pass through. Anybody else can move about freely, no questions asked.

Between Finland and Sweden, the police forces on both sides of the border have an agreement so that they can move about, even in official business. A pursuit of a criminal can be extended across border, and e.g. ambulances are shared. The crew is bilingual or works in Finnish (the population in the border area is traditionally mostly Finnish-speaking even in the Swedish side).

The countries each have a nation-wide public safety digital radio network (TETRA) and while the networks are strictly distinct, the police cars or ambulances in each country may have a radio unit from the neighbour so that they can connect to the neighbour network when they need to, or their radios can be roamed to the neighbour network through ISI interface.

<comment><USER13> _" A pursuit of a criminal can be extended across border, and e.g. ambulances are shared"_

The examples you mention all involve cases where one cannot reasonably wait for permission to pass. Those are handled similar between Belgium and the Netherlands and, I guess/think (there may be exceptions for cases like Switzerland and Norway, who are in the Schengen area but not in the EU), everywhere in the Schengen area.

This case was different, though; the victim was clearly dead, so there wasn't much of an urgency. If there is no urgent need, would Sweden allow armed Finnish police officers to enter the country or vice versa? What if it were a few heavily armed SWAT teams in armed personnel carriers?

<comment><USER3> The police in the border region in both countries is currently being trained for patrolling in the other country (there are some differences in legislation and operational procedures). So not just pass-through but actual law enforcement work.

After WW II, when Finnish Lapland's roads were largely destroyed by retreating Germans, Finns used the roads on the Swedish side of the river to get up north. Today it is perhaps the other way round; the Finnish side has more population and the roads up river are bigger so it might be feasible for the Swedish to use Finnish roads even if the destination is again across river on Swedish side.

SWAT teams no, not without urgent need and/or official request, and people would rise eyebrows also for SWAT teams driving around _in their own country_ for no reason.

</comment></comment></comment><comment><USER14> I got an apartment a few kilometers away. It's on the outskirts of Maastricht and to properly enter Belgium, you need to cross the Maas first (or Meuse I think the article called it?). Everyone there speaks Dutch, works in Maastricht, shops in Maastricht... It's the same city but officially you're in a different country. And so I'll be missing out on 291 euros a month in rent subsidies... But it was the only thing available and reasonably priced so better than nothing. I just wish they'd swap more land where it makes sense :/

<comment><USER15> In all likelihood, your landlord is missing out on 291 euros a month in rent subsidies.

<comment><USER14> Nah it's not that cheap. This is not a normal price lowered by 291, though maybe half.

</comment></comment></comment><comment><USER16> How do countries usually handle rivers moving around? It looks like in the US for state borders they go with where the rivers used to be. Makes for some messy borders

[https://firstname.lastname@example.com,-91....](https://email@example.com,-91.0512079,11.94z/data=!4m5!3m4!1s0x8620a454b2118265:0xc955f73281e54703!8m2!3d30.9842977!4d-91.9623327)

<comment><USER17> in the US whether the state border changes depends on the specific agreement between the states, which may in some cases go back to the colonial era (for virginia vs. maryland, along the potomac, it's the low tide mark on the virgina side except for tributaries - so if VA builds out into the river it doesn't become part of maryland). It gets quite difficult to amend these because both states have to agree, plus congress, for any revision of borders. It's usually just not worth it.

One well known dispute was between new jersey and new york, over ellis and liberty islands; the supreme court had to intevene on that one and I think currently both states have jurisdiction.

<comment><USER18> In the 19th century, New York was given jurisdiction over the original land and New Jersey had jurisdiction over the surrounding waters. As the island was expanded through land fill, the newly created land was treated as part of New York until the Supreme Court ruled that the original agreement between the states gave the filled in land to New Jersey which is most of the island today.

[http://www.washingtonpost.com/wp- srv/national/longterm/supco...](http://www.washingtonpost.com/wp- srv/national/longterm/supcourt/stories/wpellis052798.htm)

</comment></comment></comment><comment><USER19> Perhaps CGP Grey will add a footnote to his "Holland vs the Netherlands" video: [https://www.youtube.com/watch?v=eE_IUPInEuc](https://www.youtube.com/watch?v=eE_IUPInEuc)

</comment><comment><USER20> I don't understand. Given Schengen and free movement, why couldn't the legal team from Belgium just 'freely move' through the Netherlands to get to their area of jurisdiction?

<comment><USER21> It's a good question. My understanding is that Schengen doesn't allow for free movement of police or military. You have to travel as a civilian.

<comment><USER13> There are exceptions for emergencies ([https://en.wikipedia.org/wiki/Schengen_Area#Police_and_judic...](https://en.wikipedia.org/wiki/Schengen_Area#Police_and_judicial_co- operation): _" The Schengen Agreement also permits police officers from one participating state to follow suspects across borders both in hot pursuit and to continue observation operations"_), but that's correct.

(And aside: for the Belgian-Dutch border, that has been the case for decades; border controls in the Benelux ([https://en.wikipedia.org/wiki/Benelux](https://en.wikipedia.org/wiki/Benelux)) were abolished in 1970)

</comment></comment></comment><comment><USER22> This story prompted me to wonder [0] how mapping organisations (Ordnance Survey, Google Maps, et al) are made aware of the specifics of such details.

Grateful to hear from anyone with insight!

[0] - [https://news.ycombinator.com/item?id=13065514](https://news.ycombinator.com/item?id=13065514)

</comment><comment><USER23> India and Bangladesh swapped territories in 2015 during which India lost 40 sq km of land.[1]

[1] [https://en.wikipedia.org/wiki/India%E2%80%93Bangladesh_encla...](https://en.wikipedia.org/wiki/India%E2%80%93Bangladesh_enclaves)

</comment><comment><USER24> Illinois and Missouri need to do this after years of floods and the New Madrid earthquake have left enclaves on the wrong side of the river in several places.

</comment><comment><USER25> Reminds me of this interesting case of a piece of land between Egypt and Sudan that neither side wants, because having ownership would mean conceding ownership of a second larger piece of land to the other country:

[https://en.wikipedia.org/wiki/Bir_Tawil](https://en.wikipedia.org/wiki/Bir_Tawil)

<comment><USER26> Oh my god

[https://www.theguardian.com/world/2014/jul/16/american- claim...](https://www.theguardian.com/world/2014/jul/16/american-claim-africa- science-jeremiah-heaton-egypt-sudan)

<comment><USER27> Similar case: [https://en.wikipedia.org/wiki/Liberland](https://en.wikipedia.org/wiki/Liberland)

</comment></comment></comment><comment><USER28> And then there is Liberland [1], a micronation proclaimed on land unwanted by both sides of a border dispute.

1\. [https://en.wikipedia.org/wiki/Liberland](https://en.wikipedia.org/wiki/Liberland)

<comment><USER29> I got to see the border of this country a while back:

[https://en.wikipedia.org/wiki/Transnistria](https://en.wikipedia.org/wiki/Transnistria)

Here's a photo:

[http://journeyofkhan.us/photo/transnistria-across-the- nistru...](http://journeyofkhan.us/photo/transnistria-across-the-nistru- river/)

[http://journeyofkhan.us/photo/bridge-to- transnistria/](http://journeyofkhan.us/photo/bridge-to-transnistria/)

<comment><USER30> In case some folks here on HN don't already know it, I like to tell the story of this "country".

Basically, it starts in the Middle Ages when there are 2 small states under Ottoman (Turkish) influence: Wallachia and Moldavia. As it happens for all small states, they get bullied around by their bigger neighbors, primarily the Ottomans, Russia and Austria. For our story only the Ottomans and Russia are important.

So the Ottomans and Russia keep fighting over the region and they keep trading territories here, completely disregarding the history of the region or the ethnic groups present there. Imperialism at its finest.

Wallachia is left more or less unscathed, but around 1812 Moldova is basically split in two, Russia taking the less developed Eastern part, called Bessarabia.

After all this abuse, Wallachia and Moldova decide to unite and form modern Romania, around 1860. So Moldova basically disappears as a state.

Fast forward to the end of WW1, Romania is getting bigger by incorporating all the lands around it that are majority Romanian, so Bessarabia becomes a part of Romania.

Meanwhile, Russia becomes the USSR. It's not hard to guess that Romania and the USSR are not the best of friends. The Soviets never recognize the Romanian takeover/reclaiming of Bessarabia, but can't really attack Romania (yet). So they form a sort of "Moldovan Soviet Republic" outside of the historical territory of Moldova. Basically "Moldova in exile". I like to compare it to Mexico creating a "New Texas" state after being kicked out of Texas.

Then, before WW2, as part of the Ribbentrop-Molotov Nazi-Soviet pact, the Soviets are allowed to attack Romania. They don't attack, instead they send an ultimatum and Romania is forced to give up Bessarabia, which is incorporated into the Moldovan Soviet Republic.

So now the new Moldova has both Bessarabia and that extra part that was never Moldovan. That part is across the river Dniester, ergo "Trans" (across) "nistr" (Dniester is "Nistru" in Romanian) "-ia" (particle for "land of" in Romania).

The USSR falls, Moldova becomes independent. Transnistria has Russia armies on its territory, so it declares its independence. Russia signs an agreement to pull back its army. 20 years after the agreement, the army is still there...

</comment></comment></comment><comment><USER31> Hope India and Pakistan resolve [https://en.wikipedia.org/wiki/Kashmir_dispute](https://en.wikipedia.org/wiki/Kashmir_dispute) in similar manner

</comment><comment><USER32> India and Pakistan should have done the same over the sloppy border line in the north done by the british, crossing natural rivers. A constant source for terrorism.

</comment><comment><USER33> Does anybody live on that land? I mean, this would be like a country kicking its own citizens out. Suddenly your government your house is not in their country anymore.

<comment><USER34> According to OSM, there are no houses in the swapped areas, so they probably just swapped some grass, sand and roads.

</comment></comment><comment><USER12> Maybe Canada and the US can come to a similar agreement at Estcourt Station.</comment>


Ask HN:What would you like a list of links about? - <AUTHOR> <comment><USER2> You mean that one can't retrieve a list of links about using google?</comment>


Slack vs. HipChat vs. Hall for project communication - <AUTHOR> http://usersnap.com/blog/productivity-tips-web-developers-manage-project-communication/

<comment><USER2> It's a slim review really. Like a typical software review they include a feature comparison table, but don't dive very deep into daily usage of the platforms, and really only show demos for Slack.

The overall tone I got was, if you aren't using chat software now, you should really be looking into it and here are three options.</comment>


Company rejected my idea, now they want piece of my new company - <AUTHOR> Here is the situation. I cofounded a company with group of friends. Our product failed to attract beta customers. I informed the CEO that we might be building a product that customers dont want and I presented him with what I think is the right product. He completely rejected the idea and dismissed me for not believing in his vision.
I talked to lawyers in the field of employment, IP and corperate to make sure that I am not doing anything wrong. Since There was no employment agreement in place, they advised me that I am entitled to continue.
After 3 months of working on this product. I was able to attract customers immediately. Now the company is coming after me.
Has anyone had issues like this? <comment><USER2> When you say that the company is "coming after you," do you mean to say that the CEO is calling you saying that he/she expects you to grant some ownership interest in your new venture? Or do you mean that you have received a communication from an attorney representing the company, and that in the communication the attorney makes an argument as to why the company has an ownership interest?

When you say you had no "employment agreement" in place, does this mean that there was nothing in writing? Or do you mean that there was no agreement whatsoever regarding relative contributions, time and financial commitments, division of equity, etc.?

If you would like to discuss this matter, feel free to contact me at (805)259-5778. I am an attorney with a computer engineering degree, and I specialize in IP, corporate, and HR related matters and would be willing to discuss the matter with you.

Kenneth Stein, Esq.

<comment><AUTHOR> Hey, I really appreiate your reply. This case is taking plae in Toronto and I have little to no money to pay you. I really do appreciate your feedback.

CEO is saying that since I got the idea while working for him, he believes that he deserves all of my project. They have money so they got a lawyer sending me mail saying I should stop it.

I talked to lawyers in the field of employment, IP and corporate to see if there is any issues. So I am clean. But they want to stop me from raising money by sending me statement of claim.</comment></comment>


Proving that Android's, Java's and Python's sorting algorithm is broken - <AUTHOR> http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/ <comment><USER2> Their corrected version:

        if (n > 0 && runLen[n-1] <= runLen[n] + runLen[n+1]
        || n-1 > 0 && runLen[n-2] <= runLen[n] + runLen[n-1])

In the first clause they add earlier to later runLen elements, but in the second they add later to earlier elements. Switching the order just makes the expression harder to understand, like reusing variables within a scope. Addition is commutative and there's nothing technically wrong with it, but this construction makes it appear like there may be something special about element n when there's not.

The programmer also has to do mental arithmetic to check the bounds. Bounds checks can be written so that the largest index subtracted is also the value tested:

        if (n >= 1 && runLen[n-1] <= runLen[n] + runLen[n+1]
         || n >= 2 && runLen[n-2] <= runLen[n-1] + runLen[n])

This makes it easier to see that the bounds are correct. Formal methods found an important bug that would not have been found otherwise, but a lot of lesser bugs can be prevented just by writing clear and consistent code.

<comment><USER3> Question: Formal methods found the bug, but now that we know about it, what lessons can human programmers learn? That is, in the spirit of "20/20 hindsight," can we see the false assumption that made the bug possible as an instance of a certain kind of mistake, which we can look for and avoid in the future?

Or do you think the only lesson here is "Never fully trust anything that hasn't been formally verified"?

<comment><USER4> To me, [https://mail.python.org/pipermail/python- dev/2002-July/02689...](https://mail.python.org/pipermail/python- dev/2002-July/026897.html) reads like the bug was known (err, the possibility of running out of slots in the bookkeeping stack was understood). That indicates that it could be an intentional tradeoff.

<comment><USER5> The source code for the python version has this comment:

      /* The maximum number of entries in a MergeState's pending-runs stack.
      * This is enough to sort arrays of size up to about
      *     32 * phi ** MAX_MERGE_PENDING
      * where phi ~= 1.618.  85 is ridiculouslylarge enough, good for an array
      * with 2**64 elements.
      */

According to the linked article that's incorrect, it is only good enough for an array with 2^49 elements. So, if the (implicit) design guarantee is for lists up to 2^64, the implementation is technically bugged. But it'd be pretty hard to run into it in practice. 2^49 of python's plain integers consume more than 4.5 petabytes of RAM.

<comment><USER6> 2⁴⁹ item references in a Python list will consume about 2⁵² bytes of RAM on an LP64 system, which is 4 pebibytes (4.5 petabytes, as you said). You might also need to allocate space for the objects that the references refer to, which will almost certainly be bigger. I don't understand the bug well enough to know if it will work with large numbers of duplicate/identical items.

Nitpick: the idiomatic term is "buggy", not "bugged". Something is "bugged" if it has a hidden microphone in it transmitting to spies, not if it contains a software "bug".

</comment></comment></comment></comment></comment><comment><USER7> From the article: The reaction of the Java developer community to our report is somewhat disappointing: instead of using our fixed (and verified!) version of mergeCollapse(), they opted to increase the allocated runLen "sufficiently". As we showed, this is not necessary. In consequence, whoever uses java.utils.Collection.sort() is forced to over allocate space. Given the astronomical number of program runs that such a central routine is used in, this leads to a considerable waste of energy.

<comment><USER8> It looks like the Python version that's good for 2^49 items uses 1360 bytes for this array, allocated on the stack.

I wouldn't worry about an extra couple of bytes nearly as much as I would worry about changing the behaviour of a function used in an "astronomical number of program", so this looks like a pretty reasonable and conservative choice, at least for an immediate patch.

<comment><USER9> Does it "change behavior"? As far as I understand, the only change is that this test wouldn't crash anymore. Which certainly is "changing the behavior", but allocating more memory is as well. If so, I don't see any reason not to use formally verified version. Not that it is really important, but quite reasonable.

</comment></comment></comment><comment><USER10> This is actually really cool. They tried to verify Timsort as implemented in Python and Java using formal methods. When it didn't seem to work, they discovered that there was actually a missing case in both implementations, which could lead to array out of bounds exceptions.

Really shines as an example of how important proof is in computer science.

<comment><USER11> In the Haskell community there is a tool called QuickCheck [1]: it is able to generate inputs based on preconditions, and you provide a function to verify the postconditions.

I am not sure why these kind of testing methods aren't used more often in other communities, since it makes it really easy to catch corner cases; in the Haskell world, at least, using QuickCheck is somewhat pervasive.

[1] [https://wiki.haskell.org/Introduction_to_QuickCheck2](https://wiki.haskell.org/Introduction_to_QuickCheck2)

EDIT: I have never done this before, but could anyone explain why I am being downvoted? I wasn't making the claim that this is a substitute for a formal proof, I was merely adding this information to the discussion since it seemed relevant.

<comment><USER12> There's a difference, though: QuickCheck randomly tries a large-ish number of possible inputs to try and disprove a postcondition, whereas formal methods prove the postcondition for all inputs. Practically, if the number of possible inputs is huge compared to the number of failure cases, it's unlikely that QuickCheck will find it. I strongly suspect - and this would be an interesting experiment - that QuickCheck wouldn't have found this particular bug, since Timsort has been in use for years without anyone noticing it.

<comment><USER13> What about AFL?

<comment><USER14> What's AFL?

<comment><USER15> American Fuzzy Lop ([http://lcamtuf.coredump.cx/afl/](http://lcamtuf.coredump.cx/afl/)), a fuzzer which instruments programs at compile-time to help find interesting inputs faster than brute-force fuzzing.

It may have indeed found a bug in TimSort, since it's more apt at exercising branches, but I think AFL is C/C++ only.

</comment></comment></comment></comment></comment></comment><comment><USER16> Has anyone used the KeY project in industry?

I'd certainly prefer proofs over unit tests. However, I don't understand formal proof systems sufficiently well to know whether these they would work in terms of your typical "app" that makes RPC calls, DB changes, and generally has lots of moving parts and statefulness.

<comment><USER17> I have tried to use Coq and Frama-C to prove commercial OCaml and C programs, without, it has to be said, any success. I'm waiting for someone to write the brilliant tutorial.

<comment><USER18> You're waiting for a tutorial on Coq integrated with Frama-C, or for a tutorial on any of them?

<comment><USER17> I'm waiting for a tutorial on how to prove correctness of either OCaml or (especially) C code in real programs.

<comment><USER18> Research tools are getting closer to what people consider "real" programs, even if they often focus on embedded systems software. CompCert already deals with a quite large subset of C. Frama-C can deal with most if not all syntactic features of C, but then the next challenge is the C standard library, e.g. specifying every useful function (and even "simple" ones such as memcpy can be quite tricky). Afterwards, you have to deal with the glibc, then other high-level libraries, etc...

Most of these tools are either still in a mostly-academic setting (where "documentation = conference paper"), or do not have enough funding to pay for the development of more user-friendly features and extensive documentation. But with the ever-increasing security issues receiving media attention lately, we can hope more funding will allow these tools to reach a more mainstream status.

By the way, could you give an example of a small program that you would consider "real"? Just to have an idea of its size and complexity.

<comment><USER17> Firstly I don't expect to prove a whole program. However being able to prove the correctness of key functions or algorithms in a program could be useful.

You mention glibc, and it would be great to prove that (for example) 'qsort' is correct. That wouldn't be entirely trivial:

[https://sourceware.org/git/?p=glibc.git;a=blob;f=stdlib/msor...](https://sourceware.org/git/?p=glibc.git;a=blob;f=stdlib/msort.c;h=4e17a8874736d7a653b3f589fbb7ad253225c939;hb=HEAD) [https://sourceware.org/git/?p=glibc.git;a=blob;f=stdlib/qsor...](https://sourceware.org/git/?p=glibc.git;a=blob;f=stdlib/qsort.c;h=304fbed8e5d8d2edfd606ab7db6412e408c1d479;hb=HEAD)

For an example of a larger program, I'd like to prove various invariants of C programs, such as that 'reply_with_error' is called exactly once on every error path in this program:

[https://github.com/libguestfs/libguestfs/tree/master/daemon](https://github.com/libguestfs/libguestfs/tree/master/daemon)

</comment></comment></comment></comment></comment></comment><comment><USER19> Oh, crap. I suppose this also means we have the bug in our GNU Octave implementation:

[http://hg.savannah.gnu.org/hgweb/octave/file/0486a29d780f/li...](http://hg.savannah.gnu.org/hgweb/octave/file/0486a29d780f/liboctave/util/oct- sort.cc)

Well, time to patch it there too.

</comment><comment><USER20> Nice. As usual, entry and exit conditions aren't that hard to write; it's loop invariants that are hard.

      /*@ loop_invariant
        @  (\forall int i; 0<=i && i<stackSize-4;
        @             runLen[i] > runLen[i+1] + runLen[i+2])
        @  && runLen[stackSize-4] > runLen[stackSize-3])
        @*/

It's surprising how close their notation is to our Pascal-F verifier from 30 years ago.[1] Formal verification went away in the 1980s because of the dominance of C, where the language doesn't know how big anything is. There were also a lot of diversions into exotic logic systems (I used to refer to this as the "logic of the month club"). The Key system is back to plain old first-order predicate calculus, which is where program verification started in the 1970s.

For the invariant, you have to prove three theorems: 1) that the invariant is true the first time the loop is executed, given the entry conditions, 2) that the invariant is true for each iteration after the first if it was true on the previous iteration, and 3) that the exit condition is true given that the invariant is true on the last iteration. You also have to prove loop termination, which you do by showing that a nonnegative integer gets smaller on each iteration. (That, by the way, is how the halting problem is dealt with in practice.) #2 is usually the hardest, because it requires an inductive proof. The others can usually be handled by a simple prover. There's a complete decision procedure by Oppen and Nelson for theorems which contain only integer (really rational) addition, subtraction, multiplication by constants, inequalities, subscripts, and structures. For those, you're guaranteed a proof or a counterexample. But when you have an internal quantifier (the "forall int i") above, proof gets harder. Provers are better now, though.

A big practical problem with verification systems is that they usually require a lot of annotation. Somebody has to write all those entry and exit conditions, and it's usually not the original programmer. A practical system has to automate as much of that as possible. In the example shown, someone had to tell the system that a function was "pure" (no side effects, no inputs other than the function arguments). That could be detected automatically. The tools have to make the process much, much easier. Most verification is done by people into theory, not shipping products.

[1] [http://www.animats.com/papers/verifier/verifiermanual.pdf](http://www.animats.com/papers/verifier/verifiermanual.pdf)

</comment><comment><USER21> This is an excellent use case of formal verification. I can definitely see the merit in using tools like this in my code.

They mention KeY [http://www.key-project.org/](http://www.key-project.org/) . Is anyone using this here? Are there any good resources on it except for the official site (and this blog post)?

<comment><AUTHOR> Hi, I will ask the authors of the blog post and corresponding academic paper about additional KeY resources and follow-up.

<comment><USER21> Thanks and thanks for the interesting read. I've been looking for real uses of formal verification for a long time. I've played a lot with code contracts in C# and I've played some with languages like Eifel - the advantage of this approach is that it's static and it performs actual proof rather than enforcement.

These forms of formal verification could really help with building robust software and if someone makes them easy enough to use I can definitely see them as useful alongside if not instead of unit tests.

</comment></comment></comment><comment><USER22> Everyone here is talking about Timsort, but you should also check out the materials they've published about the KeY project, which they used to carry out this verification. [http://www.key- project.org/~key/eclipse/SED/index.html](http://www.key- project.org/~key/eclipse/SED/index.html) describes their "symbolic execution debugger", which lets you debug any Java code even if you don't know all the inputs (by simply taking the input as a symbolic value). The screencast is very accessible.

</comment><comment><USER23> Spark 1.1 switched default sorting algorithm from quicksort to TimSort as well in both the map and reduce phases- [https://databricks.com/blog/2014/10/10/spark-petabyte- sort.h...](https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html)

</comment><comment><USER24> Where can I get that PS1 from the shell in the video? It appears to have a green check mark if the previous command returned 0 otherwise some red symbol. Looks cool

<comment><USER25> for bash [http://stackoverflow.com/questions/16715103/bash-prompt- with...](http://stackoverflow.com/questions/16715103/bash-prompt-with-last- exit-code)

</comment></comment><comment><USER26> This is a good example of why formal verification is incredibly useful. I tried to invest some time in learning some of the proof verification languages and tools, but so far I wasn't too successful.

It looks like that to formulate a proof, I always have to rewrite the algorithm/problem first in the tool's language, which is often _not_ easy. I could see myself making mistakes in writing the proof just as well as I do when I'm programming.

Proof validation is also tricky. Coq isn't fully automatic as I initially was expecting. I actually used "prover9" which is first-order only, but does automatic validation. I guess Coq is really useful when you need to understand the proof and interactive validation can guide you, whereas prover9 could help with automation.

The thing is, it's still too much work, even for seemingly simple algorithms, to write a proof in either system in order to improve on the current situation of unit testing (that is: if I wanted to get something with more intrinsic value than a test case).

Formally verified languages are nice, but for a gazillion of reasons you still need to verify what's running currently.

<comment><USER27> With tools like Coq, you may even have the benefit of extracting an implementation from your proof! Some assembly may be required though, and extraction only works to functional languages.

</comment></comment><comment><AUTHOR> The authors have posted a follow-up posting: about KeY - the tool used to prove the bug in TimSort - [http://envisage-project.eu/key-deductive- verification-of-sof...](http://envisage-project.eu/key-deductive-verification- of-software/)

</comment><comment><USER28> And I always thought sort functions are always tested with millions of randomly generated sequences and the results are compared with the results of other implementations known to be good.

<comment><USER4> That was done. It's talked about here:

[http://svn.python.org/projects/python/trunk/Objects/listsort...](http://svn.python.org/projects/python/trunk/Objects/listsort.txt)

The bug would only be triggered by generating a truly massive array (the implementation mentions that it will work for up to 2^64 elements: [http://svn.python.org/projects/python/trunk/Objects/listobje...](http://svn.python.org/projects/python/trunk/Objects/listobject.c) search for MAX_MERGE_PENDING).

<comment><USER29> As mentioned by another commenter, the bug is that it's (for python) documented to work for 2^64 elements, but "only" works for 2^49. 2^49 is still pretty big... note that for 64-bit integers, 2^49 integers is 2^(49+3)=2^52 bytes... or 4 _petabytes_ of raw data. Even if you're sorting single bits (one and zero) it's quite a bit of data to chew through.

[ed: Hm, that's not quite right. A 64-bit integer is 2^6, so that should be 2^(49+6)=2^55, or 32 petabytes). I think :-) ]

</comment></comment></comment><comment><USER30> This is bad-ass. Thanks for taking the time and investigating something that so many overlook yet use every day (myself included)

</comment><comment><USER4> Fixed in python:

[http://bugs.python.org/issue23515](http://bugs.python.org/issue23515)

</comment><comment><USER31> anyone know the list of bugs for this?

[https://bugs.openjdk.java.net/browse/JDK-8072909](https://bugs.openjdk.java.net/browse/JDK-8072909)

It seems they havn't submitted to any other trackers, which is a bit unfortunate

<comment><USER27> There is also [http://bugs.java.com/view_bug.do?bug_id=8011944](http://bugs.java.com/view_bug.do?bug_id=8011944), where IIUC the suggested "fix" was to use a VM switch to enable the old (and slower) sorting...

</comment></comment><comment><USER32> I'm surprised that modern implementations don't use something like quicksort, since it has linear space requirements (sorts in-place) and good constants on average for its n*log2(n) running time.

<comment><USER33> Quicksort has poor worst-case behaviour, so it's easy to carry out a DoS attack on network services that use it. Language maintainers don't like restricting their library to input from friendly users, at least not when there are other algorithms that work reasonably with unfriendly input.

<comment><USER34> Isn't the malicious attack fought by just picking a random pivot?

<comment><USER35> There are still inputs that will make it go quadratic, that just obscures it slightly. A proper fix is to fall back on a different sort if it looks like quicksort is doing that: [https://en.wikipedia.org/wiki/Introsort](https://en.wikipedia.org/wiki/Introsort)

<comment><USER36> > There are still inputs that will make it go quadratic

Can you elaborate on this? To force quicksort into a quadratic running time, you need to ensure that each pivot splits off a bounded number of elements (e.g. no more than three, or no more than twenty million) from the rest of the list. If the pivot is being chosen at random, then it looks to me like the guarantee you'd need to make is "every single element of this list [because any of them might be chosen] is larger, and smaller, than no more than _k_ other elements of the list". But as the size of the list grows, that condition forces it to be mostly composed of the same element repeated over and over again, which is really easy to sort, and in particular is really easy for quicksort to handle.

<comment><USER37> See my bug report on libc++: [http://llvm.org/bugs/show_bug.cgi?id=20837](http://llvm.org/bugs/show_bug.cgi?id=20837) .

<comment><USER36> But that adversary works by causing the sorting agent to run arbitrary, adversary-supplied code every time it makes a comparison. It has to do that because it invents the values in its list on the fly when it detects them being accessed by the sort (also, in order to detect them being accessed by the sort). That's not really the same thing. I mean, if you want to tie up a process, and you can already make it run arbitrary code that you supply, just give it something like while(1); .

If this adversary were forced to realize all the values it fed to the sorter before the sorter did any work, or if it were unable to supply its own code to the sorter, random pivot selection would be a defense.

edit:

I feel like pointing out that a quicksort implementation could defeat this adversary, without hurting its O(n log n) running time, by just comparing the first element of the sublist it was working with to every other element in the sublist -- and throwing away the results -- and then proceeding as normal. This is O(n) comparisons, which violates the vulnerability criterion of making only O(1) comparisons per call, but doesn't affect the big-O running time at all. What it does do, with an eye to this particular adversary, is realize all the values before doing any sorting work. It still doesn't fix the actual vulnerability the paper identifies, which is that you're running adversary- supplied code. I'm growing to feel like your bug report was frivolous.

<comment><USER37> What you fail to realize is that this attack is simply an academic proof that shows the libc++ implementation is broken and can have quadratic performance. This is a bug, because the C++ standard mandates a worst case of O(n log n).

What you also don't seem to realize is that this attack merely uses a comparison function to _find_ the worst case. Once the worst case is found you can feed this input to any program using libc++'s std::sort, without comparison function, and trigger the worst case.

So no, this is not frivolous at all.

<comment><USER36> > What you also don't seem to realize is that this attack merely uses a > comparison function to find the worst case. Once the worst case is found you > can feed this input to any program using libc++'s std::sort, without > comparison function, and trigger the worst case.

This is true iff pivots are selected deterministically. In which case, why did you post it as a response to "how can an adversary force quadratic behavior when pivots are chosen randomly?"

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER38> Wish they'd put that up on Github or Bitbucket. There are so many things to be learned from that. I don't want to just download things or just use a tool (although it's pretty awesome that they made the tool available.)

<comment><USER27> What are you looking for?

<comment><USER38> To watch how change sets are going into the tool. To see what is being done and why as it happens. I find that one of the best ways to learn these things. Especially when it is theory heavy and watching the implementation congele around the theoretical framework.

<comment><USER27> Sorry to keep following up, but we'd really like to know what we can improve -- into which tool? Are you talking about the KeY tool, or the various libraries they are now catching up on this issue?

</comment></comment></comment></comment><comment><USER39> Just a nitpick, but the article makes a mathematical mistake. In the last paragraph of section 1.2, it says

        For performance reasons, it is crucial to allocate as
        little memory as possible for runLen, but still enough to
        store all the runs.  *If the invariant is satisfied by all
        runs, the length of each run grows exponentially (even faster
        than fibonacci: the length of the current run must be
        strictly bigger than the sum of the next two runs lengths).*

However, fibonacci growth is strictly faster than exponential. In fact, this is why n * log(n) is the lower bound on the number of comparisons a comparison-based sorting algorithm must use: because n! is approximately n * log(n).

<comment><USER40> None of this is true.

 _> fibonacci growth is strictly faster than exponential_

There is even an explicit formula for F(n) that shows fibonacci growth is _exactly_ exponential[1]

 _> In fact, this is why nlog(n) is the lower bound on the number of comparisons a comparison-based sorting algorithm_

It's true that lower bound for comparison-based sorting is nlog(n) but it has nothing to do with fibonacci

 _> n! is approximately nlog(n)_

Stirling's formula[2] says otherwise (that n! is exponential)

[1] [http://en.wikipedia.org/wiki/Fibonacci_number#Closed- form_ex...](http://en.wikipedia.org/wiki/Fibonacci_number#Closed- form_expression)

[2] [http://en.wikipedia.org/wiki/Stirling%27s_approximation](http://en.wikipedia.org/wiki/Stirling%27s_approximation)

<comment><USER39> How embarrassing! It looks like I flipped around Fibonacci and factorial in my head when writing my reply.

You're completely right that the comparison-based sorting bound has nothing to do with Fibonacci numbers. And in fact log_2(n!) is about n*log_2(n), which is where the comparison-based sorting bound comes from. And Fibonacci growth is absolutely exponential, just with an exponent larger than 2.</comment></comment></comment>


Creating a deepfake took two weeks and cost $552 - <AUTHOR> https://arstechnica.com/science/2019/12/how-i-created-a-deepfake-of-mark-zuckerberg-and-star-treks-data/ <comment><USER3> A few weeks ago my girlfriend made me discover Instagram filters. Obviously they're not in "deep fake" territory but I was still massively impressed by the quality of the face recognition and tracking coupled with some rather advanced 3D reconstructions on top.

15 years ago that would've been science fiction, today it's a free gimmicky feature in an app running on commodity hardware. There are also apps that will smooth your skin or change the shape of your eyes in real time.

All these technologies are evolving at a ridiculously fast pace, I'm sure that a decade from now you'll be able to do a much more convincing fake in a fraction of the time. We're getting deeper and deeper into a post-truth world.

<comment><USER4> I don't really understand this argument at all. We've been in this "post truth" world for a long time already for every form of media other than video. Text quotes, photos, and audio can all be easily faked. If I post a ridiculous quote here and say it's from Obama, you won't believe me. But if the NYTimes does the same thing, it carries a lot more weight. We've been here for a long time already, now those standards will just apply to video as well. Doesn't seem that crazy.

Also, I'm fascinated by the idea of using AI to detect the use of AI. How good will your deep fake have to be in order to fool a deep fake detection AI?

<comment><USER5> > But if the NYTimes does the same thing, it carries a lot more weight.

Sure, and that's part of the danger. Convincing deep fakes will be used to delegitimize mainstream news organizations, as with what happened with Dan Rather - a trusted source presented faked documents.

<comment><USER4> But why is this the danger now in a way that it hasn't been previously? You could have presented fake documents for hundreds of years now, but it's not like we don't have any credible news organizations as a result.

<comment><USER6> It gets increasingly cheap to manufacture convincing fakes, but it doesn't get any cheaper to do quality journalism. The risk as I see it is that there is such an overwhelming tsunami of manufactured reality that it becomes impossible to tell truth from fiction.

<comment><USER7> It was never that expensive to manufacture fake text. Minorly expensive to manufacture fake images.

And we've been hit by tabloids, spam and so on before. None of that is new. Overwhelmed by content is/was/will be a problem that is solved very simply: people limit the distribution lanes they consume, and naturally establish "trust networks" and hierarchies.

Nyt didn't become "trusted" by accident. People aren't as dumb as you seem to think.

Anyways the problem is of distribution, not that a lie can be told. If we can't trust our distribution lanes to actually reflect the institutions we're trusting, then we can't establish our trust networks.

And this is a real problem: our distribution lanes are divorced now from our actual information distrubuters; medium, Facebook, Twitter, etc fuck around with our trust networks, randomly injecting their own bullshit into our feeds and messing around with feed-order based on non-trust metrics (eg money paid) such that they've become fairly unreliable.

And our classical trustable institutions have become less trustworthy, as they flounder about trying to make sense of the "digital age", and have so far done so in a pathetic fashion

<comment><USER6> In the past you had to pay a person to write that fake text. Wait a few more years and you'll see convincing text coming out of some algorithm at the cost of an EC2 instance.

<comment><USER7> That's fine and well, but again, I don't think the existence of it is new; if they generate 1 lie a day, or 1000, or a million, it doesn't change much. It gets resolved by the trust distribution network.

When that network breaks down, you only need a single good lie to do the damage.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> The central technique used in DeepFakes is fascinating. Initially I assumed they were using Generative Adversarial Networks (GANs). But — and someone can correct if I'm wrong — Deep Fakes use two Autoencoders.

So essentially you're training a network how to compress an image down to a very tiny representation, and then uncompress it as accurately to the original as possible.

You train two of these: one for the original face, and one for the target face. Then, you compress with the "source" autoencoder, and then _uncompress_ with the "target" autoencoder. And, voila, 'source' face becomes 'target' face.

<comment><USER9> How do you guarantee that the two trained autoencoders speak the "same language", i.e. the code learned by the autoencoder of the original face gives a reasonable output when uncompressed by the target autoencoder?

<comment><USER10> This is a good question. I'm not sure which paper the parent comment is referencing - and in general, DeepFakes is a very general term for a variety of techniques including GANs. The problem you describe is called the alignment problem and it's studied in several areas of applied ml: translation, multi- modal learning, etc.

For example, the canonical way to train deep machine translation systems is to make use of large parallel corpora - large set of translation pairs. In the case where this data may not exist, for structurally similar languages, one approach is to learn unsupervised word embeddings independently (i.e. using word2vec) which only require monolingual data and to find a transformation that aligns the two spaces. In many cases, neural networks still struggle with this task. However, some work in the last few years has demonstrated that a few seed word-translations is all you need to apply simple algorithms that can learn a linear or simple nonlinear transformation that work pretty well!

[1] Here is a method that proposes locally linear transformation maps for alignment: [http://nakashole.com/papers/2018-emnlp- norm.pdf](http://nakashole.com/papers/2018-emnlp-norm.pdf)

</comment></comment></comment><comment><USER11> That's a terrible Deepfake to be honest. This is a much better example.

[https://www.youtube.com/watch?v=bPhUhypV27w](https://www.youtube.com/watch?v=bPhUhypV27w)

<comment><USER12> More convincing for the face, sure, but last I checked Schwarzenegger had much, much more body mass than in that video. Does anyone know if the technology for faking body shape (and accessories too maybe) is anywhere near as good as for faces yet?

<comment><USER11> > More convincing for the face, sure

Uh.. That's what a DeepFake is? Look up the definition.

<comment><USER12> From [https://en.wikipedia.org/wiki/Deepfake](https://en.wikipedia.org/wiki/Deepfake):

> Deepfakes (a portmanteau of "deep learning" and "fake"[1]) are media that > take a person in an existing image or video and replace them with someone > else's likeness using artificial neural networks.

"Likeness" refers to appearance in general, not just the face.

</comment></comment></comment></comment><comment><USER13> Well Zuckerberg and Data do look like they were separated at birth, which I guess would make Zuckerberg Lore - which might also explain a few things.

<comment><USER14> So FB is the crystalline entity or the rogue Borg?

<comment><USER13> Facebook does seem to have a strategy of assimilating every successful social platform.

</comment></comment></comment><comment><USER15> interesting conclusion:

 _> In the long run, the larger risk may be that public attitudes swing too far in the opposite direction: that the possibility of deepfakes completely destroys public trust in video evidence. Certain politicians are already in the habit of dismissing media criticism as "fake news." That tactic will only become more effective as public awareness of deepfake technology grows._

Deepfakes themselves will not _disrupt_ democracies or cause the much anticipated chaos. Deepfakes are in fact pointless for agitprop / disinfo campaigns:

 _" cheapfakes beat deepfakes"_ \--@thegrugq

[https://twitter.com/thegrugq/status/1206404680223358976](https://twitter.com/thegrugq/status/1206404680223358976)

[https://medium.com/@thegrugq/cheap-fakes-beat-deep- fakes-b1a...](https://medium.com/@thegrugq/cheap-fakes-beat-deep- fakes-b1ac91e44837)

What is fascinating (and uncommon) is, that the damage isn't delivered with any specific deepfake itself. But the mere awareness that the technology exists, and that they could _potentially_ be used, is enough to change our risk perception. Even if the ratio cheapfakes/deepfakes usage in actual disinfo campaigns remains tiny percentage, deepfakes make much better headlines. (in a similar way the infosec industry is obsessed with 0days - <USER2> most of the risks are coming from people clicking on links in email)

Maybe one day we will look at deepfakes like "art". If pictures and videos can be art, and if "all art is propaganda" (see Orwell), then deepfakes too should be considered art, no?

</comment><comment><USER16> I mean. No.

I understand the point of the article is "look how much I did with so little." And it's really good in terms of how much was done.

But the final deliverable isn't at all convincing. Clickbait title.

<comment><USER17> Made by a dude who has next to no experience in the field. Now imagine what a guy who knows what he's doing can do with a bit more money. That's what the allusion here is; the fact that a Reporter with 500$ even comes close should be an alarm signal to you.

<comment><USER18> This is also an impressive DeepFake:

[https://www.youtube.com/watch?v=Wm3squcz7Aw](https://www.youtube.com/watch?v=Wm3squcz7Aw)

Voice actor reads a poem in voices of famous people; his face gets swapped with the actor's face; the combination of voice and face makes it very believable.

<comment><USER19> This one is fantastic. It seems like just the most characteristic traits are carried over and the result is seem less.

</comment></comment></comment></comment><comment><USER20> You can do these overnight on high end gaming GPU boxes with [https://github.com/iperov/DeepFaceLab](https://github.com/iperov/DeepFaceLab)

</comment><comment><USER21> Can content providers not include a digital signature in the video metadata? Then you can authenticate the video against the original provider, a bit like the green padlock next to the URL bar in Chrome.

Even better (though I'm not sure this technology exists yet), some some kind of "rolling signature", available running along side the video in metadata, that a media player can use to validate the authenticity of the video at that point in time. Then you don't need the full video to be able to verify the source.

</comment><comment><USER22> I couldn't tell it was Data from Star Trek until I read the story.

</comment><comment><USER23> Thinking ahead - will Hollywood in 50 years time be actor based or digital rights based when your film is computer rendered and you pick your own actors that get digitally placed. Effectively reducing actors to DLC!

Remember, technology to fake voices has already be done. [https://www.youtube.com/watch?v=I3l4XLZ59iw](https://www.youtube.com/watch?v=I3l4XLZ59iw)

Certainly interesting times ahead.

</comment><comment><USER24> There's a newer model that can be used pre-trained and doesn't need to be retrained for each face pair: [https://nirkin.com/fsgan/](https://nirkin.com/fsgan/)

But for now deepfakes seem like a 2018 meme that died out.

</comment><comment><USER25> This work is just at the research stage right now and isn't available as consumer software, but it probably won't be long until similar services go public.

</comment><comment><USER26> Brent Spiner is by far the best actor on TNG. Even Zuck wearing a Cmdr Data skin is an inferior copy.

</comment><comment><USER27> That just looks like Mark Zuckerberg to me.</comment>


Ask HN: Looking for email delivery for 500k subscribers - <AUTHOR>

Hey HN,
I run a site that has 500k email subscribers, and were looking to do a big mass mailing campaign in the upcoming months.
We are planning on sending close to 6 campaigns (so 3 million emails) in one month. What is the best solution for us to role this out? Were looking for something our non-technical marketing guy can use, but that I can set up. <comment><USER2> MailChimp allows you to buy credits as you go, or pay them monthly by subscriber count. It's also built for marketers so after importing your list there your marketing guy should be fine.

If you're looking to do it cheaper, SendGrid offers a newsletter function that's less easy to use and has less features but will allow you to do it cheaper.

<comment><AUTHOR> Mailchimp is really, really expensive for anything this size. I contacted SendGrid & so far they look like the front runners

</comment></comment><comment><USER3> I would recommend Amazon SES (<http://aws.amazon.com/ses/>) Take a look at <http://sendy.co/> as it's a PHP script if you don't want to write your own.

<comment><AUTHOR> I use sendy for a personal project, but amazon SES limits you to 10k emails a day. For our use case, we need to send 500k in one day. Anyone have experience requesting a higher send limit with SES?

<comment><USER3> What about <https://postmarkapp.com/pricing> would that be an option for you?</comment></comment></comment>


These problems were designed to prevent Jewish people from passing (2011) - <AUTHOR> https://arxiv.org/abs/1110.1556 <comment><USER2> An alternate title would be: Jewish students were given more difficult math problems than their Russian non-Jewish counterparts

Otherwise, the title implicitly implies that Jewish and non-Jewish students received the same questions. That would be interesting!

<comment><USER3> An example of such a test, taken from the Old Testament, is a shibboleth. From the book of Judges:

Jephthah captured the shallow crossings of the Jordan River, and whenever a fugitive from Ephraim tried to go back across, the men of Gilead would challenge him. "Are you a member of the tribe of Ephraim?" they would ask. If the man said, "No, I'm not," they would tell him to say "Shibboleth." If he was from Ephraim, he would say "Sibboleth," because people from Ephraim cannot pronounce the word correctly. Then they would take him and kill him at the shallow crossings of the Jordan. In all, 42,000 Ephraimites were killed at that time.

<comment><USER4> "In all, 42,000 Ephraimites were killed at that time."

Reminds me of the old joke which I can't remember enough details of to preserve the humor, where a dumb FBI agent is given the task of hunting communists, and his boss gives him a test which he humorously misunderstands, and at the end of the week he proudly reports "I've already killed several thousand communists sir. And you sent me here just in time, the communists were really entrenched, I haven't found a single loyal american." Sorry I botched it.

But anyway, this is a fascinating archaeological record of a classic bias, where you assume your test was 100% accurate because the opposite would be horrifying...

<comment><USER5> Like why the death penalty in the USA has never been applied to someone who didn't deserve it: because the alternative would be horrifying.

</comment></comment></comment></comment><comment><USER2> This situation slightly reminds me of the interview process at some tech companies. Over the past year, I have interviewed on-site at 10 different companies. Two of those companies had one interview where the questions were extremely difficult and uncommon. When I told my friends those questions, many also thought the questions were bad interview questions.

It is important in any selection mechanism, like interviews, that we have a fair and consistent method of selecting applicants. Yet the questions need to be varied enough so that candidates cannot cheat by hearing the questions ahead of time.

<comment><USER6> _This situation slightly reminds me of the interview process at some tech companies._

This is _exactly_ the same as the whiteboarding algorithms questions. They all have a simple answer which is obvious if you have been recently cramming for your CS finals. But you are unlikely to remember them in sufficient detail if you are over a certain age, because no real actual working programmer needs to know that stuff by heart, it's either been abstracted into libraries or you just reach for a book when you need one. A perfectly plausible way to do ageism, right under the noses of any regulators.

<comment><USER2> I actually would not agree. There is a common set of knowledge that one is expected to know, even if it isn't actively used daily. During the interview process, it was clear that many of the companies asked similar questions. It was possible to prepare for.

However, the two companies I referenced asked questions that were close to impossible to prepare for. These were also not whiteboarding questions (both did have separate whiteboarding sessions which were fair)

<comment><USER7> What were the impossible questions? (You're not really expected to keep them secret, so it's fine to list them.)

To your point, it's strange that there's a common set of knowledge we're expected to know which we don't use, ever. Not just "in 99% of cases," but literally ever: If it ever comes up in the field, you can look up what you need to know in a few minutes.

Few would think that's a sensible way to hire someone, so I've spent a long time wondering why this is the case. I think it's because the hiring process implicitly selects for whoever makes it through. Meaning, if someone is giving you a test, they're biased into thinking it's a good test: After all, they passed it. That's why they're in a position to give it to you.

So it almost doesn't matter whether it's a good test or a bad test. Just that it's a test that the interviewer happens to like. There's enough talent floating around that people are willing to do whatever strange rituals you demand they do, regardless of whether it's efficient for them to know that knowledge. Since it's ambiguous whether the knowledge actually helps anyone, you get the present situation.

</comment></comment></comment></comment><comment><USER8> Pretty much exactly the same thing happens with Asian students in US universities today - admission standards vary widely by ethnicity/race. Jews were 2% of population but 10-20%+ of university students. Reverse discrimination was a thing then and is a thing now.

<comment><USER9> >reverse discrimination

No such thing. Discrimination is discrimination, no matter which way it's applied. I wish people stopped saying "reverse (discrimination|racism)" whatever as if these words meant discrimination against specific groups, but not others.

<comment><USER10> I think it's meant to imply the reasoning behind it. "Standard discrimination" is done out of fear or hatred, and "reverse discrimination" is done out of ignorance and a misguided desire to "help."

</comment></comment></comment><comment><USER11> My dad was a mechanical engineer in USSR who got his education in late 70s. The discrimination was worse in some areas. Moscow - forget about it. My dad got his degree in Minsk, and had many jews from other "states" move there just so they can get into college.

That is why there are somany known Soviet mathematicians with Jewish names. They still had paths to persue higher education, just not in Moscow and other desirable areas.

</comment><comment><USER12> For those who are interested, Edward Frenkel's book "Love and Math" chronicles his personal experience of taking such a test, and being denied entrance to Moscow State University despite superb performance in the exam. (He later got Math PhD in Harvard at age 23 and became a professor at UC Berkeley)

<comment><USER13> Some of his story is in this article from the _Notices_.

[http://www.ams.org/notices/199910/fea- saul.pdf](http://www.ams.org/notices/199910/fea-saul.pdf)

</comment></comment><comment><USER14> Reminds me of the white australia policy. You were given tests in european languages that the immigration officer chose. It was primarily meant to keep out the chinese and japanese, but also used to keep out southern/eastern europeans and of course jews.

A famous example was of Egon Kisch who was fluent in many european languages and kept passing the dictation tests in many european language. So they finally gave him a test in scottish gaelic and he failed. He was able to sue the government and win the right to stay in australia eventually.

"Jewish political activist Egon Kisch from Czechoslovakia, who was exiled from Germany for opposing Nazism, arrived in Australia in 1934. The Government of Joseph Lyons went to extraordinary lengths to exclude Kisch, including using the dictation test. Kisch was fluent in a number of European languages, and after completing passages in several languages, he finally failed when he was tested in Scottish Gaelic."

[https://en.wikipedia.org/wiki/Immigration_Restriction_Act_19...](https://en.wikipedia.org/wiki/Immigration_Restriction_Act_1901)

</comment><comment><USER15> I grew up in the 90s and wondered why every single Russian person I knew was Jewish. The wikipedia link is interesting. [1]

One non-Jewish person claimed there was no anti-Semitism. Everyone wanted to escape the Soviet Union and people just resented Russian jews because it was easier for them to leave. So I thought maybe anti-Semitism is so engrained that they didn't even notice it. Kind of like how some white Americans don't believe racism exists.

[1] [https://en.wikipedia.org/wiki/History_of_the_Jews_in_Russia](https://en.wikipedia.org/wiki/History_of_the_Jews_in_Russia)

<comment><USER16> It depends on what part of the Soviet Union he was from. If he was from the place with low Jewish population then what he said is actually true.

I am originally from central Russia and I've learned about anti-Semitism in Soviet Union only after moving to US. Yap, never heard or witnessed anything like that while living there.

Another interesting thing is that several of my Jewish friends told me that their parents asked them to lie about anti-Semitism when they were interviewed at the US embassy in Moscow in the late 90s.

</comment></comment><comment><USER17> Also worth looking at are literacy tests for voting rights given to African Americans in the Jim Crow-era southern United States.

[http://www.slate.com/blogs/the_vault/2013/06/28/voting_right...](http://www.slate.com/blogs/the_vault/2013/06/28/voting_rights_and_the_supreme_court_the_impossible_literacy_test_louisiana.html)

<comment><USER18> The reasons for and way in which they administered that are appaling. Still it made me think doing this the other way around would be interesting: would making would be politicians pass some form of unbiased well-designed test on the constitution, political system, financial principles, ethics, etc. have a positive effect on their performance? My guess is probably not but I do wonder what you could do if you let anything go.

<comment><USER19> > would making would be politicians pass some form of unbiased well-designed > test on the constitution, political system, financial principles, ethics, > etc. have a positive effect on their performance?

Who would be the authority that either makes these test questions, or monitors that they are unbiased?

</comment></comment></comment><comment><USER20> It wasn't just the USSR, though AFAIK it was much worse there (and still is in Russia and parts of E. Europe). In the U.S., I know a Jewish family that in the 1950s changed their last name to something not associated with Jewish people so that a child could get into college; there were quotas.

</comment><comment><USER21> Makes you wonder what kind of similar things that are designed to prevent access that we have from other groups in the past, present and future.

<comment><USER22> Probably all the typical bad programming interview questions. The ones that target people who can set aside a month of no work, no family, and no 'other responsibilities' to study tangentially related riddles and similar.

Culture fit probably. Though it'll just be replaced with something similar.

<comment><USER23> I think you have to just assume that there is a lot of stupid in any system, so it can't be _all_ of those questions.

The people out there who drive drunk or without insurance aren't doing it with the intent of killing people or not having to pay for their damages: they're doing it because they "didn't think about it".

Hanlon's razor tells us that a similar phenomenon is at play in many "bad interview questions".

<comment><USER24> Or, the emergent result of our hiring processes (alienating and near-untenable interviews coupled with fast tracks for sponsored candidates) is that cultural outsiders (women, people of color, older workers) are effectively locked out of higher-status resume-building jobs, and when companies try to fix those processes even a little, engineers are confronted with culturally unfamiliar potential coworkers and experience the human default of hypervigilance in unfamiliar social situations, which creates a false perception that reformed, rationalized hiring processes are "lowering the bar" when in fact the opposite would probably be happening, with a long term net result that nothing changes, and our industry remains structurally biased against people who can't pass for proto-Zuckerbergs.

<comment><USER25> I call affirmative action policies lowering the bar. If you are intentionally seeking people based on criteria like race or gender, then you are less likely to find the best candidates, unless you believe those are relevant criteria.

<comment><USER26> What does intention have to do with it? Are you saying that if you unintentionally seek people based on criteria like race or gender you're more likely to find the best candidates? Affirmative action, as you call it, is an intentional effort to overcome the sort of unintentional bias that tptacek described.

By the way, one reason people react strongly to objections like yours is that a very direct interpretation is that you believe the distribution of people in high status jobs actually does accurately reflect differences based on race and gender. Not sure if you intended that.

<comment><USER23> > By the way, one reason people react strongly to objections like yours is > that a very direct interpretation is that you believe the distribution of > people in high status jobs actually does accurately reflect differences > based on race and gender.

I think this is partly true. But then, a similar interpretation of "pro- affirmative action" is that one believes that whatever bad thing happens to a minority is never their fault.

I think both are only partly true.

The problem with affirmative action is that it (I don't know if it's always the case, but it seems to be often so) specifically considers minority status. It is obvious, I think, that this is not the kind of criteria a computer, say, would consider if it were to look only for the most qualified applicants.

On the other hand, you or someone else in the thread made a good point: this pro-minority bias can be seen as an effort to counter the systematic bias against "outgroups".

The problem is that I don't think we know the effect size of each of those: how strong is this systematic bias? In a protected status agnostic society, would all professional fields perfectly mirror population rates?

I don't think so. We know there are metal disorders/illnesses that impact cognitive functions (Down's syndrome) and obviously genetics is partly responsible (do Down syndromers have similar rates of Down syndrome babies?). Similarly, I don't think the success of Asians and Jews is due to some form of societal bias in their favor.

In the end, the question is whether the anti-bias bias is correcting towards 0 or in the other direction (i.e.: protected status is very important, just the other way).

I don't know what the answer to this one is.

</comment></comment></comment></comment></comment></comment></comment><comment><USER27> Just sent this to my mom who's told me how this happened to her.

> Wow wow wow. I knew I was not the only one, but it still hurts. I looked at > some problems: the most horrific part was that the guy next to me was asked > what is the tangent. He got 4(/5). I got 3.

</comment><comment><USER28> What's the funniest here? Jews that were rejected from MSU went straight across the street to Institute of Oil. And were promptly admitted. Guess it helped them in post-Soviet times.

</comment><comment><USER29> Previous discussions:

[https://hn.algolia.com/?query=jewish%20problems&sort=byPopul...](https://hn.algolia.com/?query=jewish%20problems&sort=byPopularity&prefix&page=0&dateRange=all&type=story)

</comment><comment><USER30> Today, Asians are required to achieve higher grades to be accepted to colleges. How is this different?

<comment><USER31> Asians are accepted to colleges. The point of these problems is to prevent any undesirable groups from passing the exam ever.

</comment></comment><comment><USER32> I was hoping for questions they couldn't answer right because of their beliefs.

<comment><USER33> Most of the kids applying were atheists or agnostics. They were Jews by ethnicity, not by religion.

<comment><USER32> I thought every ethnicity had jews

<comment><USER33> This is a source of major confusion. Ultimately, it boils down to how one identifies him- or herself or, alternatively, how one is identified by others. Jews in USSR were officially considered an ethnicity like any other, based on blood. The Judaism as a religion was suppressed under the Communist rule. But even before that, there was a lot of assimilation and secularization going on. Ultimately, in the 1970's there were very little Judaism practitioners left.

The word "Jew" ("Yevrey") in Russian usually refers to the ethnicity. They use a different word ("Iudjey") when they want to describe someone as a Judaism practitioner.

</comment></comment></comment></comment><comment><USER34> Unfortunately, these "ideas" for the problems are low quality. Problem 2: "use derivatives". How the heck that helps if F does not have to be continuous, let alone differentiable.

<comment><USER13> It suggests that F is differentiable, and as soon as you form the difference quotient you know everything. With the hint I think it's a really cool problem.

<comment><USER34> Certainly, but if the examiners would be accused of asking a too difficult question, as the article alleged, this would be defended by showing a more elementary approach. Or both of them, wouldn't hurt to have more ways of solving it. Unless the Soviet science was banned from using reductio ad absurdum proofs in some math version of Lysenkoism.

Out of curiosity, I've searched it and this article was posted to HN at least four times. Opinions aside, turns out it was merely a footnote in the exam pipeline [https://news.ycombinator.com/item?id=4761846](https://news.ycombinator.com/item?id=4761846)

</comment></comment></comment><comment><USER35> From the pdf we know these were still in use in 1975.

Anybody knows until when this went on?

<comment><USER36> Similar problems were given to everyone on the oral entrance exams in late 90s. They has nothing to do with ones ethnicity though and were given in case if there was disagreement if contestant deserves "5" or should be given "4". If problem(s) were solved "5" was given. (5 was max at that time)

</comment></comment><comment><USER37> Strange to hear, my father got into Michmat (Moscow State University Math Department) in the 70's. In 80's we immigrated to Israel.

</comment><comment><USER38> And now they are given to programmers at large companies during interviews... We should revolt as well ;-)

</comment><comment><USER39> What was the percentage of Jewish applicants passing these examinations despite all these efforts?

<comment><USER40> I recall reading elsewhere it was zero, they simply weren't allowed to admit Jewish students and these problems were just to try and cover up the discrimination.

<comment><USER36> Not true at all. How that explain then world known physicists and mathematicians from USSR with Jewish names?

<comment><USER41> If not for this quota, you would have much greater number of "known physicists and mathematicians from USSR with Jewish names" LOL In some places (Moscow University math dep-t) the target was about 2%, but I heard some technical schools didn't accept anyone (after Natan Scharansky affair. Long story...)

</comment></comment></comment></comment><comment><USER42> Jesus' full fish basket has a capacity of 0.1 cubic meters. Each fish in the basket has a volume of 0.0008 cubic meters and a packing density of 1.25. Jesus then distributes 33 fish. How many fish remain in Jesus' basket ?

Since the title of the post doesn't properly describe the article, this comment feels appropriate albeit insensitive.</comment>


Ask HN: Why do social apps try to scan my local network? - <AUTHOR> <comment><USER2> Port scans can detect fraud? [https://blog.avast.com/why-is-ebay-port- scanning-my-computer...](https://blog.avast.com/why-is-ebay-port-scanning-my- computer-avast) [https://www.komando.com/security-privacy/stop-ebay-from- prob...](https://www.komando.com/security-privacy/stop-ebay-from-probing-your- computer/740035/) Behave! [https://chrome.google.com/webstore/detail/behave/mppjbkhgcon...](https://chrome.google.com/webstore/detail/behave/mppjbkhgconmemoeagfbgilblohhcica?hl=en) Decentraleyes [https://chrome.google.com/webstore/detail/decentraleyes/ldpo...](https://chrome.google.com/webstore/detail/decentraleyes/ldpochfccmkkmhdbclfhpagapcfdljkj?hl=en)

</comment><comment><USER3> They use the local network environment as yet another input into the tracking/stalking algorithm to both track you and also track others (let's say you are on the same network as someone else and the social network doesn't yet know the association, based on network data being the same the social network can infer a connection between both of you).

</comment><comment><USER4> I'd like to know too. On Wireshark I saw that Facebook and Amazon are both involved in my computer somehow despite my never using those services. So I too want ab answer to this question. Mark</comment>


Sssgen: Make static websites simple again - <AUTHOR> https://ehuber.info/blog/why-sssgen.html <comment><USER2> Meta: Looking at the code at [0], all of the code is at the root level. Please, don't do that. It makes it quite hard to do static analysis on the code (i.e. pylint), since merely importing the code to do the code tree walk will execute the code logic. Ditto generating documentation from the code.

It can create a fair bit of boilerplate (relative to your average Python boilerplate), but the `if name == "__main__":` guard keeps this under control.

[0] [https://github.com/edmund- huber/sssgen/blob/master/bin/sssge...](https://github.com/edmund- huber/sssgen/blob/master/bin/sssgen)

<comment><USER3> Pylint doesn't actually import the code, just parses and processes the resulting AST, which is in my opinion how it should be done.

The guard is useful if your code is already composed of multiple functions/classes, which could be called from outside without running the full logic, but in this case I'm not sure it applies.

<comment><USER2> Fair enough, pylint and pep8 do not. Pychecker, pydoc, doctest, unittest... many other tools which are part of the typical Python development flow _do_.

</comment></comment></comment><comment><USER4> Firstly this is super cool and much simpler than other SSGs, but whilst I applaud the stand against "convention over configuration", I read the docs and find all the _magic_. Files starting with underscore are ignored? Where was the configuration for that? Oh yeah, that's just a convention. Then there's the magic for "_inherit.yaml", which -- by convention of this tool -- is treated specially. You can't have it both ways.

<comment><AUTHOR> You're absolutely right. I guess it's hard to be entirely magic-free. In this case, I'm pretty happy with the underscores thing since it's one pretty simple rule. Happy to hear alternatives, though. :)

</comment></comment><comment><USER5> Leave it to us developers to over-engineer easy stuff. Static Websites _are_ simple. That's the entire point of them.

Every single change is instantly reflected on the page. A non-technical person can easily debug HTML (given countless resources ranging from MDN to W3Schools ~_~) than any templating language or what have you.

That said, for us devs this is yet another tool to play around with and maybe extract something useful for our own setups. Thank you.

</comment><comment><USER6> Looks nice, but I don't think it will replace my current setup. I simply use makefiles (or rather mkfiles [0], because of the neater syntax) to administrate the generation of the site from markdown. That also makes it easy to add a deploy-target which deploys the site.

[0] [http://doc.cat-v.org/plan_9/4th_edition/papers/mk](http://doc.cat-v.org/plan_9/4th_edition/papers/mk)

<comment><USER2> Mind documenting and displaying your toolchain? Would be interesting to see how you do it with Makefiles.

<comment><USER6> I'll do a more thorough write-up some time, but I do the obvious thing — use meta rules to generate the files using the appropriate tools.

I use Pandoc to convert to html and glue together the files. An index file lists which files go to the server. Some files are html only, others are also available as PDF.

Some of the things I do are less obvious. There is a rule to make the target «deadlinks» which iterates through the resulting html files and attempts to fetch the (first bytes) of each linke. The links which fail go into deadlinks.

There is also a target which lists spelling mistakes.

The deployment target is configured to fail if there are dead links or spelling-mistakes (there are also a few other sanity checks).

</comment></comment></comment><comment><USER7> It seems a case of "I didn't understand this engine's documentation, so I'm making my own engine to do this exact same thing".

<comment><USER8> Nope. The whole point of a "static site generator" should be for it to be _SIMPLE_. If it takes me more than 15 min to understand almost everything about it, I consider the said generator _a failure_. This is why I never bothered with Jekill or Pelican after trying to get them to do _what I wanted_ and couldn't in less than 30min.

...and I fall back to my script which does "php + php -S to temporarily serve + wget to generate the actual static site".

 __sssgen seems like a breath of fresh air to me and I will actually use it. Thanks OP!!! __

Also, thanks again for the sound decision of using Mako so I can actually have python code in my templates instead of wrestling with another "crippled" template language. And for jumping off the maddening "convention over configuration" wagon... I _love_ configuration and explicitness for anything non-obvious

<comment><USER9> > The whole point of a "static site generator" should be for it to be SIMPLE.

This is incorrect. The point of a static site generator is to generate static site content. _You_ may use it in a simple manner, but there are many projects and companies that maintain huge, complex websites using static site generators like Middleman and Jekyll (along with various proprietary static site generators), in combination with content managers like Contentful. I'm talking hundreds if not thousands of pages, ecommerce, massive blog, multiple content types, etc. etc. I've worked on such projects, as have many others around here.

I'm glad that enlightened companies are increasingly open to using static generators for such websites, since many (possibly most) of them don't need to run PHP or Rails if the site doesn't have dynamic content on the frontend.

For such organizations, the point of static site generators is to produce vastly more secure and faster sites that are much easier to maintain and less expensive to host. Not to mention the ease of caching.

The OPs project, nice as it is for a small, simple project, would be impossibly simple for such projects. Anyone who tried to use it for a project of the scale I've described would just end up re-writing an inferior version of Middleman.

ssgen sounds like a cool project, and it also sounds like you're using the right tool for the job in your projects, but don't make the mistake of discounting a tool as useless or too complex just because it's not the right one for your purposes.

<comment><AUTHOR> Not taking a side here. :)

What do you think sssgen is missing that I could rip off of Middleman?

Or in general, what do you think is missing from sssgen to make it feasible to use for 100K+ -page sites? I'd really like to hear your thoughts.

<comment><USER9> I think your approach is great. There's clearly a demand for simple static site generators that favor configuration over convention, so unless you _really_ want to pursue the Middleman audience, I'd keep on keeping on. But I'll offer a few thoughts for your purposes:

1\. I think a small, well-tested core like you're doing is great. I also think for your purposes, Mako templates are the right way to go, since they allow super flexible logic in the templates.

2\. If you want some inspiration from a kinda sorta similar approach, take a look at Metalsmith and its plugin ecosystem if you haven't already.

3\. A great asset to any static site generator would be a collection of "starter themes/sites" that already have things like a static asset pipeline set up (using Gulp, Grunt, or whatever floats your boat).

If you _really_ want to accommodate the bigger static sites, a few critical features:

1\. Super fast compile times. This is why Hugo is rapidly gaining ground despite not having a plugin API. When you have thousands of static files, it's not fun waiting 10 or 20 minutes for each compile (Middleman and Jekyll have gotten faster, but they're still slow relative to Hugo and the Node-based static site generators).

2\. With huge sites that are built by large teams, flexible templates like Mako almost become a liability, since careless team members can overload logic into templates. For these teams, you may wish to offer an alternate template choice, like Jinja.

3\. Some kind of feature similar to Middleman's dynamic proxy pages, which would allow users to dynamically generate hundreds of pages basic of Contentful's API or their own JSON files. See: [https://middlemanapp.com/advanced/dynamic_pages/](https://middlemanapp.com/advanced/dynamic_pages/)

Finally, features Middleman's url helpers have proved to be a significant help on large static sites I've worked on, since the URLs are no longer hard-coded. There are a lot of small but significant features like this: localization extensions, pagination, etc. But if you have a good plugin API, all of these things can be created and offered as plugins.

But again, I think your current approach is just fine. If I were you, I'd perfect a simple core generator and then try to build up the ecosystem.

</comment></comment></comment></comment></comment><comment><USER8> Thanks! Finally a static site generator that you can actually fully understand in 5min!

And the whole Mako + yaml + tree variables is so sweat and simple.

</comment><comment><USER10> I like this project, mostly because it resembles similarities to my static site generator (not open source), with very similar concepts (except I'm using mustache), and slightly different internals, but about the same length, in Python.

Coincidentally, I've just started to re-do it as a python web app, that runs some limited dynamic processing (e.g. forms, logging, kudo-tracking, everything to keep it 100% self-hosted). I'm really interested if there are projects in this space that I'm not aware of, that I could just use...

<comment><USER11> Nikola has a plugin called webapp that's similar. it's kinda broken though and takes some fiddling, and is a bit buggy. I would be very interested in your project. I love static site generators but I hate having to remember all the commands. a web interface would be really nice.

</comment></comment><comment><USER12> I built my own simple static site generator in ~150 or so lines of Ruby. It's a fun thing to do.

That kind of inspired me to start a simple static website hosting service which is still in beta: [http://www.statichosting.co/](http://www.statichosting.co/)

I've tried all the other hosting options, and they are fine. I just never fell in love with any of them, so I made my own with a couple friends.

If you are interested in trying the beta, please sign up. We'll be opening it to more sites soon.

<comment><USER13> This is interesting, but even for a beta you should explicitly tell prospective customers about the limitations of your service:

\- how big are the zip files they can send?

\- any limitation on the outgoing bandwidth?

\- will the static websites be public or will it be possible to configure some sort of basic authentication?

\- etc. (I'm sure you get the idea)

<comment><USER12> Excellent points. I don't have all the answers to those questions. This is the best I got for now:

\- I think the zip file size is nginx configurable and it's set to like 4 or 8 MB right now. \- I don't have any bandwidth limitation at the moment, but if someone has a very high bandwidth site or wants to use the service to host/share files I'd have to cap those users somehow or maybe just charge more. \- Right now they are public, but I'd be willing to explore basic auth if that's a requested feature.

In general. We are making a service that we want to exist and are trying to do the best for whoever decides to show up.

Thank you for those questions, I am too close to the product to even realize what people might ask, but those all seem obvious in retrospect.

</comment></comment></comment><comment><USER14> > I've been asked to implement fairly complicated websites using static site > generators

What does "fairly complicated" mean here? An example would be nice.

<comment><AUTHOR> Sorry I didn't make that more clear. As an example, [https://docs.fastly.com/](https://docs.fastly.com/) was my handiwork. More recently, [https://teforia.com/](https://teforia.com/) .

</comment></comment><comment><USER15> It is simple (for developers?), but not easily extendable. Simplicity is not an excuse for no plugin support nor not writing good documentations. And more powerful tools need better documentations. If you think other SSGs are not well documented, just call for help and improvement. Why do you reinvent the wheel without _good_ documentations in mind?

</comment><comment><USER16> Any way to run the site not compiled through python built-in server ? So I don't have to generate over and over to check out tag or template mistakes.

</comment><comment><USER17> All of this has happened before and all of this will happen again.

</comment><comment><USER18> So I have to create my own templates for atom feeds?

</comment><comment><USER19> and so the cycle begins again

</comment><comment><USER20> It's not documented and I don't want to read 200 lines of code.

<comment><AUTHOR> Yeah, it's pretty funny that on the one hand, I'm saying "conventions make it necessary to eventually have to read the code", and on the other hand, I'm saying "read these 200 lines of code to understand how to use this static site generator".

That being said, for either Jekyll or Hugo, I've had to read >>200LOC in order to really understand a problem. :)</comment></comment>


London's First Open Brewery - <AUTHOR> http://www.ubrew.cc/pressrelease

<comment><AUTHOR> The UBREW concept is simple but effective. The idea is to develop an open brewery where beer lovers alike can learn how to make their own beer. Once a member, aspiring brewers are given training, space, professional equipment, and ingredients to learn how to brew craft beer.</comment>


Cities heatmaps based on tourists vs. locals pictures - <AUTHOR> https://www.flickr.com/photos/walkingsf/sets/72157624209158632/

<comment><USER2> Erik Fisher(@enf) has a good track record of finding amazing things.</comment>


The book every new American citizen – and every old one, too – should read - <AUTHOR> https://www.washingtonpost.com/news/book-party/wp/2015/12/17/the-book-every-new-american-citizen-and-every-old-one-too-should-read/ <comment><USER2> Here is a free audio book (from Librivox) in case anyone is interested: [https://archive.org/details/democracy_america_vol1_0709](https://archive.org/details/democracy_america_vol1_0709)

And here is a video I made about Caucusing for Bernie Sanders (equally important for the future of Democracy in the US): [https://www.youtube.com/watch?v=s0vO1j9ZhwQ](https://www.youtube.com/watch?v=s0vO1j9ZhwQ)</comment>