The Evolution of Trust - <AUTHOR> http://ncase.me/trust/ <comment><USER2> Wow that is extremely insightful.

Now I understand why globalisation is dangerous; if people start to favor short term relationships (both personal and business) with many different partners (smaller number of rounds <interactions> per match <lifetime>), then society will evolve to become full of cheaters.

<comment><USER3> Interestingly enough, seemingly to compensate for this phenomenon, almost every new platform that requires people to interact with each other has a built in trust and reward system with ratings (e.g. Uber).

And for physical businesses, we have things like Yelp. We are effectively trying to anticipate our opponents behavior through a trusted intermediary.

There was a Black Mirror episode on giving every person on the globe their own rating. It seems that may be the actual destination that we're headed for.

<comment><USER4> Trust, however, is not a miracle drug.

For one thing, people can build trust with the intend to abuse it after a longer period of time. An abuser, for example, could plan to offer 20 "good rides" on Uber until s/he strikes. For another thing, many people have learnt to skillfully simulate trust which may pose a problem for you. For instance, you may believe to be in a loving relationship but all of a sudden you find out that the other person isn't interested in you at all and just didn't want to be alone etc.

<comment><USER5> The book "Phishing for Phools" has a good chapter on "reputation mining", which is essentially that, with the added wrinkle that you can buy a company that has a good reputation and cut corners until the customers wise up. I'm looking at you, InBev.

<comment><USER6> I believe the term of art is "Extracting the value from the brand".

</comment></comment></comment></comment></comment><comment><USER7> This is by far one of the coolest things I've seen on this site.

Informative, engaging and educational. Would be very keen to learn more subjects through this method.

<comment><USER8> Nicky Case has a lot more where that came from :D

[http://ncase.me](http://ncase.me)

<comment><USER9> Wow. I played "we become what we behold" just last week. I had no idea it was the same author. I'm gonna check out the rest now too!

</comment></comment></comment><comment><USER10> Of course, this all breaks down when you have a cartel :) I recall in one of the major prisoners dilemma tournaments, the winning entry was actually a coalition of users who would use the first few moves to establish a code for membership in the cartel, and then one entry was the master while the others were slaves. The slaves always cooperated and the master always defected, to their victory.

<comment><USER11> Indeed, the major weakness of the linked piece is that it doesn't account for any kind of social structure or any class behaviour. Which, in my book, makes it fairly worthless as an analysis of actual social behaviour.

<comment><USER12> Not worthless. This has some clear assumptions on what holds. He looks at individuals playing a certain game, with no outside information. What the parent describes is two games: 1. Prisoners dilemma between each "slave" and the "master" 2\. The actual tournament

</comment></comment></comment><comment><USER13> This is great and really fun.

10 rounds with changed payoffs has some really interesting results. Punishing cheating from both sides, regardless of how high, still leads to the cheaters taking over. But if you increase the payoffs for both sides cooperating, even if it's just by +1, you already eliminate all the cheaters very quickly and even the copycats are gone pretty quick, leaving a population of only cooperate.

Sure that's for the 10 round game without any of the other players, it's still very interesting how even in that setting it seems more beneficial to reward good behavior vs punishing bad behavior.

</comment><comment><USER14> The execution of this visualization was rather disappointing.

I didn't like the overly cute text (the description of the Simpleton algorithm was almost incomprehensible), the low-contrast captions and colorblind- unfriendly color scheme, and the limited navigation (there was no way to go to the previous slide within a chapter, for example).

But more importantly: If you are going to design an entire interactive exercise like this, graphs are a much better way to explore the effects of varying different parameters. Trying to experiment (as instructed) with different parameters by watching animations in the various chapters and the "sandbox mode" included in this simulation was not only tedious, but prevented effective comparisons. If you just run each iterated tournament (from chapter 4 and onwards) by pressing the "Start" button, there is too much going on simultaneously at a high speed to follow along – I would recommend a sorted table or bar chart rather than many multi-digit numbers arranged in a circle – while stepping through is too slow to keep everything in your head.

I noticed that some of the other "explorable explanations" by the same creator include graphs; I think omitting them from this visualization was a mistake. [http://explorableexplanations.com/](http://explorableexplanations.com/)

</comment><comment><USER15> Very cool simulation.

Would love to see it with inter-agent communication between rounds, where those you cooperate with tell you who cheated them.

</comment><comment><USER16> This caught my attention:

"So, it seems the math of game theory is telling us something: that Copycat's philosophy, "Do unto others as you would have them do unto you", may be not just a moral truth, but also a mathematical truth."

Simpleton and copykitten for the win!

<comment><USER17> I also found it interesting how the success of cheaters requires a limited number of interactions (so their opponents don't catch on that they are cheaters). Perhaps that's why certain occupations such as used car salesmen have a reputation for being sleazy -- most people are not buying cars very often and so don't get the chance to get to know an individual seller over the course of many transactions. So while you might know that the corner-store merchant is screwing you and end up avoiding him, the used car salesman has a steady stream of suckers who don't know him.

<comment><USER18> Used car sales has its direct antecedents in horse trading and dealing.

There was a famous-for-its-time story called _David Harum_ , (1899), made into a film in 1915, and a radio serial in the 1950s. Its principle legacy today is the use of the term "horse trading" to mean "underhanded dealing".

[https://en.m.wikipedia.org/wiki/David_Harum](https://en.m.wikipedia.org/wiki/David_Harum)

I'd run across it via H.L. Mencken's essay, "Bayard v. Lionheart", itself best recalled for its concluding sentence:

 _As democracy is perfected, the office represents, more and more closely, the inner soul of the people. We move toward a lofty ideal. On some great and glorious day the plain folks of the land will reach their heart 's desire at last, and the White House will be adorned by a downright moron._

... but containing oh so much more, in the foil of a critique of the 1926 U.S. Presidential Election. Really, read it:

[http://amomai.blogspot.com/2008/10/hl-mencken-bayard-vs- lion...](http://amomai.blogspot.com/2008/10/hl-mencken-bayard-vs- lionheart.html)

Both used cars and horses have several characteristics in common:

* They are anti-commodities. That is, individual items for sale are highly disuniform, complex, and not readily assessed.

* They are expensive. For the ordinary consumer, they are not purchases likely to be made frequently. (A horse and a car each have fairly equivalent useful working lives: about 3-10 years, depending on use and care afforded.)

* Sellers of quality instances are much inclined to stay away from the general market. It's a case where selling to someone who is specifically aware of the qualities of what you're selling is a far better customer.

This has other cognates. Software and consulting services come to mind.

The treatment in the economic literature is somewhat disappointing. There's Ackerloff's "The Market for Lemons", of course (won him a Nobel prize), but it's a generally underserved area of theory.

</comment></comment></comment><comment><USER19> > Despite strict orders not to chillax with the enemy, British and German > soldiers left their trenches, crossed No Man's Land, and gathered to bury > their dead, exchange gifts, and play games.

Is this actually true? I'd love to read more about it.

<comment><USER20> Absolutely true: [https://en.wikipedia.org/wiki/Christmas_truce](https://en.wikipedia.org/wiki/Christmas_truce)

<comment><USER19> Thanks - that was a heart-warming read.

</comment></comment></comment><comment><USER21> I've read it multiple times, I still do not understand the simpleton strategy...

<comment><USER22> Essentially, the simpleton is treating your previous move as either a reward or punishment.

If you cooperated, then the simpleton thinks the thing it did last time must have been good and does it again. If you cheated, the simpleton thinks the thing it did last time must have been bad and does the opposite.

It follows these rules based on its last move even if that last move was flipped from what it should have been due to the chance of mistake.

</comment></comment><comment><USER23> The creator of this project has a really amazing talk about his work, including interactive explainers, narrative games and chaos theory. Really interesting: [https://www.youtube.com/watch?v=Zl9m0AQInBk](https://www.youtube.com/watch?v=Zl9m0AQInBk)

</comment><comment><USER24> This is super impressive. I have been asking myself these questions for _years_ , it is awesome to see such a nice visualization of the answers. The "30 minutes to play" almost made me click away but you got me hooked at the next screen.

</comment><comment><USER17> One extension I'd like to see to this simulation is the exploration of group/tribe dynamics, as a way of exploring questions regarding in-group loyalty (anthropomorphized in as "ethnocentrism" or something analogous) vs egalitarianism. It seems like much of the success of these agents can depend on whether Agent A can determine what sort of actor Agent B is -- communication that can be conveyed beyond just the framework of actual transactions between A and B.

Are there any existing generic frameworks out there for testing out a wider range of agent/group-based game theory scenarios?

<comment><USER25> Another important missing factor here is the modeling of empathy and altruism. It can often simply _feel good_ to help someone else, even if you get nothing tangible in return. This is a biological fact, as far as I'm aware.

Incredibly crude approximation: some situations that appear as [empathetic helper +0, receiver +3] on the surface, might in reality be [empathetic helper +1, receiver +3].

There is a ton of complexity in how this works in the real world. The magnitude or presence of the non-tangible reward may depend heavily on a certain aspects of one's value system. It may be heavily contextual -- only certain situations foster the non-tangible reward. Sometimes there is a potential tangible reward that may come much later, after some weeks, months, or years. And so on... I think this is barely scratching the surface.

Lastly, none of the people or dynamics in the system are static.

The linked interactive tool was very cool. However, I'm increasingly skeptical of these types of analyses as they generally are forced to ignore most of the complexity of the real world.

EDIT: I completely forgot about technology... things like the internet which completely change the nature of an interaction between people, or apps that mediate or flavor in-person interactions in ways that society has never seen before.

<comment><USER26> Have you thought that perhaps things like emotions, both empathy and revenge have evolved to emulate Copycat / Copykitten play styles?

That is, I don't think they're missing from the simulation. Rather, the simulation shows why those things exist.

</comment></comment></comment><comment><USER27> "Nonviolent Communication" by Marshall Rosenberg teaches a structured framework enshrining the takeaways from this lovely bit of art.

</comment><comment><USER28> From Netflix's "Sense8" :

What is the advantage of the Homo Sapiens over the Homo Sensorium (the telepathic humans)?

Homo Sapiens is capable of lying.

</comment><comment><USER29> We like to think we are complex, free-will wielding creatures capable of doing complex and unpredictable things.

It's striking to see how simple we really are and honestly - how easy it is to explain how we behave. I get it - this is a simulation and it leaves out alot of colors to human behavior. But it is the essence of our value-oriented decision making and really quite sad that a better world is actually? quite easy to make.

I think software has the potential to create that world. I propose the creation of a formal group dedicated to exactly such a purpose.

</comment><comment><USER30> As a co-incidence, I just read this relevant article today - [http://www.economist.com/news/economics- brief/21705308-fifth...](http://www.economist.com/news/economics- brief/21705308-fifth-our-series-seminal-economic-ideas-looks-nash-equilibrium- prison)

Edit : to say - This site was a beautiful insightful experience.

</comment><comment><USER31> More reading on this: There is a paper by David Axelrod and W.D. Hamilton called "The Evolution of Cooperation". There is also a book by Axelrod of the same name.

[https://en.wikipedia.org/wiki/The_Evolution_of_Cooperation](https://en.wikipedia.org/wiki/The_Evolution_of_Cooperation)

</comment><comment><USER32> This blew up on reddit: [https://www.reddit.com/r/philosophy/comments/6q1i1e/game_the...](https://www.reddit.com/r/philosophy/comments/6q1i1e/game_theory_and_the_golden_rule/)

</comment><comment><USER33> this is incredibly fascinating and insightful.

side question: who composed the music? the guitar track is excellent. :)

<comment><USER34> there's a list of music/sound credits on the github page: [https://github.com/ncase/trust/#play-it-here- httpncasemetrus...](https://github.com/ncase/trust/#play-it-here- httpncasemetrust)

</comment></comment><comment><USER35> 27 points, how'd you do HN?

</comment><comment><USER36> The little animated chums strike me be as a bit _too_ cute.

One has to take care to not anthropomorphize models. Even models ostensibly created to model aspects of human behavior, such game theoretic models.

Anthropomorphism might induce people either to think there's more under the hood than there actually is, and thus lead them to a hold a wrong mental representation of the model, or to naively assume that it's conclusions are robust to specification changes by making the conclusions a bit _too_ familiar, even though even somewhat simple model extensions might significantly change their behavior (e.g. going from single to repeated games, which makes cooperatives strategies viable).

Comparing tit-for-tat strategies with the Golden rule, while mixing in references toe the Christmas truce somehow seems to go in the opposite direction of being cautious regarding model interpretation.

<comment><USER37> > even though even somewhat simple model extensions might significantly change > their behavior (e.g. going from single to repeated games, which makes > cooperatives strategies viable).

He explicitly examines this in the application, did you not run through it? He actively encourages you to modify the rules and see how it changes the results.

<comment><USER36> I ran through it. I didn't suggest he didn't cover it. I just used it as a trivial example, because other readers supposedly also ran through it. Some other commenter suggested communication between agents as an extension, I could've used that if I knew what results from it.</comment></comment></comment>


MIT Free Courses - <AUTHOR> http://ocw.mit.edu/courses/#electrical-engineering-and-computer-science

<comment><USER2> Incredible. I think I'll start with this one [http://ocw.mit.edu/courses/electrical-engineering-and- comput...](http://ocw.mit.edu/courses/electrical-engineering-and-computer- science/6-080-great-ideas-in-theoretical-computer-science-spring-2008/lecture- notes/) The notes were typed up by students of the class, as part of some obligation/requirement, apparently. Not sure if that applies to the other courses. So much stuff on there to learn, it'd make you weep.

</comment><comment><USER3> Love the MIT free courses, really learnt a lot from all these materials. I know for sure, I could never get admitted to MIT CS program. Now I just hope all the rest of the top 10 uni does the same too.. Education should be FREE for everyone.

</comment><comment><USER4> Are there any sites to let people collaborate while working through these courses?

<comment><USER5> I don't think there are sites that do this currently, but the idea is definitely in the air, so there are probably guys working on it. For example, it seemed something like 3 of the 5 guys from the Google IO VC panel from late last week talked about wanting to fund collaborative educational startups.

EDIT: This is the panel I was talking about ... <http://news.ycombinator.com/item?id=1405320>

</comment></comment><comment><USER6> I really enjoyed the EECS courses available there.</comment>


Google Email Spying Ruled Unlawful - <AUTHOR> http://cryptome.org/2013/09/google-gmail-spy.pdf <comment><USER2> That's not what the document says though! Scroll to the end.

It says that one of Google's motions to dismiss has suceeded, striking out part of the case as inapplicable, and that the case is being allowed to proceed. This is nowhere near even a first ruling. And it relates to "spying" in the sense of processing email to show adverts, not the NSA.

</comment><comment><USER3> IANAL but the Court seems to have granted Google's motion to dismiss only for claims under CIPA Section 632 while denying it for all the other sections.

> the Court hereby GRANTS Google's Motion to Dismiss with leave to amend with > respect to Plaintiffs' CIPA section 632 claims

This is Section 632:

> Section 632 prohibits unauthorized electronic eavesdropping on confidential > conversations. To state a claim under section 632, a plaintiff must allege > an electronic recording of or eavesdropping on a confidential communication, > and that not all parties consented to the eavesdropping.

However, the Court has _not granted_ Google's motion to dismiss claims related to other sections:

> The Court DENIES Google's Motion to Dismiss with respect to all other > claims.

So one of the sections that still stands is Section 631:

> Section 631 prohibits wiretapping or "any other unauthorized connection" > with a "wire, line, cable, or instrument." The California Supreme Court has > held that section 631 protects against three distinct types of harms: > "intentional wiretapping, willfully attempting to learn the contents or > meaning of a communication in transit over a wire, and attempting to use or > communicate information obtained as a result of engaging in either of the > previous two activities."

Which _probably_ explains the title. I still don't agree with the title though.

<comment><USER4> Interesting, re: Section 632

> and that not all parties consented to the eavesdropping.

Are they indicating that when someone sends an e-mail to a Gmail account that they are implicitly consenting to the 'eavesdropping'?

As another poster pointed-out on HN a few weeks ago, sometimes it's not even clear if a domain is using Gmail. One would have to check the MX RRs first, before making a decision on sending.

Sounds like a useful postfix plug-in, actually...

<comment><USER5> > Are they indicating that when someone sends an e-mail to a Gmail account > that they are implicitly consenting to the 'eavesdropping'?

No, not at all.

There are, per the order, three requirements identified for a 632 violation: "(1) an electronic recording of or eavesdropping on (2) a 'confidential communication' (3) to which all parties did not consent."

The order finds the plaintiffs' case deficient on this point because: "Plaintiffs have not established that the communications at issue are confidential pursuant to section 632."

</comment></comment></comment><comment><USER6> Good. Google shouldn't be able to even "analyze" private communications and data, at the very least not without the user's _explicit_ consent (opt-in). Being in the ToS doesn't count since nobody reads those.

Maybe if they aren't allowed to do it anymore, they'll finally take requests to encrypt their services end-to-end seriously. Because right now they probably aren't even considering encrypting Hangouts if that means they can't analyze it for ads anymore, which is just sad and frustrating.

<comment><USER7> Analyzing emails is what email service providers do. If you can classify a message as spam, you can also classify it as being about shoes and target ads against it.

</comment></comment><comment><USER5> The headline is a lie, nothing Google has been found to have done has been ruled unlawful. The order here simply allows the case against Google to proceed in part (while throwing out part of it in its current form). This is a _very_ early ruling in the case, before _any_ evidence has been presented, and is purely a ruling as a matter of law about the standing of the parties to bring the case, and whether the claims made are even valid legal charges.

</comment><comment><USER7> Taken to an extreme, this idea of applying wiretapping law to federated services seems to be a danger to open systems. A closed messaging system can get all users, both senders and recipients, to agree to whatever is needed to make the system work (e.g. if they need to examine the content and not just the envelope)

However, when you have federation, two different sets of users on two different services may have agreed to differing terms. Applying terms like "interception" in these cases also seems strange.

It's bad enough that silos and vertical integration seem to be winning over the internet of old, but to have the legal system create a situation in which the open systems are much more risky to run seems to be a bad precedent.

(I also have to wonder who is really behind these cases. It's not very plausible that any of the plantiffs are really being harmed by automated ad serving, and then there's the whole Scroogled campaign...as Arsenio Hall used to say...things that make you go hmmmmm.)

</comment><comment><USER8> no such thing as a free email account

<comment><USER9> "Free Is The Most Expensive Price" or another popular expression on these days: "If You're Not Paying for It, You're the Product"

<comment><USER10> Popular but wrong

<comment><USER9> Wrong as not ethic or wrong like false?

<comment><USER10> Wrong like false, you are not the product, the product is ad space

<comment><USER11> I feel like you have something more you want to say that makes you point more clear. I think this audience would say ad space is not far removed from the eyeballs that make the ad space valuable.</comment></comment></comment></comment></comment></comment>


JavaScript Array.push is 945x faster than Array.concat - <AUTHOR> https://dev.to/uilicious/javascript-array-push-is-945x-faster-than-array-concat-1oki <comment><USER2> These functions do different things though, right?

push() mutates an array in-place, and concat() completely duplicates the existing array and adds another array to it.

        > x=[]; x.push(...[1,2]); console.log(x);
        [1, 2]
        > x=[]; x.concat([1]); console.log(x);
        []

So clearly push should be faster in general in usecases like these, since it does not need to copy.

Edit: grammar and copy paste failures from my console

<comment><USER3> Yes, they are completely different functions with completely different purposes. If you're still doing mutable programming for some god-forsaken reason using concat to mutate an array is a misuse of the function.

<comment><USER4> > If you're still doing mutable programming for some god-forsaken reason

What is mutable programming? Using mutable objects is something I do everyday. Am I doing things wrong?

<comment><USER3> The idea of "editing" stuff instead of creating new versions of them.

Whilst mutable programming is faster on write, it is much more difficult to figure out if something has changed, so any function that needs to only do work when stuff has changed (e.g. a React component), it is much much better to use immutable style programming because you only have to see if the memory address has changed as opposed to deeply compare current and previous objects.

<comment><USER5> I disagree that mutable code is inherently faster to write. Like any paradigm you can adopt, it probably feels that way igfyou start injecting immutability into an existing project, but sooner or later you settle into different design patterns which support it better, and it's not faster or slower to write. Probably faster to debug though.

<comment><USER6> Until hardware changes away from being an intrinsically imperative machines, immutable approaches will be slower simply because the hardware doesn't really support that.

We saw FP hardware leading to performance boosts kind of happen with GPUs (pixel and vertex shaders are just transformers), but then they got back to imperative again with GPGPU.

</comment></comment></comment></comment></comment></comment><comment><USER7> Am I seeing this right? Someone is creating 10000 arrays of increasing size and then wonders why it's slow? Why does this even need benchmarks? It should be obvious that copying 0.5 billion elements is far slower than only copying 10000.

<comment><USER8> And some people object to testing knowledge of the basics of algorithm complexity analysis with big-O in programming interviews because they "never need that theory".

IMO complexity analysis is possibly the single most important bit of theory to understand if you work on apps where there is any n which grows to a large number. More important than being able to reverse a linked list or whatever.

</comment></comment><comment><USER9> Yeesh... this is why it's good to have some experience with a language where you have to manage your own memory, even if you only write JavaScript at your day job. To anyone who's written C++ before this difference is incredibly obvious.

<comment><USER9> As a rule of thumb: anything done in a pure-functional/immutable style is probably going to take a performance hit. This tradeoff is often worth it for maintainability, especially if you're using true immutable data structures like Immutable.js or Clojure's structures which reduce copying to the absolute minimum. But always be aware of this tradeoff. If you're doing a simple operation over 15,000 of anything, just use a for loop for goodness' sake.

<comment><USER8> It really depends, I wouldn't be so forthright. In more complex apps, mutable structures may be shared with other bits of code, and you may end up copying for safety rather than because immutability forces you to. Non-local reasoning is less reliable as complexity and team size (concurrent development!) scales up, and the rules you adopt to ensure development can scale with larger teams can hit you on the other side.

This is a distinct issue from the maintainability argument. I'm saying that if you don't have functional code throughout, you might end up copying more often than you think. Immutability means you can hang on to things and share them with confidence.

For example, consider a method which returns an array in Java or C# or some other language without type-based constness. It's not safe to cache that and return the cached instance, because the array is mutable. So almost invariably such arrays are constructed afresh every time - copying induced by lack of immutable style.

<comment><USER9> > I'm saying that if you don't have functional code throughout, you might end > up copying more often than you think. Immutability means you can hang on to > things and share them with confidence.

But if you need to copy you're still following an "immutable" pattern: needing to share data with the guarantee that it won't be changed. Copying is just a very blunt way of accomplishing that. Immutable data structures will of course be a big improvement over simple copying whenever you need to do this sort of thing, but if you can avoid it altogether by modifying things in place instead of relying on their lack of mutation, you'll have even less overhead.

Clojure's transients, for example, were created for exactly this purpose: [https://clojure.org/reference/transients](https://clojure.org/reference/transients)

</comment></comment></comment></comment><comment><USER10> This is nonsense. You're micro-benchmarking some detail without reflecting on how to use these methods. I take it you have this code (sort of)

      const all = [[1,2,3,4,5], [1,2,3,4,5], ...] //15000 items
      let ret = [];
      for (item of all) {
        ret = ret.concat(item);
        // Or ret.push(...item);
      }

The point of using concat is of course that you don't have to do the loop. You're shuffling bytes for no use like crazy there. Not concat's fault. You should be doing:

      [].concat(...all);

This runs in ~1ms on my machine (latest Chrome). Custom versions, and loop + push both come it at around ~4ms.

</comment><comment><USER11> Wondering how .flat() compares. [https://developer.mozilla.org/en- US/docs/Web/JavaScript/Refe...](https://developer.mozilla.org/en- US/docs/Web/JavaScript/Reference/Global_Objects/Array/flat)

<comment><USER9> .flat() should be much better, because it "knows" it's flattening. In the author's case, each individual step was treated as stateless, so a new array had to be created each time. In a .flat() function, you could have a single persistent, stateful destination array that everything gets copied into. It would be equivalent to the performance of using push(). Actually, it would probably be faster, because flat() is presumably implemented in native code instead of JS.

</comment></comment><comment><USER12> Check out Funkia List [0], an persistent O(log n) random access list implementation. I use it pretty extensively when I'm writing typescript.

It actually can beat native arrays on concat and push, often by very wide margins, while also being immutable. These kinds of operations are actually much easier to optimize if you can assume immutability.

[0] [https://funkia.github.io/list/benchmarks/](https://funkia.github.io/list/benchmarks/)

<comment><USER6> It's obvious how to make concat fast if one is willing to give up O(1) random access operations. Push is already O(1).

<comment><USER12> Modern persistent tree-based data structures are usually O(log32n) random access, which is essentially O(1) for all practical purposes. With a mutable array of references, random access follows one pointer, while these structures follow usually less than 5. Sequential access is usually O(1).

These kind of persistent data structures are already the standard data structures in Scala and Clojure, and they are fast enough for the vast majority of non-numerical purposes. In typical access patterns, they are faster than mutable arrays.

<comment><USER6> As N approaches infinity O(log32 n) approaches O(log n). And really, we don't care much about N until it gets very large.

> In typical access patterns, they are faster than mutable arrays.

Only if typical is extremely random on large lists that consume many pages/cache lines.

<comment><USER12> Sequential access is O(1) just like mutable arrays. These structures really are fast and practical. The benefits of immutability are enormous. That's why they are so popular.

Edit: That's true about log32 and log2 become close in the limit, but that's irrelevant for practical data sizes. For example,

log2(10^6) ~ 20,

log32(10^6) ~ 4

That's a 5x difference.

<comment><USER13> log32n/log2n is always 5, highlighting why this is such a meaningless comparison. O(logn) (basically) refers to a constant multiple clogn, by which logic O(log32n) = O(5log2n) = O(log2n) = O(logn). It's all the same.

Case in point, if your algorithm takes log32n operations but each OP takes 5x longer its exactly the same as log2n. This is true for any value n, not just large values.

<comment><USER6> Ah! My explanation was wrong in that sense, thanks for the correction.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> [https://doesitmutate.xyz/](https://doesitmutate.xyz/)

</comment><comment><USER15> How about [...array1, ...array2] vs Array.concat. Or does 1 just use 2 under the hood?

<comment><USER16> The issue here is that the author wants to merge one array into another, so they wrote a = a.concat(b), but that is implemented as:

temp = a.concat(b) a = temp

In creating `temp`, JS copies all the elements of a and all the elements of b.

Whereas,

a.push(...b)

Only copies all the elements of b.

The code you propose copies all the elements of a and b, so it wouldn't be faster.

<comment><USER17> I believe you misunderstand. The question refers to whether or not using the spread syntax only would result in similar performance gains as seen when moving from .concat to .push with the spread.

const arraysToMerge = [ arr1, arr2, arr3, ... arrN ];

const spreadMergeFn = (reduced, arr) => [ ...reduced, ...arr ];

const spreadPushMergeFn = (reduced, arr) => reduced.push(...arr);

const merged = arraysToMerge.reduce(________, []);

Pure spreading will be slower. Using push lets you skip that first spread of what you've accumulated so far in favor of mutating a reference. I suspect that the above code using push will not run correctly though. Would need to get under the hood of .reduce to see, but it should break.

My current personal opinion is to use flat() if possible.

// these produce identical data structures

[ arr1, arr2, arr3 ].flat()

[ ...arr1, ...arr2, ...arr3 ]

arr1.slice().push(...arr2.push(...arr3))

<comment><USER16> If you think a misunderstanding may be afoot, let me be as explicit as possible. I do not believe that moving from:

a = a.concat(b)

To:

a = [...a, ...b]

Would give the same performance gains as moving to:

a.push(...b)

Because [...a, ...b] creates a new array, and copies the elements of both a and b into it.

Old-timers will say that we had this exact same conversation about Java and strings way back in the day. Using a StringBuffer was faster for this kind of thing up and until Java started detecting when your use of string catenation could be replaced by a StringBuffer.

</comment></comment></comment></comment><comment><USER18> Quadratic vs linear?

</comment><comment><USER19> (push using geometric pre-reserve) vs ((exact-per-element growth + concat) x N times) vs (exact reserve and then add the elements) . Being the third case analogous to the "builder" concatenation technique (resize the element to receive the concatenation for the final size, and copy the N strings into the resized space).

The concat privitive not doing geometric pre-reserve is not a bad thing, in my opinion, even if in this case is slower, because of memory saving for the most frequent case. Of course, the concat operation should be always efficient when destination container has enough space.

</comment><comment><USER20> Watch out for this even more obscure gotcha – if you're dealing with large enough arrays, V8 will throw an error:

       [].push(...new Array(120520))

results in `RangeError: Maximum call stack size exceeded` on node v10.15.3.

        [ ...new Array(120520)]

works fine though!

</comment><comment><USER21> Wouldn't a call to Array.prototype.splice act as a language defined (and hopefully optimized) means of adding elements in-place?

Slightly ugly because you'd need to specify the start index, a command to delete 0 elements, and a call/apply to inject the array, but it should do.

</comment><comment><USER22> Look into Immutable.js if you want concatenation and the performance of mutating

</comment><comment><USER23> Now I have to change all my Redux reducers!

<comment><USER9> Redux is designed to be used with this kind of pattern. Immutable.js, which is often paired with it, mostly solves this particular type of performance problem. You'll still have a performance ceiling when you use Redux, but it's very unlikely that you'll hit it.

Edit: I was using Immutable.js as a stand-in for immutable data structures in general; apparently there are other JS libraries you can use

<comment><USER24> Hi, I'm a Redux maintainer. I personally would recommend _against_ using Immutable.js, for a number of reasons [0].

Instead, I highly recommend the Immer library [1], which lets you do immutable updates with "mutating" logic in callbacks.

Even better, try out our new Redux Starter Kit package. It uses Immer internally to let you simplify immutable updates in your reducers [2].

[0] [https://www.reddit.com/r/javascript/comments/4rcqpx/dan_abra...](https://www.reddit.com/r/javascript/comments/4rcqpx/dan_abramov_redux_is_not_an_architecture_or/d51g4k4?context=3)

[1] [https://github.com/immerjs/immer](https://github.com/immerjs/immer)

[2] [https://redux-starter-kit.js.org/](https://redux-starter-kit.js.org/)

</comment></comment></comment><comment><USER25> I have often preferred concat over push for reasons of perceived elegance.

<comment><USER5> Performance optimization often comes at the cost of elegance. One is about logical and conceptual purity, the other is about doing whatever is required to achieve a result.

</comment></comment><comment><USER26> It's not 945x faster, it's O(N) faster.

<comment><USER27> 945x is the actual measured difference, O(N) is the theoretical difference. Many times quadratic time complexity is ok because you are working on very small N.

<comment><USER28> The point is 945x is just the measurement for this particular test case. If the author made the test larger it would appear even faster.

This is the underlying problem with people who don't understand what big-O complexity is saying.

</comment></comment></comment><comment><USER29> well of course, one modifies the array and the other creates a new copy

</comment><comment><USER30> Great research, great read.

<comment><USER31> I couldn't stand the repetitive way she was presenting the information. Sure it was like a monologue or conversation, but the info could've been given in a paragraph or two.

</comment></comment><comment><USER32> OK, so i'll make a concat function which uses push under the hood. Best of both worlds.

<comment><USER25> They do different things so it's more like the worst of all worlds.</comment></comment>


Egypt five years on: was it ever a 'social media revolution'? - <AUTHOR> http://www.theguardian.com/world/2016/jan/25/egypt-5-years-on-was-it-ever-a-social-media-revolution <comment><USER2> Legacy social networks won.

1st the Muslim Brotherhood, 2nd the Army.

Rinse, repeat, your tech is a fad.</comment>


We need a new generation of source control - <AUTHOR> https://www.rookout.com/cant-git-no-satisfaction-why-we-need-a-new-gen-source-control/ <comment><USER2> Are we mistaking a dependency control problem as a revision control problem?

In a previous life, before microservices, CI/CD etc. existed, we did just fine with 20-30 CVS repositories, each representing a separate component (a running process) in a very large distributed system.

The only difference was that we did not have to marshal a large number of 3rd party dependencies that were constantly undergoing version changes. We basically relied on C++, the standard template library and a tightly version controlled set of internal libraries with a single stable version shared across the entire org. The whole system would have been between 750,000 - 1,000,000 lines of code (libraries included).

I'm not saying that that's the right approach. But it's mind boggling for me that we can't solve this problem easily anymore.

<comment><USER3> You're not wrong. Part of it is the willingness of people to reach for a dependency that amounts to a few lines of code to avoid.

It would be nice if there was a tool that could help you identify just how much of each dependency you actually depend on so you could trim it.

<comment><USER4> These things all exist if you use something like bazel/pants/buck to manage your dependencies. When you can construct a DAG of the entire dependency structure you can see exactly how much you depend on any given thing (and get fun dot-graphs of it!). But that requires being precise with dependency declaration in a way that a lot of people don't want to be.

<comment><USER5> > But that requires being precise with dependency declaration in a way that a > lot of people don't want to be.

Some programming language stacks already fix that problem in a transparent way. Take Microsoft's .NET Core+Nuget stack. Developers can add packages to a project without specifying a version number (implicitly it's the latest release) and dependencies are checked when all dependencies are restored.

IIRC Rust's cargo also follow a similar approach, and so do npm and yarn. So, that's pretty much standard at this point.

</comment></comment></comment></comment><comment><USER6> The source control system is not the piece of the equation that matters to most people. The build system is the important part. That's what prevents you from rebuilding the repository when you only change one Kubernetes config file, or what causes 100 docker images to be built because you changed a file in libc.

I think the tooling around this is fairly limited right now. I feel that most people are hoping docker caches stuff intelligently, which it doesn't. People should probably be using Bazel, but language support is hit-or-miss and it's very complicated. (It's aggravated by the fact that every language now considers itself responsible for building its own code. go "just works", which is great, but it's hard to translate that local caching to something that can be spread among multiple build workers. Bazel attempts to make all that work, but it basically has to start from scratch, which is unfortunate. It also means that you can't just start using some crazy new language unless you want to now support it in the build system. We all hate Makefiles, but the whole "foo.c becomes foo.o" model was much more straightforward than what languages do today.)

</comment><comment><USER7> The argument of why monorepos suck seems to largely rely on "CI Sucks" in this article. But I beg to differ. Monorepos only work in combination with a build system that tracks dependencies carefully.

I contribute a lot to Nixpkgs, which is a monorepo with almost 50000 subcomponents [1], but because the build tool and CI track changes through hashes, changing a package only triggers rebuilds of other packages that depend on it and builds are super quick. It accomplishes this by heavily caching previous builds and sharing those between all builders.

No, monorepos are not going to work with a CI and build tool that always builds everything from scratch and does no caching. Instead, you should pick the right tool for the job, and go with a build system like Nix, Buck, Bazel or Please which were designed with monorepos in mind.

I think the second point the author makes, but only very briefly, is way more important to look at. Is git itself up to the job for such large repositories? One problem I've started running into in nixpkgs is that `git blame` takes considerable time to even execute, due to the enormous volume of commits in the repository. I would love to see a version control system that is optimised for storing lots of loosely connected components, and has better support for partial checkouts. I haven't found it yet, and I would love to hear what others are using for this.

I hear facebook has some modification of mercucurial. And Google probably created something themselves in-house. But is there anything open-source that supports these workflows at scale?

[1] [https://repology.org/repository/nix_unstable](https://repology.org/repository/nix_unstable)

<comment><USER8> I agree with the first part of this. If by CI you mean something like Circle or Google Cloud Build or Travis, then your CI is pretty much limited to whatever you can fit in a YAML file, and what the CI service will support in that.

YAML in and of itself is not the easiest thing to parse when you have multiple layers of nesting and a lot of lines.

I don't really want to see what a CircleCI config would look like for Nixpkgs.

Once you get to the point of scaling your CI you're looking at tailored infrastructure to make sure you're only building what needs to be built.

<comment><USER7> I'm honestly surprised that Google Cloud doesn't offer a "CloudBazel" product!

<comment><USER9> Something along those lines seems to be in the works:

[https://blog.bazel.build/2018/10/05/remote-build- execution.h...](https://blog.bazel.build/2018/10/05/remote-build- execution.html)

<comment><USER10> More information on how to get access to Remote Build Execution for Bazel on GCP: [https://docs.bazel.build/versions/master/remote- execution.ht...](https://docs.bazel.build/versions/master/remote- execution.html)

(Disclaimer: I'm a engineer on the Bazel team)

</comment></comment></comment></comment></comment><comment><USER11> I like the idea of creating a source control __protocol __that can be implemented with any number of tools rather than having wars over particular implementations of source control products.

(And would Git really have beaten Mercurial if GitHub had been HgHub instead? GitHub's success was more about process than the technology of Git, IMO.)

<comment><USER12> > And would Git really have beaten Mercurial if GitHub had been HgHub instead? > GitHub's success was more about process than the technology of Git, IMO.

Hg is a much better user experience than git, that's for sure. Git won because of Github, which may have beaten any HgHub simply because Git has an actual API while Mercurial's "API" is "use subprocessing". In other words, if Mercurial gave a damn about the _developer_ experience earlier on, it might well have won the war.

<comment><USER13> > which may have beaten any HgHub simply because Git has an actual API

Git doesn't though. A bunch of shell scripts calling shell scripts calling a few native binaries is pretty much "use subprocessing". libgit came much later, it wasn't part of the original git.

 _However_ what git _did_ provide was an open, stable, fairly simple and officially supported physical model with which you could easily interact directly, and protocols which either worked on that (file and "dumb http") or a relatively simple exchange protocol (the "pack protocol" [https://github.com/git/git/blob/9b011b2fe5379f76c53f20b964d7...](https://github.com/git/git/blob/9b011b2fe5379f76c53f20b964d7a3cecb9d8c79/Documentation/pack- protocol.txt)).

Hell, if anything hg's always provided more _API_ than git, the extension model wouldn't be possible without it e.g. stdout coloration could be an hg plugin while it had to be implemented in each git command.

<comment><USER12> It looks like you're right about the history. According to the git repo, libgit's first commit was in October of 2008 while Github was incorporated in early 2008 (according to Wikipedia).

Github's popularity was probably due to Git's popularity in the Ruby community which may have been due to the official support of the physical model and simple protocols.

That said, an "officially supported physical model" _is_ an API even if I originally had libgit in mind. Also, none of this invalidates the broader point, which is that Git won because of Github, not because of user experience.

</comment></comment></comment></comment><comment><USER14> The title felt a bit misleading- this is more a gripe of git and the approaches to mono and multi repos with git.

I'm not following the call for something new, though:

> A source control that treats CI, CD, and releases as first-class citizens, > rather than relying on the very useful add-ons provided by GitHub and its > community.

I'm not a die-hard every-tool-should-do-exactly-one-thing-the-way-the-unix- gods-intended type of person, but in this case, I really feel that source control should stick to being source control. Hooks and add-ons are great precisely because things like CI and CD came after, and who knows what the new rage will be 5 or 10 years from now.

Building everything for today's workflow into a single tool means that by the time it's ready, the "today's" workflow won't be cool anymore, and we'll have other newer tools and processes that this new source control can't support :/

<comment><USER15> You can already adapt mercurial to do different stuff very much like the author suggests. As a user (but not developer) of such adaptations, I think it works really well.

</comment></comment><comment><USER16> I don't know if I really follow the conclusion from this blog post although I sympathize with the complaints. Let's take one point: " In fact, not only will Git CI tools rebuild and redeploy your entire repo, they are often built explicitly for multi-repo projects." This seems patently wrong. On buildkite, which we use, you can explicitly set up build steps to trigger based on directory patterns.

In my experience on teams at growing companies, I've seen pain points around continuous integration, configuration management, integration testing, dev/prod parity, feature flagging and releasing, provisioning staging servers in terms of pure tech/infra issues. Beyond that, I've seen more pressing general organizational issues around tech debt, software design collaboration, architectural debt, code review processes -- these are all pressing and valid concerns. But I just find the conclusions of this blog post flat out wrong. To conflate an unsatisfactory CI choice and configuration (which is totally reasonable) with a failure of version control is a pretty serious one. It doesn't fully disprove the thesis, but it certainly doesn't lend it support.

If you've installed a wheel onto a poorly set up suspension and get handling issues, does it mean you should reinvent that wheel, or does it mean you should check if your suspension may need some tuning?

</comment><comment><USER17> This is such a pain point for me.

I would LOVE something between subtrees and submodules.

I have explored this many times, and if I had the ability to write something like this, I would.

I would love it if I could have a child repo that did not require an external remote and could be bundled and stored within a parent repo, unlike a submodule. But I would also like it if it could be more decoupled from the history unlike a subtree.

I can get most of what I want from submodules and subtrees, but not really enough.

It might be possible without even having to change git. Perhaps if there were a way to have branch namespaces of some kind, and I could have a subtree have completely separate history, but have it checked out within the same working tree. Many of my projects that are submodules only make sense within their parent repo, and it is really redundant to have an external repo for them. But I also don't like to have to do expensive surgery to deal with subtrees, and it would be nice to not have it be completely merged.

My dream is to be able to drop a repo inside another repo and have git just treat it as if it were part of the parent repo. And then to be able to bundle the child repo to the parent and push it.

I know that it is mostly possible to do this already, but it is not easy or intuitive.

<comment><USER18> _> My dream is to be able to drop a repo inside another repo and have git just treat it as if it were part of the parent repo. And then to be able to bundle the child repo to the parent and push it._

I'm not sure if I understand you right, but I think I made what you describe: [https://github.com/feluxe/gitsub](https://github.com/feluxe/gitsub)

It's a simple wrapper around git, that allows nested git repositories, with almost no overhead.

I use it for a private library (the parent-repo), which itself contains modules (the child-repos) that I open sourced on github. It works fine for my use case. I wrote it, because I found "submodule" and "subtree" too complicated. 'gitsub' is still in alpha.

<comment><USER17> Thank you for sharing that! That's really cool!

I'm just very attracted to the idea of bundling repos together. I frequently use git-annex and datalad, and try to keep binaries and helper scripts in different repositories.

</comment></comment></comment><comment><USER19> I know it's wrong of me. Genuinely, I know. But when your second paragraph states that Google and Netflix pioneered horizontally scalable processes...

It makes it so hard to read the remainder untainted by a certain amount of scepticism.

Fortunately, he's not actually saying anything much in the article so I don't think my irrational reaction to ignorance will mean I've missed something important. But still...

</comment><comment><USER20> I'm not sure the article supports its thesis with anything concrete. It's the author's opinion that submodules and some scripts are inadequate, but in my experience, a variety of reliable and flexible developer experiences are possible.

Many devs barely scratch the surface of what git can do anyway. Onboarding them on a few extra scripts seems better than an entirely new scm tool.

</comment><comment><USER21> I'm just now getting comfortable with Git, please don't do this to me.

<comment><USER22> Your opinions should not be swayed by one article. Have some resolve in what you do.

Git is great. It also has issues. Scaling has issues. Changing a tool won't solve scaling issues.

Anecdotally I think submodules work just fine, although the git submodule tool is not intuitive. Then again, I work in a very small team on small projects compared to these mammoths being discussed with monorepos and the like.

<comment><USER23> > Your opinions should not be swayed by one article.

Opinions should ideally not be swayed by the volume of text making arguments, but by the coherence and logic of those arguments.

<comment><USER22> Sure. But in order to see all sides of the argument and make an informed decision, one should read at least a few different sources, no?

</comment></comment></comment></comment><comment><USER24> > Take no prisoners: mono-repos suck too

We are transitioning to multi-repos because we have been burned so hard by our mono-repo. Builds used to take 3 hours on the monster and we managed to get them down to 30 minutes, but we are truly at the bottom of the barrel. God help us if there is a build failure, every subsequent build fails while we scramble to identify the problem (and we can only sample success or failure every 30 minutes). It's a house of cards and it's horrible.

> Shots fired: multi-repos suck

We've already had debugging woes with this combined with internal package feeds (you have to pull down the code, build it, remove the package and replace it with the local code), which has made us very bearish on code re- use. That rigmarole sucks way less than mono-repos.

> You can't have your cake and Git it too

Combine version control and package managers IMO. Go does one half of this. If you work under GOPATH with _all_ your code, you can easily jump across repos to make changes and have those changes immediately propagate to the initial repo. Your hard-disk becomes the mono-repo. What Go doesn't do is pull binaries down from package feeds. There needs to be some simple mechanism to switch between builds and code.

</comment><comment><USER15> Isn't the argument here that Git is not good at mono-repos rather than that mono-repos suck? This seems true to me, but there are already other options that suck less if you want the advantages of a mono-repo.

I would also suggest that mono-repos work better with statically typed languages with module boundaries and visibility control. The problem of anything being able to touch anything else is not so bad when you can hide implementation details behind small APIs.

I have definitely felt some pain with having Ruby projects in a single repo using git, but much less so with Java projects using Hg.

</comment><comment><USER25> I don't really understand the problem with multi-repos. Maybe because I'm a FE developer, I'm shielded from some of the pain of launching an external service locally, but I find the process of needing to update an external module to be rather similar to open source dev: clone, link, fix, PR, approve, merge, update package.json. It's that simple. The only part that can present itself as somewhat tricky is how your environment handles linked dependencies, but that can be resolved if you've configured your webpack or whatever correctly.

</comment><comment><USER26> The answer to your problem is Nix:

[https://nixos.org/nix/](https://nixos.org/nix/)

<comment><USER23> Fun fact: "Nix" is a colloquial variant of the German word for "nothing". Whenever we talk about NixOS at our hackerspace, hilariousness ensues.

</comment></comment><comment><USER27> Google uses a monorepo for most of its code and I find it a much better experience than what I've had in the past. But that good experience is predicated on a lot of Google only internal tools. If Google open sources enough of those such that people outside can have the same experience, maybe the debate will end decisively in favor of monorepo.

<comment><USER28> What kinds of tools?

<comment><USER29> You can read about all the tools in this article [https://cacm.acm.org/magazines/2016/7/204032-why-google- stor...](https://cacm.acm.org/magazines/2016/7/204032-why-google-stores- billions-of-lines-of-code-in-a-single-repository/fulltext)

</comment></comment></comment><comment><USER23> > Each side of this debate classifies the other as zealous extremists (as only > developers can!), but both of them miss the crux of the matter.

I take it the author has never had an interest in politics.

</comment><comment><USER30> I think that Git submodules are to blame. I think that the idea is great, but the implementation is cumbersome. If they were better to use it would solve a lot of these problems.

<comment><USER6> I think it's irrelevant. If your build depends on inputs from two repositories, it's the same level of complexity as having the inputs all come from one repository. You might have two .git directories, but if the code is coupled, it's just a monorepo in two directories.

Ultimately the problem is in scaling the number of build inputs, not the number of .git directories.

</comment></comment><comment><USER31> The field does suffer a bit from version control, dependency management, language compilers, and build and packaging tools all being single-purpose tools that are layered, where one can't introspect others beyond the public API, and manual effort or simplistic not-always-true assumptions have to be used to bridge information from one to the other.

It's tempting to imagine an integrated system where making changes to a piece of source code automatically commits every change, every commit will attempt to compile and build, every successful build auto-packages into a new artifact with a new build version. The language and the build system would ensure that all builds are reproducible. Because of this, all builds can be addressed by identity (content hash) too, not just a name and a build number within some namespace.

When any dependency of the current project has newer builds, one could choose to pull up an interactive diff experience to step through the code of newer versions. This would aid in selecting a different version on which to depend, if desired. If a different version of a dependency is picked up, a new build gets triggered too, and a successful build gets a new build version.

The strong linkage between source code revision and build version, the deterministic builds, and content-based artifact addressing work together to ease the traceability of changes and the reusability of artifacts, and sidesteps concerns about the hosting and namespacing of source code and build artifacts interfering with the project's "single source of truth", because any copy of an artifact, known by any name, irrelevant of its location, will share the same hash.

There will still be usability problems with such a system too. There would be no way to strip data out. A shelve, replay, and cherry-pick frontend would be necessary to allow the doctoring of input before it's committed permanently -- but in such a system, only permanently committed code can be built. The workflow to prepare a project for public consumption would be to author and test all the changes in a 'scratch' project that doesn't auto-disseminate its build artifacts elsewhere, and cherry-pick the changes into a public project. Public projects could only have public dependencies.

Configuration files, data files, and pieces making up a larger environment may need a different approach. Nonetheless, a lot of these problems take the same shape: some input should deterministically produce some output, and a running system may choose to alter its own state by interfacing with a stateful outside world (e.g. load or write files, communicate through a network). The sensible places of drawing a boundary between the inside world and outside world will differ for every use-case.

</comment><comment><USER32> What a bullshit pointless article. And oh god the cringey git wordplay... I'm starting to feel this is basically blogspam.

<comment><USER33> It IS blog spam. The article doesn't say anything besides "everything sucks." There is not a single constructive point being made besides some fantasy world where our tools are drop dead gorgeous and the build pipelines are well oiled and never have to be paid attention to.

</comment></comment><comment><USER33> As I've stated in another post on here, what's the point of these articles? It just says everything sucks, but doesn't really dive into why or how we could possibly fix any issues they may directly point out. Also it kind of sounds like the author really doesn't have any idea what GitLab is or does, so maybe he should check it out.

But allow me to retort these bald assertions presented in the article:

Monorepos are great.

Multirepos are great.

Git is the best source control system ever. And if you think it could do something better, well have I got news for you. It's completely open source and extendable with various script entry points and an easily accessibly API.

Thanks for reading my blog.

<comment><USER34> > Git is the best source control system ever.

To be clear, I'm not disagreeing. But it is simply not good enough. Any new generation of source control needs to be able to do things that are difficult with Git, and Git simply isn't extensible enough. Microsoft has a Git VFS, and there's Git LFS, but this just doesn't go far enough.

There are good technical reasons why you would use Perforce or even Subversion these days.

The people who made Git made it for working on large, but not huge, open- source code repositories with a traditional model. It doesn't work so well for vendoring, it doesn't work well for artists, it doesn't have locking, it doesn't have access controls (and there's only so much you can add). You can argue that these features don't make sense or we're using Git "wrong" or I can write a bunch of hooks but at some point I just want them to work and I'm tired of fighting with Git to make it happen.

Just personal background, these days I work with closed source and open source, monorepos and multirepos, Git, Subversion, and Perforce all on a regular basis (and sometimes use weird custom setups). Git is by far the most familiar of the three, and I've published some tools for Git repo surgery.

<comment><USER20> > There are good technical reasons why you would use Perforce or even > Subversion days.

Can you say more? What are some of those reasons? Or link to some data or examples?

<comment><USER35> With a monorepo, how do you avoid the situation where almost every time you want to commit, you have to pull-and-rebase first? Because somebody has always pushed a change, every minute or two.

<comment><USER23> In a repo that large, you don't want to have random people pushing to master anyway. Have people commit to branches, and then automation merges the approved branches into master. ("Automation" may be as simple as the "Merge" button in Github's UI, or more complex if necessary.)</comment></comment></comment></comment></comment>


How SGX Fails in Practice - <AUTHOR> https://sgaxe.com/ <comment><USER3> This is one of two lovely side-channel vulnerabilities in Intel processors released today. Apparently they also leak RDRAND and RDSEED data across cores and took 18 months to fix it: [https://www.vusec.net/projects/crosstalk/](https://www.vusec.net/projects/crosstalk/)

<comment><USER4> > The mitigation locks the entire memory bus before updating the staging > buffer and only unlocks it after clearing its content.

Entire memory bus as in entire access to any kind of shared memory (shared CPU caches, RAM, ...)? If yes, wow, that's a pretty desperate mitigation. Doesn't that starve large parts of the whole CPU whenever an affected, but mitigated instruction is executed, as all other cores can't interact with memory anymore for the duration of this execution? Doesn't that also open the ability to "DoS" CPUs by constantly executing such instructions?

<comment><USER5> Exactly what I wonder too. Can't the fix be used as an attack vector. At least infer disruptions to trigger timeout events, add execution time variance.

And wouldn't you be able to detect RdRand instruction execution for (for example) key generation. Pinpointing when processes perform sensitive operations.

And, would it be possible to use RdRand execution as a time based exfiltration channel? Run a timing loop in one process on one core. Use RdRand reading with different frequency in a second process to (using morse code or similar) to encode information. The first process use its timing loop to recover the information.

</comment></comment></comment><comment><USER6> At this point SGX is just so broken that it seems like its only purpose is to provide PhD students something to write a paper on :)

I'm hesitantly excited for AMD's SEV enclave to roll out. Anyone know if it's shaping up to be any better?

<comment><USER7> SEV is exciting because it has a much better cost-to-benefits ratio.

It provides useful defense in depth without requiring any changes to the application stack - you can run regular VMs with syscalls, plenty of memory and high-bandwidth IO.

SGX, on the other hand, is extremely limited and notoriously hard to target. It's even harder these days - you need specialized compilers and coding techniques to mitigate a number of attacks that can't be fixed by a microcode update.

I reckon it's almost impossible to do serious SGX work these days without being under NDA with Intel such that you can work on mitigations during embargoes for the never-ending stream of vulnerabilities.

<comment><USER8> _SEV is exciting because it has a much better cost-to-benefits ratio._

I think that's not actually true.

The problem is if you believe SGX "needs" these sorts of defences/mitigations then so does SEV, because SEV VMs are not magically immune to side channel attacks and in fact suffer far more than just micro-architectural side channels because they also leak all disk and memory access patterns, network access patterns and so on. These sorts of side channels aren't the responsibility of any CPU to fix but are remarkably powerful.

Sometimes it feels like SGX gets placed under a rather nasty double standard. Enclaves are "hard" because you "must" mitigate side channel attacks. SEV VMs are "easy" because nobody even tries at all. Indeed they cannot try - <USER2> software isn't built to eliminate app level side channels. That's why enclaves are special programs to begin with.

If you are happy to use existing non-hardened software though and just take the defence in depth argument, well, no problem - you can use SGX too. There are things like SCONE that let you run existing software inside enclaves. Unlike SEV, SGX is actually fixable when security bugs are found so it's meaningful to talk about using it. SEV has been theatre so far. It's not renewable, there's no equivalent of TCB recovery so nobody bothers trying to attack it because it's already been broken in unfixable ways before.

<comment><USER9> You are right about the double standard but using enclaves means restructuring your application. Even SCONE requires porting. SEV gives you the warm and fuzzy feeling that you are doing something to improve security without having to do a lot of work, assuming that your favorite OS version has been ported to run in a "secure" VM.

<comment><USER7> It's apples and oranges.

SEV is useful for defense in depth and absolutely not comparable a secure enclave like SGX or TrustZone.

However, it results in significant security gains with almost no extra effort.

Secure enclaves like SGX are designed with much stronger security guarantees and are therefore hard and expensive to use, and they're regularly broken due to design issues that may be impossible to fix, making it a bad investment.

<comment><USER8> The problem is, it's not really clear it results in significant security gains.

SEV (a hypothetical version that wasn't broken) would give you encrypted memory and some basic remote attestation protocols.

The point of this is to stop a cloud vendor from peeking into your VM.

But the problem is, none of the software inside your VM is designed to resist attacks by a malicious hypervisor. This isn't just side channel attacks but also so-called Iago attacks. This is where a formerly higher privileged piece of software is trying to break into a lower privileged bit of software by manipulating the API responses.

For SGX it was shown that running a whole standard Linux process inside an enclave and relaying system calls outside (i.e. a similar arrangement to what SEV does) could lead to the process being hacked the moment it did something trivial like call malloc(). The reason was, the software didn't expect the kernel to try and manipulate it via syscall return values because there is no point normally.

In SEV we have the same issue. Hypervisors were previously more privileged than kernels. Therefore kernels and apps aren't designed on the assumption of a malicious hypervisor. It's highly likely that not only side channel attacks but also Iago attacks apply there too.

Now you could say, well, one step at a time, maybe it doesn't matter, better than nothing etc. Sure. All true. At the very least it complicates the attack process and is more likely to generate an audit trail inside the cloud vendor.

But actually Intel had a somewhat comparable solution for a long time called TXT. It lets you start a trusted/measured hypervisor that can then supervise VMs on the box. This is in some ways stronger as you can actually check the hypervisor won't attack the VMs. But it's hardly used because cloud vendors use proprietary hypervisors, and actually the threat model people care about is "malicious cloud vendor trying to break into the VM", not "incremental security improvement of unknown utility".

I suspect Intel will implement encrypted memory for VMs at some point anyway because of the viewpoint you raise here being quite common - "I'll encrypt the RAM for free and then I'm done" although of course it needs tooling integration as well. It's not actually quite for free.

But I guess if this takes off then AMD will start to see a lot of research papers where people break into the encrypted memory space in various clever ways, and of course, you're also vulnerable to any exploits in the software stack on the VM itself. That was the main reason the security community ended up with the SGX type model. When your trusted computing base is an entire virtual machine it doesn't give you much because VMs have so much software in them, they get hacked all the time. The idea of enclaves is you design your app so the bulk of the software stack is untrusted, and only small parts get to directly touch sensitive data.

<comment><USER7> I fully agree with all of what you said. There's a big difference between "hey can you dump this VM for me please" and "hey please implement this paper to dump this customer's VM" and it provides cloud providers with plausible deniability when law enforcement comes knocking. It's certainly useless once your threat model includes grad students with lots of time at their hands.

The "trusted hypervisor" approach is actually what my company is working on, with SEV just as a convenient attestation mechanism and nice defense in depth.

Yes, with SEV the guest OS has to assume that the emulated hardware is untrusted and I'm quite sure that no OS was built with this threat model in mind. It's not as bad as proxying syscalls from SGX to the host kernel because there's a lot smaller attack surface, but I bet you could find a hundred ways to compromise a stock Linux kernel running in SEV.

I'm more optimistic about unikernels written in say, Rust, which would still be a much friendlier API than SGX.

<comment><USER8> Ah, my company is working with SGX. Perhaps that explains our difference of view ;)

The hypervisor ABI is quite large, perhaps not as large as a kernel's but it doesn't really make sense to expose the same API to secure code anyway. For instance an encrypted VM that then uploads its data somewhere else via raw TCP sockets doesn't make sense conceptually, even though the API allows it. You have to use SSL. Likewise an encrypted VM that downloads software updates from the repositories of a cloud provider, also doesn't make much sense, even though nothing in the tech stops it.

The nice thing about an enclave is you can understand completely the data flows in and out by examining the interface. That does mean compiling existing software for it will die with unimplemented functions. But those errors are probably telling you something meaningful - i.e. if the code attempts to call socket() or open() then you _can_ just relay them outside the enclave, but it makes more sense to think about what you're really trying to do.

It's a more purist approach with worse compatibility, I agree. It's really focused on finding the minimal TCB of your program and excluding all else, like how Chrome moves the rendering engine out of its TCB. I suspect many apps can be designed in such a way that almost all of the code is untrusted, but it's a bit of a frontier and takes quite a bit of skill.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> So this breaks SGX completely.

Signals new PIN thing relies (almost) entirely on SGX being secure to make their encrypted profile and contact backups secure.

This attack reduces the security of the Signal encrypted backups to just the PIN.

Edit: Indeed the authors point this out explicitly in the SGAxe paper.

<comment><USER6> > Signals new PIN thing relies (almost) entirely on SGX being secure to make > their encrypted profile and contact backups secure.

This just seems irresponsible. How could they excuse this? It seems like anyone who has even peripherally been working with TEEs recently is _well_ aware that SGX is broken beyond repair. It's not just a matter of patching bugs, this whole model seems bunk.

<comment><USER11> Seems unlikely they were mentioned in the paper in this depth without being made even casually aware that this paper would be published.

Which means they just went ahead with it anyway.

</comment></comment></comment><comment><USER12> Remember, SGX is not a security feature. It is a DRM feature meant to take control away from the owner of the machine and give it to content owners. Happy to see every successful attack against this module, and hoping that Intel decides to just ditch it entirely.

<comment><USER13> Oh boy, here we go again.

No, SGX is a technology allowing for the trusted execution of code in an untrusted environment. DRM is one potential application of that technology.

See also the comment section on literally _every other HN submission with SGX in the title_, for example here: [https://news.ycombinator.com/item?id=22495251](https://news.ycombinator.com/item?id=22495251)

<comment><USER12> Yes, indeed here we go again, because I fundamentally disagree that "allowing the trusted execution of code" is _any_ different than DRM. A rose by any other name.

I believe a user should have full control over the hardware they own. SGX and other enclaves are a direct affront to that, and I won't give up the fight. If you need a "trusted execution" of code, run it on a server you control.

<comment><USER13> Trusted computing isn't so much about control as it is about trust/attestation. You can run privileged malware on your computer without trusted computing. On the other hand, you can also sandbox trusted computing.

If you don't want to run somebody else's code on your computer – don't! (Chances are you're not a cloud provider and nobody is asking you to, anyway.) If you oppose DRM – don't consume content protected by it, whether the DRM is implemented in SGX, TrustZone or just in obfuscated software.

Don't get me wrong, I am not the biggest fan of DRM either (I personally see it as an evil, maybe a necessary one though). But shunning the entire field of trusted computing is a bit like opposing theoretical physics because it ultimately has brought us nuclear weapons.

<comment><USER12> I'm not entirely against the concept of trusted computing, but it's hard to believe that "trusted computing" as we know it today is so innocuous when it's included on consumer CPUs. Why not keep it as a premium feature for server- class CPUs instead?

<comment><USER14> I don't think SGX is available on any consumer CPUs.

<comment><USER15> My laptop has an Intel i7-8750H CPU.

[https://ark.intel.com/content/www/us/en/ark/products/134906/...](https://ark.intel.com/content/www/us/en/ark/products/134906/intel- core-i7-8750h-processor-9m-cache-up-to-4-10-ghz.html)

> Intel® Software Guard Extensions (Intel® SGX)

> Yes with Intel® ME

I have it disabled in the firmware settings.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> Fixed in January.

[https://blogs.intel.com/technology/2020/06/ipas-security- adv...](https://blogs.intel.com/technology/2020/06/ipas-security-advisories- for-june-2020/)

> Finally, in January 2020 we released INTEL-SA-00329 addressing two side- > channel issues. Specifically, CVE-2020-0549 addressing an issue called "L1D > Eviction Sampling" which has been referred to by researchers as "CacheOut". > The CacheOut researchers recently informed us of a new paper referred to as > SGAxe. It is important to note that SGAxe relies on CVE-2020-0549 which has > been mitigated in microcode (confirmed by the researchers in their updated > CacheOut paper) and distributed out to the ecosystem.

And from the CacheOut paper:

> Finally, Intel had indicated that microcode updates mitigating the root > cause behind CacheOut will be published on June 9th, 2020. We recommend > these be installed on all affected Intel platforms to properly mitigate > CacheOut.

The above blog is indeed dated june 9, however it indicates that the fix was made available, and verified, back in January.

I find fault, or at minimum a conflict, between the researchers publication as of june 9, and Intel's statement on the same date. Pretty horrible. I especially dislike that the paper has a section called "Current Status", while the paper is undated as is the norm for academic papers of this type. The "Current Status" isn't 100% clear but appears to be current as of November 2019, so yes this would be in advance of both the June 9 supposed date and the apparent January actual date. Perhaps the paper linked on the sgxaxe website isn't the "updated CacheOut paper" that Intel refers to.

</comment><comment><USER17> Boy, I'd love to see Moxie's comment on this after years of shilling SGX. I was always disappointed by that and never understood why someone as otherwise- bright as him went for it.

<comment><USER18> There have been several attacks like this already (and more likely more to come). The nice thing about the SGX design is many of these issues can be fixed immediately with a microcode update. Now if an attack is announced that can't be fixed with a microcode update that is another story :). In general, protecting against these kinds of attacks for any enclave is a hard problem, but it is an active area of research and there are already research proposals for more side channel resilient enclave designs (e.g. see the Keystone project from Berkeley). I expect some of these mitigations will be incorporated into future iterations of SGX, but it will take time.

<comment><USER17> You're right, and the idea of a secure enclave is not an inherently bad one. What is bad is treating it as an unassailable fortress, as in signal's address book feature. There's a difference between using it to make critical parts of existing computations more secure and using it to do stuff one wouldn't otherwise do.

<comment> dane-pgp > You're right, and the idea of a secure enclave is not an inherently bad one.

Maybe it is when you're trusting private keys and proprietary technology controlled by a single entity that may be subject to commercial and political pressures that make their motives not aligned to yours.

<comment><USER18> I hope you're not typing that on an Intel processor :)

<comment> dane-pgp

      $ grep vendor_id /proc/cpuinfo
      vendor_id       : AuthenticAMD

</comment></comment></comment></comment></comment></comment><comment><USER19> > Our bot provides Attestation as a Service (AaaS)

slow clap</comment>


Yahoo shuts down six products - <AUTHOR> http://ycorpblog.com/2013/04/19/355356/ <comment><USER2> What I find even more interesting that these shutdowns is that an entry from the Yahoo corp blog has made it to the front page of Hacker News. This may be a sign that people are starting to find Yahoo related news interesting again.

<comment><USER3> I probably titled it poorly, but my earlier post[1] about Yahoo's new mobile weather app (Yahoo doing _mobile_ right?!?) didn't get past a single point. :P

[1]: <https://news.ycombinator.com/item?id=5577427>

<comment><USER4> Have you checked forecast.io? It is even more interesting because it is a website which looks like a mobile app. Besides, the weather data it provides is awesomely simple.

<comment><USER3> Thanks for pointing that out!

Like the sibling commenter here, I find that I almost exclusively check the weather on my phone, but I'll keep this at the back of my mind for more thought-out planning. :)

</comment></comment></comment></comment><comment><USER5> Error 999 Copyright © 2006 Yahoo!

\---

Ironically, that's exactly how I think of Yahoo: stuck in 2006 and "are they dead yet?"

<comment><USER6> People used Yahoo in 2006?

<comment><USER7> Not sure if you're being sarcastic or not but a decade or so ago, before Google arrived on the scene, Yahoo! was actually very popular.

<comment><USER8> Google was on the scene well before 2006. I think that comment was meant to be interpreted as

"people still used Yahoo! as late as 2006?"

</comment></comment></comment></comment><comment><USER9> Verge article, since it seems dead: [http://www.theverge.com/2013/4/19/4243852/yahoo-shuts- down-d...](http://www.theverge.com/2013/4/19/4243852/yahoo-shuts-down-deals- kids-upcoming-and-more-in-attempt-to-focus)

Here's the list:

>Deals, Yahoo Upcoming (and its API), Yahoo Kids, Yahoo SMS Alerts, and the J2ME feature phone versions of Yahoo Mail and Yahoo Messenger, all on April 30th.

</comment><comment><USER10> Yahooligans was my introduction to the web. I'm sad to see it go. I hope it remains well-archived.

<comment><USER11> Yahooligans was my elementary school computer classes! :(

</comment></comment><comment><USER12> I don't understand why Yahoo! never properly promoted Upcoming.org. In fact, I don't understand why companies like Yahoo! go out of their way to acquire nice things only to abandon them when the next new, shiny thing comes along.

<comment><USER13> It's part of the dysfunction.

</comment></comment><comment><USER14> It's probably a good call to shut down these services. Yahoo has to change to survive.

What's not good is the amount of notice. 3 months should be the absolute minimum notice for shutting down a service. Otherwise you lose users' trust and that closes doors on future opportunities.

</comment><comment><USER15> Not pertinent to the discussion, but why does some of the links use <http://google.com/url?q=> prefix for redirecting URLs? Any advantage of doing so instead of directly linking to the posts?

<comment><USER16> Because whomever wrote this article was Googling the results (yes, the irony is palpable) and simply copied the URL, which is prefixed in Google's SERP.

</comment></comment><comment><USER17> Whatever happened to Upcoming as a concept? It took quite a while for Lanyrd to fill the gap for tech conference goers, but that's basically just "us".

I just wonder why this turned out not to be a viable service. I would think there would have been a market for a global event calendar, and Upcoming at one time seemed well on it's way to being that until it got "Flickr-ed" by Yahoo.

<comment><USER18> I wish it had gotten Flickr-ed. Flickr grew 2-3 orders of magnitude under Yahoo and is still the dominant site for photos you might want to look at again.

</comment></comment><comment><USER19> If Mayer's playbook is to make Yahoo like Google, and to do what Page does from quarter-to-quarter, I don't think that's a bad plan actually.

Yahoo exists strictly on momentum at this point, so it's not a matter of re- org to fix - it's a matter or org. Might as well clone Google for the seed and go from there.

<comment><USER20> Agreed. For one, it seems that Yahoo mail app on my Android syncs and works much faster than GMail. Maybe GMail has nailed the top spot for webmail, but it looks like Yahoo is ahead on Android. Don't know about iOS apps though, anyone?

</comment></comment><comment><USER21> Thankfully I don't use these products so the shutdown doesn't personally affect me. The only issue that jumps out is the post went up 4/19 Friday afternoon and the shutdown date is 4/30, including the API. Anyone relying on these services has 7 business days to respond.

<comment><USER22> "Like we announced last month" (link to post from 01.03)

<comment><USER21> If you click through, that post announces a different set of services which have now been shut down. "Last Month" was avatars, Yahoo on Blackberry, Clues, App Search, Sports IQ, boards, and Updates API.

That blog post was March 1st and the services shutdown April 1st with the Updates API shutting down April 16th.

<comment><USER22> oh, right, sorry.

well, pretty irresponsible then. you come back from a short vacation and your data is gone.

<comment><USER23> This is an important point. Not only should your online services have an easy way for you to get data out of them, but you probably should have a reasonably sane backup schedule, just like you would for hard drives/physical media/etc.

That being said this is a very short amount of time, and hopefully future shutdowns like this at yahoo and other places give more of a warning.

</comment></comment></comment></comment></comment><comment><USER11> Did anyone notice that a number of the links in the blog post are being routed through Google? (via google.com/url?q=theurl) What's up with that?

<comment><USER24> Maybe copy & paste of those links from a search results page, without paying attention to what URL actually gets pasted?

</comment></comment><comment><USER25> Brace yourselves for some posts in the next days...

* "How I've been living with my mail server at home for six months"

* "Yahoo pushed for the use of technology INSERT_TECHNOLOGY_NAME and now they're trying to kill it by closing INSERT_SERVICE_NAME!"

* "Yahoo INSERT_SERVICE_NAME is closing, use INSERT_APP_NAME-ly to replace it (and import your current configuration)."

* "Show HN: This weekend I built an app to replace INSERT_SERVICE_NAME. It will never close, I promise."

</comment><comment><USER26> I always imagined yahoo kids was profitable for them. maybe disney/angry- birds/whomever are just too dominant?

</comment><comment><USER27> Upcoming and Yahoo Deals seem like good product ideas for improving personalization for users' local news. Does Yahoo have replacements for these services?

</comment><comment><USER28> I'm surprised that Yahoo! directory (<http://dir.yahoo.com>) is not in the gang of six.

</comment><comment><USER29> Wow, did not realise that a) Yahoo kids existed and b) it was so embarrassingly terrible.

</comment><comment><USER30> Hmm, think it might be time to get around to backing up my Flickr photos.

</comment><comment><USER31> Might one say that they are putting more wood behind fewer arrows?

</comment><comment><USER32> Probably a good call. I had never heard of any of those six.

<comment><USER33> Upcoming.org was a big deal once upon a time.

</comment></comment><comment><USER34> Hey Google, this is what it really means to focus.

<comment><USER35> Yahoo could commit genocide and you idiots would compare it favorably to closing Google reader.

<comment><USER36> I regret that I have but one up-arrow to click on for this comment.

Google does this annually and they get criticized for it every time. Yahoo does it at the behest of their new ex-Googler CEO and now it's a great idea.</comment></comment></comment>