Designing very large JavaScript applications - <AUTHOR> https://medium.com/@cramforce/designing-very-large-javascript-applications-6e013a3291a3 <comment><USER2> I love Malte's take on what makes a senior engineer with regards to API design:

> The way I would talk about myself as a senior engineer is that I'd say "I > know how I would solve the problem" and because I know how I would solve it > I could also teach someone else to do it. And my theory is that the next > level is that I can say about myself "I know how others would solve the > problem". Let's make that a bit more concrete. You make that sentence: "I > can anticipate how the API choices that I'm making, or the abstractions that > I'm introducing into a project, how they impact how other people would solve > a problem."

<comment><USER3> I think "How confusing is this going to be to the one consuming this" (assuming you're not writing the front end as well) is the more important question to ask

<comment><USER4> And the one most rarely asking in my experience too

<comment><USER5> In my opinion, asking this question is the key benefit of test-driven development. You get to be the consumer, write down your expectations, and then work backwards from yours-as-consumer expectations.

<comment><USER6> I find the opposite tends to happen in Java. In the pursuit of testability, people make libraries much harder to use and understand.

<comment><USER7> Writing good tests is a skill that has to be developed, just like anything else.

</comment></comment></comment></comment></comment></comment><comment><USER8> I've always disliked the "BIG TEXT CHOPPED-OUT BUT YOU'LL READ IT BELOW AGAIN IN JUST A SECOND" paradigm in magazines and newspapers. Like, I'm reading the article anyway, why suddenly spit some of the words in my face super big? I'm guessing it's for skimmers, catch their eye and bring them into an interesting paragraph that's further down from the top.

But this article does it at the beginning... we read the sentence "Hello, I used to build very large JavaScript applications" three times. First as an image, then as a label for the image... and then as the _first sentence_ of the article. Why lol

<comment><USER9> > Why lol

Because, even closer to the beginning, in the real first sentence of the article (by which I mean, the first sentence after the title), it says, "This is a mildly edited transcript of my JSConf Australia talk."

This is not an example of the "big text chopped out" paradigm from magazines, it's an example of the, "each slide contains a brief of the most important part of what you'll be saying while that slide is being displayed" paradigm from conference lectures.

<comment> et-al The problem is those slides hinder the readability of the actual article.

For a reader, you get into the rhythm of the writer's voice, only to be interrupted by a giant image of text every other paragraph. It's a battle between just skimming the slides, or trying to read the content.

If the author had just swapped out the "loud" text images for subtle headers, the article would actually flow.

</comment></comment></comment><comment><USER10> More like designing very long Medium posts.

Is there any substance to this "we invented 'enhance', a reverse dependency pattern" thing or is it just old wine in new, javascripty bottles?

<comment><USER11> This is just basic dependency injection.

I agree it is mildly amusing to watch Javascript devs excitedly discover these principles but it does show that these principles are universal. Virtually everything in this talk applies to building very large Java and C++ applications. (And here I don't mean 10mb... rather global systems made up of dozens of services and hundreds of components.) All of this is about modularity -- of configuration, of state, of logic, and of concerns.

<comment><USER12> Is there a single place to learn about all of the common software engineering principles, as opposed to rediscovering universal principles over and over again?

<comment><USER13> Martin Fowler's site is a good start, especially this collection: [https://martinfowler.com/eaaCatalog/](https://martinfowler.com/eaaCatalog/) . And then read his books...

</comment></comment></comment></comment><comment><USER14> It is tragic that the world standardized on Javascript. They are just now discovering some of the module issues that Prof. Wirth solved in his great Modula-2 langauge in 1974. His system allowed separate compilation, and a way to detect when the interface API changed so that it could be cross-checked. I used Modula-2 to great effect on very large projects, and am greatly relieved that the Modula-2 module system has more or less made it into JavaScript.

<comment><USER15> Don't worry, they'll discover types eventually.

<comment><USER16> Aren't most large-scale javascript applications using typescript or flow these days? I definitely wouldn't try building a large scale javascript codebase without either of those anymore.

<comment><USER15> Frontend developers at our company vetoed TypeScript for being too difficult to comprehend for JS devs.

<comment><USER17> The downside of Typescript is the constant friction from bolting a type system on top of a dynamic ecosystem.

It's like Babel and Webpack where you incur time spent on debugging your own tooling and getting these systems to collaborate. By the time I put in the effort, I decided it was simpler to go all in and use a compile-to-JS language instead of something that tried to be Javascript with types.

<comment><USER18> This is 100% spot-on. With our web development product, Elevate Web Builder, the furthest that we went in the compiler with "giving in" to the dynamic typing in the target JS was a variant type. The rest has to be given an external interface declaration (classes, functions, and variables) that the compiler uses to type-check all calls to external JS or the built-in interfaces in the browser APIs. Other products have tried to allow for in- lining JS or simply include-ing it via special declarations, but that really handicaps what the compiler can do in terms of statically analyzing your codebase, which kind of defeats the whole purpose of using types in the first place.

</comment></comment></comment></comment></comment></comment><comment><USER19> This was posted multiple times in the last few days

[https://news.ycombinator.com/item?id=16880991](https://news.ycombinator.com/item?id=16880991) [https://news.ycombinator.com/item?id=16844285](https://news.ycombinator.com/item?id=16844285)

<comment><USER20> There should be a feature that matches the url of your link against posts within the last 7 days, and alerts you to possible reposts prior to submitting.

<comment><USER21> There is.

</comment></comment></comment><comment><USER22> I'm really appreciative of the mention of the inverted import.

Inverting the flow really lets components "register" themselves and thus be fully contained rather than be "required" by another component. It can be powerful but it causes the problem that you never really know what all is registered to another component.

Also, yeah, code-splitting isn't easy!

<comment><USER23> I have a talk about these ideas in the Context of a React Application last year at React.js Conf - [https://youtu.be/395ou6k6C6k](https://youtu.be/395ou6k6C6k). It might be worth a watch if you're interested in how to build applications like this on your own.

<comment><USER24> Does anyone knows if this can be done in Vue.js?

</comment></comment></comment><comment><USER25> I agree with much of what is written here, but the emphasis on code splitting seems like it risks hitting the wrong target. My first question is usually what caused even relatively simple web apps to need multi-megabytes of "optimised" JS code in the first place.

All too often, the answer is not the application code itself but rather all the direct and transitive dependencies, where the developers yarn-added first and didn't ask questions later. And while obviously that offers some advantages in terms of reusing code, it does also have a very high price in terms of bloating our deliverables, particularly given the still relatively immature tools for optimising JS compared to traditional compiled languages.

Maybe we should be looking at that problem first, before we start assuming that any large JS application will necessarily produce some huge monster outputs that need to rely on code splitting and all the overheads that brings just to get acceptable performance?

<comment><USER26> I think the author still has a point when talking about "very" large applications. His example of all the Google widgets is a good one. Even if you were to hand-craft all of them with optimised code (which I imagine is what they do at Google), it would still likely be too much code for a single package to load in a reasonable amount of time.

But I agree completely with you as far as other projects go. If you drop the "very", it should be entirely possible to build a large project without the need for code splitting. Almost nobody is doing anything to the scale of the example talked about in this article, and almost everybody is using code splitting.

In my experience, there's two root causes for this.

Cause #1 is that barely any front-end developers understand or consider the cost of abstractions. Something like code-splitting is commonly seen as "free" because it takes a couple of lines to implement. The permanently increased complexity of the software and all that extra brain power required to grok it over the development lifetime is never taken into account. At least half of the devs I know are happy just banging in code-splitting at the start of a project with zero thought and I guarantee you they'd read this article and not understand the "sync -> async" example given to explain the downsides of code- splitting.

Cause #2 is that devs are too eager to add new dependencies. By definition a popular library is going to be extremely bloated relative to your use case. When you 'yarn add react-redux-awesome-date-picker', what you're usually adding is an overly generic solution with a large amount of customisability and functionality that's not relevant to your use case.

My go-to example from my own experience is with rc-collapse. rc-collapse is a container that animates to and from 0 height. It's like 200 lines and has 4 dependencies (god knows how many KB that adds). I've been using my own version that's 50 lines and 0 dependencies in production code for years and never run into a problem with it. I'm sure rc-collapse works around some fancy edge cases or supports older browsers or something, but I'm almost positive the extra weight isn't necessary in 90% of the projects that use it.

This is kind of tangential, but another real problem with this mentality is that by implementing my own collapsible container, I learnt some important lessons about React, the DOM, browsers etc. Devs that play npm lego aren't generally going to get that extra knowledge, which will cost time in the long run.

</comment></comment><comment><USER27> The framework he is talking about is trash compared to modern JavaScript UI frameworks.

There are very few established patterns and each team seems to make up their own rules. Some of the worst code I've ever seen was a result of this framework.

The code-splitting is probably the best feature and worked great when this was released, but smart lazy-loading of bundles is much easier in 2018.

The most frustrating thing is that the framework developers consistently point to benchmarks created over 3 years ago to show how great it is.

<comment><USER28> This article isn't about the framework though. It's incidental.

This seems a very out of place comment.

<comment><USER27> I get what you're saying and maybe I'm being harsh, but the whole backdrop of the article is this supposedly amazing framework that was built. It undermines a lot of his points that it is poorly used.

<comment><USER29> The apps listed in the talk (Photos, Sites, Plus, Drive, Play) all seem to have gotten much better lately? Are you actually sure you are talking about the same framework as the author? Google has so many it can be confusing.

If you'd be talking about the closure library and dreaded goog.ui.Component, I'd agree but that isn't it.

</comment></comment></comment></comment><comment><USER30> It's interesting how in 2018 web development === react. React this, react that.

Anyway, I think beginner empathy is when you anticipate the usage of your API, or modules. Pro empathy is when you talk about it and share your feelings and needs and then listen to those other users' feelings and needs.

Fractured pluggable routing may be a good idea, but you know, at a certain point a centralized route definition may be a good idea too. There's no one solution fits all strategy.

Major routes === modules (oh boy, enhance). We're all MVC (with islands of goodness) again. Frontend is the new backend, but it's fronted too. Doing frontend is hard.

Dependency injection. Malte talks about the google search result page as an example for ultra complexity - Angular tries to sell itself for very high complexity ("enterprise") projects, yet I would eat my hat if Google plus or even the search result page would be better in Angular. The frontend landscape is fractured and everyone tries to sell you his or her snake oil.

Base bundles: yes, they suck. That's why common chunk creation is delegated to the bundler (like webpack) and then you believe that if a datepicker used more than three times it's okay to load it all the time. Or you can be a smartass and do require-ensure like logic all yourself (maybe it's not your fault, maybe just a very clever manager made you do it). And then you will feel very smart and good and after a year or so you realize how it was stupid and then someone comes along and relegates it back to the bundler.

Large applications in javascript are like Frankenstein's monster, but it's better to have a monster than a pile of dead flesh jolted by electricity with each and every release. My two cents.

<comment><USER31> > It's interesting how in 2018 web development === react. React this, react > that.

Its quite popular for sure, but after just doing a job search for about 6 months, I came across a surprising number of firms using ng, ember, and even vue in a couple places.

modern ruby on rails with ujs is still hugely popular

<comment><USER30> You're lucky. I've been through a jub-hunt cycle, my experiences here are: lots of legacy stuff (mostly angular1), tons of react, some nu-angular and one vue (which is going to be my next job, not just because I love vue, but also because they understood the tech stuff I had been talking about, unlike many others).

</comment></comment></comment><comment><USER32> This was a very interesting article!

I'd currently consider myself a mid-level engineer, and I have recently been put in a position where I must lead a team of many devs and incorporate all their code. I did understand about half of this article, but it also indicates that there's a lot left for me to learn.

I'm curious what kind of blogs or online courses or books I should search for if I wanted to better develop this skill within myself, and to design and architect these sorts of large applications with multiple contributors?

</comment><comment><USER33> My company also rolled our own framework to handle the case of 'very large web applications'.

In our case, every component is a file (which is becoming more typical) -- but it differs in that every file dependency is automatically resolved and downloaded in a lazy, on-demand fashion. With HTTP 2.0, this means you can update a single file without a build process -- and everyone receives the update as a minimal change.

Also, all paths are relative or built dynamically using something like C #define, so it's easy to refactor by moving entire directories about.

<comment><USER34> Don't the popular web frameworks handle lazy loading things?

<comment><USER33> Most of the popular frameworks generate a single JavaScript bundle that comprises an entire application. Some people break these bundles into multiple parts manually in a process called 'code-splitting' in order to do a really chunky kind of lazy load.

<comment><USER27> Code splitting via webpack is not a manual process.

Does your hand rolled framework handle minimization and obfuscation? Probably not because any changes would require a rebuild.

Are your assets cached? If they are, I'm curious how you handle the cache busting when a single file gets updated.

<comment><USER35> With HTTP/2, you can send files down independently, without creating a large bundle, so each file can be cached. You couldn't obfuscate function names across files though unless you made the process deterministic based on module name.

<comment><USER36> But you need some cache-busting functionality anyways (otherwise how does the client know that file "something.js" has changed?

<comment><USER35> Etags could be used to cause the browser to revalidate the contents without too much overhead since the connection is already establishd, but good point.

</comment></comment></comment></comment></comment></comment></comment><comment><USER37> I though the article would have just one line saying "Use Elm"

<comment><USER38> After 1 year using Elm, we've had a lot of great and not-so-great experiences. For a large app I still think choosing Elm was overall a better choice than javascript, but it's not without downsides.

Recently a new dev joined the front-end team. He had js background and no previous functional programming experience. In less than one month he was already shipping production code with confidence on our ~40kloc app. That goes to say that 1) I don't think lack of Elm programmers is a problem if the company is willing to train its employees and 2) I'm sure the front-end lead and the new dev would have a much harder time if we were using javascript.

Surely this is _our_ experience. YMMV.

<comment><USER17> The killer feature of Elm is something I discovered a year after taking a break from Elm: I revisited a production Elm application I hadn't touched in a year, approaching it almost regretfully with "why did I have to use Elm?", certain I'd have no idea what was going on after such a long absence.

What actually happened was that I was immediately productive.

Without having to credentialize in the code, I was able to stub out new features and push an update. An infrequent experience for me in other languages. I had only a figment of an idea of the code I had to write, but I got started and the compiler helped me the rest of the way.

I still don't understand how to confidently organize an Elm app. But that describes my relationship with every front-end Javascript framework as well. I regularly choose poorly between component vs upstream state in React. But what I _can_ do is refactor Elm code at a level that would be downright expensive in Javascript.

As far as the sour graping going on in some nooks of the Elm ecosystem, I'm reminded of this post by Rich Hickey: [https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_au...](https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_authority/do1olag/)

</comment></comment></comment><comment><USER39> I really dislike these frivolous images that can be easily represented by blockquotes. It just wastes data.

</comment><comment><USER40> what is the metric to consider a JS app as "VERY large" as the author puts it? If SLOC is used as a proxy then how code from libraries (i.e. code that the team didn't write) are considered? Recently when I generated a Ruby-on-Rails app with webpack enabled, it downloaded 8148 JS files in the node_modules folder. 675K SLOC. I thought it was absolute madness.

<comment><USER25> The size of node_modules probably isn't a great metric to use if it includes development tools and libraries (and all their transitive dependencies) as well as any libraries you're bundling into the files your user will download and run.

<comment><USER41> Well, I still have to have those on my computer and in my Docker container. And from a trust, auditing and security perspective it's pretty horrible, even if multi-stage builds can help

With Buble, Flow, and Roll-up, I can get a decent modern environment with as few dependencies as possible. Upsides? Much faster, too. Less disk space. Docker containers build faster, which is better for CI/CD. And it's easier to understand the codebase and dependency tree, including development tools.

<comment><USER25> Just to clarify, I'm not disagreeing with you that the current dependency- heavy JS world is madness or that the bloat during development activities is unhelpful. I'm just saying that having a large/complicated development environment is probably a separate issue to having a large application for most other practical purposes, but looking at the size of node_modules will often conflate the two.

</comment></comment></comment></comment><comment><USER42> which modern JS framework offers component level lazy loading? Angular offers only route based lazy loading, which as is limited as the article pointed out.

<comment><USER43> Pretty much trivial to implement in react.

        class Loader extends React.Component {
          state = {
            Component: null
          };

          componentWillMount() {
            import('./Component').then(Component => {
              this.setState({ Component });
            });
          }

          render() {
            const { Component } = this.state;
            if (!Component) {
              return <div>Loading...</div>;
            } else {
              return <Component/>;
            };
          }
        }

Might want to add a few lines of error handling for production use, but that is pretty much all you need.

<comment><USER30> Like the article (presentation) said it's not the best idea to do this with hundreds of components (I think he mentions latency) - in fact you "somehow" have to (should?) bundle things together and then you are back to square one or at least have to hink about pluggability/configuration again.

<comment><USER43> Obviously you don't want to use it for every component, you put it in key places where you want lazy loading to happen.

</comment></comment></comment></comment><comment><USER44> Does Google really use React? I got that impression from skimming the article. That'd be interesting given they develop at least two competing tech like polymer and angular.

<comment><USER16> The talk mentions they use an in-house framework, but react was probably substituted in to make it more approachable for the audience.

</comment></comment><comment><USER45> Isnt React actually open source? [https://github.com/facebook/react](https://github.com/facebook/react)

<comment><USER46> You were misled by the slide image interrupting the flow of the text:

 _So, I build this JavaScript framework at Google. It is used by Photos, Sites, Plus, Drive, Play, the search engine, all these sites. Some of them are pretty large, you might have used a few of them._

 _[Slide]_

 _This Javascript framework is not open source._

<comment><USER45> Oh, thanks for the clarification.

<comment><USER47> I found it a tad bit confusing as well, thanks for asking.

</comment></comment></comment></comment><comment><USER48> > This Javascript framework is not open source. The reason it is not open > source is that it kind of came out at the same time as React and I was like > "Does the world really need another JS framework to choose from?". Google > already has a few of those–Angular and Polymer–and felt like another one > would confuse people, so I just thought we'd just keep it to ourselves.

Given the current state of affairs, an open-sourcing of this framework would be welcome.

<comment><USER49> Do you have some specific reason for this (genuinely curious)?

Think we've put ourselves in a corner, so to speak, with the Angular, react, preact, vue, ember, etc etc options and this would offer something compelling from the authors examples?

<comment><USER48> The article discusses one of the major problems: "React component statically depend on their children." There are workarounds, sure, but there are all kinds of pitfalls with the dynamic workarounds, to the extent that I find myself breaking the react model regularly in applications (e.g. utilizing state outside of redux and forceUpdate)</comment></comment></comment>


How Gen Z Is Different, According to Social Scientists - <AUTHOR> https://psmag.com/ideas/how-gen-z-is-different-according-to-social-scientists <comment><USER2> This article reads like every other article about generation<variable>. In the early 2000s people at my university were also 'fully at home in cyberspace, moving seamlessly between the "real" world and the one online.' and they (according to articles like these) didn't like humanities classes, preferred to not have leaders, spent too much time in front of devices, agreed with various trendy campus politics of the day just like they did in the early 1980s or 70s or 1997. Nothing has changed.

Of course these terrible articles wouldn't be complete without a misleading picture to project the author's politics, not the generation<variable> politics. Every summary is the same too 'time will tell if this generation can confront these big challenges, which are the exact same challenges of the last 4 generations meaning workplace automation, increasing living costs, pandemic illnesses, environment problems...'. You could write an abstract function to just spit these articles out every 10 years, changing the varibles to whatever headlines of the moment.

<comment><AUTHOR> Interesting view. The issue you raise of "misleading picture" is important.

Hopefully, this iGeneration research project will address these types of weaknesses head-on, possibly in the limitations section of their report? For instance, when the authors explain their data collection process and analysis methods, it would be useful if they describe the steps taken to minimize or at least acknowledge possible biases that may affect validity.</comment></comment>


Debunking the EU Commission's Lies About ACTA - <AUTHOR> http://www.laquadrature.net/en/debunking-the-eu-commissions-lies-about-acta

<comment><AUTHOR> For those HN readers who are interested in original source material to make up their own mind about ACTA straight from the horse's mouth, the following links are a good starting point:

Full final treaty text as signed by the EU, including translations in languages of member states:

[http://register.consilium.europa.eu/servlet/driver?page=Resu...](http://register.consilium.europa.eu/servlet/driver?page=Result&lang=EN&typ=Advanced&cmsid=639&ff_COTE_DOCUMENT=12196/11&ff_COTE_DOSSIER_INST=&ff_TITRE=&ff_FT_TEXT=&ff_SOUS_COTE_MATIERE=&dd_DATE_DOCUMENT=&document_date_single_comparator=&document_date_single_date=&document_date_from_date=&document_date_to_date=&dd_DATE_REUNION=&meeting_date_single_comparator=&meeting_date_single_date=&meeting_date_from_date=&meeting_date_to_date=&fc=ALLLANG&srm=25&md=100&ssf=DATE_DOCUMENT+DESC)

"January 18th, 2010 consolidated text" including country positions makes interesting reading (PDF and transcript thanks to lqdn):

[http://www.laquadrature.net/wiki/ACTA_20100118_version_conso...](http://www.laquadrature.net/wiki/ACTA_20100118_version_consolidated_text)</comment>


Bitset Decoding on Apple's A12 - <AUTHOR> https://lemire.me/blog/2019/05/15/bitset-decoding-on-apples-a12/ <comment><USER2> "Apple A12 in my iPhone is limited to 2.5 GHz, so I multiply the result by 2.5 to get the number of cycles."

I can guarantee that unless he was going out his way to force it, the processor was not running at 2.5GHz... it would have been scaled way back as mobile processors are very aggressively tuned for power-saving.

Unless he can show how he managed to disable frequency scaling on the iPhone, the results are kind of meaningless.

If he wants to get the cycle count of the instructions, you don't have to run it, you need the datasheet from the manufacturer (which may be difficult to source admittedly).

<comment><USER3> > If he wants to get the cycle count of the instructions, you don't have to > run it, you need the datasheet from the manufacturer (which may be difficult > to source admittedly).

A12 is an out-of-order superscalar core. Counting the exact cycle counts of a sequence of instructions for such architectures by hand is basically a fool's errand, especially if you start running into issues such as exhausting register ports or issue stations.

<comment><USER4> I'm quite sure Apple has a tool for profiling iOS applications. ARM has PMU (Performance Measuring Unit) as part of the architecture, for exactly this sort of testing.

<comment><USER5> Instruments probably has this available; I know they do for Intel processors.

Edit: Just checked, and my A10 shows a small but standard set of PMCs, many of which show up here: [https://opensource.apple.com/source/xnu/xnu-4903.221.2/osfmk...](https://opensource.apple.com/source/xnu/xnu-4903.221.2/osfmk/arm64/kpc.c)

</comment></comment></comment></comment><comment><USER6> > On recent x64 processors, we find that it is beneficial to decode in bulk: > e.g., assume that there are at least 4 set bits, decode them without > checking whether there are four of them. The code might look as follow: > [...]

I find this assertion surprising, and so I decided to test it. I wrote a naive algorithm, and found the generated assembly even more surprising. [https://godbolt.org/z/ah3msM](https://godbolt.org/z/ah3msM)

        int bits(unsigned long n, char* b) {
            int retval = 0;
            while(n) {
                *b++ = __builtin_ctzll(n);
                n &= n-1;
                retval++;
            }
            return retval;
        }

First of all, the business with incrementing retval is discarded entirely. It simply performs popcnt on n and returns that. Second, it performs the calculation in two sections. First it does the naive, simple loop until the remaining bits are a multiple of 8. Then it unrolls the loop into groups of 8. It unrolled the loop even with just -O2 set. In essence, the compiler does precisely what his code does, but with groups of 8 instead of 4.

Clang seems to have changed behavior between versions 6 and 7; version 6 and below use the naive loop, as does GCC. Clang 7 and 8 unroll it. I'm curious if this had any effect on his results; his results for the A12 presumably are using Clang, while his linux box presumably uses GCC.

There's not really a whole lot of room for improvement in that unless you go whole hog and unroll the entire thing and table jump into it: [https://godbolt.org/z/qWzfHl](https://godbolt.org/z/qWzfHl)

        int bits(unsigned long n, char* b) {
            const int retval = __builtin_popcountll(n);
            switch(retval) {
                default: for(char i = 64; i--;) b[i] = i; return 64;
                case 63: b[62] = __builtin_ctzll(n); n &= n-1;
                case 62: b[61] = __builtin_ctzll(n); n &= n-1;
                case 61: b[60] = __builtin_ctzll(n); n &= n-1;
                // etc
                case 2: b[1] = __builtin_ctzll(n); n &= n-1;
                case 1: b[0] = __builtin_ctzll(n);
                case 0: return retval;
            }
        }

Which has its own macabre beauty to it.

Interesting distinction between Intel CPUs and AMD: the blsr instruction has a latency of 1 on Intel, and 2 on AMD, while tzcnt has a latency of 3 on Intel, and 2 on AMD. But the performance is limited by the blsr instruction, so Intel processes 1 bit per cycle, but AMD can only manage 1 bit per two cycles.

<comment><USER6> I looked into it a little more, and coded up some benchmarks, and it turns out I was wrong. His solution is like this:

        int bits(unsigned long n, char* b) {
            int retval = __builtin_popcountll(n);
            while(n) {
                *b[0] = __builtin_ctzll(n); n &= n-1;
                *b[1] = __builtin_ctzll(n); n &= n-1;
                *b[2] = __builtin_ctzll(n); n &= n-1;
                *b[3] = __builtin_ctzll(n); n &= n-1;
                b += 4;
            }
            return retval;
        }

Unrolled four times, this is faster than the naive code unrolled by the compiler, but slower than the jump table. Unrolled eight times, that is ~10% faster than the jump table, which really surprised me. How can a loop be faster than non-branching code? The crucial difference is that his version begins filling the pipeline immediately. The compiler unrolled version needs to wait on the popcnt before it can begin working, and the jump table needs to wait on both the popcnt and the subsequent jump. Despite the fact that the jump table is 100% branchless after the initial jump, its speed advantage down the stretch can't make up for the author's early start, _nor_ can it make up for the author's eventual branch misprediction and subsequent pipeline stall. Very strange. Cool, but strange.

I think the lesson here is to not jump to conclusions and post your faulty misconceptions about performance on the internet before you benchmark it.

</comment></comment><comment><USER7> What sort of application shows a noticeable speedup if we manage to make bitset decoding, say, ten times faster?

<comment><USER8> In this case, Mr Lemire previously design with collaborators a fast JSON parser (GB/s) that required such bit detection. You can find previous papers in his blog.

<comment><USER9> Prof Lemire

</comment></comment></comment><comment><USER10> Could a lookup table be faster even for fairly sparse bits? I'm imagining something like

      uint8_t *result;
      uint8_t numpositions[];
      uint64_t positions0[];
      uint64_t positions1[];
      ...

      *(uint64_t*) result = positions0[byte0]
      result += numpositions[byte0]
      *(uint64_t*) result = positions1[byte1]
      result += numpositions[byte1]
      ...

Or probably accumulating results in a register until you get 8.

<comment><USER11> Even if those arrays are entirely in cache they require more cycles per lookup than the computation from the article.

That's been true of lookup tables for a long while now.</comment></comment>


Why Do Americans Hate Android And Love Apple? - <AUTHOR> http://readwrite.com/2013/01/29/why-do-americans-hate-android-and-love-apple?

<comment><USER2> "I tend to think that open beats closed, that choice beats lack of choice, that diversity beats homogeneity, that offering a range of models from low- cost to high-cost ends up being a winning strategy."

Look at the world around you and realize that this is exactly the kind of model that we have in reality. Diversity is how the human race thrives, which is probably why it is also the winning strategy for almost anything.

</comment><comment><USER3> The author omitted the fact that Apple propaganda struck preemptively in the possible social immune response against their tactics. If you'd searched in 2009-2011 materials like "anti-apple" or something similar, you would have found all kinds of pseudo-critics meant to ridicule anti-Apple policy/sentiments (which might include you of course). In fact I think a lot of Apple-related critique was carefully directed by Apple (like say, the term "Apple fanboy" which is the mildest option of that could occur otherwise). Now the pressure is fading and it bothers less, but, but I still remember Apple and perceive the past years because of it like nightmare, a dark chapter in the industry.

</comment><comment><USER4> This is a very poignant article. personally, it doesn't bother me as much since I actually know what each platform is capable of. Most people don't bother to research and rely on friends or media telling them what to think. Either way, all that should probably matter is that they spend their money on apps and spend their free time using apps :)</comment>


The Beauty of Lisp-1 - <AUTHOR> http://www.lispcast.com/lisp1.html

<comment><USER2> On the mechanics of the post itself: I like putting the footnotes off to the side of the body. The alternative - using hyperlinks to jump back and forth from the location tot he end - is more intrusive.

<comment><USER3> Concur.

I think footnotes are an artifact of dead tree printing; that it has been translated directly to interactive media is ludicrous. I predict its extinction, but the wait is very surprising.

<comment><USER4> If only there were an easy, standard way to pop up a temporary "tooltip window" within a page. At that point, clicking on a footnote would pop up the actual footnote, rather than scrolling the browser to the bottom of the page.

(Putting footnotes within the page itself causes the page to be wider, which could potentially make the article harder to read on mobile devices. But at this point, a Javascript "popup window" solution would probably be worse.)

<comment><USER5> <abbr title="it is usually used for abbreviations, but why not?">abbr tag</abbr>

</comment></comment></comment></comment><comment><USER3> Aside:

The beauty of that logo!

Clojure's logo is beautiful too.

<comment><USER6> Didn't pay close attention at first, but the kanji brush-stroked lambda certainly says "Zen of Lisp."

</comment></comment><comment><USER7> People prefer arrays to singly-linked lists. Python gets it (but doesn't get many other things Lisp gets)

<comment><USER6> One of the advances of Clojure over other Lisps, is that vectors and maps are just as well supported as lists. They have their own reader syntax (using [] and {} as delimiters, respectively), and all three implement the Collection interface, which means a lot of Clojure functions work equally well on all three. There is also syntax for destructuring. And the use of vectors in places where destructuring is possible (like function arguments and let bindings) makes it easier to identify code structure at a glance (a complaint against other Lisps).

<comment><USER8> Common lisp sequence functions work on lists, vectors (including those specialized for bits, bytes, etc), and strings.

<comment><USER9> Some of them do. There are lots of redundant functions in CL, for reasons of backwards compatibility.

<comment><USER8> All of the sequence functions and special forms work on all of the sequence types.

The "redundancy" is that there are also functions/forms that work on specific sequence types. It would be surprising if there weren't because if two data types have exactly the same set of operations, one of them may be unnecessary.</comment></comment></comment></comment></comment>


Spotify Bets Big on Podcasting to Power Battle Against Apple - <AUTHOR> https://www.bloomberg.com/news/articles/2019-02-06/spotify-rises-on-podcast-acquisitions-robust-subscriber-growth <comment><AUTHOR> This is potentially very bad for podcasting as an open, RSS-driven medium. Welcome to a world of "Spotify exclusive shows" not available on your favourite podcast player.

<comment><USER2> What if Spotify is my favorite podcast player? Since I pay the subscription it's much better than other one's with ads. I got as a music player first, getting all of my podcasts there as well is a nice benefit. I don't really see how this hurts podcasting though, pretty sure you can still host podcasts elsewhere, just that with Spotify you could reach a greater audience.</comment></comment>


BMW Modified Car with Clay - <AUTHOR> https://youtu.be/YIG2PaBzxgk <comment><AUTHOR> I am Ahmad, 10years old, If you like my work plz comment. Thanks</comment>


How do you encourage users to share/copy/evolve data while preventing competitors from copying it? - <AUTHOR>

What sort of license would you use? <comment><USER2> If I understand you correctly, you want people to be able to use your data for whatever they want except to compete with you. So why not just make everyone agree to some form of a non-compete with regards to the use of your data? Just leave it broadly defined with the right to terminate their license at any time if you feel they are violating the provisions or spirit of your TOS.

<comment><AUTHOR> I don't want anyone to get a copy of a significant portion of the data. Moreover, I don't want competitors to use any of the data.

</comment></comment><comment><USER3> So what you're trying to do is prevent wholesale copying from competitors?

The trick here is the _agency_ doing the copying. You want to say that _users_ can copy it, but that _third parties_ can't.

My suggestion would be to be very clear that what you're talking about is bulk copying by competitors, and prevent it by requiring _users_ to initiate copies, rather than third parties.

</comment><comment><USER4> By data do you mean source code?

As a start you can review the open source licenses commonly used: <http://en.wikipedia.org/wiki/Open-source_license>

Or in theory you could always write your own license.. But once you open your source it will be impossible to completely protect your IP. People will find ways to copy it anyway..

<comment><AUTHOR> No, I mean the data contributed by users. For example, consider a wikipedia- like site.

<comment><USER5> _"... A Creative Commons license is based on copyright. So they apply to all works that are protected by copyright law. The kinds of works that are protected by copyright law are books, websites, blogs, photographs, films, videos, songs and other audio & visual recordings, for example. ..."_

Maybe you should re-phrase this to ask, _"How could I sell competitors the data if they want to buy it, but restrict them if they do not?"_. To me the licenses found in the Creative Commons accommodate this. They give you the licensing tools to flexibly control content and can let users to share content. Safe that in the knowledge competitors do not have the copyright rights to use/abuse licensed content without recourse to the law (in a lot of cases with laws adopted in your home country).

You cannot however use CC for 'facts and ideas' (as per CC FAQ, _How does a Creative Commons license operate?_ ) ~ <http://wiki.creativecommons.org/FAQ>

Having said that I know of no known tools or tool sets to automatically check for content, it's licensing (if any) and contact details for buyers. Buyers would grab at this seeking for _free_ content (images for instance) clear of any copyright issues.

If you really don't want competitors (are your users competitors as well?) to take away your content then don't create API's allowing users to use their own content. But then you risk making another "Roach Motel" allowing users to create but not export. Is this what you mean, lockout of users data?

<comment><AUTHOR> My service already has an export feature. I have no problems with users exporting their data provided that this constitutes only a small fraction of the entire data set AND that it is not used in a competing service.

<comment><USER5> _"... My service already has an export feature. I have no problems with users exporting their data provided that this constitutes only a small fraction of the entire data set ..."_

Excellent. Do you intend to go as far as say flickr and use an open API as well at some stage? This allows you to integrate third party services but at the potential competitor raids. Stuart Butterfield has lots to say on this particular problem ~ [http://flickr.com/groups/central/discuss/72157594165399644/#...](http://flickr.com/groups/central/discuss/72157594165399644/#comment72157594166309763)

Out of curiosity can you define _"data"_? I notice smugmug undertakes the type of restrictions you are describing in their pro account.

</comment></comment></comment></comment></comment><comment><USER6> You make it a royal pain in the ass to do it beyond the typical user scale to the point where it's cheaper to create their own similar data than to attempt to copy yours. Myspace does it for example. You can search their enitre user database with just clicks, but good luck in trying to scrape their entire user database.

Anything beyond that and your talking DRM.

</comment><comment><USER7> share alike, non commercial

<http://creativecommons.org/licenses/by-nc-sa/2.5/>

<comment><AUTHOR> I'm looking for something commercial that would prevent competitors from getting all the user data from my service.

But at the same time, I don't want to put major limitations on what users can do with their data -- even if it is obtained by copying/modifying data from others.

<comment><USER8> That's impossible.

If you're trying to stop automated scraping of your data, then limit the access rate per IP. That won't really help things since someone can just tell the script to slow down to the pace of a normal user.

If a user can access unauthenticated data, so can a computer.</comment></comment></comment>


Trunk-Based Development - <AUTHOR> https://trunkbaseddevelopment.com <comment><USER4> This workflow, like all others, is just a formalization of some reality. The reality is that many organizations and teams run on nothing but trunk - and it more-or-less works on them. Some teams work by building blocks separately and then joining them together; other prefer to hammer away on a problem all together. Both approaches work. What worries me are emotional claims that the other approach is fundamentally flawed, or that some git workflow is The One True Way.

I've seen feature branches get stuck in review for _weeks_. I've seen trunk- only developers not knowing how to merge a branch back to trunk. I've seen trunks brought into a state of disarray and never fixed again. I've seen svn- only devs get completely bogged down and confused by a git wofkflow.

Trunk-only is a reality. Not a reality some of us would want to live it - <USER3> one that nevertheless exists.

<comment><USER5> _Trunk-only is a reality. Not a reality some of us would want to live it - <USER3> one that nevertheless exists._

My background in Smalltalk made me accustomed to a style where everyone would continuously merge everyone else's changes as they worked. This style makes you aware of what your team mates are doing. In fact, it facilitates communication pretty much when communication is most called for.

As always: context. The above approach is only going to work well, when there is convenient high bandwidth communications between teammates. (Not only for your version control, but also for communication between people.)

 _I 've seen trunk-only developers not knowing how to merge a branch back to trunk._

Well of course. If they never had constant practice at merging, they wouldn't be very good at it. Merging isn't trivial.

 _I 've seen trunks brought into a state of disarray and never fixed again._

Can't they roll back? This strikes me as a sign of badly designed process, or developer incompetence. This should be seen as something like breaking the build. (In a CI environment, it would be breaking the build, yes?)

<comment><USER6> How does a team of Smalltalk developers work? How do you merge the binary Smalltalk program image?

<comment><USER7> With time, Smalltalk also got its own version control tooling.

Monticello for example,

[http://www.wiresong.ca/monticello/](http://www.wiresong.ca/monticello/)

<comment><USER5> You could consider Monticello as one of the precursors of git.

</comment></comment></comment></comment></comment><comment><USER8> _" The core requirement of Continuous Integration that all team members commit to trunk at least once every 24 hours."_

Continuous integration is a means to an end, not an end in itself.

<comment><USER9> Indeed. There's a wide spectrum of software complexity and while committing to trunk every day might be reasonable for lower complexity projects, it's definitely not for higher complexity ones.

Having such a requirement also means that people can't undertake major refactorings/rewrites of significant subsystems, leading to long term tech debt.

<comment><USER10> In my experience, the "refactor the world in a separate branch" strategy is always a giant mess. The initial work is perhaps simpler but the final merge is always horrible and the full impact of breaks isn't found until the merge is committed and you're stuck in "fix forward" mode because reverting is an unacceptable choice due to the cost of merging.

I'm not sure the overall cost to the team wouldn't be lower if these sorts of rewrites were done in place. (Also it's not really a refactoring if it's so big you can't do it in branch.)

<comment><USER11> Not disagreeing, but a way to avoid this is to pull changes from the main branch into the development branch every day, and write your new subsystem 'beside' the old one until it is ready to go. If you do it like this, it's not much easier for the branched developers than working on the trunk - it's still a moving codebase - but it does mean that if you cancel the rewrite there isn't any pollution to the main branch.

<comment><USER10> The scenario you called out, where the rewriting work might be abandoned/cancelled, is the only scenario I can think that makes a separate branch _actually_ better. For cases where you're confident that the work will not be cancelled, you have the same amount of work to push the rewrite back into the main branch constantly as you do to constantly keep the child branch in sync. i.e. If you can pull from the main branch into the rewrite branch and build/test successfully, you can do the same in reverse. If you _cannot_ pull from main into rewrite and build/test successfully, you're just postponing the pain to the end when you'll dump it on the rest of the team when you finally merge back and break everything in the main branch.

<comment><USER12> The disadvantage is that if your rewrite branch is halfway finished, you _can_ merge in the latest main branch, but if you do the reverse (which is exactly as much effort), you're blocking the other developers until your rewrite is done.

<comment><USER10> This is only true if you assume that the rewrite branch will be broken the entire time. You're only blocking the team if your work will be unusable until it's done. If that's the case, then you're also assuming at some point in the future you'll complete the work, get everything working again, and then merge everything back to master. It won't likely play out that way, though.

More likely is that:

* Your child branch will be test-broken and likely build-broken almost the entire time you develop.

* You'll stop taking frequent merges as they become painful because you're clearly not doing the work to patch everything up (and can't, because you've broken everything and likely can't even run tests).

* You'll be "almost done" and then spend three weeks trying to get everything building and testing again.

* You'll tell everyone to freeze checkins because you need to get merged and it's impossible because the branch is so far out of sync and and you're now conflicting on every tiny change.

* You'll finally push your merge back with half the tests disabled.

* The entire team will spend another month cleaning up the mess.

I've never seen a big feature/rewrite branch play out any other way.

<comment><USER12> True, in that case you're screwed :)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> > Trunk Based Development is a key enabler of Continuous Integration, and by > extension Continuous Delivery.

Quite false. How do you expect developers to take you seriously when you essentially say "you can't properly do CI/CD with your current approach"? I sure am enjoying CI/CD right now.

Trunk-based development seems to rely heavily on feature flags, which are a huge source of complexity and inconsistency. Even when useful, they're liabilities; not assets.

git-flow (and variations) is simple, enables consistent codebase state, makes no assumptions about your codebase/infrastructure, and allows you to know the exact state of production/staging.

Finally, as it happens so often with open source projects and initiatives, first thing I should see is a big "Why". Why do I want this? Why are the alternatives inferior?

<comment><USER14> > Quite false. How do you expect developers to take you seriously when you > essentially say "you can't properly do CI/CD with your current approach"? I > sure am enjoying CI/CD right now.

Quite true actually! : [https://en.wikipedia.org/wiki/Continuous_integration#Everyon...](https://en.wikipedia.org/wiki/Continuous_integration#Everyone_commits_to_the_baseline_every_day)

<comment><USER15> That's in a "Best Practices" section with this note at the top:

> This section contains instructions, advice, or how-to content. The purpose > of Wikipedia is to present facts, not to train. Please help improve this > article either by rewriting the how-to content or by moving it to > Wikiversity, Wikibooks or Wikivoyage. (May 2015)

<comment><USER14> Are you questioning the daily part (continuous) or the mainline part (integration)?

I'm also interested in your better source. Every book I've read on the subject and the top 4 search results on google say the same thing.

<comment><USER15> I'm not an expert, and I've read no books on the subject, so I'll refrain from suggesting sources. The implication of my comment was that it's disingenuous to use that link as proof when it's marked with the equivalent of a FIXME.

<comment><USER14> First off, I have no motive to be disingenuous. I really don't care how you collaborate on software with your team.

Secondly, CI itself is considered a best practice. I don't know how you could expect wikipedia to mark it as anything else?

Here's the top link from google if you really care to learn and aren't just here to be a contrarian: [https://www.thoughtworks.com/continuous- integration](https://www.thoughtworks.com/continuous-integration) . There is a wealth of information on this subject that I promise all says the same thing. We can argue about whether or not CI is useful, but the practice of integrating continuously is in all the literature as well as the name itself.

</comment></comment></comment></comment></comment></comment><comment><USER16> Puppet Labs' recently published State of DevOps (2016) report specifically calls out trunk-based development as a leading indicator of high performing organizations [1].

[1] [https://puppet.com/resources/white-paper/2016-state-of- devop...](https://puppet.com/resources/white-paper/2016-state-of-devops- report)

<comment><USER17> Amount of info required to download free report makes it not free. Can you quote?

<comment><USER18> All you have to do is enter an email address. It doesn't even have to be a real email.

<comment><USER19> It's pretty rude to ask every visitor to a free report to hand over contact details. What are you planning to do with all those contact details?

I can only think of plans that range from annoying to malicious. None that are good.

Don't normalize rude behavior. Conditionally free is not free.

<comment><USER20> It's their right as a producer to require email to view their content.

It's your right as a consumer to not want to make that trade.

Not rude. Rude is posting the raw file link to subvert the agreement or complain about it on hacker news.

<comment><USER21> You have a very odd view of rights. It's /within their means/ as a producer to require email to view their content. Just because you can do something doesn't make it your right to do something. Doesn't make it wrong to do it either, but that's god damn miles from a right.

In addition, just because you can do something doesn't mean you should, and it's within my means to assert that anyone who asks for my contact information but doesn't know me personally /definitely/ wants to spam me, there's no other reason for them to ask for that information.

Being polite, spamming is rude. Being rude, people who send or enable spam are worthless scum. I have no time for anyone who chooses willfully to be part of that cycle.

My freedom to express that opinion actually /is/ a right[1].

[1] [http://www.un.org/en/universal-declaration-human- rights/](http://www.un.org/en/universal-declaration-human-rights/)

<comment><USER22> > My freedom to express that opinion actually /is/ a right.

Sure, but [https://xkcd.com/1357/](https://xkcd.com/1357/)

<comment><USER21> It's odd that you would link that. From my perspective, I'm the one doing the boycotting and showing of the door. How do you see it?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> Correct me if I'm wrong but isn't this just a form of centralised version control? If this is a better approach for many teams, perhaps this an indication that DVCS weren't the silver bullet they were hyped up to be.

Aside from that, in my opinion the whole centralised version control vs. decentralised version control, and all the variants thereof, miss the bigger issue. Keeping track of the history of a file, and the authors of the changes, is trivial for any version control system. The real challenge is in resolving merge conflicts, and it's the low level of sophistication in merge tools that is the real bottleneck here. That's why I think tools like Semantic Merge are far more important to a development workflow than SVN vs Git vs Mercurial. It relies on the merge tool understanding the structure of code rather than treating it as an ordinary text file. Similar tools could be built for any language that offers a 'compiler as a service' (such as RLS for Rust).

[https://www.semanticmerge.com/](https://www.semanticmerge.com/)

<comment><USER24> SemanticMerge mentioned on the site: [https://trunkbaseddevelopment.com/game- changers/#plasticscm-...](https://trunkbaseddevelopment.com/game- changers/#plasticscm-s-semantic-merge-2013). It got it's name after I blogged about it too - [http://paulhammant.com/2013/02/14/features-i-would-love- sour...](http://paulhammant.com/2013/02/14/features-i-would-love-source- control-tools-to-have/) \- and am still owed a beer.

<comment><USER23> Thanks for the links, I'm glad you identified this as an issue and inspired further development of a solution (though I'm surprised you're owed a beer for such an obvious name, must've taken all of about 5 seconds to think of that one ;-) ).

Also found this article in the comments of your blog to be interesting:

[http://blog.plasticscm.com/2013/04/put-your-hands-on- program...](http://blog.plasticscm.com/2013/04/put-your-hands-on-programming- language.html?m=1)

This article you linked to was interesting too, is this a common workflow on CI solutions now?:

[https://blog.snap-ci.com/blog/2013/11/07/automatic-branch- tr...](https://blog.snap-ci.com/blog/2013/11/07/automatic-branch-tracking-and- integration/)

</comment></comment></comment><comment><USER17> The suggestion that feature branches hinder CD is just untrue, at least in my experience. To me, this seems like a silly restriction to put on such a critical tool (the scm) which is designed to support unlimited flexibility.

<comment><USER18> You must not understand what CD really means then, because full CD is impossible with feature branches. CD means every commit gets built and delivered all the way prod if it passes all tests. The key word is "continuous" as in, every single commit. Branch based means you only deliver once you merge. Merging is a manual step, so your not doing full-CD. The merge is basically your "trigger" and your doing "delivery when I am ready"

<comment><USER25> We deploy every commit in master and use short lived feature branches and pull requests.

I would still consider this continuous delivery.

<comment><USER18> By "short lived", how short lived? Is the branch only for one developer? Or is it shared? How do you enforce "short lived"? If the branches are not collaborated on, you might be doing trunk based without realizing it.

<comment><USER17> >you might be doing trunk based without realizing it.

Then trunk based is poorly named and apt to start a fruitless holy war. Nobody wants too many people on a feature branch, or for one to live too long. (But it happens, like rebranding an entire app or rewriting all the auth code, which is a good thing to be able to do. )

Is there a threshold of time/collaboration that makes it not-CD?

<comment><USER18> My point is that the article says that branches for the purpose of WIP and code review is OK as long as you don't share the branches. Trunk based is trunk "based" as in your git "base". Your based on trunk, not a branch. It's not called "commit directly to trunk only" development.

</comment></comment></comment></comment></comment></comment><comment><USER26> With a sentence like:

" It has been a lesser known branching model of choice since the mid-nineties, and considered tactically since the eighties. "

Given:

The Release Engineering of 4.3BSD [https://docs.freebsd.org/44doc/papers/releng.html](https://docs.freebsd.org/44doc/papers/releng.html) 1989

concerning a key software system and part of software history, released in 1986, which states:

" For the 4.3BSD release, certain users had permission to modify the master copy of the system source directly. ... The development phase continues until CSRG decides that it is appropriate to make a release. "

,that the rcs tools of the time basically didn't even support branches natively, and this model is followed to this day by the direct descendants of this project

I'm prone to think that.. well. Hmm.

</comment><comment><USER27> We currently use short-lived feature branches, merged via Pull Requests (+ review / automated testing) into the main development line. This way, we can communicate changes in a detailed manner before they are added to the product and make sure there is no unfished or bad code in the main branch. (The dev team is small, 5-8 devs).

I don't see (yet?), what benefits TBD would provide in such a setup.

<comment><USER28> With TBD you can reduce the overhead of coordination between committers.

Edit for clarification: "Branching in code", á la Feature Toggles is much better, because everything around it can be automated. With CVS branches, you shift that process into an earlier development stage where you need more human collaboration. So TBD can shift the focus of collaboration to things that matter more: the code.

<comment><USER27> I am very concerned that feature toggles add unnecessary complexity to the code.

With Lisp Macros, for example, that would actually be quite elegant but in a language like JavaScript, we'd end up with a bunch of if'a here and there, wouldn't we? One might simplify it a bit with a different architecture, though.

<comment><USER29> Long lived branches add complexity too, it's just hidden in GIT. The main issue with feature branches are merge conflicts: they are the most evil operation one can imagine. First you have your code, that you understand. Then you have others code that you have no idea what it does, so first you need to understand it. This might take hours when done properly. Usually people skim over this part BC it's boring. This leads to superficial merges that leads to very very evil and hard to find bugs. And these bugs are super hard to find in git logs, since the merge commit is 2 or more people's code.

Whereas the alternative, feature toggles are much better: first you don't need one for every small thing, add them to bigger features that take weeks to implement. With a team of ~10 developers you should not need more than 2 at one time (if you have more the management is at fault). Then feature toggles enforce modularity: by principle you should use them in as few places as possible, this will make the interface between the new feature and your current app as small as possible. This is a good thing!

</comment></comment></comment></comment><comment><USER30> This is what I do when I can get away with it. Suitable for smallish teams.

All developers sit in the same branch; changes and conflicts are visible (and resolved) immediately. All work takes place there including bug fixes.

When we branch out for release; the focus is on disabling unwanted features. Actual development work + "hardening" is done in the main trunk before the branch.

<comment><USER24> Small teams like Google's 25000 devs and QA automators? :)

<comment><USER30> I think it can work well with maybe 10-20 people.

Depends on how well you software is modularised.

<comment><USER31> I assure you it works well with 25000 people, I've worked in such an environment.

It seems to me that in the various threads I read here that there seems to be a bit of misunderstanding. If you work on short lived feature branches and merge them to trunk/master and other people branch new feature branches from trunk/master and so on you might be just doing TBD without calling it that way.

Many things are just common sense consequences of that, e.g. if other people are spawning their feature branches from trunk/master; then you don't want to merge something that breaks trunk/master, otherwise you'd be making the life for your colleagues harder, since now they cannot know whether the code they added breaks some tests or they are broken because somebody else broke them.

The feature-flag thing comes into play only when you want to break down a larger feature into smaller branches, each landing in trunk/master before continuing the next step. This get more important the more people can actually make changes on the same part of the codebase that you big feature is going to touch.

If it's only you that touches a section of your code base (and you can do it because it's well modularized), you won't feel much pain of making your feature branch last longer.

However, TBD shows its strengths precisely when the team grows; when you no longer can make the assumption that only you will work on a given piece of code.

<comment><USER32> > you might be just doing TBD without calling it that way

I think this happens an astounding amount. In particular, i think Git Flow is TBD with the names swizzled.

As far as i know, all the source control disciplines that people seriously advocate involve:

1\. Developers working on local copies of the code

2\. A shared copy of the code where developers integrate their work when they're done

3\. A copy of the code used to make releases, which is refreshed or recreated from the shared copy

These copies get different names, which are meaningless labels. In TBD, the shared copy is called 'trunk', and the release copy a 'release branch'. In Flow, the shared copy is 'develop' and the release copy is 'master'. They're just names. They don't matter.

The precise mechanism for moving work from local copies to the shared copy varies, but not in a way which really matters. In TBD, developers usually push straight to the trunk. If they're doing XP, they've done code review via pairing. In some teams, there might be pre-merge code review through a tool like Gerrit, Reviewboard, Phabricator, or some such, but still with an expectation of a merge happening quickly. In Flow, i guess there are pull reviews, but those are basically the same, a code review before merging. The Not Rocket Science Rule is this, but enforced by a robot [1].

The mechanism for moving work from the shared copy to the release copy varies a bit more. Strict CD shops release the trunk, but they will only push the latest point on the trunk which has passed all the checks, which is a bit like having an implicit branch. Conventional TBD cuts release branches. Flow does, that, and then adds a bit of a dance around merging individual release branches into the long-lived master branch, but it doesn't ultimately matter, because any changes made on an earlier release branch have already been merged back into develop. There's variation in where you fix urgent bugs: on the shared copy, followed by a copy to the release copy, or on the release copy, followed by a copy to the shared copy. I'm not sure that this is very significant.

Which leaves the only significant tuneables being how often developers integrate their changes, and how often the team makes releases. If you integrate often, you get easy merges and rapid integration feedback. If you integrate infrequently, you don't get those things. If you integrate more frequently than you release, then you will need disciplined incremental development, feature toggles, or some other way of making incomplete features unavailable in production.

[1] [http://graydon2.dreamwidth.org/1597.html](http://graydon2.dreamwidth.org/1597.html)

</comment></comment></comment></comment></comment><comment><USER25> Trunk based development works well on any scm that sucks at branching. Perforce, subversion etc.

But, on git, using GitHub-Flow is far superior.

The two poster children for TBD do not use git. Don't cargo cult their process without understanding the unique problems they have that you don't.

Edit: downvotes on HN? This isn't Reddit, and I'm advocating github flow, not git flow.

GitHub flow is trunk based development but with all work in feature branches that live for less than s day or two, instead of branch by abstraction in the core code. That's it!

<comment><USER33> GitFlow and it's derivatives have very poor handling for continued bug fixes of old released versions. It might be great for a truly continuous model such as Facebook and Gmail where there is only "one true version", but in a case where there is a current version and three old versions requiring bug fixes, it fails miserably.

<comment><USER34> I've implemented GitFlow with multiple support versions. We just added a "version" dimension to each of the main branches.

1.1/master 1.1/release 1.1/development

1.2/master 1.2/release 1.2/development

When we fix bugs in 1.x, we work off 1.1/development and merge into all later versions.

This doesn't fit for a "web" or "continuous delivery" model. It does fit very well for software houses with many customers on 6 monthly upgrade / development cycles.

</comment></comment></comment><comment><USER35> I described something that seems very similar years ago[1] (in response to git-flow), however I've since come up with something better: I think knowing how to get things live is important, and the _entire_ development process should reflect that -- too many people have a "code cut" that takes days or even weeks, and that's despite using the latest and greatest CI tools.

A lot of the problems may sound familiar: How do we test something? How do we know it's good enough? We have a UAT step, but it doesn't find all the problems, so do we need more tests? To be more careful?

My approach[2] turns the entire problem into a software problem, and it's proving very successful to me (faster request turnaround, fewer problem reports post-deploy, and so on). Being able to select a "git branch name" for a specific user and get acceptance is powerful.

[1]: [https://news.ycombinator.com/item?id=6125964](https://news.ycombinator.com/item?id=6125964)

[2]: [https://news.ycombinator.com/item?id=11190540](https://news.ycombinator.com/item?id=11190540)

</comment><comment><USER36> We have a smallish team working on web apps and use a trunk-based-development branching strategy. Not really to do with helping CI/CD.

The advantage we see is that new developers can start developing as soon as they clone the repository, without needing to switch to the 'development' branch. This is particularly useful when on boarding new developers who are inexperienced with Git. I found that when using Git Flow that the (stable) master branch frequently ends up out-of-date when developers are required to remember to merge into it releases into it.

Diagram of our branching strategy: [https://gist.github.com/CameronWills/abf9e307669b1005c88ef82...](https://gist.github.com/CameronWills/abf9e307669b1005c88ef82e55102cd0)

<comment><USER24> I looked at your diagram. I don;t think you're doing trunk based development.

</comment></comment><comment><USER37> TLDR: instead of using diffs of changed lines in branches, use 'if' statements in code to make Enterprise Continuous Integration gods happy.

<comment> UK-AL The advantages of feature branches is that you can turn them on and off easily without trying merge in, or revert and dealing with the resultant conflicts this can cause.

Feature switches also have more advanced features like being able targeting specific users, if you want a gradual rollout.

We use both feature branches and feature switches, since branches short be very short lived. However large areas of development require months.

</comment></comment><comment><USER38> In subversion the only way I can possible develop with sanity is in an "unstable trunk". Branch off releases N weeks before shipping, and have only the requirement that trunk passes _automated_ tests while release branches are manually tested. Obviously you don't release more than say once every month or two, but that's more than enough for most.

I'm honestly not sure if the site is satire?

No one develops in long lived feature branches in svn (rather, everyone does, but most only try it once before they realize the pain of trunk based dev is much smaller than that of svn merging).

<comment><USER29> If you release only a month or 2 you are doing something really wrong (or need to work based on some insane spec). Read the basics about CI and CD, read the clean code book, read about modern organisation practices (e.g. Github releases several times a day). Releasing every month or so is (thank God) a relic of the old waterfall development times.

<comment><USER38> We do CI, not CD. Lots of software doesn't do CD and lots of it can't. Games. Embedded. Desktop. Store-apps.

In my case it's large desktop software for structural engineering. Customers don't want new versions more than 1-2 times per year because of deployment, training and code compliance requirements. If we bork a release we have had users make a long download+install again. We don't just deploy the fixed one in prod to solve problems. Thats why a release goes through hundreds of hours of manual testing.

Not to mention data: a feature is a new file format. If we ship one new feature we basically ship a new format. Now you know why Autocad, Word and Illustrator ships once per year and not once per commit.

I think what a lot of people (who presumably do web dev only) forget is that web dev is just one of many software development disciplines, and a very young discipline too.

I know of the practices you mention. I like clean code (both the book and the idea). I'd do CD if I had a project some day that runs on a _server_. I hope I won't have to do web dev any day soon though.

<comment><USER29> You can do CD without releasing, its not pressing the release button that matters. CD means that you can press that button any time. Do small changes, test continuously, work on master.

If you make small changes and test them (using automation - unit tests, integration tests etc) you are catching bugs early, avoiding merge conflict (the most evil operation of them all). This is not my idea, its in the clean code book and its the basic of the CI/CD/TDD disciple.

And you can apply this to projects of all kind - we used it on the server, for iOS projects where we had to wait for Apple for weeks etc.

<comment><USER39> This is an important distinction. Another one is that just because you aren't deploying to production regularly doesn't mean you aren't deploying to other environments.

</comment></comment></comment></comment></comment><comment><USER24> One of the authors here. Ask questions :)

<comment><USER40> Paul Hammant? Thank you for your blogs and thanks for putting this site together, they're very informative. I work at a company that has been doing "Cascade" model for close to a decade [0], very similar to TBD but not quite the same. In addition to having a mainline trunk repo which everyone's work is pushed to, we have started utilizing Phabricator in the past few years -- it enables us to have short-running feature branches which aren't public to the upstream repository but can be shared if necessary and used for review (I'm not sure how similar this might be to GitHub flow? We use Mercurial).

[0] [http://paulhammant.com/2013/12/04/what_is_your_branching_mod...](http://paulhammant.com/2013/12/04/what_is_your_branching_model/)

</comment></comment><comment><USER41> Trunk only goes well with continuous deployment.

If each increment in deployment is small enough, then you know exactly when things break and rollbacks are easy.

The 3-2-1 system where you develop for 3 months, then merge for 2 months, then debug in production for 1 month, is a real drag, and doesn't lend itself to quick feedback at all.

For a software looking for product market fit with ultra short cycles, I would absolutely choose trunk only.

For engineering projects in areas where cycles are months, or years, more traditional branching makes more sense.

</comment><comment><USER5> The Envy configuration management system for Smalltalk worked like this. As you were working, you could see if someone had changed code "nearby" (same class, same category, same protocol) and you could review it and merge it in.

Perforce provides something with the same effect. If you periodically get the most recent revision, you can see files with collisions in your change sets, then merge them in.

</comment><comment><USER42> I read Drunk-Based Development and was a bit disappointed. I can get awesome ideas while very tired and usually write them down, but a few weeks later, even though I was super exited when I wrote it down, it no longer makes any sense ...

<comment><USER43> I read Trump-Based Development...

</comment></comment><comment><USER44> If there was indeed a superior way of developing software, why doesn't anyone take this knowledge and out-compete for example Linux? It has literally hundreds of branches of all kinds (stable branches, feature branches, test branches), this should be easy pickings! Or out-compete Google or Microsoft which, contrary to whats said in this article, use branches and probably in all their big projects. Sorry, I don't buy it. I suspect that if you are religiously against branches you will just have to re-invent them by some other name.

</comment><comment><USER45> > Google do Trunk Based Development and they have 25000 developers and QA > automators in that trunk

For some stuff, but

Is Android trunk-only? No.

Is V8 trunk-only? No.

Is Chrome trunk-only? No.

Those have release branches.

\---

Some software works with trunk-only paradigm, some does not. A rule of thumb is whether your customer-facing distribution is SaaS/web-based.

If you're making Google Search, you do it in a trunk. If you're making Google Chrome, you don't.

<comment><USER24> > Those have release branches.

Some Trunk Based Development teams do "branch for release" [https://trunkbaseddevelopment.com/branch-for- release/](https://trunkbaseddevelopment.com/branch-for-release/) and some do "release from trunk" [https://trunkbaseddevelopment.com/release-from- trunk/](https://trunkbaseddevelopment.com/release-from-trunk/)

Google gave numbers themselves - 95% of their devs are in one big trunk (formerly perforce, but re-written in-house in 2012). It's true that the open source facing teams are outside that, and have processes that are less lock- step than what they do in-house.

Android - Samsung, on each release (maybe not anymore), used to check all the composite parts of Android into one big trunk for themselves. Back into their Perforce.

> If you're making Google Chrome, you don't.

There's nothing stopping the Chrome team from doing Trunk Based Development on Github, and accepting unsolicited pull requests like any other team.

<comment><USER45> You can have long lived release branches in "trunk-based development"?!?

In that case, I guess I haven't ever seen a branching strategy that _wasn 't_ "trunk-based." What would that even be?

<comment><USER14> Anything involving long-lived feature branches especially ones with multiple collaborators, eg git-flow.

<comment><USER45> If I understand git-flow correctly, it's a trunk (develop) with a release branch (master).

<comment><USER24> We wrote about that: [https://trunkbaseddevelopment.com/alternative-branching- mode...](https://trunkbaseddevelopment.com/alternative-branching- models/#gitflow-and-similar)

<comment><USER45> Guess I should have RTFM.

Well written.

</comment></comment></comment></comment></comment></comment></comment><comment><USER46> Developing on master is something I favor, with tagging for releases, though it makes sense to create release branches if you are delivering software packages (that others will install) and not just deploying it to .com websites. Hotfixes can then be applied to both the maintaince branch and master.

</comment><comment><USER47> I think some people are getting confused about terminology between Trunk Based Development[1], Git Flow[2], and GitHub Flow[3]. TBD and GitHub Flow are nearly identical, basically if you use TBD on GitHub you use GitHub Flow. Git Flow (not GitHub Flow) is a model that is all about cutting releases, and supporting bug fixes to releases. Git Flow is nonsensical when doing Continuous Delivery[4] of a web application.

If you are delivering web app code to production every day, it doesn't make sense to cut versioned releases, with for example semantic versioning, instead you just release the code that has passed tests. It also doesn't make sense to have hotfix branches, because there is only one version in production[5], and you release every day, so there is no need to backport a fix to older versions, or do anything special to get a fix out quickly.

Trunk Based Development has branches. If you are working with a team you pretty much never want to commit straight to master. The difference is that in TBD the branches are short-lived and only exist for code review purposes.

This is what TBD would look like with GitHub and GitHub "forks".

        git remote -a
            origin    email@example.com:<your company>/<repo>.git (fetch)
            origin    firstname.lastname@example.com:<your company>/<repo>.git (push)
            <github username>    firstname.lastname@example.com:<github username>/<repo>.git (fetch)
            <github username>    firstname.lastname@example.com:<github username>/<repo>.git (push)
        git checkout master
        git pull # make sure you are starting with the latest master
        git checkout -b my-cool-feature # create a branch for you change
        vim <file>
        git add <file>
        git status # check that all expected code is staged
        git diff HEAD # review your changes
        git commit # write a meaningful commit message
        git fetch # get the latest changes from origin
        git rebase origin/master # put your changes on top of the latest changes from master
        git push <github username> my-cool-feature:my-cool-feature # push to a branch on your "fork"
        # Open a PR to merge from <github username>/my-cool-feature to origin/master
        # Once the PR is approved, use the GitHub UI to merge the PR into master
        git checkout master
        git pull # pull down your code that just merged to master
        git branch -d my-cool-feature # delete your feature branch

[1] [https://trunkbaseddevelopment.com/](https://trunkbaseddevelopment.com/)

[2] [http://nvie.com/posts/a-successful-git-branching- model/](http://nvie.com/posts/a-successful-git-branching-model/)

[3] [https://guides.github.com/introduction/flow/](https://guides.github.com/introduction/flow/)

[4] I'm defining CD to mean delivering to production at least once a day.

[5] Unlike on-premises software, with a SaaS product you only have two versions: (1) what is in production, and (2) what is going out to production (in staging, canary, etc.).

<comment><USER25> This is all good, if you are on git.

Like I said in my other downvoted post... too many TBD zealots look at big companies that use it sans branches, and assume that it is the one true way, instead of understanding the larger picture (branches outside of git suck, these companies don't use git for various reasons)

<comment><USER24> Many of Google's 25000 Trunk-Based-Developers use Git on their local deevloper workstations. They choose their own workflow on their own machine, and then cooperate with the Mondrian submission rules when attempting to get their commit(s) through code review and automated tests/lint/findbugs etc.

</comment></comment></comment><comment><USER48> Yup, as the other comment stated, this is par for the course in modern quality software development. Its actually rather shocking to me to see my comment get downvoted. The quality of the HN audience is declining dramatically. Feels like all we have is a bunch of web agency lifers at this point.

<comment><USER49> Feature flags are used by essentially every major software company. Hell, the browser you're using to post this has quite a few. A bunch of major software companies use them heavily as well.

You may disagree and think they're terrible, but calling their users just "web agency lifers" is objectively incorrect.

</comment></comment><comment><USER50> No wonder they call it trunk, it's "branches in SVN are pure suck" elevated to some sort of grand vision. Must have been feeling left out with all the cool kids doing Git.</comment>