Ask HN: How do you manage state in your React application? - <AUTHOR> Is it recommended to use redux to share state across different components while you are using react-apollo client to sync remote data between client and server?<p>Looking forward to your views <comment><USER2> Been using MobX for about three years and I think it is wonderful. Incredibly easy to use (mark your variable as observable, modify it like you would any normal variable, and any components which use those variables in their rendering and are marked as observers automatically update), very performant out of the box (only components affected by a change re-render so no shouldComponentUpdate needed) and some excellent features like computed variables for derived values which automatically update. I really urge everyone to try it out. There is the odd time when the abstraction us a bit leaky (e.g. working with arrays is a bit different to plain JS) but with proxies in MobX 5 this is much improved.

I started out with Redux and I do think it's worth learning - not just because it's so widespread, but the concepts behind it are worth knowing and have influenced the way I design some stuff, but I found it really verbose and slow to work with compared to MobX (and more error prone) - there probably are libraries that help with this.

React Hooks look interesting for more simple use cases but I expect I'll stick with MobX for the foreseeable future.

<comment><USER3> Try mobx-state-tree. It's basically MobX with Redux-like tree structure written by the guy who did MobX. It combines best of both worlds. We use it at work and it's just such pleasure to work with, highly recommend

<comment><USER4> It is indeen wonderfully easy to work with. However, it has some performance penalties. I found typescript in conjunction with pure MobX just as efficient.

Especially complex state de-/serialization is a breeze with serializr.

<comment><USER2> Yeah exactly, for my use case (mobile device, frequent state updates) I couldn't justify the overhead of state-tree after doing some profiling. Shame as it looks great and would have been a good fit ofberwise.

Thanks for the pointer to serializr - I hadn't seen that before!

</comment></comment></comment></comment><comment><USER5> We use Redux at work for larger projects, React setState/Context for smaller ones, and I'm personally a huge fan of mobx-state-tree which I think is the best solution, and can be plugged into Redux if needed.

I think Redux should not be used for everything (there are libraries like Redux-Form which I think are an anti-pattern as ephemeral state like that should never go through the root reducer on every key change)

<comment><USER6> I hate redux-form so. much.

</comment></comment><comment><USER7> In the past I used Redux extensively - and I have only good things to say about Redux. I recently started using React Context and found I was much more productive, to the extent that I went back and ripped out Redux code. The new hooks thing with useContext makes using context even easier.

Learning Redux, especially the "single source of truth" thing was very helpful, but my code is now simpler, easier to understand and faster to write.

I would recommend checking out React context and hooks. I would also recommend watching the Redux videos by Dan Abramov, even if you don't use Redux.

</comment><comment><USER8> At my current job we use Redux, though I wouldn't choose it myself (it had been adopted already when I joined). I think Redux is best suited to very large and complex applications, with very disciplined and skilled engineering teams, who talk to each other, refactor code, and have the time to think through things like the correct arrangement of application state, providing reusable selectors. Those teams, of course, would probably also do fine without Redux.

If you have a team that writes spaghetti code, or never has time to refactor, or doesn't have effective communication then things will be worse with Redux, because it makes the code much harder to follow (good luck stepping through things in a debugger to see how they work, for example).

This is mostly intended as a criticism of people/teams who think adopting Redux will solve problems with the messiness of their code. But I also think Redux is overly complicated and too abstract for 95% of projects. And don't get me started on Sagas...

I would recommend setState as far as you can push it, then choosing something conceptually simple like MobX (Or MobX State Tree), React Easy State, or whatever the one from Formidable Labs is. Using Redux correctly is not as simple as it seems on the surface, and it requires real thought and real understanding o t

<comment><USER9> FWIW, we recently added a new "action stack trace" feature to the Redux DevTools Extension that helps show exactly where each action was dispatched from [0].

Also, I agree that sagas are not necessary for most apps, and we have some advice in the Redux FAQ on how to choose a side effects handling approach [1]. For most apps, thunks are more than sufficient.

[0] [https://github.com/zalmoxisus/redux-devtools- extension/blob/...](https://github.com/zalmoxisus/redux-devtools- extension/blob/master/docs/Features/Trace.md)

[1] [https://redux.js.org/faq/actions#what-async-middleware- shoul...](https://redux.js.org/faq/actions#what-async-middleware-should-i-use- how-do-you-decide-between-thunks-sagas-observables-or-something-else)

</comment></comment><comment><USER9> I'm a Redux maintainer. I'd specifically encourage folks to take a look at our new Redux Starter Kit package [0]. It includes several utilities that add good opinionated defaults around the small Redux core to simplify common use cases, including store setup, defining reducers, immutable update logic, and even creating entire "slices" of state without writing any action creators or action types by hand.

Long-term, I want this to be the standard way that most people use Redux.

[0] [https://redux-starter-kit.js.org](https://redux-starter-kit.js.org)

</comment><comment><USER10> I created my own library for it.

Compared to redux it's far more simple and requires none of the boilerplate. It also allows perfect type support with typescript.

[https://github.com/gunn/pure-store](https://github.com/gunn/pure-store)

</comment><comment><USER11> Mobx is my go to. It's fast, and a much easier mental model to integrate. Just inject the store you want into your components

@inject

and you can call/set @observables easily.

It's much easier to onboard people. Redux is just a spaghetti mess of files upon files, trying to find the actual implementation of something. A mess.

\---

I also looked into Apollo and it's obscenely good! Terrific dev UX. I'm heavily researching it currently.

</comment><comment><USER12> Using redux v4 with react-redux v5 for a large and complicated project. Was looking into react-redux v6 but it looks like there may be a significant performance hit ([https://github.com/reduxjs/react- redux/issues/1164](https://github.com/reduxjs/react-redux/issues/1164)) so will hold off on it now.

react-apollo is very interesting to me, but my understanding is that we'd need a graphql compatible backend, which we don't have

<comment><USER9> Hi, I'm a Redux maintainer.

I wouldn't call it a _significant_ performance hit. There's a measurable difference in our artificial stress tests benchmarks, and some users have reported slowdowns in specific scenarios (especially when large forms are connected to Redux).

I recently posted a roadmap issue with our plans for addressing the perf issues, and moving towards the ability to ship a hooks-based API for React- Redux:

[https://github.com/reduxjs/react- redux/issues/1177](https://github.com/reduxjs/react-redux/issues/1177)

I've specifically spent the last week experimenting with reworking the internals of `connect` to come up with an alternative implementation that relies on direct store subscriptions again, and as of a couple days ago, I seem to have come up with something that is actually _faster_ than both v6 _and_ v5 in our benchmarks:

[https://github.com/reduxjs/react- redux/issues/1177#issuecomm...](https://github.com/reduxjs/react- redux/issues/1177#issuecomment-461699799)

Still needs more investigating and testing, but this approach looks very promising.

<comment><USER12> I'm not sure how you guys qualify "large" forms, but our application is pretty much all inputs (dynamically generated somewhere between 25 and ~300 visible on screen at one time)

We're also investigating hooks, we think we'd get some benefits from not having so many HOCs... specifically as our application is a _large_ form, unmounting components can sometimes be very expensive for us (common example: switching between two React tabs). Excited to see when you guys can release this change since it IS breaking, hopefully a major version change will suffice here :)

<comment><USER9> If you've got any public examples of apps that are showing perf issues (especially with React-Redux v6), please let me know. Ditto if you can come up with a smaller repo or CodeSandbox that demonstrates the issues.

Our artificial stress test benchmarks are better than nothing, but I really want to get some more "real-world"-type scenarios put together that we can use to compare behavior.

</comment></comment></comment></comment><comment><USER13> apollo has a local state link ([https://www.apollographql.com/docs/react/essentials/local- st...](https://www.apollographql.com/docs/react/essentials/local-state.html)) but I did not find it useful.

Redux is fine but I guess with the new hooks functionality one can combine usereducer and contexts to create a much simpler state manager.

</comment><comment><USER14> Back when Redux was fairly new, we used a "mixin" architecture with a single component refreshing the entire global state of the application. Unfortunately, this has been a horrible decision and we're currently moving towards a Redux based solution. Nonetheless, the rendering is still impressive.

[https://cellcollective.org](https://cellcollective.org)

</comment><comment><USER15> there are a lot of options out there and this thread kind of surprised me was expecting more interesting submissions. you might want to look at the original flux:

[https://github.com/facebook/flux](https://github.com/facebook/flux)

from facebook if you app has a lot of global state needs multiple stores can be handy.

Mobx is great too:

[https://github.com/mobxjs/mobx](https://github.com/mobxjs/mobx)

With the current project I am working on we actually just vanilla javascript with singeltons with subscriptions (note: I did not write this, but I think it is rxjs).

anyways someone should come along with some answers.

Redux is still the easiest and with Sagas is really cool. One start up I interviewed with was using something similar to hooks, but do not remember the name of it.

Also datomic (used by om.next in clojurescript) has some really clever optimizations and is a joy to use.

</comment><comment><USER16> Full disclosure, I'm the author of Breezy, [https://github.com/jho406/Breezy](https://github.com/jho406/Breezy). An opinionated state shape and PJAX library for Redux.

I think "how you shape your state to make it easy to reduce and share between components" is an important topic that's taken a backseat when discussing state management tooling.

Should I repeat my backend models? Should I use a redux ORM? How much should I denormalize or normalize?

I found it much easier to just denormalize everything and shape state for presentational purposes, e.g, I have a bunch of page nodes in my redux state each with a duplicated header. If I need to make an update to the header in response to, say, a change in user email, I change each duplicated header individually.

Sounds insane but it really works for me and let's me keep more business logic in the backend where it belongs.

</comment><comment><USER17> If you don't have to support IE11, there has been a very interesting trend of JS Proxy based state managers (immer, react-easy-state, overmind, etc.). It reduces the boilerplate compared to redux while giving basically the same architecture. I have been using my own implementation, because react-easy- state did not work well with styled components ([https://github.com/UgnisSoftware/lape](https://github.com/UgnisSoftware/lape)), while I was building [https://github.com/UgnisSoftware/Ugnis](https://github.com/UgnisSoftware/Ugnis) if you want to see this approach in use.

</comment><comment><USER18> The concept of having one store with associated actions is a great one, but Redux itself has an enormous amount of boilerplate having to do with the fact that you do all calls into your state functions as events which is completely unnecessary. Every project I've worked on using redux ends up having a ton of different abstractions just to reduce the redux event boilerplate overhead. I use unstated [https://github.com/jamiebuilds/unstated](https://github.com/jamiebuilds/unstated) , which gives you the beneficial central store structure without the silly event concept.

</comment><comment><USER19> I prefer mobx to redux bc it's simpler.

</comment><comment><USER20> Redux. I really like keeping my state flat as the app I mainly work on has a lot of pages which use the same data. Complex component props are derived in mapStateToProps. I find reselect complex and hard to use with typescript, so I simply use memoized functions with memoize-one.

</comment><comment><USER21> For my thick client robot management applications: Redux and an Immutable.Record. A very flat state. The only nested elements are GeoJSON features. Been doing this successfully across half a dozen deployed applications. It's really simple.

</comment><comment><USER22> Just to offer a different viewpoint, my app is rather lightweight in features and a rather hierarchical structure. So I have bern able to manage just fine with props and state. It rather feels quite clean that way

</comment><comment><USER23> Hooks + Context seems like a win, as does xstate [https://github.com/davidkpiano/xstate](https://github.com/davidkpiano/xstate)

</comment><comment><USER24> May I suggest giving Muster a look? ([https://news.ycombinator.com/item?id=19134930](https://news.ycombinator.com/item?id=19134930))

</comment><comment><AUTHOR> I would like to thank each and everyone for their response. I will go through the suggestions and see how I can incorporate them in my app and post an update later :)

</comment><comment><USER25> Pass the things returned from useReducer() hook (state object + dispatch function) down via context. Then useContext() in child components.

<comment> mustafa-alfar How can you implement that, the useReducer must be inside the hooks component, so how can I export the value returned from useReducer to pass it into createContext ?

</comment></comment><comment><USER26> Been using the new Context API which has been super easy and simple to use. Though our product is not a giant SPA.

</comment><comment><USER27> Using Redux at work, looking forward to trying out the newly released React Hooks for less sophisticated needs

</comment><comment><USER28> [http://meiosis.js.org](http://meiosis.js.org) \- this is the best way to manage state

</comment><comment><USER29> react-easy-state for smaller projects, apollo for bigger ones.

</comment><comment><USER30> Redux is the best!</comment>


Hubris and GHC 6.12 + dynamic linking - <AUTHOR> http://www.shimweasel.com/2009/09/10/hubris-and-ghc-6-12-an-experiment-in-dynamic-linking

<comment><USER2> Wow! All the dynamic features of Haskell _and_ the type safety of Ruby? Where do I sign up?

(Yes, I'm being sarcastic. It's actually a neat accomplishment)

<comment><USER3> The code's pretty simple and there's still a lot to do in terms of making it easily installable, but hopefully it's useful to some people.</comment></comment>


Let's Break Up Adobe - <AUTHOR> https://tedium.co/2019/03/12/adobe-history-antitrust-concerns <comment><USER2> >Photoshop alternatives ... are not designed to handle animated GIFs.

>Which means ... it looks like I'm stuck in that ecosystem for a little while longer

Gimp handles animated gifs just fine imo.</comment>


Web Crawling: Why is Service a Better Option than Software? - <AUTHOR> http://www.grepsr.com/blog/web-crawling-why-can-service-be-a-better-option-than-software/

<comment><USER2> This kind of article is honestly insulting to the intelligence of anyone who has written more than a dozen lines of code. _It literally says nothing_ while "magically" hiding the fact that "services" are just "software" handled by someone else. Seriously?

<comment><USER3> Hmm?

 _" Grepsr (...) uses software as a service"_

 _" With service mode, however, the software that runs the service (...)"_

 _" A powerful crawling software runs at the back of our system (...)"_

</comment></comment><comment><USER4> Having had direct contact with all the issue of massive crawls (talking about tens of thousands of websites daily), I can say that it depends on the scale and "business-criticality" of the crawl. For the types of crawls that their product seems to be targeted at (quite small scale and with a very precise "point and gather" approach), it might be efficient to go the service way.

But the author fails to mention that the outsourcing decision holds true only for all non-mission critical crawls (whether it's for marketing, accounting or any other operations) while it's totally unfeasible if your company is doing big number crunching. When the product is the data, you need to control the accuracy and timeliness of the crawl without a doubt.

<comment><AUTHOR> Right now most of our revenue is from bigger projects servicing products which are data centric. We cater both type of projects -- small and big.

Not all companies (consultant firms, investment banks, equity researchers) have the expertise to do a in-house solution for bigger projects. This is where a service has an upper hand over using a tool.

</comment></comment><comment><USER5> [http://www.bbc.co.uk/news/technology-23866614](http://www.bbc.co.uk/news/technology-23866614)

        Google handles more than 100 billion searches each month or more than three billion per day.
        Google has to crawl more than 20 billion pages a day to keep up with the ever-changing web.
        Google has found 60 trillion web addresses so far, up from one trillion in 2008.
        Google has indexed pages from over 230 million domains.
        A query on Google travels 750 miles each way, to and from the data centre.</comment>


How to spy on teen drivers - <AUTHOR> http://money.msn.com/frugal-living/post.aspx?_blg=2&post=b8f30e0b-09f2-40c7-8758-c49ee95a9ecf

<comment><USER2> A new statup just launched that handles this situation quite well. <http://thecanaryproject.com/>

I was at a fireside chat for the launch with one of the founders. I like the app and it gives parents an amazing tool to combat teenage rebellion.

</comment><comment><USER3> > _Suddenly an onboard camera doesn't sound like such an invasion of privacy._

Why of course! Once there's an _upside_ to an invasion of privacy, it doesn't seem like such an invasion of privacy.</comment>


How Yelp Crushed Citysearch & Yahoo Local - <AUTHOR> http://searchengineland.com/yelp-crushed-citysearch-yahoo-why-google-stealing-yelps-playbook-78623

<comment><USER2> "Firsts" ... Game over.</comment>


Ask HN: Trust equity promises? - <AUTHOR> Howdy,
2-3 months ago I joined a startup that had & still has some problems. I was promised the CTO position along with some equity & wage on the next round. So I've been doing the work because it's a crucial time, but still nothing on paper, I told them and say to wait a bit more till we get the next funding, since the cap table will change. I've been told some numbers, but they keep changing, and I don't really trust them. Am I irrational to demand that I know what my equity & salary will be and on writing somehow?
P.S: It's not a US company.
Thanks in advance <comment><USER2> Having equity means being in business with the other people who hold equity. If there's no trust, it may make sense to question going into business with the other people. The changing offers, and I am assuming that they are becoming less attractive not more [though it really doesn't make any difference when nothing is in writing, and no much difference even then] are not a good sign.

My general experience is that people who enthusiastically let someone else work for free rarely change direction and pay well later.

Good luck.

<comment><AUTHOR> Thanks for your reply. It's not really the chiefs that I don't trust, but rather one notorious investor that holds the biggest piece.

<comment><USER2> That might be a less than ideal structuring of ownership. An investor with a track record of businesses where people work for free with nothing in writing may not be an omen of your prosperity.

</comment></comment></comment><comment><USER3> You are definitely not irrational to ask all this in writing, i would want to do that from both points of view. Even if it's not legally binding, it's always much easier to discuss promised things when you have them written down in some form.

</comment><comment><USER4> No, you are not irrational asking upfront. And get everything in writing. Whether in the U.S, outside the U.S, on Mars,etc.. get it in writing. Better be safe than sorry.</comment>


2k drones light up night sky in Shanghai to welcome new year - <AUTHOR> https://m.youtube.com/watch?v=Zh3-Mvrp1Lc <comment><USER2> I -used- to love fireworks, but over the years it dawned on me the huge negative impacts that they have have (pollution, terrifying wildlife and pets, etc) and I'm _now_ unable to see them as much other than an extravagant, arrogant, complete waste.

While I'm aware that there are a lot of eco-costs involved in building drones, it's nice to see alternatives to fireworks - that are at the very least are reusable.

<comment><AUTHOR> I was thinking the same thing as well. It's very interesting that fireworks originated in china and in a way it's now full circle

</comment></comment><comment><USER3> Singapore has 500 drones too! [https://m.youtube.com/watch?v=zmpB3SEZ2Rk](https://m.youtube.com/watch?v=zmpB3SEZ2Rk)

<comment><USER4> Does anyone know what kind of tech is used for this and the original content?

All radio frequency communication?

What drones are used?

How do the drones learn about one another's position mid-flight?

Looks like Intel has an offering from $150k to $350k+: [https://www.intel.com/content/www/us/en/technology- innovatio...](https://www.intel.com/content/www/us/en/technology- innovation/aerial-technology-light-show.html)

</comment></comment><comment><USER5> There are many reports coming out that this was faked. People who were there posting videos of waiting for the show, and then nothing happening. I believe it as some of the animations look a bit too fast or precise to be done by real drones.

<comment><USER6> > many reports

The only thing I can find is a Reddit post saying it's fake. Got something substantial instead?

Given the Chinese previously broke the record 2 years ago with 1300 and Intel broke it again just recently with over 2000 why exactly do you think it's fake?

[https://www.guinnessworldrecords.com/news/commercial/2018/5/...](https://www.guinnessworldrecords.com/news/commercial/2018/5/video- dazzling-display-over-chinese-city-as-more-than-1-300-drones-take-to- the-s-524413)

<comment><USER7>
       They admitted it that it was fake.

[https://www.bbc.com/news/world-asia- china-50979557](https://www.bbc.com/news/world-asia-china-50979557)

</comment></comment></comment><comment><USER8> I thought about doing this about ten years ago when our town decided to discontinue fireworks shows due to fire risk. At the time it would have been too expensive to attempt. It's really neat to finally see this happen.

</comment><comment><USER9> Maybe I am the only one feeling this way? These drone shows are just silly elementary level computer visual arts with a huge size.

Fireworks are visually way way better than these, and more creative, more exciting and holiday-ish.

<comment><USER4> There's definitely something to it existing at scale, in the real world, but I mostly agree.

I'd be interested to see what visuals can happen if the drones had additional props, like fog and directable light beams.

</comment></comment><comment><USER10> I wonder if sound from these 2k drones was deafening.

<comment><AUTHOR> Curious too. If they're nearby maybe so but high up in the air they may not be too loud.</comment></comment>


Stack Overflow: How We Do Deployment - Nick-Craver http://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-do-deployment-2016-edition/ <comment><AUTHOR> How people manage git and source control tells you a lot of things about a company's culture. They said that most commits go directly onto master and this works for them, which indicates:

\- Good rapid communication about who is working where. People are generally not touching the same code or else you'd run into frequent collisions (solvable via rebasing of course but they would be doing more branching if it were a thing to happen very frequently I'd suspect)

\- The developers are given autonomy and have assumed some level of mastery over whatever their domain is. Trust in each developer's ability to commit well-formed and considered code.

\- They have a comprehensive test stack, which helps verify the above point and keep it sane

<comment><USER2> I found this very curious - by their own admission, this also means that most code _does not get reviewed_ before it lands in production. To me, this is quite scary, and I would be very hesitant to adopt this for any large-scale project or company.

IMO, code review is a cornerstone of code quality and production stability - the number of dumb (and smart!) mistakes in my code that have been caught in CR are numerous, and it's a big portion of my workflow. There are times when I feel it's redundant (one-line changes, spelling mistakes, etc), but I wouldn't trade those slowdowns for a system where I only got review when I explicitly wanted it.

Of course, for pre-production project and/or times when speed is of the utmost concern, dropping back to committing to master might make sense, but for an established and (I'm assuming) fairly large/complex codebase, I would think that it would be best for maintainability and stability to review code before it's deployed.

<comment> Nick-Craver If we want a code review on anything risky, we may push a branch or we may just post the commit in chat for review before we build out. Which is chosen depends on how big or blocking the change may be.

We ask for code reviews all the time, we simply don't mandate them - I think that's the main difference.

<comment><USER3> > or we may just post the commit in chat for review before we build out.

Isn't that 'after the fact', considering your teamcity polls the gitlab repo a lot, so a commit will trigger a build right after it, and if everything goes well, deploy it too?

So you have to know up front whether a thing is 'risky', but that's a subjective term.

<comment> Nick-Craver It only deploys to our development/CI environment automatically. Deploying out to the production tier is a button press still.

So yes, it will build to dev, but we're using this in situations where we're _very_ confident the changes are correct already. I'd argue blind pushes are the problem otherwise. If the developer is not very certain: they can open a merge/pull request or just hop on a hangout to do a review.

<comment><USER3> > It only deploys to our development/CI environment automatically. Deploying > out to the production tier is a button press still.

Ah missed / overlooked that!

</comment></comment></comment></comment></comment></comment><comment><USER4> I was always wondering if there is some way of database deployment which does not suck. And all I see - every professional team ends with bunch of 074-add- column-to-table.sql files. I mean, code deployment can be organized much better. You can make graceful restarts, transactional updates, etc. Actually, almost nobody backups old code version because deployment of new one may fail, but database upgrades are so fragile and making backups is a must not only because you afraid of upgrade process may be interrupted and leave database in inconsistent state, but because properly done upgrade may result in corrupted state.

<comment><USER5> It's called NoSQL, which removes the need for schema migrations for things like adding or deleting columns.

This could be solved for relational databases if you implemented application- level abstractions that allowed you to store all your data using JSON storage, but create non-JSON views in order to query it in your application using traditional ORMs, etc.

So, store all data using these tables, which never have to be changed:

\- data_type

\- data (int type_id, int id, json data)

\- foreign_key_type (...)

\- foreign_keys (int type_id, int subject_id, int object_id)

(we'll ignore many-to-many for the moment)

And then at deploy time, gather the list of developer-facing tables and their columns from the developer-defined ORM subclasses, make a request to the application-level schema/view management abstraction to update the views to the latest version of the "schema", along the lines of [https://github.com/mwhite/JSONAlchemy](https://github.com/mwhite/JSONAlchemy).

With the foreign key table, performance would suffer, but probably not enough to matter for most use cases.

For non-trivial migrations where you have to actually move data around, I can't see why these should ever be done at deploy time. You should write your application to be able to work with the both the old and new version of the schema, and have the application do the migration on demand as each piece of data is accessed. If you need to run the migration sooner, then run it all at once using a management application that's not connected to deploy -- with the migration for each row in a single transaction, eliminating downtime for migrating large tables.

I don't have that much experience with serious production database usage, so tell me if this there's something I'm missing, but I honestly think this could be really useful.

<comment> Nick-Craver > With the foreign key table, performance would suffer, but probably not > enough to matter for most use cases.

Citation needed :) That's going to _really_ depend.

I'm not for or against NoSQL (or any platform). Use what's best for you and your app!

 _In our case_ , NoSQL makes for a bad database approach. We do _many_ cross- sectional queries that cover many tables (or documents in that world). For example, a Post document doesn't make a ton of sense, we're looking at questions, answers, comments, users, and other bits across _many_ questions all the time. The same is true of users, showing their activity for things would be very, very complicated. In our case, we're simply very relational, so an RDBMS fits the bill best.

<comment><USER5> Sorry for being unclear. I'm not proposing NoSQL. I'm saying that many NoSQL users really mainly want NoDDL, which can be implemented on top of Postgres JSON storage while retaining SQL.

\- data (string type, int id, json fields)

\- fk (string type, int subj_id, int obj_id)

      select
        data.id,
        data.fields,
        fk_1.obj_id as 'foo_id'
        fk_2.obj_id as 'bar_id'
      from data
      join fk as fk_1 on data.id = fk_1.subj_id
      join fk as fk_2 on data.id = fk_2.subj_id
      where
        data.type = 'my_table'
        and fk_1.type = 'foo'
        and fk_2.type = 'bar'

What would the performance characteristics of that be versus if "foreign keys" are stored in the same table as the data, if fk has the optimal indexes?

</comment></comment></comment></comment><comment><USER6> Additional anecdata: At my place of employment, after the required code review, we must write an essay about the change and have at least two coworkers approve it. Then we must email the essay to a mailing list of several hundred people. One of those people is the designated external reviewer, who must sign off. However, lately, this final reviewer has been requesting 10+ modifications to the deployment essay due to a managerial decision to "improve deployments". Moreover, deployments are not allowed 1/4th of the year unless a vice president has signed off.

Any code change requires this process.

<comment><USER7> What industry sector? Medical? Defense?

<comment><USER6> Logistics

</comment></comment></comment><comment><USER8> This feels really clunky to me, but maybe I'm just not getting it. I'm trying to implement a more automated build/deploy process at my current place of employment and am basically modeling it off of Github's [0], which seems to have a better feel.

Obviously the quality of the process needs to be high, but when it's effortless and "fun" then everybody wins.

[0] [http://githubengineering.com/deploying-branches-to-github- co...](http://githubengineering.com/deploying-branches-to-github-com/)

</comment><comment><USER9>
       Fun fact: since Linux has no built-in DNS caching, most of the DNS queries are looking for…itself. Oh wait, that's not a fun fact — it's actually a pain in the ass.

Surely that should just be a very fast lookup in /etc/hosts?

<comment> Nick-Craver The problem here is that these services move - so if it's in /etc/hosts, our failover mechanisms (to a DR data center which has a replica server) are severely hindered. We're adding some local cache, but there are some nasty gotchas with subnet-local ordering on resolution. By this I mean, New York resolves the local /16 first, and Denver resolve's its local /16...instead BIND doesn't care (by default) and likes to auth against let's say: the London office. Good times!

<comment><USER10> but thats what DNS scope is for surely?

we had _n_ datacenters each named after their city: ldn.$company.com, ny.$company.com etc etc. in the DHCP we pushed out the search order so that it would try and resolve locally, if that failed try a level up until something worked.

This meant that you'd bind to _service_ it would first look up service.$location.$company.com, if thats not there it'd try and find service.$company.com

This cuts down the need for nasty split horizon DNS, moving VMs/services/machines between datacenters was simple and zero config.

If you were taking a service out of commission in one datacenter, you'd CNAME service.$location.$company.com to a different datacenter, do a staged kick of the machines, and BOOM failed over with only one config change.

On a side note, you can use SSSD or _shudder_ NSLCD to cache DNS.

<comment> Nick-Craver We do, but in the specific case of Active Directory, we _want_ to fail over and auth against another data center if the primary is offline. This means for our domain, the local (to the /16) domain controllers are returned first and then the others. The problem is BIND locally doesn't preserve this order and applications are suddenly authenticating across the planet.

DNS devolution isn't a good idea here, since the external domain is a wildcard. We'll be paying for that mistake from long ago until (if ever) we change the internal domain name.

This is a pretty recent problem we're just now getting to because the DNS volume has been a back-burner issue - we'll look into permanent solutions for all Linux services after the CDN testing completes. Recommendations on the Linux DNS caching are much appreciated - we'll review each. It's something that just hasn't been an issue in the past so not experts on that particular area. I am surprised caching hasn't landed natively in most of the major distros yet though.

<comment><USER10> Aha gotcha. I was under the impression that SSSD chose the fastest AD server it could find(either via the SRV records, or via a pre-determined list)? I've not had too much trouble with it stubbornly binding to the furthest away server. (thats with AD doing the DNS and delegation to BIND )

NSCD (name service caching daemon) is in RHEL and debian, so I assume it'll be in ubuntu as well. The problem is that it fights with SSSD if you're not careful. [https://access.redhat.com/documentation/en- US/Red_Hat_Enterp...](https://access.redhat.com/documentation/en- US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/usingnscd-sssd.html)

out of interest, what are you using to bind to AD?

</comment></comment></comment></comment></comment><comment><USER11> I think if you're deploying .net code you're almost certainly going to follow similar build architecture with TeamCity doing most of the grunt work. We have a very similar build structure but a bit more polished I think. Our TeamCity build agents build nuget packages, deploy them to Octopus, run unit and integration tests. Octopus handles promotions of code from dev to qa, to staging and all production environments. We also write migrations for database updates using FluentMigrator which works with just about any database driver. It's a joy deploying code on an environment like this.

<comment><USER12> Agreed on the Octopus bit. TeamCity + Octopus is practically magical. Until literally yesterday, I'd yet to find something that didn't work with minimal effort between the two.

</comment></comment><comment><USER13> > _Fun fact: since Linux has no built-in DNS caching, most of the DNS queries > are looking for … itself._

This is wrong in two ways, and isn't factual at all.

First, the cause of the queries is nothing to do with whether DNS query answers are cached locally or not. There is no causal link here. What causes such queries is applications that repeatedly look up the same things, over and over again; not the DNS server arrangements. One could argue that this is poor design in the applications, and that they should remember the results of lookups. But there's a good counterargument to make that this is _good_ design. Applications _shouldn 't_ all maintain their own private idiosyncratic DNS lookup result caches. History teaches us that applications attempting to cache their own DNS lookup results invariably do it poorly, for various reasons. (See Mozilla bug #162871, for one of several examples.) Good design is to hand that over to a common external subsystem shared by all applications.

Which brings us to the second way in which this is wrong. A common Unix tradition is for all machines to have local proxy DNS servers. Linux operating systems have _plenty_ of such server softwares that can be used: dnsmasq, pdnsd, PowerDNS, unbound ...

One of the simplest, which does _only_ caching and doesn't attempt to wear other hats simultaneously, is even named "dnscache". Set this up listening on the loopback interface, point its back-end at the relevant external servers, point the DNS client library at it, and -- voilà! -- a local caching proxy DNS server.

* [http://cr.yp.to/djbdns.html](http://cr.yp.to/djbdns.html)

* [http://cr.yp.to/djbdns/run-cache.html](http://cr.yp.to/djbdns/run-cache.html)

* [http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/djb...](http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/djbdns-problems.html#wrong-icann-root)

* [http://cr.yp.to/djbdns/install.html](http://cr.yp.to/djbdns/install.html)

I run server machines that have applications that repeatedly look up the same domain names again and again. Each runs a local dnscache instance, which ameliorates this very well.

<comment> Nick-Craver I don't believe your assessment is correct. I _very specifically_ said built- in. This remains true. If curious, we're on CentOS 7 specifically. I didn't say there aren't _any_ options, only that there aren't any built-in. What you described as alternatives are totally true, but they still aren't built-in. It's a manual/puppet/chef/etc. config everyone has to do.

As for the applications - we have little _direct_ input to TeamCity of Gitlab (the problem children here). And even if we did, I think we agree: the application level shouldn't cache anyway.

That being said, we're looking at `dnscache` as one of a few solutions here. But the point remains: _we have to do it_.

<comment><USER13> You are employing a faulty concept of what constitutes built-in. This is not Windows, or one of the BSDs. You're using one of the Linux distributions where _everything_ is made up of installing packages. There is no meaningful "built- in"/"not-built-in" difference between installing one of these DNS server packages from a CentOS 7 repository and installing any other CentOS packages from a CentOS 7 repository.

> All software _on a Red Hat Enterprise Linux system is divided into RPM > packages which can be installed, upgraded, or removed._

\-- [https://www.centos.org/docs/5/html/Deployment_Guide-en- US/pt...](https://www.centos.org/docs/5/html/Deployment_Guide-en-US/pt-pkg- management.html)

A (very) quick check indicates that the CentOS 7 "main" and "updates" repositories have at least three of the DNS softwares that I mentioned. Ubuntu 16 is better endowed, and has all of them that I mentioned, and an additional "Debian fork of djbdns" that I did not, in Ubuntu's "main" and "universe" repositories.

</comment></comment></comment><comment><USER14> It's funny to see that stackoverflow came to exactly the same solution for database migrations on the Microsoft stack as my team did, even down to the test procedure.

Simple, safe and very effective :)

<comment><USER15> its also pretty much the same solution colleagues and I came up with a few years ago for a migration tool we were working on. It's kind of abandon-ware at this point, but this version is pretty far along: [https://github.com/jdc0589/mite-node](https://github.com/jdc0589/mite-node)

</comment></comment><comment><USER16> > If Gitlab pricing keeps getting crazier (note: I said "crazier", not "more > expensive"), we'll certainly re-evaluate GitHub Enterprise again.

Shots fired :P

</comment><comment><USER17> When did everybody decide that chatbots were the new hotness for deployments?

<comment> Nick-Craver If you mean pinbot - that's literally _all_ it does. It takes a message and pins it, knocking the old one off the pins.

The build messages build...that's also literally all it does. It simply puts handy notices in the chatroom. Why _wouldn 't_ you want that integration? Everyone going to look at the build screen and polling it to see what's up is a far less efficient system. A push style notification, no matter the medium, causes far less overhead.

I doubt we'll ever build from chat directly for anything production at least, simply because those are 2 different user and authentication systems in play. It's too risky, IMO.

</comment></comment><comment><USER9>
       A developer is new, and early on we want code reviews
       A developer is working on a big (or risky) feature, and wants a one-off code review

This implies you don't normally do code review??

<comment><USER18> Code review is overrated. Over my career I've built 8/9 figure revenue platforms without code review or serious test setups.

A lot of this cruft is unnecessary when compared to good domain knowledge and solid coding focus.

<comment><USER8> > Over my career I've built 8/9 figure revenue platforms without code review > or serious test setups.

This does not diminish code reviews. Would you be a better engineer today if you had regularly participated in code reviews? Would your coworkers?

<comment><USER19> One of the problems with code reviews is they're often done really poorly, for a number of reasons:

1\. Reviewers are often poorly trained to provide good design reviews and default to nit-picky stuff a code linter should pickup. Human linting is just a poor use of time and money.

2\. Nobody seems to ever have time for them to deep dive into the code.

3\. Few engineers seem to ever actually want to do them.

4\. Reviews can become hostile.

Code reviews are probably really important in some fields, for example, medical equipment, aviation, etc, but for the vast number of projects where we're shoveling A bits to B bucket or transforming C bits into D bits it's overkill and companies would be better off investing the massive amount of wasted time in better CI/CD infrastructure.

</comment></comment></comment></comment><comment><USER20> SE/SO is for it's size an amazingly high proformance team. It's rare to hear them say it, in fact, I don't recall ever hearing them say this.

The importantance of the cohesion and trust amoung their team is critical to their deployments. In fact, I would say it's vital to how they're able to get away with minimal amounts of code reviews for example.

It's dangerous to believe this is easy or reproducible. New teams needs extensive controls in place to make sure the quality of the deployments will not negatively impact the group.

</comment><comment><USER21> A few things strike me as odd/sub-optimal:

        - migration id's freeform in chat -> why not central db with an auto-increment column?
        - Use chat to prevent 'migration collisions' -> Same central db/msmq/whatever to report start/stop of migrations and lock based on that...

<comment><USER22> I guess this is because using a team chat is 'good enough' for them without adding another layer of tooling

</comment></comment><comment><USER23> The article says:

> there is a slight chance that someone will get new static content with an > old hash (if they hit a CDN miss for a piece of content that actually > changed this build)

Anyone has a solution to this problem?

<comment><USER24> Push new static content first to all servers/cdn, _then_ bust the CDN cache / push a cache bust

<comment><USER25> This doesn't resolve the issue. The fundamental issue here is that two versions are running in parallel.

* If you push static content and web pages together, you get V1 and V2 of _both_ static and web, and you end up with incorrect static resources served in both directions. This approach is only reasonable if your deployment strategy is to take a service outage to upgrade all machines together.

* If you push web first, you get the ugly scenario described in the article where V1 resources get served with V2 hashes and cached for 7 days.

* If you push static content first, you still have V2 static content being served for V1 web pages. The "cache bust" doesn't matter. Somewhere a cache will expire and someone will get V2 static resources for a V1 page.

You have to deal with the two versions somehow if you want to resolve the issue fully.

</comment></comment></comment><comment><USER26> Stack Overflow uses windows? Any particular reason to do this?

<comment><USER27> The founders and first devs were proficient in it.

It works well enough for our needs (e.g. C# has one of the best GCs on the market), and no one is a platform/language zealot, so we keep on using it.

Stuff we added later runs on other platforms, as needed (e.g. we run redis and elastic search on CentOS, our server monitoring tool Bosun is written in go...)

</comment></comment><comment><USER28> I was surprised they weren't using Kiln [[https://www.fogcreek.com/kiln/](https://www.fogcreek.com/kiln/)], a Fog Creek product. I know SO is independent from Fog Creek now, but still a bit surprised at it. I wonder if there was a migration off at some point.

<comment> Nick-Craver Yep...for the on-premise reasons listed in the article. Once upon a time a lot of projects were on Mercurial, hosted by Kiln. The Stack Overflow repo specifically has always been on an internal Mercurial and then Git server. Originally this was for speed, now it's for speed and reliability/dependency reduction.

</comment></comment><comment><USER29> Did you choose polling over webhooks for a reason? Or was webhooks recently added as a feature to Gitlab?

<comment> Nick-Craver Webhooks didn't _used to_ work well for many builds off a single repo, but I think this changed very recently in TeamCity. Thanks for the reminder - I'll take another look this week at _adding_ web hooks. We'd still want the poll in case of any hook failures.

At the moment, Gitlab knows nothing about our builds - and we'd want to keep it simple in that regard. If we can generically configure a hook to hit TeamCity to alert of _any_ repo updates though, that's tractable...I need to see if that's possible now.

</comment></comment><comment><USER30> TeamCity is a great CI system.

</comment><comment><USER31> What's with all the backslashes?

<comment> Nick-Craver Yeah! What's wi...wait, what?

<comment><USER13> Enjoy [https://news.ycombinator.com/item?id=11433139](https://news.ycombinator.com/item?id=11433139) .</comment></comment></comment>


Algorithm That Tells the Boss Who Might Quit - <AUTHOR> http://www.wsj.com/articles/the-algorithm-that-tells-the-boss-who-might-quit-1426287935

<comment><USER2> Sadly, this has the perverse effect of forcing otherwise happy people to _act_ like they might be a flight risk, otherwise they risk being passed-over for interesting opportunities, raises etc. If you're not playing that game then you're losing out.

It's yet another mechanism that devalues loyalty to a firm, which I'm sure they realise is already in short supply. I'd be happier if I'd read that this was also a way to identify and _reward_ the dedicated, productive employees — but no.

</comment><comment><AUTHOR> The algorithm seems to run on the following factors per the article:

"job tenure, geography, performance reviews, employee surveys, communication patterns and even personality tests to identify flight risks".

I wonder if employees can try to game the algorithm by artificially ranking high on their score for leaving - with the sole purpose being to get a pay raise.</comment>


Paul Rand - Failure by Design - <AUTHOR> http://www.paul-rand.com/index.php/site/thoughts_failureByDesign/

<comment><USER2> Originally published in 1993. While I think the value of good design is still discounted, it's being discounted to a much smaller degree (1) post-Apple and given (2) how much more critical good UI/design is to the web/software.</comment>


How I got a job from a Hacker News post - <AUTHOR> http://bostinno.com/channels/a-little-bit-of-luck-and-a-whole-lot-of-backend-as-a-service/

<comment><USER2> This is Networking 2.0. Good on you, but hypothetically if you hadn't gotten "lucky" by having the predictable consequences of directed effort come to pass, you (and similarly situated geeks) still had options.

1) What if big names in your space hadn't retweeted you? Well, since you know who the big names in the space are, you take an hour or two to lookup their email addresses. Then, you send a one-paragraph email:

Hiya Bob,

My name is $NAME and I really like your work on backends as a service at $COMPANY. In particular, your post on the blog last year about $TOPIC was really insightful. I was able to apply a few ideas from it to my work.

I just wrote a post on BaaS myself. $LINK

I'd appreciate any thoughts you had on it.

Regards,

Anyone who writes you back is _no longer a stranger_. Instead, they think you're intelligent about a field where they currently cannot hire for love or money. You can then followup with "As it happens, I'm looking for opportunities to put further my interest in BaaS professionally. Do you happen to know anyone who is hiring?"

2) What if he wasn't really close for coffee?

Planes, we've got them. I am totally serious. A day off and a few hundred bucks versus a career upgrade, which has the higher NPV? "I will be in Boston for one day only on XX/YY (or, alternately, week of XX/YY). Would you let me buy you breakfast to talk about this?" is a very compelling offer psychologically. Your time is scarce, and hence valuable, plus you've already demonstrated three things they'll get out of saying "Yes." (Food, interesting conversation, and "the ability to get to know someone whose comportment suggests that _he is going places_.") Note that that is the only decision they need to make - after you have yes, then you can schedule the details.

3). What if he wasn't hiring?

He knows someone who is. Get warm intro.

<comment><USER3> Patrick, this is great for our field where there are tons of influential folks blogging and/or tweeting. But what if you don't work in IT industry?

My very dear friend works as an Internal Auditor. She earlier worked for KPMG and now she is working for a Fortune 50 company. She is looking for other opportunities as she feels that she is not growing in her current position. Now, there aren't that many Internal Audit bloggers and certainly no hiring manager is blogging/tweeting. What approach should she take to get noticed by the right people?

<comment><USER2> There's _so many_ possible answers to this question but the easiest possible one is to go to a conference in the industry, on your company's nickel if possible (trivial), and start making friends. Or you can use existing contacts in your Fortune 50 network and get warm intros to the right sort of people at peer firms, clients, vendors, etc. Or, since I'm guessing one does not become an Internal Auditor the first day out of high school, you work the alumni network at your college.

Also, even if nobody in your industry blogs, doesn't mean you can't start to generate a portfolio for yourself. How many blogs have you ever read by Japanese salarymen who do Enterprise Java development? Can you think of any which had an article or two that suggest maybe the author is smart enough to take out for coffee? (You can _totally_ do this without bringing down the wrath of God on yourself for spilling company secrets. Just talk in very general terms about challenges in Internal Auditing and how one could overcome those, optionally with heavily anonymized anecdotes.)

<comment><USER4> Almost hate to say it, but this is where a place like LinkedIn can really shine. Almost any profession has one or more interest groups on there where you can share stories with folks of a similar interest. You do have to be careful as more popular professions have their share of groups that are little more than recruiters and desperate job hunters in a mortal spiral. But reading for a while before joining can usually help you steer clear of those.

<comment><USER5> Agreed. If you don't have the network to find a job, create it. Cold calling/emails is a pretty inefficient way to do it, but it does work. Figure out what kind of job you want, find people that have that job and start talking to them. The most important thing is DON'T ASK THEM FOR A JOB. Just ask them how they got there and what advice they have for you. I've gotten more job offers that way than any other method.

</comment></comment></comment></comment></comment><comment><USER6> "Opportunity is missed by most people because it is dressed in overalls and looks like work." Edison was kind of a jerk in a lot of ways, but he was spot on with this.

<comment><USER7> A far better quote than the one I used in the article - thanks for this one, I'll be hanging on it!

</comment></comment><comment><USER8> I too got my current job from a HN post. I posted about Bay Area salaries as I was flying out here for an interview. I included some of my background information and got a reply saying if I wasn't gonna be making X, they would love to meet and talk while I was in town. Later that night I had an offer. I love Hacker News.

</comment><comment><USER9> "Given enough exposure, the impossibly lucky becomes the inevitable."

Nice story, but I don't know if I would go that far. You don't have an unbiased sample, so you don't know how many do exactly the same thing and nothing comes of it. It's the classic "what do successful people have in common" fallacy. An equally justified conclusion, based on "the hundred ways I failed leading up to this moment" might be "the chances of success are about 1%".

<comment><USER10> The "what do successful people have in common" fallacy is more commonly referred to as survivor bias.

<http://en.wikipedia.org/wiki/Survivorship_bias>

<comment><USER7> I agree that survivorship bias is something to watch out for. Stories that tend to do well on HN are the ones that end well.

However, I don't think that should hold back someone who is in the same position I was (a longtime lurker trying to break out). For me, a primary takeaway from this experience was the small downside compared to the large upside of each of my decisions.

Worst case: your post goes nowhere on HN, the person you reach out to ignores you, or your conversation turns up nothing fruitful.

Best Case: your post pins it on HN, the CEO agrees to meet, and it turns into a job offer.

For 30 minutes of effort - that calculation is a no brainer to me. It just takes getting over that lurker inertia, and a lot of patience for failure!

</comment></comment></comment><comment><USER11> My last full time job I got by leaving a comment on Fred Wilson's blog. He shared my info with a bunch of investors in my area and they connected me with people in their portfolio who were hiring. About 3 weeks from comment to my first day on the job.

<comment><USER12> can you tell more about it? sounds like a story.

</comment></comment><comment><USER13> After reading a monthly job post a while back, I had a couple of phone interviews with factual.com. While I didn't get that job, I can tell you that Ron, their recruiter, is still the nicest and most down to Earth guy I've met in my job hunt.

A month or two later, I actually got a cold email from someone who apparently really liked the way I replied in a thread. (I never found out which.) Sadly it didn't happen either; the person assumed that I could program simply because I was here. I went to school for it years ago, but can not with any proficiency whatsoever.

For all the talk that occassionally pops up about community quality, as I see it, this place is doing all right. Better than the profession my experience resides in anyway; journalism. Then again, I'm the genius that didn't see visiting 4chan as a big deal, in search of a direction to find an irc for Anonymous talk. Be warned, your boss may think differently!

(Yes, that last sentence was slightly tongue in cheek. I realize most would take theobscene material seriously rather than as childish outlashes. But I didn't think a journalist would.)

</comment><comment><USER14> Congratulation this is a great bored for people to discover each other and cool projects. I actually learned about another provider in that post who supports a technology stack that I have been looking for a BaaS provider that supports it. The company you got the job from was contributing to it too and seemed like a great bunch of guys, I would have definitely used their product if it supported what I am trying to do. I love reading the win-win storied on HN.

</comment><comment><USER15> I got my job from a reddit post I made (a year ago), and we were acquired two days ago ! high five from a fellow 'employee from the internet' :)

</comment><comment><USER16> I was at one point skeptical that frequent activity on the Internet would ever amount to anything worthwhile. I was offered an informal interview for my current job through a follower I occasionally talked to on Twitter.

I knew about the opening a few weeks before. What surprises some is that if I wasn't asked to interview I probably would have never applied for the role.

</comment><comment><USER17> Would you like to meet up? I will buy you some breakfast.

</comment><comment><USER18> sounds like an interesting article, wish i could have read it, but the floating social toolbar on the left side covered a quarter of the article on my phone.</comment>


Great breakdown on what's wrong with most iPad magazines - <AUTHOR> http://www.justatheory.com/computers/apps/conde-nast-ipad.html

<comment><USER2> The New Yorker digital assets are a completely technically suboptimal solution. Obviously in life other non-technical issues can take precedence.

The New Yorker online edition (archives.newyorker.com) consists of a javascript interface that displays 760x556 jpegs for browsing. Now we get to the good part: if you click on a page to zoom, it brings up a 1228x900 png overlaid on a 1228x900 jpg. The png is the high frequency information (abrupt black and white transitions) and the jpg is the low frequency information (large swathes of colour). I'm confident the data is processed by DjVu to perform this image separation (if you dive the image urls 'djvu' is one of the directory names). The hilarious part is that they OCR their images and this text is in the html. Even more fun is that there is a separate 1613x1181 jpg when you print from the online edition but as we know jpg is totally inappropriate for black and white text.

So to summarize the data is already passed through a DjVu toolchain to generate a djvu file but they render all of these ridiculous image sizes in various formats to fit their terrible viewer.

I don't have an iPad so I can't confirm but I'm pretty sure the Kindle New Yorker app runs on the same Adobe AIR platform. For the Kindle, each issue clocks in at around 100 MB and consists of a 1024x600 png for each page (I assume the iPad images are 1024x768) and a whole bunch of other HTML/XML assets. I'm running this on an android phone and the issues are stored under "/mnt/sdcard/The New Yorker/.library/The New Yorker/" so the data structure is pretty transparent.

Harper's is much more sane: they serve gifs or pdfs. From the pdf properties it's obvious it's direct output from QuarkExpress.

</comment><comment><USER3> The Economist has the best iPad experience of any magazine I've checked out to date. A great browsing experience, vector text, etc. Apparently it was done by an Australian development house; I wish more magazines/newspapers would do something this good.

</comment><comment><USER4> That's a great technical analysis, but I think it misses the primary business reason why Conde went with images for most magazine apps - it's cheaper and faster for them to create content during each 4-week release cycle.

Oddly enough, I know a lot of people (myself and roomate included) who have been specifically waiting for an iPad version of Vogue to see Conde's implementation decisions.

<comment><USER5> > it's cheaper and faster for them to create [image] content during each > 4-week release cycle.

Would you elaborate on how it is cheaper and faster for them to create content, esp. text content, by going the image route rather than simply keeping the text as text (given that the text has to start out as text in order to be rendered into any image format in the first place). I simply do not see any reasons how performing a text->image transformation can be cheaper and faster than simply keeping the text as text.

<comment><USER6> I work with a children's book publisher that faces a lot of the same challenges. They're a much smaller company in a completely different market, but their roots are providing printed product to customers.

It's easy to take for granted everything leading up to the final product. For publishers who create physical books, electronic publishing is a challenge, largely because their talent doesn't necessarily have _any_ skills in this area. Any freelancer web designer who has worked with a print designer knows what I'm talking about. For the people who live in the print world, rendering the page to a giant image is analogous to what they do with their print books. The destination of the printing process is, after all, a rasterization of the layout on paper. They see the final product, and everything beyond that is a black box to them.

So, everything a print publisher does leads to one final resting place: rasterization on paper. When you decide to make the jump to an electronic platform, the point in the process you diverge dictates the expense. The choice, from the publisher's perspective, is to:

A) Hand off their layout to this tool that Adobe provides and be done with it

B) Create some custom toolchain and process that converts the content to a better format

Keep in mind that option B is completely opaque to many of them. The commercial printing process has been around a long, long time, and for many of the managers that work in the publishing industry, the process is as intuitive as riding a bike. They don't even think about it. This means that choosing option B is more than just changing what they do, it's changing _who_ does it. This means hiring more help, which is expensive.

Then there's the distribution problem. Many publishers don't handle their own distribution, and publishing each title as an individual app isn't a desirable option either. Choosing option A, or something like Zinio (also image based), means not solving "yet another hard problem".

Many publishers are in survival mode. The publisher I work with is fortunate to be so small. They had some early success with web-based products that returned huge margins, so the management decided that putting resources there made sense. Their staff have attacked the electronic-publishing learning curve and are coming around. I suspect that for many large publishers, the shift in direction won't happen. Their momentum is too great, it may ultimately lead to their demise.</comment></comment></comment>


World Bank Open Data - <AUTHOR> http://data.worldbank.org/

<comment><USER2> 50 upvotes and no comment so here is my "I hope someone will make nice analysis on this data, might be very interesting, at least nice to look at" comment.

</comment><comment><USER3> A subset of the World Bank development indicators is available in Google's public data explorer here: <http://www.google.com/publicdata/directory></comment>


In Defense of Copy and Paste - <AUTHOR> http://zacharyvoase.com/2013/02/08/copypasta/ <comment><USER5> I love copy & paste! I also defended it in this scholarly article: [http://harmonia.cs.berkeley.edu/papers/toomim-linked- editing...](http://harmonia.cs.berkeley.edu/papers/toomim-linked-editing.pdf) with a video: <http://youtu.be/1wo_7MTdWWI>

<comment><USER6> No idea why you were downvoted. That paper looks like it has far more thoughtful things to say than most comments here. :-)

<comment><USER7> I didn't downvote it, but I was just going to skip that comment and continue reading.

The comment is too short, apparently slightly off-topic, looks like self promotion and give no reason to look to the linked content. But as you said, this is a good comment!

How I would rewrite this comment (with some parts stolen from the abstract and some parts just invented):

 _I love copy & paste! I think that the use of programming abstractions like functions and macros has inherent cognitive costs. A few years ago, as part of my research we proposed "Linked Editing": A technique for managing duplicated source code with the help of the text editor. We implemented it in a prototype editor as a XEmacs extension. More details in this article: [http://harmonia.cs.berkeley.edu/papers/toomim-linked- editing...](http://harmonia.cs.berkeley.edu/papers/toomim-linked-editing.pdf) and we made a video of the editor: <http://youtu.be/1wo_7MTdWWI> ._

P.S.1 I still prefer refactoring to this kind of multiple edition. I use Racket, so I love functions and macros. But sometime they are very complex to edit, so perhaps sometimes this multiple edition can be a good idea.

P.S.2 Another possibility is that some users are too eager to downvote. I saw many good comments downvoted, but usually they bounce later.

</comment></comment></comment><comment><USER8> _> This may come across as a straw man argument_

Big time. The refactoring in this case was ill advised. When things started getting hairy, it should've been backed out.

Piling too much flexibility in one function is a common mistake. A justification for copy/paste it does not make.

I worked at a shop with this rule: don't try DRY until you've seen at least three repetitions. I think this saves one from premature refactoring.

Another way to put it: Refactor when the code speaks to you, that is when need is evident. Keep the result only if its a significant improvement. Avoid refactoring only because you are enamored of refactoring. (Or enamored of a rule.) Goes for any programming technique/tool, really.

<comment><USER9> > don't try DRY until you've seen at least three repetitions

A significant number of bugs I've had to fix over the years were the result of just two repetitions. Invariably someone updates one of them and not the other one. Or copies the first one, then updates, then forgets.

This happens over and over and over and over. I'm so sick of it that I want to punch my coworkers (I don't usually get angry quickly but this has been wearing on me for a few years now). So I will continue to refactor at the second copy.

<comment><USER10> As Djikstra said, "two or more, use a for."

<comment><USER8> That's not entirely relevant in this context. We're talking about two similar bits of code in entirely different parts of the codebase.

</comment></comment></comment></comment><comment><USER11> That example under the _When Tools Make It Worse_ section - uggghhhhh. Why would anyone actually _do_ that? That isn't DRY refactoring, that's cargo cult refactoring.

DRY is not, was never, and should never be about unnecessarily replacing clean, well-factored code with @$2!% shared mutable state. The goal is to normalize your code, not to micro-optimize for keystroke count. No. Nonononononono. Just no.

<comment><USER12> Oh man, I've been fighting a similar battle this month. I have to deal with a codebase which has to talk to a handful of proprietary protocols, which don't have the best interoperability between implementations. However the all the protocols look fairly similar at a macro level (general concepts, data structure layout, etc). So somehow this mutated into a terrible inheritance tree 3-9 classes deep, with absurd placement of special cases 4 classes upstream, or special case handling for a specific device in each class all the way up the inheritance stack, and worse.

So I am trying to convince the cargo cult inheritance people that there are patterns like strategy and "library of useful functions" to handle a lot of the code better. But they keep sticking on cargo cult DRY. No matter how many times I explain to them that we need to organize with repeated code, because as the future happens, and new versions of "interoperable" code will diverge in their special case handling and whatnot, we still can't repeat ourselves.

/rant

So yeah... Cargo-cult DRY is just as bad for readability and maintainability as spaghetti and big balls of mud.

<comment><USER11> Sometimes I really wish there were a way to make violating the Liskov Substitution Principle throw a compiler error.

<comment><USER13> Learn Haskell.

</comment></comment></comment></comment><comment><USER14> Knowing when to refactor is obviously an art, not a science. And with an example as trivial as this, it's not really a very "real-world" example.

A lot of the time, you don't even _know_ if two swaths of code are "coincidentally" identical (don't refactor) or identical in a "deep" way (refactor), even when the program is yours -- you just don't know how the program will evolve.

In the absence of additional information, I usually refactor only when I see _three_ similar code paths, since by that point a project rarely goes back. Over the years, it's turned out to be a surprisingly good rule of thumb.

<comment><USER15> Same here. Doing the same thing three times is always my rule of thumb for when to take a look at refactoring, automating, etc.

</comment></comment><comment><USER16> As I ponder this more, I think it's useful to consider the concepts of simplicity and complecting as articulated by Rich Hickey in his talk "Simple Made Easy". As he explains it, to complect is to braid multiple things together, whereas in a simple system, multiple things are composed. He has often pointed out that simplicity does not necessarily mean fewer things; as I understand it, it's not about how many things they are, but how they interact.

In that light, the single flexible tweet list function presented in this post is indeed problematic because it has a few things braided together: a tweet list, a profanity filter, and pagination.

So we should be suspicious of repetition, but at the same time avoid complecting.

</comment><comment><USER6> One rule I try to follow is to avoid refactoring when the shared code is "coincidental." Perhaps this is another way of expressing what the author says about business logic.

I've definitely worked on projects where developers created large, unwieldy, hard-to-grok, buggy abstractions in the name of DRYing code. I'm pretty aggressive about making code DRY, but simplicity and readability are more important.

The effort I'll tolerate in pursuit of DRY also varies by language. I've been doing some Android work lately, and I'm finding that things I would have done DRY in Ruby require too much added complexity to make DRY in Java.

<comment><USER17> I think that's exactly what the author tries to say. I've also joked expressing it as "you don't refactor your twins."

I'm curious (and I think it can be a useful sub-topic) what you think made Java worse for DRYing. The rigid type system? Added verbosity?

<comment><USER6> One easy example: no higher-order functions like `map`, `select`, and `inject` because of no lambdas, so I wind up repeating the same looping code all over the place.

I took a look at a few "functional programming in Java" libraries, but their solutions were still pretty verbose, and it didn't seem worth adding a dependency for a tiny smartphone app.

<comment><USER6> EDIT: After thinking about it some more, there are several places where two methods are almost the same except for one line stuck in the middle. I don't know how to turn them inside-out in Java without spawning a bunch of tiny classes, and it's just not worth it. But with Ruby I could turn them into a single method, throw a yield in the middle, and then call it with a block. So again it comes down to no lambdas.

(Hmm, this was supposed to be an edit, but somehow I did a reply instead.)

</comment></comment></comment></comment><comment><USER18> Straw man thinks DRY applies to two-line function. Straw man is a straw man. Also, less code > DRY. In fact, less code -> DRY. If refactoring makes for more code, not really DRY. More like taking a principle to its illogical conclusion. Compression is a process of diminishing returns.

Although there are certainly times when a factoring two lines into one line is better. Like when it's self-documenting, or when those lines otherwise add noise to part of another function.

Sometimes a new function is not the right approach to avoiding repetition. If you can't write a function to adhere to DRY, use a macro or equivalent. In C/etc, macros are wonderful if used well.

<comment><USER19> I feel like people in this thread (not just you) are particularly caught up with an oversimplified example that the author even acknowledged may not have been the best example within the article itself.

This article sheds light on something I also encountered frequently when I was doing contracting, and also have to put the brakes on myself when I see I'm going down a bad road: creating more generalized code is not always better than creating code that repeats trivial pieces of functionality but accomplishes distinct tasks.

Part of the difference between "conscious competence" and "unconscious competence" is innate awareness of places where refactoring or normalization will actually create technical debt. I found myself thinking "no duh" when I read the article, but that's only because it was explaining things I was unconsciously very familiar with.

I think this article would be a great read for less experienced programmers. I think the examples may have been lacking, but it would be hard to simplify any application to a point that would make sense to illustrate this issue in a blog post, so attacking it as a "straw man" is actually a "straw man" in and of itself if you fail to account for the author's intended purpose by including the examples. lol

<comment><USER20> I don't think the article serves any purpose. It uses an obvious strawman to try and argue; this is not going to convince anyone of anything, novice programmer or no.

</comment></comment></comment><comment><USER21> One of the things that people don't get about refactoring is that it is not just a matter of extracting things or removing duplication. Sometimes you merge things or re-introduce duplication to get someplace better.

When you look at refactoring examples online, they often make that mistake. There's a straight arrow toward a "better solution" but without any backtracking. It's a hobbled view of refactoring.

To bring it home, in the blog example, I think is perfectly fine to remove duplication in the way listed as "bad", as long as you reintroduce the duplication when you have a bit of trouble. Much of the time, you're lucky and you don't.

</comment><comment><USER22> I really really like this take. Refactoring is usually pitched as something that is completely orthogonal to solving the actual problem you were given. I think too many of us (clearly, I'm projecting) are weary of anyone else going on a refactoring spree because we see it break down things that were just fine separate. Often with only "warm fuzzies" being the actual gain. The progression shown in this post is really really good.

<comment><USER23> Agreed. This is -- or should be -- called "premature refactoring."

<comment><USER19> I don't even think it's always premature refactoring. Sometimes it is just stupid refactoring.

A correlating result to stupid refactoring is the existence of over- generalized functions that try to do so much that they need an absolute crap- ton of parameters passed in and still end up locking you down to a limited set of functionality. Adobe's ColdFusion scripting language (anyone remember that) used to have functions that would automatically generate huge and specific pieces of client-side JavaScript functionality. Stuff to the effect of:

      cfCreateShoppingCartWithPopupSummaryWhenUserHoversOverLink({
        supportsPaypal: true,
        dontShowLinkOnCategoryPage: true,
        doShowLinkOnProducePage: false, ...},
        'myShoppingCartElement',
        ...)

Okay maybe I'm embellishing a little bit. But the end result was loading hundreds of kilobytes of proprietary JavaScript libraries to support these weird built-in functions that would create very specific bits of client-side functionality that would then need a billion parameters passed in to allow remotely useful customization. Maybe it would have been better to just learn JavaScript instead of being locked in this way.

</comment></comment></comment><comment><USER24> I think the first step the author took on the refactoring path was one I wouldn't take. It breaks the "do one thing" rule and the rest of the post is the pain that naturally follows from having an over-generalized method that tries to do too much.

</comment><comment><USER25> These articles are a dime a dozen. This popular philosophy isn't right, because look at my poorly coded example of it.

You're dry code, is only dry is the laziest of senses, and represents a lousy programmer cluttering the system. A really lousy implementation of any of these programming paradigms would make one side look wrong.

In your example, the refactored code would look excellent if it implemented OOP and the Strategy Pattern. The two different feeds can inherit their similarities from the same place, and their differences implemented in separate places. Which feed to produce can be chosen dynamically, rather than one crappy grab-all function.

</comment><comment><USER26> Not directly related - but it's something I come across so often when mentoring newbie devs that I thought I'd mention it in passing just in case anybody has this problem.

A pattern I sometimes see with newbies who understand the value of DRY is - <USER2> soon as they get to the point when they're about to repeat something or about to copy and paste - they stop themselves and start refactoring to remove the duplication they haven't typed into existence yet. They see adding the code that will produce the duplication as bad / waste.

Don't do that.

It's hard - because the code that they've not typed or copy/pasted doesn't exist or work yet. It's still in their head.

Make the duplication explicit first.

Type it out. Copy and paste. Change those two branches so they have exactly the same structure.

When you've done that - and everything is working and all tests pass - <USER3> refactor the heck out of it.

Much simpler, faster and less error prone.

<comment><USER27> I don't know if I agree 100%. Sometimes the extra abstraction helps you solve the problem in the first place.

In my experience the OP is right. The worse problem is when you eliminate duplications in the wrong place or using the wrong abstraction, leading to brittle abstractions that break in the future.

<comment><USER26> Yeah - there are probably exceptions ;-)

However what I've seen happen on multiple occasions is somebody merrily driving along churning out code then suddenly hitting the "ohhh - <USER4> is bad" wall and halting as they feel their way around the duplication and abstract they may need (or may not - since they've not written the code yet).

Duplication is not a mortal sin. Having it sit there for a few hours while you work out the meat of the problem isn't going to kill any kittens.

And often the easiest way to fully grok the abstraction that you need is to make the duplication really, really obvious.

</comment></comment></comment><comment><USER28> I expect most of us would agree that a single function should ideally have exactly one main job and do it well.

Two functions are really doing the same job, and should probably therefore be combined into a single function, not when their behaviour _is_ the same but when it _should always be_ the same. As the article suggests, that determination is generally more about the software design or domain model than the mechanics of the current implementations.

Having said that, there is also a middle ground: create some sort of utility/helper function(s) to contain the code that _is_ the same, coincidentally or otherwise, and then rewrite the two higher-level functions in terms of common helpers for now. If those higher-level functions need to diverge for good reasons later, at least it will be an active decision to separate the behaviours.

IME that sort of breakdown is unlikely to be beneficial with very short functions such as the examples here. There's not enough commonality to justify the overheads of breaking everything up. However, in more realistic code, if you've got, say, 80% common operations between multiple cases, there are often some underlying concepts that can be extracted into their own functions. Those then become informatively named building blocks for the original functions.

Put another way, you might not want to consolidate the functions' _interfaces_ if they serve logically distinct purposes, but you can still consolidate some of their _implementation_ details.

</comment><comment><USER29> I see a lot of comments here of the type "You have to know when to refactor". I don't do it this way. Instead, I rely on a willingness to undo a refactoring when I see that something else might work better -- and even to undo _that_ when I decide that I've got that wrong.

I have no problem extracting as in "WHEN REFACTORING GOES BAD" -- although I might wait for a third copy because removing the duplication -- because I want to see whether a useful abstraction would emerge. On the other hand, as soon as I recognise that one of those copies wants to change in a way that the other does not, I'd simply inline the method and let them diverge. I don't consider this a problem.

It seems as though some programmers believe that, one they extract something, it needs to remain extracted. No. It's only "cargo cult refactoring" if you stop thinking.

Most importantly, refactoring is experimentation. It's a kind of Mechanical Turk-based genetic programming-oriented style of designing, except that you have heuristics you can follow. That means that you'll go down the wrong path. THAT'S OK! as long as you allow yourself to backtrack. Remember: refactorings are small, _reversible_ design changes. That means not just that one _can_ undo them, but that one is _willing_ to undo them.

</comment><comment><USER30> OK, I'm obviously missing something, and part of the problem is that I'm not a Python programmer so my brain is obviously in "skim-mode".

Couldn't the problematic DRY pattern be alleviated by refactoring the following call:

        filter_profanity = kwargs.pop('filter_profanity')
        tweets = Tweet.objects.filter(**kwargs)
        if filter_profanity:
            tweets = itertools.ifilter(lambda t: not t.is_profane(), tweets)
        return render(request, template, {'tweets': tweets})

Into something like:

        def tweet_list(request, **kwargs)
           ...

           tweets = get_filtered_tweets(kwargs)
           ...

        def get_filtered_tweets(**args)
           filter_profanity = args.pop('filter_profanity')
           if filter_profanity
              etc....
            end
           return tweets
        end

Why does the logic for the Tweet filtering have to be encapsulated in the rendering function?

// edit:

What might help is if the OP showed how the non-refactored code would look with the profanity_filter and pagination features. I agree that his refactored proposal is confusing...I'm just having a hard time imagining how the non- refactored version would be less so.

<comment><USER31> I believe they intended the 'proper' code to look like:

        def global_feed(request):
            tweets = Tweet.objects.all()
            tweets = itertools.ifilter(lambda t: not t.is_profane(), tweets)
            return render(request, 'global_feed.html', {'tweets': tweets})

        tweets_per_page = 20
        def user_timeline(request, username):
            tweets = Tweet.objects.filter(user__username=username)
            page = request.GET.get('page', 1)
            offset = (page - 1) * tweets_per_page
            tweets = tweets[offset:offset + tweets_per_page]
            return render(request, 'user_timeline.html', {'tweets': tweets})

<comment><USER30> So the problem with that is seems to be...what if there was another view that required pagination?

So the two unpleasant scenarios seem to be this:

1) The OP's assertion that DRYing the code may unintentionally break functionality in all the places that use it.

2) The DRY assertion: copy-pasting functionality, such as pagination, makes it more likely that the pagination functionality won't be properly updated across all the modules that use it.

I guess it's a case of YMMV...because in this hypothetical app, it doesn't seem likely that the number of views will multiply, thus making it easier to update the copy-pasted code. But that seems like a mindset as prone to future problems than one that is more DRY-minded.

</comment></comment></comment><comment><USER32> My only real concern with this essay is that the OP bothered to refactor out the duplication, but didn't bother to refactor his internal refactoring when it got too complicated, instead claiming: "look, now it got messy", threw up his arms, and said there's nothing more that can be done, blaming DRY as the culprit.

Except we CAN do something about it.

It would have been just as easy to continue refactoring the tweet_list() method to pull filtering, pagination, and profanity checking out into sub methods-- at which point you've built a strong reusable component that can support many more combinations of those extra requirements. So by the time you get more feedback saying, "we need a new page that only shows 5 tweets per page and hides profanity, but does not filter", you can now easily take that reusable component, pass in those options and be done rather than starting from the top because you refused to clean up your internals. That's why we strive for reusable components in the first place.

In other words, if the argument is that refactored code is messy, it really means you aren't done refactoring.

</comment><comment><USER33> Refactoring / DRY to me is not about creating monolithic, generic do anything functions. It decomposing code into layers of abstraction somewhat like mini-"DSL"s. The top level functions are tying together next level "down" helper functions. Which may themselves be higher level tools over something like DB api. More than 3-4 layers is probably a smell.

</comment><comment><USER16> In defense of the single flexible function, I think the hypothetical business requirements are pathological. Or perhaps the hypothetical developer is taking a pathologically literal interpretation of them. Who would want pagination in one view but not another? As for the profanity filter, that should probably be a preference of the currently logged-in user which is applied to all feeds which that user views. (It should probably be enabled when an anonymous user is viewing any feed.)

I suppose some developers don't have the freedom of suggesting alternative specified behavior that is nicer to implement. In some cases I have not had that freedom. But in this hypothetical case, when pressed, the person setting the requirements ought to value consistency.

My own experience has been that I tend to do copy-and-paste because it's easier, but then regret it later. I don't think I've yet erred too far on the side of trying to follow the DRY principle.

</comment><comment><USER34> I did a self-study on a project that lasted several months. I wrote down everything, including mis-typed characters, grammar errors, syntax errors, and semantic errors. I did a root cause analysis of the result. One general result is that I have a 3% error rate, regardless of activity. In fact I find that the delete key is by far the most important key on the keyboard, representing 3% of all of the characters I type.

One observation is the fully 1/2 of ALL the programming errors I made were due to copy/paste. Your mileage may vary but I doubt it.

Copy/paste is evil but it is so "low level", like the delete key, that you probably don't even think about it.

You may find it worthwhile to do a deep analysis of your personal error rate on some project. It is very enlightening. In fact, we ought to fund studies so we can get industry wide statistics.

</comment><comment><USER35> The thing is:

"If you are using copy and paste while coding you are probably committing a design error" doesn't conflict at all with what he says. The fact is that copy and paste is the point when one looks and says "is refactoring appropriate here?"

One thing I would point out is that premature optimization is the root of all evil. You can get a pretty good sense that if your refactor adds more lines than it deletes and functionality remains the same, that you have added complexity in refactoring which means very likely that you are doing it wrong. This is particularly true if you can't say it is reducing the number of lines of code generally, or compartmentalizing state changes.

(This leaves aside the fact that the most pernicious use of copy and paste in the world is "sample code.")

</comment><comment><USER36> If your code is starting to look ugly, refactor it. If your refactoring is looking ugly, stop, you're doing it wrong. If a test breaks because of your refactoring, stop, you're doing it wrong. I call this the Don't be Stupid principal.

<comment><USER37> principle

<comment><USER36> Lol. Do I look less stupid if I say it's my principal principle?

</comment></comment></comment><comment><USER38> Isn't this what functional programming is for?

You have several pieces of code that follow a very similar structure and logic but perform very different purposes for the program. So you try and generalise the structure of the code?

</comment><comment><USER39> Refactor if doing so will give you an advantage.

Copy paste when you aren't sure if the requirements will change. Nothing is worse than building an abstraction only to find out it's useless given this new project requirement and that the two abstractions should really be separate.

</comment><comment><USER40> I don't remember where I read it, but it's good advice:

Copy the first time, only start refactoring if you need the code a third time.</comment>