What on earth are Google thinking? (re: Binggate) - <AUTHOR> http://www.puremango.co.uk/2011/02/what-on-earth-are-google-doing/ <comment><USER2> > At first I thought there could be three possible explanations to Google's > handling of this situation:

I think he forgot option 5: whether Bing intentionally copied Google's search results or not, if Bing continues this practice then their index will contain a _de facto_ copy of Google's index for tricky queries. Bing will inherit Google's spelling correction, it's long tail, its top results, and any other enhancements Google develops to return more relevant results.

Google invests so much into these things; if Bing absorbs these improvements without lifting a finger, Google loses its ability to stay ahead through technical merit and innovation.

I think Amit is telling the truth: "And to those who have asked what we want out of all this, the answer is simple: we'd like for this practice to stop." [http://googleblog.blogspot.com/2011/02/microsofts-bing- uses-...](http://googleblog.blogspot.com/2011/02/microsofts-bing-uses-google- search.html)

<comment><USER3> It's nice that Google would like Microsoft to stop copying them. Everyone else who generates unique content (news site articles, TripAdvisor reviews) would like Google to stop copying their content into Google News and Google Places.

Apparently Google only cares about sites copying other sites content when their it's their own stuff getting copied.

<comment><USER4> Any site that wants Google to stop copying their content to Google news can ask Google news to stop doing so. It's very simple.

<comment><USER5> The real issue is that they don't want their content in Google News, but they also don't want to kill the golden goose that is directing traffic to their site. I.e. Google News is generating traffic for them, but they don't want the content to be on Google News either. They want to have their cake and eat it too.

edit: 'they' here refers to the old media newspapers, more specifically to Mr. Murdoch.

</comment></comment></comment></comment><comment><USER6> If it were just click data, how would they get the terms?

They're either parsing the query out of the url, or violating robots.txt to fetch the result page, almost certainly the former. This seems like a pretty clear indication that they've special-cased clicks from google. It's theoretically plausible that they are treating all query parameters the same for all sites, but very unlikely given how much noise that would introduce into their results. Even so, they would have to know that most clicks with meaningful query parameters come from Google. This isn't something that's going to happen by accident.

<comment><AUTHOR> > parsing the query out of the url

Or parsing any words found in the URL. Not unreasonable to believe.

<comment><USER7> Actually, unless I'm mistaken, parsing the URL may well yield good search terms, but the query string is much less likely. Look at eg HN -- you'd parse out id = some long number. I may be wrong, but I don't recall many sites that would have good information for a search engine in that string.

What seems far more likely is bing is using click tracking on G's results. This was explicitly _not_ denied by their VP on the search engine panel today. If not many sites except search engines have useful keywords in the query string, that pretty much validates Google's complaint.

In fact, if you go to Google's blogpost [1], the bing toolbar specifically calls out monitoring "the searches you do, the websites you visit, [...]" [1]. And the MS guy doesn't deny using clicks on G's search results [2]. In fact, the pretty much just says they copy G on long tail searches.

[1] [http://searchengineland.com/google-bing-is-cheating- copying-...](http://searchengineland.com/google-bing-is-cheating-copying-our- search-results-62914)

[2] [http://www.bing.com/community/site_blogs/b/search/archive/20...](http://www.bing.com/community/site_blogs/b/search/archive/2011/02/01/thoughts- on-search-quality.aspx)

<comment><AUTHOR> > I don't recall many sites that would have good information for a search > engine in that string.

Well, Google is one such site - as in, if you have bing toolbar installed and are on google.com/search?q=keyword and click on example.com, then Bing can easily extract "google com search q keyword" and associate it with example.com - without anything explicitly or intentionally relating to Google in their code.

They may also be looking at referrer info.

</comment></comment></comment></comment><comment><USER8> Google's outrage seems silly to me. They made a fortune harvesting _other people's_ judgments on relevant webpages. That's what PageRank is.

So Microsoft takes their search results - it's all in the game! And Google wrote the rules!

<comment><USER9> But this is a matter of core functionality. Bing should not be using clickstream data from Google searches as a signal. The core offering of Google search is getting used by its competitor, without any citation. Yes, it's a side effect of the Bing bar's clickstream tracking, but it's still an observable, quantifiable effect.

It's like if Developer A takes Developer B's code and passes it off as his own work without any citation. Does it matter if it was malicious or incidental? Not really, it's still wrong. And if Developer A gets caught doing it, he should own up to his mistake and fix it.

<comment><USER8> The way I see it, it's like if B builds entire applications by pasting snippets from Stack Overflow without citation, and then pitches a holy fit upon realizing that one of his/her amalgamations of Stack Overflow snippets has in turn been cut and pasted.

The boundary just seems arbitrary to me, which makes the whole thing seem hypocritical.

</comment></comment></comment><comment><USER10> OK, out of curiosity I actually tried installing the Bing bar (on IE9 beta, which, btw, makes you manually activate add-ins after they're installed. The installer for the addin itself is pretty upfront about it sending your click data and stuff - it's right there on the one and only options page, next to one of three checkboxes - though the box is checked by default which I think is dubious).

I haven't been able to influence the Bing search results (no surprise there, since I've only spent a few minutes on it and not weeks like the Google folks) but one thing I did find _very_ interesting was that if you search for something on another site, the Bing bar actually lights up and populates its own search field with your query so that with another click you can search for it on Bing.

As far as I can tell, the bar doesn't seem to be using any heuristic to tell what is a search query but just has a list of sites/URL patterns it knows about. Besides Google and Bing itself, these include Wikipedia, Yahoo, Ask.com, Amazon, Facebook, eBay, YouTube, MSDN and IMDB, but not Twitter or DuckDuckGo. This doesn't prove anything about how it's feeding search results of course but it does at least suggest that the Bing bar is very interested specifically in search queries, though not only on Google.

</comment><comment><USER11> I think the key to all of this lies in the code for the bing toolbar (and the code that parses its data).

If I search twitter for "binggate" then click on a link, the referrer would be:

<http://search.twitter.com/search?q=Binggate>

It wouldn't be hard to write a generic parser to detect URLs that look like search queries, and it'd be a novel way to gain a lot of information from private indexes (stack overflow, twitter, lucene/solr setups, reddit, etc).

If this is what Bing is doing, then kudos to them for clever thinking, no foul play.

However, if the toolbar has any logic to specifically parse Google's URL syntax, or if they're filtering and correlating google.com URLs against their own algorithms, then it's copying and foul play.

The surprising thing here is how much publicity Google's giving the issue knowing that Microsoft could shoot them down in no time with a few code snippets (if there's no foul play going on).

<comment><USER12> > It wouldn't be hard to write a generic parser to detect URLs that look like > search queries

vs

>However, if the toolbar has any logic to specifically parse Google's URL syntax

So they're either generically copying from ANY search engine, or they're specifically copying from Google. Either way, the perception and actual outcome is the same. I'm not sure either is inherently good or bad, but I'm surprised that there is such a delineation in peoples' minds.

<comment><USER11> Interesting.

Take a site-specific search index like the one that reddit uses. Such an index can prioritize by votes, comments, users, fine-tuned spam filters, and so on. A search engine specifically optimized for reddit has a lot more information available when returning links than a generic web crawler ever could.

If Bing came up with a generic way to leverage site-specific search engines and help drive traffic to those sites, is that still perceived with negative connotations?

</comment></comment></comment><comment><USER13> This piece is so bogus. If, as he suggests, Google's query data is accidentally getting caught up in some larger Bing project, then why doesn't Bing just say that in their numerous posts and tweets about the scandal? Instead, they've just thrown mud at Google.

</comment><comment><USER14> I think the biggest question here is, if a site robots.txt file prohibits bots from gathering data, should that also prohibit bots that piggyback on real user sessions ?

<comment><USER15> >if a site robots.txt file prohibits bots from gathering data,

Pages in a robots.txt file still get put into Google's index they just don't get parsed. Surprised me to find that out; <http://www.seomoz.org/learn- seo/robotstxt> (see "Why Meta Robots is better than robots.txt").

</comment></comment><comment><USER16> Can we please, please, PLEASE not call this Binggate?

Not everything needs a pithy one-word term of endearment.

<comment><USER17> Bingapocalypse?

Oh wait, that's still one word.

</comment></comment><comment><USER18> Very good post. I'd like to suggest another explanation: they really ''do'' see what Microsoft is doing as cheating, and expect others to share their outrage. When you're "in the bubble", talking only with people who share your perspective, it's easy to believe everybody things they way you do. And Google's known for being smarter than everybody else when it comes to search, so at some level people there probably believe that the only way anybody could get results as good as they are is by cheating.

<comment><USER19> A bubble like Microsoft Research for instance?

The comments on this subject seem very polarised and it is interesting to look at the background of the commenters.

<comment><USER18> Yes, although in some ways MSR is less in the bubble than the rest of the company -- people go to conferences and are aware of what's happening elsewhere. Microsoft's bubble applies just as much here as Google's: I'm sure a lot of folks there can't understand why Google or anybody else would think there's anything wrong with what they did here. When bubbles collide.

My charter back in 2006-7 was "game-changing strategies", which meant getting people to think outside the bubble. So we did a lot of work analyzing Google's, Yahoo's, and Microsoft's corporate culture. One of the things that's core to Google's identity is preferring algorithms to anything that has to do with people, and that's very much on display with Binggate.

</comment></comment></comment><comment><USER20> My first thought was that it was click data, and not outright copying. If a bunch of Google employees with Bing toolbar start clicking on links to some made-up term, that should spike the data enough to change results within a few weeks. How can Google positively rule that out without internal knowledge of how Bing works?

<comment><AUTHOR> With a control. Launch a fake search engine with similar spiked results but with no traceable connection to Google. If the results only show up on Bing from google-clicks and not from clicks on the control, then it's a good indicator that Bing have Google-specific code. Then run the test a few more times with greater than 100 queries (which in SE land is a miniscule test set).

From what they've said, it seems like they only tested against fake clicks on google.com. That tells us Bing are using click data but nothing more. This is a pretty simple debugging technique, which is why I'm shocked if Google didn't think to do that. I really wish I wasn't the one saying this, wish I didn't have these doubts. But I can come to no other conclusion than the ones I've outlined in the post.

<comment><USER21> If you can't come up with any other conclusion, I'm not sure you're really sorry about having those doubts. Here's a few alternatives off the top of my head:

0\. There might or might not have been a control group, but its results didn't matter since:

0a) the whole purpose of revealing this was to make sure that blackhat SEOs could start abusing the system, and MS would thus be forced to stop doing it.

0b) the whole purpose of revealing this was to shame MS into stopping the practice, and for that purpose it didn't matter whether the system was specific or generic.

0c) describing the full experimental setup and the gazillion things that were tested would just have distracted from the core story

1\. There was a control group and it suggested that the mechanism wasn't generic, it just wasn't mentioned because:

1a) it was held back as a gotcha in case Microsoft started lying about what the system actually did.

1b) positive evidence from the control group couldn't actually prove anything, it'd be suggestive at best.

2\. There was no need for an explicit control group since:

2a) they actually observed the network traffic of the toolbar, and it only sent the relevant information for Google and not other sites.

2b) they disassembled the toolbar and found out it had Google-specific code related to this.

3\. There was a control group and it suggested that the mechanism was generic, but:

3a) they thought that the the mechanism being generic didn't matter, and what MS were doing was still equally dodgy.

3b) they thought that MS would not be keen on trying out a "oh no, it's not just Google whose algorithms we're leeching off when spying on users, it's every other site too" PR strategy.

I have no idea of what actually happened. In all likelihood it was something not listed here, since these were just random ideas. But at least I think many of them are way more plausible than the silly "maverick super-senior engineers botch the job, leak a flawed story, PR coverup follows" theory.

</comment></comment></comment><comment><USER22> I like to imagine that the bing algorithms are so smart, that it realizes the preciousness of the signal in clicktrack-logged visits to websites with a referrer "<http://www.google.com/search?q=%s.> This single feature is given so much weight that bing unintentionally gives the appearance of wholesale google duplication.

</comment><comment><USER23> The article was interesting, but using Google and Bing as plurals really disorients me. I had to stop and mentally substitute every time he said something like "Google are thinking".

<comment><USER24> That's standard practice in the UK when referring to companies.

<comment><AUTHOR> thanks, I was beginning to wonder if I was 'right' or not. Didn't know the standard was singular in US.

</comment></comment></comment><comment><USER25> Finally some sense in all this.

</comment><comment><USER26> Really interesting comment at Reddit about what Google might be thinking:

[http://www.reddit.com/r/programming/comments/fd3g9/google_bi...](http://www.reddit.com/r/programming/comments/fd3g9/google_bing_is_cheating_copying_our_search_results/c1f1bvx)

Apparently by creating fake results they've published something "creative" and thus potentially able to be copywrited. Just speculation but interesting.

<comment><USER27> The Reddit commentator appears to be operating on the idea that trap streets and the like are copyrightable under US law, which is a fairly common urban legend.

In reality adding a fake "fact" to a collection of real facts does not allow you to sue someone if they copy the fake fact along with the real facts; from the decision in _Nester's Map & Guide Corp. v. Hagstrom Map Co._, treating "'false' facts interspersed among actual facts and represented as actual facts as fiction would mean that no one could ever reproduce or copy actual facts without risk of reproducing a false fact and thereby violating a copyright ... If such were the law, information could never be reproduced or widely disseminated"

<comment><USER15> In Europe there are specific database IP rights.

Re the decision you cite it seems to be spurious. The notion of whether the data is factual or not is irrelevant. Was the presentation copied?

>would mean that no one could ever reproduce or copy actual facts without risk of reproducing a false fact

Unless they actual did some work and checked the facts rather than making a slavish, infringing, copy of someone else's work.

>If such were the law, information could never be reproduced or widely disseminated"

For limited terms of never (not limited enough mind you but nonetheless limited).

This is interesting though - if you can copy facts with impunity then can't I copy the "fact" of the score for Katy Perry's Firework for example?

</comment></comment></comment><comment><USER28> 9/10 desktops run Windows and IE still has almost 60% market share, despite being much worse than Firefox or Chrome (yes IE9 is a huge improvement). Microsoft could do a lot of damage to Google by leveraging their desktop monopoly. I think Google has around 70% search market share, but they're easily replaceable. By the time the courts sort it out, the damage will be done.

</comment><comment><USER29> Why does he think that Google were not aware of the clickstream data when they set up (several) experiments where they specifically install what they thought were sources for the clicks and then purposefully clicked?

</comment><comment><USER30> It's okay for MS to parse user input through it's toolbar, fine. However, it's not okay for Microsoft (or anyone) to use that toolbar to figure out what response any given server sends in reply to any kind of requests - unless it's part of a documented feature. You're only seeing the part where Microsoft is grabbing user input and not the part where Microsoft is grabbing Google data.

</comment><comment><USER31> I think Google is right to be upset here. Google's biggest asset is their search results and Microsoft's response seems tepid at best. Effectively MS is claiming that they don't scrape google search results, instead they merely constructed an automated device which essentially does exactly that. If Bing doesn't see what's wrong with that then they are not terribly smart.

</comment><comment><AUTHOR> I've written a followup post addressing some of the common reactions to my first: [http://www.puremango.co.uk/2011/02/what-are-google- thinking-...](http://www.puremango.co.uk/2011/02/what-are-google-thinking- part-2/)

On HN: <http://news.ycombinator.com/item?id=2169690>

</comment><comment><USER32> I have a feeling there is more to this story than meets the eye. Either Google did more tests then they mention on the blog and proved irrefutably that Bing is literally scraping their results, or there's some underlying political stuff going on that we're not privy to.

<comment><USER33> Never mind about proving it - google isn't even _saying_ it.

It's not scraping, it's click tracking.

</comment></comment><comment><USER34> Can't someone just analyze the Bing toolbar binary and figure out what's actually going on here?

<comment><USER35> The Bing toolbar only sends information back to Microsoft, it doesn't decide what to do with it. For this kind of a task it would be simpler to use a packet sniffer anyway.

<comment><USER34> Thanks for the clarification. Does the toolbar send a notification to Microsoft on _every_ click? Or only clicks on Google SERPs?

</comment></comment></comment><comment><USER36> I'm in Venezuela and it just didn't happened, I've parse HTML results from Google a couple of times and Google discriminates results from Address IP, Browser and Languague. So far, I think it's pretty wrong to say something like this without being 100% sure.

</comment><comment><USER37> Whether it's wrong or not, it does make you wonder where Google's priorities are. How much time (cumulatively) did they spend on this and could they have spent this time improving their algorithms instead? Maybe it took a few hours, but still.

</comment><comment><USER38> Is the next push in SEO to pay users to navigate from search results or other sites to your site?

</comment><comment><USER39> <http://www.google.com/xml> is wrong

</comment><comment><USER40> it's tangentially interesting that the number 2 result on bing for 'google' is a washington post semi-hit-piece regarding spam. google's results for 'bing' contain marginally less transparent attacks .

</comment><comment><USER41> I'm surprised google didn't just use this to poison the results.

</comment><comment><USER36> I just tried what the blog said and it's simply not happening.

</comment><comment><USER42> I honestly don't get the controversy of this thing.

Let's say Microsoft via its bing-toolbar is checking what pages people are visiting, what search term they are using on various sites (including google) and what they deem are the most useful results of these queries, and incorporating this into their search engine. Is this really so bad?

As it stands, Google is collecting this data about you almost _everywhere_ on the internet, even if you use google or not. Besides straight Google search, think websites which uses Google's CDN for stuff like jQuery, Google analytics, etc etc. Google is everywhere and they are collecting data to incorporate in their search and ad-platform from everyone. End of story.

Heck, with google instant and their new JS-based search-gui you can't even get the referer information on your own website to see what search terms lead your users to your site. You now _have_ to use Google analytics to get that information, and in getting that information you are helping Google even further in tracking everything everyone is doing on the internet. WTH?

Relatively speaking, is really Microsoft the bad guy here?</comment>


Write code that is easy to delete, not easy to extend - <AUTHOR> http://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to <comment><USER2> I think this highlights a common issue when development is (or feels) rushed. You either end up with developers having only done the first part of each of these pairs (repeating themselves, ball of mudding, etc), without time to clean up as part of each iteration, or, you find developers immediately shooting for the latter half of each pair (DRY, modular, etc) without having done the former, and so you get abstractions that make no sense, overly complex interactions in a shared function as they attempt to be DRY, etc.

This latter is also, I feel, what informs a lot of the monolithic frameworks used for 'enterprise' development, Spring and the like, where a predetermined architecture and structure of the app is imposed by the framework, and which leads to, once you get down in the weeds of dealing with odd edge cases and things, hackery on the part of the developer, or framework bloat if the framework attempts to address the most frequent of those cases.

<comment><USER3> Couldn't agree more. Sandi Metz has an excellent talk touching on this topic. Developers exaggerated willingness to keep things DRY and elaborates a bit on why: it's one of the easiest things for a not-so-experienced developer to identify and one of the easiest things to teach.

Edit: One of the best quotes from that talk is (paraphrased): "The wrong abstraction is a lot more expensive than duplicated code". [https://youtu.be/OMPfEXIlTVE](https://youtu.be/OMPfEXIlTVE)

</comment></comment><comment><USER4> The trick really boils down to: be messy, but clean up. If you don't do the second half of each thing (copy and paste / don't copy and paste), then you end up with that unmaintainable mess. If you do the second part too early, you end up with the wrong abstraction.

Sandi Metz also talks about this: [http://www.sandimetz.com/blog/2016/1/20/the-wrong- abstractio...](http://www.sandimetz.com/blog/2016/1/20/the-wrong-abstraction)

<comment><USER5> Yes yes yes. When I'm coaching junior developers (or developers that like over engineering) this is what I try hammering home more than anything.

If you looked at my code at 10-70% solution complete you'd find so many egregious DRY errors, magic values (likely with a comment above for my own memory), functions that are too long, and a whol ball of mess. If you looked at it 90-100% complete it would be nearly unrecognizable in writing style to what you saw earlier.

Figure out how to solve the problem first, spend all your efforts on that, then figure out how to clean up your code and make things nice/performant/secure/etc.

I've found that as I've gotten better my sloppy is better than a lot of people's clean, which means I'm spending less time on cleanliness, and getting done faster.

<comment><USER6> In the real world, at least mine, once it works you get pressure to move on. Both my own mind saying, it works so it's good enough, and external pressure from a stake holder who says it looks good lets move on. I don't feel like I get to that latter 20% - 30%. As such, I try to be thoughtful on my design, but I also realize the need to get stuff done. It's a balancing act.

<comment><USER7> The pressure you put on yourself is your responsibility. Don't pretend it's something you have no power over.

The stake holder should be your customer, not your boss. Even if that dynamic is broken, you don't need to tell them it's shippable until you're happy with the code.

<comment><USER8> True, we are the ones responsible for our integrity and we often live with our mistakes, paying for poor design choices during later maintenance cycles.

Just be aware that it can be easy to get lost in a yak-shaving exercise doing your cleanup refactoring. I recommend timeboxing this phase to no more than an hour, or prioritizing one or two specific tasks so that you don't go off the deep end.

Don't forget that others are depending on you to get their work done. There may be a 2-hour cleanup task somewhere else in the code that will help your teammates far more than the itch you want to scratch here and now.

<comment><USER9> > I recommend timeboxing this phase to no more than an hour, or prioritizing > one or two specific tasks so that you don't go off the deep end.

When I read comments like this I think to myself "How can they possibly be so productive that refactoring can be done inside of an hour?".

Then I remember that some developers split up projects/problems into really small components and can complete a feature or two in a single day. Well then an hour is actually a fair bit of time to spend refactoring.

</comment></comment></comment></comment></comment></comment><comment><USER8> This is extremely validating to read. How many times have I battled with DRYists over which solution is "better."

I've happened upon the pattern of code growth described here after years of encountering and resolving pain points in code, often as the maintainer. DRYness for DRYness sake might feel satisfying when writing the code, but when changes need to be made, it often scatters constraints and requirements throughout the codebase, making any change an unestimatable mess of sneaky traps. Try writing your boilerplate initialization code straight some time... no metaprogramming, no helper functions, just get your configuration, create your objects and wire them together. It is oddly liberating to have your bootstrap code flat and unmagical.

<comment><USER10> > DRYness for DRYness sake might feel satisfying when writing the code, but > when changes need to be made, it often scatters constraints and requirements > throughout the codebase, making any change an unestimatable mess of sneaky > traps.

This also applies to "one-liners" when writing code. For example in ruby:

Model.where(...).foo { ... }.flatten.bar { ... }.join(...)

Looks nice and all, but it can be confusing when you need to fix a bug that you aren't sure of the source (foo or bar? maybe we shouldn't flatten yet? maybe we should join sooner?). It's much better, IMHO, to write more lines of code here so that you can easily test each step manually when a bug occurs.

<comment><USER11> I'm not addressing your overall point, but one thing I learned about recently that could help in debugging those Ruby one-liners is the `tap` method. You can stick it in the chain and set a breakpoint.

[http://ruby-doc.org/core-2.2.3/Object.html#method-i-tap](http://ruby- doc.org/core-2.2.3/Object.html#method-i-tap)

<comment><USER12> Further off-topic, but I almost exclusively see tap used as some kind of clever way around a return line, e.g. instead of:

        def build_hash_or_whatever
          hash = Hash.new
          hash[:foo] = some_generating_method
          hash[:bar] = "some value" if whatever?
          # More hash construction nonsense
          hash
        end

With tap it is:

        def build_hash_or_whatever
          Hash.new.tap do |hash|
            hash[:foo] = some_generating_method
            hash[:bar] = "some value" if whatever?
            # More hash construction nonsense
          end
        end

I think the tap-form is significantly worse and I don't understand the aversion to a single line that clearly shows what you are returning.

</comment></comment></comment></comment><comment> V-2 Makes sense - you can't know the right abstraction upfront.

As Hejlsberg said:

 _" If you ask beginning programmers to write a calendar control, they often think to themselves, "Oh, I'm going to write the world's best calendar control! It's going to be polymorphic with respect to the kind of calendar. It will have displayers, and mungers, and this, that, and the other." They need to ship a calendar application in two months. They put all this infrastructure into place in the control, and then spend two days writing a crappy calendar application on top of it. They'll think, "In the next version of the application, I'm going to do so much more."

Once they start thinking about how they're actually going to implement all of these other concretizations of their abstract design, however, it turns out that their design is completely wrong. And now they've painted themself into a corner, and they have to throw the whole thing out. I have seen that over and over. I'm a strong believer in being minimalistic. Unless you actually are going to solve the general problem, don't try and put in place a framework for solving a specific one, because you don't know what that framework should look like."_

[http://www.artima.com/intv/handcuffsP.html](http://www.artima.com/intv/handcuffsP.html)

So of course write it dirty, clean it up afterwards seems like a better idea, because it lets you feel what the right abstraction is. However it takes discipline to always refactor something that works fine already

</comment><comment><USER13> I feel like this is a little like learning to cook. At the very beginning, I would just create a huge mess as I was going, use a different pan for every item, a different mixing bowl for each thing, and I didn't know how to prep effectively in advance.

This left a train wreck in the kitchen after each meal that I was forced to clean up before I could cook again.

I learned how to do a little up front prep, and that saved time and mess, and made the whole process smoother.

As I learned more, I started trying to clean up as I went and conserve the number of pots and pans, I made a lot of mistakes washing everything after I used it, which slowed the whole process down, and I didn't need to reuse some of it.

Now I'm in a place where I have enough experience that I know in advance what I can reuse for this meal and whether it needs to really be washed (maybe you just deglaze a pan and wipe it down with a paper towel instead of hauling it over to the sink and scrubbing it), when I'm actually done with an item, what the approximate cooking times are, etc.

Now I can cook complex multi-course meals to a good restaurant quality and have just a couple of things left to clean up by the time dinner is ready to serve.

It doesn't take a genius to figure these things out, just the acknowledgement that these things are important, and the more people you are working with, the more important they are.

Sure, I could have spent the rest of my life cleaning the huge mess after every meal at home, and it wouldn't really matter.

But if you are going to be a line cook at a large restaurant, you must get a hold of these concepts.

<comment><USER14> And this requires introspection on your process, whether the domain is programming, cooking or really any human endeavor. Unfortunately not many people can do this effectively (and this is related to the 10k hours to mastery dictum - it requires effortfull practice which includes exactly this sort of introspection)

</comment></comment><comment><USER15> To add to his point about copy and pasting...

Notice the contradiction implicit in these bits of accepted dogma:

1\. Copy and pasting code is evil (DRY).

2\. Tight coupling is bad, loose coupling is good.

A system written with no abstracted functions (ie, where copy/pasting was the _rule_ ) has no coupling. Of course, I'm not advocating such a style. But it's worth keeping in mind that practicing DRY through abstraction necessarily increases coupling, which means that it's always a tradeoff and that you have to get your abstraction right so that the coupling, on average, makes change easier rather than harder.

<comment><USER16> That's not really true, is it? Coupling is independent from explicit code dependencies. Take, for instance, two programs. One writes to "/tmp/file". The other reads from "/tmp/file" but neither shares the code to determine the path ("/tmp/file", in this case). They're code independent, but very tightly coupled.

In fact, I'd hypothesise that copy-paste codebases are more tightly coupled than otherwise.

<comment><USER15> It's certainly true. Coupling measures the degree of interdependence between modules/functions/classes. If everything is copy/pasted, no module depends on any other -- it simply has its own copy of the code. This will result in a terrible system, but it will be very loosely coupled. When you change something, you won't have to worry about it's impact on anything else.

In your example, if the content of the file is changing the behavior of the program reading it, then the programs are sharing code via the file. It just happens to be going on at runtime rather than at compile time.

</comment></comment></comment><comment><USER17> Totally agree. One thing that I still find really hard to maintain/delete is CSS code. More and more I feel like it should be included with components rather than in a plain .scss or .css files. It feels good to be able to delete a component knowing that there isn't css crap left behind..

<comment><USER18> To solve this issue I keep component-specific .scss files alongside each component's JS file, in a component-specific folder. A root index.scss file imports all of the components' .scss files.

Coupled with BEM, this also helps prevent component style interdependencies.

</comment></comment><comment><USER19> This article could be summarised as "abstract only when you see a clear need to", something that seems to be the exact opposite of what a lot of programming courses teach; especially those dealing with object-oriented design. I think abstraction should be viewed not as a technique to be applied generously and whenever possible, but a necessary evil, resorted to only when nothing else can simplify the code.

A bonus of this style is that it often also makes the resulting code more efficient for the machine to execute, reducing the need for optimisations later.

</comment><comment><USER20> The importance of DRY is proportional to what could go wrong if the repeated code has an issue. For instance, if the code is performing some vital calculation or is part of your security architecture, it had better not be repeated anywhere because somebody will need to patch it later and will need to guarantee that the update has been applied consistently.

Sometimes, it's just clearer to rewrite something yourself. For instance, just because you _can_ express just about anything in terms of algorithms in the C++ standard library, you should instead write the simpler stuff by hand. (A great example is this silly idea of "copying to an ostream iterator" just to print out some data; I don't care if that combination of standard functions _happens_ to produce the desired result, because the code is painful to look at!)

It's also helpful to using aliasing (e.g. C++ reference variables) to make similar code look as similar as possible; i.e. rather than have two similar blocks using entirely different variable names throughout, declare references at the top to give them the same names so that the similarities are obvious. This also makes it easier to later pull common parts into functions if desired.

</comment><comment><USER21> Easy to _replace_ not simply to _delete_.

Anything that is easy to delete (and just leave it deleted) is superfluous. Don't write superfluous cruft, obviously.

Easy to replace is important in all engineering. A product or structure with easily replaceable parts is better than one without easily replaceable parts, all else being equal.

We'd never say "design a brake caliper with brake pads that are easy to _delete_ ". :)

</comment><comment><USER22> From his about: "I am not a very good programmer. I forget to write tests, my documentation is sparse, and i'm pretty apologetic about it at any code review. "

Guy who doesn't write unit tests suggests work around to dealing with problems in code that has no unit tests...

<comment><USER23> The point of a lot of tef's writing and speaking isn't that he's a bad programmer, but that he is tearing a large hole in people who claim to be great programmers. tef is actually a pretty good programmer; good enough Heroku employ him.

It's dry British humour: "I'm a terrible programmer, but that's okay, because programming is terrible, the whole industry is terrible, and pretty much everyone is terrible at programming, especially if they are lecturing you about how to do it because they are probably full of quite a large amount of shit; now let's try and do better".

If you are used to chest-thumping evangelical "rah rah, everything is awesome" sermons, the British cynical attitude of "no, this is all shit, and I'm awful too, but maybe, if we make a nice big cup of tea, and thought about it a bit more, we might be able to be a bit less rubbish" sounds grating. The opposite certainly is true.

</comment></comment><comment><USER24> Corollary: Write code such that it's easy to use _grep_ to find where things are used.

One of my pet peeves is when OO programmers take class's separate namespaces as a license to use super-generic method names. Method names like "add", "update", etc. Makes it near impossible to figure out where those methods are used!

<comment><USER25> That's why you should use an IDE with proper "find all references" instead of coding in notepad.

<comment><USER26> Bit trollish, but: IDEs which let you ignore the complexity of the code let you write complicated code. Writing in less-fully-featured environments forces you to consider simplicity as a desirable feature (so you can, y'know, work out what's going on when using a 80x24 terminal).

</comment></comment></comment><comment><USER27> Interesting. Thanks.

This looks like it's from a very functional (FP) view of the world. I like a lot that's here, but I fear it will make my OOP friends' heads explode.

Also, as a nit, the essay feels a bit on the "thrashing" side. I know when I'm trying to express complex concepts many times I will hit the same topic a few different times from multiple angles until I get something that's tight. This essay feels like on of those attempts -- nothing here to throw rocks at; it just doesn't feel "done" yet.

<comment><USER28> Hopefully, he has--cough--written paragraphs that are easy to delete, not easy to modify. Ha. Ha. Ha.

But seriously, this general idea strikes me as being applicable to writing words as well. Write in such a decoupled way that it is relatively easy to excise paragraphs and sections without destroying the rest of the piece.

That may introduce a small amount of redundancy, but it also makes it easy to read in smaller chunks, reorder, or remove outright rather than edit portions that no longer fit.

<comment><USER29> Do remember to actually _delete_ paragraphs though. I've seen long-form articles that were really just several short-form articles that are linked together by a listicle format. If I'm lucky, there may even be a few transition phrases or a narrative that would justify the listicle format.

All these articles may have been good if the writer wasn't so wedded to the long-form format and started deleting paragraphs. But since he was wedded to the format, all the reader sees is a rather meandering and dull piece. It's an example of the sum of pieces being greater than the whole.

</comment></comment></comment><comment><USER30> Some of the good advice in this blog post is taken too far, like the advice about intentionally writing shitty code in order to learn from your mistakes. Coding is like any other skill. You learn from your mistakes only if you're trying your best not to make them. Otherwise there's no differentiating between your real mistakes and your carelessness.

> A lot of programming is exploratory, and it's quicker to get it wrong a few > times and iterate than think to get it right first time.

It's really hard to get it wrong a few times if you have 10 other developers building on top of your mistakes. Then you're pretty much stuck with the code you thought you were going to get rid of. In accordance with Murphy's law, it seems like the code you push out knowing it sucks always ends up being the foundation for something really important. So yes, build the simplest thing possible. Yes, don't abstract pre-maturely. No, don't write shitty code on purpose.

Similarly advice with "copy-paste 10 times". While I agree that you should copy-paste 2-3 times, 10 times is way too many. By the time you've copy pasted something 10 times, it's too expensive to refactor. Or if you do refactor it, 5 of those 10 instances have changed beyond recognition and will be missed. They continue to evolve and now for the rest of the life of your software, you're fixing 6 times the amount of bugs.

</comment><comment><USER31> Step 8: Listen what other programmers say

Step 9: Don`t listen to other programmers

</comment><comment><USER32> This all goes to how hard code reuse is - is it worth having two very similar functions that do the same thing instead of having one function that's DRY but more complex than both.

Maybe you should have a shared function that has the similar bits of both but then when you remove pieces you have this added complexity.

Everyone can give you arguments either way...

I remember reading this beautiful explanation about an idea for a functional language where you simply install collections of methods that do specific things and each method you install should be as simple as possible and do just one thing etc. - maybe it was somewhere on the Elixir mailing lists but I can't find it!

</comment><comment><USER33> This is a super frustrating article, so I'm going to post the quote I think that sums up the art:

"To write code that's easy to delete: repeat yourself to avoid creating dependencies, but don't repeat yourself to manage them."

This is such a hard thing to explain to someone, so I'm impressed the writer described it so accurately and succinctly.

</comment><comment><USER34> Rule Of Three covers steps 2 and 3. This they used to teach in our 1st year computer science program: [https://en.wikipedia.org/wiki/Rule_of_three_(computer_progra...](https://en.wikipedia.org/wiki/Rule_of_three_\(computer_programming\))

</comment><comment><USER35> I love this article. I have been at war with some of the tech teams at my company that force reusablilty. Personally I find it a nucance, and in most cases can write a new function in less time. You have articulated this ideology very well, thanks for the insight

</comment><comment><USER26> Another behaviour which makes these steps hard to follow is: someone wrote code and ships it, and thinks, because it works and is being used, it is great and doesn't need to be changed. These feeling get even more in the way when the code took a while to write!

I keep telling my team to write bad code: write something that's correct (but not great) quickly, with an eye to replacing it with something better next sprint. Then, once you have something which works end-to-end, work out which bit is most terrible, and replace it. Repeat & launch when the quality is acceptable (and keep repeating after you launch).

</comment><comment><USER36> I might not have understood the article, but avoiding to write code is dangerous, and will lead to code that after a while will be impossible to understand because there will be hacks upon hacks. Your boss loves this though.

And about cleaning up the code after you got it working, ha ,ha, like that would ever happen. It doesn't have to look nice, but make sure you get it right from the start, and that it covers all those twenty edge cases. It will cost more time ... So most software projects fail anyway!? Maybe it's because of your shitty code ? (grin)

</comment><comment><USER37> Over the past year I can't stop thinking about this article: [http://250bpm.com/blog:51](http://250bpm.com/blog:51)

</comment><comment><USER38> A neat response to the same problem posted yesterday:

[http://akkartik.name/post/wart-layers](http://akkartik.name/post/wart-layers)

Basically, layers with numbers and preprocessor directives to inject (tangle) code in the proper places without worrying too much about language abstractions.

</comment><comment><USER39> I stil think it would be better to guide programming using principles, not practices. To avoid fragile code, remember the open closed principle. To avoid wrong abstraction, think single responsibility and substitution principle. The author is right about interface segregation. :)

</comment><comment><USER40> This is awesome. Easier said then done, but, yes. Coding is a craft, and always will be.

</comment><comment><USER6> I read this and thought it sounded a lot like my reality. Then, I expected to come to these comments and hear arguments against these thoughts. Glad to see a lot of us agree with much of it.

</comment><comment><USER41> really appreciated this b/c I've lately felt bad a/b not "getting it right the first time," and one point here seems to be that that never happens, exactly; that development always involves some amount of evolving a body of code, at one level or another

</comment><comment><USER42> _Jean-Paul Sartre's Programming in ANSI C_

I am pretty sure that is at least a misattribution.

<comment><USER43> I believe all (or at least most) of the "quotes" have been re-purposed/altered by the [blog's] author. e.g. " _Every line of code is written without reason, maintained out of weakness, and deleted by chance_ " is based on Jean-Paul Sartre's actual quote[1] " _Every existing thing is born without reason, prolongs itself out of weakness and dies by chance._ "

[1] [https://en.wikiquote.org/wiki/Jean- Paul_Sartre](https://en.wikiquote.org/wiki/Jean-Paul_Sartre)

<comment><USER44> It's kinda sad that no one else seams to be picking up on this. To lead off with that intentionally altered quote and looking at code in terms of how existentialists frame reality is the most significant insight.

edit: Radical Freedom in my programs intensifies

</comment></comment></comment><comment><USER45> "Every line of code is written without reason, maintained out of weakness, and deleted by chance" Jean-Paul Sartre's Programming in ANSI C.

I just started the article and I already have problems with it, not a good sign. While it may be obvious when you research the timelines of JPS (he died a few years before ansi c 89 was established) and C , not to mention the miles of metaphorical distance between Computer Programming and JPS's work) . I guess the author was trying to be cute?? but that fabrication should be made clear as such. he's undermining his own inherent credibility as an author, however much the reader decides to put in. Serious problem in my book.

<comment><USER46> I thought it was pretty funny. Lighten things up from the start.

<comment><USER47> Agreed, I usually warm up the engineering crowd with a few quotes from Shopenhauer or Malthus, but Sartre is always good for a few yucks.</comment></comment></comment>


Kiln - Version Control and Code Review Software - <AUTHOR> http://fogcreek.com/kiln/ <comment><USER2> With choice of version control so important for any project, how can they be so light on details about what theirs is or how it compares to the very strong competitors already out there?

<comment><USER3> Obviously FogBugz did it themselves because they are the most highly documented sufferers of NIH syndrome.

<comment><USER4> They are? I know they used VNC for their remote help thingie. What's the charge?

<comment><USER5> Writing their own language/compiler surprised a lot of people:

<http://www.joelonsoftware.com/items/2006/09/01b.html>

<comment><USER6> Why? It worked. It solved a problem they had. And they are successful as a business. It's only surprising if you think their problem was a non-issue. Sometimes NIH just means your doing something no-one else has a solution for.

Google gets accused of NIH quite a bit too but when your serving billions of requests per second you find issues with libraries, frameworks and services that no one else sees. It's not NIH when you are solving a problem no one has solved or at least solved and made publicly available.

</comment></comment></comment></comment></comment><comment><USER7> Sorry for the confusion! The updated homepage has been pushed. That should clear up a bunch of questions, but feel free to yell if you are still unclear about stuff. By yell, I mean tweet @kilnfc or hop on over to <http://kiln.stackexchange.com> and ask.

</comment><comment><USER8> Hosted source control for closed-source projects? I am intrigued by this idea. I would have thought this would be a small market.

<comment><USER9> github (you can pay for a private account), unfuddle, I'm sure I'm missing at least another major one.

<comment><USER10> assembla is the other bigger one. Very slick too</comment></comment></comment>


Why We Moved from AWS to Google Cloud - <AUTHOR> https://lugassy.net/why-we-moved-from-amazon-web-services-to-google-cloud-platform-726c412fd667#.hxaimrey9 <comment><USER2> Almost all of the subheadings begin with the phrase: "Why we moved?"

I appreciate the article's content and all, but that "Why we moved?" repetition is probably just a waste of space and user attention.

<comment><USER3> good point, fixed (and added a bunch of other updates).</comment></comment>


Windows Task Manager Runs Doom (896 Cores) - <AUTHOR> https://m.youtube.com/watch?v=hSoCmAoIMOU&feature=youtu.be <comment><USER2> As with their previous videos, no reason to believe they aren't fake. (i.e. in earlier videos the "Task Manager" somehow didn't _quite_ look like it does on real Windows machines...)</comment>


Misinformation on NSA programs includes statements by senior U.S. officials - <AUTHOR> http://www.washingtonpost.com/world/national-security/misinformation-on-classified-nsa-programs-includes-statements-by-senior-us-officials/2013/06/30/7b5103a2-e028-11e2-b2d4-ea6d8f477a01_print.html

<comment><USER2> I think it says a lot that the defense of these programs relies on what amounts to the redefinition of words, sometimes redefinitions with practically the opposite meaning - like how "collect" has been changed to mean access already collected data.

If they can't defend the programs with straightforward language it means they simply aren't being straightforward with us.

<comment><USER3> _A week earlier, President Obama, in a television interview, asserted that oversight of the surveillance programs was "transparent" because of the involvement of a special court, even though that court's sessions and decisions are sealed from the public._

I just don't even know what to say to this. There is no way to have a discussion if they are going to redefine words arbitrarily.

<comment><USER4> one definition of transparent is, "cannot be seen."

</comment></comment></comment><comment><USER5> I would place some part of blame on the journalists for not asking specific enough questions. All the discussion around "direct access" was a typical example where more precise questions would say more than hundred pages of discussion.

For example, do Google allow NSA to run computer code on google's servers? More specific, do NSA supply database queries when demanding data? Do NSA ever provide Google with tracking code like JS or links to NSA webbugs (1px imgs) that Google later put in targeted ads? Do Google ever provide physical (or remote) access to servers, hard drives (like backups) or network devices to NSA?

Either of those would to me be equivalent with "direct access", while I can perfectly see how Google would not define it as such.

<comment><USER6> I can imagine that the current state of affairs is that the private tracking networks sell the data to NSA. It's not just the problem if the private company or the government institution does something, the bigger problem is if the whole system which includes the private companies is outside of checks and balances and prone to misuse. The honest and not hidden laws should regulate both private and government uses of citizens' data.

</comment></comment><comment><USER7> It's unfortunate that the rule of law bifurcating in this country - swat teams and militaristic police for us common folk and the ability to lie to congress, etc not being punished for the rulers. Hopefully a committee will be started to investigate these crimes and people will involved will be prosecuted to the full extent of the law.

</comment><comment><USER8> As I said at almost every article where the "NSA officials" were defending the programs in some way - _Do not trust them_! They _can 't_ be trusted.

We need a real external investigation done into this (and even that needs to be done by very _credible_ people), and then we'll see how it all goes. You can't be naive enough to trust anything the NSA or the administration is saying now.

<comment><USER9> Who should investigate this? The same government guys? Will you trust this investigation?

Private investigators? Will multi-layered bureaucracy let any outsider to just go in and look into papers? Or you suggest going to congress and mr. president to ask permission to let you investigate? (Basically, asking the same gov. guys you never trust.)

Here's suggestion: I don't want to play this game. Let NSA do whatever they want to do, but I don't want to pay for this. Let me not pay my taxes and use different currency, so they cannot tax me by inflating it. Then we'll see if they'll have enough funds to continue doing what they are up to.

The real fix is total removal of any moral justification for the government. Give voluntarily to those who deserve it (in your view) and expose violence when they try to extract money from you by force (explicitly via taxation or implicitly via inflation). When people see the "gun in the room", they'll be more informed about how shit like NSA happens to them and how to _actually_ prevent it.

However, if you in favor of some government policy, I respect you decision and will never violently prevent you from sponsoring it. But will you give me the same respect if I disagree with you and not violently prevent _me_ from withdrawing my participation? In other words, will you blame or at least not justify a violent action against me if I decide to not pay for some government programs I dislike?

<comment><USER10> You're welcome to withdraw from our social compact, but in doing so you can't be allowed to stay on U.S. soil because as a practical matter we can't help but defend you from invasion if you do. I'm sure there are lots of other countries that would love to have you, though.

<comment><USER9> So you can point out legitimate owners of the soil? Was there a contract among some folks? Maybe you have some theory of property rights. I'd be very interested to hear it and see how consistent it is.

Btw, slavery once was legal too. It must be covered by your theory of justice.

<comment><USER10> Nobody "owns" the soil, not really. "Ownership" is a fiction--nothing more than an implied agreement between you and a big enough group of other people that if someone tries to displace you from a particular patch of soil, the group will forcibly remove that person. The U.S. governments (the federal and states) don't "own" the soil within the borders of the U.S., but have exclusive jurisdiction to enforce ownership agreements on U.S. soil for no greater or lesser reason other than the fact that they can defend those borders against outsiders.

It is interesting that you mention slavery. No amount of bleeting about "natural rights" will keep armed people from enslaving you, as has happened countless times throughout history. Your "right" to freedom is nothing more or less than your agreement with a sufficiently large group of other people that they will defend you against people trying to enslave you. The history of slavery in the U.S. actually exemplifies this. The 13th amendment didn't end slavery. Constitutional recognition of peoples' inherent right to freedom didn't end slavery. Union soldiers burning down southern cities ended slavery.

<comment><USER9> See my comment below. I don't advocate any definition of "ownership". Call it however you want. It is you, not me who is using those concepts when you tell me that some group of people have some sort of agreement by which I must pay taxes, use certain paper bills as a legal tender etc. I am only saying that if I don't hurt anyone and do not want to participate in certain activities, what is the justification for violent action against me?

So please explain what you meant by your original comment.

<comment><USER10> The justification is that you are on soil that was staked out and is defended by those other people. You're welcome to leave, but you're not welcome to stay without abiding by the rules of the group.

You are advocating a particular definition of ownership: ownership that gives you the right to occupy soil independent of any compact with some other group of people.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> I hope that in the future states of the Earth, they teach disinformation and doublespeak detection/eradication techniques to the teenagers of the tomorrow. It sure would help to know thats going to happen..

</comment><comment><USER12> > I assume most people are acting in good faith

He's talking about himself of course, the NSA is premised upon the exact opposite.</comment>


Counterparty (XCP), first trust-free decentralized asset exchange, is now live - <AUTHOR> http://blockscan.com/order_book.aspx Here are some links to get started:
Block explorer: http://blockscan.com/
Official site (under construction) https://counterparty.co/
Installation instructions: http://counterpartyd-build.readthedocs.org/en/latest/
Github: https://github.com/PhantomPhreak/counterpartyd <comment><AUTHOR> Here are some links to get started:

Block explorer: [http://blockscan.com/](http://blockscan.com/)

Official site (under construction) [https://counterparty.co/](https://counterparty.co/)

Installation instructions: [http://counterpartyd- build.readthedocs.org/en/latest/](http://counterpartyd- build.readthedocs.org/en/latest/)

Github: [https://github.com/PhantomPhreak/counterpartyd](https://github.com/PhantomPhreak/counterpartyd)

Discussion thread: [https://bitcointalk.org/index.php?topic=395761.0](https://bitcointalk.org/index.php?topic=395761.0)

<comment><USER2> So thing is this, you're using proof of burn and/or some derivative of proof of stake or whatever.

Thing is what proof of * you use don't matter. You're not in control of the chain, and anybody can just spam the bitcoin blockchain with fraudulent transactions (double spends, etc.)

And the only way for anybody want to accept any of those assets, would be to calculate the probability of having a real asset in hand from the entire blockchain history that this amount derives off of.

So, very sorry to say that, but, this'll crash&burn. That's just how it is if you forgoe the iron guarantees of proof of work.

<comment><AUTHOR> Could you give an example of a double spend attack on XCP?

From what I understand, XCP transactions guarantee atomicity and funds are swept from addresses to fund any bets/orders immediately. The risk of double- spending XCP is the same as the risk of double-spending BTC as XCP relies on bitcoin's proof of work.

Please correct me if I'm wrong.

<comment><USER2> XCP would rely on the blockchain, and therefore it's atomic transactions are executed. However, the ones putting the transactions in the chain, are not XCP miners. They're bitcoin miners, they don't care what extra information you attach. And by extension, they don't care if they violate some rule in your protocol.

This basically means that XCP-looking transactions will appear in the blockchain, however, they will be invalid, because by the rules of XCP their outputs would already be spent.

In Bitcoin this is an invalid transaction, and it never enters the public ledger. This important because, it allows you to shortcut the path to figure out if an address has unspent outputs. Figuring out quickly if an address has unspent outputs or not is important for miners (they need to produce valid blocks), and it's important for participants to transactions, because it means the recepient can quickly verify that the sender is intending to make the transaction, and that a transaction, sufficiently confirmed, is not double spent. The recepient relies on the work of everybody who produced a block. And in turn, everybody who produces a block relies on the work of everybody before AND on the work of everybody after (who bases their block on his).

This is what I mean by iron-clad guarantee against a double spend.

But if nobody can reject an invalid TX for XCP, that means verifying that something has unspent outputs is an extremely expensive task. And the longer transactions go on, the more expensive this task becomes. And nobody can put in some kind of "block" that could be rejected by others to "proove" that some balances on addresses are now valid up to that point.

This means that in a short time, XCP will become unusable for everybody as the XCP-TX spam rises and everybody is checking the entire bitcoin blockchain over and over and over for XCP-looking TXes to figure out if the coin he thinks he got, aren't actually double spends.

<comment><AUTHOR> Ok I think I understand your point that XCP-invalid-TX spam cannot be filtered at the point of origin like invalid BTC transactions and a motivated attacker could spam the network to make it extremely slow to use XCP.

I think this is a valid concern, even if somewhat mitigated by the 0.0001 BTC transaction fee. I'll post this to the BTCTalk thread and see if someone smarter than me has an idea to defend against it.

</comment></comment></comment></comment></comment><comment><USER3> Can someone "explain like I'm 5" what some of the use cases are for this service? I've been following bitcoin and distributed services for a while and I'm still confused (or lazy).

The website copy is extremely abstract and non-specific. It's in need of a copywriter.

<comment><AUTHOR> Here are a couple of uses cases off the top of my head. This is by no means comprehensive.

* The ability to issue shares in your company/startup and have them be traded from Day 1. Unlike BTCT/BitFunder/Havelock, Counterparty is a P2P protocol and cannot be shut down by regulatory agencies. Kind of like BitTorrent.

* The ability to trade XCP for Bitcoin and vice versa without relying on any fallible counterparty. There is no centralized exchange to steal money from you or fail like MtGox. There is no escrow required to trade with peers because Counterparty guarantees atomicity of transactions.

* The ability to place a variety of different bets. Currently the protocol has support for simple contracts-for-difference (CFDs) and binary Equal/NotEqual bets. Why is this useful? You could for example hedge your exchange rate risk completely by being short a XCP/USD contract. This allows you to hold your wealth in cryptocurrency but have no exchange rate risk. Another use case is sports betting. There was a friendly bet on the Super Bowl, for example.

</comment></comment><comment><USER4> You might need to read this though (I sure did, still don't really get it)

[https://counterparty.co/about/](https://counterparty.co/about/)

<comment><USER5> 2 virtual objects + secure peer to peer exchange of those objects.

what is so hard about getting that...</comment></comment>


Using Machine Learning to Load Balance Elasticsearch Queries - <AUTHOR> https://underthehood.meltwater.com/blog/2018/09/28/using-machine-learning-to-load-balance-elasticsearch-queries/ <comment><USER2> I am biased, as I am working for the same company, but I really appreciate the level of detail that we are sharing here about how we are approaching the query load balancing challenges in Elasticsearch.</comment>


The Rule I Followed to become a Great Developer - <AUTHOR> http://vitalflux.com/one-rule-i-followed-to-become-a-great-developer/

<comment><USER2> I wasn't familiar with this rule until a few months ago, and I'm only starting to implement it. What I find interesting is that it makes code easier to test and initially, simpler to read.</comment>


The New Era of Multimessenger Astronomy - <AUTHOR> https://multimessenger.desy.de/ <comment><USER2> This is a fascinating scientific discovery and i love the visualization on their website!

13 observatories and 5 satellites contributed to the discovery.

If you are interested in the topic and understand German, check out Episode RZ073 of Raumzeit Podcast, topic: IceCube Neutrino Observatory which covers this (no affilication).

[https://raumzeit-podcast.de/2018/06/06/rz073-icecube/](https://raumzeit- podcast.de/2018/06/06/rz073-icecube/)

</comment><comment><USER3> Can someone please explain?

> Neutrinos are the lightest and least interacting of all known particles. > They easily fly through planets, stars and whole galaxies, completely > unaffected. Because of this, neutrinos can reach us from extreme > environments in the cosmos, inaccessible by other messengers.

Two slides later:

> It spies for flashes of light generated when a cosmic neutrino hits an atom > of the ice, sparking a cascade of secondary particles that leave their trace > in the detector.

So why do neutrinos interact with atoms of ice in detectors but pass through planets and galaxies?

<comment><USER4> There are so many of them, that occasionally one must interact with other matter. One in a billion billion billion, sure, but there are trillions of trillions of trillions of neutrinos flying past over a period of time.

Neutrinos are normally created by beta decay[0]. So when they are detected, are they going through "beta reabsorbtion"? Does the (anti-)neutrino hit an electron, form a W boson, then collide with a proton to form a neutron? Or is there another interaction that causes the light?

[0][https://en.wikipedia.org/wiki/Beta_decay](https://en.wikipedia.org/wiki/Beta_decay)

<comment><USER5> The primary channel is the inverse beta decay where an anti-neutrino interacts with a proton to produce a neutron and a positron (via a W boson like you said). It can also just scatter of an electron (via either the W or the Z boson).

But the important point is that it's only via the weak interaction (meaning only via the W^\pm and Z^0 bosons).

<comment><USER4> At first, it sounds like an anti-neutrino can "spontaneously" emit a positron to form a W boson, instead of finding an electron.

On re-reading, it's like the neutrino "bumps" the positive charge out of the proton. Like the neutrino interacting with a down quark will convert them into an up quark and a positron.

Edit: _Inverse beta decay, commonly abbreviated to IBD, is a nuclear reaction involving electron antineutrino scattering off a proton, creating a positron and a neutron.

... The IBD reaction can only be initiated when the antineutrino possesses at least 1.806 MeV of kinetic energy (called the threshold energy). This threshold energy is due to a difference in mass between the products ( positron and neutrino ) and the reactants ( antineutrino and proton ) and also slightly due to a relativistic mass effect on the antineutrino. Most of the antineutrino energy is distributed to the positron due to its small mass relative to the neutron. _

[https://en.wikipedia.org/wiki/Inverse_beta_decay](https://en.wikipedia.org/wiki/Inverse_beta_decay)

To state the obvious, since the Neutrino is massless, its energy can only come from momentum. Hence the distribution of the 1.806 MeV from the Wiki article above:

      P+ to N:  1.293 MeV
      Positron: 0.511 MeV
      total:    1.804 MeV (rounding error?)

</comment></comment></comment></comment><comment><USER6> Man, this is really high production value website that Ice Cube has managed to produce. Would love to have one about the real multi-messenger breakthrough of GW170817!

Now we get one about a milestone for a particular experiment instead : /</comment>


Hello admins of EU AWS systems – DNS outage in eu-west-1 - <AUTHOR> https://status.aws.amazon.com
Looks like there's a DNS outage in AWS Dublin
All DNS request coming from `eu-west-1b` fail completely if you're using the default AWS DNS resolution itself - if you switch to `22.214.171.124` it's all totally fine, but that's of no use if you have private services that need internal AWS resolution
`1a` and `1c` are also affected, but to a much lower extent from what I can tell <comment><USER2> Question for the more experienced in the group - is there a way of caching the DNS locally within you VPC in case something like this happens again? Or does that just cause more issues than it solves?

When I discovered it I tried switching my resolv.conf to 22.214.171.124 but of course none of my internal stuff worked because of how my security group / subnet / ip restrictions are setup internally.

<comment><USER3> I'm not a big user of AWS but I don't know that there's anything stopping you from running your own recursive resolver (in your VPC) that your other machines use (instead of using an external resolver or whatever Amazon provides).

You could, for instance, stand up an instance or two running Unbound, forward to Amazon's own resolvers for internal stuff, and forward to external resolvers for external resolution (or just handle it yourself instead of using forwarders).

<comment><USER4> The main reason to use AWS is so you don't need to do anything except calculate profit from your app, at least in theory. If you need to set up own resolvers or anything else within the AWS there is no reason to use AWS in the first place.

</comment></comment></comment><comment><USER5> It's weird, I see actually most of my issues coming from instances located in `eu-west-1a`, whereas `eu-west-1b` and `eu-west-1c` look fine.

<comment><USER2> I recall reading somewhere many years ago that AWS randomise a, b, c etc per account. Otherwise people have a tendency to pick A and be done with it.

<comment><AUTHOR> That's true - they loadbalance instance creation unless you specifically ask for an instance in a zone

That doesn't apply here though. Instances that are already up are experiencing DNS resolution issues because the Amazon-provided DNS service (which is distinct from Route53) is failing

<comment><USER2> I don't think I explained that very clearly.

What you see as zone A is not what I see as zone A (maybe). When you sign up for an account, AWS assigns your zone A to one of the 3 available zones:

        zone X (real) : your A (virtual) : my C (virtual)
        zone Y (real) : your B (virtual) : my B (virtual)
        zone Z (real) : your C (virtual) : my A (virtual)

That's why people see different outage characteristics between the zones.

Edit: one of the few articles (other than HN comments) that explains it [https://alestic.com/2009/07/ec2-availability- zones/](https://alestic.com/2009/07/ec2-availability-zones/)

<comment><AUTHOR> I've been using AWS for years and this is the first I've ever heard of this.

All I can say is "WHAAAAAAT?!"

You've made me question my reality. Take an upvote for that alone.

<comment><USER6> It's in the documentation [0].

        An Availability Zone is represented by a region code followed by a letter identifier;
        for example, us-east-1a. To ensure that resources are distributed across the Availability
        Zones for a region, we independently map Availability Zones to identifiers for each
        account. For example, your Availability Zone us-east-1a might not be the same location
        as us-east-1a for another account. There's no way for you to coordinate Availability
        Zones between accounts.

[0] [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using- reg...](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions- availability-zones.html#concepts-regions-availability-zones)

</comment></comment></comment></comment></comment></comment><comment><USER7> 6:29 PM PST We have identified the root cause of the DNS resolution issues in the EU-WEST-1 Region and continue working towards resolution.

</comment><comment><USER8> same here :) let's see how long it takes

</comment><comment><USER7> sigh</comment>


Show HN: Juicero+ – juicing without hands or a $400 machine - <AUTHOR> http://juiceroplus.com <comment><USER2> Your images are in a very low definition.

<comment><AUTHOR> well it's a parody site i made in two hours so i dont care</comment></comment>


Free course for building your business in the Digital Space - <AUTHOR> https://opensourcecapital.co/landing1/ <comment><AUTHOR> I put together everything I know about building projects in the digital space into a 10,000 word course.

It is geared towards no-code and brings you from idea through to scaling.

It's broken down into 4 lessons.

You may be familiar with some of the concepts and ideas but I think there is a bit of value in it for everyone.

You can grab it at the link attached :)

<comment><USER2> You do realise that you are infringing on the registered mark of the Open Source Initiative [https://opensource.org/logo-usage- guidelines](https://opensource.org/logo-usage-guidelines)

<comment><AUTHOR> Thanks for this @barbegal. This is just a temporary and is being changed today. Thanks for your input.</comment></comment></comment>


Crafting the perfect container to play with a Headless Chrome - <AUTHOR> https://dev.to/jlandure/crafting-the-perfect-container-to-play-with-a-headless-chrome-ilp <comment><AUTHOR> A tiny Chromium Headless docker image reaching 1 000 000 pulls on the Docker Hub!</comment>


Decentralized Microgridding (2018) - <AUTHOR> https://www.vice.com/en_us/article/vbngmd/decentralized-microgridding-can-provide-90-of-a-neighborhoods-energy-needs-study-finds <comment><USER2> How does something like this work at the density of a town or city, with people living in apartment blocks?

</comment><comment><USER3> One wonders what centralized microgridding would look like.

<comment><USER4> Probably the fever dream of many Oligarchs. :P But in all seriousness:

I think the term "microgrid" started out because that's exactly what they were... small grids in places like islands or rural areas.

We learned a lot about how to make a sustainable grid in this contexts, and so the term sticks a bit, even as we start talking about macro-scale microgrids, or just decentralized generation. Especially because the "micro" element of these new grids plays a much larger role.

Also... regulations, protocols and standards are a form of centralization, which will all play a large role in these "smart" grids.

<comment><USER3> From what I can recall of the Tesla/Edison disagreement, Edison was building microgrids for wealthy individuals. Tesla had solutions that worked over larger distances, and Westinghouse enabled him to do it (starting with the hydro power plant at Niagara Falls)

<comment><USER4> DC, simple and easy. AC, complex and scalable...

Except, my surface level (pun intended) understanding of this, is that it's actually not that simple. AC may have been easier to deal with at high voltages (and thus long distances) at one point, but due to the skin effect and RMS power not determining the safe tolerances for transmission lines (I think this is true), there are some issues with this approach.

Nowadays, it's possible to build safe HVDC systems, I think partially due to the existence of HVDC breakers.

That said, you lose a channel of communication assuming your power is DC, and that signal is used in AC to help shape the generation / switching / etc. We could always try superimposing signals on DC power lines, but I digress.</comment></comment></comment></comment>


Amazon launches a cloud-based robotics testing platform - <AUTHOR> https://techcrunch.com/2018/11/26/amazon-launches-a-cloud-based-robotics-testing-platform/ <comment><USER2> Nice pet project, perhaps I'm in the minority or is there increasingly a disconnect between where we want to be and where we are as an industry?

At one end we are talking about autonomous intelligent robots and then you have what I and everyone I know is actually working on, which I suspect is the same sort of thing you reading this are working on. Oh we might have a thing where we bolt on tensor flow and call it AI, but thats about it.

Or is it just me?

I suspect its like social media, where everyone pretends they are dining at fancy restaurants and jet setting all the time, when really they are just bored.

<comment><USER3> I build industrial robots and work with ROS daily. Yeah. I'm not sure if I see where AI fits in to any of the platforms and products I've been a part of. We just aren't there yet to be worrying about the kinds of things AI may lend itself to. At least that's my opinion.</comment></comment>


Unix System Call Timeouts - <AUTHOR> https://eklitzke.org/unix-system-call-timeouts <comment><USER3> This is done right in QNX. In QNX, everything that can block can be given a timeout, using TimerTimeout().[1] This is useful in real-time code with moderate time constraints. If you're logging to disk, and the disk stalls because of a hardware problem, you may need to get control back so you can keep doing your real-time work. It's more important to, say, issue steering commands than log.

After a timeout, you don't know how the request that blocked ended. If you need that, a multi-thread or multi-process solution is indicated.

In general, QNX is much better at timing, hard real time scheduling, and inter-process communication than UNIX/Linux. In the Unix/Linux world, those were afterthoughts; in QNX, they're core features. Because all I/O is via interprocess communication, it takes an extra memory to memory copy; this seems to add maybe 10% CPU overhead. That's the microkernel penalty, a moderate fixed cost.

[1] [http://www.qnx.com/developers/docs/7.0.0/#com.qnx.doc.neutri...](http://www.qnx.com/developers/docs/7.0.0/#com.qnx.doc.neutrino.lib_ref/topic/t/timertimeout.html)

<comment><USER4> A better solution is to have async syscalls, this helps both the scheduler (you could batch return) and given a proper language the developer as well. The microkernel approach still suffers from the high context switch overhead and IMO it would be better to use something similar to an exokernel while implementing most things in userspace.

<comment><AUTHOR> The problem with asynchronous operations is that there has to be a buffer somewhere. Once you have a buffer, you have to answer the question: what happens when the buffer is full? If the answer is to block, then you haven't really solved anything.

<comment><USER4> This is why you want something like an exokernel.

</comment></comment></comment></comment><comment><USER5> I'd like to point out here that the basic Unix assumption is that we don't care about blocking on local disk. So mkdir() should not have a timeout because it should always be fast enough not to need it. Yes, that's not true, local disk can be very slow, and there's NFS and FUSE, but it's the assumption Unix is built on.

Have you ever had a process waiting for disk that you couldn't kill? Like running "ls" in a big directory, which hangs, and you pound ^C over and over again in vain? That's this same assumption at work here. It's the difference between D and S states on Linux. So if you want timeouts on mkdir(), what you really want is a complete reengineering of the Unix system with the philosophy that local disk IO is interruptible.

<comment><USER6> Pretty much, yes. I think that assumption turned bad a really long time ago?

Interruptible sounds like a very strong requirement. Just being able to wait for the result sanely is enough, I think. A process may then decide to lose interest halfway, but having the result be undefined and fds invalid is just fine, from what I've seen in e.g. node.js/libuv.

<comment><USER7> Keep in mind that local storage is getting lower latency year over year. NVMe was a big step, but there could soon be more hardware memory mapped flash storage (like DIMM SSDs), and 3D XPoint. I think the overhead will soon exceed the operation in terms of cost, especially if it's asynchronous.

<comment><USER8> That assumes uncontended access.

</comment></comment></comment></comment><comment><USER9> You can effectively get wait() with a timeout by setting a handler for SIGCHLD, spawning the child, and using select(). This works on all UNIX and gets you everything signalfd would. And as long as you set the handler before you start the child, you don't have to worry about the self-pipe trick or pselect, since you avoid the race condition. The sigtimedwait() approach also seems fine.

The big trouble you have is that there is no approach to signals that composes; if someone else is waiting for a SIGCHLD via some other means, you're likely to eat the signal they were waiting for. If you're particularly unlucky and both children terminate while you're not scheduled, you'll only get a single SIGCHLD - even with siginfo_t, all that means is _at least_ this child exited, maybe others did too.

Perhaps the better approach is to just spawn a thread for this, which is also the general-case answer for every other syscall, whether it's wait(), mkdir(), readlink(), or sync_file_range(). On the other hand, threads are a "now you have two problems" sort of solution.... I'd like to see a generic API where you can submit _any_ system call to the kernel, as if it were on another thread, and select on its completion.

<comment><USER10> I would agree with you. The most natural way to do it seems to be SIGCHLD and a timeout function like select - when the signal hits, the timeout returns early.

I'd add that the most 'Unixy' way to do it probably _is_ to use `alarm()`, though like threads it is a bit of a "now you have two problems" solution as well. But if you blocked all of your signals, spawned your child, called alarm(), and then did a `sigwait()` on SIGCHLD and SIGALRM I think that would work fairly well and be decently simple. You just check which of the two hits first and then handle them accordingly.

I'd also add that even with the threaded cases it's not actually guaranteed to work. Lots of these syscalls may put the program into uninterruptible sleep, which will prevent you from killing the thread until the action is done (And for lots of things this is a necessity, since otherwise you could leave things in an unknown state. Userspace shouldn't be able to corrupt the file-system just by specifying a weird timeout.).

IMO, like others have said I think the real solution to a timeout for things like `mkdir` is in the kernel: If you're attempting to access something like a NFS mount and it's taking too long, it will fail and you get an EIO, which is not extremely graceful, but does mean that you won't sit there forever - <USER2> will return after a period of time the kernel considers sufficient to know that it's not going to happen.

In userspace you ideally have no idea what you're actually accessing, and thus have no basis for what an accurate timeout should be anyway. It doesn't make sense to pass a hard-coded timeout to `mkdir()` when that value will probably be stupid long for some things and not long enough for others, and that's a much bigger usage issue then certain programs taking a long time on slow filesystems.

<comment><USER11> You may also interrupt most operations using pthread_cancel and a few other pthread functions.

<comment><USER9> pthread_cancel to abort a syscall is usually internally implemented by sending the thread a signal (on glibc, it's one of the real-time signals below SIGRTMIN), causing the syscall to return EINTR. So it's the same logic as SIGALRM, essentially.

(And for some system calls, like certain NFS operations, they aren't interruptible by a signal of any sort other than perhaps SIGKILL.)

</comment></comment></comment></comment><comment><USER12> _> there's a trick to turning these kinds of I/O operations into something you can put into an event loop: you run the desired operation (mkdir() in this case) in another thread, and then wait for the thread to finish with a timeout_

This isn't a trick, this is expected. Calls return when they're supposed to return, or not at all. If you need to return BEFORE the system call is done, you need to do it somewhere else (like in a new thread or process, or node). This is also not limited to I/O, but basically any system call: if you return before it's done, it may break something, so it might not provide a good timeout method.

Something to ask yourself is also why you need to return before the call is done? It's similar to the NFS hard-vs-soft-mounting argument. Soft mounting can cause damage when improperly interrupted; hard mounting prevents this by waiting until the system is behaving properly again, with the side effect of pissing off the users.

<comment><USER13> In some cases, you have knowledge that you're about to do a whole bunch of I/O operations, sometimes all of the same time, sometimes not. (It doesn't really matter.) _Ideally_ , I'd like to transfer this knowledge — that is, the list of I/O operations — to the kernel wholesale, so that it _also_ has complete knowledge of the task at hand, and can better figure out the most optimal way to go about completing that (which it really can't do if it can't see the whole picture). This might mean scheduling disk operations more efficiently, to avoid seeking, or batching multiple network requests into a single packet, etc.

You can't do that with synchronous APIs without hackery, since the very structure of the API is self-defeating when it comes to getting the complete picture to the kernel. If I have a 1000 I/O operations, I do not want to spawn 1000 hardware threads: relative to the amount of information required to describe an I/O op, threads are incredibly expensive.

I don't want the syscall to represent the entirety of the work, simply, the request to have the work performed. The kernel's response is then essentially "Acknowledged, beginning this I/O. Here's a handle/means¹ to obtain the result of the operation." Then, I can batch-request notifications of results through some kernel I/O event queue … e.g., kqueue or epoll.

¹if handles are too much, you could also agree to have it stuck in some sort of queue of results, that might be usable with kqueue/epoll.

<comment><USER12> Do you want this bulk i/o syscall to inform you every time an operation is complete, or in stages? Do you want it to prioritize latency over bulk operations? Do you want it to take up more or less CPU? Will interrupts get thrown each time you query the status? Do you want to know when the operation is complete on spindle, in on-disk cache, in the filesystem cache? Do you want it to handle network filesystems differently? Do you want it to take advantage of multichannel NCQ and other features or implement your own in the kernel? Do you want this new i/o scheduler to affect the rest of the system's i/o, or only your application's? Do you want multiple applications to use different command queues or for yours to trump them (priority) ? Do you want the kernel to implement it's own batch ordering or rely on vendor firmware? (It sounded at first like you were describing vectored i/o but I assume you want something more abstract than that, kinda like a more generalized blk-multiqueue?)

<comment><USER13> All good questions, but none of these seem possible in today's POSIX APIs either. (Most, I feel, probably are best just implemented as "options" to the syscall in either the sync or async view of the world.) The point was more to have async operations be _possible_ , whereas today, they're not.

> _It sounded at first like you were describing vectored i /o but I assume you > want something more abstract than that, kinda like a more generalized blk- > multiqueue?_

Asynchronous I/O, not so much vectored (though vectored is similar, but I want to stay away from that term as most of the APIs (e.g., readv writev) I've seen for that aren't actually asynchronous and are just more efficient user-to- kernel bindings).

</comment></comment></comment></comment><comment><USER14> Shouldn't this work?

\- In the parent, open a pipe. Fork.

\- Parent closes the write side. Child closes the read side, clears FD_CLOEXEC on it, and execs.

\- Parent does a select(2) on the pipe's read side with the desired timeout. If the child exits before the timeout, the pipe becomes "EOF readable." Either way select(2) returns on or before the timeout.

I get that this trick won't work for imposing timeouts on other syscalls (the author points to mkdir(2)) but isn't that the purvey of an RTOS?

<comment><USER11> Poll and epoll also work.

RTOS should prefer asynchronous calls and lockfree queues instead.

<comment><USER7> That sounds like a nightmare. I doubt many mortals could do hard realtime on a fully-asynchronous RTOS.

</comment></comment></comment><comment><USER15> FreeBSD has pdfork, which works with process descriptors instead of signals. No more signal handlers needed, and you can use process descriptors as you'd use file descriptors with calls to poll, select, kevent, &c.

</comment><comment><USER16> If you really need this kind of abstraction for syscalls, I might suggest looking at Erlang, or Golang. Both of these languages and run times provide abstractions for these "bugs" in the underlying APIs.

In Erlang, all I/O goes through things called "ports", and does not block, but it's all asynchronous. There is no reason you couldn't do the same in C -- spin up an external process with a SIGALARM, and maintain a socket that you use, instead of dispatching the syscalls in the same process / thread you run the rest of your code.

In Go, syscalls are scheduled on their own thread (typically), and you can run a goroutine, and wait for the output of the call.

Yes, these bugs exist, and they sometimes make writing code a pain, but they're mostly "solved" if you use the right tool.

<comment><USER17> Go doesn't have a general abstraction for that either. If you put something in a goroutine, you usually cannot cancel it. If you decide to timeout, the goroutine will continue to run. This may not be important for an mkdir(), but this may be for other stuff, like for example for accepting a connection. The not-cancelled goroutine will have an important side-effect on your program (stealing a connection).

Moreover, Go wrapper functions automatically restart a syscall when it is interrupted. Therefore, you cannot cancel a syscall by sending yourself a signal.

This may change in the future when everything will take a context (at the cost of an API change), but this is currently not the case.

</comment></comment><comment><USER18> Yeah, wait() doesn't have a standard way to wait for a timeout. I would wait (with timeout) for the SIGCHLD with some OS-dependent interface (signalfd+poll/select/epoll on Linux, or kevent on BSD).

But: What possible semantics could you want for mkdir() such that a timeout is sane?

<comment><USER19> In a distributed systems problem, almost everything should have a timeout - mkdir and all filesystem operations should be prepared for the possibility that backing store will be slow, unresponsive, or simply broken.

On the other hand, the application layer is probably the wrong place to implement such. Most NFS clients have the option to set a timeout, and system calls that take longer than it instead receive EIO. This isn't perfect, but if you're building an application that needs more complex behavior than that, you probably do need to be building on a more filesystem specific API. Still, there's a point to be made here - There should be a general template or recommendation for what that API is.

<comment><USER20> Exactly. I'm kind of surprised to see the comments arguing that all potentially blocking syscalls should have a timeout field or something. That kind of thing just begs to be factored out into a more general framework.

<comment><USER19> Something similar: [https://golang.org/pkg/context/](https://golang.org/pkg/context/)

</comment></comment></comment></comment><comment><USER21> Why does he say alarm is always the wrong answer? I've used alarm to kill the current process after a user-selectable timeout if it's taking too long, and it seems to work fine.

</comment><comment><USER22> His use case should be solvable with a timer and a kill. Then asking for a timeout version for the syscall is for convenience. But that's a bad guiding principle for system interface design. You want to keep things simple and orthogonal and that rules out a timeout parameter. Maybe a general way to cancel in-progress syscalls would do it?

Anyway, as my late systems professor used to say: timeouts are always too short or too long.

</comment><comment><USER23> In the case of waiting on a child started with exec, I've used the crappy workaround of using the timeout command. It makes checking the return value a bit trickier though...

</comment><comment><USER11> I've found a similar thing. The best way it seems is to open a pipe and poll on it.

</comment><comment><USER24> Not saying the syscall APIs are not broken, but what is a use case for wait() with timeout? Just curious...

<comment><USER13> From the article's first paragraph:

> _The use case is something like this: you spawn a subprocess, and you expect > the subprocess to complete within ten seconds. If it doesn't complete in > that time, you want to treat it as an error (and perhaps kill the child)._

I've had this exact use case myself, several times. It's painful.</comment></comment>