Tehran tracked, captured, studied, copied RQ-170 - <AUTHOR> https://medium.com/war-is-boring/f3fc9b2c087a <comment><USER2> There seems to be a lot of comments to the effect of, "They'll never get their version to work.". First, this probably isn't true and, second, it's not what's really important here.

How many parts inside these drones are off-the-shelf parts manufactured in China? Of the parts that aren't off-the-shelf, how many can be reproduced exactly just by sending a design files and a check to the right place? How much of the software do they now have binaries for? Iran could probably produce a drone configured to run on the U.S. TDRSS system fairly quickly. All they'd have to do is reproduce the drone that crashed exactly. Unfortunately, this wouldn't do them a whole lot of good. The big problem is probably going to be to reverse engineer and adapt the drone's software/hardware so they can be used with Iranian systems, which probably don't exactly have _global_ coverage. It's another challenge still to learn enough from this design for Iran to be able to produce different designs. Copying is a great way to learn though.

The drone's stealth tech may be compromised for a variety of reasons, such as the difficulty of reproducing paint coatings, etc.. This brings us to what is really important. This same stealth tech was already compromised by the Chinese radar installations Iran used to detect and hijack the drone in the first place. Iran is already using Chinese radar systems and is probably not working alone. The opportunity to reverse engineer the latest stealth drone hardware is going to make China (or perhaps Russia) _very_ generous to Iran.

Chinese radar systems can already penetrate the latest U.S. stealth tech given the right circumstances, but now they are likely to get even more effective. Stealth requires such huge compromises from aircraft designs that I wouldn't be surprised if stealth designs are soon abandoned. This incident is not the beginning of the end for Stealth aircraft, it's closer to the end of the end!

<comment><USER3> Stealth will end up as a footnote in the history of military technology just like reinforced concrete. Prior to WWI, reinforced concrete gave rise to an era where networks of large forts could hold against any attack with proper coordination. Then early formulations of "modern" high explosives combined with advances in metalworking gave rise to artillery that could easily blast any fort to pieces.

Stealth worked well against radar systems lacking the computing horsepower needed to filter an aircraft out of the noise. Military hardware lags far behind the capabilities of consumer hardware since the rate of progression has been so rapid while military projects are planned over decades but if systems from 1998 are capable of tracking the latest in stealth technology the next few years will be the bookend of the stealth era.

Personally, I think the future will be an environment where swarms of small drones are thrown against these advanced radar systems in an effort to overwhelm their tracking capability. It'll be an updated strategy along the lines of Baldwin's "the bomber will always get through" ideology. The difference is that drones don't cost lives so the weak point of WWII era bombing strategy won't be of any concern. The winning side will be the one that can afford to keep throwing 1,000 drones at every mission.

<comment><USER4> There will always be more weapons to shoot down each drone, at a lower cost, than there will be drones. The throw 10,000 drones at them approach will not shake the super powers.

Aircraft bombing runs against high powered military nations is long since over. It will never see a return as a broad strategy method.

The swarm strategy won't produce a winning side in any scenario except for ones in which a high-powered military takes on a weaker military. Nothing will fundamentally change about who dominates the global military sphere.

<comment><USER5> Maybe, but you're not considering (a) the value of the target or (b) the effectiveness of drone defense. Even if defense is cheaper than offense, it will never be perfect. If all it takes is one drone to destroy the target and the target is sufficiently high-value, then the 10,000 drone strategy may be worth it.

For comparison, see ICBMs and missile defense.

</comment></comment></comment></comment><comment><USER6> There's a lot of claims in this article, but from what I can tell they're taking the Iranian media's reports entirely on good faith. Given Iran's propensity in the past to stretch the truth(putting it lightly) with regards to their military capabilities, I'd take this article with a huge grain of salt.

This was linked in the Further Reading section, which seems to debunk many of those claims(similar arguments have been made in the comments here):

[https://medium.com/war-is-boring/ed9dd24dffa8](https://medium.com/war-is- boring/ed9dd24dffa8)

> That a drone landed in Iranian hands is still an intelligence failure—the > CIA is widely believed to operate the Sentinel. But one rule of drones is to > not fly anything you're not willing to lose. This is because drones are > finicky and crash at higher rates compared to manned aircraft.

> Aviation Week also reported in 2011 that the RQ-170's sensor package was > already obsolete. And it's unclear if Iran managed to copy the drone's > sensors. Even if the Iranian drone can fly, it doesn't mean it can spy.

<comment><USER7> obsolete for us isn't obsolete for everyone.

plus we are the great satan so anytime they triumph over us its a victory for their leadership, let alone the proof they need to rightly claim we are doing what they claim we do

</comment></comment><comment><USER8> I'm prior Air Force.

Few pieces of military technology are so individually valuable to overall tactical and strategic capabilities that knowledge or possession of such technology would tip the scales one way or the other. An exception would be the nuclear bomb.

We may have embarrassingly given the Iranians a freebie, but their ability to dissect and understand it is far different from their ability to reproduce or manufacture it for any tangible military advantage.

Keep in mind, the global dominance of the U.S. Air Force isn't rooted in any one aircraft or technology, rather it comes from the _massive_ network of supporting technologies (e.g. global radar networks, manufacturing technologies, communication and satellite tech, etc.) and personnel (e.g. intelligence agencies, engineers and scientists, pilots, etc.) that support such advanced aircraft.

<comment><USER4> People seem to miss the part where Iran won't be able to build a thousand of these, supply them, fly them, train for them, maintain them, upgrade them, equip them with the appropriate technology, and build the next generation of them.

It'd be like saying: oh look I stole an Intel processor, and was able to figure out how it works. Now all I need is to build a fab and a global supply chain so I can churn out thirty million of them. Or hey, I know how PageRank works, Google better watch out.

</comment></comment><comment><USER9> Strange comments here. It saddens me that the US lost a drone like this. Maybe not everyone is American here, but the technology in this vehicle will be in China's hands Russia's hands etc. The whole world will miss the old world order some day. By that I mean today's world order. the US will some day will not be able to police the world for one reason or another. Then the world will realize how valuable what the US stood for despite its shortcomings.

<comment><USER10> "Then the world will realize how valuable what the US stood for despite its shortcomings."

The West, maybe.

Not if you're living in a Middle Eastern country, with oppressive rulers kept in power by US funds, forcing the country to sell its oil resources cheaply.

Not if you're living in Eastern Ukraine, where the US toppled an elected government to install a government that is hostile to your people. [http://www.presstv.com/detail/2014/03/25/355978/nuke-8-milli...](http://www.presstv.com/detail/2014/03/25/355978/nuke-8-million- russians-in-ukraine/) You could pretty much tell her attitude to them by the way she suggested the 8 million ethnic Russians living in Eastern Ukraine should be nuked.

Not if you're living in Iraq, Syria, Lebanon, Libya, Somalia, Sudan, Iran. [https://www.youtube.com/watch?v=9RC1Mepk_Sw#t=79](https://www.youtube.com/watch?v=9RC1Mepk_Sw#t=79) All countries where the U.S. has sought to destroy domestic, independent, public institutions if and when they arise.

If you're living in an African country where the U.S. has previously tried to destabilize your government, and where China instead have offered to build infrastructure for your country, you might not be so sad that the US has lost a drone, which it was probably using for spying purposes.

If the world was Panem, then U.S. is the Capitol, and its closest allies are the Capitol's adjacent districts.

Of course, you're probably from the U.S., if not, then probably, U.K or Australia, or N.Z. Of course you'd have something to lose if the US will some day not be able to police the world.

EDIT: What's the downvote for? A lack of citations?

<comment><USER11> The US did not bring down an elected government in Ukraine. It was taken down by Ukrainian protesters. While I admit that having the threat of force cause elected albeit very corrupt politicians to flee is very sketchy its a million times better then what happened in Crimea and eastern Ukraine and especially in Crimea Russia orchestrated quite a bit of it.

<comment><USER10> Here's a video of Victoria Nuland of the Department of State claiming U.S. has spent $5 billion since 1991 to 'build democratic institutions' in Ukraine. Take note of the sponsorship flags by Chevron and ExxonMobil in the background. Note how she uses the word 'invested'.

[https://www.youtube.com/watch?v=2y0y-JUsPTU#t=448](https://www.youtube.com/watch?v=2y0y-JUsPTU#t=448)

IMO that is evidence of U.S. interference in Ukraine's political affairs.

It is crazy to suggest U.S. is not involved in encouraging Ukraine's aspirations to join NATO.

On one hand the U.S. supports a government installed via a coup, toppling the previous elected government, and on the other give no comments on the Kiev government sending army units to crack down on Anti-Kiev protesters. [http://en.wikipedia.org/wiki/Mariupol_standoff](http://en.wikipedia.org/wiki/Mariupol_standoff)

The U.S. media calls armed men storming Pro-Russia government buildings "protesters", while armed men storming Pro-US government buildings are "separatists", "militants". I'm not for U.S. or Russia, I see the Ukraine situation for what it is, a small country whose people are being manipulated by larger ones.

Russia's move into Crimea isn't even illegal, it was specified clearly in the terms of the treaty leasing the Russia naval base. It was also a defensive one. If Ukraine would join NATO, the lease for the base would terminate, and Russia will lose a naval port. Crimea was also historically Russian, the fact that Ukraine had it after Soviet Union's collapse was pure accident; the Soviet Union 'awarded' the Ukraine state with Crimea in 1954 as a meaningless proclamation, probably for domestic propaganda purposes. It was the people in Crimea who decided to join Russia. ([http://en.wikipedia.org/wiki/File:UkraineNativeLanguagesCens...](http://en.wikipedia.org/wiki/File:UkraineNativeLanguagesCensus2001detailed- en.png)) U.S. (and probably all governments if given the chance) favors self- determination when it favors them (e.g. [http://en.wikipedia.org/wiki/Falkland_Islands_sovereignty_re...](http://en.wikipedia.org/wiki/Falkland_Islands_sovereignty_referendum,_2013)), calls them illegitimate when it doesn't.

<comment><USER12> $5 billion is less than $10 per Ukrainian per year. That doesn't necessarily say much about how much influence the money would buy, but it can be interesting to try to stick a denominator under big aggregate numbers.

($5 billion / 45 million -> $111, I left lots of wiggle room for changes in population)

</comment></comment></comment></comment></comment><comment><USER13> Kind of a strange exercise they are doing. Not that I can blame them, I'd be pulling it apart to see what makes it tick myself.

As a practical exercise, however, this is rather futile; they aren't going to be able to replicate the stealth technology, nor the electronics, nor the engine...or anything else important, for that matter.

What they do have, however, is a large scale model making hobby. Maybe they should contact Airfix or Revell for a licensing deal.

<comment><USER14> The best way to find weaknesses in a system is to analyze, and then to see if you can replicate it. This has been done just about forever with weapons systems. Piloting a drone will teach you about the blind spots of a drone, that's golden knowledge if you need to take them down.

<comment><USER15> If Iran is approaching this the same way they have approached "replicating" U.S. aircraft carriers then I don't think the West would have much to worry about with this. So I suppose that's the big question here, what exactly is this replicated drone—an accurate copy of at least the airframe, or a propaganda piece designed to appeal to the local populace?

<comment><USER14> Replicating an aircraft carrier versus replicating a drone is a challenge of a totally different level, never mind the practical applications. Iran is much more likely to encounter a US built drone on its territory than it is to encounter an aircraft carrier within the 12 mile zone.

Most likely it is more than you think it is and less than they would want, and its only practical uses right now are analysis of soft spots and propaganda (and that's working well by the looks of it).

</comment></comment></comment></comment><comment><USER16> _Sentinel has two separate hard drives, both encrypted. After deciphering the hard drives, the IRGC-AF_

And that's how we know most of what the Iranian military says regarding the Sentinel is completely fabricated. Correct me if I'm wrong, but wouldn't it be nearly impossible for them to decrypt a properly encrypted hard drive unless they had an powerful supercomputer?

<comment><USER17> Depends entirely on why and how it was encrypted. If we are talking about the flight control software, the airplane must(edit,not necessarily) have the keys to boot clearly.

If we are talking about recorded data, it's less clear. The imagery data may be encrypted to provide access control after the drives are removed from the plane and handed somewhere else. In which case, there's little reason the drone wouldn't have the decrypt keys.

You could encrypt the data with a public key and store it so that the drone itself can't read it. About the only reason to do this would be to ensure if it was captured, the target wouldn't see what was on it. I'd suspect the way the military planed to handle that eventuality, however, would use on an exothermic random number generator (a.k.a. a self destruct mechanism/ explosive charge) rather than encryption. There's a lot of way more sensitive thing on the aircraft than just what it took pictures of that you need to destroy.

<comment><USER18> > If we are talking about the flight control software, the airplane must have > the keys to boot clearly

My _laptop_ doesn't have the keys to decrypt its own hard drive; it boots to initrd and waits for my smart card, which does the decryption. I would imagine the military can do better.

<comment><USER17> True. On the other hand, the military may need to be able to reboot the thing remotely/automatically, in which case it's possible they don't want to risk having to send the keys first.

<comment><USER19> If I were to design such a system, booting to full functionality would require human intervention prior to launch, but would allow a reboot to a separate partition with enough functionality to fly the drone back to base. Data, as was pointed before, would be asymmetrically encrypted and would require a separate key that would never reach the drone.

</comment></comment></comment></comment></comment><comment><USER12> The money quote:

 _At the end of the presentation, Hajizadeh said that Iran's 1:7-scale copy of the drone has already flown, while half-scale and full-scale models would fly this year_

So they haven't finished matching the capabilities of it yet (matching the capabilities is sort of implied by "copied").

</comment><comment><USER20> I'm part way through James Fallows' "China Airborne" and he talks about what it would mean for China to have an airline industry like the US. It means it would have the technical know-how in designing engines, manufacturing high- precision parts for engines, advanced metallurgy for body, wings, etc., advanced electronics, and software to tie it all together. In addition, it must have the organizational ability to direct all the flights around the country and do advanced scheduling (computationally difficult) to minimize wait time, etc.

China is part way there but not completely. It will be a while. Even their fighters today use Russian engines.

[http://en.wikipedia.org/wiki/Chengdu_J-10](http://en.wikipedia.org/wiki/Chengdu_J-10)

If someone is going to copy the RQ-170 and use it operationally the same way the US does, it won't be China and much less Iran anytime soon.

</comment><comment><USER15> Similar to how the Russians bootstrapped their strategic bomber (by copying, rivet-for-rivet, some interned B-29s).

<comment><USER21> >(by copying, rivet-for-rivet, some interned B-29s)

Not quite. The Tu-4 _was_ a very close copy of the B-29, but the sheet metal the Soviets had available was slightly thicker (and thus heavier)than American sheet metal, which required some engineering to work around.

<comment><USER14> It was the difference between 'metric' and numbered gages that had this effect. They took the next standard size up but still ended up with a plane that weighed almost the same as the B-29. Makes you wonder how much weight the B-29 could have still saved.

</comment></comment></comment><comment><USER22> Very interesting. I would love it if the Iranians open-sourced the drone that they captured. :-)

<comment><USER23> Put it up on Kickstarter. iDrone ;)

</comment></comment><comment><USER24> The bottom line is that you can't fly a bunch of planes around an airspace without losing a few, and the people who build and fly these planes know this.

Second, the technology behind bare-bones drone surveillance isn't that special. For $250 you can get an "FPV" remote control plane (first-person view) that lets you fly around and see a real-time video-feed from the plane. Of course the RQ-170 is a million times more powerful than a Walmart toy, but the fact of the matter is that any government that wants to can put surveillance drones in the sky.

</comment><comment><USER25> A bit of a click bait headline. The end of the article summarizes by stating "even if the Iranian Sentinels fly on the schedule, they probably won't be truly stealthy." The intent of the RQ-170 is the be stealthy, a copy that's not stealthy isn't a copy. It's propaganda.

</comment><comment><USER26> I think their copying is more propaganda for them than anything else. Just look at how they're displaying these models for show in a plastic tent. Really?

</comment><comment><USER27> I'm more disturbed that they managed to remotely "trick" a US drone into taking new orders.

Imagine if that was a Reaper.</comment>


New Apple music subscription $10/month? - <AUTHOR> http://arstechnica.com/apple/news/2010/10/new-apple-music-subscription-rumor-pegs-price-at-10month.ars

<comment><USER2> New _rumoured_ Apple music subscription: $10/month. (The Ars Technica headline says it's a rumour. The HN headline doesn't. Naughty.)

<comment><AUTHOR> Agreed: Added a '?'...</comment></comment>


French Scientists Claim to Have Created Metallic Hydrogen - <AUTHOR> https://www.universetoday.com/142654/french-scientists-claim-to-have-created-metallic-hydrogen/ <comment><USER2> > "As a synthetic material, metallic hydrogen would also have endless > applications. First off, it is believed to have superconducting properties > at room temperature and is meta-stable (meaning that it will retain its > solidity once it has been brought back to normal pressure)."

Will it burn? Detonate? Fascinating.</comment>


Bayesian Optimization with Uncertainty - <AUTHOR> https://blog.sigopt.com/posts/bayesian-optimization-with-uncertainty <comment><AUTHOR> Hi, I'm one of the founders of SigOpt (YC W15). This is part 2 of a 3 part series we'll be doing on uncertainty in modeling and optimization (Part 1 here [0]).

Let me know if you have any questions about this post or SigOpt in general. Gustavo is a Research Engineering Intern with us and wrote this post with our research team lead, Michael Mccourt. If you're a student looking for internships please check out our careers page [1]. Our platform is also free for academics [2]. You can find more of our research (including NIPS, ICML, AISTATS, etc papers) here [3].

[0]: [https://blog.sigopt.com/posts/modeling-with- uncertainty](https://blog.sigopt.com/posts/modeling-with-uncertainty)

[1]: [https://sigopt.com/careers](https://sigopt.com/careers)

[2]: [https://sigopt.com/edu](https://sigopt.com/edu)

[3]: [https://sigopt.com/research](https://sigopt.com/research)</comment>


Ask HN: Anybody want a hosted email API? - <AUTHOR>

Its really hard to set up an email host, I have tried doing it myself and it was really too crazy to set up. I had a problem where I was serving email across about 40 domains and I needed something that would work for everyones different DNS settings.
So we outsourced, got it done but its still very clunky, its hard to administrate and monitor and I couldn't build on top of it.
I was thinking of maybe building an API service where people can build their own email hosts really easily without buying hardware, getting a system admin etc.
If you dont know, IMAP and POP are so hard to work with programmatically but if you had an API where you just ask for /users/134234/mailbox and got all your data, you pretty much just cut out 2 months of programming.
Plus: People could build apps on top, like shopify Plus: We wont have to rely on google anymore! Plus: Google doesn't seem to let you do this or anything similar, I might be mistaken though.
Just to clarify, this is not a transactional email service, its a fully hosted imap server.
Thanks! <comment><USER2> It's not clear where you're trying to innovate here. IMAP mailboxes on my domains are hosted by Rackspace Mail [1] at $1/mailbox/month. I don't use any other Rackspace services, just point my MX records there so I don't have to manage mail servers and spam filtering myself.

Where I need an API for both inbound and outbound mail, there's SendGrid and SendGrid Parse [2], among others.

1: <http://www.rackspace.com/apps/email_hosting/rackspace_email/>

2: <http://sendgrid.com/docs/API_Reference/Webhooks/parse.html>

<comment><AUTHOR> At the core I will be building an API that hooks straight into an IMAP server. So instead of making an IMAP call from your application you make a much faster API call.

I always thought that Sendgrid was a transactional email service, not so much a business oriented and more full featured email server.

This would make building new applications that use email as a feature much faster to build because you only have one endpoint to hit.

</comment></comment><comment><USER3> The problem is that mail-clients speak IMAP - if you present your own new API you'll need to wrap it in an IMAP-presentation layer before anything can use it.

Sure the simple case of reading/polling/creating mails is not difficult, so your API idea seems attractive initially. But when you say setting up IMAP is hard I suspect you're getting in over your head - IMAP is not hard, even at scale, certainly not compared to writing a damn IMAP-server (I've done it.)

(PS. Besides your API will need more granularity than "GET /user/foo/mailbox" because my mailbox is _very very large_.)

</comment><comment><USER4> <http://context.io/>

</comment><comment><USER5> Why not just use Google Apps/FastMail/any other IMAP provider?

<comment><AUTHOR> Other providers don't allow you to extend the platform and if you tried doing it through imap, its sometimes slow and becomes unmanageable to interpret the mail it gives you.

I am building this for my own webmail project and it really helps to have a fast api so I don't have to worry about the data.

Having an api call to build a new mailbox would be quite handy for small web hosting companies too.</comment></comment>


Revolights: An innovative bike light solution - <AUTHOR> http://www.kickstarter.com/projects/revolights/revolights-join-the-revolution

<comment><USER2> Another advantage they don't mention: revolights should be easier to secure than bracket-mounted lights. Just pass a cable through the wheels, like you probably already do.

Not so excited about the external battery though. But they're apparently working on something that would power the lights via wheel rotation.</comment>


KickStarter: Doing versus Planning - <AUTHOR> https://medium.com/@floathub/kickstarter-doing-versus-planning-21c057fdfc98 <comment><USER2> This reminds me of our experience as well; seems like in the modern age of the internet, it is really hard for a quality engineering signal to break out over marketing hype.

Might be something more profound about people's lack of trust in institutions. Happy 4th :-)

<comment><USER3> I disagree, at least partially. While it might be hard to get out over something short like a KickStarter campaign, eventually quality always matters more than hype. But it can take a while to get there. We all know cases where the lack of good marketing means a great idea runs out of runway before it has a chance to really get noticed.</comment></comment>


Ask HN: How do you improve skills of programmers in your team? - <AUTHOR> As a lead programmer, which approaches of improving skills in your programming team had the most positive feedback? I presume some activity not related to day-to-day job that may be considered boring, especially by junior developers. Local hackathons? Doing some over-the-weekend collaborative projects? One-hour-a-day projects? Forcing to reading some tech books? Examinations? Providing local classes, with or without homework? Going to external conferences?<p>It's interesting to hear your experience and attitude to these approaches. <comment><USER2> I like to teach the more junior members of my team.

They may have a very basic understanding of some library or concept. I sit them down in a room with a whiteboard and go over the concepts.

I usually have a brief 1-2 page handout with some code examples as its easier to read just from the paper rather than having to write all of the code on the whiteboard.

</comment><comment><USER3> You (or your predecessor) hired your team based on the skills they already have. Focus on improving performance, not skills.

If you try to force your employees to develop their skills in a certain direction, even if you think it's a win-win, they are going to resist or even quit. Entire teams have quit their company based on a programming language change. People don't like giving up any autonomy when their level of autonomy is already low. But chances are, they will respond positively to pressure on things like improving performance benchmarks, turnaround time on requirements, test coverage, resource consumption, etc. if you approach it in a uniform and consistent way. Programmers see themselves as professionals and experts, and if you raise your expectations, they will meet them if they can.</comment>


Abusing new technologies to create ASCII tables - <AUTHOR> http://ionicabizau.net/blog/7-le-table-create-ascii-tables-with-node-js

<comment><USER2> *ANSI... the box-drawing characters aren't ASCII.

<comment><USER3> *Unicode… the box-drawing characters aren't ANSI.

(The confusion comes from the MS-DOS's ANSI.SYS driver, which provided some ANSI-compatible control sequence handling, together with IBM Code Page 437.)

<comment><USER2> Ah, yes, good point.

</comment></comment></comment><comment><USER4> Now we just need an easier input method.

<comment><AUTHOR> It can be any visual editor - a HTML page (with some js) or even some CLI tools.</comment></comment>


Tell HN: Atrium Academy – free fundraising bootcamp March 24-25 - <AUTHOR> Founders who are fundraising: I'm hosting our next Atrium Academy - a free, two-day event March 24 and 25 for entrepreneurs raising their Series A in the next 1-3 months. You'll get 1-1 help around pitching, fundraising, and legal negotiation from seasoned entrepreneurs and top VCs. Since starting this three months ago, we've had 6 founders receive term sheets. Alumni have continued perks and mentorship from the Atrium network as they grow too. Apply here and share this with other promising founders (best to get a referral from an atrium academy previous attendee/angel/VC investor): https://www.atrium.co/academy/. <comment><USER2> Great feedback and advice from VCs and peers! It never seems to amaze me that after a single conversation, it can change what you should be focusing on for your startup. Also, a great way to get a foothold in the valley if you're outside SF.

#FirstCohortAttendee

</comment><comment><USER3> Any alumni here? Would love to get a referral if possible</comment>


Superfluous Patterns of JavaScript - <AUTHOR> http://h3manth.com/new/blog/2015/superfluous-patterns-of-javascript/

<comment><USER2> These strike me as more syntax tweaks than "patterns".</comment>


Bottle Cap Thread Calculator - <AUTHOR> https://www.noupoi.net/bottle-cap-thread-calculator.html <comment><USER2> Seems cool, I'd love it generated or was OpenSCAD code though

<comment><USER3> You will likely be interested in this then,

        https://github.com/adrianschlatter/threadlib

Lifted from the comments of this HackADay article,

        https://hackaday.com/2020/07/12/learn-the-secrets-of-matching-bottle-cap-threads-to-one-another/

</comment></comment><comment><USER4> why the heck does the spec cost money</comment>


Show HN: Bored Chat – one-on-one chat app where all chats are public - <AUTHOR> Bored Chat's a one-on-one chat app where you meet people...where all chats are public. The idea is that you can suss someone out before you chat to them. I.e. see if they're interesting or boring or a creep etc. You can tell a lot about someone's behaviour by seeing how they interact with others.
Currently I have a small little community happening.
iOS only at this stage: https://itunes.apple.com/us/app/bored-chat-chat-with-strange...
Keen to hear feedback and suggestions :). <comment><USER2> Neat. Reminds me of diewithme:[https://techcrunch.com/2018/02/02/die-with-me- is-a-chat-app-...](https://techcrunch.com/2018/02/02/die-with-me-is-a-chat- app-for-sharing-your-phones-last-gasp/)

Is your chat app asynchronous? As in sending a public message for an offline user?

<comment><AUTHOR> > Is your chat app asynchronous? As in sending a public message for an offline > user?

Ah, so DieWithMe is an interesting, but different concept.

With Bored Chat, it's simply a chat app where you interact via an iMessage like UI. The difference is that all your chat logs (your chat history) are viewable to the world on your profile page...and on a main "activity" feed.

So Bored Chat is basically chat, but with no privacy whatsoever!</comment></comment>


Typed, modular macros for OCaml - <AUTHOR> https://oliviernicole.github.io/about_macros.html <comment><USER2> This is awesome, and a big step forward for OCaml. Only criticism is that it seems difficult/awkward that you need different functions/values for each phase. In other staged macro systems (Scala/LMS, Rust/Compiler Plugins) you can refer to the same functions in any phase, and you can also splice values from the staging phase into the runtime phase without explicit use of a function like Expr.of_int.

<comment><USER3> There are some difficulties with moving arbitrary values between phases: it's easy to move an int or a list, but what about a mutable reference or a closure?

However, I don't think this will ultimately be a problem in practice, for two reasons. First, global values can be used in different phases via "module lifting". Second, there's a separate proposal for adding overloading to OCaml in the form of modular implicits:

[https://www.cl.cam.ac.uk/~jdy22/papers/modular- implicits.pdf](https://www.cl.cam.ac.uk/~jdy22/papers/modular-implicits.pdf)

Modular implicits will make it possible to use a single overload function ('lift', say) in place of a family of functions 'Expr.of_int', 'Expr.of_float', etc., which will make things much less awkward. And it's only a small step from there to having 'lift' called implicitly/automatically at appropriate points. Here's a message from an earlier discussion with a few more details:

[https://sympa.inria.fr/sympa/arc/caml- list/2015-05/msg00032....](https://sympa.inria.fr/sympa/arc/caml- list/2015-05/msg00032.html)

<comment><USER4> > There are some difficulties with moving arbitrary values between phases: > it's easy to move an int or a list, but what about a mutable reference or a > closure?

Wasn't this already answered by "Closing the Stage" [1]?

[1] [http://lambda-the-ultimate.org/node/2575](http://lambda-the- ultimate.org/node/2575)

<comment><USER3> "Closing the Stage" is about a different interaction between staging and closures: that (with some care) staging constructs can be elaborated into closures in an unstaged calculus.

The problem with moving arbitrary values between phases with macros is that values can refer to bits of the runtime environment that cease to exist when compilation is finished.

<comment><USER4> > that (with some care) staging constructs can be elaborated into closures in > an unstaged calculus.

Exactly, which seems to provide the necessary semantics for references that you mentioned. Clarifying staging semantics for difficult abstractions like refs by elaboration into well understood closure semantics was the point of the paper.

</comment></comment></comment></comment></comment><comment><USER5> Can someone tell me why I would want statically typed macros? Why not just simply type check after macro expansion phase? Seems a lot simpler and more intuitive.

It strongly feels like a case of everything looks like a nail when you really like hammers (static typing), but I am willing to be proven wrong.

<comment><USER6> There may be a bunch of other reasons, but the one that springs to mind is localisation of errors. You'd ideally like to know about any mistakes you've made in your macro before you run your macro. And when you do learn about a mistake you'd like the error to describe where the mistake is in macro-land rather than generated-code land.

<comment><USER7> Why just type errors? You might want to trace any error that occurs at runtime back to the original source code anyway.

<comment><USER6> I agree, you may wish to do that but I was answering the question "What is the utility of typed-templates", not "contrast type and runtime error tradeoffs".

</comment></comment></comment></comment><comment><USER8> This seems well done, the sort of design I'm pushing for Rust and Haskell.

The money question is: does it work with cross compiling? This is a proper phased design so it should, but that doesn't mean it doesn't.

<comment><AUTHOR> Well what macros only ever do is generate OCaml ASTs, so the should work whatever the compilation target is.

<comment><USER8> Things like the ^ quoting get more complex. You need to quite the target's version of the module.

<comment><AUTHOR> Oh I see what you mean. For now, we have made the choice to compile static code to OCaml bytecode, whatever the compilation target is. While this enables the use of macros regardless of the target (e.g. it works in `ocamlopt`, the native compiler), it does make it necessary to compile a module to bytecode if you want to lift it. It's not a big deal with an adapted build system, but a distant future we might support native compilation of macros on some architectures.

</comment></comment></comment></comment><comment><USER9> Having never tried OCaml, what advantages does it have over other languages? Two languages I don't know seem more interesting, Clojure and Rust. Generally I work in Swift these days.

<comment><USER10> IMO, the single most interesting thing about OCaml is its object model. On one hand, you have a fairly conventional setup with classes, methods, inheritance, generics etc. On the other hand, the entire system is structurally typed, with pervasive type inference. Recycling bits from my earlier comment on the subject...

        # let f obj x y = (obj#m x)#n y;;
        val f : < m : 'a -> < n : 'b -> 'c; .. >; .. > -> 'a -> 'b -> 'c = <fun>

Here I defined a function taking 3 arguments, called method m on obj, passing x as argument, then called method n on the result of that, passing y as argument.

Note the inferred type signature for this: the first argument is of type <m : 'a -> <n : 'b -> 'c; ..>; ..> \- i.e. any object that has a method named m (and possibly some other unspecified members - that's what ".." means), with said method m having a signature that allows us to pass some 'a, and returning another object of type <n : 'b -> 'c; ..> \- i.e. something with a method named n that accepts 'b and returns 'c. 'a and 'b' are then used as types of x and y, respectively, and 'c is the the result of f.

<comment><USER11> This is actually row polymorphism rather than structural subtyping. The difference is rather small, but important:

Under structural subtyping (or any kind of subtyping, for that matter), the subtyping relation admits a rule called "subsumption", which can lead to information loss. Consider this behavior, under structural subtyping:

        fun f {x : real, y : real} = {x = 2 * x, y = 2 * y};
        (* f : {x : real, y : real} -> {x : real, y : real} *)

        f {x = 2.0, y = 3.0, z = 4.0};
        (* it = {x = 4.0, y = 6.0}; *)

Above, when `f` is applied to a subrecord with an additional `z : real` field, the result is missing the extra field! This is because the subrecord was _subsumed_ by the (super-) record type taken by `f`, leading to information loss. This is the reason why you often see people using up-casts in languages like Java (especially before generics were introduced) and Go (which is lacking generics for some befuddling reason): up-casts allow you to circumvent the type system and force a particular value to take a particular type, thus recovering the lost information, but can be unsafe.

Under row polymorphism, however, you can achieve basically the same effect, but you get to save any unused/irrelevant type information inside of a _row variable_ (the `..` in your example) and include it in the result. Then, no information is lost, and no safety is compromised:

        fun f {x : real, y : real, R} = {x = 2 * x, y = 2 * y, R};
        (* f : {x : real, y : real, _ : ..a} -> {x : real, y : real, _ : ..a} *)

        f {x = 2.0, y = 3.0, z = 4.0};
        (* it = {x = 4.0, y = 6.0, z = 4.0}; *)

Users of languages like Lua or Python might be reminded of "duck typing", and not without reason: row polymorphism enables many of the same programming patterns/techniques/abstractions in typed languages that are enabled by duck "typing" in untyped ones. It's a shame that more typed languages don't incorporate the idea, because it's a really wonderful one, and it can lighten the relative "burden" of writing typed code in some cases :)

The folks at INRIA has come up with some really neat stuff, and, if you haven't already, it's definitely worth it to check out their publications, too!

<comment><USER10> Thank you for the clarification!

And, indeed, "static duck typing" is exactly what I'm thinking about when I'm dealing with OCaml objects.

FWIW, while it was relatively obscure, TypeScript is currently popularizing this concept (albeit with a lot less type inference). Although they also refer to it as "structural subtyping":

[https://www.typescriptlang.org/docs/handbook/type- compatibil...](https://www.typescriptlang.org/docs/handbook/type- compatibility.html)

But if I understand correctly, they're basically doing the same thing that's going on here - the only difference is that the ".." is implicit in any TS interface type. Or am I wrong?

<comment><USER11> > Thank you for the clarification!

No problem! The difference between the two was something I personally struggled with at first, so now I feel like I've got a pretty good grasp on it. Funny how that works ;)

I don't see anything here in the TypeScript docs that looks like row polymorphism. Without a row variable _somewhere_ in the type (even if it's inferred), I'm not sure it could be considered row polymorphism. However, they _could_ be doing something fancy with their notion of type compatibility that avoids information loss without introducing row variables (which, I think, would similarly imply a lack of subsumption[0], and _might_ not be sound, but the docs admit the system is unsound in places). It's also possible that _every_ record type in TS carries an implicit row variable.

The important part is the (lack of) information loss, and good way to tell would be to construct a test case like the one I wrote above and see if you wind up losing your `z` field (information loss => subsumption => "by-the- book" subtyping).

This all reminds me that I've been meaning to spend some time with TypeScript :)

[0]: I've known type theorists who claim that without subsumption, there _is_ no subtyping. I disagree because I think the fundamental/important part of subtyping is the subtyping relation itself.

<comment><USER10> I actually didn't fully understand your example originally (I missed the part where ".." now has a name!). I grok it now, and yeah, TS doesn't have it - there's simply no type-safe way to write a function like that (of course, TS being a strict superset of JS, you can still write it, it just won't typecheck).

It appears that this was discussed in the context of supporting the ES7 "spread" operator, since row variable is the obvious typed counterpart to that:

[https://github.com/Microsoft/TypeScript/issues/2103](https://github.com/Microsoft/TypeScript/issues/2103)

but they ended up implementing just the operator, without a way to reflect it in a function signature:

[https://github.com/Microsoft/TypeScript/pull/11150](https://github.com/Microsoft/TypeScript/pull/11150)

However: "I expect a spread type or something similar to be available in future versions of TypeScript."

<comment><USER11> Ah! Good to know :) TypeScript seems like a neat language, and I've got a JavaScript project coming up so I intend to look into it more deeply soon.

Yeah, in my example, `R` is the row variable itself, and it represents all those other, irrelevant values in the record, while I gave it the name `_` in the type itself, since rows themselves don't have field labels, but rather represent sets of labels. In ML, the labels of the _fields_ are part of the type, but the labels of the _rows_ are _not_ part of the type. Then, `..a` is a type variable representing the type of the row (it gets a distinct name because you can imagine a function which takes two separate records with two separate rows[0], or even a function which takes two records but constrains them to the same type with a single row[1]), and is akin to a type variable used for parametric polymorphism, such as `'a`, and, in Standard ML with eqtypes, `''a`.

To the best of my knowledge, the term "row" comes from _The Definition of Standard ML_ , where the grammar given for the core language includes productions called "TyRow", "PatRow", and "ExpRow", which correspond to the syntax of record types, patterns, and expressions, respectively, but only the part in between the `{` and `}`:

        TyRow  <- Label `:` Ty [ `,` TyRow ]
        PatRow <- Label `=` AtPat [ `,` PatRow ]
        ExpRow <- Label `=` AtExp [ `,` ExpRow ]

(The `At` prefix means "atomic", and has a specific meaning in the definition.)

So, then, a row _variable_ is one which literally holds a sequence of label×type or label×value pairs. It sometimes even _feels_ like it's a metavariable which holds a branch of the syntax tree the way a variable in a macro does, which might be another reason why row polymorphism feels so cool ;)

Sorry if I'm boring you at this point, I just find this stuff really fascinating and fun :)

———

[0]: Any two records of any two record types, as long as they each have both an `x : real` and a `y : real`, with _intentional_ information loss:

        f : {x : real, y : real, ..a} -> {x : real, y : real, ..b} -> {x : real, y : real}

        (* f {x = 1.0, y = 1.2, z = 1.3} {x = 1.0, y = 2.0, w = 42}; *)
        (* ^ typechecks okay!
         * row type `..a` can be different from row type `..b`,
         * no problem
         *)

[1]: Any two records which both have the _same type_ , which can be any record type that at least has an `x : real` and a `y : real`:

        f : {x : real, y : real, ..a} -> {x : real, y : real, ..a} -> {x : real, y : real, ..a}

        (* f {x = 1.0, y = 1.2, z = 1.3} {x = 1.0, y = 2.0, z = 0.0}; *)
        (* ^ typechecks okay! *)

        (* f {x = 1.0, y = 1.2, z = 1.3} {x = 1.0, y = 2.0, w = 42}; *)
        (* ^ type error!
         * the second record has a different type than the first,
         * but the type of `f` demands that they be the same type,
         * since the single row `..a` must match itself
         *)

<comment><USER10> A largely unrelated question: do you know why most ML dialects have this weird scoping rule for record members, where they share the same scope as the record itself (and so another record cannot easily reuse the same field name)? It seems awfully inconvenient, and it feels like it could be trivially resolved by allowing reuse with an explicit disambiguation syntax. Or am I missing something?

<comment><USER11> Which dialects? Both Standard ML and OCaml can handle multiple record types with homonymous fields just fine:

        $ poly
        Poly/ML 5.5.2 Release
        > type point2 = {x : real, y : real};
        type point2 = {x: real, y: real}
        > type point3 = {x : real, y : real, z : real};
        type point3 = {x: real, y: real, z: real}
        > let val pt = {x = 1.0, y = 2.0} in #x pt end;
        val it = 1.0: real
        > let val pt = {x = 3.0, y = 4.0, z = 5.0} in #x pt end;
        val it = 3.0: real
        >

        $ ocaml
                OCaml version 4.03.0

        # type point2 = {x : float; y : float};;
        type point2 = { x : float; y : float; }
        # type point3 = {x : float; y : float; z : float};;
        type point3 = { x : float; y : float; z : float; }
        # let pt = {x = 1.0; y = 2.0} in pt.x;;
        - : float = 1.
        # let pt = {x = 3.0; y = 4.0; z = 5.0} in pt.x;;
        - : float = 3.
        #

The only nuance with record labels in ML, as far as I can remember, is that the type of projection operations can be ambiguous if the exact type of the record is unknown (e.g., in `(fn pt => #x pt)`, the type of `pt` cannot be inferred since it could be `point2`, `point3`, `{x : int}`, `{x : real}`, ...; but this can be disambiguated with a type annotation: `(fn (pt : point3) => #x pt)`).

Perhaps I've misunderstood the question. If so, can you give me an example?

<comment><USER10> This is weird. Last time I really did anything with ML was ages ago, and I distinctly remember fighting with the fields... I wish I could remember the context, though.

Anyway, good to know that it's definitely not OCaml!

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> Love the design! Maybe not the simplest to use, and fairly verbose at times... but it's very clear what's going on at any given point - and given how hairy macros can get in production code, I think emphasizing clarity is the right approach.

</comment><comment><USER12> What's the story for error messages? I read it's inspired by racket; the big thing about racket is that from what I hear it seems to be the only language that has good support for syntactic extension without sacrificing error reporting.

<comment><AUTHOR> Since macros aren't just a plugin but are directly baked in the compiler code, error messages can be anything we want… what are you suggesting?

<comment><USER12> Let's take your static printf as a concrete example. Here's what a not so great error message looks like (ocaml's built in Printf):

      utop # Printf.sprintf "Some int: %d another int: %d" 1 "2";;
      Error: This expression has type string but an expression was expected of type int

Here's what a good error message looks like:

      clang -Wall printf.c
      printf.c:6:52: warning: format specifies type 'int' but the argument has type 'char *' [-Wformat]
         printf ("Some int: %d some other int: %d\n", 1, "2");
                                               ~~        ^~~
                                               %s

Notice how it also highlights the mismatched part in the string-based DSL.

<comment><AUTHOR> The second message is arguably clearer, but I don't think anyone would be really hindered by the first one, especially given that, unlike in C, application of printf have an explicit type:

        # Printf.sprintf "some int: %d another int: %d";;
        - : int -> int -> string = <fun>

I think a general tendency in statically typed languages is that we like to write stuff using only the type system, rather than adding code to the compiler. Same for macros: we will rather have them proved correct by the typechecker than make them raise errors (also it is possible, through exceptions).

But maybe this is detrimental to clarity of errors in some cases. I'd be interested to see how Racket handles errors.

<comment><USER12> Is it really arguable? I'd be surprised if anyone took longer to fix a bad format string based on clang's output than on ocaml's.

Printf is a trivial example – good output is mostly useful because of ubiquity and C's unsafety.

I believe what sets racket apart is mostly their mix of tech and culture:

\- their macro system, whilst being extremely flexible, preserves enough source location information that you can give good error messages

\- due to their pedagogic background they care about good error messages took effort to tool for them (e.g. syntax-parse: [http://www.ccs.neu.edu/racket/pubs/icfp10-cf.pdf](http://www.ccs.neu.edu/racket/pubs/icfp10-cf.pdf))

By contrast, common lisp and to a lesser extend clojure are also flexible but produce pretty bad error messages (and both have a culture that strongly de- prioritises approachability). This comes at a real productivity cost that doesn't disappear completely even with experience.

In lisp you can write a nice SQL like language or a math DSL without resorting to string parsing. In ocaml you kinda can't because the base syntax isn't that flexible. So I'd expect the problem to be worse and I don't think types are a magic fix. The worst and most voluble compiler error output known to man must be C++ template metaprograming errors.</comment></comment></comment></comment></comment>


I'm Sure It Will Only Take You A Few Days To Code - <AUTHOR> http://danshipper.com/non-technical-people-cant-estimate-developmen <comment><USER2> The Lifecycle of a Project Manager:

Phase I: (Idealism) Treat building software like building a bridge. Analysis --> Design --> Development --> Testing --> Implementation.

Phase II: (Pragmatism) Realize that Phase I doesn't work. Try to figure out why. Decide that the weak link is Analysis, i.e. we're no good at estimating and hitting deadlines because we never have good enough specs. Devote your life to the art and science of getting the perfect spec. Implement the methodology du jour to make that happen.

Phase III: (Realism) Realize that Phase II doesn't work. Decide that no existing Analysis methodology works because no one knows how to conduct analysis, no matter what the methodology. Focus on optimizing the practitioners with better education and 10,000 hours of real-world experience.

Phase IV: (Enlightenment) Realize that none of the other phases works because we've been focusing on the wrong constraints all along. Instead of optimizing scope, quality, or quantity of the product, optimize the schedule. Adopt a new model: define the deadline first, then build whatever you can by that deadline. Hit your deadlines every time, not by estimating better, but by not caring how pretty your deliverable is. Eliminate the problem of estimating by making the deadline a constant. Eliminate the problem of analysis by replacing it with prototyping and revising. And eliminate the problem of blueprints by adopting a strategy of inventing, not engineering.

<comment><USER3> I once interviewed a project manager.. asked how he came up with deadlines. He said "ask the programmers and subtract 20%". Talk about a red flag in an interview!

<comment><USER4> That's funny, because as a programmer I always estimate how long it'll take and add 20%.

<comment><USER5> Only 20%?

A recent conversation I had with my MD, regarding how long to estimate for a project, involved me adding 6 months for each feature added to the specification.

We're also careful emphasise how inaccurate these estimates are, and we don't really know until we start.

Of course this is the initial "finger in the air" stage. We try to be a little more concrete with more immediate requirements. Of course, we've gotten those badly wrong too.

</comment></comment></comment></comment><comment><USER6> P = programmer; NP = non-programmer

      NP: How long will it take?
      P:  It'll take as long as it takes.
      NP: I don't understand. What does that mean?
      P:  When you tell me that the product is finished, it's done.
      NP: You mean you can't estimate how long it'll take?
      P:  I can't estimate when you'll decide that it is done, therefore an
          accurate prediction is impossible.  From experience I know that a
          project like this can take from three to nine months.  A lot
          depends on you, the choices you make and how well you communicate
          them to me.  A lot also depends on how many times you change your
          mind during the process and how many times you change the
          specifications during the job.  Major changes can even cause the
          project to be scrapped and start over.  Rest assured that I will
          not be the one slowing this down.
      NP: How much will it cost?
      P:  It will cost as much as it costs.
      NP: I don't understand.  What does that mean?
      P:  You just learned some programming!
          Go back to your first question and re-read the entire thing.
          This is called an endless loop.

<comment><USER7> What are the implications on the universe if P=NP?

</comment></comment><comment><USER8> Disclosure: I am currently a software PM

I actually think that the same principles apply to _any_ form of creative activity.

Let's say I'm going to write a 300 page book. I can take my typical wpm speed, do some math on it, and say that it will take me X hours to do this. A book, on the surface, should fit into the "size and speed" heuristic. But of course, this is nowhere near the case. No decent piece of writing ever comes out without many many iterations of rewrites.

Writing a piece of music? Same thing. Making a clay pot? Even great masters take many tries before coming up with something worthy of the public eye.

Software, even to my novice eye, is similar. But the difference is that the person with _zero_ experience doesn't know instinctively that software also falls into this iterative, creative process. We've been trained and educated to _know_ that writing, composing, and sculpting are iterative creative processes that don't have a linear measurement scheme. We spent hours upon hours as children and young adults writing and rewriting, drawing and redrawing. We know from personal experience that things hardly go "as planned".

But for something completely foreign to us, such prior knowledge is absent. We have _no_ idea. So we take what we _can_ see, and extrapolate. "Hey, I see just one web page, that can't be so hard... (insert mysterious thought process) how about XYZ days?"

The fact that I took a few years of programming courses in college (I was an EE Major) has probably helped me in keeping things sane. I've written enough code myself to know instinctively that both development and debugging are far from linear processes. I _know_ that I have no way of estimating how much time it will take devs who are much more capable than I am to develop/fix something. So I cede to the guys who know best. I take their word and estimates in good faith, and focus on swatting the 'flies' away so the team can focus on the task. What if we don't hit our estimates? Well too bad, but stuff happens. Time to reassess, buckle down, and go again.

<comment><AUTHOR> I agree completely but I think there are two things that make software slightly unique from any piece of art:

1\. We have a cultural context that tells us how hard it is to produce something like a song, or a painting. There's a reverence for it. And so people tend to back off of their estimations for the amount of time required to produce it.

2\. Software isn't physical. You can hold a big book in your hand say CRAP! that must have taken a long time to do. But you're just looking at the result of a program, if it's done well, it's a tiny streamlined user interface. The result of one button click could have taken thousands of programming hours and thousands of lines of code but you would never know it. Software obfuscates complexity from the user in a way that someone looking at a huge huge painting or a big book doesn't run in to as much.

So if we're talking about the efficiency of natural heuristics in estimating things, I think software is still slightly more difficult to estimate than art. But I agree with your overall point.

<comment><USER9> And one of the things that throws some software estimates out the window is when a developer/programmer gets in the zone. There have been times I've hammered out a problem that I had spent days on in a few hours when the inspiration/flow/etc. hit. Something I would have guessed would have taken me a few days more, I end up hammering out in 3 hours. And when you pull these kinds of tricks...

People think you can do it again... and again... and again as if it was something you could just conjure out of thin air. There are many papers written on this topic. I've been playing around with a few ways to achieve this more frequently, and it has worked to some degree, but it's not something I could rely on in the heat of battle.

<comment><USER10> Easy way out - if you estimated 3 days and it turns out to be 3 hours ... simply sit on it for 2 more days ;-)

</comment></comment></comment></comment><comment><USER11> I'm a music producer, and this is a constant problem for me that after many years still boggles my mind.

The root of the problem is that producing a modern style recording of music is resource expensive. Far more than anybody would like it to be.

The resources can be time/money/talent. So there is an intense bias on everyone involved in the project to under-estimate.

I usually work on low budget projects. You can actually accomplish a lot with a low budget. But as you start to climb the "quality" scale the costs increase much more quickly than the quality. To get a product that you might consider subjectively twice as good might cost ten times as much.

At core, I feel like there is a fundamental principal of information organization. It takes a certain amount of resources to undo entropy and organize information. When we experience a piece of work we can intuitively feel that entropy has been significantly reduced, we appreciate the accomplishment. (Even if we are not in the field of endeavor we still feel it though we don't know how it is done.)

One specific point of difficulty in estimating time in music production is that there is an x-factor. It derives from the need to create something exceptional, which has both subjective and objective elements.

With an experienced production team you can predict how long it will take to produce something competent. But usually there is a point where we have all done the things we know how to do: the song is arranged, the players are good, the production environment is good, the instruments are good. But the song is missing something. What is it? It is not known yet. So an iterative trial and error process begins. Let's try a different singer. Maybe we should borrow my friends guitar. Let's keep recording. Let's rewrite the song.

Hopefully a solution is arrived at, but since it is a trial and error process it is impossible to estimate how long it will take.

</comment><comment><USER6> One of my favorite ones was a fellow who wanted an iPhone and iPad game done. He proclaimed that it had to be done for $10K. This game easily required six months of work. It also required server-side development and support. He also had no idea that every so often you need to fix, I mean, update, your apps because new iOS releases might break them.

I was open-minded and decided to invest some time educating him on the process and the needs of the platform. Once he had enough information he networked and hired someone out of India to build him the game. I was not happy about that at all. Then I learned that he was having all kinds of problems with the process (and the app) being a total mess. He's learning his lesson.

<comment><USER12> Come on now. Don't blame the programmer being Indian for the problems your friend is having. Good programmers anywhere cost almost as much as they do in the US now. Globalization means that an Indian programmer has the same access to salary information in the west. He/she might knock off 10% to be competitive.

You're friend could have easily got a cheap, inexperienced programmer anywhere else in the world, including the US, and still had the same problems. He had a budget, you knew it wasn't enough for what he needed, and he came up snake eyes.

<comment><USER6> Who says I am blaming the programmer for being Indian? I'm not. Excellent work being done out of India. Here's the problem, lots of people like this fellow look at India and China to find the lowest possible bidder out of complete ignorance. And, when they find the lowest possible bidder they get exactly that: crappy inexperienced programmers.

If they searched for the lowest possible bidder in the US or Europe they'd get exactly the same thing.

I did not intend to imply that Indian programmers are not good. An ignorant fool looking for rock bottom prices will, more than likely, find them in India rather than the US or Europe.

</comment></comment></comment><comment><USER13> Another thing which I think kills accuracy in estimating development time is the ratio of time needed for "figuring out what to do" and "actually doing it".

If you're learning a song, the first stage is very quick. You think about the problem for a moment and you decide on a plan like "Read the sheet music and try to play the song. Focus on areas you have trouble with. Repeat". The bulk of the time taken is spent "actually doing it", not "figuring out what to do".

The thing about the "figuring out what to do" stage is that often in order to estimate the time needed for that stage, you basically have to have already completed it. The answer could come to you out of the blue, it could take weeks of thinking into dead ends. And with software, that first stage can be the bulk of the time. Software is a more like writing a novel than learning a song. The thing that takes longest isn't the actual writing, it's the figuring out what to write. If someone asked you how long it would take you to write a novel that gives insight into the human condition by following the story of a family caught up in a bloody civil war, what time estimate would you give them? I expect your typing WPM wouldn't be a big part of the calculation.

You only know how long it's going to take once you know how you're going to do it, and once you know how you're going to do it, you've already done most of the hard work.

</comment><comment><USER14> _As a programmer I know I can use my prior experience building similar things to estimate how long each feature will take to implement._

Here's where this breaks down for me: it's never similar. Sure, the functionality from the user perspective might be similar, but with the pace of change in development tools, platforms, and frameworks, it's never the same. Seems that every project involves some major piece of kit that I've never used before. In the last five years I've used ASP, ASP.NET, C#, PHP, Lua, Python, Erlang, Oracle and PL/SQL, MySQL, PostgreSQL, SQL Server and TSQL, OpenLDAP, XML, XSLT, JSON, javascript, jQuery, Scriptaculous, and other things built out of those, and probably a dozen others I'm forgetting at the moment.

Yeah at some level databases are all basically the same and languages are all basically the same but at another level they are not and that's where the time sinks are.

</comment><comment><USER15> You are in a closet, looking at a closed door. How big is the house?

<comment><USER16> The problem is no one has locked you in the closet - just open the door and look around. Heck, you have a whole hour to look around the house and estimate how long it takes to build the thing.

You are limited by how long you can spend inspecting every nook and cranny of something, but you're only limited by time. The "door" metaphor is misleading. Nothing is standing in your way from exploring the space.

If you can find the "hard parts" of the house faster than someone else, you're likely to be a better estimator. In fact, my favorite trick is to focus on one hard part and extrapolate up. For example, write down how tricky it is to build the kitchen, or heck just a cabinet in the kitchen, so everyone gets it when you say "oh, by the way, there are 10 more rooms."

<comment><USER15> At Bigco, you can't start opening doors until you give them your estimate and get it approved. There's no way to know how many more doors in how many more rooms there will be until you've been in all the rooms and opened all of the doors.

Its fractal. That "last" door could lead to 100 more. Oh and its almost always a horror flick with axe wielding maniacs(1) and booby traps(2) in each room intent on your destruction.

(1) The boss's boss's nephew who's a "computer whiz".

(2) $300,000 "developer tools" you have to use by fiat policy because, dammit, we spent a lot of money on them.

</comment></comment></comment><comment><USER17> Funny to see this on the same front page as <http://news.ycombinator.com/item?id=3795064>.

"[Jobs] pushed Steve Wozniak to create a game called Breakout. Woz said it would take months, but Jobs stared at him and insisted he could do it in four days. Woz knew that was impossible, but he ended up doing it."

<comment><USER18> Yeah and Linus coded git in a weekend...

The infuriating thing for me is that people don't get that these monumentally fast development achievements still involved a massive expenditure of personal effort. Thus the "It'll take a few days of coding" customers of the gp are only more annoying if they cite examples like yours, expecting a complete product in a few days.

<comment><USER19> Not only that, but they require all the skill and experience of truly world- class experts in their fields.

Yes, Linus can throw together an amazing dvcs that scratches his particular current itch in a weekend. The just-graduated CS major you're pitching your idea to? The one with three whole php and/or ruby websites under his belt? He'll be able to do that too - after spending 20 years running the worlds most popular open source project and knowing exactly what he needs his version control system to do. Give him a weekend to do it right now? You'll get _something_, but it sure as hell wont be git.

Same with Woz - there was probably nobody else on the planet who could have written breakout for the AppleII in under a month. I don't know whether Jobs was brilliant enough to know what he was asking was possible for Woz, even when Woz told him it wasn't, or whether Jobs just strongarmed Woz with unrealistic expectations and took credit for the success (and laid blame for any failure); but if he'd asked anybody else, they would have failed.

Whatever your idea is, chances are _very_ high that if you think it could be coded "in a few days", you don't actually understand the problem (and it's solution) yourself.

<comment><USER20> I can/have built in some amazing stuff over weekends. But generally the requirements for that are, and in no specific order.

        1. I must be in a flow.
        2. I am using a technology very familar, so I know
        the tools and many things.
        3. I am doing something new. Not maintaining some ones
        code or interacting with an external team.
        4. No disturbances and other office distractions like
        office emails.

Building new things are different than maintaining. Many a times I'm waiting for some other team's inputs to proceed. Or I'm required to fix a list of X number of bugs which can't be fixed until I understand the underlying code base well.

Create this webpage in 15 mins kind of arguments don't work because you are adding something into a working system and you can't generally do so until you understand everything about the system.

Doing something new is different. You definitely can get to the first prototype if you know what you are doing, what you are using to do it and no distractions.

<comment><USER21> Mark Bernstein is recoding a program called StorySpace which dates from 1980s?. He is blogging about it. Day 6 will give a flavour

<http://www.markbernstein.org/Mar12/Day6.html>

He is working on a familiar design, and knows the old code. He is recoding in newer tools.

<comment><USER20> Look Mate, By no means Am I downplaying anybody's accomplishments nor am I trying to say that anything can be built quickly.

Rome was not built in a day. Neither was Git as we know now. But given a narrow set of requirements you can get a prototype out pretty quickly. Especially today when so many reusable things are available. Linus coded up a distributed vcs(Not with all the feature we know now today) sufficient enough to host itself on a weekend.

Its also depends on the Individual. Some people can sit for long hours at a stretch and focus on problems till something worthwhile comes out. Others can only do little per day.

In this case it looks like a _rewrite_ and rewrites are by no means doing _something new_. By now he would have had a perfect idea of what he needs to do in the entirety so he would be writing per that design one thing at a time. That is totally different that writing a self hosted vcs from scratch on weekend and then iterating and adding features over years.

<comment><USER21> I was agreeing with you and providing an illustration of the working practice of a 'craftsman' programmer!

</comment></comment></comment></comment></comment></comment></comment><comment><USER22> Reminds me of "open textured" in legal philosophy: the idea of precedent in law is you can use it to predict how a court will decide in future. This mostly works, but if you have a bunch of cases (data points in multidimensional space), simple interpolation isn't always accurate. You need to look at the specifics of the present case (there's still uncertainty because each judge has an individual perspective, but that's a different issue). It's a little bit like a fractal: just knowing some data points won't always predict - even roughly - what happens in between (but in flatter regions, it can). And Turing showed that (in general), you can't predict the result of a computation except by actually doing it.

The _experience_ the article mentions only helps when a specific domain reoccurs, and known tools have been developed for it, with known characteristics. Now, if you know that domain and the tools, you can do prediction pretty well. I think of it like an experienced tradesman.

However, when you go into a new area (and a new one is created every decade or so in programming), you're back to zero in many respects. e.g. the existence of the right library can save 10,000 hours.

<comment><USER23> That makes me wonder what other disciplines there are that have the unpredictability and open-endedness of software. I read that on large construction projects, a 1% overrun is considered failure. That is clearly a different universe. Who else is in our universe?

<comment><USER22> I think it includes anything involving new knowledge: e.g. law, science/mathematics, invention. Maybe also in the academic Arts (I don't know enough to be sure), and also in the creative arts - painting, sculpture, writing (there's exploration and massive overruns there). Also, exploration in prospecting, oil discovery - though that's not really open-ended.

OK, I think it's being able to open up/zoom into finer detail (like knowledge, recursive structures, fractal) + discovery.

<comment><USER23> Law, I'm not so sure. Any variations posed by new legal cases are still limited by the basics of human nature.

Oil discovery? Don't think so. People may look in different places but they don't invent new ways of doing each project.

The other examples are all art. (The practice of science itself is an art and not a science.) Is software an art?

</comment></comment></comment></comment><comment><USER24> Is Pachelbel's Canon considered complicated as far as music performance goes? Reading that knocked me right out of the flow of this article. What's he thinking of? Maybe someone's complicated improvisations on top of the (really simple) canon?

On topic: the post didn't really cover where the layperson's base estimate comes from -- a layperson sees X pages, or X features, and multiplies that by... what?

The reference they use varies hugely, in my experience; many are trying to _imagine_ what's involved in developing a "feature" -- which yes, will generally be a massive underestimate... not always, though.

There are also plenty of laypeople who imagine that everything they see is incredibly complex & difficult to create (I have a neighbor who imagined that writing software meant _typing in_ the gunk he saw when accidentally opening a binary file in notepad), and all written from scratch -- so a simple blog would take ages to build, because it's obviously complicated stuff.

In real life, some quite complicated things come for free, other seemingly simple things take gobs of time, sometimes unexpectedly... it's a big mess. And just as some of the harshest wildcards (like nasty bugs in old browsers) are calming down, more form factors, interaction modes, etc. are popping up, so I'm not sure it's going to be a problem we can stop talking about anytime soon.

<comment><USER25>
      Is Pachelbel's Canon considered complicated as far as music performance goes?

Consider the more complicated violin part.

<comment><USER24> Still, you hear grade schoolers play it.

And wow, if you've ever been forced to play the part of anything _other_ than violin -- I played the viola part innumerable times in HS orchestra, and the cello part was even worse -- it's almost a punishment.

</comment></comment></comment><comment><USER26> Thanks for this, it's a great article. I have this problem with my boss occasionally, which is funny because he's a programmer. "Oh, you just have to do this and this, shouldn't take you too long."

You're right, there's always those bottlenecks that crop up that you didn't expect. It's impossible to truly know all the limitations of the software you're working with. I work with Drupal, and it has A LOT of those problems. Sometimes I'll think something will be easy to do, only to find out that the popular modules I thought would work great don't cater to the specific use- case I'm working on. Then it's back to schlepping.

<comment><USER27> As bad as it is for non-technical managers to estimate, I've found that in some cases, programmer managers can be worse, especially if you're new on the team. Why? They already know the code, so often their estimates state how long _they_ would take to implement the feature. They either forget or don't realize that you have to spend additional time to learn how the code works and how to best integrate your change. They don't realize that sometimes it's harder to modify software than it is to build anew.

<comment><USER26> Very insightful, that's my case I believe. I had never used Drupal prior to starting at the place I am now, and part of every project so far has been finding out that Drupal does not, in fact, do what I want. Luckily my boss is a good guy, so he's understanding when I don't estimate correctly.

</comment></comment></comment><comment><USER28> Nice write up.

Want to add two other reasons we're horrible at making estimates.

1\. Expectations. 2\. No widely accepted language for writing requirements.

Re: 1. Expectations are just hard to manage. Even when you do get "So the site's pretty simple" - the set of features in their brain is totally different from the set of features in your brain. Alas, it'll take more than "all it needs to do is X, Y, Z" to fix this process.

Re: 2. Is there a professional software consulting company that has solved this problem? I say there isn't. I also say that with the advent of Agile/Lean Startup methodologies, it'll probably never happen - "No, we don't write reqs, we have stories you need to approve." uh huh, okay. For writing requirements, I got really damn fond of Cucumber, but then the lean-ux took off, and that shifted everybody's priorities.

Bottom line, until there's a widely accepted way to communicate design and functionality, making bad estimates will always be easy to do

</comment><comment><USER29> I think it's much more domain-expert vs non-domain-expert rather than programmer vs non-programmer.

I've regularly seen very experienced developers make exactly the same sort of mistakes when talking to people outside of their field (Sales, Design, etc.) because they miss things like:

* Sales is about building stories and relationships - it's not just listing features and meeting people.

* Design is about understanding the user, iterating, testing and experimentation - it's not just making things pretty.

I wish technical folk were naturally better at this sort of thing - but they (and I :-) keep making similar sorts of mistakes.

(It's also amusing to see the "it's not like building a bridge" analogy turn up. Go talk to some engineers, architects and builders - and find out how many of those projects ended up matching the original time and cost estimates :-)

<comment><USER30> > _(It's also amusing to see the "it's not like building a bridge" analogy > turn up. Go talk to some engineers, architects and builders - and find out > how many of those projects ended up matching the original time and cost > estimates :-)_

That's an important point.

I used to work in sub-contract electronic engineering. We had a lot of information about how long it took to do things, and how much items cost. In theory, estimating the cost of projects should have been straightforward. And then you add 15% for wiggle room.

And still it's hard to keep things going out on time and on budget.

</comment></comment><comment><USER31> I very often hear this comment from my fellow developper colleagues.

In our startup, I am responsible for the frontend coding. Very frequently, I get requests from other developpers to implement this or this, and very often those requests are annotated by a "c'mon, it is just CSS, it'll only take you 5 minutes" when they have absolutely no idea what they are talking about, because they never ever did real frontend developpment. They simply think that everything boils down to HTML and that frontend is NOT really programming. Attempting to explain that a frontend also has logic similar to backend, and that no, plotting graphs when you don't even have the data modeled is not just a 2 liner with Google Visualisation.

I am always offended by people estimating time for work they never did. So I find myself very rarely giving estimations. Sometimes when I have a very precise idea on how to implement an idea right when I'm being told about it, I usually say "yeah this is definitely doable in a short time span". Meaning things that do not take more than half a day to implement, because you already compiled the code in your brain.

Now, when a colleague of mine estimates stuff for me, I tell them "No I don't think this is going to take X minutes to implement. If you think it is implementable so fast, why don't you go ahead and do it.". When they take it to the word, and I see them still struggling after many hours on some problem that are trivial to me, I will go ahead and help them out. After a few times, they usually won't bother me with estimations anymore... NOT.

</comment><comment><USER32> "But it brings up another more interesting question: why does the way we naturally measure complexity stop working when we apply it to programming?"

Ah and I had high hopes for that next paragraph,

There are two fundamental differences between, lets say, estimating how long it will take to make a sandwich, and estimating how long it will take to create an online noticeboard for your village.

The first difference is that one task is manufacture, the second task is design. Writing code IS design, and it is inherently much harder (or even logically impossible) to estimate because implicitly you do not know what done means.

The second difference is that (broadly) how satisfactory a sandwich or other real world object is, is proportional to how long it takes to make. This is not true for software, because it is not a physical entity. It can be tripped up by a single incorrect character, and then, forget tasty - your software sandwich doesn't even exist.

</comment><comment><USER33> Same fallacy that convinces people to think they can change the world as young whippersnappers.

I'm sure it'll only take you one life time to change the world.

How you do guess at something without knowing the full size, weight, depth, and complexity of something?

</comment><comment><USER34> A thought that occurred to me - another metric we use in the real word would be what I'm calling "solidness" or "weight." I.e. if you see a house and a similarly sized tent, the apparent difference we see between the two is weight - we can probably pick up the tent and walk away with it.

In a sense, that's a useful way to describe software to non-technical users. We can throw something quick and dirty together, but it would be light as a feather, and carried away by any kind of load whatsoever. Or we can build something more solid to last longer, but it should be clear that that will take more effort.

<comment><USER35> Yeah, let's just go with the quick and easy version. It's really all we need anyway, I mean it's not like we're going to release it as a product to the public. We just need a demo for the boss right now.

<three months later>

Why is that thing not working? I thought we finished it three months ago!

</comment></comment><comment><USER36> _Why does the way we naturally measure complexity stop working when we apply it to programming?_

Lots of projects that have nothing to do with programming end up taking longer than expected or are over budget.

</comment><comment><USER37> Let's take it a step further, shall we. Say you get this type of request thrown at you, which is obviously a mis-estimate and you don't end up delivering it with expected pace. You weren't supposed to, by the way. It was a trap. Now, the PM or whoever says this "Well, Joe, what if we punt it to Bob over there and see if we can get you concentrated on another problem X". Does that precipitate your balls into shrinking or what?

Estimates are just that, estimates and not contracts. We should expect them to have a float time, otherwise rename them to contracts. I do X in Y amount of time if Z is true. Usually, there are unknown variables and I'm running out of letters to name them all.

Furthermore, I don't work on a deserted island. I have a phone, QA keeps on asking me questions, clients want clarifications and my version control is burning up from constant merging, and yes I have included all that in the time estimate.

</comment><comment><USER38> Any project management software out there that takes an ML / probabilistic approach to this? One thing I found at one of the mega-corps that I worked at is that both project managers and regular managers were blind to history. Note: this was before agile became accepted by mega-corps.

Manager: How long is this going to take?

Me: Virtually impossible to predict accurately

Manager: I need a date

Me: 3 months

Manager: That's too long, we need to get it done in 1 month

Me: OK, 1 month

... 5 months and several iterations later ...

Manager: Project has been deprioritized, we have a new project, how long is this going to take?

Me: Virtually impossible to predict accurately

Manager: I need a date

Me: I need you to be stateful.

</comment><comment><USER39> so reverse the question - "why does programming small things take so long?" (assuming x, y and z were relatively standard). is there a niche in this somewhere that maybe some % of the guru class should be focusing on 'making typical, trivial'? ie, we found 70% of websites include these features - download this template, push run and you have a website running. change text on front page manually might actually be good enough for 20% of the population...

<comment><USER19> Because "x, y, and z" are "standard" only in the "the great thing about standards is; there's so many of them to choose from" sense of the word.

Recent example. Client: "We just want an online shop" - great clickety click, Magento is installed, stick their logo in, tweak the stylesheet to match, call it "done".

Over the next few weeks/months: "Where's the blog?" "How do I enter auctions?" "I want to display products with individual pricing, but sell in cases of a dozen." "My SEO consultant says I need some 'landing pages' - where do I make them?" "This doesn't sign people up to out MailChimp email lists!" "It doesn't work right on my wifes iPhone."

Discuss needs with client, install various plugins, a wordpress blog, tweak the templates to make single-pricing/bulk-selling "work" for boxes of 12 (and kick myself knowing it'll bite me in the ass one day doing it this way…)

Then: "How do I make the auction extend the time if someone bids in the last minute?" "How do I enter products that come in boxes of 6 or 8 instead of 12?" "It works on the iPhone now, but not on a friends 2 year old cheapo Android 1.8 phone!"

Then:

"How do I auto calculate freight costs based on both weight and cubic size, for orders with multiple products?"

(at which stage I start explaining combinatorial complexity and the knapsack problem to the client, and run a warm bath and remind myself "down, not across…")

<comment><USER40> So good strategy would be provide customers with almost free standard solution and then customize it step by step for ever increasing fees, right?

<comment><USER41> That sounds like the business model of most of the agencies selling WordPress solutions who have been hiring me for the last two years.

</comment></comment></comment></comment><comment><USER42> I think a lot of clients offer estimates based on how much money they have to spend regardless of the complexity.

<comment><USER43> No shit. It's always like this. They have a wish list and a budget. These two things hardly ever match.

It's hard to give a quote when there are no detailed specs yet. Most clients still want a price tag. Of course afterwards they'll nail you on this premature conjecture.

As a business your quote needs to be competitive and still be profitable.

From experience, we usually throw things out and create boundaries in our offers / quotes. We've learnt the hard way that often times clients implicitly assume certain features to be there without ever communicating them to us, so you need to think ahead and have it in black and white that if they want X they have to get another offer / quote at later stage.

So imo it's not just about what work a quote includes, but also what work it does not include.

</comment></comment><comment><USER44> Such a noob topic. If you don't know what you are doing, just admit it.

For seasoned programmers and PMs, this is a non-issue. Perhaps you are in the wrong profession if you can't handle the demands without whining like a brat.

</comment><comment><USER45> non-programmers, in my experience, tend to think software is "just that stuff you see on the screen". So if they create some screen mockups of the UI, they think the programmer pretty much needs to sit down at the computer, and do some simple keyboard/mouse clicky-clicky to make the screen match his mockups. and then they're done. To them, it's just visual, and static. They don't understand there are processes, threads, hardware contraints (cpu, memory, disk, network bandwith), libraries, languages, configuration, edge cases, error conditions, etc. I've seen this kind of perspective a lot with non- technical stakeholders, especially first-time clients of software contractors, or first-time managers of software developers.

"Just make it match these mockups!"

Dragging a little Facebook icon into place and poof all the code and edge case handling and database work needed to integrate with Facebook is all done, put to bed, etc. Just put that icon in the right place.

<comment><USER20> Well in such a case those people should not opt to manage technical people.</comment></comment>