Apollo 11 Saturn V Launch - <AUTHOR> http://vimeo.com/4366695 <comment><USER2> Is a version of this footage available directly from NASA, in the public domain and without the commentary/etc.? I've searched around the NASA video archives a bit and haven't found a version on their site. Or did this company, Spacecraft Films, do the transfer from the 16mm film themselves?

<comment><USER3> The NASA video gallery is really heavier on current/recent videos, but you can give a quick search. [http://www.nasa.gov/multimedia/videogallery](http://www.nasa.gov/multimedia/videogallery)

They really should come up with a more comprehensive central repository.

The other source would be Archive.org : [http://archive.org/search.php?query=nasa%20apollo%2011%20AND...](http://archive.org/search.php?query=nasa%20apollo%2011%20AND%20mediatype%3Amovies)

</comment></comment><comment><USER4> As alien as this video is, I must say it's one of the most beautiful things I have ever seen as it directly represents the power of human knowledge. If I would fit into that camera's case, I would give anything to watch something like this in person.

<comment><USER5> You'd just see some blinding white light for a few seconds and then maybe nothing else for the rest of your life. Our human eyes and brains limit us from perceiving even everyday things with the majesty they deserve.

Even popping water balloons looks amazing in high speed video. On YouTube, there is a community of people doing silly things with high speed cameras that makes for stimulating watching:

Smarter Every Day: weird glass phenomenon [http://www.youtube.com/watch?v=xe-f4gokRBs](http://www.youtube.com/watch?v=xe-f4gokRBs)

Smarter Every Day: cat righting itself in midair [http://www.youtube.com/watch?v=RtWbpyjJqrU](http://www.youtube.com/watch?v=RtWbpyjJqrU)

Lots of high speed videos [http://m.youtube.com/user/theslowmoguys](http://m.youtube.com/user/theslowmoguys)

<comment><USER6> I wouldn't dismiss our perception as lacking majesty :) That would be very depressing. Subjectivity aside, however, you're right – what we experience is determined in part by the constrains of our nervous system and perception organs (remember the rapidly spinning fan blades that look like a disk?). But, this is just the conditions on life, a fact to which we need to adjust, for we cannot change it.

It's also worth pointing out that for humans, this is where science comes in :)

The slow motion videos illustrate very convincingly that we are always omitting data. I love them for that. One cannot watch them and not be persuaded into a sense of how unavoidably probabilistic our inferences are, as well as into a sense of wonderment and curiosity. They are both desirable responses, because they are key to sanity.

</comment></comment></comment><comment><USER7> This footage reminds me of a post some time back about why in 1969 it was actually easier to fly to the moon than fake all the video footage.

</comment><comment><USER8> [http://vimeo.com/4366695](http://vimeo.com/4366695)

Great commentary!

<comment><USER5> Thanks for the non-blogspam version of the link. It's really a great video with no need for editorializing.

</comment></comment><comment><USER9> Those who like this will probably like the short documentary "Ascent - Commemorating Shuttle" produced by a some NASA engineers. It uses similar footage from launchpad cams, with lots of technical commentary.

[http://www.youtube.com/watch?v=vFwqZ4qAUkE](http://www.youtube.com/watch?v=vFwqZ4qAUkE)

</comment><comment><USER10> This video is well worth watching. I almost skipped it, expecting the same old launch footage, but it is a slow motion closeup of the engines and exhaust with very interesting commentary. (You can skip the last couple minutes without missing much, though.)

</comment><comment><USER11> Koyaanisqatsi, the 1982 movie uses similar footage from the Apollo 12: [http://www.youtube.com/watch?v=g9K0013nITs](http://www.youtube.com/watch?v=g9K0013nITs)

</comment><comment><USER12> Nothing, I believe, better illustrates the human potential than the space race that culminated with the Apollo lunar landings. We should be doing more of that.

<comment><USER13> A well defined objective, successfully completed, is a well-defined objective terminated.

What a fantastic mountain of effort, focused dedicated enthusiasm, and skill- sets, left adrift and carelessly abandoned. I have read schematics of some Apollo entire sub-systems, have completely disappeared.

And then there is the decades long disappointment. We see few Shuttle posts on HN.

<comment><USER12> Indeed. New objectives should have been set. The shuttles, while a marvelous achievement of engineering, were ruined by the politics around them - so many different mission profiles they were designed to do that nothing was left that could be done practically. And yes - going to LEO is not nearly as inspirational as going where no one has gone before.</comment></comment></comment>


The Powerball Jackpot is $425M. Should you play? - <AUTHOR> https://medium.com/p/28c5a31cd41d <comment><USER2> The math for split jackpots assumes that people pick randomly distributed numbers in their tickets. But that's clearly not true, and that means by choosing numbers carefully you should be able to significantly reduce the chance you'll have to split a win _without_ changing your chance of winning.

I'd love to see an analysis of how beneficial this could be and which numbers you should pick.

<comment><USER3> Choose 1, 2, 3, 4, 5, 6. Nobody _ever_ chooses that set.

<comment><USER4> The laws of large numbers make any edge supposedly gained by "picking what nobody else picks" almost entirely moot. The probability is still incredibly small that any given sequence will come up. Getting the right sequence is the _much_ taller hurdle to clear before tackling the problem of how many others have also picked correctly.

<comment><USER5> Of course no one is saying that it's easy to get the winning numbers, the question is whether it is worth it to play. If the prize is big enough, then the expected value of a lottery ticket is more than $1, and it makes financial sense to play. By reducing the chance of sharing the jackpot you can effectively increase the size of the prize, pushing the odds towards your favor.

<comment><USER4> _" By reducing the chance of sharing the jackpot you can effectively increase the size of the prize, pushing the odds towards your favor."_

In theory, sure, but in praxis, whatever edge you gain on expected value by doing this is still infinitesimal. Furthermore, it's highly unlikely that you could choose an unlikely-to-be-duplicated number combination any better than random chance could. I'm not convinced a randomly generated string of numbers is any more likely to be duplicated by someone else's pick than an intentionally chosen sequence. Probably less likely, in fact. I would guess there's someone else out there picking 1,2,3,4,5,6 (for example) more often than there's someone else out there whose random number generated matches your random number generated.

<comment><USER6> But once you accept the small chance of winning, you may as well set it up so that if this unlikely result came out you'd maximize your winnings. Its a tiny edge because it'll happen so infrequently, but if it did it would make a huge difference in the result.

<comment><USER4> _" But once you accept the small chance of winning, you may as well set it up so that if this unlikely result came out you'd maximize your winnings"_

If there was a way to do so, sure. I'm not convinced there is.

Like I said, I get the logic behind it. I just don't think it's any more likely to produce an unduplicated sequence than a randomly generated string would. If anything, an intentionally chosen sequence (e.g., 1,2,3,4,5,6) seems _more_ likely to be chosen by someone else. Assuming that nobody else (or, at least, fewer people) will ever choose a specific sequence is a pretty naive assumption, IMO. I totally understand the thinking behind it, but I don't think you're likely to do better than a random number generator at outmaneuvering the X hundred million other tickets in play.

<comment><USER5> By definition, if there are numbers which are chosen more often than those randomly generated, then there must also be numbers which are chosen less often. I'm not saying that those numbers are 1,2,3,4,5,6 - just that they exist.

So we are left with an exercise in psychology - should you pick a sequence that seems common, because the people thinking about it will pick something else because it seems common? Or should you not pick the sequence because people will pick the sequence because they think that people thinking about it will pick something else because it seems common? Some actual data would solve this, but I imagine the lottery folks would be reluctant to release it.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> In my opinion, using expected value to evaluate lotteries is not correct. Realistically, there are only two outcomes. Either you win millions, or you lose a few thousand over several years. The future where your outcome matches the expected value never actually occurs.

This is unlike games like blackjack, poker, or roulette, where the expected value does match the eventual outcome.

So if you can live with the second outcome, playing the lottery is fine. But using expected value does not seem appropriate to me.

<comment><USER8> > In my opinion, using expected value to evaluate lotteries is not correct. > Realistically,

Expected value is realistic. The relation to casino games is silly seeing as the timelines for a few hands of blackjack are several orders of magnitude shorter than buying a ticket for a $300+ million dollar lottery.

</comment></comment><comment><USER9> Sad to see it not make the Kelly criteria argument for ruin, even where there is a positive scalar expectation: [http://r6.ca/blog/20090522T015739Z.html](http://r6.ca/blog/20090522T015739Z.html)

<comment><AUTHOR> O.P. here. The Kelly criterion is a product of the logic outlined in the section about utility being the log of wealth. In short, assuming a jackpot of under $1 trillion, you'd need to have a bankroll on the order of eight digits to justify spending $2 on a ticket under the Kelly criterion.

Also, the Kelly criterion doesn't exactly apply here because this is more like a one-off event than a repeated game. But a lot of the same principles apply.

<comment><USER10> not sure I agree with you.

Kelly says, if there is a positive expected value, there is an optimal amount to bet, which maximizes the growth rate of your stack per bet, if you were to bet repeatedly. If you overbet, your expected growth rate is negative... because over the long run, by the time you hit the jackpot you've lost too much of your stack to get back to even.

Basically, I think Kelly says you can't just go by expected value, ie your edge, you have to look at your stack. Overbetting turns a +EV bet into a -EV bet and eventual loss of your whole stack.

So 1) I don't think Kelly is a product of log utility, it just requires that you are trying to maximize your growth rate (As an aside, in many settings people are much more risk averse than log utility implies; and in many settings people don't have consistent cardinal utility, a time-inconsistent utility like prospect theory is more predictive) and 2) since you are presented with interesting bets every day, if not this particular one, Kelly generally applies, since if you don't heed it you fall victim to gambler's curse.

<comment><USER11> > Kelly says, if there is a positive expected value, there is an optimal > amount to bet

Yes, and the optimal amount to bet here is for all practical purposes equal to zero. In short, you don't have a large enough stack to keep on taking this bet until it pays you back.

I tend to agree with your wider point, that you can treat life as a long-term sequence of taking risky bets and use Kelly to approximate that.

However, even if you did have an equivalent opportunity to this all day every day (which is itself hard to assert), Kelly is saying you _definitely_ shouldn't take it here. If you don't have that repeated opportunity, Kelly- type thinking just leads to an even stronger conclusion - you _definitely_ _definitely_ shouldn't take it.

Good luck to everyone if you're playing the lottery :) Until the numbers are drawn, you still have a chance ;)

<comment><USER10> yes, agree with you and OP! just Kelly makes the OP's advice not to play stronger, even if you get +EV, unless you're a millionaire you will overbet and go broke before you win !

</comment></comment></comment></comment></comment><comment><USER12> I love this analysis. Growing up in Las Vegas we did probability analysis on Casino games as a homework exercise. And it explained _exactly_ why Casinos are a license to print money. It's even weirder when you consider the case where you gamble $10/week versus save/invest $10/week in a compound interest account. Guess who is happier 20 years later?

But of course our gambler is more entertained so there is something to be said for that too.

<comment><USER13> Lots of investments are very risky, almost like gambling but the odds aren't supposed to be stacked (return related to risk). Many people are addicted to playing the market would be (and often are) the same ones who would be addicted to games of chance.

I don't see why we have casinos at all. Just replace them with exquisite trading houses with over blown transaction fees.

</comment></comment><comment><USER14> The odds are beyond terrible, but they are exactly zero if you don't play.

My minimum required jackpot for me to play is a $320,000,000 jackpot. I increase the minimum every year. Keeps my maximum spending to about $4 or $6 a year, which is worth it for the entertaining daydreaming it provides.

<comment><USER15> In fact the return is 100% if you don't play.

<comment><USER16> $0/$0

The return is undefined actually.

<comment><USER13> I thought that was infinity? Ah, but that would cause problems.

<comment><USER17> This concerns real things, not mathematical approximations. So, we divide 0 dollar over 0 people, that means 0 people get 0 dollar. The answer is clearly 0.

<comment><USER18> But _each_ person, of which there aren't any, gets an undefined amount.

</comment></comment></comment></comment></comment></comment><comment><USER19> I've never really thought of playing as an investment. I've always thought of it as a donation to the education system that also has a potential for a large payout. I don't play very often -- 2 or 3 times a year. I never feel bad about losing the money. The way I see it, I would've donated it to some cause. That, to me, makes it worth playing for.

<comment><USER20> Until you realize just how little of that money makes it to the schools :-/

<comment><USER21> Especially as it becomes an excuse to fund the schools by that much _less_ from the general budget. It's an exquisite accounting lie.

</comment></comment></comment><comment><USER22> Interesting article.

In the past I've used the powerball simulator to get a feel for things. Buying two tickets a week for hundreds of years, and tracking profit/loss:

[http://justwebware.com/powerball/powerball.html](http://justwebware.com/powerball/powerball.html)

</comment><comment><USER23> In the 'but then it gets even more complicated' spirit:

When the author mentions buying a ticket in the end, for the entertainment of fantasising about a potential windfall, I was reminded of this interesting Less Wrong article: "Lotteries: A Waste of Hope" which argues even that is a bad reason to play lotteries.

[http://lesswrong.com/lw/hl/lotteries_a_waste_of_hope/](http://lesswrong.com/lw/hl/lotteries_a_waste_of_hope/)

<comment><AUTHOR> OP here. Interesting article. "Fantasizing" isn't what I was trying to describe in the "entertainment" section. I 86'ed a longer explanation of what I meant, but here goes:

When I say, "I also enjoy letting my mind wander and think about what I'd do with a nine-figure windfall," I actually mean that I think about it, not that I fantasize about it. Some of my problems would go away. Others wouldn't. I'd have a whole new set of issues to think about. And I'd still come in to work [0] tomorrow.

When I think about what it would be like to win the Powerball jackpot, I tend to reflect on some of the issues raised in PG's "Cities and Ambition" essay [1], when he talks about the things different cities value: "New York is pretty impressed by a billion dollars even if you merely inherited it. In Silicon Valley no one would care except a few real estate agents. What matters in Silicon Valley is how much effect you have on the world."

The utility I get from buying a lottery ticket isn't about dreams of private jets and caviar. It's the perspective I gain: if I didn't have to think about money ever again, what would I want to do with my life? I think I'd keep working on startups. What would I do differently? I'm not sure, but I'd start with buying a better set of wheels for my bike. In the bigger picture, I'd probably spend more time on projects that have the potential to have huge impact, and think less about whether a particular idea can be a profitable business.

It's easy to get caught up in the patterns of daily life. Thinking about winning a jackpot makes me evaluate my life from 30,000 feet. I guess I could do that for free, but at least for me, that $2 gets me thinking a little differently. And to me, that's where the value is.

[0] [http://zeromailer.com](http://zeromailer.com) [1] [http://www.paulgraham.com/cities.html](http://www.paulgraham.com/cities.html)

<comment><USER24> Your end result is useful, but it's bad that the most effective way to get there is wasting money to trick your stupid human brain.

<comment><USER25> I often think about finding a winning lottery ticket for the same reason.

I find the odds indistinguishable from winning after purchasing a ticket.

</comment></comment></comment></comment><comment><USER26> You should play in a workplace pool situation. $5~$10 is a fair price for psychological insurance.

If most of the people in your office win the jackpot, no matter how Vulcan you think you are, it will negatively affect you psychologically for many years.

</comment><comment><USER27> This post shows one of the limitations to medium.com, namely, a lack for math/LaTeX formatting. This post was well written, but occasionally difficult to parse.

<comment><AUTHOR> Agreed. Can you recommend a better alternative?

<comment><USER28> functionspace.org looks nice

</comment></comment></comment><comment><USER29> It's darn near impossible to justify buying a ticket as a rational, wise investment (as the article points out) due to the odds, split jackpots at that size, and then taxes. But, if you include the intangible of a night or three of "dreams" of what you would do... then maybe one ticket (and only one ticket) might be okay... $2 for 60 hours of dreaming or a soda?... But then again you might be better without that money as most go broke and their relationships and lives are ruined over money. I'd recommend purchasing a $1 pick 5 game, often times with odds around 1:170,000 if you want to but a dream and not win enough to completely ruin your life. Or don't buy a ticket at all.

</comment><comment><USER30> Betteridge's Law of Headlines continues to apply. [http://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines](http://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines)

</comment><comment><USER31> I don't think a high enough dollar value is enough to make it worth playing. Over a certain amount, the winnings are simply just "arbitrarily large". If the prize was a trillion dollars, but each ticket cost $100 to play, does that mean it's now worth it for everyone to buy a ticket? You're still not going to win.

I think stuff like this should be weighted towards chance of winning. Given the choice between a 1 in a million chance of winning $2 million, or a 50% chance of winning $4, I'd pick the $4 one every time. They have the same expected value, but one of them gives me a chance of winning anything at all.

</comment><comment><USER32> The return is so huge compared to investment that you should play based on... what ticket cost means to you. Would you play if ticket's cost would be $0.01? Of course you should - $0.01 won't make any difference to you.

</comment><comment><USER33> This doesn't take into account the value of the "hope" that buying a lottery ticket gets you. Specifically, the owner of a lottery ticket has bought the right to dream about how their life could suddenly be turned around if only their numbers came up.

I admit that this "hope" may not be worth much to some people, but to those who are scraping by it might be worth quite a lot. (I will leave discussion about whether lotteries exploit the poor to another day.)

</comment><comment><USER34> > [https://medium.com/p/28c5a31cd41d](https://medium.com/p/28c5a31cd41d)

What kind of URL is that?

<comment><USER35> "permanent"

</comment></comment><comment><USER36> Something interesting for the mathematically inclined.

"Statistical auditing and randomness test of lotto k/N-type games", available freely at [http://arxiv.org/abs/0806.4595v1](http://arxiv.org/abs/0806.4595v1)

Powerball is not like a typical k/N lotto game though, but you might still want to skim the paper.

<comment><AUTHOR> Interesting, I'll have to give that paper a deeper look. I have noticed seemingly skewed lottery results in the past, but I've never found a case where the skewness of the results made a material difference to the optimal strategy (excluding cases where computer RNGs were configured incorrectly).

</comment></comment><comment><USER37> I've always felt like the value of playing the lottery is not in the numerical outcome, but rather the enjoyment of thinking about what could happen if you win. How much enjoyment you get from dreaming about the outcome can, for some people, be well worth the few dollars it costs for the ticket.

</comment><comment><USER38> More and more blogs moving to Medium for some reason. This makes me sad, because it means horrible formatting with anything code or math related. At least until they fix that...

</comment><comment><USER39> Does anyone actually consider whether they will be happier if they win the lottery? (I don't play the lottery on purpose.)

</comment><comment><USER40> Interesting article, but hard to put mathematical facts over psychology in cases like these.

</comment><comment><USER41> If you don't play, you _definitely_ won't win it.</comment>


Is having a good grasp of computer science basics required for web programming? - <AUTHOR>

There's lot of talk about computer science basis in hackernews, but in most job descriptions I see only mention of web frameworks and other stuff like javascript, dbs and what not.
Though I do see computer science degree as requirement in most of them and fully understand that with Rails and other frameworks its easy to create a web app thanks to numerous open source libraries and tutorials online.
The reason I'm asking is I was wondering if I should spend at least the next 3 months learning about the computer science basics or spend that time getting better at Ruby and learning new technologies that I'll directly use.
Sure learning both is better, but I'm more interested in web programming than writing a compiler or other intensive tasks. (True the experience and critical thinking would be helpful.)
Thanks. <comment><USER2> Yes and no. In a vacuum, I think the more you know about CS the better--for instance, if you want to be a "full-stack"[0] programmer, reading a bunch of Rails tutorials won't be enough. Unfortunately, studying CS for three months probably won't help much either.

It's kind of a dumb analogy, but I think learning how to do "pedestrian" programming is a bit like learning to translate a Chinese newspaper. There are broadly speaking two main algorithms: (1) read until you get stuck, ask for help, and then repeat until finished; (2) go study Mandarin in college for four years, and _then_ translate the newspaper.

If you need to translate the newspaper by tomorrow, then yeah, do (1). If you think you'll need to translate a ton of newspapers and have the luxury to study for a while, the second method might yield better results in the long run.

[0] [http://www.facebook.com/notes/facebook-engineering/the- full-...](http://www.facebook.com/notes/facebook-engineering/the-full-stack- part-i/461505383919)

<comment><AUTHOR> That's actually a really good analogy and a great link. But I don't think "full stalk developer" is what really suits me. Maybe my definition of web developer is more limited than its original meaning, I was thinking more in terms of designing a decent sized, complex web app, but not going as far as building hulu or facebook singlehandedly without an actual scalability engineer.

<comment><USER2> Yeah, then I think you're better off relentlessly building things. You'll have plenty of opportunities to learn CS incrementally as you go.

Personally, I know that I have a pretty extreme bias towards wanting to "study forever", which is really irritating. I have to keep reminding myself that building things takes practice too!

</comment></comment></comment><comment><USER3> From experience, I can tell you that you will feel a lot better about what you create and you will feel a lot more confident in your code if you are creating things according to proven design patterns and not just hacking together applications from tutorials.

<comment><AUTHOR> That was more of a general example rather than personal practice. I completely agree when you say using proven design patterns makes you more confident, just the other day I was implementing something I read from Design Patterns for Ruby. What I'm more curious about is whether I should stay the course to try to become an intermediate Ruby programmer or try to learn some basic computer science stuff, since everything I know about programming comes from Ruby related books. Not just half pay online tutorials, but "real" ones like "Meta- programming for Ruby" and "Ruby Way." At least from what I can tell, they are one of the most advanced Ruby books around.</comment></comment>


Ask HN: How to teach my kids programming? - <AUTHOR> Has anyone tried doing this? I want to them to be interested but not force it on them. <comment><USER2> When I was in the 3rd grade some time ago, an amazing teacher decided to bring in his personal apple IIe and allow interested students to hack after school. He taught us "Logo", a simplified lisp variant that I remember being very intuitive. After we understood the basics, we began building robots with Lego/Logo, a Logo variant you could use with an external controller that operated motors an sensors embedded in lego bricks. It was a transcendent experience for me. What's more, as a kenyan immigrant living in midwestern suburbia in the early 90's, it was one of the first times I felt as though I had a meaningful place in the world. I never stopped programming and its why I'm an engineer today.

Logo Project: [http://el.media.mit.edu/logo- foundation/logo/programming.htm...](http://el.media.mit.edu/logo- foundation/logo/programming.html)

Lego Logo Project: [http://www.ucls.uchicago.edu/students/projects/1994-95/Lego-...](http://www.ucls.uchicago.edu/students/projects/1994-95/Lego- Logo/ProjectDescription.html)

<comment><USER3> Logo is very old, and interesting, and since its invention it was about robots, the first logo robots usually were a circle with wheels and a pen in the middle, and logo students made geometry or trigonometric applications that you would see the results drawn on the floor.

</comment></comment><comment><USER4> Well, what age are they? I'd say age and abstractness of their thought process dictate how to teach them.

I'd argue that you start learning programming concepts in everyday life, early on, just nobody explains them to you.

The lunch line at school. Line Up, first in, first out. Right? -Array/Vector (depending on if you know the number of kids in line, like a single class of, say, 30 or if kids just all get in a line together.) What about "Budging", kids getting kicked out of line for bad behaviors....

For the cafeteria workers this is a While Loop \- While there are kids in line, serve them.

I can think of so many everyday life happenings that would teach kids computer science fundamentals if presented in such a manner to them.

I have used: [http://www.amazon.com/Hello-World-Computer-Programming- Begin...](http://www.amazon.com/Hello-World-Computer-Programming- Beginners/dp/1933988495/ref=sr_1_1?ie=UTF8&qid=1358563128&sr=8-1&keywords=Hello+World) (Not an affiliate link, FYI)

<comment><USER5> The lunch line is awesome example of real world programming concepts.

Myself I have taught few people how to understand object orientation like this. Though my example is not good for kids, but older people.

Told them to think about a beer can and asked them what kind of properties they have. Amount of beer in it, the color of the can, the material of the can, the labels and icons. After they realize these properties I have explained them that how they could code these as class variables.

If they understood that I asked them what kind of functionality beer can has. They have answered, opening it and drinking from it. I told them these are the two functions for the class. The opening of beer can is void method with no parameters and drinking from it is a method with parameter and possibly with a return value.

If they understood that I could go into the relationship of beer can and beer crate.

</comment></comment><comment><USER6> Create a development "playground" for your children. The way children learn anything is by trial and error, remember them trying to learn how to walk? Remember how many times they fell before being able to walk?

Do the same with teaching them to code. Pull HTML source code from a simple website and paste it to your text editor. Then let your child play around with the code and see how it renders in the browser. It will be fun and exciting for them because they will actually get to see their work right away (instant gratification).

Once you have done this, move on to css.

Good luck!

<comment><USER7> The reasons I recommend against HTML+Javascript+CSS are because they don't teach algorithms and data structures. If a kids learns HTML first he/she will not now anything about algorithms. To learn that she/she will have to move on to another language: javascript.

Another reason is that those web language combos are forever being extended. There is no way children will be able to hold the entire language in their heads. Some people may not agree but even C's success is due to the ability of one person to remember the entire language all at once.

Still another reason is compatibility or the way those languages behave in different browsers. Everyone has his own proprietary css extensions these days, what will a kid need to learn next? JQuery? Ext-JS?

Start with simlicity. Learn a language that really offers instant gratification, with graphics, sound,and file keywords included. Have them learn BASIC. They'll perfecly be able to move on from there.

<comment><USER8> _If a kids learns HTML first he/she will not now anything about algorithms._

That's just weird to me. Do you have children? Did you teach them this way, it worked successfully, and now you're passing that information along as a public service? Or are you just talking on a public forum about something in which you have no first hand knowledge? I'm always suspicious of taking advice from someone who just _thinks_ something should work rather than from someone who has actually practiced what they are preaching (and, in particular, I'm suspicious of "parenting advice" from someone who doesn't have children). HN seems to be full of "This is what my parents did for me and I turned out okay, therefore this must be the right way" type of talk.

 _Start with simlicity. Learn a language that really offers instant gratification, with graphics, sound,and file keywords included._

I fail to understand how HTML/JS/CSS does not meet this guideline. I don't understand why you think that programming in BASIC is a better start than JS. JS has algorithms and all the basic implementation that something like BASIC is going to have.

 _There is no way children will be able to hold the entire language in their heads._

There is no way you or I at our age can keep the "entire language" of any sufficiently advanced language in our head. I don't see how this matters.

Sorry - I just don't like it when someone positions them self as an expert during a reply yet there is no context provided therein that would improve/prove their position.

<comment><USER7> I've learned by meself when I was 12, I had no one to teach me. It took me two days to understand BASIC and one week to adapt to DBASE III+

As for context, says Paul Graham: "Use succinct languages. More powerful programming languages make programs shorter. And programmers seem to think of programs at least partially in the language they're using to write them. The more succinct the language, the shorter the program, and the easier it is to load and keep in your head.

You can magnify the effect of a powerful language by using a style called bottom-up programming, where you write programs in multiple layers, the lower ones acting as programming languages for those above. If you do this right, you only have to keep the topmost layer in your head."

More here:<http://www.paulgraham.com/head.html>

How many months does it take to master the HTML+JS+CSS mess?

</comment></comment></comment></comment><comment><USER9> This question gets asked alot. I've never seen a good answer.

The only good reason I've seen for teaching a child programming is that the child is interested in it...and that tends to be several orders of magnitude less common than a parent who wants to teach their child programming.

</comment><comment><USER7> Teach them BASIC first, not the HTML+CSS+Javascript mess. <http://www.freebasic.net/get> Then let them loose on this site: <http://www.petesqbsite.com/>

<comment><USER10> What what you say is a good age to try and introduce a child to programming?

<comment><USER11> I've seen people claim to have started as early as 4 so really whenever both you and the kid feels ready

<comment><USER10> Well my daughter is 5, and she's above average, developmentally speaking.

The problem we keep running into is that my daughter sounds excited about learning something, but when it comes down to actually doing it, she loses interest fast. We haven't tried programming yet, so far we have been experimenting with other stuff because we plan on homeschooling her.

I'm hoping that isn't a trait she inherited from me. I was a terrible student from about 3rd grade until I "finished" high school. Sometime in my early 20s, I suddenly became interested in electronics, math, and programming. Now that I'm a 30 year old, 3rd year college student, I wish I hadn't wasted so many years.

<comment><USER11> I agree, it's a tough balance. I'm a dad and a technical geek myself so I completely understand. Just remember, while we may wish our kids to be interested, they might just have plans of their own. Best we can do is try to introduce them to these things, show how cool it can be, and hope they are willing to learn. I don't believe in forceful learning (not suggesting that's what you are doing).

I know even from personal experience most people don't retain majority of things they've learned growing up or in college unless its something they care about or have done continuously. So maybe your daughter might be ready now, maybe not quite yet. Maybe she'll never be interested. Do the best you can but don't take your ideals upon her if she isn't interested. Believe me, more than anything I hope my kids will have a love for these things too, but its something they must discover to be interesting to them.

On that note, perhaps you could try it from a few different angles and see how it works. Five is still a pretty young age and the average is probably usually a few years older so don't lose hope if she isn't interested just yet

</comment></comment></comment></comment></comment><comment><USER3> When I was six, my father gave me a GWBASIC and those magazines with game source codes.

</comment><comment><USER12> Just let them enjoy Minecraft.</comment>


Monetizing your Web App: Business Model Options - <AUTHOR> http://www.boxuk.com/blog/monetizing-your-web-app-business-models

<comment><AUTHOR> I should point out that graph at the bottom of the post took ages to research; I hope someone appreciates it!</comment>


Jenkins pipelines as YAML - <AUTHOR> https://jenkins.io/blog/2018/07/17/simple-pull-request-plugin/ <comment><USER2> This is highly confusing to me. Pipelines in Jenkins are fragile. You can spend entire days trying to do _very_ common things and run into tons of friction. The differences between scripted and declarative are severe, in terms of how erros and continuation are handled, the way that plugins interact depending on how you've nested stages and steps. Even things like environment vars and CWD can be a pain to coordinate. Trivial pipelines take far, far, far too long to setup and get working well.

This just appears to be a layer on top of the still-limited declarative pipelines. I'm not sure trading a groovy DSL for an extra layer (of yaml, no less) is a good deal.

It doesn't solve the weirdness around multibranch, it doesn't address the mismatch between having a parameterized pipeline where that parameterization has to live outside the Jenkinsfile (or if it is encoded in the Jenkinsfile, it acts really oddly, like running the Jenkinsfile actually reconfigures the job that invoked it, etc). This is why everyone has to continue using Groovy DSL and/or JJB to reinstantiate parameterized jobs or handle jobs that deal with multiple Jenkinsfiles in a project.

I know Jenkins isn't going anywhere but it's legacy shows a lot.

<comment><USER3> First, full disclosure, I'm the creator of Jenkins.

I'm sorry to hear that you had a bad experience with Jenkins Pipeline. I can see that you know a lot about it.

There's actually no difference between how continuation is handled between scripted and declarative, so I'm curious to know more about what hit you, because I suspect it's something else (though obviously equally frustrating!) Similarly curious about improvement to error reporting, because I think that's something the Pipeline team cares about, and it's one of those things where how errors in real world are made is always more interesting than what we can imagine. I used to work on compilers, so I know the frustration of a poor error message pushing you down a wrong lane, only to discover a few hours later that all it took was one line fix! Modern improvements in Pipeline (like declarative and this one) is in no small part motivated by making those error checks more thorough, easier, and more upfront. So I think this is a change in the right direction.

My perception has been that parameterized jobs are in decline, in part because more people are triggering automation implicitly through commits, and not through explicit "run this" button. Parameters are more implied from the context (branch, commit message, creation of tags, etc), as opposed to explicitly given from UI.

Stepping back from those specifics, I think regrettably software has bugs, and there's always more usability improvements that can be made, so we are just working on those one at a time, which kinda summarize my entire journey with Jenkins :-) So in that spirit, I want to make sure we learn from your suffering.

<comment><USER2> Thanks for the response. Mine is a bit terse as I'm on mobile for a few days.

For one, the always/finally block in declarative pipelines flatly doesn't work in scripted mode. You have to set/maintain job status yourself and throw/catch. It's very unpleasant.

I am shocked to hear that parameterized builds are in decline. Every project I've seen or touched that used Jenkins had multiple pipelines in a repo that needed to be executed with a number of different configurations. If there was better native support for pipelines calling other pipelines, this _might_ not be such a problem.

Example: a kube related project. It has a Jenkinsfile for release builds and another for general checkin/PR builds. Each of those need to be tested with two versions of kube, with flannel and calico, with and without RBAC. I don't know of a clean way to do that without Job DSL/JJB and parameterized jobs. In my experience, this type of scenario is not uncommon. We need visibility at the config level. It's not helpful to shove all of that into a single job with coarse -grain reporting up.

Even if you put parameters aside, there is just a fundamental mismatch because the pipeline only (today) encodes steps. What about triggers? Having them be part of the pipeline and having the pipeline modify it's own job in Jenkins feels very, very, very wrong to me. There's a reason Job DSL and JJB kept those separate. In fact, if it weren't for the Pipeline visualization, I could've gotten the same functionality with JJB /Job DSL alone, with some bash scripts, with considerably less heartache.

I don't know that I know a "lot" about Jenkins, but I do know that I strive for easily repeatable setups that need git triggers, need to post back status, and need to run bash scripts. This most recent attempt was the third time owning it all-up and it was still very tedious. And it was really much easier to achieve this simple requirements with k8s's Prow... Even when having to dive in and hack some of prow's plugins.

All of the mature, repeatable Jenkins setups I've seen are small mountains of groovy scripts with their own workarounds embedded. There needs to be a system config DSL too, because maintaining huge blobs of XML and writing groovy.init.d scripts keep me awake at night.

Anyway, Jenkins is crazy good stuff, especially for it's age, but for newer smaller projects that don't need the 10,000 Jenkins plugins, it feels unwiedly. I had hope that Jenkins X was going to tackle some of these things, but so far I'm not sure.

Thanks again for taking the time, I hope my further feedback is constructive.

<comment><USER3> Thanks, this is really helpful.

I need to find someone from the pipeline team to pull into this, so in the mean time, just responding to what I can contribute,

Just to make sure, I'm not saying parameters are disappearing. I'm just making an observation that less people seem to be using it. Take your example of release vs general checkin/PR builds. I see increasing people doing releases as automation that kicks in after creating a tag, or by cutting a release branch. Or the master branch is always deployable.

I agree with you that reporting capability needs to be better in order for one Jenkinsfile to pack lots of different test cases. I believe the team totally gets this importance.

The "system config DSL" you mention has evolved into "Jenkins config as code", and I've referred to it in my other comment. I think we are on the same page that it's a crucial part of a repeatable mature Jenkins setup. I think I also totally get what you mean by "unwieldy", and Jenkins Essentials in that comment is making steps to attack that challenge.

I'd love to hear from you where Jenkins X fell short for you, because I think it should speak to some of the challenges by embracing a certain best practices.

</comment></comment></comment></comment><comment><USER3> Creator of Jenkins here.

First of all, this is a Google Summer of Code project. Abhishek, who is driving this work, is doing a great work, so I hope people can give him encouragements and feedbacks to push him forward. I'll make sure he sees those feedbacks and will stop by to answer questions you might have.

This is one of the efforts that is pushing the envelop of Jenkins that solve problems people had with Jenkins. Reading some of the reactions here, I wanted to use this opportunity to introduce other bigger efforts going on currently in Jenkins that I think addresses various points raised in this thread.

* Jenkins Essentials is aiming to be the kind of "readily usable out of the box" Jenkins distribution that is a lot less fragile, because it's self-updating appliance that has sane defaults and obvious path to success.

* There's architecture effort going on to change the deep guts of Jenkins so that data won't have to be on the file system, and instead go to managed data services.

* Jenkins configuration as code lets you define the entire Jenkins configuration in YAML and launch Jenkins as a docker container to do immutable infra. Jenkins Pipeline lets you define your pipeline in your Git repo, so that's the other part of immutable infra, and between modern pipeline and efforts liek this one, there's no need to write Groovy per se. It's just a configuration syntax based on brackets like nginx, which happens to conform to Groovy syntax, so that when you need to do a little bit of complicated stuff you can, but you don't need to

* Finally, Jenkins X is focused on making CD a whole lot easier for people using and developing apps for Kubernetes. It's a great example of how the community can take advantages of the flexibility & the OSS nature of Jenkins to the advantages of users.

* A few people mention about container-based build environment, which is very much a central paradigm with modern Jenkins (and thus obviously with Jenkins Essentials and Jenkins X.) See our very first page of the tutorial! [https://jenkins.io/doc/pipeline/tour/hello-world/](https://jenkins.io/doc/pipeline/tour/hello-world/)

<comment><USER4> Jenkins Essentials sounds great. I've become very good at not breaking Jenkins installations. Jenkins is a powerful tool, but it's hard to recommend when that's a skill you must acquire.

I remember when I installed the git plugin, which forced the credentials plugin to update, which caused runtime failures in communication with some other core plugin, so we reverted the install/update, which broke the whole system because the update had renamed fields in the configuration and the old version didn't understand them...

Since then, I have always updated all core plugins together in lockstep. Based on the name and description, that sounds like what Jenkins Essentials would do too. If so, that's a good sign.

Simpler, more reliable administration is exactly what Jenkins needs and you seem to have a credible way of achieving it, so I'm excited to see the results.

</comment></comment><comment><USER5> Looks interesting, although it seems very similar to another project, Jenkins Job Builder ([https://docs.openstack.org/infra/jenkins-job- builder/](https://docs.openstack.org/infra/jenkins-job-builder/)) that can describe jobs as YAML or JSON, and is designed one level higher as a meta-jobs creator that can create many jobs all with slight variations, with all of the config kept in version control (no manual fiddling with the Jenkins GUI, other than troubleshooting). With JJB, most jobs aren't pipelines, but traditional Groovy-based pipeline scripts can be included and run.

It would be great if there was a pre-run remote lint step like can be run on Declarative Pipeline jobs ([https://jenkins.io/doc/book/pipeline/development/#linter](https://jenkins.io/doc/book/pipeline/development/#linter)).

<comment><USER6> Also highly recommended is Jenkins Job DSL, which, despite also using Groovy like Jenkins Pipeline, takes a far different approach.

In Job DSL, you write Groovy scripts to declare your job. These scripts, in turn, write the config.xml that Jenkins uses to actually define and execute jobs.

One of the big benefits to us is that it is _not_ an alternate execution engine like Jenkins Pipeline is. It just takes over the job of writing the job XML for you, instead of editing it through the GUI.

We had a large, complex, manually 'versioned' job chain that did not lend itself to being re-written in Jenkins Pipeline, as it made heavy use of plugins that were not supported by Pipeline at the time.

By using Job DSL, we were able to incrementally replace the manually maintained jobs with script-generated ones. Importantly, with nothing breaking for end users! All download links, URLs, etc of the new jobs exactly matched the old ones, so it was a very painless, incremental adoption that people did not even notice!

If you've got an existing job chain you are trying to get under control, Job DSL comes highly recommended.

<comment><USER7> Agreed, I love the groovy jenkins job DSL. It's so awesome and simple. Easy to put everything on github and forget about it.

<comment><USER8> Besides all the special cases in the normal use of Apache Groovy (e.g. definition of `==` different to Java), you also have to remember all the Jenkins-specific special cases (e.g. Groovy's collections methods don't work). I wouldn't call it simple or easy at all.

<comment><USER6> Are you sure you're not thinking of Jenkins _Pipeline_?

Jenkins _Pipeline_ definitely has these issues. Jenkins _Job DSL_ has always worked fine, including all the special collection method operations (grep, find, collect, each...)

</comment></comment></comment></comment></comment><comment><USER9> I think it totally makes sense to do stuff like this as long as you keep a very thin layer of abstraction between your YAML declarations and Jenkins config. Like, someone familiar with Jenkins and the naming conventions should easily be able to come in and map what buttons and boxes they fill in the UI to what is declared in the YAML. I've worked with several build systems that try to get way too clever with this and all you end up with is huge amounts of config files that only a few people are able to understand and manage (even if there is nice documentation!). Build systems are just as important for following the Principle of Least Astonishment as codebases are.

</comment><comment><USER10> This is just lowering themselves to the same level as every other CI tool. For sure YAML is an easier barrier to entry to overcome, but it's also a low ceiling.

YAML doesn't let you transcribe logic easily compared to any programming language, and build logic always tends to go up. Programming languages also allows you to create abstraction to remove some of the complexity from the users.

<comment><USER11> My experience with yml based builds and deployments is basically using the yml file to tell the build system what code to run to do the build - usually other shell scripts or some other scripting language that can also be in the same git repo.

<comment><USER12> I read yaml for ansible like an ordered array.

It's a namespace with the values you want inline, in my head

They're INI files or .cfg to me

Not code

The code is the Python in Ansible, or Go in TF, that applies those values

Using config mgmt tools like they're vanilla programming languages is weird.

<comment><USER11> Exactly. Why would I want to learn another pseudo scripting language when I have Powershell, bash, Python, etc at my disposal?

</comment></comment></comment></comment><comment><USER13> I am so grateful for this. As a non-Java person, using the Groovy pipelines was incredibly painful to incrementally fix on the server. Nothing like pushing 20 commits in a row and forcing a build, only to find each time that an odd syntax error.

<comment><USER14> Still have to do that to build the thing in the first place. I do not understand at all how every CI solution out there makes it entirely impossible to trivially test a pipeline. Here we are building something whose entire point is that it can run _wherever_ , but the one place it can't run is my local machine? It's like they are doing it out of spite.

<comment><USER15> I seriously don't understand how anyone can use most of these CI solution.

Here is what everyone should demand from a CI solution:

* Actual code that specifies a pipeline (not a GUI/config-file/DSL)

* Supports any SCM, workflow, build system, and language

* Supports complex mixed-language, mixed-build-system projects

* Doesn't make assumptions about project's structure and processes

* Able to run locally (but scales to clusters, etc)

* Written in a common scripting language

* Completely end-to-end customizable in the scripting language it's written in

The only one I know that fits is BuildBot, but practically nobody uses it.

[edit: formatting]

<comment><USER11> Well if the whole world has a different perspective than you, have you thought that maybe their wants/needs are different?

<comment><USER15> Sure that could be the case, but I think most people that set up and use CI just pick Jenkins or whatever on a whim, then we all read about horror stories and bad experiences because the solution they picked is a square peg and they're forcing it in a round hole.

<comment><USER11> Why do I need all of that bespoked complexity, my yml file is just there to tell the build system the scripts or shell commands I want to run - that are also in my git repo.

Fundamentally, a CI system for a simple application just has to gather dependencies, build a package/run automated tests and store the artifact somewhere.

A deployment system just has to copy the deployment package to a server/group of servers, and do some type of installation process based on an automatic (Dev/integration environment) or manual approval process (who ever is responsible for pushing to any other environment has to approve the release). There are a million ways to skin the cat but if your build process or deployment process is too complex, maybe it's more of a question of the maturity of your framework or tooling.

My build process for .Net web apps is:

\- nuget restore

\- msbuild

\- run nunit

\- zip artifacts based on version.

My deployment process is

\- run a CloudFormation file to configure AWS resources

\- deploy code to a VM

\- run one or two commands to configure IIS or install a Windows service.

\- run a few AWS CLI commands to start autoscaling, reconfigure API gateway, etc.

No build servers to maintain, just an agent running on the target VM for deployment and a slightly custom Docker image for builds.

—-

For Lambda functions and scripting languages it's even simpler...

Build:

\- import deprndencies \- create zip file

Deployment

\- run CloudFormation YML file to deploy lambda

The CF file can be the deployment step. If I need some programmatic, I can create a custom lambda backed resource that is run when the CF file is run.

</comment></comment></comment></comment></comment></comment><comment><USER16> We have Jenkins connected to ECS as build agent pool. Every team has it's own Jenkins instance running, secured with AD login. Pipelines are using Jenkinsfiles. My only critique is that it's not fully ci as code because you need to manually create the pipeline, connect it to fit repo, etc.

<comment><USER17> It may not work for you but "organization folders" (a feature of multi branch) were designed to discover projects from bitbucket, github etc (ie a new repo appears, with a Jenkinsfile, a new project will automatically configured). Some people have had luck with them (they don't always work how you want though, but when they do, don't need to manually create the pipeline other than dropping in a Jenkinsfile in a new repo).

</comment></comment><comment><USER18> I'm wondering, who uses Jenkins nowadays and why?

A few years ago, at one of my first gigs in an enterprise environment, I used Jenkins for the first time to test and build my stuff. When I needed a newer version of my compiler or specific linter, some fellow had to install that on the VM that was running Jenkins.

Later, working in a more modern environment, we started using Gitlab CI. It was a bliss. I specified the docker image with my favourite tooling and my stuff got built in there. When some tooling changed, I updated my image.

At my current gig, again an enterprise, it is Jenkins everywhere. They do the most complex things with it, orchestrating entire releases, integration tests, etc. I don't know what to think of this yet.

How does the HN crowd see this?

<comment><USER19> We're using it because it costs nothing and it's what everyone already knows. We're making a big architectural shift to microservices and I made a pitch for the Travis / CircleCI style workflow, but after Jenkins pipelines were discovered, that was the compromise that was made.

We've only got our toes in it now, but from what it looks like, you theoretically _can_ use a Jenkins pipeline (with a Jenkinsfile) to get some of the benefits of those systems, but the problem is that they also allow you to rule out the others. Your Jenkinsfile can assume certain plugins are installed, assume other jobs are configured on the same Jenkins instance... basically all the things that led to Jenkins becoming what it has in terms of a carefully configured sacred cow that must be meticulously backed up and everyone is scared to update.

Having builds trigger on push isn't easy if you're not using GitHub or BitBucket, and having a series of pipelines that trigger off of each other is... not clean. You can certainly trigger another job as a "post" action just like you could in any other Jenkins job, but now your upstream job contains the logic for whether or not a downstream job is triggered. What if your downstream project (like a VM image) only wants builds from a certain branch? Or should hold off on new builds from a certain microservice while QA completes their testing? I guess you'll need to edit the Jenkinsfile for the upstream project (likely someone else's project) and be careful not to break it.

<comment><USER20> >> our Jenkinsfile can assume certain plugins are installed, assume other jobs are configured on the same Jenkins

I think what worked for us in this case was using Jenkins shared library[1]. We provide a common template for the common stacks and expose only few configurable options. This would really help in maintaining sanity across the jenkins env and since you maintain the shared lib, you can control the dependencies.

[1] - [https://jenkins.io/doc/book/pipeline/shared- libraries/](https://jenkins.io/doc/book/pipeline/shared-libraries/)

</comment></comment></comment><comment><USER21> This would be great if you didn't have to use scripted pipelines for so much stuff. Declarative pipeline isn't there yet.

<comment><USER5> From what I can tell, this creates a mapping of YAML to Declarative pipeline syntax, so it's unlikely to improve on that.

<comment><USER21> Yeah that's what I meant, though I guess it was ambiguous.

</comment></comment></comment><comment><USER22> ha YAML, when mentioned, always reminds that taking a look at the specs[1] is a great reminder that things can derail badly event without a large committee

[1] [http://yaml.org/spec/1.2/spec.html](http://yaml.org/spec/1.2/spec.html)

</comment><comment><USER23> Amazing. I had this project at my previous job but left before getting it done. It's so cool to see that someone, somewhere, had the same need.

</comment><comment><USER24> Maybe they will be cleaner than groovy pipelines..they look really ugly once your building process involves parallel steps

</comment><comment><USER25> There are technologies that are build using COD - consultant oriented development. That is one of the principals is that only a trained consultant would be able to configure it and that would require knowledge impossible to find in freely available documentation. It also needs multitude of gotchas that couldn't be solved using common sense. Jenkins I believe is one of such tools. We have completely moved away from it.

<comment><USER26> I guess rather than having to deal with the agony to move away from a sufficiently large and complex Jenkins setup, people rather delude themselves in that its actually good software.

Just the amount of obtuse setup that thing requires is crazy and always ends in a mess. I have no idea why there seem to be so many people praising that thing in this thread. I don't fault Jenkins though, its just old obsolete software that should be replaced.

</comment></comment><comment> gm-conspiracy Isn't this how drone.io works?

</comment><comment><USER27> I'm excited about this! I would love for pipelines to get easier to configure.

</comment><comment><USER28> I like yaml to edit preexisting files, but they should come with a generator.

<comment><USER29> Jenkins has a declarative pipeline generator, it should be relatively straightforward for it to output YAML rather than the standard groovy based declarative pipeline DSL.

</comment></comment><comment><USER30> Why does everything always need to be in the root of a git repo?

<comment><USER31> I never put my Jenkinsfile in the root. And it works just fine.

If I left it in the root, I wouldnt have to configure it... but I can live with that.

</comment></comment><comment><USER32> What are the reasons one would use Jenkins over Drone?</comment>


A faster way to compute edit distance does not exist - dnt404-1 http://www.bostonglobe.com/ideas/2015/08/10/computer-scientists-have-looked-for-solution-that-doesn-exist/tXO0qNRnbKrClfUPmavifK/story.html <comment><AUTHOR> The title of the Boston Globe article and of the HN submission is misleading.

The article does not prove that subquadratic edit distance algorithms do not exist. It only proves it conditional to another unproven complexity hypothesis (SETH). In other words, it shows that if such algorithms exist, then faster algorithms exist for another problem (SAT for CNF formulae with certain size bounds), where we haven't been able to find them yet. See the original article [http://arxiv.org/abs/1412.0348](http://arxiv.org/abs/1412.0348) whose title is more accurate.

This is important work, of course, and provides more evidence that faster edit distance computation is not possible, but it is not proven yet.

<comment><USER2> There are really two notions of proof here [and hence truths and facts]. The article uses proof in a scientific sense, where P!=NP is accepted as empirical fact by the community of computer scientists in the same manner that biologists take natural selection as empirical fact.

To put it another way, representing the world as if P!=NP is no less reasonable than representing the world as if Newtonian physics are applicable.

<comment><AUTHOR> Theoretical CS is usually not thought of as an experimental or empirical field. When a paper in that field claims "X does not exist", this has to mean that the inexistence of X was proven, not that a body of empirical knowledge against the existence of X has been collected. By contrast, such things are called "plausibility arguments", "heuristics", etc.

There is such a thing as experimental truth, but it applies to experimental fields, and theoretical CS is usually not thought as one of them. I still think it is misleading to claim that the inexistence of something has been proven if it is conditional to another hypothesis.

<comment><USER2> Computer scientists behave as if P!=NP, and Donald Knuth will send a check to anyone who comes up with a polynomial time general solution to 3-sat.

Or to put it another way, P!=NP is a _de facto_ if not _de jure_ axiom of theoretical computer science. And the burden of justification is on anyone whose conclusions are not premised on P!=NP just as it is a biologist who takes issue with evolution by saying "it's only a theory."

</comment></comment></comment></comment><comment><USER3> >The fact that edit distance is only computable in quadratic time has big implications for genomics. The human genome contains about 3 billion base pairs. To compute the edit distance between two human genomes takes 3 billion- squared operations (to appreciate how big that is, write a 9 followed by 18 zeroes). That's a number of operations, explains Piotr Indyk of MIT, that is "computationally infeasible." Which is to say that our best computers chugging away for a really long time still won't generate an answer.

As was commented last time this was posted: Just because it's not possible to do better in the _general case_ doesn't mean it isn't possible to do better in _your case_. For all we know, human genomes may not actually be 100% random strings, but may have some kind of structure that makes it possible to do it faster. Even when solving the problem exactly.

<comment><USER4> Human genomes are very much not random strings, and in any case if calculating the edit distance between two human genomes they are likely largely similar, and if memory serves there are suffix tree/array based algorithms for bounded edit-distance calculations.

Moreover, I'm not sure if it would ever actually be useful to compute the edit distance between two full genomes. Edit distances/local/global alignments are useful for short fragments (gene's/reads etc) but not the full sequence.

<comment><USER5> The challenge with comparing genomes is that you can't just do edit distance using a naive substitution table. It has to be probabilistic and contextual (for it to be scientifically useful), because genome sequences are actually sampled estimates of the true genome sequence. You have to tolerate errors, and when you do that, the exploration space gets huge.

Instead of aligning two genomes this way, people normally start with a very well calibrated and fairly accurate reference sequence, then align to that structure probabilistically, and "call the joint variants".

This is actually far more interesting that doing the raw hard problems that computer scientists love to talk about theoretically. One of the reasons it's so interesting is that different genomes differ by operations that are much more complex than simple edit distance- for example, large regions can be excised, and placed, in reverse order, in completely other parts of the genome. Trying to use a DP edit distance calculation to detecet those is futile.

[http://bcb.io/2014/10/07/joint-calling/](http://bcb.io/2014/10/07/joint- calling/)

</comment></comment></comment><comment><USER6> The name "Wagner-Fischer" is new to me. I usually cite this as Levenshtein [1965]. Even the Wagner-Fischer wp page notes that Levenshtein was first by a long shot. [https://en.wikipedia.org/wiki/Wagner%E2%80%93Fischer_algorit...](https://en.wikipedia.org/wiki/Wagner%E2%80%93Fischer_algorithm)

<comment><USER7> Yeah this struck me as well. I've always heard it described as Levenshtein edit distance.

</comment></comment><comment><USER8> It's a nice theoretical result. In practice there are some quirks though:

\- I assume this is worst case complexity. While this is useful, maybe the average time can be much lower for certain data (like the genome).

\- I assume the theorem doesn't restrict the input strings in any way. If you put restrictions on the input then there could be a faster algorithm than the general one. I don't know if this can be applied to the genome though.

\- People are interested in the constant factor too.

<comment><USER9> Also, a fast approximation with a good error bound might be good enough for many applications in biology.

</comment></comment><comment><USER10> As a quadratic algoritm, assuming 13 million base pairs can be solved in 7 seconds on a GTX275 (according to a paper on edit distance computation), is it correct to assume that 3 billion base pairs can be solved in (3bil * 3bil / (13mil * 13mil/7)) in 373443 seconds, or about 103 hours. Isn't that fast enough for a quadratic algorithm that samples DNA to be useful?

<comment><USER11> I think the article is talking about the "full fledged" problem, while the computation you talk about could be using a set of optimizations aimed particularly at base-pair edit distance problem.

For example, I could assume that there no deletions and insertions, only changes and then the problem becomes embarrasingly paralell and extremely easy to express in assembly-like operations. So I could paralellize it on a GPU by splitting the string into fixed length substrings and get huge gains. Problem is, can it be safely assumed that there are no deletions and insertions?

Wikipedia has a list with some modifications that can also be used to speedup the calculation under certain constraints: [https://en.wikipedia.org/wiki/Wagner%E2%80%93Fischer_algorit...](https://en.wikipedia.org/wiki/Wagner%E2%80%93Fischer_algorithm#Possible_modifications)

</comment></comment><comment><USER12> Previously here: [https://news.ycombinator.com/item?id=9698785](https://news.ycombinator.com/item?id=9698785)

</comment><comment><USER13> That was refreshing; a regular newspaper publishing an article about computer science. And it does a fairly good job, I wish I could see something related to CS rather than to web apps and startups in the Canadian newspapers.

</comment><comment><USER14> An article on edit distance without mentioning the person who invented it - Levenshtein. [https://en.wikipedia.org/wiki/Vladimir_Levenshtein](https://en.wikipedia.org/wiki/Vladimir_Levenshtein)

</comment><comment><USER15> This is interesting, but it looks like this proves nothing according to the article since we don't know if P==NP. We are no closer to understanding whether or not a faster way to compute edit distance exists. All we know now is that it matches up to a class of problems that we _think_ are probably hard. It is still possible that we might find a faster solution to the problem.

I'd like to request a title change - "New proof that a faster way to compute edit distance might be tied up in the P vs. NP problem"

</comment><comment><USER16> Hence the need and desire for approximation algorithms. Often, you can give up a tiny amount of precision in exchange for dramatically increased performance - if you can make an assumption.

For instance, computing cardinality of a value over a huge dataset (distributed or otherwise) that is assumed to be evenly distributed would take memory proportional to the size of the dataset for 100% precision. Implementing a cardinality approximation algorithm like HyperLogLog++ lets you get a pretty close to accurate result for this calculation but with far fewer resources. There are many others and I think it is important to consider this trade off when it is appropriate.

Of course, it makes an assumption that the data is distributed more or less evenly.

</comment><comment><USER17> Now we can start looking for a more precisely defined problem, that _can_ be solved faster than in quadratic time.

Or in other words, for certain classes of inputs, we can find a faster algorithm. We only need to find/define those classes.

</comment><comment><USER18> Does a similar result hold, if we don't consider inserts, deletions and substitutions (definition of edit distance), but only inserts and deletions?

<comment><USER19> How do you define an edit distance (the number of operations required to transform one string to the other) if you don't allow substitutions?

Eg. what's the edit distance between these two strings?

AAAA BBBB

Traditionally, it would be 4, but it's unclear if you allow only inserts and deletions.

<comment><USER18>
      > Eg. what's the edit distance between these two strings?
      >
      > AAAA BBBB
      >
      > Traditionally, it would be 4, but it's unclear if you
      > allow only inserts and deletions.

It is >= 8, since you surely have to remove 4 As and insert 4 Bs. Since these 8 transformations indeed transform AAAA into BBBB, it is exactly 8.

It is very easy to generalize the Wagner-Fisher algorithm ([https://en.wikipedia.org/wiki/Wagner%E2%80%93Fischer_algorit...](https://en.wikipedia.org/wiki/Wagner%E2%80%93Fischer_algorithm)) to this "simpler" variant of edit distance, but this algorithm will still have a (worst-case) runtime of O(m*n), where m, n are the lengths of the two strings.

Of course, the question remains: Is there a (worst-case) faster algorithm?

<comment><USER6> Don't know, but you still end up aligning common subsequences (advanced monotonically in both seqs) and you still have the same order-of-computing- cells allowed in the standard m*n dynamic program. Your restricted problem is basically asking for the standard unix line-diff.

</comment></comment></comment></comment><comment><USER20> > Edit distance is useful, but also very time-consuming to calculate.

So I guess we will soon see a lot of crypto based on calculating edit distances?

<comment><USER21> No. Quadratic speed is way too fast for crypto - besides, asymmetric cryptography relies on problems that are difficult to solve but _candidate solutions are easy to validate_! This is what the complexity class NP is about. As far as I know there's no way to check a solution for edit distance any faster than it takes to compute it in the first place. Also, the new proof just shows that edit distance is quadratic _if P!=NP_.

<comment><USER22> I get what you mean but 'easy to validate' in the context of NP means 'can be validated in polynomial time'. It seems to me that you imply that edit distance is harder to validate than problems in NP. Edit distance itself is in NP since P is in NP.

<comment><USER21> I mean that there needs to be a fundamental difference between the difficulty of finding a solution and validating one. There's none in case of edit distance.

<comment><USER18>
      > I mean that there needs to be a fundamental difference between the
      > difficulty of finding a solution and validating one. There's none in
      > case of edit distance.

If you consider the decision problem "Is the edit distance <= d?", there is (according to this result) a fundamental difference - though only a polynomial one.

To see this, consider the following argument:

Clearly the edit distance between strings of length m, n is bounded by m+n (>= d). So if we found a way to transform the string s1 into s2 using d steps, we can construct a certificate of length O(m+n) (worst case) steps, which can be verified in linear time.

On the other hand (according to this result) you need Omega(m*n) (worst case) steps to find such a certificate.

</comment></comment></comment></comment></comment><comment><USER23> This is misleading twice over. First, as a3_nm notes, this proof depends on unproven conjectures, and so this isn't proven yet.

Secondly, it _is_ possible to do better than the Wagner-Fischer algorithm in certain cases. I figured this out myself and was about to write a paper on it when I realized that other researchers had beat me to it by a couple of decades.

If the two strings are equal you can establish that the edit distance is 0 in linear time, obviously. If you consider how the algorithm fills out the square down the diagonal you'll see that if there's only a one-letter difference you only actually need to fill out small parts of the grid around that difference.

This is generalizable, so the complexity actually depends on the distance between the strings.

<comment><USER24> In practice, yes, the computation time required to compute edit distance will depend on the actual data. Similarly, 'sorting' a list that is already sorted can be done in linear time, and near-sorted lists can be sorted much faster than a randomly ordered list.

This is certainly useful in practice, but it doesn't affect whether the worst- or average-case complexity is quadratic. I'd like to see a quadratic (or exponential etc) time problem that couldn't be solved faster in many cases.</comment></comment>