Every NFL play for the past 10 years in CSV format - <AUTHOR> http://www.advancednflstats.com/2010/04/play-by-play-data.html <comment><AUTHOR> From Line 42536 of the 2008 CSV file:

 _20090201_PIT@ARI,2,30,18,ARI,PIT,1,1,1,(:18) (Shotgun) K.Warner pass short middle intended for A.Boldin INTERCEPTED by J.Harrison at PIT 0. J.Harrison for 100 yards TOUCHDOWN. Super Bowl Record longest interception return yards. Penalty on ARZ-E.Brown Face Mask (15 Yards) declined. The Replay Assistant challenged the runner broke the plane ruling and the play was Upheld.,7,10,2008_

They forgot: for(i=0;i<92;i++){yell('edw519','GO!')}

Seriously, I had plans for the next 4 days, but I just scrapped them. Funny how jazzed I get when it's data that I can really relate to...

I've already structured my data warehouse and started the loads. (I'll probably need a whole day just to parse the text in Field 10.) Then I'm going to build a Business Intelligence system on top of it. I will finally have the proof I need that I, not the offensive coordinator, should be texting each play to Coach Tomlin.

See you guys on Monday.

EDIT: OK, I'm back, but not for long. I'm having way too much fun with this...

fleaflicker: Cool website & domain name. Thanks for the tips. I expect shortcomings in the data, but it looks like it's in a lot better shape than the usual free form enterprise quality/vendor/customer comments I usually have to parse. We'll see...

MattSayer & sjs382: I don't plan to do any analysis. I prefer to build an app that enables others to do their own analyses, answering questions that nobody else is asking. Like "Which Steeler makes the most tackles on opposing runs or more than 5 yards when it's 3rd down and longer than 8 yards to go, the temperature is below 38, and edw519 is twirling his Terrible Towel clockwise?"

jerf: Nice thought. I've spent years trying to earn enough money to buy the Pittsburgh Steelers just to fire the slackers and fumblers and win the Super Bowl every year. Maybe I should just take an easier route and solve that problem like any self-respecting hacker should: with data & logic. No Steeler game this weekend; I may have found my destiny </sarcasm>

<comment><USER2> You'll find that the text descriptions aren't consistently formatted. It's tough to extract structured data from all play descriptions.

For example, first initial plus last name does does not uniquely identify a player. You'll need accurate roster data first, and even then there are clashes.

We store play data by its structured components (players involved, play type, player roles, etc) and then derive the text description. This allows us to reassemble pbp data from different pro games to show a "feed" for your fantasy team.

Baseball has a smaller set of play outcomes/transitions so its easier to model this way. As your example from the Steelers Super Bowl shows, football plays can be very complex.

<comment><USER3> "It's tough to extract structured data from all play descriptions."

Which means you can treat it a bit like a text mining program. NASA had a text mining contest in 2007 as part of the SIAM conference on data mining which was really similar - instead of football plays it was textual descriptions of aeronautics incident reports and their classification. There were several papers that came out of that (I was with a group that did one of them, using an approximate nonnegative matrix classification approach - got beat out by some ensemble approaches).

Anyway - if you'd like to do something with unstructured football play descriptions, text mining might be able to empower you to some extent without going through a full manual analysis, and those papers could be a good starting point. I think some of them ended up in a volume titled _Survey of Text Mining II_.

<comment><USER4> Incredibly interested in your work here. For small-dimensional problems (or problems with features that can be engineered to be small-dimensional), ensemble methods through random forests and bagging and the like are incredibly useful.

But for high-dimensional text problems that're pure classification, I tend to rely simply on 1NN classifiers (against a single centroid of training data of a target category, of which there tend to be many). I've spent a lot of time with NMF, for its potential as an incredibly interesting data-exploration tool ("There's a pronoun cluster! There's a Spanish cluster! There's a 404 Error axis!") or low-dimension projection step. I've even spent a good amount of time on implementing the algorithm in a number of memory-efficient ways.

Could you expand a bit on how you used NMF for these problems in practice (similar to how a sparse autoencoder captures reduced-dimensional features en route to supervised learning), or how others used ensemble methods?

<comment><USER3> Afraid it's been a while, and I wasn't really at the core of the project design - if you're REALLY interested look up _Anomaly Detection Using Nonnegative Matrix Factorization_ and contact Michael W Berry (whom I assume still teaches at the University of Tennessee, Knoxville).

The main idea, though, is to generate a term-by-document matrix (count words, maybe throw out stopwords, normalize counts), then do Math to factor your matrix (approximately) into two: term-by-feature and feature-by-document. When you want to classify a new document, you can use its contents (more terms) to calculate a feature vector.

(The math seems to typically involve random initialization followed by iterative improvements. Other work in the field discusses the specifics.)

The matricies are "nonnegative" because, conceptually, features are a _positive_ thing, and you can't say that a certain term makes something _less_ a member of a feature cluster (only more).

The tricky part is figuring out how to map features to things which are semantically interesting to your application, and I don't want to comment too much on the state of that because it's been five years and I honestly forgot what exactly we did there, and it was all done in Matlab (which I'd never used before), and there's probably more recent work in the field. But if you fiddle with it manually, you can come up with your matrices and essentially have a nice little classifier.

</comment></comment></comment></comment></comment><comment><USER5> Looking through the 2002 season, there's an oddity around touchdowns and extra points. It seems that the 6 points for the touchdown are bundled with the extra point, and the score is not updated until the extra point is complete.

It seems this might result in bugs, as in the Oct 20, 2002 game between Dallas and Arizona. In the third quarter, with a score of Arizona 6 - Dallas 0, Dallas scored a touchdown (row 13900) but "aborted" the extra point (row 13901). The 6 points for the Cowboys are not recorded in the data.

The game eventually went to overtime, with the Cardinals kicking a winning field goal in OT for a final score of Arizona 9 - Dallas 6, but the data here records it as Arizona 6 - Dallas 0.

</comment><comment><USER6> There's a FAQ for this data that is on the site's main nav:

<http://www.advancednflstats.com/2007/02/contact.html>

Of particular interest:

 _Where did you get your data?_

Most of my team data comes from open online sources such as espn.com, nfl.com, myway.com, and yahoo.com. It's easy for anyone to grab whatever they're interested in from those sites.

 _My play-by-play data comes from a source that's not publicly available, and at this time I regret that I cannot share it._ However, I am working hard to develop a way to spread the wealth. One of my biggest goals is to help create a larger, more open, and more collaborative community for football research.

\----

There's no real terms of service so I'm curious as to the constraints in using this for commercial purposes. I most definitely want to use this for teaching purposes (how to text-mine, how to build a web app from data, etc) but want to know what terms the data can be redistributed.

<comment><USER7> IANAL, but it has been ruled that NFL player names and statistics are protected by the First Amendment, i.e. no one "owns" it and anyone is free to use it for any purpose.

[http://blogs.trb.com/sports/custom/business/blog/2009/04/cbs...](http://blogs.trb.com/sports/custom/business/blog/2009/04/cbssportscom_wins_fantasy_game.html)

However, you do have to get the data, and unauthorized access of computers (which constitutes trespassing) can be a legal gray area. I'd love to hear a lawyer weigh in on the legality of scraping the data directly from espn.com.

<comment><USER8> Last time I checked, the play by play data on espn.com was pretty error- ridden. This was three or so years ago, so it might have changed, and I was hypothetically interested in the score columns, so it may not matter depending on other hypothetical uses. But I'd hypothetically avoid scraping ESPN for that reason alone.

</comment></comment></comment><comment><USER9> This seems like as good a time as any to share something I've been working on which uses the same source data, even though it's pretty rough at the moment (slow, bad data, only currently goes through week 8 of 2012, etc.):

<http://nfl-query.herokuapp.com/>

The basic syntax is [stats] [conditions] : [row] / [column].

There's some autocompletion to try to make it possible to discover what is accepted.

Examples:

passing yards : team / season

first downs / first down attempts : down / distance

rushing yards min 100 rushing yards : player, game, quarter

rushing yards / carries min 200 carries : player

One of the biggest problems is that it's currently way too easy to shoot yourself in the foot by making a really slow query.

</comment><comment><USER10> I'm frankly surprised that this information is allowed to be distributed. I spent awhile in the financial services industry, and while it was really easy to obtain "public" information like stock quote data, I recall that we weren't allowed to simply scrape data from public sites... we had to pay a license fee to get a feed of the data if we were planning on repackaging & distributing it.

It seems to me that the NFL would want to have exclusive rights to distribute this data and charge people a fee for access to it. Clearly I'm no expert in these legal affairs though.

<comment><USER11> IANAL, but I asked one about this while ago; let's see if I can remember: It's complicated. The NFL broadcasts are copyrighted, and come with a statement that (among other things) distributing descriptions of the game is not allowed. That could be considered a derivative work.

On the other hand, a live performance is generally not protected from copyright, so if you attend a live game to collect the data, you may be in the clear.

The data isn't owned by the NFL, but all recordings of the games are, and so any data obtained by watching recordings of the games could potentially be controlled by the NFL.

<comment><USER12> It might not even violate the NFL's copyright if extracted from tapes. For one thing, something is only a "derived work" for copyright purposes if it's a "creative work" subject to copyright at all, and in the U.S., data sets comprising factual information aren't typically considered "creative". For another, it's not clear whether data about a recording is derived from the recording for copyright purposes. For example, a re-edit or mash-up of a film is clearly a derived work, but is a count of how many minutes each character speaks a derived work? Or is a Spotify-style algorithmic analysis of a song's musical style a derived work of the song?

I wouldn't want to put a large bet on where exactly those lines are drawn, though.

</comment></comment></comment><comment><USER13> Here is an idea: build a predictive model of an offensive coach that predicts the play he will call, given a game situation (and based on that, build a predictiveness quotient for a coach).

<comment><USER2> It doesn't work like that in practice. Football is very dependent on matchups. Coaches will vary gameplans from week-to-week to exploit weaknesses they see on film.

<comment><USER13> Matchup would be a part of the model. My experience with predictive modeling in various domains has taught me that people tend to underestimate how predictive they are (NFL offensive/defensive coaches are no exception).

<comment><USER14> I'm interested in doing some predictive modeling for a couple of project ideas I've been kicking around. Are there any specific resources you would recommend as good starter material?

</comment></comment></comment></comment><comment><USER15> How many of you are thinking right now: I'm going to generate an HTML page for every game and throw ads on it? Be Honest!

<comment><USER16> Is anyone going to try a 'moneyball' style Fix_Your_Fantasy_League_LineUp site with ads?

<comment><USER17> I might try to use the data to create mock drafts.

</comment></comment></comment><comment><USER18> The CSV file format is nice, but if you're looking for a Python API to play with NFL stats without having to parse play-data fields, check out nflgame [1]. I've written up a quick primer. [2] It also includes the ability to get play-by-play statistics live.

[1] - <https://github.com/BurntSushi/nflgame>

[2] - <http://blog.burntsushi.net/nfl-live-statistics-with-python>

</comment><comment><USER19> I've started uploading these CSVs to a public Google BigQuery dataset called [nfl], so you can run queries over them like this:

        SELECT off, COUNT(off) AS count
        FROM [nfl.2012reg]
        WHERE description CONTAINS "INTERCEPTED"
        GROUP BY off
        ORDER BY count DESC

(This counts the number of plays that resulted in an interception by the team that threw the interception, sorted from most to fewest INTs)

<comment><USER20> I'm new to BigQuery, how do I access a public dataset? I ran the query and got the error Not Found: Dataset 578707073226:nfl

<comment><USER19> Sorry, I didn't actually make it public it seems. Should work now.

<comment><USER20> thanks!

</comment></comment></comment></comment><comment><USER21> Here's some soccer data, doesn't include play-by-play though (soccer generally isn't suited to that kind of breakdown, although Opta Sports do track it).

<http://www.football-data.co.uk/downloadm.php>

Tons of European leagues, going back to 1993 in some cases.

Here's some sites that give detailed stats and match reports:

<http://www.eplindex.com/>

<http://www.whoscored.com/>

<http://www.soccerstats.com/>

<http://www.soccerway.com/>

<http://www.squawka.com/>

Man City use their petro-dollars to open up Opta Sports (detailed match stats) to all: <http://www.mcfc.co.uk/the-club/mcfc-analytics>

Someone needs to compile stats equivalent to these NFL ones for european football! Hmmmm...

</comment><comment><USER22> The comments on that are awesome too - great advice for parsing, categorizing, and such. I couldn't download 2010 though - "Sorry, we are unable to generate a view of the document at this time. Please try again later."

<comment><USER22> If you click the little down arrow (top left), it will download the file. Just a heads-up in case others see this message as well.

</comment></comment><comment><USER23> Amazing!!! Thanks to www.advancednflstats.com for doing all the leg-work. Highly recommend their site too. Their in-game win probability statistics are always a must-have for me on game-day ^_^

<comment><USER5> I really like his 4th Down analysis:

[http://www.advancednflstats.com/2009/09/4th-down-study- part-...](http://www.advancednflstats.com/2009/09/4th-down-study-part-1.html)

The tl;dr version can be found at:

<http://www.advancednflstats.com/2010/05/4th-down-briefs.html>

The conclusion is that teams should go for it on 4th down much more often than they currently do.

He also has a calculator where you can get the exact values:

<http://wp.advancednflstats.com/4thdncalc1.php>

<comment><USER24> The Patriots have used this analysis:

[http://www.math.toronto.edu/mpugh/Teaching/Sci199_03/Footbal...](http://www.math.toronto.edu/mpugh/Teaching/Sci199_03/Football_game_theory.htm)

I also believe Belichek/Adams have funded some football economics research.

</comment></comment></comment><comment><USER6> This looks like great fun...Judging by some of the sample entries, it will also be an instructive example of the limitations of CSV and why serious analysts who want to work with unstructured data need to know a scripting language, or at least regexes.

Sample description field: > _20020905_SF@NYG,1,59,20,NYG,SF,3,11,81,(14:20) (Shotgun) K.Collins pass intended for T.Barber INTERCEPTED by T.Parrish (M.Rumph) at NYG 29. T.Parrish to NYG 23 for 6 yards (T.Barber).,0,0,2002_

In the comments section of the OP, someone posted this sample Excel function:

        	=IF(ISNUMBER(SEARCH("right   tackle",J2)),"rush",IF(ISNUMBER(SEARCH("right
    	guard",J2)),"rush",IF(ISNUMBER(SEARCH("left
    	guard",J2)),"rush",IF(ISNUMBER(SEARCH("up                              the
    	middle",J2)),"rush",IF(ISNUMBER(SEARCH("left
    	tackle",J2)),"rush",IF(ISNUMBER(SEARCH("left
    	end",J2)),"rush",IF(ISNUMBER(SEARCH("right
    	end",J2)),"rush",IF(ISNUMBER(SEARCH("pass",J2)),"pass",IF(ISNUMBER(SEARCH("kneel",J2)),"kneel",IF(ISNUMBER(SEARCH("punt",J2)),"punt",IF(ISNUMBER(SEARCH("kicks",J2)),"kickoff",IF(ISNUMBER(SEARCH("extra
    	point",J2)),"extrapoint",IF(ISNUMBER(SEARCH("sacked",J2)),"sack",IF(ISNUMBER(SEARCH("PENALTY",J2)),"penalty",IF(ISNUMBER(SEARCH("field
    	goal",J2)),"fieldgoal",IF(ISNUMBER(SEARCH("FUMBLES",J2)),"fumble",IF(ISNUMBER(SEARCH("spiked",J2)),"spike",IF(ISNUMBER(SEARCH("scrambles",J2)),"rush","rush"))))))))))))))))))

Dear god, at what point do people finally realize that it's worth learning some simple scripting to work with text files?

<comment><USER25> The Excel function looks ridiculous, but it probably didn't take more than 10 minutes to make, tops. Nested conditionals are easy.

At any rate, what would you recommend most to accomplish the task? I'm learning Python and know R a bit, so I was just wondering how I was going to about combing through the data.

<comment><USER6> Python or Ruby is fine...the main trick is to be able to process those fields with regular expressions...which, IIRC, requires throwing in VBscript if you were to handle it solely in Excel.

Python and Ruby would also allow for more elegant-looking -- i.e. more maintainable -- functions to handle that field.

<comment><USER8> R is fine too: it has regular expressions and probably excels if you plan to do statistics using all of that data. Python seems to have reasonable statistics functionality as well (with pandas, etc) but I haven't used it personally.

<comment><USER25> Thanks for the note. I've not really looked into how one would do that with R (doing it in Python seems more clear), but am checking it out now. If anyone else is looking I'm finding this PDF helpful: [http://biostat.mc.vanderbilt.edu/wiki/pub/Main/SvetlanaEdenR...](http://biostat.mc.vanderbilt.edu/wiki/pub/Main/SvetlanaEdenRFiles/regExprTalk.pdf)

</comment></comment></comment></comment></comment><comment><USER26> God this is such interesting stuff. How do we still not have a fully featured open source NFL stats-rosters-game charting API? Who wouldn't want to contribute to that project?

Other than cool data visualization stuff, the obvious implication is the potential to devise a profitable system to pick games against the spread. The guys at Football Outsiders have done a decent job at it and made a proprietary algorithm that picked games at 58% this year ( which is over the threshold you need to be profitable in Vegas ). But even those guys are still having some trouble getting access to and aggregating the data in a usable format.

I really want to sit down and start playing around with some of this data so I appreciate you putting this together for everyone. The NFL needs an open source API and this is definitely a step in the right direction.

<comment><USER18> I believe my library, nflgame [1], would fit the bill. Features all play-by- play data back to 2009, and includes the ability to track play-by-play data live.

[1] - <https://github.com/BurntSushi/nflgame>

</comment></comment><comment><USER27> For my ML class, I used this dataset to play around with predicting plays. Im obviously no expert, but wanted to share my report anyways:

<https://www.dropbox.com/s/cy04oxaq83mxvoz/report.pdf>

</comment><comment><USER28> A bit OT, but I thought this might be a good opportunity to mention the upcoming SportsHackDay in Seattle from Feb 1-3 which culminates in a group viewing of the SuperBowl. <http://sportshackday.com/>

</comment><comment><USER29> I wrote a small wrapper around the 4th down calculator on that site, which should help you figure out if your team should go for it on 4th down:

<http://downanddistance.herokuapp.com/>

<comment><USER30> Need a bounds check or two in there. Tried setting the number of yards you need to 1 and the yards away from the endzone to 99 and it threw up a nice exception. Cool calculator though!

<comment><USER31> As a Madden (game) aficionado my first thought was "ALWAYS go for it on 4th!" but then again I play super recklessly...

</comment></comment></comment><comment><USER32> I've used data from Brian Burke's site before. I think it's the exact PBP data the NFL has, but you'll find that the structure and common phrasings change over the years. I had to write a lot of regular expressions and I was still catching edge cases for weeks.

btw, pro-football-reference has pbp data now too, and it probably goes back a lot further, but I think they discourage mass scraping of their site.

</comment><comment><USER33> There is a lot to have fun with here. I would imagine though that in a lot of NFL coaching rooms there has to be a balance between coaching and analysis.

Like someone else said, it's about match-ups.

Semi-related : [http://profootballtalk.nbcsports.com/2013/01/03/polian- think...](http://profootballtalk.nbcsports.com/2013/01/03/polian-thinks- moneyball-wont-work-in-nfl/)

</comment><comment><USER34> As a French not interested in sports at all this would have made no sense at all to me before I watched the TV series The League [1]. Now I kind of enjoy the fact that these stats exist and are available in an open format, even if I don't really care myself.

[1] <http://www.imdb.com/title/tt1480684/>

</comment><comment><USER35> There's an interface into this sort of play-by-play data (since 2000) at <http://pro-football-reference.com/play-index/play_finder.cgi> \-- lets you do queries on down/distance/position on field/score differential, all that sort of stuff.

</comment><comment><USER36> This is slightly off topic but does anyone know of a resource to get the odds at gametime historically for NFL games?

<comment><USER37> [https://www.dropbox.com/s/ikczgv737lllh0a/nfl_spreads_1985-2...](https://www.dropbox.com/s/ikczgv737lllh0a/nfl_spreads_1985-2010.csv)

<comment><USER36> thanks!

</comment></comment></comment><comment><USER38> If you live in the vicinity of Seattle, there is a sports-themed hackathon going on Superbowl weekend. Google, ESPN and a bunch of tech companies are sponsoring. The grand prize will be passes to the Sloan Sports Conference. More details to come:

<http://sportshackday.com>

</comment><comment><USER39> You should send this to the Buffalo Bill's new analytics department. Maybe a Hacker could get hired to an NFL team.

[http://www.nfl.com/news/story/0ap1000000121582/article/bill-...](http://www.nfl.com/news/story/0ap1000000121582/article/bill- polian-moneyball-does-not-work-in-the-nfl)

</comment><comment><USER40> It would be interesting to take this data and build an app around it for fantasy football. If you have all the tendencies, and how players like your player have played against certain teams, you could make better guesses on who to play.

<comment><USER41> I did something like that for an AI class in college. We used FuzzyCLIPS to write an expert system for drafting fantasy football teams. A ruby script pulled the CSV data from some other site that had the previous three years of NFL data, and then converted the CSV to fact files which the system then read in.

When all was said and done it worked, but made some pretty crappy draft picks! I should find that code....

</comment></comment><comment><USER42> FYI, for baseball fans, you can get similar data about each play in the MLB from <http://www.retrosheet.org>

</comment><comment><USER43> Shame it does not have player ids. I wonder how many players with the same surname and the same first initial play for the same team.

</comment><comment><USER44> Now all we need is some historical line & associated movement data...

</comment><comment><USER45> What ... no NHL? Oh ya.

</comment><comment><USER46> Would be great if there were columns:

Head Coach \- Offensive Coordinator \- Defensive Coordinator \- Formation \- Play

</comment><comment><USER47> wow

</comment><comment><USER48> Wow. Simply. Wow.</comment>


Neopets HTML Guide - <AUTHOR> http://www.neopets.com/help/html1.phtml <comment><USER2> Every once in a while I run into another programmer who started on Neopets, and it makes me happy. :) When I interviewed for my job, they asked me my "programming story." I remember struggling to make my petpages compatible with Firefox, Safari, and IE5. I remember learning Dreamweaver so I could apply to adopt cool Neopets. I wrote tens of thousands of lines of HTML in TextEdit to make an interactive Lite Brite, and more recently I built Neopets-compatible Minesweeper. ([http://www.neopets.com/~wire](http://www.neopets.com/~wire)) These are the projects that made me give CS a shot even though I had no idea what it would be like and even though I was nervous to enter a field with a bad reputation for its attitude towards women. I'm so glad I did.

So many people learned the joy of coding from that site. Sadly, it can't be that for the next generation because its code filters are almost as obsolete as this guide. HTML5, CSS3, multiple classes on an element, and the word "position" in a stylesheet aren't supported. TNT, if you're reading this, please take some time to invest in the part of your site where people learn and create.

<comment><USER3> To this day I will still cite how Neopets' pet pages were my introduction to the creator's web; later MySpace would help me accumulate some spare lunch money. The good ol' days. =)

</comment></comment><comment><USER4> Neopets is the reason I'm a software engineer. Along the way looking for "cheat codes" for Neopets, I discovered some people hacking the games. I eventually ran a forum on hacking Neopets, made money from ads, got into PC World Magazine, and had a great time along the way.

Sorry to Adam and Donna for any headaches I caused during my teenage years. <3

<comment><USER5> Neopets was where I first discovered hacking.

There was a user named "blueyoshi82" or something, I wanted to see if I could get into their account (I don't remember why). So I tried guessing their password, and lo and behold, it was just "yoshi"!

Being the white-hat that I was, I tried to PM them a message saying "hey you really need to change your password, it's too easy to guess", but the word "password" was blacklisted so I couldn't tell them.

<comment><USER6> I remember during school showing my Math teacher (who was a big Ultima gamer but also played Neopets) how I set up a fake neopets shop that prompted you for login. <form action="mailto:xxxxxx"> When you submitted the login form it sent the form inputs in an email to a hotmail address :D. One of my first experiences with HTML. I had no idea there were so many shared Neopets experiences in this community.

</comment></comment></comment><comment><USER7> Low key, this guide is how I got into programming and probably plays a big role in why I'm now a Software Engineer. Throwback.

<comment> 52-6F-62 I never saw the Neopets guide. For me, it was Joe Burns' site: HTMLGoodies[0]. It was indispensable for me. That and a lot of help from FrontPage Express[1] & viewing the source code of whatever I'd just made.

[0] [https://web.archive.org/web/19990429010831/http://www.htmlgo...](https://web.archive.org/web/19990429010831/http://www.htmlgoodies.com:80/)

[1] [http://www.jegsworks.com/lessons/web/html/fpx/fpx.htm](http://www.jegsworks.com/lessons/web/html/fpx/fpx.htm)

<comment><USER8> HTMLGoodies helped me make terrible Geocities sites.

<comment> 52-6F-62 You and me both

</comment></comment></comment></comment><comment><USER9> First time I saw a program written was when I asked my dad to help me get #1 on one of their mini games. He helped me by making a java swing app (at least I think that is what it was).. which I could put letters into to make words with. Made it to top 10 for in about a day & then got banned for 'acting suspiciously' :D. Definitely a throwback for me too.

</comment><comment><USER10> And for when you wanted to dive a bit deeper and make your shop really pop, there was the Funky Chickens HTML Help Page ([http://www.funkychickens.com/main.asp](http://www.funkychickens.com/main.asp))

<comment><USER11> Also, [http://www.lissaexplains.com/](http://www.lissaexplains.com/) was really big at the time. Taught me everything I needed to know to code up petpages and the like.

</comment></comment><comment><USER12> This is a major throwback, and definitely one of the main reasons why I'm where I'm at today.

</comment><comment><USER13> This got me into programming as well! The biggest problem I had was when I wanted to expand and build a forum, and tried to understand why Geocities didn't run my .php file (which was from a tutorial). Answer: PHP support was paid option AFAIK

<comment><USER14> Yep, back then it was very hard to get any free hosting that supported any sort of scripting language. Of course, as a 12 year old on the internet, paying for anything was unheard of.

I remember wanting to set up some forum that required CGI. I had some sort of hosting account that required using their own dialup service to connect to their FTP. Probably cost more on the phone bill than some cheap hosting service, but it was easier to get that by the parents than asking to use their credit card when the internet was still a mystery to them.

<comment><USER15> Tripod offered Perl CGI scripting on their free service at one point.

I wrote a terrible "messageboard" that only one other person ever used.

</comment></comment></comment><comment><USER16> The bandwidth at which they produced new flash games was amazing. They were pretty detailed too. I can't think of any html5 gaming community website that even comes remotely close to them.

One of my favorite events with Neopets was participating in a global takedown of enemies during war. I spent so many hours with that website.. from tending my shop, training my pets and tricking out their pages w/ HTML, to collecting map pieces (I remember the economics w/ Secret Laboratory Map Pieces exploding when people realized how valuable they were)..

</comment><comment><USER17> I too used this, and more generally credit Neopets as a source of a lot motivations to learn by solving interesting problems. I reproduced a couple of the site's games in my first couple high school programming classes, and then eventually had a minimal logic but very reliable "Battledome" fighting script using autohotkey partway through undergrad. That autohotkey experience was helpful senior year in very quickly automating data collection from a non- automatable program for estimating atmospheric radiation dispersal.

</comment><comment><USER18> This is so nostalgic... I remember spending several days of my youth playing and, er, "boosting" my scores to earn more neopoints. Usually modifying the POST request with something like TamperData did the trick :)

<comment><USER19> I actually went beyond that. There was a fun game that involved overlaying tetris-like shapes on a board to cycle images through a set. It started out easy, but soon got rather hard. I wrote a program to help solve them easier, and did quite well at first. Eventually, I think other people did the same, but better. That, or they were really, really savants at that kind of problem.

</comment></comment><comment><USER20> I know a handful of developers started to code thanks to Neopets. Personally it was bbcode that got me interested in what was going on behind the scenes.

</comment><comment><USER21> > If the user viewing your page is using Internet Explorer, your music can > play in the background when the page loads.

This brings back memories ...

</comment><comment><USER22> I got introduced to HTML through Neopets as well. I remember joining a Legend of Zelda guild. Another member and I decided to start our own guild. He showed me how to use an image tag so we hotlinked a bunch of sweet images and created a bunch of new accounts to boost our member count and that helped us organically get members. Good times!

</comment><comment><USER23> Chiming in as another person who got into web development thanks to Neopets.

</comment><comment><USER24> Ah neopets.... your character limits forced me to ditch frontpage and actually learn how to make html with notepad.

</comment><comment><USER25> Now you too can make web pages straight out of the 90's!</comment>


Anonymouse - <AUTHOR> http://blog.rapleaf.com/dev/2010/07/20/anonymouse/

<comment><USER2> I'm a bit confused about how this article fits in with the previously posted article about the company (<http://news.ycombinator.com/item?id=1817631>). There, they are harvesting PII data (e-mail) and re-selling access to related information. In this article, they explain a machine-learning-type algorithm to anonymize data to contain no PII. I don't really see how the two fit together. What's their deal?

<comment><USER3> Personally Identifying Information (PII) is defined as data which can uniquely identify an individual--name, social security number, facebook ID. Rapleaf's personalization service--which is designed to let websites personalize based on who is viewing the website--does not serve PII. Instead, it serves targeting data like "Age 40-50" and "Interests Basketball".

The idea behind the Anonymouse project is that a person should not be able to be personally identified based on the targeting information served about them. Only sets of targeting data which cannot be traced back to a unique individual is stored in the personalization cookie.

For example, it would be okay to serve a targeting cookie about a person which contained "Male, Wealthy, looking to buy a Ferrari", because thousands of people fit that description, and the ad network or website cannot identify the person with any reasonable specificity.

It would not be okay, however, to target a person as "52 year old Male, makes $256,000, lives in Sometown OH, and was born on April 12th", because in all likelihood, only one or two people fit that description, and the data would effectively serve as PII, and this would be no better than dropping a facebook ID in the cookie.

Let me know if anything's unclear, or if you want more details. From a CS perspective, it's a really cool/hard problem, and something we've spent a lot of time on. We're planning on writing an update blog post on where we are... as soon as we're done coding : )

<comment><USER4> I still manage to marvel every time it seems Hacker News is community to the creators of pretty much everything.

<comment><USER5> ... that gets extensive coverage on Hacker News.

</comment></comment></comment></comment><comment><USER6> This is pretty fascinating. I was expecting another cover-your-ass blog post, but this is a pile of interesting data work which is new to me.

</comment><comment><USER7> A central concept in the article is k-anonymity, which I only learned about maybe a year or two ago. Researching, it appears though that it has been introduced in a conference article in 1998 and has been published in 2002 ([http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.163...](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.163.9182&rep=rep1&type=pdf)), which I found rather surprising. It's a shame that the knowledge of this concept seems to be mostly restricted to CS circles. Maybe the rather steep technical barrier (research seems to be published mostly in technical context - non-scientific metric: half of the url in the resuls of googling for 'k-anonymity' has 'cs' and 'edu' in the domain name; and no literature on anonymizing datasets for people with only basic schooling in statistics and database theory seems to be available) is preventing the uptake of better database privacy protection.

Of course there's also the issue that incentives in collecting and storing sufficiently anonymous data are not aligned (and sometimes in direct competition) with the goals for which the data is collected. This company seems to be making great strides in this field with this research, I hope they keep pushing the edge and publishing their results.

</comment><comment><USER8> Technically, this is really interesting, but what's the business justification for the engineering work?

I'm skeptical that these efforts will protect Rapleaf or any other company from public relations disasters or class-action lawsuits or harmful regulation, because the solution is too complex for reporters, lawyers, and politicians to understand. (The cynic in me thinks that even if they _did_ understand it, that's not going to get in the way of a good story / lawsuit / feel-good cause for the public.)

While I'm sympathetic, declaring that your dataset is 16-anonymous due to cluster-based suppression isn't going to persuade anyone that's already decided that behavioral targeting is the devil.

</comment><comment><USER9> I assume that this was just a feel-good article written in response to their name appearing in WSJ recently. The project itself has been around a few months, at least publicly. The engineering described isn't particularly novel, but it is a high-level overview of a large-scale project.

<comment><USER3> We wrote this blog post back at the end of July about the project, not in response to the WSJ article. There have been a few posts since then discussing other aspects of the project, and there will be an update coming quite soon.</comment></comment>


$3.2B to Fix Tampa's Aging Pipes? From Where? - <AUTHOR> https://www.strongtowns.org/journal/2019/8/30/32-billion-to-fix-tampas-aging-pipes-from-where <comment><USER2> Well, had the city budgeted prpoerly andpaid lower wages to politiciand and workers, but still the local average wages, not minimums, and rational politiciand and worker pensions they would be able to deal with their capital needs, but the politiciand over pain themselves and the workers - because it was not their money - remember that - not their money.</comment>


Machine learning and data science advices - <AUTHOR>

I am writing to ask for career advices, I am 27 years old with a master degree in electrical engineering. I was working as a ruby developer for almost 5 years and 1 year (during my master thesis) with C++ developing a project related to Computer Vision using Image processing and Machine learning.
During master thesis year I realize that I like to work in Machine learning and Data science areas but I can't work on companies that are looking for people in these areas because I am junior. When they look to my resume, they always sort me as ruby developer.
I am reading and following the OSDSM (http://datasciencemasters.org/) but I know it's not enough to get a position in related areas. Can you give me some advices? Do you know some companies that may be interested on internship/junior positions? <comment><USER2> Hi, I was in a similar situation to you last year. I was working as backend developer and took the online course on Machine Learning on Coursera and realised that I want to work on Machine Learning in the future.

One of the myths is that you can learn to use toolkits and programming languages (R, Python) and become eligible for Machine Learning jobs (I certainly couldn't). It's only when you begin to understand the underlying maths behind the algorithms, you can be successful in interviews.

I would say getting another degree is the best way to go about it since it is very much an academic field. However, if that is not an option, I'd recommend looking at some online courses such as:

\- [https://www.coursera.org/specialization/jhudatascience/1](https://www.coursera.org/specialization/jhudatascience/1) \- [https://www.udacity.com/course/nd002](https://www.udacity.com/course/nd002)

In addition, I would supplement the courses with a good Machine Learning textbook such as [Pattern Recognition and Machine Learning by Bishop].

Also, see [https://news.ycombinator.com/item?id=1055042](https://news.ycombinator.com/item?id=1055042)

</comment><comment><USER3> One option is to go get an additional MA in Statistics and become really proficient using R. Continue to work on C++ in Machine Learning projects and take a course or two specifically in Machine Learning either from the Statistics department or if they don't offer those courses pick them up as electives from the CS department.

Obviously the other is to continue self-study while working FT using your Ruby development skills and EE credentials.

If it were me, I would just bite the bullet and pick up the second MA degree while supporting myself with freelance Ruby development projects.

<comment><USER4> Good point, work as a freelance on his current path to pay a MA degree seems a good idea. Can you recommend a MA? Any of them with online options?

I am on similar situation and I live in Europe. I didn't saw a lot of MA in related areas.

<comment><USER2> See

-[http://www.csml.ucl.ac.uk/courses/](http://www.csml.ucl.ac.uk/courses/)

-[http://www3.imperial.ac.uk/computing/admissions/pg/specialis...](http://www3.imperial.ac.uk/computing/admissions/pg/specialism_ml)

-[http://www.cl.cam.ac.uk/admissions/acs/](http://www.cl.cam.ac.uk/admissions/acs/)

-[http://www.cs.ox.ac.uk/admissions/grad/MSc_in_Computer_Scien...](http://www.cs.ox.ac.uk/admissions/grad/MSc_in_Computer_Science)

</comment></comment></comment><comment><USER5> My suggestion is just to pivot your ruby skills (I assume are web dev) into Python. Make something awesome in Flask - there aren't any real good data science tutorials in Flask which sucks but it seems to be a tool that a lot of data scientists use.

<comment><USER4> I don't see your point. I don't think so that it's an programming language problem. What can he do if he starts to use Python and a web framework using the new language?

<comment><USER5> Help build and integrate data tools - this could be done in Ruby but I'd imagine that working with data people using the same language could be helpful.</comment></comment></comment>


Varnish 6.0 denial of service - <AUTHOR> https://varnish-cache.org/security/VSV00003.html <comment><USER2> Strange. Downloading varnish-6.2.1.tgz with Safari 12.1.2 resulted in a 3,203,620 byte file that failed the shasum check on the three Macs I tried. The file does not appear to be a valid tar.gz file. Downloading with curl and Chrome on the two Macs resulted in 3,207,791 byte files that passed the shasum check. Changing the User Agent in Safari did not fix the problem. Open "safe" files is disabled.</comment>


Factor's implementation of polymorphic inline caching - <AUTHOR> http://factor-language.blogspot.com/2009/05/factors-implementation-of-polymorphic.html <comment><USER2> If my personal opinion counts for anything: I think Slava is a first-rate compiler hacker working on a pet language. Somebody should snag him ASAP and put him in charge of something more useful; dunno, a tight compiler for an embedded ARM platform for mobiles or something.

The world didn't need "the best forth-dialect" which he created, and still doesn't. It just needs another standards compliant C compiler. Nvidia or Intel should put an end to his fun-mongering and suit that boy up for a nice Lead FOO Architect position :-)

<comment><USER3> Couldn't disagree more. The world already has enough von-Neumann-style languages and people working on them; we need to explore different ways.

<comment><USER4> "The world already has enough von-Neumann-style languages and people working on them"

I believe mahmud was being ironic.

<comment><USER5> Slava's very smart, but I think that the basic idea of a 'stack based language' is simply not a great one, because keeping track of the stack in your head is a pain in the ass. So I'd say that mahmud is right on. Worth learning for fun, but it just doesn't strike me as practical.

<comment><USER3> Good Factor is more about combinators than stack juggling.

</comment></comment></comment></comment></comment><comment><USER6> I dunno, I kind of like the energy that he gives off, and stack-based languages are a nice complement to high-level languages such as lisp.

I would certainly find myself using factor in very small footprint systems rather than C or asm.

Seems like this is useful innovation, and there is not much innovation happening in the C language world, relatively speaking.

<comment><USER7> You're thinking of Forth, not Factor- Mr. Pestov seems to want to compete more with high level languages like Python than with C. Factor has garbage collection, and compiles to images [ a la smalltalk ], not really attributes that scream 'embedded' to me.

<comment><USER8> I like how you put [a la smalltalk] in square brackets, rather than in parentheses (a la lisp).</comment></comment></comment>


100 FinTech Problems to Solve for Singapore - <AUTHOR> http://fintechnews.sg/3268/fintech/mas-100-fintech-problems-to-solve <comment><USER2> Great post! Enough substance here to fund the search for startup ideas</comment>


Ask HN: College student looking for freelancing advice - <AUTHOR> Thanks for reading! I'm looking for advice about how to get started freelancing. I'm a master's CS  student in college looking to make some extra money, and I'm not sure where or how to look for freelancing work.  I have three years of experience with backend python web development, using several frameworks, and would like to find companies looking for that particular skill.
Does anyone know of a way to find such opportunities?  There are so many freelancing websites out there (Gigster, Toptal, Freelancer), and it's so hard to tell which to try, and which I would have a shot at getting work from.
Thanks!! <comment><USER2> I started freelancing full-time 8 months ago. What I found really supportive on my journey was the Creative Class from Paul Jarvis ([https://creativeclass.io](https://creativeclass.io)).

Remember that you have something valuable to bring to the table, namely yourself and your unique experiences. I would not recommend freelancing websites for one simple reason: race to the bottom. On those websites you will often be competing with super low rates.

I think one of the most effective "techniques" for finding the opportunities you are looking for is to connect with people. Spend time where other people with work opportunities are spending their time, talk to people and help others.

All the best!

</comment><comment><USER3> Previous comment on this subject: [https://news.ycombinator.com/item?id=12744624](https://news.ycombinator.com/item?id=12744624)

In general, avoid freelancing websites. Networking is key. Go to industry and tech meetups. Talk to other freelancers.

If possible find a more specific niche. Freelancers offering generic web development skills are ten a penny. Specialising in problems and industries is preferable to specialising in technologies. If you're the "Python guy" you're essentially positioning yourself as a fungible commodity. If you can for instance be the guy who uses software to solve specific problems in supply chain management that'd be a much more lucrative and sustainable niche.

</comment><comment><USER4> I'd say: network network network. Meet people that work at companies that might want freelancers.

Or look on remote job boards and email those companies directly. If you're just beginning, be willing to work for a bit less to get your foot in the door.

</comment><comment><USER5> I get a lot of leads going to local programming meetups. I would recommend staying away from freelancer sites, though. The rate will be low and the quality of client will be awful.

<comment><USER6> Seconded. I've hired a freelance through the university network (though, if others are like me, don't expect stellar rates :) ). You might want to ask at a local accelerator or meetups, plenty of people need a helping hand.

</comment></comment><comment><USER7> Hi Chris,

Picking up freelance work will be more difficult (but certainly not impossible!) if you're a student without a lot of professional experience. Most of the freelancing websites are a bloodbath race to the bottom where everyone wants work done as cheaply as possible. Toptal is a bit different, as the barrier to entry is a lot higher, so you might find it more useful than sites like Freelancer and Upwork.

You might also find the freelancer "Who's Hiring" posts here on HN useful. They pop up on the first day of every month, and you can find them archived at [http://hnhiring.me/](http://hnhiring.me/). Posting that you're available on there could be a good way to pick up some work.

Having a good portfolio page of work you've done can help tremendously. I've always liked Michael Fogleman's projects page: [https://www.michaelfogleman.com/projects/](https://www.michaelfogleman.com/projects/)

To stand out from the crowd, you don't need nearly as many projects as he has. 3-5 would probably be enough. They don't have to be huge! I've always liked downloading the GTFS schedules for a local transit agency and creating a site that makes it easy to look up schedules and stop times. It usually isn't difficult to make one that looks better and is easier to use than the actual transit agency site!

You could also do something interesting and unusual like implementing a cat detector using OpenCV and Python: [http://www.pyimagesearch.com/2016/06/20/detecting-cats-in- im...](http://www.pyimagesearch.com/2016/06/20/detecting-cats-in-images-with- opencv/)

If you learned how to do that from the blog post, reimplemented it yourself, and then put some of your own customizations in it, you'd have something good to add to your projects page. Someone who had a project like that they could show me and talk about would stand out (in a good way) from 100% of the developers I've interviewed in my career. So maybe try getting a few projects up and then post on the next HN 'Who's hiring freelancers' post. There are no guarantees, but plenty of people have found good freelance gigs that way.

If there are any local developer meetups, those can be a great source of work, too. My first ever paid programming work happened because I went to a meetup and showed off a transit app I had created. I didn't have any professional experience at the time, but showing that I could actually execute and ship something convinced them to give me a chance. So having a projects page can help you here too. Other devs that go to meetup often work for companies that sometimes have extra work they need done. If they meet you and like you, _and_ can see that you're able to take on projects and complete them, you'll be at the top of the list of people they'll contact to do work for them.

Hopefully that is helpful!

</comment><comment><USER8> If your school has a mailing list where people send out jobs, subscribe to that and start replying. I had much more success with that than freelancing sites, in fact the only other method that really worked was Craigslist.</comment>


Show HN: Web app to send email notifications when a Dropbox file has changed - <AUTHOR> http://boxification.com

<comment><AUTHOR> Hey HN, a friend (current business partner as well) of mine built this. I thought it was cool, so I decided to pass it along.

The main use case is when you're away from your computer or don't see the desktop notifications, you can still know if a file was modified. Ideal for small collaboration of files on shared folders.

</comment><comment><USER2> I like this, but whats the point of individually selecting files?

<comment><USER3> I built it mostly because my biz partner and I wanted to know when a file changed in a shared folder. We use markdown a lot for meeting minutes, discussions, etc...and sometimes i missed (or just didn't pay attention) to growl on my mac.

This way I had a persisted copy so it reminded to check out the changes.

Dropbox only has a RSS feed for activity that I just wouldn't pay attention to either.

Any feedback is much appreciated. Just a small side project.</comment></comment>


Ask HN: How to debug webpage crash on ipad? - <AUTHOR>

I have a search page that search for deals online. There are various filter options and a result view pane where results are shown. I am using backbone.js . Any change in filters remove all the previous results and load new results, that is removing old views and adding new views. I have ensure that there is no memory leak still page is getting crashed. How can i debug it ? <comment><USER2> Not sure if it fits for your problem but I've been using weinre to debug on mobile devices: [https://people.apache.org/~pmuellr/weinre/docs/latest/](https://people.apache.org/~pmuellr/weinre/docs/latest/)

</comment><comment><USER3> This is from Chrome but the same goes for Safari: [http://www.chromium.org/developers/how-tos/retrieving- crash-...](http://www.chromium.org/developers/how-tos/retrieving-crash- reports-on-ios)</comment>


Macedonian E-Health – Live stats dashboard - <AUTHOR> http://livedashboard.zdravstvo.gov.mk/ <comment><USER2> It returns 502 bad gateway. What should this be?

<comment><AUTHOR> It's fixed now. Apologies.

<comment><USER2> Yep, I see. I was asking because Serbian Ministry of Health just signed contract few days ago with as it appears to be the same company that did this sw in Macedonia.</comment></comment></comment>


787 Dreamliner teaches Boeing costly lesson on outsourcing - <AUTHOR> http://www.latimes.com/business/la-fi-hiltzik-20110215,0,442445.column

<comment><USER3> _Sure, it's immoral to abandon your loyal American workers in search of cheap labor overseas._

I know a lot of people who talk like this. However when it comes to spend their own money, they always go for whatever's cheapest, whether it's clothes, electronics, whatever. You can literally see these people ranting about worker's rights while wearing Primark and talking on a Huawei OEM'd handset.

Me, I'm a capitalist, and proud of it, but y'know, I'm the one willing to spend a little extra for a UK PLC product. Bullshit talks, money walks.

<comment><USER4> It wasn't the foreign suppliers that were the problem - Mitsubishi delivered the wings on time. It's was the excessive outsourcing to layers and layers of subcontractors without enough oversight. So a one mall plating shop in Albuquerque goes out of business and a critical path component is missing.

This is fairly common in defense, highly specialized components are subbed out to small shops - there are lots around LA - the problem here was that there was so many layers of sub-sub-sub contractors all being bidded out to the cheapest supplier nobody had any idea who was making what. That works for Walmart - it doesn't work for Boeing.

How they thought they were going to ensure40years supply of spare parts is anybodies guess

<comment><USER5> Just to elaborate on your point, the outsourcing model works for Walmart because Walmart is selling numerous separate products, rather than an integrated whole. It seems that Boeing forgot that lesson and started thinking of their airplane as a "collection of parts flying in close formation".

<comment><USER4> Also very few of Walmart's products are critical path - if they run out of lemon scented cleanup squares the store still opens (gratuitous hitch hikers reference)

</comment></comment></comment></comment><comment><USER6> In Boeing's case wasn't there a secondary goal in outsourcing of drumming up foreign demand? I seem to recall reading an article a few years back where Boeing agreed to outsource the wing sub assembly to some Asian country and in return that country agreed to buy some number of Boeing planes. I can't seem to find the article now though.

<comment><USER7> This is part of it. Its also part of the reason why Airbus/EADS quite successfully builds bits of aircraft all over Europe (and even proposed expansions of operations in Alabama as part of their bid for USAF tankers)

Also, it may surprise the article author to learn that there are outside specialists who have capabilities that Boeing couldn't effortlessly replicate in Seattle. Italy and Sweden certainly don't have low labour costs and they don't have much of a domestic market for long range airliners either.

Ultimately Boeing's logistical cockups have more to do with the overall ambition and complexity of the project than cost cutting.

<comment><USER4> Airbus's distributed manufacturing is more about pork-barrel politics, it's spread around the countries that own shares in the airbus consortium

Ironically Several of the components of the 7e7 are made in europe by Airbus suppliers. The undercarriage, tail and bits of the fuselage are made in the same plants that are building them for airbus.

AFAIK the only certified engines at launch are Rolls-Royce

</comment></comment></comment><comment><USER8> The title should really be 787 Dreamliner teaches Boeing a costly lesson in due-diligence. This sort of thing has nothing to do with outsourcing and everything to do with poor decision making by management.

<comment><USER9> Here's the problem, though: Who is capable of doing the due diligence? By outsourcing, you end up losing a lot of the in-house knowledge necessary to spot problems early on.

<comment><USER10>
      Who is capable of doing the due diligence?

Even if they're everybodies favorite wipping boy right now. I'd enter Nokia as the winner in the due dilligence category.

There's a classic business case study about two handset manufacturers, a Philips fab in Mexico and a small fire in that fab. Details can be found here :

<http://www.ftpress.com/articles/article.aspx?p=1244469>

<comment><USER11> In light of Nokia's latest struggle with iOS and Android, aren't they both a success story and a cautionary tale for due diligence and proactive management. Nokia had a second fire-- a burning platform.

By the logic of the book excerpt that I finally had time to read, they should have read the tea leaves about the iOS and Android universes much quicker than they did. Nokia's burning platform memo would not have been so dire if their response had occurred one year earlier. Nokia was already considerably left behind by the application universe at that point. iOS had momentum then and should have been considered a direct threat even though the Android userbase was only 1/10th of the size it is now.

I still find that corporate cultures tend to sideline Android to this day because they underestimate the value of an open platform on commodity hardware with infinite hackability that intrigues developers. The same incredible growth occurred with Linux and Apache on the web as the 888% Android growth in the last year. Android will win partly for the same reason Microsoft beat Apple's superior UI in the 80s and 90s-- Apple is still tied to expensive proprietary hardware and the need to completely control their platform.

</comment></comment></comment></comment><comment><USER12> This makes it sound like US companies were better than foreign countries just because they are american.

Me, I feel slightly insulted by that. Just look at american cars. I kid.

There is nothing wrong with patriotism and there is nothing wrong with outsourcing, if both is done with due diligence and proper planning. But there is no way one company is any better than any other company just because it is registered in a different country. The 787 did not have problems because of outsourcing, it has problems because of bad planning--at the all-american Boing HQ itself.

<comment><USER13> You're only insulted because you misread outsourcing to mean offshoring. Boeing farmed work out to other companies, and it bit them. The fact that some of those companies were based in other countries is not a point mentioned as a contributing factor.

<comment><USER12> Oh! My bad, you are right! I repent!

</comment></comment></comment><comment><USER14> _Among the least profitable jobs in aircraft manufacturing, he pointed out, is final assembly — the job Boeing proposed to retain._

...and...

 _"I warned Boeing not to make the same mistake. Everybody there seemed to get the message, except top management."_

If top execs aren't keeping an eye on where the profits are, you're screwed. Are they compensated too well to care?

<comment><USER15> No. They are given stock options which means that they have an incentive to pump the price up short term even if it means that long term the company would loose more money.

<comment><USER16> Precisely, and this explains a wide swath of seemingly nonsensical decisions being made in western countries in the last decade.

<comment><USER17> I feel like most execs are still concerned about their long term credibility and legacy ove their ability to pump some of their stock at a higher value in the short term. And I say that with full cynicism - it's simply suboptimal for them personally to do so.

</comment></comment></comment></comment><comment><USER18> I'm surprised no one is talking about the fact that this is one of the most complex projects ever undertaken by human beings. It was being designed and orchestrated in the only way we know how, and it broke down. I don't believe bringing in the people 20 layers below the top management and giving them a paycheck that says 'Boeing' on it would solve all the problems. They were building something too complicated for their basic process (blueprints, documents, proposals, etc.)

Many of the problems such as parts that don't fit and supply chain problems could be solved by current machine learning techniques. Too bad Google and Facebook hire all those people and sets them to work optimizing ad placement.

<comment><USER9> The surprising thing is that they managed to design and build the 747 in the 60s in record time, without access to machine learning or any CAD tools whatsoever.

Although the Dreamliner is a completely new design using new materials, I'd say the 747 was a far more impressive accomplishment at the time.

<comment><USER19> The problem I saw working on the 787 and other modern aircraft is that we now bury ourselves in analysis and data in the name of accuracy. Just like in programming, when the code gets complex and you have more and more dependencies. One small change impacts lots of other things and you go round and round in circles and iterations.

Back in the day they used to do more conservative, more basic calculations and conventinoal designs that they knew would work.

If there was a small change they didn't have to go and revisit everything because everything was still well within their original conservative calculations.

It's basically Paretos Law, they got 80% of the results for 20% of the effort. We are now chasing that other 20% of the results and the complexity to do so has balooned.

<comment><USER18> I appreciate someone in your position posting this comment more than you can imagine. The research project I'm working on is an attempt to confront this problem. As you say in another comment on this thread, GA's are good for small problems, but don't scale well as more _people_ are involved on the problem.

The problem is more about software engineering and project management, in my opinion. What would we replace CAD systems with if we could? What does the IDE look like when everyone is trying to encode their expertise in such a way that the final design is determined by a machine learning algorithm? That is, it's conceptually easy to imagine those final iterations being done by a machine learning algorithm, but how do we do that?

<comment><USER19> Geez, where do I begin... :)

It's a fascinating problem and on very small engineering projects with a small team of people willing to experiment with new methods it would be great to try and take things in this direction and see what works.

The reality of how aircraft projects and pretty much any engineering project really, is undertaken is currently so far from lending itself well to this type of thinking it's not funny.

Best that can be done is to chip away at little sub problems at a time tyring to improve things.

In terms of CAD data, there is a shared workspace amongst the different teams around the world. You can see what everyone else is up to.

The problem is that 90% of what goes into that design is not something that lends it self easily to an optimisation problem of any kind. At least not without gathering a ton of metrics that are currently in various documents from the 40s, people's heads and difficult to quantify metrics like what manufacturing options are currently available, what things people already have experience at etc etc.

A huge part of engineering is almost black magic. It's peoples experiences and judgements.

Also, many things are not done by analysis because we don't understand all the factors involved fully, they are done based on emperical evidence of what worked in the past. This emprical data is spread all over the place in tables and documents or just some old timers brain.

Reducing much of it to an optimisation problem is an enormously difficult task.

Then there is the analysis side of things. This is where it could be more akin to programming and could be "solved" potentially.

However, the current processes involved in doing engineering analysis makes life difficult. Basically, hundreds or even thousands of engineers produce mountains of Excel spreadsheets, words docs, text files of data and hundreds of other formats that all relate to one another somehow to tell a story of whether the aircraft is safe to fly or not.

This is far from computer code where the variables all neatly reference one another and a computer can understand it. Picking through it and figuring out how the numbers in one file match the numbers in another is a nightmare at times. I do not envy the senior engineers that have to check it and sign their name on it to say it is correct.

As you say, if everyone was working in a common IDE to do this and forcing things to have relationships the world would be a much better place for engineers. The mountain that has to be climbed to get there in the aerospace industry is enormous however.

So that I don't sound like a complete skeptic though, there is hope!

On the design side, things are most simple early on in a project where people are working on the early conceptual designs. This is were the problems are simple enough that things can be solved. It's once the conceptual design is thrown out to the large engineering departments to fill in the millions of details that the trouble begins.

And on the analysis side, the world is moving toward more and more finite element analysis. This is basically the brute force approach to analysing a structure. This is where it is in machine readable form and optimisation is possible. However, the answers the computer gives you are always layered and compromised from the engineers looking over them and having to turn them into something that's actually practical, can be built and takes into account the many requirements the computer does not know about.

<comment><USER18> I agree almost 100% with what you're saying. I'll just add 2 comments:

1) Regarding the fact that at any given time, there is knowledge that people have that hasn't been encoded in a form usable by the optimizer: I think this can be addressed by generating a set of decent designs with machine learning (on the pareto front, if you will) and then providing user interface tools to allow the humans to pick the designs that best handle the knowledge that wasn't optimized against. These user interfaces will often have to be quite elaborate but that's a tractable problem.

2) When working with Genetic Algorithms, and any machine learning technology, really, my experience has been that the first results only show what is wrong with the problem setup. The designs that are returned by a genetic algorithm are invariably nonsense at first and the then it's a matter of playing whack- a-mole as the optimizer exploits inaccuracies in the problem statement, a fix is made, a new exploit is found, repeat. This is often viewed as a deficiency of genetic algorithms but in my opinion the fact that the metrics being used can be gamed is valuable information that needs to be addressed as early as possible.

Finally, I understand my original comment came across as somewhat naive. If we ever get to a point where we can design physical products, I am imagining starting with something along the lines of a mechanical pocket watch, or maybe a nice desk chair. Airplanes are gonna be last.

<comment><USER19> Oh absolutely, if the problem is simple enough it's definately possible :)

Even in aerospace finite element optimisation techniques have led to some very effecient structures.

The problem is when you pass the limits of what is manufacturable and have to start breaking it up into simpler pieces joined together based on time, budget, abilities, what you know will work etc.

Also when you have to meet hundreds of requirements like having to fit lots of systems that people haven't designed yet and don't fully know what they'll need yet. It's the problem of compounding unknowns etc.

An awesome chair is definately doable :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER20> Boeing has been late on almost all their projects. The Australia tanker is behind by 2 years, the Japanese shipment is behind by 3 years and counting.

Its sad to see a company that is politically corrupt still win contracts by the U.S. Government even though it delivers years behind schedule.

<comment><USER21> Unlike software, which is always on schedule.

<comment><USER22> Not if you are part of the highway bandits!

</comment></comment></comment><comment><USER23> I'll take planes I don't want to get in for 1000 Alex!

Seriously, think of software projects that took years longer than predicted and went way over budget. What kind of product was finally delivered? I don't want to be a seller of fear (I hate that about the US) but I won't be surprised to see serious issues with this thing when it enters service. Anyone here ever work on a 3 year long 1 year project? I have and its rare to see the attention to quality at the end of that 3 year death march that you see in the beginning.

</comment><comment><USER24> Off topic, but I'll keep this short:

At the time I post this, there are just 52 comments on the story at LATimes site, while there are 75 here !!!

I guess that'll surprise a few.

<comment> T-hawk Not too surprising. Any talk of outsourcing will hit a pretty personal nerve among the hacker and programmer community.

<comment><USER24> A small clarification: :-)

Being fairly new here at HN, I was not fully aware of the popularity of this community and actually speaking, the only context in which I made my observation was a comparison of general traffic/interest/engagement levels between HN and LATimes.

</comment></comment></comment><comment><USER25> It depends upon how the outsourcing is managed. By outsourcing you're adding extra complexity, unknowns and unmeasureables to the problem - increasing the variety in cybernetic terms. Unless you have a sufficiently good way of managing that additional complexity you're going to get into trouble, which in business terms means overruns or in the worst case failure of the enterprise. See Ashby's law for details.

</comment><comment><USER26> Aim to establish India in global supply chain: Boeing [http://timesofindia.indiatimes.com/business/india- business/A...](http://timesofindia.indiatimes.com/business/india-business/Aim- to-establish-India-in-global-supply-chain-Boeing/articleshow/7458158.cms)

</comment><comment><USER27> Interesting viewpoint from hat retired employee who said the final assembly has the least value. Kind of put the fact that the final assembly of electronics such as the iPhone is outsourced while some of the component are home mad or at least home designed.

<comment><USER28> Why do you believe that components are home made? They are only home-designed.

Nearly all of the iPhone is built and assembled outside of Apple and outside of the US. The (non-standard parts of the) hardware are "designed by Apple in California", and the bulk of the software is created in Cupertino as well.

[http://texyt.com/iphone+manufacturer+supplier+assembler+not+...](http://texyt.com/iphone+manufacturer+supplier+assembler+not+apple+00113)

</comment></comment><comment><USER29> Not much different from what happened to airbus only a couple of years before.

<comment><USER10>
      Not much different from what happened to airbus only a couple of years before.

Uhh, no. That had much more to do with the structure of EADS (the airbus parent) and how it's a joint company between multiple European countries. Notably France, Germany, Spain and the UK.

Airbus also had French / German co CEO's at that time, which didn't help decision making.

The problem stemmed from the fact that manufacturing is divided between various plants in various countries. Notably between Toulouse (France) and Hamburg (Germany). This is certainly not the same thing as outsourcing critical components to external suppliers (which I'm sure Airbus does too to a lesser extent).

Upon final assembly of the A380 in Hamburg they bungled the cabling which hit them back another six month or so. They where other problems of course, which there are always bound to be on such an ambitious project and with the (likely) unrealistic time scale for delivery.

The A380 was two years late, which is probably not bad for such a project. I wouldn't bank on it that the Dreamliner takes of in 2011. At least not commercially.

<comment><USER4> The airbus style big block integration actually worked quite well - <USER2> from RR, undercarriage from airbus and wings from Mitsubishi all worked well.

The problem was that they eg. farmed out the avionics to mega corp, who farmed out the hardware to company A and software to B. Company B farmed out bits to company C and company D. Nobody realised that company D was a single contractor shop until they were late which went all the way up the chain.

There was an article on here at the time, Boeing literally couldn't find out the name of the company doing a vital part of the avionics software.</comment></comment></comment>


Apple Encouraging Progressive Web Apps by Rejecting Apps - <AUTHOR> https://love2dev.com/blog/apple-encouraging-progressive-web-apps-by-rejecting-apps/ <comment><USER2> > "Is Apple about to dump apps built wit frameworks like Ionic, React Native > or Cordova?"

> For now they are not impacted. But as you will see there is a good sign they > soon will be. Per the language Apple uses in their guidelines, the future of > hybrid apps rather fragile.

That's like arguing that apps built with Apple's own UIKit are soon to be subject to removal, because UIKit is a "framework."

<comment><USER3> Pretty much. And there are some pretty important apps built with react.

I think this is more intended to remove the plethora of cookie cutter apps.

</comment></comment><comment><USER4> Apple already clarified these rules and cleared up a lot of what this article is concerned about.

<comment><USER3> Link?

</comment></comment><comment><USER5> And it's an ad.

</comment><comment><USER6> Sales pitch at the end of article.

</comment><comment><USER7> I imagine Basecamp's iOS app could well be affected by this. A company which revolutionised web development fails to get the Apple seal of approval. Walled garden is too tame a label. I hate Apple's arrogance. In some ways they're worse than the Microsoft of 2000. Guess whose browser is the only one still not supporting progressive web apps?</comment>


Dropbox 100GB Plan - <AUTHOR> https://www.getdropbox.com/upgrade?plan=100 <comment><USER2> I'm not certain if this is "news" per se. It's interesting, and I'm glad that Dropbox is staying kickass, but there's not much discussion to be held, nor is this a particularly newsworthy subject.

<comment><AUTHOR> I posted it for two reasons:

a) The 100gb choice isn't listed as a normal upgrade option. People here are likely to be users, and also potential paying users. Classify this as more "useful link" than "news story".

b) This is news.YCombinator. There's nothing wrong with YC companies gaining a large amount of attention here. On the flip side, if it isn't interesting/useful then don't vote it up.

<comment><USER3> Thank you for posting it. 2 of my friends have already upgraded as they were hitting the cap on their 50GB accounts.

</comment></comment></comment><comment><USER4> At first glance, it seems a touch expensive, as you can get Github and GMail accounts that offer far more storage for less expense. But on the other hand, Dropbox don't have the advantage of being able to assume that most of their customers won't use the full 100GB, as is the case with emails or source control. Also, looking at Amazon's S3 pricing, it isn't much cheaper to use something like Jungledisk with S3 directly, and Dropbox is much nicer.

<comment><USER5> Dropbox also retains revisions and deleted files, which on free accounts at least does not count against your quota. If this is true for paid accounts, their users can end up using _more_ than what they think they are paying for.

<comment><USER6> i think i read on the forums that neither free not paid users have deleted files counted to their quota :D

</comment></comment></comment><comment><USER7> For those not wanting to create an account or sign in to find out, it's $20/mo or $200/yr.

<comment><USER6> more accurately $19/mo or $199/yr </anal> hehe

</comment></comment><comment><USER8> Dropbox is great when you need realtime access to your files. I have a free account and use it for syncning some music and important files between home and work.

If you're only using it for backup purposes, there are a lot of cheaper alternatives. I've been using Mozy (home vesion) for a year and it has unlimited space for $5/month. I don't get realtime access, but for backup it's great. As a bonus, their sync program lets you specify your own encryption key so they can't even view your file data (file names/directory structure is still open).

<comment><USER9> _[Mozy's] sync program lets you specify your own encryption key so they can't even view your file data_

... assuming you trust their security, that is. Given that they proudly advertise using blowfish -- an algorithm which even the inventor has expressed surprise that anyone is still using -- and they don't make their client source code available for audit, it's hard to place much trust there.

<comment><USER6> thats a good point.

at some time you're gonna have to trust any cloud provider, irrespective of the encryption(/lack) or the implementation - you send ur data off over your dsl and it's up for grabs. simple as that!

</comment></comment></comment><comment><USER10> What's the technical advantage of dropbox over, say, jungledisk?

Differentiators:

\- Jungledisk uses S3 which I, in principle like.

\- Jungledisk adjusts what they charge me with what I use versus charging me a flat rate when I want > 2GB frame

Non-differentiation

\- Both use a snazzy multi-OS drag-and-drop widget \- Storage is storage

It seems like strapping me into a one-size fits all "pro" account is a rather dated modality of charging. Anyone clear on that?

I'm evaluating this critically because I need to have a 'follows me' net-disk solution, but have not, as yet, found one that just bowled me over. This link was timely.

</comment><comment><USER11> Very interesting. I didn't even realize this. Thanks for sharing Jason.

With that Dell Mini 9 Hackintosh thread some hours ago, I'm strongly considering doing the Mini + Dropbox combo for my second machine.

</comment><comment><USER12> I might have something to say if the page would ever load.

</comment><comment><USER13> I'd like to see what it's about, but I am not about to make an account just to find out.

<comment><USER14> What dropbox is about? Screencast here: <https://www.getdropbox.com/screencast>

<comment><USER13> No, the provisos of this 100GB.</comment></comment></comment>