Boeing's software fix for the 737 Max overwhelms the plane's computer - <AUTHOR> https://www.moonofalabama.org/2019/06/boeings-software-fix-for-the-737-max-problem-overwhelms-the-planes-computer.html <comment><USER2> From the article

\</comment><comment>-----

 _Boeing says that it can again fix the software to avoid the problem the FAA just found. It is doubtful that this will be possible. The software load is already right at the border, if not above the physical capabilities of the current flight control computers. The optimization potential of the software is likely minimal.

MCAS was a band aid. Due to the new engine position the 737 MAX version had changed its behavior compared to the older 737 types even though it still used the older types' certification. MCAS was supposed to correct that. The software fix for MCAS is another band aid on top of it. The fix for the software fix that Boeing now promises to solve the problem the FAA pilot found, is the third band aid over the same wound. It is doubtful that it will stop the bleeding.

The flight control computers the 737 MAX and NG use were developed in the early to mid 1990s. There are no off-the-shelf solutions for higher performance.

Boeing's latest announced time frame for bringing the grounded 737 MAX planes back into the air is "mid December". In view of this new problem one is inclined to ask "which year?"_

\</comment><comment>-----

Ouch...

Introducing a new CPU into airplane like 737 will require another whole series of test/certifications.

And right about now, Boeing management is thinking _we could use some of those senior engineers we laid off because we thought they were not needed as our products are now mature._

<comment><USER3> I know military jets to have unstable flight characteristics to get advantages in maneuverability that need an operating flight computer.

But I have problems accepting this fact for civilian planes, especially if the manufacturer just wanted to save development costs. Because you could simply design a more stable plane. Not trivial, but also not impossible.

<comment><USER4> Then you probably don't want to fly in a T-tail jet. [0] [1] Or a helicopter [2]. Or in a jetliner at cruise. [3] [4]

The fact is, every single airplane ever built is unstable in the wrong circumstances. They stall, they spin, they lose lift, they suffer mechanical failures. Aeronautical engineering, rigorous maintenance, and thorough pilot training have resulted in the safest transportation system that has ever existed, but the only completely stable aircraft is one that's sitting in a hanger.

[0] [https://en.wikipedia.org/wiki/T-tail](https://en.wikipedia.org/wiki/T-tail)

[1] [https://aviation.stackexchange.com/questions/1400/how-do- con...](https://aviation.stackexchange.com/questions/1400/how-do-conventional- and-t-tails-differ)

[2] [https://aviation.stackexchange.com/questions/35764/are- helic...](https://aviation.stackexchange.com/questions/35764/are-helicopters- aerodynamically-stable)

[3] [https://en.wikipedia.org/wiki/Coffin_corner_(aerodynamics)](https://en.wikipedia.org/wiki/Coffin_corner_\(aerodynamics\))

[4] [https://www.faa.gov/regulations_policies/advisory_circulars/...](https://www.faa.gov/regulations_policies/advisory_circulars/index.cfm/go/document.information/documentid/1020859)

<comment><USER3> By stable I meant gliding properties. If the center of mass is moved too close to the center of lift, planes tend to pitch up and then are in danger of stalling. Some people suggest Ed the heavy engines might be responsible for this, but I believe this was misinformation.

</comment></comment></comment></comment><comment><USER5> Does this mean that one of the reasons MCAS relied on a single sensor could be the already saturated CPU? Mindblowing.

<comment><USER6> I thought the extra sensor was an optional upgrade?...

<comment><USER7> No, that was the AoA indicator in the cockpit. The second AoA sensor was always there, MCAS just didn't consult it.

</comment></comment></comment><comment><USER8> Simple question: Why doesn't Boeing upgrade to i386 for their new planes? I heard it's available for aerospace applications.

<comment><USER9> Forget about upgrading - they could just press the Turbo button on that 286. That's what it's there for!

<comment><USER10> On my boxes, pressing the Turbo button donwclocked them to make Zaxxon playable.

</comment></comment></comment><comment><USER11> This smells like more lives are about to be lost if Boeing doesn't get their act together

<comment><USER12> No.

A few executives might shave a fraction of a penny off their multi-million dollar annual bonuses. And a lot of lower level workers are going to lose their jobs.

But no more lives will be lost if Boeing fails to consolidate their defecation on this matter.

</comment></comment><comment><USER13> Could this be considered unacknowledged technical debt?

<comment><USER14> Maybe we're just ignorant of the available chips for this use case. Otherwise just reads like they've been kicking the can of system redesign/refactoring down the road for decades. It appears that short term thinking abounds in every industry.

<comment><USER15> I'm a developer with extensive experience in SIL-4 programming, specifically for life-critical systems (rail transport). I've built software subsystems that are used in 38 countries around the world to keep the trains safe.

The issue is the extensive costs of changing from one CPU type to another. Certification for these systems is a multi-year process and can cost millions of dollars even before any kind of success is guaranteed.

There are still very old CPU's out there, functioning just fine for decades. Safety programming doesn't just swap out parts like consumer computing does - it takes a lot of work to change CPU's.

I don't think Boeing has been kicking the can down the road on the upgrade. I do think they've been trying to cut costs and exploit their customers by offering extra safety features as upgrades, rather than making them standard. Its interesting to note that some of their customers don't require such stringent safety features in the regions they operate - i.e. this is as much of a legislative issue as anything else. It could very well be that Ethiopia doesn't have the same safety requirements encoded in its laws governing flight as France does, so Boeing offers different features not just according to budget but also legislation - although we are sure to see that change rapidly now.

</comment></comment></comment><comment><USER16> As has been discussed on HN, this whole 737 MAX flaw is not a software problem. Boeing wants you to believe it is. The flaw is actually a physical design problem of the aircraft. Even if the software were to work perfectly, the plane is still flawed. The only real fix is not to fly the 737 MAX.

<comment><USER17> This is entirely false. Stop spreading disinformation.

Myth: the 737 MAX 8 is not inherently stable, has relaxed stability, etc. Fact: it's very much inherently stable.

Myth: the 737 MAX 8 is easier to stall than other planes. Fact: no it isn't.

The pitch-up characteristic of the MAX 8 is less strong than of e.g. the 757 and that plane flies just fine.

The actual problem with the MAX 8 is that Boeing added MCAS to allow it to share a type rating with the rest of the 737 family (allowing existing 737 pilots to fly the MAX 8 without additional training), and they fucked up MCAS. There's a number of solutions on the table, including removing rather than fixing MCAS and giving up the 737 type rating.

I am continuously astounded that even on HN people are focusing on news cycle bullshit about inherent instability instead of the actual issues with Boeing/FAA that caused this situation.

Evidence so far suggests that MCAS was originally a non-critical system that was found to be too weak during flight testing, and given significantly more pitch authority. For whatever reasons, this didn't trigger the reclassification of MCAS as a critical system and it all went downhill from there.

Here's a pair of sources slightly more credible than the bullshit news cycle:

[https://www.youtube.com/channel/UCphqjYZxxzjNbONVmY-0J7Q](https://www.youtube.com/channel/UCphqjYZxxzjNbONVmY-0J7Q)

[https://www.youtube.com/channel/UCwpHKudUkP5tNgmMdexB3ow](https://www.youtube.com/channel/UCwpHKudUkP5tNgmMdexB3ow)

<comment><USER4> Ironically, I believe that your information is also inaccurate. MCAS is required _in order to meet federal airworthiness requirements_. Without it, in certain flight conditions the back-pressure on the control yoke gets less as the yoke is pulled further back. It's like over-steer in a car, and is simply not allowed. Yes, the plane is dynamically stable; if you leave the yoke in one position it won't pitch up even more. However, the forces must be corrected somehow, regardless of type rating concerns.

 _" The 737 MAX was a bit too easy to pull into a stall when flying with high AoA and making abrupt maneuvers. The larger engines for the MAX hung further forward from the wing, added a destabilizing aerodynamic area ahead of the center of gravity, destabilizing the pitch moment curve at high AoA.

Boeing and the certification authority, FAA, decided added margins was called for. Boeing added a pitch augmentation at high AoA called Maneuvering Characteristics Augmentation System, MCAS.

The aircraft should trim nose down to increase the stick force needed once it passed into the light grey area where the base aircraft had a region of less stability. Before the augmentation, the pilot felt if the aircraft wanted to fly into the stall, it got easier to increase the AoA after 12°AoA. With the augmentation the felt extra force was the same for the first and last part of the curve before the maximum lift was achieved at stall (and stall warning kicked in)."_ [0]

The _manner_ of the fix (MCAS transparently pushing the nose down) was designed to avoid pilot retraining and thus keep the same type rating.

Edit: The fact that the 737-Max needs a handling tweak is _not_ a failure. Modern planes have all kinds of these tweaks, whether aerodynamic (such as strakes), mechanical (stick shakers) or enabled in software. As the cited article continues: _" So far so good. It's common an aircraft's flight control system has fixes to stability margin changes in different parts of the flight envelope."_ The problem is that Boeing had a pretty severe collapse of its systems engineering regime.

 _" The implementation for the 737 MAX had two problems, however:

\- The fault checking of the triggering AoA signal was not rigorous enough. This problem has been discussed a lot. No need to add anything.

\- The judgment the pilots would identify a problem with the augmentation as a trim runaway and shut the trim off was wrong. Why the pilots didn't see MCAS rouge actions as a trim runaway is poorly understood."_

(The article was published in February. Since then lots of information has come to light about how MCAS determinedly fought correction, and the huge mental and physical loads imposed on the pilots.)

Edit 2: FAA regulation mandating increasing elevator forces for _all_ transport aircraft: FAR §25.253 High-speed characteristics, (a) Speed increase and recovery characteristics, (3):

 _With the airplane trimmed at any speed up to VMO /MMO [maximum operating airspeed], there must be no reversal of the response to control input about any axis at any speed up to VDF/MDF [maximum airspeed demonstrated in testing]. Any tendency to pitch, roll, or yaw must be mild and readily controllable, using normal piloting techniques. When the airplane is trimmed at VMO/MMO, the slope of the elevator control force versus speed curve need not be stable at speeds greater than VFC/MFC [maximum control airspeed], but there must be a push force at all speeds up to VDF/MDF and there must be no sudden or excessive reduction of elevator control force as VDF/MDF is reached._ [1]

[0] [https://leehamnews.com/2019/02/08/bjorns-corner-pitch- stabil...](https://leehamnews.com/2019/02/08/bjorns-corner-pitch-stability- part-9/)

[1] [https://www.ecfr.gov/cgi-bin/text- idx?node=14:18.104.22.168.11#se1...](https://www.ecfr.gov/cgi-bin/text- idx?node=14:188.8.131.52.11#se14.1.25_1203)

<comment><USER17> "MCAS is required in order to meet federal airworthiness requirements"

Cite this. Specifically this. The rest of your comment agrees with mine without this being true, and I have not seen any evidence of this being true.

<comment><USER4> Hmm. OK, better wording would be "Handling mitigation such as MCAS is required in order...". I've seen several news articles, such as the one cited, which reported that the yoke forces decreased near stall AoA in certain flight regimes. Here's one from the NYT: [0] Originally MCAS was implemented for rather extreme maneuvers, and required both the AoA sensor and a G-force sensor to agree. Later, it was discovered that low-speed stalls also had yoke- force problems, and the control authority was increased, and the G-force requirement dropped.

[0] [https://www.nytimes.com/2019/06/01/business/boeing-737-max-c...](https://www.nytimes.com/2019/06/01/business/boeing-737-max- crash.html)

<comment><USER17> And as stated in countless well cited articles, this was done in order to maintain the 737 type rating and make the MAX 8 handle like other 737. Not to meet regulations that were otherwise unmet.

</comment></comment></comment></comment></comment></comment><comment><USER18> Is the 80286 still in production? Or is it old stock on a shelf somewhere that is used?

<comment><USER19> AFAIK, Airbus has stockpiled huge amounts of legacy CPUs in climate controlled conditions. Boeing must have done the same

<comment><USER20> Was really curious as to what Airbus was using, thanks for this! Why not just use the latest CPUs though? I can't imagine the cost would be relatively significant in the greater scheme of things. Are the legacy CPU's really better for this use-case?

<comment><USER19> Aero industry moves at a very slow pace. The state of the art F-22 Raptor uses intel 386 cpu and the F-35 lightning upgraded to more "modern" PowerPC CPUs, similar to the ones on legacy Macs.

Planes don't need insane multitasking processing power like our smartphones or PCs. They mostly do signal processing and sensor fusion in a tight loop which is quite trivial even for legacy CPUs as it's basic flight math equations which results in highly optimized code.

In terms of aero chips, basic is always better as you want a silicon that's tried and tested for decades to have a deep understanding of it's quirks and bugs so you know the code execution is reliable.

</comment></comment></comment></comment><comment><USER21> They may find a physical fix for it later. Like adding weight where it can help. Even then, perhaps nobody wants to fly with it plus it can't offer the same performance.

</comment><comment><USER6> It amazes me how Pilots are required mainly for the purposes of being able to act as a last resort against a computer which has gone haywire, and the public feel better thinking that the pilot can "take control" and then at the same time aircraft manufacturers are removing the ability for the said Pilots to override the computer. That's progress!

</comment><comment><USER22> Mods: Can you remove the #more from the submission URL? The fragment makes the browser scroll down which skips the first half of the article.

<comment><USER23> Sure.</comment></comment>


Can feds be fired for adultery? Feds search AM leak - <AUTHOR> http://www.washingtonpost.com/blogs/federal-eye/wp/2015/08/25/can-feds-be-fired-for-adultery-the-government-is-combing-through-thousands-of-e-mail-addresses-that-turned-up-in-the-ashley-madison-leak/

<comment><USER2> I suspect they're more interested in who might potentially be blackmailed.</comment>


Data-Oriented Design (2013) - <AUTHOR> http://www.dataorienteddesign.com/dodmain <comment><USER2> > Data is all we have. Data is what we need to transform in order to create a > user experience. Data is what we load when we open a document. Data is the > graphics on the screen and the pulses from the buttons on your game pad

I've been trying to explain this concept to UI-focused devs for years, it's nice to have a (seemingly) well written resource I can point them towards instead.

<comment><USER3> Yeah, it's such a critical concept that I feel gets lost in the mix of frameworks, languages and fads of the day.

In the end it's just Data all the way down, just Data.

Want to update something in your app on the fly? Just change the Data. Need adjust your UI? Again, data. Once you grasp that then all sorts of composability and techniques become possible.

Lisp takes this to a nice logical conclusion where everything is S-expressions and code is just data.

</comment></comment><comment><USER4> This looks interesting but is hard to read in the online form (the black on gray is particularly taxing on the eyes). Do the authors plan to make a PDF available, esp if more readable?

<comment><USER5> I believe this is the source: [https://github.com/raspofabs/dodbook](https://github.com/raspofabs/dodbook)

So you can browse the Markdown or potentially run it through some of the "markdown to webbook" services that exist.

</comment></comment><comment><USER6> I guess I'll read the markdown a little bit later because I can't read those pages, but I recall a talk like this given at some game developers confence (something Sony I think) a while back.

But since most of us don't worry about cache misses, I'd like to see some more information about this for a more general programming audience.

I think the clojure folks subscribe to something along these lines, but mostly because they eschew OO.

</comment><comment><USER7> If you are looking for something to use this good theory on take a gander at Drools [http://www.drools.org/](http://www.drools.org/) (Apache v2 license)</comment>


The Pushbutton Web: Realtime Becomes Real - <AUTHOR> http://dashes.com/anil/2009/07/the-pushbutton-web-realtime-becomes-real.html <comment><USER2> This is a superb explanation. PubSubHubBub is a beautiful piece of technology - I hadn't realised it was supported by FeedBurner. Should really help bootstrap further adoption.

<comment><USER3> How on earth can one pronounce "PubSubHubHub" in a conversation?

<comment><USER4> Are you kidding? It's a perfect rhyme: Pub. Sub. Hub. Bub.

It distinguishes it as a specific standardized implementation of the PubSub architectural pattern.

It's no worse than Aitch. Tee. Tee. Pee.

Or Doubleyou. Doubleyou. Doubleyou.</comment></comment></comment>


Creepy 'Geofence' Finds Anyone Who Went Near a Crime Scene - <AUTHOR> https://www.wired.com/story/creepy-geofence-finds-anyone-near-crime-scene/ <comment><USER2> For people wondering why this is bad: Let's say you have a DNA test that's 99.9% accurate. If you find a suspect through normal investigative means and then test his DNA and he matches, you've almost surely got the right guy.

If you geofence the crime scene and test/Ancestry-scan the 10,000 people who went through the crime scene that day, you'll get a DNA match for 10 innocent people, and a lay jury won't have the statistical fluency to understand how the two scenarios are different.

</comment><comment><USER3> "Creepy policing investigates the evidence at crime scenes."</comment>


Books for experienced software developers? - <AUTHOR>

Are there any good books aimed towards experienced software developers? I've read through all the classic books on code quality, architecture, unit testing, domain driven design, etc. I find most of these books explain what should be common sense, I yearn for a book that goes deeper than "Don't Repeat Yourself, Write your tests first!, Isolate your dependencies and inject them! Use Composition rather than Inheritance!".<p>What I'm looking for is a book that offers a deeper perspective, or reveal new ways to apply core knowledge to build higher quality software. Something for a developer who is already competent and experienced. <comment><USER2> On physical design:

Large Scale C++ Software Design, John Lakos. [http://www.abebooks.com/servlet/SearchResults?sts=t&tn=large...](http://www.abebooks.com/servlet/SearchResults?sts=t&tn=large+scale+c%2B%2B+software+design)

</comment><comment><USER3> Have you read Structure and Interpretation of Computer Programs?

<comment><AUTHOR> Yes. It's one of the classics, if not THE classic. Good book, but again there's nothing really novel in there for an experienced developer.</comment></comment>


An Essay for Programming Students  - <AUTHOR> http://hardforum.com/showthread.php?s=5b832586c573a1a7051c5fb14f69c90a&p=1028953760

<comment><USER2> That's really not an essay.</comment>


Cheerp – A C/C++ compiler for web applications - <AUTHOR> http://leaningtech.com/cheerp/blog/2014/08/11/Cheerp-1.0/

<comment> pierre-renaux If anyone from Cheerp reads this, I use emscripten ([https://github.com/kripken/emscripten](https://github.com/kripken/emscripten)) at the moment, is Cheerp any better ?

Specifically does it compile faster, generates a smaller final .js output, or produce "faster" code ?

<comment><USER2> The Emscripten author wrote a blog post when Cheerp (then called duetto) was first announced:

[http://mozakai.blogspot.co.uk/2013/11/c-to-javascript- emscri...](http://mozakai.blogspot.co.uk/2013/11/c-to-javascript-emscripten- mandreel-and.html)

<comment><USER3> As far as I know that comparison is still up to date, but if the Cheerp devs are here maybe they can comment.

I may do a followup post now, since Cheerp is at 1.0. Seems like a good time to do some benchmarking.

</comment></comment></comment><comment><USER4> From my understanding it also allows you to write your front and back end in one application and language. Then it compiles the front end to JS and the backend to C++ executable. I think this is where a large advantage can be seen. edit: it automates the connection between the front and backends tying the two together.

</comment><comment><USER5> Reminds me of Wt [http://www.webtoolkit.eu/wt](http://www.webtoolkit.eu/wt)

</comment><comment><USER6> This is somewhat off-topic but they turned user-scalability off, making it impossible to zoom out (of the pre-zoomed in) mobile site.

Other than that, it's an interesting proposition and I look forward to trying it out.

<comment><USER7> Thanks for the report, we have fixed that issue today.

</comment></comment><comment><USER8> "Cheerp compiles to native binary code backend, JavaScript frontend, and automatically generates RPC communication code"

The RPC feature sounds like it could be immensely useful for day-to-day web development. But the C/C++ is a deal-breaker for most web developers. Are there any projects that only do the RPC autogeneration stuff?

Also can frontend/backend code be shared (if there are no dependencies to the browser/unix system)?

<comment><USER7> Yes, code without client- or server-specific dependencies can be shared.

</comment></comment><comment><USER9> I wonder what advantages choosing C++ to write web apps would bring. At the end of the day it's all JavaScript so performance is rather moot point.

<comment><USER10> That's actually not true. Emscripten compiles to code that is much faster than hand-written JavaScript. In this case they are not using Emscripten or a similar technique but are compiling to regular old JavaScript, so performance probably is the same, rather this might be for people who just like C++, like what GWT is for Java devs.

<comment><USER9> You misunderstood. People mostly write with C++ when performance is critical. Why? C++ offers deterministic memory management, Specialize to CPU instruction set and better use of CPU caches etc. Those performance benefits would not translate when you cross compile to something like JavaScript. I get the point about converting the legacy desktop apps to web apps, but how would it handle the architectural differences ( Desktop apps don't scale)

<comment><USER11> > People mostly write with C++ when performance is critical.

Performance is a big reason people use C++, but there are many others. For example, people also write C++ when correctness is critical (bank software, safety-critical systems, etc.).

<comment><USER12> Writing for correctness in a language with no memory-safety, weak type system and, till not very long ago, no standardized memory-model, and UB in every other paragraph of the specs? Good joke. If they are really doing this, they have no idea what they are doing. There are plenty of languages better for safety-critical systems than C++; even widely hated, boring Java is much better.

<comment><USER11> It's a bit off topic, but I'm being descriptive (based on professional experience) and not normative when I say C++ (and C and Ada) are used for safety-critical software. Javascript and functional languages certainly aren't.

The vagaries of the standard aren't issues since safety critical software is validated on a per-platform and per-compiler basis. Memory safety is certainly a concern during the development process but more importantly (!) GC languages (like javascript and Scala) do not have provably (for some value of provably) deterministic execution times. Determinism is also a problem for lazy languages (like Haskell).

I say that to say the zero-cost abstractions of C++ is a huge benefit when writing safety-critical software. The level of control over emitted binaries is essential and fairly rare in programming languages.

> If they are really doing this, they have no idea what they are doing.

Well, I can't speak for entire industries, but that's a pretty sweeping generalization. You might be surprised what the challenges are when writing safety-critical systems software. That being said, I'm sure large portions of the industry are ripe for innovation.

<comment><USER12> > GC languages (like javascript and Scala) do not have provably (for some > value of provably) deterministic execution times

This is not unique to GC languages, but any languages with dynamic memory management. C++ new/free and STL abstractions built on top are not provably deterministic either. And if you program without ever touching dynamic memory (statically allocating and pooling everything) then GC is not a concern.

> I say that to say the zero-cost abstractions of C++ is a huge benefit when > writing safety-critical software.

You're talking about performance now, not correctness. All those benefits get lost once translated to JS.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> A different idea, but still an interesting C++ Web development framework: [http://cppcms.com](http://cppcms.com)

</comment><comment><USER3> I'm a bit puzzled by the rebranding. First of all, the logo and the name sound a bit similar to Twitter. And I'm not sure what bird noises have to do with a compiler?

The previous name, Duetto, made sense to me, since they compile to both client and server, so it's like two things running in harmony, which is a duet in music.

</comment><comment><USER14> I know C and C++ pretty well; I wonder what this "C/C++" language is like.

<comment><USER15> The subset that is the interception of both? which would be a subset of C. (Although I am pretty sure that wasn't what the author had in mind)

<comment><USER16> Bjarne has always maintained that C++ is (mostly) backwards compatible with C. And that's what most people mean by C/C++.

<comment><USER14> I know that C++ is mostly, but not entirely, backwards compatible with C. That doesn't answer the question of what "C/C++" means.

They're two different languages. The problem is that different people mean different things by "C/C++". Some people mean "C _and_ C++"; others just seem to be unaware that they're distinct.

<comment><AUTHOR> Cheerp, like clang (upon which it is based), can compile both C and C++ files.

</comment></comment></comment></comment></comment><comment><USER17> Interesting although if you want massive performance on a web application i'd probably vie for erlang unless you're doing something computationally complex on the backend. It would be nice to be able to write code once and use in multiple places but theres often so much extra work involved in that, that for many applications it makes more sense to just use interfaces etc.

Neat project either way.

<comment><USER18> If you want massive performance, you most likely want to avoid erlang; if you want massive scalability, erlang might very well be a great fit. I think it's quite different.

<comment><USER17> Performance in terms of request throughput you can serve per minute for some given set of CPU and RAM.

Erlang processing is slower than C by quite a bit but the reduced threading costs tends to make up for it when you are in a scenario where you need to serve multiple requests simultaneously.

On an apples to apples comparison well written C is going to beat erlang on this every time but once you start getting into threading, mutex locks etc. the equation starts to shift more towards Erlang favor _as long_ as each request you are serving is not highly computational in nature and would involve semaphores, and so on for thread management in c.</comment></comment></comment>


Computer solves a major time travel problem - <AUTHOR> https://cosmosmagazine.com/physics/computer-solves-a-major-time-travel-problem <comment><USER2> How did clickbait like this article make it onto HackerNews? That article is void of any actual scientific content, and is pure garbage.</comment>


WSJ: AT&T Plan Would Let App Makers Pay for Subscribers' Data Use - <AUTHOR> http://www.one37.net/blog/2012/2/27/att-plan-would-let-app-makers-pay-for-subscribers-data-use.html <comment><USER2> I see too many problems with this idea.

(1) App makers that transmit data already pay for their web services' bandwidth. So they are basically getting double charged for the same data. (2) If app makers transmit data to other open services (Twitter API, Facebook API, YouTube API, etc.) then does the app maker pay for the subscriber's data use or does the API provider? Looks like a lot of room for confusion. (3) It shields the true cost of the subscriber's usage in much the same way I see company paid health care shielding employees from the true cost of health care. I would imagine this would result in AT&T raising their fees more and more against the app makers without subscriber's being aware and taking the blame on any increase in the cost of the app on the app maker rather than AT&T.

<comment><USER3> Given metered pricing, it's not really double-charging. Today the server pays for the packets to travel half way and the customer pays for them to travel the rest of the way. If the app pays for both halves then (in theory) the total charge is the same; it's just being paid by a different party. I doubt that AT&T would charge an app the same ridiculous overage fees they charge customers, though.

Regardless, the market distortion caused by the cost-shifting is real and problematic.

<comment><USER4> > If the app pays for both halves then (in theory) the total charge is the > same

It sounds like you're assuming the carrier (AT&T) would stop charging the consumer for the bandwidth as well. As in AT&T would offer a data plan without a data fee. I'd wage that AT&T's gambit is to get paid twice for the same carriage, and not what you have in mind.

<comment><USER3> "A feature that we're hoping to have out sometime next year is the equivalent of 800 numbers that would say, if you take this app, this app will come without any network usage."

</comment></comment></comment></comment><comment><USER5> So, in other words, unless you're already rich enough to pay for your customers' data usage, your app will be at a disadvantage on AT&T's network. This is why we need source-neutrality in networks. Otherwise existing players can subsidize their services and "dump" their product in order to drive new competitors out of the market.

<comment><USER6> > This is why we need source-neutrality in networks.

No, we do not. There are already laws on the books against "dumping" their product to drive out competitors. It's called price bashing and it is illegal.

If Netflix is able to subsidize their data usage and remain profitable, great. If, on the other hand, Netflix decides to go with this to kill off the competition with the calculus that they will lose money temporarily and hike prices after the competition is gone ... well, that's illegal and on top of everything, won't even work.

Also, AT&T's ploy here won't work either. They are trying to increase data usage while hiding the cost of that usage in 3rd party fees. Yeah. Watch content providers offer two price tiers. One for AT&T and one for non-AT&T. Won't work and a waste of time. You're just shifting costs.

Also, 800-numbers became popular during a time when every street corner had a pay phone. It was about actually letting the customer contact you from anywhere, not about shifting costs.

There is no need for regulation. The market will take care of this stupidity by itself.

<comment><USER7> Sounds great on paper. In real life there's a certain degree of collusion between the major carriers, and it's possible AT&T could convince Sprint and Verizon to play the same game. It's also possible one of them had the idea to begin with, and AT&T won't even be the first to implement it.

After a couple of years you might get a class action lawsuit far enough along that the FCC says the whole thing is perfectly legal and it doesn't see the problem. Or the carriers might backstab each other for secret reasons. But this matter cannot be trusted to the market.

</comment></comment></comment><comment><USER8> This is the equivalent of free shipping. I predicted it a while ago (patting back): [http://richvsreach.wordpress.com/2008/06/03/the-long-game- on...](http://richvsreach.wordpress.com/2008/06/03/the-long-game-on-metered- pricing-free-shipping/)

What it's not is net-neutral. I'm OK with that.

<comment><USER9> Good call. I called it too, much later than you though.

[http://blog.altudov.com/2011/08/27/the-upcoming-amazons- andr...](http://blog.altudov.com/2011/08/27/the-upcoming-amazons-android- tablet-will-eclipse-the-iphone-app-store/)

</comment></comment><comment><AUTHOR> The cost for data does not dissipate, it is merely redistributed. Rather than giving your money to AT&T for your service, it will just be charged by data- heavy services - particularly as this would increase their data costs.

Furthermore, reliance upon such a system would likely give AT&T license to reduce data package size, thus harming the average smartphone user.

Although AT&T might frame this as innovation, such maneuvering is transparently self-serving. Rather than having the customer blame them for high costs and decreasing data allowances, AT&T can place the blame elsewhere.

Knowing AT&T as we do, the potential for good in this model is rather woefully undercut by its obvious potential for bad.

</comment><comment><USER10> So instead of giving up on the flawed idea of data caps, they are putting the onus of caps on app makers instead of customers? I can only imagine that this new spin on an old strategy will have all the same problems.

</comment><comment><USER11> Perhaps this technical post from AT&T labs (<http://bit.ly/zFyXTw>) discussing their work with Pandora to optimize power & bandwidth consumption might shed some light on AT&T's thinking. It might be a win-win for AT&T to partner with such app producers, optimize their network utilization in exchange for some end user subsidy -- extending the subsidy carriers give to device manufactures to the app developers, essentially?

</comment><comment><USER12> This is Amazon's Whispernet tied to service instead of device, and it is certainly, "fair."

We don't expect users to pay service fees to get Kindle books delivered.

</comment><comment><USER13> I think this is really a smart move, and would create a brand new revenue stream for the carriers coupled with (ideally) cost-savings for the customer. It'd also encourage efficiencies by app developers in how they use the network.

</comment><comment><USER14> It would be REALLY interesting to flip this model.

In other words, AT&T redistributes a portion of their subscription revenue to app makers based on the bandwidth they get people to use.

Then, app makers would have a direct financial incentive to drive usage, without all the hassles of explicit billing.

<comment><USER15> But more bandwidth usage is _exactly_ what AT&T doesn't want.

<comment><USER16> What a messed up system. AT&T is in the business of selling bits. Most sellers want to sell more of what they sell, not less...

<comment><USER17> No, AT&T is in the business of selling _phone contracts_. The actual bits themselves are loss leaders to help sell those subscriptions.

<comment><USER16> OK, you're right. Maybe it was wishful thinking: they should be in the business of selling bits.

</comment></comment></comment></comment></comment><comment><USER18> Should restaurants pay for the gas I use to get there?

<comment><USER19> This is a poor analogy, because you don't pay a flat monthly fee for your gas usage. The customer is already incentivized not to drive too far to eat at a restaurant because of the fuel expense and the time required to get there.

There are few analogies that work well outside of examples involving utilities, but most utilities are metered rate. The carriers built their current pricing models based on a usage profile that is rapidly disappearing. Basically, they priced their service for a model where the user rarely uses the actual service. Now they're desperately trying to escape their own trap.

Carriers believe that users don't want metered service, and in fact, they don't. Why would I want to expose myself to additional risk when a flat rate service is already available? Carriers captured massive profits when cellular users didn't utilize their plans, but aren't prepared to take the hit when consumers actually use what was sold to them. It's a sham.

</comment></comment><comment><USER20> That actually doesn't seem like such a bad idea. Well except that the connections should have been unmetered like residential broadband is (in most of the world) in the first place...

</comment><comment><USER21> Can someone summarize the actual WSJ article or submit a non paywall link?

</comment><comment><USER22> What the hell? I __already __pay for my data. AT &T is just making a blatant money grab.

They need some competition.</comment>


Echo 4k TV: Amazon - The First Major Tech Company to Figure Out Television - <AUTHOR> https://www-yahoo-com.cdn.ampproject.org/c/s/www.yahoo.com/amphtml/finance/news/amazon-4k-smart-tvs-having-130000335.html <comment><AUTHOR> This is huge. The idea that a tech company would figure out a way to integrate their vision of television into the central living room appliance has been enough to swing the stock of major tech companies upwards and down. However, prior to this, it was never convincing that the tech players had something useful to offer beyond just another smart television. But an Echo TV is a game changer. For the first time, a major tech company has something compelling to offer to the average household as their vision of television. Now let's see what skills developers come up with. Wouldn't surprise me if it takes a few iterations to perfect this, but if anyone understands iterating towards perfection, it's Amazon. This is an exciting moment in tech.</comment>


Show HN: Real-time Fare Estimate Comparisons for Uber, Lyft, Sidecar, and Taxi - <AUTHOR> http://www.whatsthefare.com/ <comment><USER2> I love products like these. Slices right through all the promotional noise like "fun drivers," "fast pick-up times," "clean cars," etc and gets right to chase -- price.

At the end of the day, they are all taxi services in my book. Whichever gets me from point A to point B the cheapest wins my dollars. Fast pick-up times, a sexy functional app, and other common sense differentiators are expectations now, not selling points.

<comment><USER3> > At the end of the day, they are all taxi services in my book.

Unless you live in New York, and are trying to get from Manhattan to one of the outer boroughs, in which case getting a yellow cab is a total crapshoot.

<comment><USER4> It shouldn't matter one bit. They're legally obligated to take you anywhere in the five boroughs.

Just get in before telling them your destination. If they give you flak (and you feel like fighting them), just take note of their medallion # and state clearly that you will report them if they fail to fulfill their obligation.

<comment><USER5> Or, you know, pay an extra $3 to Uber and avoid that situation all together.

It shouldn't matter, but it does.

<comment><USER6> There are even fewer regulations preventing Uber drivers from declining a fare based on location - or race, or gender, or they don't like your sports team, or anything.

<comment><USER3> Does that actually happen?

<comment><USER6> This article claims that Uber is actually better than cabs because the drivers don't know your race when they agree to pick you up. [http://www.racialicious.com/2012/11/28/cab-drivers-uber- and-...](http://www.racialicious.com/2012/11/28/cab-drivers-uber-and-the- costs-of-racism/)

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Uber API TOS says:

"You may not use the Uber API in any manner that is competitive to Uber or the Uber Services, including in connection with any application, website or other product or service that also includes, features, endorses, or otherwise supports in any way a third party that provides services competitive to Uber's products and services, as determined in our sole discretion."

[https://developer.uber.com/v1/terms/](https://developer.uber.com/v1/terms/) under 2.A

<comment><USER8> Here's one angle. If this app becomes popular and Uber sends a C&D, then Uber results will no longer show up in this app, meaning Lyft would get all the bookings from people who use it.

</comment></comment><comment><USER9> Nice.

Now just give me an iPhone app where I push a button and it sends me the cheapest, closest car from any of those companies.

<comment><USER10> One of my buddies tried this idea. It seems to have failed (their website is now down). Archive: [http://appcrawlr.com/ios/red-ride-ride-sharing-car- picku](http://appcrawlr.com/ios/red-ride-ride-sharing-car-picku)

<comment><USER11> They ended up getting a C&D from Uber.

</comment></comment></comment><comment><USER12> This how you know when something becomes a commodity. Uber/Lyft/Taxi/Sidecar they will just be competing on price in the long run, since they all compete in a commoditized market, and the player with better ubiquity and infrastructure will ultimately win, because they will be able to cut prices in a way that smaller players won't be able to afford.

<comment><USER13> This is a problem. If one of them gains a monopoly they will be able to lower service quality, raise prices and still be able to kill/buy off any new small competitor.

<comment><USER14> ...so, like taxi medallions?

</comment></comment></comment><comment><USER15> Taxi rate in NYC seems too low. I tried a couple of trips that I have taken dozens of times, and they have never, ever been as low as whats being calculated. I like the idea though.

</comment><comment><USER16> This is brilliant and one of the many reasons I love this site. Simple Bootstrap interface that cuts through the crap, the fancy user interfaces of ride sharing and taxi apps and gives me the information that I really want: the cost.

Now this has hit the homepage, it'll be interesting to see if Uber, Lyft and other providers cut you off from their API's. Presumably, some of these services don't want you to be able to compare estimated costs between competing offerings, but it also means that everyone is being kept honest and as a consumer, I think that is the most important thing of all.

</comment><comment><USER17> Question to America. Why do you tip the taxi driver but not the uber/lyft driver?

(SFO -> Pier 39 - Lyft is $31, Taxi is $49.96 + 15% standard tip = $57)

<comment><USER18> Uber fare is supposed to include tip.

<comment><USER19> Which is kind of a funny way for Uber to say "we don't do tipping."

I mean, I guess you could say that my salary "includes a tip," and that it's really 20% lower but then there's a 20% tip on top of that, but obviously I wouldn't actually say that.

Uber says that their fare "includes a tip" because people are used to tipping cabbies. But tips are only tips when they are optional amounts that customers can vary the size of.

I don't think that rides-for-hire should really be a tipping industry anyhow.

<comment><USER20> I don't know where you are from, but North America tipping is not optional in certain situations. A waitress in a restaurant does not get payed a salary, but lives solely from tips. I don't know if the same is true for taxi drivers (who are not the owner of the car).

<comment><USER19> Yes, I've seen Reservoir Dogs too. Without engaging the extremely shopworn discussion about how much tipping is appropriate, the fact remains that if you choose, you can tip a waiter nothing. It may be a dick move, but you can do it.

You can't "tip" an Uber driver nothing. It doesn't matter how bad your service was. It doesn't matter how much you don't care about being labeled a dick. You just can't do it.

Because the Uber thing is not, in fact, a tip.

<comment><USER21> Correct, they're just removing the social requirement.

The goal here being "no cash exchanged".

</comment></comment></comment></comment></comment></comment><comment><USER22> For NYC to New Jersey the fare estimate does not take into account bridge and tunnel tolls. I was getting $19, $23 and $23 for Taxi, Uber and Lyft respectively, which is very low for all of those services. A typical taxi will cost $45-$55 from Manhattan to Jersey City or Hoboken.

</comment><comment><USER23> Cool idea! FYI, when I clicked the 'locate' button and denied it access, the spinner just kept going and I couldn't manually enter a location.

</comment><comment><USER24> This is awesome. One suggestion is a permalink for a specific fare estimate that I can forward someone.

</comment><comment><USER25> Great app! Would be great to estimate "regular" price, right now it's surging and I'd like to know what the price would be for e.g. tomorrow. Hopefully Uber wont shut you off, seems like a breach of ToS to me (I'm not a legal expert).

<comment><USER26> It says +25%, so should be easy enough to guess.

</comment></comment><comment><USER21> This is alarmingly inaccurate. Says it'd take $71 to take a taxi from Alexandria, VA to NE Washington, D.C.

If you live in the area, you'd know that's a bold-faced lie. It'd probably cost $25-$30 bucks, the same price being quoted for Lyft and Uber.

<comment><USER27> I don't know the addresses you're looking at, but it's about $20 from Mill Rd in Alexandria to DCA (I've taken the route a ton of times), which is a _much_ shorter distance than to NE D.C. I don't know if $71 is entirely accurate, but it's definitely more than you're quoting.

EDIT: Just quoted it. It's about $42 in light traffic, about $69 in heavy traffic (from the hotel I stay in Alexandria to an address on Oglethorpe St in NE D.C.). So they're providing the worst possible fare, whereas they should probably provide the median.

<comment><USER21> There is no universe in which $42-$69 is remotely possible, unless there's something along the order of $20 surcharge for airport dropoff.

I do this bimonthly, and it's ~$25, with tip.

I hate to think of the nefarious reasons for overquoting taxis like this...

</comment></comment></comment><comment><USER28> I just looked up the ride I took this morning and it doesn't seem to include any add-on charges, like the toll I was charged, the Uber Safety Fee or the Seattle Accessible Driving Fund.

edit: oh yea, it says that in small print at the bottom of the page.

<comment><USER29> I'd imagine these would be added to all fares so as a price comparison tool it'd still be valid.

</comment></comment><comment><USER30> I'm confused... if I type in San Fran to Los Ang I get $608 for Lyft and over $1K for the other two.

Why is Lyft so low (like half if not more) competition makes sense, but ~400 mile trip should be more then $600.

<comment><USER31> At this range, the estimate can be wildly inaccurate. For example, there may be a maximum distance imposed, say 60 miles at about $120 [http://orangecountylyftcoupons.wordpress.com/2014/03/05/how-...](http://orangecountylyftcoupons.wordpress.com/2014/03/05/how- far-can-you-go-in-a-lyft-whats-the-maximum-ride-distance/)

</comment></comment><comment><USER32> Uber claims to be 20% cheaper than yellow cabs in NYC. What gives?

</comment><comment><USER33> It would be nice for completeness, if either start or end is an airport (e.g. "SFO", "LAX") to include an estimate of a shared-ride van such as supershuttle.

</comment><comment><USER34> Which locations does this website have actual data for? Just tried two places in San Francisco and it said it had no data.

Is this an actual site or just a one pager looking for interest?

<comment><USER35> What two places did you try? It works for me.

<comment><USER34> Tried going to a place on Market to a restaurant in Sunset, both had addresses...

<comment><USER35> I am trying "2020 Market Street, San Francisco, CA, United States" and "1330 9th Ave, San Francisco, CA 94122" and the site works for me.

I'm on Chrome 37.0.2062.94 on Ubuntu Linux 14.04 on a desktop.

</comment></comment></comment></comment><comment><USER36> How long is this going to last before one of the services shuts it down, claiming some kind of EULA noncompliance (or some other flimsy excuse to avoid competition)?

</comment><comment><USER37> This reminds me Movigo [1]. Is there any relationship or known difference?

[1] [http://www.movigo.com](http://www.movigo.com)

</comment><comment><USER38> You'll want to adjust yellow taxi fares from Manhattan to JFK and reverse. This is done on a flat fee.

</comment><comment><USER39> Looks like none of 'em hold a candle to Taxis in small-to-medium cities like mine.

<comment><USER40> Huh, in SF starting from my place to all of the places I might go Lyft wins consistently, and sometimes by as much as 1/2 of the taxi price.

<comment><USER25> Unfortunately (in my limited experience with Lyft) the quality of UberX drivers are far better than Lyft drivers. Half the time in Lyft cars I've been worried about crashing! I'd rather pay that extra dollar or two and have a comfortable ride.

</comment></comment></comment><comment><USER41> This is really cool. How do you guys get current surge/prime time rates?

<comment><USER42> Uber has an API, but I looked into Uber's TOU a week a two ago and they explicitly disallowed this sort of thing (comparison shopping). Maybe it's just an estimate?

<comment><USER43> This is probably just violating the TOU. As a consumer, I think lack of comparison shopping is absolutely the worst thing about lyft/uber, and that they should be required to publish pricing data in a manner accessible to this sort of thing, just like taxis are required to use certified meters and post rates on their doors.

</comment></comment></comment><comment><USER44> Great app. Pull in wait time and you might have something.</comment>


Ask HN: Review my startup, CampaignBar - <AUTHOR>

This is my minimum viable product, CampaignBar: http://campaignbar.com
I wrote this after getting tired of maintaining an Excel spreadsheet of which sites I wanted to contact and their current contact status. There are two parts to it: a Firefox toolbar where you can save or ignore sites (along with their contact information); and a web front-end to the data.
The site supports multiple users, so if you outsource website marketing you can set your guys loose and they won't be able to duplicate the sites they've found. Currently, any users you add can only use the toolbar - they have no access to the admin area.
When I have some more time, I want to add the ability to change the status of sites, as well as let you add different types of user accounts, and add in some useful statistics as well.
I'd love to know what you think about the functionality of the site, whether it's obvious what it does, and if you give it a try how everything works for you :)
Thanks, Tom. <comment><USER2> I would explain a bit better what it does and that it's targeted to advertisers and not publishers.

Maybe I'm retarded, but to me, it's not too clear that what it does is "helps you to track communication with webmasters to put ad campaigns on their sites".

Also, why don't you create a bookmarklet which shows a small popup (like instapaper or pinboard.in) for non firefox users and people who don't like installing bars? Shouldn't take much effort and it will lower a bit the barrier of entry.

<comment><AUTHOR> Thanks, you've made me realise I _really_ need to work on my wording :)

It's not targeted at advertisers per se. For example, I've been contacting bloggers about reviewing another web app that I run, and I've been finding it a real pain to keep track of the various websites. I know one guy who uses an Excel spreadsheet to track websites he's contacted about purchasing outright, and I think this could be useful for him as well.

So really - I guess I need to get some use cases on the site, right? :)

</comment></comment><comment><USER3> Also, just thought of something really cool. You could make it sort of like Google Webmaster Tools where a user can "verify" that they own the site. Then you can open it up so they can see who's maybe "tagged" their site and that allows the members to open up dialog with each other and communicate on the site.

Could turn it into a neat webmaster community where people can work together easier.

Just a thought.

</comment><comment><USER3> This is pretty handy, I can see the need for this tool for quite a few use cases. Good job, haven't seen anything like it before either.

</comment><comment><USER4> In 30 words or less, what does the site do? Does it solve a problem?

<comment><AUTHOR> _In 30 words or less, what does the site do?_ The site works in conjunction with a Firefox toolbar to save the details of websites you want to contact later on.

 _Does it solve a problem?_ Absolutely, yes! There are plenty of people I know using an Excel spreadsheet to keep track of potential contacts and for their marketing campaigns.

</comment></comment><comment><AUTHOR> Clickable: <http://campaignbar.com></comment>


President Obama: Pardon Environmental Activist Tim DeChristopher - <AUTHOR> http://www.change.org/petitions/president-obama-pardon-environmental-activist-tim-dechristopher#

<comment><USER2> Off-topic, but to sign, you must provide an address, and if you don't, the form becomes unusable. Windows 7-x64 running Chrome 18.0.1025.x</comment>


Show HN: Maildown – Super Simple Email API - <AUTHOR> https://github.com/chris104957/maildown <comment><AUTHOR> I'm planning to add support for other email backends. For me, AWS makes the most sense as it's basically free for 99.9% of the population with its 62k free monthly email limit

<comment><USER2> Do you know whether the "62k free monthly emails limit" of the free-tier is available to existing AWS accounts, or a new AWS account must be created to enjoy free-tier billing?

Thanks!

EDIT: BTW, nice project!

<comment><AUTHOR> I believe its 62k free monthly emails for anyone, regardless of how long you've had your AWS account

Source: [https://aws.amazon.com/ses/pricing/](https://aws.amazon.com/ses/pricing/)

<comment><USER3> Do you have to do your own queueing to make sure you're not sending too many emails all at once and they get marked as spam? Or does SES handle that for you? And if it doesn't, is there any service that will do this?

<comment><AUTHOR> AWS handles that stuff for you. You can use SNS to monitor stuff like deliverability

<comment><USER3> Wow! That's amazing! What's the purpose of having your own dedicated IP then?

</comment></comment></comment></comment></comment></comment><comment><USER4> It seems like the benefit is in sending markdown email from cli.

I'd actually love to be able to do this now that I think about it (I'm sure there's ways to do this already)

<comment><USER5> What is a markdown email? Is it a mail body written in markdown, using the unprocessed markdown for the text/plain part, and generating a html part from the same?

Pretty sure you could to that with pandoc and heirloom mailx or something similar.

<comment><AUTHOR> That's pretty much it. Maildown reduces this process to a one liner, however

</comment></comment></comment><comment><USER6> Hi I'm really grateful for your open sourcing this just because I get to ogle at how you write your python to interact with AWS SES

Thanks!

</comment><comment><USER7> Except for the Markdown part, this command line looks very similar to the most minimal old Unix email interface. (Anti-spam measures effectively ended that, and now we need to pay service providers.)

</comment><comment><USER8> I expected infinite recursion in the screenshot :)

<comment><AUTHOR> Sadly, since I fixed a typo in the readme it doesn't match at all

</comment></comment><comment><USER9> sadly just a wrapper around Amazon AWS SES

<comment><USER10> Why is it "sadly"? I would say that it is happily a wrapped around SES. There are use case for using mkdown in email and this tool helps address those use cases.

Good job OP!

</comment></comment><comment><USER11> I feel this unnecessarily complicates things. If you want to send emails via SES, why not just use something like [https://www.sendwithses.com](https://www.sendwithses.com) or [https://moonmail.io/](https://moonmail.io/)

<comment><AUTHOR> Neither of these services is free. Not sure how this complicates things any more - it's a one liner to send an email</comment></comment>


Charles Stross and the invention of robots.txt - <AUTHOR> http://www.antipope.org/charlie/blog-static/2009/06/how_i_got_here_in_the_end_part_3.html

<comment><AUTHOR> Just picked that one because I thought it was particularly interesting, but the other ones are pretty good, too, like the second start-up death march: [http://www.antipope.org/charlie/blog- static/2009/06/how_i_go...](http://www.antipope.org/charlie/blog- static/2009/06/how_i_got_here_in_the_end_part_4.html)

Or you can read the complete series here: [http://www.antipope.org/charlie/who_am_i/autobio-all- redacte...](http://www.antipope.org/charlie/who_am_i/autobio-all- redacted.html)

If you don't know Charles Stross, he's an SF author and he also has a hacker news account. I read one book by him, Halting State, which is about a break-in in an MMO vault among other things. It had the most real-feeling near future SF settings I've come across.

<comment><USER2> _Accelerando_ and my others are highly recommended as well.</comment></comment>


TV Series Episodes That Inspire Creativity - <AUTHOR> http://www.smashingmagazine.com/2009/08/13/10-unique-tv-series-episodes-that-inspire-creativity/

<comment><AUTHOR> There's been a few of these types of posts about movies here. Went looking and found this one with some interestingly creative tv show episodes.

Almost every episode of Mad Men also has a neat creative bit to it when you watch those folks copywrite. Any other tv shows/episodes you've enjoyed along these lines?

<comment><USER2> Deep Space 9, Season 5, Episode 6: Trials and Tribble-ations.

Sisko and Jax travel back in time to the period of the original series. Amazing visual effects and a clever plot that avoids breaking continuity with the original series.

[http://en.memory-alpha.org/wiki/Trials_and_Tribble- ations_(e...](http://en.memory-alpha.org/wiki/Trials_and_Tribble- ations_\(episode\))</comment></comment>


Wouldn't it be fun to build your own Google? - <AUTHOR> http://radar.oreilly.com/2014/12/wouldnt-it-be-fun-to-build-your-own-google.html <comment><USER2> [lightly modified version of a comment I put on the article as I love HN for discussion!]

Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article.

The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it only in an EC2 cluster. You don't even need to have an Amazon account! Our crawl archive blog posts give full details for downloading[1]. The main challenge then is storing it, as the full dataset is really quite large, but a number of universities have pulled down a significant portion onto their local clusters.

Also, we're performing the crawl once a month now. The monthly crawl archives are between 35-70 terabytes compressed. As such, we've actually crawled and stored over a quarter petabyte compressed, or 1.3 petabytes uncompressed, so far in 2014. (The archives go back to 2008.)

Comparing directly against the Internet Archive datasets is a bit like comparing apples to oranges. They store images and other types of binary content as well, whilst Common Crawl aims primarily for HTML, which compresses better. Also, the numbers used for Internet Archive were for all of the crawls they've done, and in our case the numbers were for a single month's crawl.

We're excited to see Martin use one of our crawl archives in his work -- seeing these experiments come to life the best part of working at Common Crawl! I can confirm that optimizations will help you lower that EC2 figure. We can process a fairly intensive MR job over a standard crawl archive in afternoon for about $30. Big data on a small budget is a top priority for us!

[1]: [http://blog.commoncrawl.org/2014/11/october-2014-crawl- archi...](http://blog.commoncrawl.org/2014/11/october-2014-crawl-archive- available/)

<comment><USER3> what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their human uploaded content and profiles, and majority of the data from the web being useless without a context and purpose of the searcher?

I'm just curious as to how commoncrawl compares with kimonolabs and import.io as they seem to have the same goal of creating an internet as a dataset, or an API. I can't help but feel like it's just solving another 'semantic web' problem that nobody asked for.

It is funny that the most demanding customers of semantic web are also the ones who are willing to pay the least amount of time and money.

<comment><USER4> Regarding copyright issues, I believe you can still use copyrighted data as long as it's transformed. E.g., building language models, or doing a search engine like Google. In fact, I can think of more computational uses for copyrighted data, while on the "banned" side, I can only think of... SEO.

Regarding point two: "monopoly businesses who viciously protect their human uploaded content". I spend a lot of time scraping these monopoly businesses, and it seems to me they do a decent job of letting their users decide what data is exposed. Facebook, linkedin, and google all are decent about letting me scrape their public info. That's all I have a right to -- private info should stay private, at the behest of the owner (the User in UCG).

You are correct regarding the third point, but I don't see that as a problem. This isn't a solution in search of a problem -- it's a problem without a solution at the moment.

Here's a toy example of something I'd like to do: calculate the positive / negative sentiment of commenters at particular baseball fan sites, so I can hide the content I don't like, and show that which I do. Having a common crawl of the site would be immensely useful (and is indeed a prereq) for this. I wouldn't need to republish it, just compute on it.

</comment></comment></comment><comment><USER5> At first Google was a search algorithm, but at some point they decided to have humans review and rank the important queries. Important as in query volume.

Why use humans? People can decide if your navigation is intuitive. They can decide if your page looks like crap. If 230,000 people are searching for "coconut oil" per month (actual numbers) then it's worth having an intern spend 15 minutes to make sure page 1 of "coconut oil" looks right.

Google can afford that. They need a human to decide if the "user experience" is actually good vs. disallowing the back button and forcing the browser to crash, which is how I suppose you could fake a "time on site" metric if this was just an algorithmic problem.

Google is now more like playing Zork. You type "Go North" like 10 million other people before you typed "Go North" and Google has already crafted that experience you'll find in next room. (Which makes me wonder, do they score how boring you are based on predictability?) This is becoming more and more obvious over time as a search for "calculator" shows you an actual calculator that a human at Google created. That's not an algorithmic response.

Similarly, I see that human touch coming more into play with voice recognition, Google Glass, Siri, etc. Call that "AI" or whatever. You ask Google a question and Google has already sculpted a slick answer based on tons of testing. That's how I see Google as a search engine now. Part of the crawling is interesting (recognizing objects in photos?) but I think human reviews of all the important websites and SERPs, that's harder for a competitor to reproduce.

<comment><USER6> _I think human reviews of all the important websites and SERPs, that 's harder for a competitor to reproduce._

Google was forced into that by improved "search engine optimization". SEO used to be about things like keyword stuffing, but as Google made their search engine smarter, SEO companies made their search spamming smarter. There are now SEO operations using machine learning to reverse engineer Google's algorithms and then automatically spam just enough to stay under the threshold.

In 2010, Google tried using "local" data to improve search. That turned out to be extremely easy to spam. A classic example of this can be found by searching for "laptop repair bradford pa". This brings up "Illusory Laptop Repair", located in the middle of a railroad crossing. A SEO expert created that phony business listing to demonstrate how bad Google was at detecting such spam. Google still thinks it's real.

In 2012, Google tried using "social" data to improve search. That worked even worse. Fake Google accounts created to create fake "+1"s may have exceeded the number of real ones. Google "+1" are still for sale; the going rate is about $0.10 each.

Meanwhile, links aren't as useful as they used to be. Who creates a link to a retail outlet other than on social media any more? Google is trying all sorts of "signals", but in heavily spammed areas, they're not doing all that well.

Yandex has been trying search that doesn't weight links at all for some heavily spammed categories in the Moscow area. It seems to be working for fake real estate ads.

(We have a partial solution - find the real-world business behind the web site and check it out in hard data sources, such as Dun and Bradstreet or Experian, which have business credit data. See "sitetruth.com/doc".)

<comment><USER7> >A classic example of this can be found by searching for "laptop repair bradford pa". This brings up "Illusory Laptop Repair", located in the middle of a railroad crossing. A SEO expert created that phony business listing to demonstrate how bad Google was at detecting such spam. Google still thinks it's real.

This doesn't seem to work.

<comment><USER8> It does work on Google Maps, I find, but not on the main search page.

This is what I found: [https://maps.google.com/maps?q=laptop+repair+bradford+pa&hl=...](https://maps.google.com/maps?q=laptop+repair+bradford+pa&hl=en&ll=41.955948,-78.643141&spn=0.0031,0.004823&sll=37.0625,-95.677068&sspn=53.477264,79.013672&hq=laptop+repair&hnear=Bradford,+McKean+County,+Pennsylvania&t=m&fll=41.955948,-78.643141&fspn=0.0031,0.004823&z=18)

<comment><USER6> If you're logged into Google, your results may vary.

</comment></comment></comment></comment></comment><comment><USER9> A really nice idea.

I volunteered a bit early this year for Common Crawl (not much, just some Java and Clojure examples for fetching and using the new archive format).

Common Crawl already has many volunteers (and a professional management and technical staff) so it would seem like a good idea to merge some of the author's goals with the existing Common Crawl organization. Perhaps more frequent Common Crawl web fetches and also making the data available on Azure, Google Cloud, etc. would satisfy the author's desire to have more immediacy and have the data available from multiple sources.

<comment><USER9> some edits:

Most of the Common Crawl data is on Microsoft Azure, but not all of it.

The Common Crawl is a great resource that deserves attention from more companies and developers.

</comment></comment><comment><USER10> I've always wanted to experiment with my own search algorithm. Unfortunately, I think this is still out of the budget of average programmers. Just the hard drives to download 1.3 petabytes would cost six-figures.[1][2]

[1] [https://www.backblaze.com/petabytes-on-a-budget-how-to- build...](https://www.backblaze.com/petabytes-on-a-budget-how-to-build-cheap- cloud-storage.html)

[2] [https://www.backblaze.com/blog/why-now-is-the-time-for- backb...](https://www.backblaze.com/blog/why-now-is-the-time-for-backblaze-to- build-a-270-tb-storage-pod/)

</comment><comment><USER11> A couple thoughts:

1) I like the idea of human curation, but in combination with some sort of automated crawler (or other tool) that helps in the browser.

2) Why can't we also distribute the act of crawling, the maintenance of the index and the map-reduce (or other algorithm) that produces the data.

I've been thinking about architectures that would allow (in essence) a P2P search system. Would anyone be interested in talking about architectures to make this work? There are millions of computers on the web at any given time ... if it's built into the browser (or plugs in), you could have human input at the same time.

</comment><comment><USER12> Yeah, this sounds all well and good in theory, but after visiting thousands of sites over the years, it might be a better idea to help engineers build a search engine for their own site/data first. I can't recall many websites that have amazing search. It's a problem when I have to use google to find what I want on xyz.com because if I go search for what I am looking for on xyz.com I cannot find it even if I know its on that site.

It would be so nice to go to xyz.com and actually find what I am looking for in under 1 second.

<comment><USER13> I'm pretty sure Elastic is on the right direction in this regard, don't you think?

<comment><USER12> I don't think so. I just went to their site and the first case study i opened was from theguardian.com. I went to theguardian.com and went to do a search. Guess who they are using to power their search function? Google.

In my opinion which means nothing, sites need to figure out how to power their own search. Using a third party isn't going to work for most. Maybe people need to focus on building custom architecture that indexes the data in a more structured way rather than cobbling systems together that ultimately hinder search efforts when its time to get the user what they want. I don't know the answer but somebody eventually will. Maybe wordpress will create a powerful search for all those wordpress sites.

<comment><USER13> Maybe I'm beating a dead horse but I feel like you could create a pretty compelling search engine using ES. I used it this summer at Goldman and it seems like it will really change the landscape... It's just so fast. I mean full-text-indexing seems to be a pretty integral part of search. Maybe it's just the first step, with the second step being a well written ranking function, but that's just my thought.

</comment></comment></comment></comment><comment><USER14> For anyone interested, there's a hilariously bitter and practical paper on the trials and tribulations of building a search engine:

[http://queue.acm.org/detail.cfm?id=988407](http://queue.acm.org/detail.cfm?id=988407)

EDIT:

Article is clearly from an earlier era, but it's really cool to see how far we've come and how much more computing power we have available now. There are entire categories of problems that simply don't _exist_ anymore.

<comment><USER6> She later designed the search engine of Cuil. While Cuil failed, it only cost them about $30 million to do most of what Google does.

It's surprising to me that there aren't search engines from Comcast, AT&T, and Apple. If you have customers, why give up all that ad revenue to Google? Google is paying some big players a lot of money not to do that. They were paying Apple $1 billion a year to be the default on Apple products. Apple switched from Google to Bing anyway.

<comment><USER15> _While Cuil failed, it only cost them about $30 million to do most of what Google does._

They raised ~$30 million in two rounds, but their _valuation_ was at $200 million by round two. I agree with your point though; the cost to develop a good search engine is dirt cheap compared to the value it brings.

<comment><USER6> "Valuation" by whom? They had no revenue, no revenue model, no VC would give them additional funding, and Google didn't buy them out. On September 17, 2010 at 1 PM, all the employees were told the company was shutting down.

Google did hire the CEO and Anna Patterson to keep them from doing another search engine.

</comment></comment></comment></comment><comment><USER16> This would be really cool to participate in, especially if it could be packaged à la Folding@Home/SETI@Home and widely distributed. I wonder if there's some clever method using crypto that can provably discourage bad actors if the network has certain properties (e.g. Bitcoin is nearly impossible to cheat unless one group owns >50% of the network).

</comment><comment><USER17> Google's power comes not from the crawling, but from the retrieval and ranking. They use many more signals than the hyperlinks and anchor text (which is all you'd have if you crawled yourself). Indexing crawled content would have been OK in the year 2000; but today, the users demand more. Relevance is the top priority, and no one does it better than El Goog.

<comment><USER18> Sorry but Google's ranking algorithm for me is far from brilliant.

To give you an example, search for "webhcat primary key" (without quotes) and note how the top three search results do not actually contain the term webhcat. Google constantly does this. It randomly ignores search terms unless you explicitly quote them.

I believe that there is still a market for a technical/advanced search engine.

<comment><USER4> Isn't google doing that because it detected the semantic information was on the page, even if the exact term wasn't? Is your issue with the fact that they're doing more than just a keyword retrieval, or is your issue with the fact that they're doing it poorly?

<comment><USER18> Isn't my issue obvious ? I wanted search results that contained the search terms. Otherwise I wouldn't have entered them in the first place.

I understand Google is trying to be clever here and appealing to novices who don't really understand what they want.

But my point is that for those of us that do it is an incredibly annoying "feature". Feature is in quotes because in the specific case above they didn't find semantic equivalents. They just dropped the "webhcat" term entirely.

<comment><USER4> >Isn't my issue obvious ?

Not touching that with a 10 foot pole... (just teasing, just teasing...)

But seriously, it sounds like what you want is a keyword matching engine. Google, for better or worse, has decided they know enough about their user's searches that they don't mind modifying the query parameter in an attempt to retrieve what people want rather than what they literally say they want.

I understand that you don't feel they're serving your needs any longer, and that can be frustrating. I think, however, that you can preceded mandatory terms with a + sign to require it to be present on page.

<comment><USER19> After Google Plus was released, Google changed its search syntax slightly so that you now need to "quote" mandatory terms instead of prefixing them with a + symbol.

[http://waxy.org/2011/10/google_kills_its_other_plus/](http://waxy.org/2011/10/google_kills_its_other_plus/) [https://productforums.google.com/forum/#!topic/websearch/3oI...](https://productforums.google.com/forum/#!topic/websearch/3oIWbew9xdE%5B1-25-false%5D)

<comment><USER20> This is really weird, because now I have to enter this:

"singer" "song" -inurl:(htm|html|php) intitle:"index of" "last modified" "parent directory" "description" "size" "m4a"|"mp3"|"ogg"|"flac"

to search for songs. _Too many quotation marks!_

original, unmodified: [http://lifehacker.com/207672/turn-google-into-your-own- perso...](http://lifehacker.com/207672/turn-google-into-your-own-personal- free-napster)

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> Maybe more people should start crawling and seeing what they can extract? I remember seeing DuckDuckGo Instant Answers and thinking what a valuable resource that would be (having a database like DDG must have, I mean).

Then one would be able to do some "stuff Google can do" \- say, analysing trends - albeit with worse sampling, and not depend that much on them.

</comment><comment><USER22> The problem with algorithmic/scraper search methods, is that they only work with existing data. For example, most Google searches gives a list of websites on one side, and some data scraped from Wikipedia on the other. There is not much meaning there. That's because Google's algorithm cannot combine the results into something original, because that would require human creativity. As such, I see the rise of different kinds of search based on what humans create, rather than what computers can scrape. I wrote a (longish) blog post on this problem: [http://newslines.org/blog/googles-black- hole/](http://newslines.org/blog/googles-black-hole/)

<comment><USER23> You really discount the value of the long tail of search, which is where you get best info from google.

<comment><USER22> You may get the best (more precise) info in the long tail, but I'm pretty sure that Google makes most of its money from the most popular searches.

</comment></comment></comment><comment><USER24> suprised not to see a mention of a bloom filter in url dedupe. Another tough problem now is the portion of the web in walled gardens or that is expensive to crawl (needs a js context).

</comment><comment><USER25> I thought Wikimedia tried this once. Big announcement, then nothing. Is that code still available?

<comment><USER26> That was Wikia, not Wikimedia, but yes, the code is still available: \- crawler: [http://sourceforge.net/projects/grub/](http://sourceforge.net/projects/grub/) \- search engine: [http://nutch.apache.org/](http://nutch.apache.org/)

<comment><USER27> If my memory serves me correctly so is it only the client part of grub that is open source. Without the server part on cannot use it to setup one's own crawl.

</comment></comment></comment><comment><USER28> Hmmm I'd think that ChuckMcM would have some interesting views about this.

</comment><comment><USER29> what about Algolia? HN uses it

</comment><comment><USER30> Good Read</comment>


A VC: The Community Conundrum - <AUTHOR> http://avc.blogs.com/a_vc/2007/05/the_community_c.html <comment><AUTHOR> I really like the way he sums it up: _When you get in the way of geeks sharing hacks with each other in a geek community, you've done something big._</comment>