The American Economy Is in a Funk, But Not for the Reasons We Think - <AUTHOR> http://freakonomics.com/podcast/american-growth/ <comment><USER2> US GDP per capita is larger then it has ever been. We are richer then we've ever been and this economist predicts we'll be richer still for the foreseeable future. The problem is that 30 years ago or so the middle and lower classes stopped receiving any of the gains.

<comment><USER3> Healthcare providers and insurers have received much of the gains, it's not just a class issue. Since 2001, employers' benefit costs have risen 60%, which is almost twice the rate salary increased. Perhaps salary increases would've outpaced inflation, leading to the real wage gains everyone's missing, if that money weren't being diverted into the employer's share of health insurance premiums.

<comment><USER4> The government's share of GDP has increased substantially.

[http://www.usgovernmentspending.com/us_20th_century_chart.ht...](http://www.usgovernmentspending.com/us_20th_century_chart.html)

<comment><USER5> In perspective its really not that odd.

[https://research.stlouisfed.org/fred2/series/FYONGDA188S](https://research.stlouisfed.org/fred2/series/FYONGDA188S)

1) 2015 is 20.55%. 1980 is 20.64%.

2) Recessions increase spikes as a share of GDP, really bad recessions have bigger spikes.

<comment><USER4> Scroll down to chart S.03t. It shows a consistent trend upwards. The chart shows total government spending, not just federal.

<comment><USER5> The point is, I can't fix the other 49 states with magical powers.

</comment></comment></comment></comment></comment></comment><comment><USER6> I've just started the book this guy wrote. It's really good so far and is very readable. It's not like the epic and unreadable Picketty.

Note that he is saying that the growth were getting will be the same as levels from 1820 to 1870, just not the same as it was from 1870 to 1970.

The book is worth reading because he has loads of data, not just Internet comment opinions about how things are going. Even if he is wrong and productivity shoots up again he's shown fairly strongly that between 1970 and 2016 growth has slowed.

The consequences are pretty serious for the welfare state. Either benefits cannot continue to grow or taxation must increase.

<comment><USER7> Per capita GDP has been going up far faster than entitlement spending per capita. The top have been getting most economic gains and fewer taxes which is the core problem.

We could talk about long term demographics, but GDP growth is loosely tied with workforce size.

PS: A lot of systemic corruption is also going on. Almost half of the lifetime gains from a 401K ends up as management and other fees, that was clearly designed to make someone money, but not working class Americans.

<comment><USER6> Hauser's law, that states that federal tax revenues since World War II have always been approximately equal to 19.5% of GDP, would suggest that tax levels are similar as a percent of per capita GDP.

[https://en.wikipedia.org/wiki/Hauser%27s_law](https://en.wikipedia.org/wiki/Hauser%27s_law)

<comment><USER7> U.S. gov spending is 20,000$ per person which takes vastly more than 19% GDP. [http://www.usgovernmentspending.com/per_capita](http://www.usgovernmentspending.com/per_capita)

Local, State, and Federal = ~32 to 42% GDP over last 30 years. [http://www.usgovernmentspending.com/spending_brief.php?brief...](http://www.usgovernmentspending.com/spending_brief.php?brief=recent). Defense for example is all over the map, 7% under Reagan down to 3.45% with Clintion then bumped under Bush to amost 6% and then back down under Obama.

PS: US tax code is designed around talking points so people can support all sorts of things. But at the end of the day Total government spending is what's important not what you call it.

<comment><USER8> Thanks for editing to point out that is Local+State+Federal. sien's point and "Hauser's law" is about Federal revenue and specifically how Federal tax revenue doesn't seem to respond to policy changes in the marginal Federal income tax rates as you might expect. They are both good metrics to consider at different times, for example sometimes you ignore certain classes of taxes and spending (like local/property taxes which are largely for K-12 education) when you want to focus on welfare spending, which is 95%+ Federal.

<comment><USER7> [https://research.stlouisfed.org/fred2/series/FYONGDA188S](https://research.stlouisfed.org/fred2/series/FYONGDA188S) Even then federal outlays go from 10 to 25% post WW2. There is little support for a law. People love to pretend marginal rates are the hole story while ignoring tax breaks, social security, gas taxes, etc. it's a political tug of war and positions hate to make things clear cut.

Ex: hand outs in the form of very targeted tax breaks are rarely considered spending even as they subsidize industries.

</comment></comment></comment></comment></comment></comment><comment><USER9> This article may (or may not:) be a little short sighted.

Basically it claims the inventions of what it terms "The third industrial revolution" (computers) affect a much narrower slice of life than the proceeding industrial revolutions. So they won't produce the growth the proceeding revolutions did and the rapid growth is over.

But earlier in the article:

"GORDON: When we had the invention of steam engines, steam ships, locomotives, factories making cotton fabrics, and then the telegraph. All of those things were invented in the century between 1770 and 1870. And they set the stage for the inventions that happened after 1870. "

So there was a lag between the large number of invention and the true realization of potential in subsequent inventions and thus improvements in life.

First of all I don't think digital technology does affect a narrower slice of life when looked at in broader terms. I think it affects an even greater slice of life. True we have a bunch of cat videos and Facebook but that's just the surface.

But the main point, going back to the lag, I believe that is where we are now. We have the technology and are really just getting started with what it can do. Digital technology hasn't really infiltrated everywhere it can and many things are still being done in inefficient ways. When the subsequent set of innovations begin to appear I think we will see an even greater increase in standard of living than the previous industrial revolutions. More efficient distribution mechanisms. Better control over biology (including our own). Automation and robotics. Things like that. Social structures need to catch up though also... likely part of the reason for the lag.

<comment><USER10> Yes! To a large extent the visible nature of digital technology is a reflection of the capability of people to take advantage of it.

If we never progressed past the Commodore 64, we could still be discovering advantages and uses for it. As it is, new tech is coming our way far faster than we can effectively apply it.

Or, to rephrase, a lot of office work could be completely replaced by a basic excel spreadsheet macro, let alone machine learning and the latest neural networks.

</comment></comment><comment><USER11> _Before the telegraph, which was invented in 1844, the fastest that news could travel was by the speed of a horse or a sailing ship._

Not strictly true, the semaphore telegraph predated the electric telegraph by about 50 years.

<comment><USER12> And very rudimentary message passing was made using signal fires.

There's smoke signals and messenger birds (eg pigeons) but I don't know whether/when/how they were used historically.

Edit: Just thought, maybe signal mirrors too?

<comment> nommm-nommm Pigeons were used to carry important military messages during war up until 1957.

[https://en.wikipedia.org/wiki/War_pigeon](https://en.wikipedia.org/wiki/War_pigeon)

[https://en.wikipedia.org/wiki/United_States_Army_Pigeon_Serv...](https://en.wikipedia.org/wiki/United_States_Army_Pigeon_Service)

<comment><USER13> Let's not forget RFC 1149 specifying "IP over Avian Carriers"

[https://www.ietf.org/rfc/rfc1149.txt](https://www.ietf.org/rfc/rfc1149.txt)

</comment></comment></comment></comment><comment><USER14> >As compared to me, where I think that new technologies are coming, but they're occurring at an amazingly slow pace.

This is problem with economists analyzing technological growth in the economy. They think there must a new google or iphone to come out every 10 years.

20 years ago very few people have ever sent an e-mail. Now I can send an e-mail/call anybody from any place on the earth. We are making significant technological progress and it's where the most growth is at the moment.

<comment><USER15> I don't think "Google or iphones" are the type of technology he is referring to.

He's thinking more plows, or cotton gins, or fiber optics

<comment><USER16> Those kinds of technologies are coming at a faster pace too. But the fact you discount search engines and cell phones kind of shows how fast new technologies are moving. They're on the same scale as cotton gins but we don't have decades to marvel at how they were gamechangers because they seem like nothing a couple years after they are invented and there is so much more going on.

</comment></comment></comment><comment><USER17> Who's reading freakonomics?? I don't think they (freakonomics) ever corrected their horrendous mistakes with respect to climate change.

[http://www.theguardian.com/environment/climate- consensus-97-...](http://www.theguardian.com/environment/climate- consensus-97-per-cent/2013/jul/08/climate-change-superfreakonomics- superfreakingwrong)

[http://scienceblogs.com/deltoid/2009/10/16/why-everything- in...](http://scienceblogs.com/deltoid/2009/10/16/why-everything-in- superfreakon/)

Embarrassing people! Here they serve up some more gossip ...

<comment><USER18> I've always kinda hated "Freakonomics" \- albeit irrationally. It just never felt like the author truly understood the difference between correlation & causation.

<comment><USER19> I too grew a distaste for them, more accurately the Freakonomics Podcast. Steven Levitt comes across like a total snob and he has all these annoying quirks and preferences, I find his personality really irritating. He reminds me of someone who won't do manual labour and thinks it's funny and charming to be someone who sees anything physically strainful as being this horrendous undertaking. He really embodies so many detestable human qualities, and that's the thing, it's like his 'image', he's this weirdo guy who gets to be reprehensible because he's a notable economist. Also their Podcasts frequently air re-runs, and their shows consist of 1 revelatory finding (i.e. legalizing abortion reduces crime rate) that they drag out for 30 mins until you get to the 'gem' of their findings. That being said these are personal gripes from previously being an avid listener who got tired of their stuff.

</comment></comment></comment><comment><USER20> I suspect we're on the verge of pretty big advances in genetic engineering, robotics and other physical technologies, including technologies that are likely to spur the development of other technologies. Genetic engineering of embryos for increased IQ is looking very feasible, for example. This is likely to increase the pace of innovation drastically in those countries that allow it.

<comment><USER21> > This is likely to increase the pace of innovation drastically in those > countries that allow it.

And create a divide between the rich who can afford it for their children, and the poor who can't.

<comment><USER20> There is already a huge wealth and status divide between those born with high IQs and those born with normal IQs.

The incentives are such that it's going to happen. Also, it looks like very- large increases in IQ will be possible once we have millions of genomes tagged with their donor's IQ and a reliable means of editing thousands of alleles. More than a few standard deviations. I would certainly use this were it available today. If a rich person has to fly to Korea to do it, so what? Any country that bans it will fall behind.

The gains from this may be such that governments will think it worth subsidizing it for the poor's children. High IQ people commit less crimes, pay more taxes, tend not to use welfare. Liberals and conservatives will likely both converge on such subsidization.

<comment><USER12> >a huge wealth and status divide between those born with high IQs and those born with normal IQs //

Are you saying that's the primary reason for the status divide? Any citations that support that, the most recent research I've seem looked at societal status/wealth of families a few generations back and then traced living descendants to find how stratified things were. They found a high degree of stratification; that across generations it was far less likely that poor ancestors had wealthy descendants. Perhaps IQ goes this way too?

>High IQ people commit less crimes //

Or get caught less often. Wealthy people (in the UK) often commit some common crimes at what seems like a far larger rate because the penalty is usually just a fine and to the wealthy it's pocket-change [speeding, parking offences, tax evasion ...].

<comment><USER22> You seriously think that high IQ people are committing as much crime as "whoever else", but get caught less often? Seriously?

Obviously a high IQ person on average will have a better job/more money and therefore have less reason to commit crimes.

You think average/below average IQ people are scared to jaywalk, illegally park because they're scared of the fine?

Does your ideology really cloud your reasoning to that extent?

<comment><USER9> There are all kinds of criminals. Hi IQ adults probably don't (generally) shoplift $3 worth of beauty products nor hold up gas stations for $56.

Bernie Madoff was probably pretty smart though don't you think?

And there are no doubt plenty of others who are too smart to get caught, and if caught not prosecuted, and if prosecuted not convicted. I'm not convinced hi IQ people commit that much less crime. Just that they probably commit less petty crime.

<comment><USER8> 'Crime' is a ridiculous term, because the argument reduces to everyone is a criminal to some degree. But I think it's well established that there is a negative correlation between IQ and crime, of course not just using raw conviction rates, but after attempting to correct for factors like arrest/conviction rates.

mikerichard's point about less 'reason' to commit crime I think is a good one. A lot of crime stems from desperation.

Now I don't know about trying to create some kind of Bentham inspired calculus to put criminality on a weighted scale versus IQ and see if it all balances out.... The most terrible outliers are often economic or political leaders, by definition, their power in society gave them opportunity to commit "worse" crimes.

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> Most evolutionary experts, like the late Stephen Gould, espouse Punctuated Equilibrium. Basically the theory says that environments are stable for long periods of time and then there are bursts of changes in small periods of time. The Cambrian Explosion is a good example (read Gould's book on the Cambrian, it's awesome).

If you consider technology to be like evolution, which it surely is, then infrequent advances in short periods is only natural.

<comment><USER24> >>If you consider technology to be like evolution, which it surely is, then infrequent advances in short periods is only natural.

What you are describing is basically Thomas Kuhn's notion of the nature of scientific revolutions.

[https://en.m.wikipedia.org/wiki/The_Structure_of_Scientific_...](https://en.m.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions)

</comment></comment><comment><USER25> My personal bet for current 1st world lack of rapid economic growth is simply lack of demand. Not absolute lack, but people in the rich countries aren't hungry, cold, physically exhausted or sick. They aren't even close, most people have some money buffer between then and hunger, called "expendable income".

At first you would think that the expendable income would breed economic growth like mad. But what exactly are you buying with your money if you are generally ~happy about your life and your stuff? Four wheeler? Snake oil? Snowboard? One off things that you know are overpriced, you won't buy another one soon, and you might indefinitely postpone that purchase if you don't feel like it.

Your purchase decision is more and more likely to be based on advertising or status than price. So ROI of advertisement is better than ROI of improving the production lines. Especially when consumption of luxury goods is so fickle. This is not really a recipe for long term exponential growth.

Compare this to Peru, Cambodia or Chad. Everybody lacks money, but everybody would have several dire needs for some capital. You have reliable demand and really top notch _consumption_ _efficiency_ from the customer side.

</comment><comment><USER26> >It sounds as though your argument is essentially that many of the inventions provided by the Digital Revolution, while exciting and appealing and sexy — and maybe, because they are so sexy, they gain more of our attention than they warrant — that as exciting and sexy, etc., as they are... they don't stand at all in comparison to the breakthroughs of the earlier technological revolution.

This is the perspective of an outsider looking into the world of tech. As someone on the inside, I would argue that those benefits simply haven't arrived yet.

I just left a start up who business model was automating a large (Multi- billion $), ancient industrial vertical and who's only technological improvement since the invention of the telephone was using excel rather than paper ledgers.

</comment><comment><USER27> There is the top 1% that own and control 90% of the worlds wealth, then the "upper" classes that are doing very well thankyou and think they are the normal "if everyone just worked hard like me", then there is the rest.. the slave class

If the upper class (used to include an upper middle and middle class) was larger, and was the majority of the voting class and the top was no less than 2%, the balance of slavery can be contained and controlled and the fiction of a fair society can be perpetuated.

The reality is the top 1% exist in their own economy and the rest of us exist in our own, so when one of us makes a bucket load we are really only ripping off our fellow man.

When the super rich throw out more "free" cash eventually they reel it back in again to suck up more wealth from our economy.

It is important to remember the US reserve is a private bank of the top 1% that is NOT controlled by elected officials, and debt is the trojan horse.

</comment><comment><USER28> Well power must have something to do with it. The 2nd industrial revolution made essentially unlimited chemical power available to many people.

Next revolution? Unlimited nuclear power available to many people...</comment>


DuckDuckGo adds zero-click recipe search with the Punchfork Recipe API - <AUTHOR> http://blog.punchfork.com/2011/07/05/announcing-the-punchfork-recipe-api-and-a-search-partnership-with-duckduckgo/ <comment><USER2> My wife and I have a small site that serves up hamburger casserole recipes ( <http://hamburger-casserole-recipes.com/> )

She was getting over 15K visits per month -- the site made almost nothing, but we enjoyed creating it together, watching the numbers and responding to emails. And the numbers kept climbing -- at least until a couple of weeks ago.

I noticed the traffic numbers starting to drop. I was wondering why. I thought about digging into it but put it off. Perhaps this new feature at DDG did it? If so, fine with me. They doing a much more awesome job than we did.

The only reason I mention it is because this is the type of question that if you knew enough, you could find the answers in SEO-land. But for a little mom- and-pop site, lots of times you don't have that luxury. You're hot for a few months then suddenly it all dies off and you never know why. You could be adding the best content you can and still all the visitors disappear. No skin off of my back in this particular case, but this has to be frustrating for lots of folks -- especially if your site is a startup instead of something silly like recipes.

Way cool UI! I think we'll add it -- looks like it might be a nice fit.

<comment><USER3> Sounds like no analytics if you're relying on hits purely. It might not be too late to check how users find the website. Look at trends.google.com, alexa.com, etc for related searches. Might also be a good idea to start webmaster accounts on search engines (Google and Bing at least I know of).

The website name is also quite a handful to remember ;)

<comment><USER2> When we put it together, if you searched for "hamburger casserole recipes" all you got were sites that tried to trick you into signing up for recurring cell phone charges or stuff like that. She thought it would be nice to make a plain site that helped folks (our idea of "plain" was not so good back then!) We figured using our target phrase as the domain name would at least tell people what we were about.

We make maybe 20 bucks a month from the site, and domain renewal is something like 140/year for all the major TLDs, so it's not a high priority right now. The only reason I shared this to try to point out how easy it is to lose traffic and not know why.

If I wanted to chase it, I'd go to SEOMoz and check out the backlinks and competition -- did anything change over the past month? Are other sites getting a bunch of links for some reason?

Thanks for the great tips! We have a Webmaster account on Google. Didn't know there were options also on Bing. Looking at Google Analytics today, we're still at 13K visitors for the previous month, which is about a 20% decline. It's all still search engine traffic, mostly from Google and mostly for "tater tot casserole" Go figure. Is tater tot casserole so popular? Why? These kinds of questions drive you crazy, because many times when you make a site for your startup or topic -- especially if it has a lot of pages, you get all this data from web analytics and it's a bitch trying to make some kind of meaning out of it.

Folks are still spending on average more than a minute on the site, which means they are taking time to read the recipes and get some value. That's all that counts for us.

BTW, if any of you startup guys want to go into recipes, good luck. She has had this site, with lots of traffic, for a couple of years now, and hell if we can figure out how to monetize it. We did books, kitchenware, magazines, AdSense -- finally writing our own ebook. Right now we're thinking about coupons or some other giveaway product, but I don't have my hopes up. Recipes, at least to me, looks like an income-free zone. One of the reasons we did the ebook was to provide a totally ad-free place to keep track of recipes. We're operating under the principle that people who are cooking hamburger casseroles probably aren't needing anything else at all, at least at the moment they're online.

Still, we did it to make something people want that can scale, so no matter how it turns out, we helped people (as evidenced by our emails and traffic stats) and we can learn something from it.

<comment><USER4> Domain renewal is something like $140 per year !?

I hope you mean web hosting.

//edit: Ah you mean for the .com + .org + .net etc. Still can't see this being 140 but makes more sense

</comment></comment></comment></comment><comment><USER5> My favorite part of punchfork (besides Jeff) are the custom url shorteners. Check out the customize dropdown on the side of a recipe page: <http://punchfork.com/recipe/Bramble-Chow>

Nice little touch.

</comment><comment><USER6> Sorry unrelated, but just realized DuckDuckGo has hash bang syntax for hackernews!

!hackernews which leads to a hnsearch.com search results!

I've been using it as my default search engine for about 2/3 weeks now and its been unnoticeable (in a good way)

<comment><USER7> I am amazed that so few people have ever heard of YubNub[1]. It's a service that provides this kind of feature (`g foo' to google, `gim bar' for google images, `am baz' for amazon, etc.), plus some advanced tricks (multiple parameters, default values, command combination).

Anybody can create commands.

It's a hidden gem, and I love it.

[1]: <http://yubnub.org>

</comment></comment><comment><USER8> I just thought: DuckDuckGo should get a shorter domain name because it takes to long to type it. It thought of duck.com so I checked what's there, and to my surprise it redirected to... google! Looks like someone there also thought of this.

Or has "duck" another (search related) meaning in english that I'm not aware of?

<comment><USER9> We have <http://ddg.gg/>

As for duck.com, see <https://duck.co/topic/duck-com-redirects-to-google>

<comment><USER10> Wow. I wonder how they justify that as a non-evil action?

</comment></comment></comment><comment><USER11> A great tool for recipe ideas is to enter two ingredients and see what comes up: <http://duckduckgo.com/?q=chickpea+potato>

</comment><comment><USER12> Jeff doesn't have it in his HN bio, but Punchfork is his (awesome) startup

<comment><USER7> who's jeff by the way?

<comment><AUTHOR> me

</comment></comment></comment><comment><USER13> Ironically, the search has a little bug. There's no where to do a two word phrase search, so if i do 'fried chicken' or "fried chicken" I get chicken fried steak as the number one result. Seems like the refinements show that it also searched for the word individually.

<comment><AUTHOR> "fried chicken" as a single term wasn't in my db, but I'll add it shortly. Thanks for letting me know.

</comment></comment><comment><USER14> I also like how the API directs you to the source of the recipe.

Click on one of the recipes in the Punchfork results:

<http://duckduckgo.com/?q=mojito+recipe>

Jeff is doing an amazing job with Punchfork and this is a great addition.

</comment><comment><USER15> I wish I had your UI skills Jeff!... Been using PF for a while now and it's great.

</comment><comment><USER16> Gabriel, if you are reading this, I am waiting on the duckduckgo stickers with a fork stuck in the bill in response to this. Gota put it on my hot rod...err... dinky netbook.

<comment><USER9> We sent out a batch yesterday and do it about once a month. If you asked before a month ago then they should have arrived already! If not, they may have gotten returned (we get returns). You could send the address again.

</comment></comment><comment><USER17> Cool update, Gabriel. Are you still working solo on DDG?

<comment><USER9> No: <https://twitter.com/#!/duckduckgo/team/members>. I'm still the only one full-time, but I think those days are dated. There is ample opportunity to get involved if anyone is interested, including plenty of stuff that can be open sourced.

</comment></comment><comment><USER18> Great API docs...some of the big boys should take note!

</comment><comment><USER19> How do you get around the legality of showing other websites' recipes on yours?

<comment><AUTHOR> I don't show the other websites' full recipes on Punchfork. Only the ingredients.

</comment></comment><comment><USER20> Congrats Jeff, this looks awesome.</comment>


Travel Writer Booted Off United Flight for Taking Picture of His Seat - <AUTHOR> http://upgrd.com/matthew/thrown-off-a-united-airlines-flight-for-taking-pictures.html

<comment><USER2> >"I want you to understand why I was taking pictures. _I hope you didn't think I was a terrorist._ Here is my business card [offering her one]. I write about United Airlines on an almost-daily basis and the folks at United in Chicago are even aware of my blog."

(emphasis added)

Reacting to a command given by a flight attendant with anything along the lines of "Whoa, sorry. I hope you don't think I'm a terrorist," is a terrible decision. Even if you're not, saying it will immediately make them wonder if you are.

Don't think of a pink elephant. How's that mental image of a pink elephant looking?

Terrorist. Bomb. Threat. 9/11. These are things you don't go saying on airplanes, these days, unless you're expecting some sort of discomfort from anyone within earshot. This is especially true for someone in charge of ensuring the safety of people aboard the flight.

I don't think the traveler would have been removed from the plane if they would have just complied and not mentioned such a charged word.

<comment><USER3> I mean, this is the problem right here. You're claiming that because a plane ran into a building 12 years ago, it actually makes sense for an American citizen to kick another American citizen off a plane while lying about it. How about we get back to being rational sensible human beings who understand context and can have a thought process beyond "If I hear the word terrorist, that person is a terrorist and I need to call homeland security" /roboticthinking

I get that it is a bad idea to say terrorist on a plane, but if after 12 years, we still are unable to understand context here, people should be getting fired, especially if they need to lie to justify their bizarre, made up fears.

<comment><USER2> >You're claiming that because a plane ran into a building 12 years ago, it actually makes sense for an American citizen to kick another American citizen off a plane while lying about it.

No, I'm not. I'm claiming that there are words that shouldn't be said on a plane anymore. A building exploding didn't do that. Decisions from organizations and individuals created the catalyst for that change. But, that's a completely different discussion.

It's common knowledge whether it's okay or not. I agree: we as a society, should strive to be more open and accepting about using words. But, unfortunately, we're not all there yet.

The title is sensationalist and misleading. The photo taking clearly wasn't the problem because the author wasn't removed from the plane after taking the photo, they were removed after they said something they shouldn't have.

"Travel Author Irritated After Being Kicked Off a Plane for Claiming 'Not a Terrorist'" is hardly a story, is it?

<comment><USER4> Except that, according to the author, he was accused of refusing to stop taking pictures. Giving him the benefit of the doubt, it seems reasonable for him to critise the airline based on their words, even if he suspects their words are not truthful.

<comment><USER3> Not even just according to the author. According to the official reason the airline gave him. I can certainly understand why one might suspect this isn't the real reason, but calling the guy sensationalist for repeating EXACTLY what he was told is pretty insane imo.

</comment></comment></comment></comment></comment><comment><USER3> The fact that we got attacked by a few Saudi nationals 12 years ago makes one American lie to another American because shes scared an American passenger on their plane is a terrorist because he said the word terrorist. Is it really debatable whether we did exactly what the terrorists hoped we would do at this point? I don't want to say they won, because its not a game and no one wins, but they certainly accomplished some objectives here if this is the way people are allowed to act.

<comment><USER5> Or in this case apparently it turns flying attendants into power tripping liars.

</comment></comment><comment><USER6> Same article 16 hours and 200+ comments ago

<http://news.ycombinator.com/item?id=5256051>

</comment><comment><USER7> I give this blog post Autism/10

Flight attendant's perspective was that she is routinely telling passengers to quit with the happy snaps, as required by company policy. Flight attendant is tired and very busy with getting the plane together for takeoff. The photo guy from before is suddenly angrily motioning her over and saying a bunch of crazy shit about not being a terrorist, and trying to make her take/keep his business card. flight attendant freaks out and runs to her boss.

Crew boss sees really freaked out flight attendant and assumes the passenger needs to go. Once that decision is made, crazy travel guy probably doesn't have any hope of reversing it.

The real issue is that the guy grossly messed up his interaction with the flight attendant and paid the price for it. If you misbehave on a flight you're gonna get kicked off - yes, airlines have a duty to have fair rules for customers to follow. But passengers have a duty to act relatively normally in their standard human interactions with staff.

Trying to air one's grievances with an airline's cabin policies with a flight attendant is ridiculous anyway, hence my rating out of 10.

</comment><comment><USER8> This sounds dubious. It's obviously one sided which is OK, but what exactly is the motivation of a flight attendant to flat out lie about continuing to take pictures. And why wait to do it while continuing to service other passengers.

</comment><comment><USER9> Even if the travel writer had done something horrible this is going to give United a big black mark. I guess he'll record his future interactions with cabin steward/stewardess'.

<comment><USER10> I'm just curious who already had a good feeling about United. I mean, I fly Delta/United/America when I have to, but I've been pretty annoyed by all 3 for years.

I don't really see anything changing from this article.

<comment><USER11> I'm reasonably happy with United out of the big airlines. They almost always have seats with extra leg room, and they have a hub at the airport that's 10 minutes from my house.

I mean, they're still an airline, and I've had problems, but overall they haven't been bad to me.

</comment></comment></comment><comment><USER12> You poked the bear and, because you're white and rich, you got growled at instead of bitten. Pardon me if I do not feel sympathy for this "elite status" privileged whine.

<comment><USER4> Social change (without revolution) doesn't come until enough of the already empowered embrace it. So it may seem trivial to those of us that have seen far worse, but the alternative is for the white and rich to never hear about the problems that have affected one of their own. Since most people only recognize a problem when they are personally at risk, this sort of "whine" is a necessary part of the process.

</comment></comment><comment><USER13> Without any information from the airline hostess, captain or company this paints a very one sided and incomplete picture. The authors account might be accurate however I'm inclined to think that it may have more to do with the fact that he uttered the word 'terrorist' rather than any of his other actions.

I mean most seasoned travellers, let alone someone in the industry, should know by now that when you're in an airport you jump through all the hoops regardless of how stupid they may appear simply because airports and airlines hold power over you. For better or worse free speech does not exist in this environment and I wonder whether the author should have just apoligised and swallowed his pride rather than try to make a point or even apologise. Some times you just need to bite your tongue.

Edit: I'm curious about why people disagree with me.</comment>


The year 2000 - from 1900 - <AUTHOR> http://morletsblog.blogspot.com/2009/05/year-2000-from-1900.html <comment><USER2> _The Moving Pavement_

They're not widespread due to energy consumption, but most large airports have moving conveyers between terminals; and escalators (vertical moving sidewalks) are ubiquitous in malls.

 _House-Moving by Train_

It's normally done by flatbed truck, but it is done.

 _Televised Outside Broadcasting_

Justin.tv.

 _Personal Flying Machines_

Um, hang gliders?

 _Weather Control Machine_

[feverish paranoid conspiracy fantasy involving HAARP, Hurricane Katrina and global warming removed at the request of the commenter]

 _Combined Ship and Railway Locomotive_

Containerization. <http://en.wikipedia.org/wiki/Containerization>

_Undersea Tourist Boats_

Disneyworld's "20,000 Leagues Under The Sea" ride.

 _Roofed Cities_

Toronto's PATH. <http://www.toronto.ca/path/>

_Personal Airships_

Luxury private jets.

 _Summer Holidays at the North Pole_

Check back in 20 years.

 _Police X-Ray Surveillance Machine_

Ubiquitous urban CCTV cameras, TSA heavies.

</comment><comment><USER3> >Summer Holidays at the North Pole

Wait about 20 more years for this one.

</comment><comment><USER4> Wow, roofed cities and personal airships?

How could they miss the obviousness of the domes and flying cars we use today?

<comment> onreact-com Shopping malls are basically roofed cities. The roofed cities concept has been a very popular nation of the utopian literature of that time. I think one day we'll reach that state. We're just not yet there.

<comment><USER5> I'm pretty sure that the main reason we're not there is that many people actually prefer roofless cities.

<comment> onreact-com Yeah, but roofed cities doesn't mean the city is a huge building like you probably imagine but the idea behind it is the ability to stay dry when it rains etc. So maybe we should talk about roofed sidewalks instead.

</comment></comment></comment></comment><comment><USER6> Funny how they think a "Police X-Ray Surveillance Machine" is a good thing...

<comment><USER7> And that's the one they got right.

</comment></comment><comment><USER8> Somehwhat more accurate predictions from mid-century

<http://www.youtube.com/watch?v=GJjUVIIYptE>

</comment><comment><USER9> What, no flip-flops or tatoos?

I find it interesting that so many of these "predictions" consider technology while completely ignoring popluar culture. Notice that no matter what the prediction, people are still wearing Victorian garb. The easiest prediction of all would have been that people would wear more functional clothing.

</comment><comment> onreact-com This is just content theft with no source given. The source is <http://www.paleofuture.com> I can't find the deep link now though.

</comment><comment><USER10> Well 50% success rate isn't too bad.</comment>


Ask HN: What is an Environment? - <AUTHOR> Internally we have are having a discussion around what actually is an environment? What are the classifications of an environment? Should all environments be monitored? Is there really a difference between a dev, ref, qa, prod environment from an operations perspective.
It seems that the operational people care about all environments and want to make sure they are up. This is because if one of them goes down, then most likely they are responsible for getting it back up.
The developers only care about prod. They do not want to notified if their development environment is having issues.
I was wondering how others handle these problems internally or if we are the only ones who are having this discussion? <comment><USER2> > Should all environments be monitored?

Yes. If you don't collect data about the environment, you can't tell whether the current behaviour is normal or abnormal. Though not every environment should send all alerts possible.

> I was wondering how others handle these problems internally or if we are the > only ones who are having this discussion?

Monitoring is a tough topic -- still, after more than twenty years of development in this field. The problem is that we _still_ treat monitoring systems as monolithic applications with hardcoded data collection and alerting logic, instead of making them what they are: event stream processing engines, with some publish-subscribe capabilities to allow attaching some external logic.

I'm working on a toolbox that helps in building such a system ([http://seismometer.net/](http://seismometer.net/)), though it's still lacks tools and its website is more of a sketch than a project homepage.

In the meantime, you should think about how to allow different people to _select_ or _filter out_ different notifications, possibly in such a manner that they can process those notifications programmatically.</comment>


Why do baseball games take so long? - <AUTHOR> http://www.bostonglobe.com/sports/2013/06/08/why-baseball-games-take-long/wikaeRMGatBDGDefpbFE1H/story.html

<comment><USER2> Since people are mentioning other sports, for an analytical study on golf's pace of play, see Golf's Pace of Play Bible, by the founder of the three/45 Golf Association -- [http://www.three45golf.org/?r=no](http://www.three45golf.org/?r=no) \-- and a colleague at Columbia Business School. They found that studies showed that the current slow pace of play is the biggest obstacle to enjoying and growing the game. They analyzed golf's play with business operations analytics, like queueing theory.

I thought people liked to play golf to relax, but waiting for others isn't relaxing. It turns out that speeding the game through a few changes improves the experience for everyone -- not just players, but the club as well. They talk about principles everyone can do to improve the pace.

</comment><comment><USER3> Though my wife says she'd be happier to go to games if they were just about six innings long, I don't mind. I enjoy the atmosphere, particularly at Wrigley Field where there's no attempt to keep everyone distracted every second with per-player at-bat song themes and audience participation games during every inning break. The game is designed to be deliberately paced and reflective. Enjoy it for what it is. Pitching and hitting will ebb and flow as it always has. Strategies will change. But the pace is fine.

</comment><comment><USER4> Also American made "documentaries".

I cannot watch a doco made by Americans, its infuriating. Maybe 13 minutes 9 seconds worth of content stretched over 55 minutes of excruciatingly repeated cut scenes of the same damn clip over and over and over again. I once made a point of counting them. The same content, exactly the same footage shown 6 times, I quit watching about then.

Now compare that to anything by David Attenborough.

</comment><comment><USER5> Baseball seems very fast paced when compared to cricket, a game that can stretch over 5 days with routine breaks for lunch and tea.

<comment><USER6> Tests match have become more and more rare though with the increase in twenty- twenty matches.

<comment><USER4> And a damn shame that is, I say ! Not that I disapprove of 20/20's mind, but they should not come at the cost of proper tests.

</comment></comment></comment><comment><USER7> same of this stuff applies to tennis as well...there's too much time spend between serves with all the little habits each player has, or using towel etc., plus the break every two games !? It's hard to even consider watching a full 5 set game.

</comment><comment><USER8> Refreshing to hear somebody talking about _too much_ dead space filling entertainment. Contrast vs reality-TV and game shows where the pace is deliberately dragged down to a crawl.

</comment><comment><USER9> Football is a much worse offender. The amount of time spent waiting around just goes to show how it was perfectly made to generate TV advertising revenue.

<comment><USER10> With TV timeouts, sure. But generally this seems mitigated in football IMO -- the combination of:

A) the play clock (requiring the offense to start their next play quickly) and;

B) the fact that every single play has one or more players interacting with the ball, and some sort of an outcome from it.

Football has a methodical pace similar to baseball, but the biggest difference IMO is that in baseball a high percentage of "plays" (pitches in this context) don't actually result in any "action" and instead are just a ball or a strike.</comment></comment>


Simple tool to plan a bachelor party (or any weekend trip to be exact) - <AUTHOR> https://medium.com/@TalkAboutJack/9-step-guide-on-how-to-plan-a-bachelor-party-aeadd0cec899 <comment><USER2> PR/Advertisement for [https://jaypad.de/](https://jaypad.de/), which seems to combine discord/slack and strawpoll.

It's not clear what the value add is. They even feature slack in a graphic, as Jaypad vs slack/whatsapp/email, but I'm left wondering what the compelling feature distinction is.

<comment><AUTHOR> If I somehow violated HN-etiquette, sorry for that and I'd appreciate to be pointed into the correct title-rules for posting an article like this (or whether this is a place to share such an post).

Regarding the value we create: JayPads are a tool you can use to quickly make a plan in a group (coordinate a weekend trip, a party, etc) or to start the process of "getting something done". In a way it serves as a mini project management tool: A group chat separated from added features (like polls, notes, etc.) which conserve more permanent information which might otherwise get lost in a typical chat conversation.

I'd love to hear if this clears things up. And especially your thoughts on how to improve the landing page as well as the app itself as we might not have succeeded in getting our points across to you.

[1] [https://hackernoon.com/how-planning-a-weekend-trip-with- our-...](https://hackernoon.com/how-planning-a-weekend-trip-with-our-buddies- changed-our-perception-of-slack-and-whatsapp-1d53058f51d0)

[2] [https://hackernoon.com/how-jaypads-help-us-build- jaypads-c66...](https://hackernoon.com/how-jaypads-help-us-build- jaypads-c66c07c6c3ae)

<comment><USER2> I'm no HN expert. The article just, well, reads like an ad/PR. And things that read like an ad, for an unknown company, that aren't obviously useful, tend not to get many upvotes.

I don't think I misunderstand your product. I just don't see why I would create another account to try another tool that doesn't add much over discord/slack and something like strawpoll.

I feel like app creators always underestimate the friction in trying new things, making new accounts, etc. And getting my friends to try a new account over using tools we already use? Unless I see something that really solves a problem I could not have possibly overcome before, I'm on to the next thing.

This is harsh but, I don't think this is going to take off. Or if it does, it needs a lot more features to get there, something that competes with Evernote but more collaborative.

<comment><AUTHOR> The points on creating new accounts to try out a new tool you are making are exactly the reason why we opted to not make having an account mandatory. Users can make a JayPad by giving the JayPad and themselves a name (that's it) and see if our combination of text notes and Slack-like discussion adds value over a current setting with Slack + Strawpoll + Evernote or whatever they are using. Right now these are teams working on side-projects, or startups in the idea- to prototype-stage, local sports teams as well as political groups.

Thank you for your input on making this more into an Evernote plus X. We have this in mind and will be implementing new features and nuances in close loops with our users.</comment></comment></comment></comment>


Ticketmaster Beats Bots and Scalpers with Verified Fan Program - <AUTHOR> https://insider.ticketmaster.com/verifiedfan-faq/?_ga=2.211053269.1924618498.1537244127-618599229.1537244127 <comment><USER2> Note carefully what is _not_ said here -- that "verified fans", or _any_ fans, will get the _best_ seats ( _or_ will get seats of a quality even remotely corresponding to their purchase time).

Ticketmaster has left their same, bot-vulnerable systems design in place for 13+ years[0], simply because it _doesn 't matter_ to them whether brokers/scalpers get the best seats -- they are hired only to move the _most_ seats in minimum time _regardless_ of who gets them. One then has to wonder what their incentive has been to let the bot operators continue to operate the same way all these years[0].

[0] [https://news.ycombinator.com/item?id=13643045](https://news.ycombinator.com/item?id=13643045) "The Man Who Broke Ticketmaster" (2017), 300+ comments</comment>


2-Year-Olds Convert Binary To Hexadecimals In Game Experiment - <AUTHOR> http://www.flippybitandtheattackofthehexadecimalsfrombase16.com <comment><USER2> WARNING: Loud and obnoxious 8-bit music on autoplay when the page loads.

<comment><USER3> "8-bit music" is a non-sequitur.

<comment><USER4> "Music which is reminiscent of a game console from the late 80s until the early 90s, where the sound was produced by dedicated 8-bit sound chips, working alongside 8-bit primary CPU and using 8-bit DAC for conversion to analog output."

Is this better?

Or how about just "8-bit music".

<comment><USER3> Or, you know, chip music. Since the music is mostly mp3 it's not 8-bit.

I'm surprised a community as pedantic as this one doesn't get it.

<comment><USER4> 8-bit _music_ is a nickname for a music genre.

8-bit _audio sampling_ is a specific level of digital audio fidelity.

Human language is context sensitive. Pedantic doesn't mean we have to be stupid.

</comment></comment></comment></comment></comment><comment><USER5> Please let this be real, and back on the iOS App Store soon. My kids would so play & learn from this, just as they've spent lots of time on DragonBox learning core concepts of algebra.

The most annoying thing about April Fool's Day is everything that should be real but isn't, and everything that is real but shouldn't.

</comment><comment><USER6> Maybe it's an april fools, but the game is actually there, and it's playable.

The solution is to memorize the 4-bit sequence of each of the 16 digits, type the left nibble with your left hand and the right nibble with the right on the number row.

It takes a few minutes of practice

<comment><USER7> You have just learned how to use chmod correctly! The only difference is chmod uses octal, instead of hex.

</comment></comment><comment><USER8> Nice game! As it stands, later 'enemies' appear higher than more recent (and therefore more imminent) ones. The downside is that you sometimes can't read a number properly because it's occluded—with that font, the bottom of a character is not sufficient to identify what the character is.

My suggestion is to have the enemy z-index start at a high number and decrement per enemy (eg, z-index = 1000 - enemy_number). Trivial fix, and that way more pertinent enemies are on top with their numbers visible.

</comment><comment><USER9> So I assume this is an April Fools? Not sure, and I hate expanding efforts on filtering April Fools.

Tried the game, it's nice. Got 1 point on my first try, somehow managed 15 points on the second. I don't think a baby would get any points, so I'll go with April Fools.

<comment><USER10> If it is an April fool, then it is not entirely off the mark as satire. It isn't uncommon to see reviews of apps like Dragonbox and Slice Fractions where bloggers claim "my 5 year old can do algebra"[ _] or "my 3 year old can do fractions".

[_] My son could play Dragonbox when he was 3, so I am not knocking the game, but I am skeptical it will be of much use when he does algebra in school.

<comment><USER11> My 5-year old daughter played through Dragonbox pretty quickly and afterward I went through some concepts with her like x/x = 1 or 1x = x.

When shown to her using fractions of a drawn circle or simple multiplication of real objects, she seemed to understand the concept in the abstract.

However, once I introduced something like ayx/yx = a, it seemed to trip her up, even though the motions in Dragonbox made sense to her.

That said, I'm not sure I understood why x/x = 1 until much after 5 years old, so maybe there's something to be said about going through the motions of computation before truly understanding the general concept.

</comment></comment></comment><comment><USER12> Nice one! Converting Binary to Hexadecimals is easy when you've got experience from coding assembler on your Commodore 64. Just remember the 4 bits for each hexadecimal digit. Then enter the 4 bits of the first digit with your left hand and the 4 bits of the seconds digit with your right hand.

</comment><comment><USER13> I can't believe that URL wasn't already taken.

<comment><USER14> lol

</comment></comment><comment> bulte-rs Just had my two year old daughter beat me at this game. Thanks for destroying my self-esteem.

<comment><USER15> Wait, are you serious? Or is this more April Fools?

<comment> bulte-rs Actually. It's - unfortunately for me - partly true. She was able to figure out that you're able to use the number keys before me, and actually hitting 1A... I was fiddling around with my mouse up to that moment :|

(Semi off topic: I don't know how far developed your offspring is, but my four days shy of two years old is only able to count up to three and recognize her own name when written down sufficiently large ;)

Edit: she's also able to mash key's like a perl programmer)

</comment></comment></comment><comment><USER16> So the game was really hard to begin with, until I realised...

The hex numbers are two digits, and you can treat them both separately, so use the first 4 bits for the first hex digit and the second 4 bits for the second.

If you were brute forcing, this reduces the search space from 256 options to 32.

A good reminder of how you can handle hex. Really nice.

</comment><comment><USER17> Apparently, I'm the only one who can't figure out how to play. It seems you're supposed to launch missiles, but all I can do is make them pop into and out of the ground. As soon as the first bug reaches the ground, I lose, regardless of how many missiles I have showing.

<comment><USER18> Bugs have a hex number of them, you need to match it with a binary representation with rockets.. I'm still slower than that kid.

</comment></comment><comment><USER19> Being able to rebind the keys would be really nice. The keys are decently intuitive, but it sucks pretty bad if you have a split keyboard.

</comment><comment><USER20> This is the next 2048 for me at least. Simple, fun, addicting. Achieved a highscore of 41 before my wrist caved in.

</comment><comment><USER21> That game got a lot more fun when I found out the number keys work ... it's like playing a hexadecimal piano!

</comment><comment><AUTHOR> This was indeed an April fools joke ;). Our idea for the joke came a little too late to get our iOS version reviewed in time. So that one is on its way to Apple as we speak. Thanks for the great comments, it was a pleasure to read them!

</comment><comment><USER22> Their update says it was removed from the Apple App Store. I can't find out why, does anyone know (or might be able to hazard an educated guess) why it was taken down?

<comment><USER23> I guess they baned the word "flappy" in all their forms, to prevent an avalanche of "flappy bird" clones, and that includes "flippy". The game mechanics is totally unrelated, but the name is "too" similar, for a very low value of similarity.

<comment><USER24> They've got a Levenshtein distance of 3 (replace 'a' with 'i', delete 'd', replace 'r' with 't').

</comment></comment></comment><comment><USER25> "Seven percent of the test group could bitshift"

</comment><comment><USER26> Well, that's easy, binary to decimal is a lot harder.

</comment><comment><USER27> Can anyone explain how to win at this game? How to you determine the binary representation of a two-character hex number in your head?

<comment><USER28> Same way you convert one-character hex to binary. Then you simply do it again.

<comment><USER27> Ah of course. I forgot :-)

</comment></comment></comment><comment><USER29> Now this is a pretty cool April Fools thing.

</comment><comment> slipstream- After a few tries my high score is 0x13 :D

Of course, I'm about to try again.

(really needs a shorter URL too. too bad flippybit.com is taken.)

<comment><USER30> My first game was 1 point. I tried clicking on the falling bugs and saw some numbers lighting up, but I didn't quite realize I had to click on the numbers at the bottom to 'prep' rockets.

My next game was a much more respectable 25.

<comment> slipstream- Heh. I keep beating my high score in almost every new game I play.

</comment></comment></comment><comment><USER31> if u mash the keys from 1-8 randomly , its pretty easy to score big ;)

</comment><comment><USER32> 5 woot!

Caught some lucky numbers FE was first!</comment>


Ask HN: Have any Expense Tracking/Approval Software Recommendations? - <AUTHOR>

Hey HN,
We're growing and IT had to buy a bunch of equipment which Accounting wasn't expecting.
How do you keep track of your IT purchases at your start-up so there aren't any surprises?
Is there a "Basecamp" for Expense Tracking and Approvals?
Thanks, Max <comment><USER2> I've liked Outright, as a simple, easy to use, "won't need Quickbooks for a while" solution. <http://outright.com/>

</comment><comment><USER3> When you say approvals, do you mean prior purchase approvals?

By the way, our expense tracking service, CostJar, is in beta. Please signup for an early invite at <http://www.costjar.com>

</comment><comment><USER4> FeeFighters use and like Expensify: <http://expensify.com></comment>


Entrepreneurs are Dead, Long Live the Hacker - <AUTHOR> http://marcgrabanski.com/articles/long-live-the-hacker-entrepreneurs <comment><USER2> Idea people started dying out around YC's launch...

Once hackers (pg and such) got capital, it was game over for people that can't execute.

The title is bad because hackers become entrepreneurs when they embark on an enterprise.</comment>


Humor: WHOIS as a bulletin board - <AUTHOR>

% whois google.com
Whois Server Version 2.0
Domain names in the .com and .net domains can now be registered with many different competing registrars. Go to http://www.internic.net for detailed information.
GOOGLE.COM.ZZZZZZZZZZZZZZZZZZZZZZZZZZ.HAVENDATA.COM
GOOGLE.COM.ZZZZZZZZZZZZZ.GET.ONE.MILLION.DOLLARS.AT.WWW.UNIMUNDI.COM
GOOGLE.COM.ZZZZZ.GET.LAID.AT.WWW.SWINGINGCOMMUNITY.COM
GOOGLE.COM.ZOMBIED.AND.HACKED.BY.WWW.WEB-HACK.COM
GOOGLE.COM.ZNAET.PRODOMEN.COM
GOOGLE.COM.YUCEKIRBAC.COM
GOOGLE.COM.YUCEHOCA.COM
GOOGLE.COM.WORDT.DOOR.VEEL.WHTERS.GEBRUIKT.SERVERTJE.NET
GOOGLE.COM.VN
GOOGLE.COM.VABDAYOFF.COM
GOOGLE.COM.UY
GOOGLE.COM.UA
GOOGLE.COM.TW
GOOGLE.COM.TR
GOOGLE.COM.SUCKS.FIND.CRACKZ.WITH.SEARCH.GULLI.COM
GOOGLE.COM.SPROSIUYANDEKSA.RU
GOOGLE.COM.SPAMMING.IS.UNETHICAL.PLEASE.STOP.THEM.HUAXUEERBAN.COM
GOOGLE.COM.SOUTHBEACHNEEDLEARTISTRY.COM
GOOGLE.COM.SHQIPERIA.COM
GOOGLE.COM.SA
GOOGLE.COM.PK
GOOGLE.COM.PE
GOOGLE.COM.NS2.CHALESHGAR.COM
GOOGLE.COM.NS1.CHALESHGAR.COM
GOOGLE.COM.MY
GOOGLE.COM.MX
GOOGLE.COM.LOLOLOLOLOL.SHTHEAD.COM
GOOGLE.COM.LASERPIPE.COM
GOOGLE.COM.IS.NOT.HOSTED.BY.ACTIVEDOMAINDNS.NET
GOOGLE.COM.IS.HOSTED.ON.PROFITHOSTING.NET
GOOGLE.COM.IS.APPROVED.BY.NUMEA.COM
GOOGLE.COM.HK
GOOGLE.COM.HICHINA.COM
GOOGLE.COM.HAS.LESS.FREE.PORN.IN.ITS.SEARCH.ENGINE.THAN.SECZY.COM
GOOGLE.COM.DO
GOOGLE.COM.CSE.DEVORAELS.COM
GOOGLE.COM.CO
GOOGLE.COM.CN
GOOGLE.COM.BR
GOOGLE.COM.BEYONDWHOIS.COM
GOOGLE.COM.AU
GOOGLE.COM.ARTVISUALRIO.COM
GOOGLE.COM.AR
GOOGLE.COM.AFRICANBATS.ORG
GOOGLE.COM <comment><AUTHOR> A very interesting mix of sites "advertising" here, to say the least. Everything from swingers and warez to bat conversancy.</comment>


Ask HN: What happened to threewords.me? - <AUTHOR>

Still lots of traffic? Did it get monetized? <comment><USER2> <http://www.quantcast.com/threewords.me></comment>


Malaysian Prime Minister: Missing plane crashed in Indian Ocean - <AUTHOR> http://www.marketwatch.com/story/malaysia-prime-ministermissing-plane-crashed-in-indian-ocean-2014-03-24

<comment><USER2> This statement seems more to be made in a "We're sick of having the world's spotlight on us and we're giving up" way more than a statement of "This must be the conclusion because X, Y, Z."

<comment><USER3> It sounds like they've found a cluster of debris:

[http://www.theguardian.com/world/2014/mar/24/flight- mh370-ch...](http://www.theguardian.com/world/2014/mar/24/flight- mh370-chinese-plane-debris-indian-ocean)

<comment><USER4> No one has found anything in person yet except some wood with straps on it that cargo is sometimes secured to, but ships use these also and they're not sure it's from the aircraft yet. Lots of debris has been sighted in many different areas via satellite but when they arrive in the area nothing gets found.

So far it seems this is entirely based upon analysis of the SATCOM ping data. Unless they have information they aren't yet sharing, I feel it's somewhat irresponsible for the Malaysian government to tell family conclusively that it crashed into the ocean with no survivors when they haven't even found wreckage yet.

</comment></comment></comment><comment><USER5> Wonder how long it will be until he retracts this statement.</comment>


Authy is joining Twilio - <AUTHOR> https://www.authy.com/twilio <comment><USER2> > "There are few companies who share our dedication to excellence…"

Authy is one of the worst-designed iOS applications I have ever used. It has been this way for a very long time. They are actively hostile to people who try to criticize their poor design choices. I would not classify it as "dedicated to excellence."

CloudFlare forces me to keep it installed, so I have to interact with it on occasion. If you have more than about 3 services set up, you have to first scroll to expand the scrolling list of icons, then scroll multiple times to find and read the 10pt font name of a service, etc. It's awful. Compare it to Google Authenticator, which has a nice, big scrolling list of numbers and names. The one advantage Authy has is encrypted backup, which I like, and service lock-in, which annoys me.

Perhaps there's hope in somebody else taking over. Please, Twilio: prioritize _usability_.

<comment><USER3> Has anyone gotten the Bluetooth stuff to work on OS X? I tried hard, but never managed to.

To be fair, Bluetooth pairing between iOS and OS X hasn't been the most pleasant experience in general.

\--

Authy is a fine app, but the main selling point to me has basically always been the typical "anything but Google's half-abandoned BS".

<comment><USER4> No it just doesn't work. It is still available on the Mac App Store and when I contacted them on Twitter they said "oh, we will look into the problem you are having" but at the same time their website says it says the OSX app is discontinued due to bugs in OSX Bluetooth driver [https://authy.zendesk.com/hc/en- us/articles/202760296-Having...](https://authy.zendesk.com/hc/en- us/articles/202760296-Having-trouble-with-Bluetooth-)

I would not trust such liars with credentials to my most important services.

</comment></comment></comment><comment><USER5> Lots of hostility toward Authy on here, I'm shocked. I'm an Authy user and I've always liked the service in that I've barely noticed it existed. Super glad to use 1 app for most services I have 2FA on.

<comment><USER6> Yeah, my sentiment on this acquisition is "one company I like/use buying another company I like/use, I hope it goes well".

I switched to Authy when Google Auth 'forgot' all my tokens in one of its updates (which ended up being restored later), and Authy has worked flawlessly since then.

</comment></comment><comment><USER7> Interesting acquisition, glad to see authy as a service living in Twilio instead of just using it. Can't wait to see it in the portal. Make sure you read Jeff's post as well.

<comment><USER8> Jeff's Post

[https://www.twilio.com/blog/2015/02/ahoy-authy-joins- twilio....](https://www.twilio.com/blog/2015/02/ahoy-authy-joins-twilio.html)

</comment></comment><comment><USER9> Founder of @Authy here. Happy to answer any questions.

<comment><USER10> Why does Authy require I provide my cell phone number and email address? Why do I have to have a user account? This is completely ridiculous. I do not need nor want cloud syncing or backup. You are making Authy a potential target for attacks by associating a user to cloud stored 2FA information.

This is not in the spirit of 2FA.

<comment><USER9> Hi, good question. The reason for the phone number is that we depend on your phone number as part of your identity. Almost all 2-FA systems today use the phone number as a way to send you the code via text/phone call. If you read my blog post: blog.authy.com/twilio you'll see we decided to build our infrastructure on top of the telecom infrastructure because it was ubiquitous.

I also understand why some people don't like clouds backups. The good news is that backups are off by default and optional. If you don't need them, you can keep them disabled.

<comment><USER10> This tweet indicates you're using TOTP, slightly modified from Google's implementation:

[https://twitter.com/authy/status/498244613766139904](https://twitter.com/authy/status/498244613766139904)

      @benmcginnes Yes we are RFC 6238 TOTP compatible.
      Same algorithm as GAuth but 7 digits, 256 bit keys and 10 seconds window.

So why do you still need my phone number? There's no network connection or SMS required to generate those TOTP codes. I'm not buying the story that you need to text me or call me unless you're storing the seed/token centrally and sending it to users upon request which I strongly disagree with. That should only be stored on the user's device.

<comment><USER11> For those interested in how TOTP is implemented, here it is in Python [1] and Ruby [2]. It is really simple and understandable. Oh, and did you know you can secure your SSH connections using TOTP [3]?

This stuff is no more complicated than storing password hashes. Having a nice client app is good, but Google Authenticator is good enough. So instead of using authy and relying on a third party, why not get something like [4] and be done with it?

[1] [https://github.com/nathforge/pyotp](https://github.com/nathforge/pyotp)

[2] [https://github.com/mdp/rotp](https://github.com/mdp/rotp)

[3] [http://delyan.me/securing-ssh-with-totp/](http://delyan.me/securing-ssh- with-totp/)

[4] [https://github.com/mtigas/django- twofactor](https://github.com/mtigas/django-twofactor)

<comment><USER12> Oh thanks for mentioning [4], I always thought 2FA using PAM disabled the key- based authentication and used passwords.

</comment></comment></comment></comment></comment></comment><comment><USER13> When I first tried Authy, I was amused by bluetooth sync for OTPs. Then, when I tried hardware TSV like yubikeys, I realized that Authy was only a marginal improvement in technology, while it would take a complete rethinking of the system - e.g. Yubikey - to scale to dozens of accounts, multiple devices, and corporate adoption.

</comment><comment><USER14> I stopped using Authy a few weeks ago after I upgraded phones and lost access to all the tokens I had on my own phone.

Perhaps I should have read the instructions more carefully, perhaps I am an idiot. But I thought the purpose of an app linked to my cell phone number is that these codes would port automatically.

As a result, I will never use Authy again.

SMS validation seems to work fine, which is good business for Twilio. Not sure I understand the acquisition, unless Authy has some good math in their code generation process.

<comment><USER15> In my experience, Authy does port your OTP tokens between devices.

The ones that use their backend infrastructure are tied to your phone number, and are ported automatically.

Ones that you import from other TOTP-based systems via a QR code (like Google Authenticator) are private to your device by default. But if you really want, you can turn on the optional "backup" feature in settings, which will upload them to your Authy account and automatically port them between phones.

Sure, it requires an extra step... but to be fair, normal behavior for TOTP codes is to keep them 100% local to the device. I'd argue this is a reasonable default, given that security is involved. TOTP services almost always require a backup method (SMS or scratch codes) in case you switch phones, anyway.

</comment></comment><comment><USER16> Can anyone with experience give an estimate of the acquisition price?

<comment><USER17> Somewhere between $1.00 and 16 billion

</comment></comment><comment><USER18> You spelt "was acqui-hired by" wrong

</comment><comment><USER10> What is the reason for all of these 2FA apps? This is completely unnecessary. If you're going to offer 2FA, _please_ offer at least the following:

      * TOTP
      * Yubikey
      * SMS

If you want to partner with someone and ram unnecessary apps down our throats, optionally add the following:

      * Authy
      * RSA
      * etc

I have places where I want 2FA but I will _not_ install the Authy app, so I go without. Sorry. Let me use TOTP or my Yubikey.

<comment><USER19> You do realise that with authy you're just registering TOTP stuff right? It's not on the same level as what you're mentioning. It's like complaining about having to use Firefox instead of google.com

<comment><USER10> Then why do they need an account or my phone number to do TOTP?

<comment><USER20> Yes, that's definitely unfortunate. I don't remember it being required with older versions of the Authy app. I really hate forced cloud service integration, especially in applications that deal with sensitive material.</comment></comment></comment></comment>


Reforming the Mathematical Language of Physics (2002) [pdf] - <AUTHOR> http://geocalc.clas.asu.edu/pdf/OerstedMedalLecture.pdf

<comment><USER2> > I have noted that perhaps a third of the students seem incapable of > reasoning with vectors as abstract elements of a linear space. Rather, they > insist on conceiving a vector as a list of numbers or coordinates. I have > come to regard this concept of vector as a kind of conceptual virus, because > it impedes development of a more general and powerful concept of vector. I > call it the coordinate virus!

It's hard to tell what the author is proposing, but from the surrounding pages it sure sounds like he's talking about teaching vectors as geometric objects with various manipulation rules and _avoiding_ coordinate descriptions.

This is actually the sort of angle that may be possible for mathematicians (although I doubt it's sensible) but is not for physicists. In math, you're studying the logical implications of various axioms, and the abstract ideas are the _goal_. It's perfectly possible for a mathematician to learn and master an abstraction that was historically inspired by a physical system without personally ever actually connecting the abstraction to the physical system.

But for physics, it is absolute essential that you are also able to think about vectors in terms of coordinates if you are to connect them to the real physical world, which has rulers and T-squares and so on. Physicists are trying to understand the world, not the abstraction. In my teaching experience as a graduate student, students adopt the geometric picture of a vector as soon as they are able to, because it's vastly easier and more pleasant to work with. (That's why the abstraction was originally developed.) The students who don't fail to do so because they struggle to build the proper abstract machinery in their brains, not because they have been "infected" with the coordinate picture. The pictures aren't exclusionary, they are both necessary.

<comment><USER3> I disagree completely. I am not a physicist but I make video games which has the same kind of constant grounding in applicability. (We are always dealing with running physical systems, it's just that they are simulated.)

The week I learned to treat vectors as abstract objects, rather than arrays of coordinates, I experienced a drastic phase shift in my ability to program geometric operations effectively and clearly. The coordinates are still there, of course, but you have a lot more power over them.

The book "Linear Algebra Done Right" is all about this, and I absolutely recommend reading it if you haven't.

<comment><USER2> I don't see how we disagree. I wasn't downplaying the importance of the geometric picture, I was just disputing the idea that you could avoid teaching coordinates, or that the coordinate picture was somehow infectious in the sense that it would displace a geometric picture.

</comment></comment></comment><comment><USER4> Previous submissions: [https://news.ycombinator.com/item?id=3284160](https://news.ycombinator.com/item?id=3284160) [https://news.ycombinator.com/item?id=9202543](https://news.ycombinator.com/item?id=9202543)

Related conversation currently on HN: [https://news.ycombinator.com/item?id=9746051](https://news.ycombinator.com/item?id=9746051)

</comment><comment><USER5> Not being a mathematician, I have no idea whether Geometric Algebra is in fact superior to the systems in general use currently, but it sounds exciting.

</comment><comment><USER6> Kuro5hin's trane is a classical linguist (ie. Ancient Greek) who knows enough about physics and math to shoot his own foot off.

For example he regards the notion that mathematics does not permit division by zero as demonstrating that mathematics is faulty.

He's a real smart guy as well as one of my very best friends but it's hard to debate someone who got honors in his Linguistics Masters degree.

<comment><USER4> > _he regards the notion that mathematics does not permit division by zero as > demonstrating that mathematics is faulty_

It's very oversimplified to say that "mathematics does not permit division by zero". Mathematics (at least in the modern sense, descending from Euclid) is just the logical extension of whatever set of axioms you want to set up. It's entirely possible to operate on a concept of numbers which allows division by zero. For example using the extended complex numbers (the complex plane plus a point at infinity). See [https://en.wikipedia.org/wiki/Riemann_sphere](https://en.wikipedia.org/wiki/Riemann_sphere)

Anyhow, what does your friend have to do with Hestenes's paper?

<comment><USER7> > what does your friend have to do with Hestenes's paper?

The comment is clearly totally irrelevant, so I did some googling, and it appears that "trane" is a bot created on Kuro5hin by MichaelCrawford, and that references to "Kuro5hin's trane" are some sort of inside joke.

Which in turn makes it look like the non sequitur above was posted by that bot, even if other posts from that account were by human.

In any case it's just adding noise to the conversation.

<comment><USER6> His website is [http://www.subbot.org/](http://www.subbot.org/)

His real name is Robert Mitchell. He has a BA in Classical Greek as well as a Master's of some sort from U of Chicago. He worked for several years as a Java coder but for reasons I don't fully understand, decided that paying work is somehow evil and so flatly refuses even to apply for a job.

In part that's because he is very shy and so cannot tolerate office politics, and in part - again for reasons I don't understand - he has a passionate hatred for women and so cannot tolerate being in their presence.

He's heavily into jazz and swing music; he likes to call himself "trane" because that was John Coltrane's nickname.</comment></comment></comment></comment>


A Python Library to extract tabular data from PDFs - <AUTHOR> https://blog.socialcops.com/technology/engineering/camelot-python-library-pdf-data/ <comment><USER2> Cool! That's a good intro too.

Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coordinates. As you can imagine it's quite a battle to get back to something "raw" (assuming you even had fonts to start with).

<comment><USER3> If PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package.

I so much want to see the day when PDF is dead like Flash.

<comment><USER4> > all functions of an OCR engine outside of character recognition per se

Actually, depending upon how 'obfuscated' an author was attempting to be, you might need that OCR engine itself.

PDF allows for defining arbitrary mappings from byte values to font glyphs. So one could define byte value 32 (decimal, usually ASCII space) to actually map to printing, say, a capitol letter Z instead. One is supposed to provide a reverse mapping table when one does this that says "a decimal 32 byte prints a capitol letter Z" to allow for search and extraction purposes. But the PDF spec. does not require this reverse table be present.

So it is quite possible to randomly assign font glyphs to arbitrary byte values, and omit the reverse mapping table. The result would be that extracting data back out of that PDF results in garbage if one does not know beforehand what the mapping from byte value to glyph was.

So, if a 'bad actor' did this, one's only recourse to retrieving data would be to rasterize the PDF to a bitmap, then OCR the resulting bitmap to extract the content back out.

<comment><USER5> Or perform frequency analysis on the simple substitution chiffre. Seriously though, we need a document format with easier to extract payloads. Like Office documents with stronger structure, an underlying schema, along the lines of react-json-schema-form for Word.

<comment><USER6> ODF - Open document format, it is not perfect but a lot better than Microsofts formats.

</comment></comment></comment></comment></comment><comment><USER7> This sounds neat. Thanks for the work, vortex_ape and others. When I last needed this, I used tabula via tabula-py. Tried camelot on the PDF [0] I worked on and unfortunately the default option returned less-workable dataframe than tabula-py. I think it's just the area detection of stream and you are working on it anyway so I'm really looking forward to see the results.

btw, I think the pip install requirements missed opencv-python (on Windows?). And in this doc [2], it should be "top left and bottom right" instead of "left-top and right-bottom".

[1] [https://www.boj.or.jp/en/statistics/set/kess/release/2018/ke...](https://www.boj.or.jp/en/statistics/set/kess/release/2018/kess1808.pdf)

[2] [https://camelot- py.readthedocs.io/en/master/user/advanced.ht...](https://camelot- py.readthedocs.io/en/master/user/advanced.html#specify-table-areas)

<comment><USER8> Hey squaresmile! Yes, right now table detection with Stream doesn't work nicely if the table is not present on the full page, for which you can use the table_area kwarg from [2].

You should use "pip install camelot-py[all]" to install Camelot (which will install opencv-python too). I had to take it out of the requirements since it wasn't available in any conda channels while I was creating the conda package. I'm looking to remove opencv as a requirement altogether by either vendorizing the opencv code that is being used inside Camelot or reimplementing the code using something lightweight like pillow.

Thanks for the catch in [2], I'll correct it!

</comment></comment><comment><USER9> Quick suggestion - you should integrate the functions to extract signature data inside PDF. This is a huge issue and everyone has to write their own.

for example, this is my sample piece of code to extract data from Aadhaar signed PDF [https://pastebin.com/dg8p98T1](https://pastebin.com/dg8p98T1)

<comment><USER8> Thanks for the suggestion sandGorgon! Can you also point me to an example of a PDF with signature data?

<comment><USER9> unfortunately i cannot share without running afoul of all the laws out there. but you can create your own here - [https://app.digio.in/#/authenticate](https://app.digio.in/#/authenticate)

<comment><USER8> Ah sorry I forgot about posting PII data online. Thanks for the link!

</comment></comment></comment></comment><comment><USER10> This is a really good example of how to briefly introduce/sell a library. What it does, why, how, how to install it, with concrete examples.

<comment><USER8> The API and docs (on which the blog post built upon) were inspired from pandas and requests!

</comment></comment><comment><USER11> A few months ago I was looking for a similar solution but couldn't find one that handles empty cells very well. I ended up writing my own program[0] that is specific to my files' layout.

This library works perfectly and could've saved me a lot of time! Looking at some of the source code, we used similar logic to parse the tables. Pretty neat!

0.[https://github.com/khllkcm/pdf2calendar](https://github.com/khllkcm/pdf2calendar)

<comment><USER8> Will check out pdf2calendar!

</comment></comment><comment><USER12> This is nice. I do quite a bit of tabular data extraction and pdf tables are often a sticking point. It is absolutely correct in describing it as a "fuzzy" problem.

My go-to solution has been 'pdftotext -layout' with a bit of hackery before giving it to pandas.read_fwf. That usually gets me 80% of the way there 80% of the time. The upside is that this tends to fail "better" than some other options.

I look forward to kicking-the-tires with this on my test cases.

<comment><USER8> Do submit bugs on GitHub if you face any issues! [https://github.com/socialcopsdev/camelot](https://github.com/socialcopsdev/camelot)

</comment></comment><comment><USER13> This is a very interesting software. In research community still many results are only in pdf tables in papers, so obtaining them in dataframe is very useful, good job!. By the way, I would like export also in Excel files in the command line.

<comment><USER8> Hey danimolina! You can export the data into an excel by specifying it as the export format, Camelot comes with a command-line interface too! [https://camelot- py.readthedocs.io/en/master/user/cli.html#cl...](https://camelot- py.readthedocs.io/en/master/user/cli.html#cli)

You can simple do: camelot --output data.xlsx --format excel lattice input.pdf (lattice can be replaced with stream based on the type of tables in your PDF)

</comment></comment><comment><USER14> > However, OpenCV's Hough Line Transform returned only line equations.

Did you try HoughLinesP? [https://docs.opencv.org/2.4/modules/imgproc/doc/feature_dete...](https://docs.opencv.org/2.4/modules/imgproc/doc/feature_detection.html?highlight=houghlinesp#houghlinesp)

Returns line segment endpoints with a probabilistic Hough Transform. I'm fully confident your solution works, just wondering if you tried this and why it was rejected.

<comment><USER8> Hi plaidfuji! I did try HoughLinesP during experimentation. I vaguely remember (since this was almost 2 years back) getting the actual line segment as a combination of multiple smaller line segments in all cases (which could then be combined to form the actual segment using some heuristic). It came down to getting the actual table line segment out which a combination morphological transformations and cv2.findContours provided (without the need for another combining step).

<comment><USER14> Interesting. I noticed you mentioned below that you're trying to get rid of OpenCV as a dependency - that's really tough. I came from a Matlab background where image processing was really well-packaged and Python is a total mess.

If you managed to vendor a small portion of OpenCV that contained image i/o, basic colorspace conversion, thresholding, scaling/rotating, shape drawing/insertion, HoughLines and findContours, I think you could release that as its own package and it would be quite popular. OpenCV is such a bloated dependency...

<comment><USER15> scikit-image contains Hough transforms and the other things you mention? Though it does depends on scipy and matplotlib, which are kinda big.

</comment></comment></comment></comment><comment><USER16> Oh that is so timely. I've been putting that part of a pipeline I built off for a while due to the complexity and now I can just plug this in. Super neat. Thank you very much!

<comment><USER8> What does this pipeline do and what software have you used to implement it?

I have used Airflow in the past to create ETL pipelines, and plugged in Camelot in one of them to extract tables from PDFs. I also wrote a blog post about it in case you might be interested. [https://hackernoon.com/how-to- create-a-workflow-in-apache-ai...](https://hackernoon.com/how-to-create-a- workflow-in-apache-airflow-to-track-disease-outbreaks-in-india-fd145575efa4)

<comment><USER16> Compress scientific papers.

Thank you for the pointers!

</comment></comment></comment><comment><USER17> > When using Stream, tables aren't autodetected. Stream treats the whole page > as a single table

I've often wondered if image semantic segmentation methods as used in the ML community could successfully identify things like "there is a table (or figure) here, it's not part of the main text". I mean, it seems that humans should be able to do this even without reading the text so I don't see why a CNN couldn't.

<comment><USER18> Yes it should work. Definitely worth trying.

</comment></comment><comment><USER19> An alternative: [https://github.com/jsvine/pdfplumber](https://github.com/jsvine/pdfplumber)

<comment><USER20> Yeah. pdfplumber is also good for digital pdfs. Curious to know the advantages of camelot over this.!

</comment></comment><comment><USER21> So many times I have wanted to get this type of data. Visa would send reporting this way and it would have to be manually copied over. They offered CSV but there were extra charges associated. There were some pretty good libraries for paragraph text extraction but the graphs were too tough to deal with.

</comment><comment><USER22> I hope this is good at extracting register maps from datasheets. That would save a lot of tedious driver work.

<comment><USER8> Hi berti! I wrote the library and the blog post. Can you point me to some PDFs which have these register maps?

<comment><USER22> Try these: Page 233 [http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-8351-M...](http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-8351-MCU_Wireless- AT86RF233_Datasheet.pdf)

Page 45 [https://ae- bst.resource.bosch.com/media/_tech/media/datashee...](https://ae- bst.resource.bosch.com/media/_tech/media/datasheets/bst- bmi160-ds000-07.pdf#page45)

Is the library able to handle cells that span multiple columns?

<comment><USER8> I assumed that you're talking about page 33 in the first PDF, since it has only 225 pages. I extracted Figure 6-23 from it and the table on page 45 in the second PDF. Here's a gist: [https://gist.github.com/vinayak- mehta/cf30a5560f1b8ab4c0b25e...](https://gist.github.com/vinayak- mehta/cf30a5560f1b8ab4c0b25e34e5c6121b)

Yes, Camelot takes care of cells spanning multiple columns! You can check out the Advanced Usage section for explanation on the keyword arguments I used in the gist! [https://camelot- py.readthedocs.io/en/master/user/advanced.ht...](https://camelot- py.readthedocs.io/en/master/user/advanced.html)

<comment><USER8> Note: I had to decrypt the second PDF using qpdf since the library I'm using to split a PDF into pages (PyPDF2) doesn't support the encryption type of that PDF.

Did this: qpdf --decrypt input.pdf output.pdf

</comment></comment></comment></comment></comment><comment><USER23> I'm curious why the authors didn't contribute this directly to Tabula instead.

</comment><comment><USER24> Interesting. I've used Tabula [0] in the past with great success. I wonder how this compares.

[0]: [https://github.com/tabulapdf/tabula](https://github.com/tabulapdf/tabula)

<comment><USER25> They have a detailed comparison with other tools (including Tabula) in the wiki:

[https://github.com/socialcopsdev/camelot/wiki/Comparison- wit...](https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other- PDF-Table-Extraction-libraries-and-tools)

</comment></comment><comment><USER26> Great work and write up! HN submissions about PDF extraction seem to be as reliably popular as threads mentioning bees or bashing Mongo, which I guess goes to show how pervasive a problem it is.

</comment><comment><USER27> Is there any decent tool for tabluar data extraction from scanned PDFs?

<comment><USER28> Hey andrew_chris, we're working on it and will be interested to help you. Please contact me: tanmay [at] inkredo [dot] in

</comment></comment><comment><USER29> A Python library to do this is cool, but there's already Tabula: [https://tabula.technology/](https://tabula.technology/)

<comment><USER30> They address Tabula in the post:

>The first tool that we tried was Tabula, which has nice user and command-line interfaces, but it either worked perfectly or failed miserably. When it failed, it was difficult to tweak the settings — such as the image thresholding parameters, which influence table detection and can lead to a better output.

</comment></comment><comment><USER31> [https://github.com/invoice-x/invoice2data](https://github.com/invoice-x/invoice2data) is another one.

</comment><comment><USER32> You should also check out pdf.js:

We use it in Polar:

[https://getpolarized.io/](https://getpolarized.io/)

for our PDF management.

It's a pretty robust library and it renders everything on canvas BUT you also get the raw text in the DOM so you can play with it more as an API for managing PDFs.

REALLY nice to be able to use web standards when working with pdf.js.

The downside is that the graphics are rendered to canvas so you're only really getting an image.

</comment><comment><USER33> Wouldn't it be easier and more generic to have an OCR solution for this task?

<comment><USER8> Hey amelius! Though OCR would provide a generic solution, it would be an overkill for text-based PDFs. I'm working on getting a OCR solution up since there's still a lot of data that is trapped inside scanned PDFs and not text- based ones.

If you have any pointers in the OCR route, do suggest them here, or on this GitHub issue! [https://github.com/socialcopsdev/camelot/issues/101](https://github.com/socialcopsdev/camelot/issues/101)

<comment><USER28> Hey vortex_ape, we're also working on extracting data trapped inside scanned PDFs and recently, we've begun to get good results using DL algos. I am based in Gurugram, would you like to catch up and exchange experiences?

</comment></comment></comment><comment><USER34> Can't wait to try this out with Percollate!

<comment><USER8> Doesn't Percollate save web pages as PDFs? If you have a web page with tables, you can directly use pandas.read_html to extract them!

</comment></comment><comment><USER35> I'm always skeptical of these kind of libraries. Whenever I try to use them, it ends up feeling like a broken promise.

<comment><USER28> Ever researched why it breaks?</comment></comment>