Why 30 year old entrepreneurs are not over the hill - <AUTHOR> http://www.ycreject.com/2011/05/why-30-year-old-entrepreneurs-are-not.html <comment><USER2> I don't think it's about money as much as lifestyle -- the OP misses this entirely by assuming that both entrepreneurs are single and without children.

Something interesting I observed at Startup Weekend here in Indy last fall:

When it came time to form teams, all the young'uns grouped with one another, and the four of us codgers with responsibilities worked together. The line wasn't drawn according to age: there was one 30+ guy in another group, but he was single; two of the four members of my group were still in our 20's, but we were the primary sources of income for families with children, and I think that led us to fit in better with our 40+ cofounders.

With a few exceptions, I'd find it uncomfortable -- if not unworkable -- cofounding with the bouncy 20-something who has no outside responsibilities. When team building means getting drunk together on a Wednesday night, work hours are noon-8pm, and the company culture is low on planning and high on "nothing to lose", a company is just not going to be a good fit for me.

I'm fine with economic sacrifice, and with working hard. Because I'm looking for a new startup to join, I've tooled down my living expenses such that I could do an extended period somewhere between the two guys in that article. Because I've always been a hard worker, I've made my work schedule part of my family's life: I work a normal 9-3:30 in the office (currently home office, but I'm flexible), then after school, my son brings his homework to my office and we do our work together for a couple of hours. I tend to get in another hour or two in the evenings after he goes to sleep. I never get drunk, let alone on a weeknight when I have to get a first grader up for school in the morning. I'll never end up in an incubator because I'm not changing my son's school, leaving his speech therapy program, leaving our dojo, and generally up-ending his whole world in exchange for a few months of help for my startup.

I often meet people at my stage in life (kids, responsibilities, etc.) who say they don't do start-ups because they perceive "bouncy without responsibilities" to be the expectation of all startups. Their loss. I have to look a little harder, but I find other "codgers" out there in the entrepreneur community. I have to wonder if the people held back by the stereotype really have whatever that spark is that makes us take risks on new things anyway.

Oh, and I almost forgot.... GET OFF MY LAWN!

<comment><USER3> There are significantly more "non-traditional" families today than there were 50 years ago and assuming that older entrepreneurs had to be in the same kind of traditional situation (married, kids, house in the suburbs) isn't much better than assuming only young people can start companies.

There are dual income, no kids couples, there's divorced people, there's never been married people.

It would be great if everyone stops assuming that everyone else is the same as you.

<comment><USER4> > It would be great if everyone stops assuming that everyone else is the same > as you.

This seems unnecessarily snarky. OP didn't say "all > 30 year olds have these issues", he was talking about his own experience and how it shapes his startup aspirations.

Kids & Mortgage is still very common.

<comment><USER3> I agree that I was overly snarky, though only slightly. I disagree that Kids & Mortgage is still very common. Maybe just common, and even then, there's often additional nuance (second marriage, renting not mortgage, etc.)

</comment></comment></comment></comment><comment><USER5> If you're <a thing> who has bought into some blogger saying you aren't cut out to be an entrepreneur because of <aforementioned thing>, then maybe you aren't cut out to be an entrepreneur.

One of the very best things about being an entrepreneur is that you get to ignore what everyone else thinks qualifies or disqualifies you. The only thing that matters is what you make (or more accurately, what you do with what you make).

</comment><comment><USER6> Not everyone over 30 has a wife, kids, 2 cars, is in debt up to their eyeballs, and lives downtown San Francisco.

I'm over 30, single, own a home, and am bootstrapping a software/consulting business. Between income from my business, savings, and profits in the stock market (I day trade on the side) I have almost unlimited runway. I don't have to pander to investors and I have the time to really sink my teeth into some hard problems. The crazy thing is when I do the math at the end of the year, I'm not even that far behind where I'd be if I had a day job and I'm still just getting started. What risk?

<comment><USER7> I am in basically the same situation: Over 30, single, own a home, bootstrapping a business. I have steady and flexible part time consulting work I can use to pay the bills if I need to, but rarely do. I have 2 room-mates I share my house with that pay 100% of the mortgage and a large chunk of utilities. I have practically 0 debt (other than the house). I have worked at several startups already, and know better than many first-timers what is involved. If anything I believe I am better off starting a company than a young 22 year old.

One important difference though: Like another commenter said, I am not as eager to accept any investor terms thrown my way, nor am I gauging my success on investor interest. I am in this to make a profitable, successful business venture, and will only accept investment on good terms when it will directly aid in the growth of my company. Perhaps that is why VCs might avoid older entrepreneurs, we are more cautious of investment deals, and don't judge success by those alone.

<comment><USER8> I"m over 30 and single too. But I dont own a home - so I do have substantial savings. What kind of business are you starting ?

I've been doing some customer development in the area of travel. I quit 3 weeks ago.

</comment></comment></comment><comment><USER9> Obviously, Arrington's article was just a fluff piece. 30 is not over the hill. 25 is also definitely not peaking. It is true that a 25 year old is more willing to take a risk on a startup than a 30+ year old (which I have learned first hand), but that doesn't mean it doesn't happen. It's all about the person. I'm 27 and am about to take a huge risk with a startup. Maybe i'm still in the "peak age" but I don't think at 30 I would think any differently. It's all about passion.

<comment><USER10> Agreed.

It's not consumer internet but I always look at how Colonel Sanders made his mark after he turned 65 as inspiration for the future: <http://en.wikipedia.org/wiki/Colonel_Sanders>

</comment></comment><comment><USER11> At 28.8 I am a better decision maker. I have better life skills, I am better at making friends, making social connections and making business connections.

As a coder, I no longer like coding for the sake of coding. I like coding because it is a means to an end. My powers of abstraction are greater. After 2 years of moderately successful solopreneurship, I fully understand the absolute need to ally with other people.

I am able to cut through people's bullshit easier. When I meet a potential business parter/coder I am more quickly able to see exactly how strong they are, what kind of drive they have, and ultimately I am more quick (and confident) to tell them to fuck off if they are a liability. As a 20-25 year old, I had no confidence and could not speak up for myself.

I focus solely on projects that I enjoy with people that I enjoy working with. My ability to create potentials for happiness and avoid pitfalls is stronger than ever. Physically, I am stronger than ever, and I eat much much healthier.

I am more positioned than ever before to hit a project out of the ballpark.

Finally, at 28.8 I realize that my ageism is just another prejudice that was hindering my ability to take advice. Now, my mind is not closed. It is more open than ever before. Age has worked well for me, and young peers, it will work well for you too.

</comment><comment><USER12> Two advantages that older entrepreneurs have is domain expertise and industry connections. For example, I know some people who started companies that contract out to the US government and make a killing doing it. I don't think an under-25 entrepreneur would be too successful doing that unless they're well connected (involves writing proposals, which most 20-somethings aren't going to be good at) .

My observation is that most young entrepreneurs stick to building consumer internet products that don't require a lot of domain expertise.

</comment><comment><USER13> I would argue that this "over the hill by 30" mentality permeates a lot of things in our culture. I know I hear it a lot in the sports I am involved in (climbing, backpacking, etc.)

Yet there are at least 1-2 guys well past 30 (likely in their 50's) at the climbing gym I go to that could out climb most of us twenty-somethings. There are also a healthy number of successful entrepreneurs well past 30.

I would posit that more than anything, being "over the hill" is a frame of mind.

</comment><comment><USER14> Flamebait.

I cannot be bothered, in the 14th hour of my workday, to find the reference on the internet but the last time I heard anything the average age was thirty six.

The average VC victim is in their twenties, sure, because they have no capital.

</comment><comment><USER15> I'm 41 with a wife and kid (another on the way) and not crazy technical: <http://pricetack.com>

</comment><comment><USER16> 1\. startups are not risky. there's plenty more

2\. it's a lot harder to negotiate terms with an experience entrepreneur.

3\. over 30s can hire under 30s.

4\. is it techcrunch or techtroll?

</comment><comment><USER17> I'm just about to hit 30 and taking more entrepreneurial risks. I feel like the knowledge I've gained in the business world is immensely helpful.

In general, it seems that the older you get, the opportunity costs for risks increase. Not just monetarily speaking but also other things you may have to cut back on - family, friends, etc.

<comment><USER18> i'm in the same boat and struggling with the same issues. on the verge of 30 it becomes more of a game of weighing the pros and cons.

does this startup idea have enough potential that it's worth leaving my full time job? and if so, how am i going to keep paying these damn student loans? etc.

<comment><USER8> True but think about when you turn 40, how are you going to feel about the work you did during the most productive years of your life ?

Your remember that 30th birthday. I bet turning 40 is going to be much much worse.

</comment></comment></comment><comment><USER19> I'm a 31 year old entrepreneur. Here are my rough living costs. I've done them on a personal basis. The household costs (including my wife) are double this. I'm not saying I'm typical, but I am saying that it doesnt _need_ to be as expensive as the article claims.

      Rent/Mortgage/Prop Tax: $765 (single family home in SF, including office)
      Utilities: $80 (fast Internet)
      Insurance (car, home): $60 (free health plan through wife)
      Transportation: $10 (bike)
      Gas: $50
      Food: $150
      Fun: $200
      Clothing: $50 (and it shows..)
      Grand Total: $1365

And I'm not living on Ramen!

<comment><USER20> I'm jealous. 28 and in NYC here.

      Rent: $1700 (tiny studio in Manhattan)
      Utilities: $200 (electric and FIOS)
      Transportation: $120 (subway + occasional taxi)
      Food: $600
      Fun: $400
      Maid: $400
      Clothing: $100
      Grand Total: $3,520

<comment><USER21> you need a maid for your tiny studio? you spend $100 in clothing every month?

<comment><USER22> $400 dollars a month for said tiny studio? How about learn to pick up after yourself and pay someone $50 to clean the bathroom and dust once a month.

</comment></comment></comment></comment><comment><USER23> His "data-driven" numbers are misinformed because they're raw. They just look at how many 20-year-olds started a company and took it to success vs. how many 30-year-olds did the same - just raw numbers.

They don't look at the percentage of 30+ people who succeed out of the total in their age group who attempted a technology-startup vs. the same for those in their 20s; and if the difference is statistically significant. If he tells me that 20% of people in their 20s succeed and only 5% in their 30s succeed then he'd have a point, I bet he does not.

</comment><comment><USER24> Didn't Arrington start Techcrunch at 35 anyway?

<comment><AUTHOR> Yes. That's about the age he started TC.

</comment></comment><comment><USER25> Also not considering the 30 somethings (such as myself) who spent a few years working for companies in order to save up and get some experience BEFORE taking the "plunge". Now I have a great portfolio, some cash and some experience that can all help me work towards this kind of goal. Doesn't necessarily put me a head of a new grad but surely doesn't put me behind either!

</comment><comment><USER26> If one actually reads 'Zen Mind: Beginner's Mind' it should be abundantly clear that enlightenment can come to anyone at any age. That said the primary reason why the 30 year old is not over the hill is that he or she can still see the 60 year old just short of the summit (and neither can see that there are an endless number of summits to come...)

</comment><comment><USER27> Why does the 30 yr old spend more on gas (in the initial estimate)?

<comment><USER28> As a 35 yo:

I used to be able to walk/catch bus/ride lots of places. Now I have a kid it's no longer possible, and things like school drop-offs double my commuting distance.

</comment></comment><comment><USER29> 30/40 year old entrepreneur is more likely to be married or in a stable relationship; fun is Netflix ($5 a month) or streaming movies (free). Use wife's health insurance. $400 a month on transportation, really? Anyone with a wife/gf knows they don't travel that much anymore. No reason for utilities to be higher either. Food is way lower, older people are more likely to cook their own and make smarter supermarket choices. Being older and boring is very cheap.

If working from home eg programming, having your own place can mean that you sleep better and are more focused on doing the work, rather than being distracted by whose turn it is to do the dishes or joining in your roommates' post-bar parties. Of course, having kids or being a caregiver would change the person's profile, regardless of age. Just a few counter-thoughts.

</comment><comment><USER30> The younger crowd is also more impressionable than the older crowd. Older investors I bet <3 to play sage and school the youngins' and the first step to get them to listen is to compliment them. lol

</comment><comment><USER31> One thing missed by this article - the house price boom over the last few years means that people who bought before the boom should have much lower outgoings for housing. A 30-something with much of their house paid off is in that respect in a better position to take risks than a 20-something paying rent and perhaps wondering if they ought to save for the downpayment/deposit on a house purchase. Another issue is that learning how to cook healthy food quickly and cheaply can take years to learn and your food costs can diminish with age.

</comment><comment><USER32> man, suddenly i feel old with 28...where has my live gone ? ;)

<comment><USER33> Turn around, it's ahead of you.

</comment></comment><comment><USER34> I'm 36 and just getting warmed up!

</comment><comment><USER35> When someone says that he will only fund or work with people in their 20s, run. He's looking for a young, inexperienced person who doesn't know his rights and of whom he can take advantage.

Where there is some truth in this is that a 30-year-old first-time entrepreneur usually cannot get terms from a VC that would merit leaving a more stable job. A person of that age with a decent career is unlikely to have much interest in the terms (participating preferred, easy to fire the founders, giving up a lot of board seats, low salary) available to a no-name first-time entrepreneur. If someone who is 35 is willing to take a 3x-liquidation preference and participating preferred, do you really want to fund him? Obviously no, because he either lacks decent options or doesn't understand business, neither of which is a good sign for someone of that age. On the other hand, there are plenty of 22-year-olds with a lot of talent but no sense of what they're worth or what rights they have.

<comment><USER36> Considering that it takes 7+ years to build a substantial, successful company, if you found a startup when you're 25 and take 7 years to exit, you WILL be > 30 when you're done and available to work on startup #2.

So by this "under 30 only" logic, you would never fund successful serial entrepreneurs. Great idea. :-/

<comment><USER10> Well said.

</comment></comment></comment><comment><USER37> Young bull viewing a herd of cows up the hill says to old bull: "Quick let's run up and ride a cow"

Old bull replies : "Let's walk up and ride 'em all"</comment>


D.C. court rules tracking phones without a warrant is unconstitutional - <AUTHOR> https://www.cbsnews.com/news/d-c-court-rules-warrant-is-required-for-stingray-cell-phone-tracking/ <comment><USER2> This is a big win, but it is going to be a long road ahead to gain back basic personal privacy.</comment>


Roblox grows 3x in 3 years – now 100M MAU and bigger than Minecraft - <AUTHOR> https://nwn.blogs.com/nwn/2019/08/roblox-second-life-revenue-users-linden-lab.html <comment><USER2> 7 days ago: [https://news.ycombinator.com/item?id=20620102](https://news.ycombinator.com/item?id=20620102)</comment>


Apache Spark building on Scala 2.12 - <AUTHOR> https://issues.apache.org/jira/browse/SPARK-14220 <comment><AUTHOR> This was a long and major effort, awesome to see it completed. I believe the underlying issue holding Flink back from 2.12 support was also the ClosureCleaner taken from Spark[0].

[0] [https://github.com/apache/flink/blob/dacc16b4fa6db6abfdbf73b...](https://github.com/apache/flink/blob/dacc16b4fa6db6abfdbf73b99f26a5fd36b12acd/flink- scala/src/main/scala/org/apache/flink/api/scala/ClosureCleaner.scala#L33)</comment>


Tech companies may have a public-image issue in political battleground states - <AUTHOR> https://www.recode.net/2017/4/29/15470916/tech-companies-facebook-uber-airbnb-voters-government-oversight-politics <comment><AUTHOR> So, it somehow takes Republican pollsters to tell us that Airbnb, Uber, and Facebook are behaving like extractive rentiers?</comment>


Fun with the Anthropic Principle (2006) - <AUTHOR> http://www.scottaaronson.com/democritus/lec17.html <comment><USER2> I go through this chain of logic every so often:

What are the chances that anything only exists right now, and that this experience I'm having is 'special' in some way (e.g. significantly more likely to be occurring)? At first, it's fairly high.

Then, by thinking about all the other things in my memory, the probability of the experience being 'special' or 'more likely to be occurring' in some way significantly decreases. Why would my memory be complicated enough in a 'special' world to remember details about totally mundane things that seem to have happened a long time ago?

Both before and after the chain, I'm in the real-world state of mind. But during it, very fundamental probabilities shift. Am I being irrational here?

Tangential, but something that gives me a lot of respect for philosophers and psychonauts is that any significant discovery you make affects your world view irreversibly (unless you are good at selective denial). If you're happy with your current world view, and there's a chance that you would make a discovery that makes you unhappy, isn't it a mistake to keep going? But as a human, maybe part of what keeps you happy is that uninhibited, unworried revealing of new experiences.

<comment><USER3> > Am I being irrational here?

Not so much irrational as not particular enough; this line of reasoning doesn't really work.

The thing is ... at any one instant, the amount of memory you are able to recall / visualize / etc is very small, and if your mind is occupied by that thought, you won't have any thoughts about the present simultaneously. It is only the apparent continuity of time that seems to link these things.

So when you say "thinking about all the other things in my memory", well, you can't experience all the things that are supposedly in your memory. You can only experience one small part of that at any time. If there is only one time, then the other stuff does not exist, you just have confidence that it exists for some reason.

</comment></comment><comment><USER4> There is a lot going on here.

1\. The part about the red-haired and green-haired people is more or less the Sleeping Beauty problem[1]. The LessWrong community (who are Bayesians) has concluded that this problem is under-specified and doesn't say what it's asking, and has two different valid answers.[2]

EDIT: Actually it's a bit different. I wouldn't assume the two problems should have the same answer.

2\. The dice room / madman experiment is ill-defined. It only works if you have an infinite population to take people from (if there is a bound on the number of rounds you can do, the effect vanishes), and it implicitly assumes that every person is equally likely to be chosen. This is nonsense, because there is no uniform distribution on the natural numbers.

Taking a closer look, put the die results in a grid, where columns are the total length of the experiment, and there's one row for each round:

                 len=1  len=2  len=3
        round 1  snake
        round 2  normal snake
        round 3  normal normal snake

Here are the probabilities of these results:

                 len=1  len=2  len=3
        round 1  1/36
        round 2  35/36  1/36
        round 3  35/36  35/36  1/36

If you take the product of each row, that gives the probability that the experiment ends in that particular round. If you sum up these probabilities, you get 1, as you'd hope. (It converges slowly.)

Here are the number of people involved in each die roll, with a * if they are killed:

                 len=1  len=2  len=3
        round 1  1*
        round 2  100    100*
        round 3  10000  10000  10000*

The reason this experiment is unintuitive is because you can calculate the probability that someone will die (i) by column, assuming that the length of the experiment is known, or (ii) by row, assuming that the round you were selected in is known, and these give different answers. (Try it.) This is because infinite sums are not commutative.

3\. There's no trouble with the doomsday argument, because SIA is correct :-).

[1] [https://en.wikipedia.org/wiki/Sleeping_Beauty_problem](https://en.wikipedia.org/wiki/Sleeping_Beauty_problem)

[2] [http://lesswrong.com/lw/32o/if_a_tree_falls_on_sleeping_beau...](http://lesswrong.com/lw/32o/if_a_tree_falls_on_sleeping_beauty/)

</comment><comment><USER5> It seems like it is playing a little fast and loose with the interpretation of hypothesis, or maybe is mixing Bayesian and frequentist reasoning in the same experiment. Either way, something seems off about the logic here (regarding the madman thought experiment). it smells a bit like a case of equivocation, but i can't put my finger on exactly what it is (and in shrouded in a sheath of mathematics)

</comment><comment><USER6> I have come to the view that the anthropic principle is a kind of egocentric reflection of the fact that selection doesn't just operate on systems that reproduce in time, but also on systems that are reproduced (with lots of variation) in space.</comment>


Ask HN: How long it took you to get your dream job. Why that long? - <AUTHOR>

<comment><USER2> From committing to the idea to getting the position, a little over 5 years.

I was working on a corporate ASP.Net site when in early 2009 I decided I wanted to be a game developer instead of pursuing my previous dream of being a musician. I had published a couple of terrible games on iOS but wasn't qualified to do AAA development. A few months later I quit my job and ended up getting a position doing research programming in the AI lab at the University of Michigan under John Laird, who teaches the game dev course there. After a year of that I enrolled in the Master's program. By the time I graduated two years later I had an offer to work at Microsoft, where I ended up doing game prototyping for almost a year before moving to the 343 Industries team, which I think marks me achieving my dream of becoming a AAA game developer.

</comment><comment><USER3> 38 years and still counting ;)

</comment><comment><USER4> Dream job = Entrepreneurship

<comment><AUTHOR> Yes, by job I didn't mean a company job - it can be whatever, entrepreneurship, non profit etc.

</comment></comment><comment><USER5> Hard to answer this question. Funny thing is, once you get your dream job, you don't stop having dreams.

<comment><USER6> I agree! Maybe, it is because of our tendency as humans to hedonic adaptation ([http://goo.gl/4EnTXL](http://goo.gl/4EnTXL))</comment></comment>


We indexed some top mobile apps and the SDKs they use - <AUTHOR> https://medium.com/@kevinleong789/here-are-the-sdks-top-mobile-apps-use-faf8e6e5cfee#.vssem43ud <comment><USER2> I analyzed the app frame works from the linked article here is the result when the count is larger than one, first column is usage count among popular apps.

"cat topmobilesdk.txt | tr "," "\n" | sed -e "s/^ //g" | sort | uniq -c | sort -n"

          2 Adjust SDK
          2 Formatter Kit
          2 Google Maps SDK
          2 Google+ SDK
          2 Hockey SDK
          2 Mantle Framework
          2 PLCrashReporter
          2 Pop Animation Engine
          2 PSPDFKit
          3 AFNetworking
          3 BPXLUUIDHandler
          3 FLAnimatedImage
          3 Google Conversion Tracking SDK
          3 Google Mobile Ads SDK
          3 Masonry
          3 SocketRocket
          3 Tune SDK (Mobile App Tracking)
          4 Card.io SDK
          4 Cocoa Lumberjack
          4 OpenCV
          4 TTTAttributedLabel
          5 Eigen
          6 Bolts Framework
          6 Crashlytics
          6 Fabric
          6 Facebook SDK

<comment><USER3> One more interesting take-away - Uber has both Adjust and TUNE(Mobile App Tracking)'s SDK. Lyft has both Kochava and TUNE(Mobile App Tracking)'s SDK. All these SDK's do Attribution analytics (where are my app installs coming from). I wonder if app installs are _so_ importnat to Uber and Lyft that they need 2 SDKs each doing the same thing to keep each other's data honest?

<comment><USER4> It's possible they are running advertising campaigns with ad networks that are only integrated with one of these attribution partners? I know I've started to see a lot of uber interstitial ads in games that I didn't use to.

<comment><USER3> That's a good point. Not all attribution vendors work with all ad networks so they might have to use a few to get coverage.

</comment></comment></comment></comment><comment><USER5> Can I make a request? Can you do a slice on the top 50 gaming apps? As a game maker, I'd be curious to see what are the most used SDKs.

<comment><AUTHOR> We'll probably throw all the data up on appsight.co site and let folks do the filtering themselves. Sign up on the site now and we'll alert you when we launch for Beta.

</comment></comment><comment><USER6> In case anyone wants a tabulated view. [https://gist.github.com/mdubbs/41c19c5384ee0386465cff19aa800...](https://gist.github.com/mdubbs/41c19c5384ee0386465cff19aa800e8d)

<comment><AUTHOR> Thanks! Btw, I think not all your counts are correct. AFNetworking has a count of 6, not 1.

<comment><USER6> Whoops! Should be fixed now, thanks.

</comment></comment></comment><comment><USER7> Hey there, nice work! I worked on a similar project for Android. Currently, there are about ~3k apps added: [http://www.libtastic.com/](http://www.libtastic.com/)

<comment><AUTHOR> Thx! Just checked out libtastic. Pretty fantastic :)

</comment></comment><comment><USER8> We have been doing SDK Recognition for a couple of years here at Apptopia, and it's absolutely not as easy as it's being portrayed here (i.e. run this GIT library). The technology requires constant attention, maintenance and improving otherwise you very quickly find yourself missing analysis for 65-70% of an app's SDKs. In fact, we have recently re-built our SDK Tech completely as we found previous methods (what we call "Binary Analysis") were extremely inaccurate and resulted in over 20% missed SDKs.

The comparison of Uber and Lyft is interesting. Our latest recognition for iOS shows Uber with 6 currently installed and 6 others uninstalled since 2013. Lyft made a lot of updates on May 9 and has 8 installed and 12 uninstalled. The only SDK that they are both currently using is SocketRocket.

Screenshots from our tool of the specific SDKs recognized: Uber: [https://www.evernote.com/l/Aro4rGREdBJI_a80JzIRKZ3z2P8BwM3SG...](https://www.evernote.com/l/Aro4rGREdBJI_a80JzIRKZ3z2P8BwM3SGrk) Lyft: [https://www.evernote.com/l/ArrWkfAF3OxGWqcsnrCRtgSnRKgSfwH0I...](https://www.evernote.com/l/ArrWkfAF3OxGWqcsnrCRtgSnRKgSfwH0I0k)

We offer free trials at www.apptopia.com where you can look at SDK data for over 1 million apps in 196 countries.

</comment><comment><USER9> Fascinating. On my wish list is which SDKs use location in some way. A real pain on iOS to understand permissions (super easy in Android)

</comment><comment><USER10> This is very similar data to what's available on our site, AppBrain. If you're interested in more like this, the top ad networks and developer tools are updated daily at [http://www.appbrain.com/stats/libraries](http://www.appbrain.com/stats/libraries) and each individual app also shows the SDKs it uses at bottom of the app page, eg. Instagram: [http://www.appbrain.com/app/instagram/com.instagram.android](http://www.appbrain.com/app/instagram/com.instagram.android)

It seems in the dataset from OP more different SDKs are detected than we do at AppBrain. We invite anyone to report missing SDKs will definitely review the current presented set to add our omissions!

</comment><comment><USER11> These lists leave out some important frameworks these apps are built on. For example, Pinterest links AsyncDisplayKit. I'm curious how this data is collected. If Keyword matching against known popular public frameworks, it might be worth expanding your queries to include all public CocoaPods.

<comment><AUTHOR> Yeah good catch. The SDK list we scan against isn't comprehensive. It doesn't have AsyncDisplayKit (I just added it to the list). Great suggestion on Cocoapods, we'll be adding those. We'll publish more comprehensive list on our site AppSight.co once we launch our closed Beta. Would love your feedback too at that point.

</comment></comment><comment><USER12> What could Uber and Lyft be using OpenCV for?

<comment><USER3> I had the same question too. I was thinking it was to process all the avatar images. Card.io dependency makes a lot more sense :)

</comment></comment><comment><USER3> This is some cool data. Am surprised how often AFNetworking and Crashlytics are used. Unsurprisingly, for advertising, FB and Google Ad SDKs totally dominates. I signed up. When will you be launching the site with full data?

<comment><USER13> Yes, I'm especially surprised that major players give their data to Twitter by including Crashlytics.

<comment><AUTHOR> Crashlytics is like the only bright spot out of the whole Twitter product suite. Or major players don't see Twitter as a threat at all anymore. Not like FB

</comment></comment></comment><comment><USER14> Surprised to see so much use of AFNetworking. Is it really necessary? Looking at the example code it doesn't seem to add much (these days), with the overhead of managing another sdk.

<comment><USER15> It used to be a dramatically better API than what the system provided. These days it's debatable as to whether it's actually better than just using NSURLSession directly, but AFNetworking is well established as the default that everyone who has ever touched an iOS application knows how to use, and there isn't really any incentive for ripping it out of existing applications (and if you have a mix of old and new applications it's easier to just use the same networking API in all of them, so it continues to be used for new ones as well).

</comment></comment><comment><USER16> Is this an indication of anything other than popularity.

With programming languages, the general rule seems to be the worst win in the popularity contests (VB, PHP, MySQL come to mind).

<comment><AUTHOR> I think it depends. You can draw some take aways if major apps are using a particular SDK - stability for example. Think about scaling. If uber has your SDK and you are logging events, you probably need a fairly meaningful infrastructure to do that.

</comment></comment><comment><USER17> Nice. Thumbs up. Would love to see a larger data set

</comment><comment><USER18> Cool. Interesting to see Starbucks make the list considering I don't see that app mentioned too much

<comment><AUTHOR> We indexed it exactly for that reason but also for the fact that it has one of the highest payment processing volume. So we were curious what payment processor they used.

<comment><USER18> I see. I remember reading about the Starbucks mobile payment system taking off even though they launched recently, I believe the Summer or fall of last year.

</comment></comment></comment><comment><USER19> Interesting insight. Interesting that Uber and Lyft are the ones with the most numeber of SDKs by far.

</comment><comment><USER20> This is pretty cool! The more data and/or apps the better!

<comment><AUTHOR> Thx! We think people will find this useful too. Feel free to spread the word and have folks sign up at AppSight.co. Will have a lot more comprehensive data when we launch!

</comment></comment><comment><USER21> Android?

<comment><AUTHOR> iOS only for now. Android coming very soon!</comment></comment>


Parents Are Forcing Tech Companies That Monitor Schoolkids to Delete the Data - <AUTHOR> https://www.theguardian.com/education/2019/dec/05/schools-monitor-students-online-activity <comment><USER3> Why do we as adults presume a child of say 6 or 7 years of age understands the context or ramifications of searching taboo terms on the Internet and subsequently put said child in trouble for being curious?

</comment><comment><USER4> > Like thousands of American public school districts, Montgomery county gives > students laptops and has hired tech companies to track students' activities > on those computers

This is one of the rare occasions where I am sympathetic to libertarians - <USER2> state takes money in the form of taxes, and returns it in the form of spyware- infected laptops. If parents could keep that money, they could buy laptops that would not betray them.

<comment><USER5> I have to kindly disagree for these reasons:

Cost: Not every parent can afford a laptop for their child, let alone children. When a state, county, or city (not sure where the taxes do come from) covers the cost, the many help the few and the cost burden is lower. Yes, some end up paying for what they don't use. (In theory this is where the balance of social programs come in).

Technical: Not every kid would end up with what they need for class. Example: Last year my perfectly good Ubuntu Laptop from Dell worked great, now my kid must use Windows because <insert proprietary software that doesn't work via wine>. Oh and the class requirements changed for the next years after one was already purchased so no planning ahead possible.

Hand-wavy Solution?: However, perhaps the People should push for anti-spying- on-kids legislation for those that already have laptops and states/counties/towns that are headed towards new laptops should have that type of legislation before the kids even get them. (IMHO this is where ideas akin to 'it takes a village to raise a child' can shine through.)

<comment><USER4> I'd be willing to accept school-issued laptops, _if_ any monitoring software is loudly announced, and the parents can modify/install/remove software as they wish. I.e. with the exception of needing to return them, it's _their_ laptop - no "it's okay if they spy on you because they're not really yours" justifications.</comment></comment></comment>


React v15.0 - <AUTHOR> https://facebook.github.io/react/blog/2016/04/07/react-v15.html <comment><USER2> The referenced pull request for removing the <span>s ([https://github.com/facebook/react/pull/5753](https://github.com/facebook/react/pull/5753)) is terrific - a real paragon of open-source professionalism.

<comment><USER3> Agreed. I'm really impressed by the professionalism shown by all members of the React ecosystem. Just the other day I had noticed a org member of the reactjs org (houses react-router, redux, etc) who was acting very unprofessional and expressed that it was a turn off seeing official members of the org basically telling a issue reporter/discussion was "annoying them" and Ryan Florence stepped up and said he was not tolerating that and even reached out to me on Twitter apologizing and saying that they do not condone such tones.

<comment><USER4> Out of interest, do you have a link to the github issue?

</comment></comment></comment><comment><USER5> Is there a reason why the file size has exploded?

react.min.js is now 145.4kb. For comparioson, angular.min.js is 155.2kb. One of React's biggest selling points was that it was much smaller than Angular/Ember/Backbone etc, but I'm not sure that argument can be leveraged anymore. I think React is a culmination of some great ideas, but in my view this is a big step back. Especially since Angular/Ember/etc. offer a lot more tools out of the box.

I don't intend to tell anyone here that React is a bad framework because it's not, but I think it should be a big priority to tighten down the file size. When React is big pretty much all their selling points go out the window.

Perhaps they can split files up so you can just import the APIs you want. I wonder how much that would help with the bloat.

I feel like some of the smaller VDOM frameworks like Cycle/Mithril/Riot are a lot more appealing now, since they focus on small file size and low bloat.

<comment><USER6> It is time to deploy common JS libraries out of band, like native libraries. Download them once, precompile them once, and then use precompiled version at any site, which requires it.

<comment><USER7> It would be kind of nice if you could import modules directly from HTTP sources, like Go. For example:

        import _ from package 'github.com/lodash/lodash/tree/4.8.2';
        import $ from package 'jquery.com/version/2.2.0';

Your browser would make the OOB requests and cache the results in a shared directory, which _any_ JS code that's running locally can use. This way, HTTP requests for 3rd-party components shared across websites aren't repeated over and over again for the same libraries. And it could even be extended to CSS:

        @import package("github.com/twbs/bootstrap/tree/3.3.6");

<comment><USER8> Better idea would be to reference it via cryptographic signature. Then you could host your own copy of lodash for example, but still benefit if a client already has it downloaded from a different site.

<comment><USER9> Subresource Integrity is the standard for that [https://www.w3.org/TR/SRI/](https://www.w3.org/TR/SRI/)

<comment><USER8> Do you know if this allows consolidation of multiple copies of the same resource from different hosts?

<comment><USER10> Not currently, because it conflicts with CSP. See [https://news.ycombinator.com/item?id=10311020](https://news.ycombinator.com/item?id=10311020)

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> If you are using typescript, you should read:

[https://github.com/Microsoft/TypeScript/issues/7270](https://github.com/Microsoft/TypeScript/issues/7270)

TLD'R: React 15.0 broke TSX in typescript but the workaround is easy.

<comment><USER12> FWIW I used TypeScript in anger with React, and hated it so much I rewrote all the React code in regular JavaScript. TypeScript and React, IMO, are a poor fit.

To be clear: I do like TypeScript and use it almost exclusively on the server, so this isn't a case of me "not liking TypeScript". It's the combination of the two. Basically, all of the stuff I did to keep TypeScript happy when using React was worthless overhead and brought nothing but pain, warping the codebase horribly. The JavaScript version with React was smaller, lighter, and had way less cognitive overhead. YMMV.

<comment><USER13> What is exactly your problem with TSX? I've been using TSX for a while so maybe I can help. The only thing that TypeScript really adds to JSX is component parameter validation, which is quite neat to be honest.

<comment><USER14> TypeScript also validates the expressions that supply the values. And if you use Visual Studio you get intellisense and validation for HTML, and intellisense and instant validation of your javascript expressions. These benefits are awesome, and by the way, all this is independent of React. You can use TSX just as a templating library using UIBuilder, see: [https://github.com/wisercoder/uibuilder](https://github.com/wisercoder/uibuilder)

</comment></comment></comment></comment><comment><USER15> Amazing. Honestly, React made me love developing for the web again. It's a huge component of our software, and is what allows us to move so rapidly.

Really happy to see the SVG support, and the continued fast-paced development of React. Onwards!

<comment><USER16> React made frontend much more enjoyale to me too, it's so quick to build nice things in it :)

</comment></comment><comment><USER3> As a recent converter from Angular to React, I'm so excited for this release. Great job Dan & React Core Team!

<comment><USER17> Angular two lost me.

<comment><USER3> Yeah I wasn't too thrilled with Angular 2.x either. I had always been watching React and liking it, but I had no experience with it. So one day I decided to give it a whirl and see how far I could get converting the Angular 1.5 app I had already started and get it to the equivalent state in React.

Long story short, I took 40+ hours of Angular 1.5 code and converted it to React code in a single 8 hour work day without any prior experience in React. I was both impressed and giddy. The next day I merged that reactify branch into my develop branch, and continued on.

I feel that React does components better. I feel that Angular 1.5's introduction of the component sugar over top directives and even components in Angular 2.x just felt shoehorned in as an afterthought.

I couldn't even spend a day trying to figure out how to componantize this app in Angular 2.x let alone 1.5.

Now I'm head deep in learning as much about the ecosystem as I can, Redux, Relay, GraphQL, all that good stuff. I've officially ditched Angular for the React ecosystem.

<comment><USER18> > I took 40+ hours of Angular 1.5 code and converted it to React code in a > single 8 hour work day

That doesn't seem like a fair comparison considering that when you first wrote that code, you probably had to figure out things unrelated to whatever framework you were using and didn't have to deal with that aspect when porting to React.

For example, if I rewrote a Django app in Pyramid, it wouldn't take as long as writing the original app even with the overhead of learning Pyramid, because I've already figured out the functional aspects of the app.

Generally speaking, learning frameworks isn't the hard part of writing an app, unless the app is trivial.

<comment><USER19> That goes without saying....

I took it as a statement of "someone new to the framework, can quickly rewrite their old code" if they came from Angular.

Which is an important metric.

When I converted a Backbone app to Angular, it took 3x as long simply because of the amount of ceremony and process that I had to go through to do things the idiomatic Angular way.

React on the other hand, as a view layer, is so dead simple that you can internalize its workings within hours of study. The complexity with React comes with everything else---Redux, React-Router, GraphQL, CSS in React, etc.

None of that stuff is defined as part of the React framework itself, but that's the only way you can make a "fair comparison" to Angular which is a full-stack framework.

</comment></comment></comment></comment></comment><comment><USER20> There's a bit of irony here, but in a good way: Switching back to DOM APIs (e.g. document.createElement) was a performance _boost_.

('cause the browser vendors have done a ton of awesome work speeding up DOM APIs in recent years)

<comment><USER21> I'm not sure how this is ironic? We were using one DOM API (.innerHTML) before; now we're using a different one.

<comment><USER22> I had no idea react used .innerHTML. That provides a significant performance drawback compared to other DOM APIs due to it requiring the browser to re-flow / re-render so much more. Glad you guys updated :)

<comment><USER21> We only used .innerHTML when creating new subtrees, which requires a reflow anyway when they're inserted. There are reasons that document.createElement is faster for us, but that's not one of them.

<comment><USER22> Even inserting new subtrees should be significantly faster via appendChild() versus innerHTML. The browser has to do a whole heck of a lot more when you don't build the nodes first.

I'm curious though. I'll have to check out the source to see how it works in React now. You know, when I find some free time :)

<comment><USER21> Yeah, you'd think that it would be a lot faster to avoid serializing and parsing everything, but it turns out to be not much difference. Not sure why.

<comment><USER23> The native API calls such as createElement are pretty expensive (in V8 it's like 10 function calls to finally reach the target C++ function) and you are making a lot more such calls when not using innerHTML.

<comment><USER21> Interesting, thanks. I had heard that calls from JS to C++ were cheap but calls from C++ to JS (such as if Array.prototype.map was implemented natively) were expensive.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER24> I just tried the new release. The data-reactid is indeed gone when rendering on the client side. But on the server side, the html string generate by ReactDOMServer.renderToString still has data-reactid. Is it because client side can validate the server result and decide if it needs to re-render the page?

<comment><USER21> Not quite -- it's so that the client can know which node is which. (When rendering on the client, it can just store a direct reference to each created node.)

</comment></comment><comment><USER25> > There were a number of large changes to our interactions with the DOM. One > of the most noticeable changes is that we no longer set the data-reactid > attribute for each DOM node.

Nice. I currently use another functional reactive virtual DOM data binding mechanism and the data attributes everywhere was one of the things that didn't appeal about react. Since react is getting crazy network effects from community I might check it out for my next project.

</comment><comment><USER26> Great to see React continue to improve and streamline as new versions come out. They've managed to avoid the huge bloat that happens to frameworks as they age. Kudos.

<comment><USER5> The thing is, they have bloated out the library quite a bit. One of their early selling points was how much smaller the React library was than Angular/Ember. Now the size of react.min.js has ballooned up to 145.4kb, which is only 10kb short of Angular, and with addons it's even bigger.

As someone who focuses heavily on small page size, that's a pretty harsh regression. Especially when compared to all the other VDom frameworks like Riot/Mithril/Cycle which are all much much smaller and achieve about as much (in my opinion).

<comment><USER21> (Replied to your top-level comment at [https://news.ycombinator.com/item?id=11451283.](https://news.ycombinator.com/item?id=11451283.))

</comment></comment></comment><comment><USER27> React 14.x was built with babel 5.x. React 15.x is compiled with babel 6.x. Must user jsx code now be compiled with babel 6.x to be compatible with React 15.x?

<comment><USER21> Nope, it shouldn't affect you at all. (Otherwise it would have been noted in the blog post -- that should have all the info you need to upgrade.)

</comment></comment><comment><USER28> Great news about the support for all the SVG tags and attributes. Now I can get rid of ugly dangerouslySetInnerHTML hacks for unsupported elements :)

</comment><comment><USER29> In ReactConf, Ben Alpert noted that there would be a ~10% performance improvement from v15.0. Is that still true?

<comment><USER21> Haven't benchmarked again, but it should be. This came from the document.createElement and reactid changes (and some associated internal refactors).

<comment><USER30> Am I correct in assuming that you're only talking about the production build being faster?

Going from 0.14.8 to 15.0 RC2 there seemed (subjectively) to be a noticeable increase in lag in some pages we're working on with a moderately large amount of content being rendered (things like tables or SVGs with a few thousand elements) when using development builds. However, looking at the changelog and how many new warnings seem to be included now, it doesn't seem unreasonable that those would slow things down a little overall even with the other changes.

As long as those aren't going to affect production, or if there were some final changes that have gone in now but weren't in RC2, it's not a big deal. However, if the React experts wouldn't expect a net slow-down even in a dev build, maybe we should look into it a bit more.

<comment><USER21> That's possible. We haven't done much optimization on the dev build but I was planning to look for low-hanging fruit soon to see if there are any significant improvements we can make. I wouldn't have expected 15.0 to be much different from 0.14 though.

<comment><USER30> OK, thanks. It doesn't seem to be a huge difference or anything, just enough to be a bit noticeable in some pages we're staring at for a disturbing number of hours per week at the moment, so I was curious. Compared to things like implementing shouldComponentUpdate in the right places, the change is tiny.

</comment></comment></comment></comment></comment><comment><USER31> Can anyone say what this SVG support means for d3.js integration?

<comment><USER32> No direct implications. But it should provide solid ground for libraries like Victory. [https://github.com/FormidableLabs/Victory](https://github.com/FormidableLabs/Victory)

<comment><USER33> Wow, this is an awesome library. Thanks for sharing this.

</comment></comment></comment><comment><USER34> Has Facebook removed the potentially dangerous patent restrictions?

[https://news.ycombinator.com/item?id=8985541](https://news.ycombinator.com/item?id=8985541)

<comment><USER21> We updated the patent _grant_ to be less restrictive in response to community feedback about a year ago (several months after the discussion you linked):

[https://github.com/facebook/react/commit/b8ba8c83f318b84e429...](https://github.com/facebook/react/commit/b8ba8c83f318b84e42933f6928f231dc0918f864)

Since that change I've heard few complaints, and I know for a fact that several large companies who were previously unwilling to use it are now content with the language.

<comment><USER35> I noticed you italicized the word _grant_. In reality the "grant" isn't granting; it is taking away. Let me explain:

If React didn't have an explicit "grant" there would be an implicit grant. Is the explicit grant better than an implicit grant? It isn't because the explicit grant has what is known as a _strong_ _retaliation_ _clause_.

More info on weak vs strong retaliation clauses: [http://www.rosenlaw.com/lj9.htm](http://www.rosenlaw.com/lj9.htm)

React's patent "grant" gives you a license to React's patents. This sounds like a good thing, however, the "grant" has a "strong retaliation clause" which says that if you make any sort of patent claim against Facebook this patent grant automatically terminates.

Which means Facebook can now sue you for patent infringement for using React. You may think this is no worse than not having a patent grant at all. But that's not the case. If React didn't have an explicit patent grant then there would be an implicit grant which does not have any retaliation clauses, and cannot be revoked.

If you work for a software company and your company has patents then keep in mind that by using React you are giving Facebook a free license to your entire patent portfolio.

<comment><USER21> > If React didn't have an explicit patent grant then there would be an > implicit grant which does not have any retaliation clauses, and cannot be > revoked.

I'm not a lawyer. My understanding is that this is hypothetically possible but stands on shaky legal ground and has no legal precedent. Whether or not an implicit grant exists seems to depend on who you ask. This is apparently why GPLv3 includes an explicit grant.

It could also be that there aren't any patents that cover React. I don't personally know of any that do.

<comment><USER36> >It could also be that there aren't any patents that cover React.

If that is the case, why have this controversial patent clause at all?

I haven't read them all, but Facebook seems to have a number of patents which at first glance are so broad they could apply to react:

[https://patents.google.com/patent/US20160092096A1/en](https://patents.google.com/patent/US20160092096A1/en) "a method performed by one or more computing devices including defining a hierarchical structure for a user interface (UI) that includes defining one or more layers of the hierarchical structure, adding one or more objects at each layer, and specifying one or more relationships among particular objects"

[https://patents.google.com/patent/US20160091953A1/en](https://patents.google.com/patent/US20160091953A1/en) "Adjusting mobile device state based on user intentions and/or identity"

[https://patents.google.com/patent/US20150277691A1/en](https://patents.google.com/patent/US20150277691A1/en) "In one embodiment, as a user is scrolling through a first series of content items and reaches the nth content items from the first series of content items, display a visual indication that there are additional content items from the first series of content items existing after the nth content item."

[https://patents.google.com/patent/US20150113066A1/en](https://patents.google.com/patent/US20150113066A1/en) "A communications system including one or more alert gates and an alert controller. Each alert gate is configured to detect a different type of alert feed corresponding to a particular kind of alert. The alert controller is connected to the alert gates and operable to receive detected alerts from the alert gates and to deliver the detected alerts to a user of the communications system."

[https://patents.google.com/patent/US20160086219A1/en](https://patents.google.com/patent/US20160086219A1/en) "methods and systems that allows for improved user navigation within a group of content items."

I actually had no idea Facebook's patent portfolio was so broad. There are countless more: [https://patents.google.com/?q=G06F3%2f048&assignee=Facebook+...](https://patents.google.com/?q=G06F3%2f048&assignee=Facebook+Inc)

<comment><USER12> That list of patents reads like an Onion article. Is there a joke I'm missing here?

<comment><USER37> Did you also read the patent's text, or just their title and high-level description?

<comment><USER38> Even the text of some are pretty much as expected. It seems that they have a patent on single page applications (SPAs) to-a-T [1], elaborating that an application may use mechanisms like XHR and append an identifier to the fragment portion of the URL to keep history.

Even if I subscribed to the idea that this was without prior art on the web in 2009, equivalent functionality already existed in desktop applications. To claim that it is a novel invention because it now works on a web page is disheartening. It's exactly this kind of behavior--claiming new ownership based on each abstraction--that leads to restrictions in some countries on patents which can only exist in software.

[1] [https://www.google.com/patents/US20110055314](https://www.google.com/patents/US20110055314)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER39> Interesting renaming choice. ReactPerf.getWasted()

:D

<comment><USER40> Yeah I was hesitant about this one. At least it will be consistent with other methods (we'll add getInclusive(), getExclusive() etc).

</comment></comment><comment><USER41> Really nice work with all this! Great blog post on update

</comment><comment><USER42> Is the `key` prop still required?

<comment><USER21> Yes, it is. React uses it to track identity of components in a list; it's significant especially when reordering children and affects both which DOM nodes get reused as well as whether this.state is preserved.</comment></comment>


Statistical Flaw Punctuates Brain Research in Elite Journals - <AUTHOR> http://blogs.scientificamerican.com/talking-back/2014/03/27/statistical-flaw-punctuates-brain-research-in-elite-journals/

<comment><USER2> The problem mentioned--mistreating "nested data" by assuming independence between observations that may actually be linked--is definitely a real one and it's important to bring up, but I don't think it's as severe as suggested here.

How did the authors come up with their numbers on which papers had nested designs? I would argue that nested data is on a continuum and it's a matter of judgment and interpretation what to call a singular "research object".

If you are studying a single protein molecule, when can you start assuming independence? Protein molecules in the same subcellular compartment? same cell? same cluster of cells? same organ section? same animal? same litter? same species? same animal tech at the animal supply company? same experimental session? same solution stock?

It's not obvious where to draw the line. As long as authors are clear and honest about where they decided to draw that line in their analysis, I can assess whether I want to put stock in their conclusions.</comment>


Robert Baden-Powell's Entomological Intrigues - <AUTHOR> http://publicdomainreview.org/2013/07/10/robert-baden-powells-entomological-intrigues/

<comment><AUTHOR> "In 1915 Robert Baden-Powell, founder of the worldwide Scouts movement, published his DIY guide to espionage, My Adventures as a Spy"</comment>


Pingdom and DeadMansSnitch vs. WDT.io – price comparison - <AUTHOR> http://docs.wdt.io/price_comparison.pdf <comment><AUTHOR> Using [https://wdt.io](https://wdt.io) for monitoring HTTP servers and cron jobs is often less expensive than a combination of DeadMansSnitch and Pingdom.</comment>


Jonah Lehrer: The Future Of Reading - <AUTHOR> http://www.wired.com/wiredscience/2010/09/the-future-of-reading-2/

<comment><USER2> I'm not familiar with the research cited in the article, however it seems the author is making some inconsistent points refering to it.

He says there are two different ways of reading(ventral route and dorsal stream) and that the latter way is preferable because it is more conscious of the actual text being read. When reading on digital devices he worries that we will only be reading through the ventral route, because the clear display quality will make it so easy for us the read the letters and words.

Now, there is a few things I'm wondering about: In practice, do we actually ever read through the dorsal stream? In the research experiments quoted, this was archived through rotating letters or errant punctuation, both of which we do not usually encounter in actual books.

The author also mentions the dorsal stream being activated because of an obscure word, or an awkard subclause. This seems more like the thing that would in practice cause us to use the dorsal stream. Is the dorsal stream active in cases of difficult texts, e.g. scientific papers?

If that is so, then it seems to me that it is not so much a question of wether you read on a digital device with great display quality or a simple book, but instead it seems to be of much greater importance what it is that you are reading. A Dan Brown book, would probably only activate the ventral route on both mediums, whereas the newest P[!|=]=NP proof would probably always activate the dorsal stream. Is my understanding of the ventral route and dorsal stream correct?</comment>


Ask HN: BigQuery vs. Redshift vs. Athena vs. Snowflake - <AUTHOR> I'm investigating potential hosted SQL data warehouses for ad-hoc analytical queries. I've read quite a bit about all 4 of these technologies and their high-level tradeoffs, but I'd love to get people's first-person accounts of the pros and cons of each.
Dimensions I care about: query performance, load performance, cost, stability, maintainability, features (e.g. views, UDF), limitations (e.g. concurrent queries, schema restrictions), and all of these at scale.
If you've used any of these and have thoughts, would appreciate hearing them, thanks! <comment><USER3> Based on random use over the past few years.

Redshift: Pros: Has the most adoption, so most integrations from SaaS services etc. are built with Redshift as their sink. Relatively fast and battle-tested.

Cons: In an awkward middle ground where you're responsible for a lot of operations (e.g. capacity planning, setting up indexes), but don't have a lot of visibility. Some weirdness as a result of taking PostgreSQL and making it distributed.

BigQuery: Pros: Rich feature set. Pay-per-TB pricing. Recently released standard-ish SQL dialect. Very fast.

Cons: JDBC driver is recent and doesn't have support for e.g. CREATE TABLE AS SELECT (as of a couple of months ago) so harder to integrate with existing systems. There are ways to run out of resources (e.g. large ORDER BY results) without a good path to throw more money at the problem.

Athena: Pros: Built off of the open-source Presto database so can use the documentation there. Pay-per-TB pricing.

Cons: Slower than the other options listed here. Very early product so lacking some in documentation and some cryptic errors. Not a lot of extensibility, but you could theoretically move to just using open-source Presto.

Haven't had a chance to evaluate Snowflake.

<comment><USER4> Just a question, which of these does have a good BI tool support?

<comment><USER5> All of the above data warehouses have good support. You can use almost any popular BI tool with all of them. All of the above have ODBC/JDBC drivers that BI tools can use. And since these are the most popular data warehouses out there, most BI tools implement their connectors.

</comment></comment></comment><comment><USER6> My perspective on a brief trial of BigQuery and RedShift:

1) RedShift is PostgreSQL 8 with some additional features, and while can significantly improve some query runtimes, comes with usual DBA burdens as an on-premise database.

2) BiqQuery is a true cloud implementation. Its SQL syntax is evolving and may not have some features older platforms do, but the concepts of the physical implementation is completely removed from the user's concern.

Were I going to start a new project without existing dependence on AWS, BigQuery would have an advantage.

</comment><comment><USER7> BigQuery for data storage + Looker for visualization.

RedShift is not a real managed service. You have to handle all the maintenance/scaling/indexing/provisioning, it's a hassle.

</comment><comment><USER8> We have ~80TB of (compressed) data in Snowflake at Celtra and I'm working with Snowflake on a daily basis. We've been using it for the last ~1 year in production. Overall maintenance is minimal and the product is very stable.

Pros:

      - Support for semi-structured nested data (think json, avro, parquet) and querying this in-database with
      custom operators
      - Separation of compute from storage. Since S3 is used for storage, you can just spawn as many compute
      clusters as needed - no congestion for resources.
      - CLONE capability. Basically, Snowflake allows you to do a zero-copy CLONE, which copies just the metadata,
      but not the actual data (you can clone a whole database, a particular schema or a particular table). This is
      particularly useful for QA scenarios, because you don't need to retain/backup/copy over a large table - <USER2>
      just CLONE and can run some ALTERs on the clone of the data. Truth be told, there are some privilege bugs
      there, but I've already reported those and Snowflake is working on them.
      - Support for UDFs and Javascript UDFs. We've had to do a full ~80TB table rewrite and being able to do this
      without copying data outside of Snowflake was a massive gain.
      - Pricing model. We did not like query-based model of BigQuery a lot, because it's harder to control the costs.
      Storage on Snowflake costs the same as S3 ($27/TB compressed), BigQuery charges for scans of uncompressed data.
      - Database-level atomicity and transactions (instead of table-level on BigQuery)
      - Seamless S3 integration. With BigQuery, we'd have to copy all data over to GCS first.
      - JDBC/ODBC connectivity. At the time we were evaluating Snowflake vs. BigQuery (1.5 years ago, BigQuery didn't
      support JDBC)
      - You can define separate ACLs for storage and compute
      - Snowflake was faster when the data size scanned was smaller (GBs)
      - Concurrent DML (insert into the same table from multiple processes - locking happens on a partition level)
      - Vendor support
      - ADD COLUMN, DROP COLUMN, RENAME all work as you would expect from a columnar database
      - Some cool in-database analytics functions, like HyperLogLog objects (that are aggregatable)

Cons:

      - Nested data is not first-class. It's supported by semi-structured VARIANT data type, but there is no schema
      if you use this. So you can't have nested data + define a schema both at the same time, you have to pick just
      one.
      - Snowflake uses a proprietary data storage format and you can't access data directly (even though it sits on
      S3). For example when using Snowflake-Spark connector, there is a lot of copying of data going on: S3 ->
      Snowflake -> S3 -> Spark cluster, instead of just S3 -> Spark cluster.
      - BigQuery was faster for full table scans (TBs)
      - Does not release locks if connection drops. It's pain to handle that yourself, especially if you can't
      control the clients which are killed.
      - No indexes. Also no materialized views. Snowflake allows you to define Clustering keys, which will retain
      sort order (not global!), but it has certain bugs and we've not been using it seriously yet. Particularly,
      it doesn't seem to be suited for small tables, or tables with frequent small inserts, as it doesn't do file
      compaction (number of files just grows, which hits performance).
      - Which brings me to the next point. If your use case is more streaming in nature (more frequent inserts, but
      smaller ones), I don't think Snowflake would handle this well. For one use case, we're inserting every minute,
      and we're having problems with number of files. For another use case, we're ingesting once per hour, and this
      works okay.

Some (non-obvious) limitations:

      - 50 concurrent queries/user
      - 150 concurrent queries/account
      - streaming use cases (look above)
      - 1s connection times on ODBC driver (JDBC seems to be better)

If you decide for Snowflake or have some more questions, I can help with more specific questions/use cases.

<comment><USER9> Here's some useful updates and additional information on some of the items mentioned above regarding Snowflake:

\- The concurrency limits mentioned above are soft limits that can be raised on customer request (those defaults are there so that runaway applications can be detected easily). Snowflake can handle very high concurrency--we have customers running hundreds of concurrent queries.

\- We've recently released a new Spark connector with a bunch of optimizations, including additional push-down capabilities that speed up performance significantly.

\- The clustering capability is currently in "preview", we're definitely taking input and have been working on incorporating feedback we've received so far into it.

\- One important thing to note when it comes to full table scans is that Snowflake allows you to choose how much horsepower you apply to the job, so you can easily adjust the horsepower to get faster scans.

<comment><USER9> Forgot to state the disclaimer that I work for Snowflake.

</comment></comment></comment><comment><USER10> What does the rest of your infrastructure look like? Are you running on AWS already? or on GCP?

<comment><AUTHOR> In AWS, so we understand there will be some extra costs sending data to BigQuery.

</comment></comment><comment><USER11> I worked on the BigQuery team and happy to answer any questions you may have. There are quite a few AWS customers running on BigQuery. (See [0], [1] [7], [8] )

I'll offer some resources:

\- A session from last Google NEXT, where Yahoo, NY Times, and Blue Apron detail their experience with migrating to BigQuery [0]

\- A session from previous Google NEXT, where Kabam Games detailed their experience with migrating to BigQuery (and running the rest of their workload on AWS) [1]

\- A blog "15 things you probably didn't know about BigQuery" [2]

\- A blog detailing what BigQuery means by "fully managed" [3]

\- A blog offering some details on most recent iteration of Dremel "In-memory query execution in Google BigQuery" [4]

\- A blog describing Capacitor, the opinionated storage format that replaced ColumnIO (on which Parquet is based) [5]

\- "BigQuery Under The Hood" [6]

\- Since you specifically asked about Load Performance, [9]

( A wholly redundant disclosure that I work at Google Cloud)

[0] [https://www.youtube.com/watch?v=TLpfGaYWshw](https://www.youtube.com/watch?v=TLpfGaYWshw)

[1] [https://www.youtube.com/watch?v=6Nv18xmJirs](https://www.youtube.com/watch?v=6Nv18xmJirs)

[2] [https://medium.com/google-cloud/15-awesome-things-you- probab...](https://medium.com/google-cloud/15-awesome-things-you-probably- didnt-know-about-google-bigquery-6654841fa2dc)

[3] [https://cloud.google.com/blog/big-data/2016/08/google- bigque...](https://cloud.google.com/blog/big-data/2016/08/google-bigquery- continues-to-define-what-it-means-to-be-fully-managed)

[4] [https://cloud.google.com/blog/big-data/2016/08/in-memory- que...](https://cloud.google.com/blog/big-data/2016/08/in-memory-query- execution-in-google-bigquery)

[5] [https://cloud.google.com/blog/big-data/2016/04/inside- capaci...](https://cloud.google.com/blog/big-data/2016/04/inside-capacitor- bigquerys-next-generation-columnar-storage-format)

[6] [https://cloud.google.com/blog/big-data/2016/01/bigquery- unde...](https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the- hood)

[7] [https://cloud.google.com/customers/sharethis/](https://cloud.google.com/customers/sharethis/)

[8] [https://storify.com/felipehoffa/dow-jones-on- bigquery](https://storify.com/felipehoffa/dow-jones-on-bigquery)

[9] [https://medium.com/google-cloud/paying-it-forward-how- bigque...](https://medium.com/google-cloud/paying-it-forward-how-bigquerys- data-ingest-breaks-tech-norms-8bfe2341f5eb)

<comment><AUTHOR> Thanks for the wealth of resources. After reading through, the only big concern I have with BigQuery is pricing. I like the model of pay-per-query, but it seems like costs can get out of hand easily.

These two issues particularly worry me:

\- No non-date partitioning, which means you'll often end up (in my particular use case) scanning 10-100x as much data as you actually want to.

\- Pricing is based on uncompressed data size, which is at least several times more expensive than the alternative (e.g. Athena).

Any thoughts on these? We'll have several hundreds of TB of data, so cost matters.

<comment><USER11> Happy to help!

On partitioning - sounds like a great candidate for sharding. BigQuery standard SQL makes it very easy to run unions on shards (ex. Select * from tables_*)

On pricing - BigQuery is a very different product functionally, and I encourage you to compare it via a poc. That said, definitely reach out to me on price.

</comment></comment></comment><comment><USER10> Disclaimer: I'm the co-founder of a start-up that provides performance analytics for Amazon Redshift. So I'm biased towards using Redshift because otherwise I can't sell you our service. With that cleared, some considerations.

One of our customers I thought made a great observation. "I see a lot of people using Redshift. But I don't see anybody using it happily". Because of all the issues that people point out in this thread. lots of operations. little visibility ("black box"). All very true. It's the problem our product solves, and let me talk about how we see companies successfully using Redshift at scale. (and that customer is now a happy Redshift user, btw).

Here's the architecture that we're seeing companies on AWS moving to:

\- all your data in S3 (I hate the term, but call it a "data lake")

\- a subset of your data in Redshift, for ongoing analysis

\- daily batch jobs to move data in and out of Redshift from / to S3

\- Athena to query data that's in S3 and not in Redshift

The subset of the data sitting in Redshift is determined by your needs / use cases. For example, let's say you have 3 years of data, but your users only query data that's less than 6 months old. Then moving data older than 6 months to S3 makes a lot of sense. Much cheaper. For the edge cases where a users does want to query data older than 6 months, you use Athena to query data sitting in S3.

Why not use Athena for everything? Two reasons. For one, Athena is less mature. And then you still need a place to run your transformations, and Redshift is the better choice for that.

Three uses cases for Redshift:

1) classic BI / reporting ("the past")

2) log analysis ("here and now")

3) predictive apps ("the future")

Your use case will determine how you have to think about your data architecture. For some it's ok if a daily report takes a few minutes to run. I'll go get a coffee. It also means my batch jobs are running on daily or hourly cycles. But if I'm running a scoring model, e.g. for fraud prevention, I want that score to be as fresh / real-time as possible. And make sure that the transformations leading up to that score have executed in their proper order.

For Redshift to work at scale, there are really only three key points you need to check the box on:

\- setting up your WLM queues so that your important workloads are isolated from each other

\- allocating enough concurrency and memory to each queue to enable high throughput

\- monitoring your stack for real-time changes that can affect those WLM settings

That's it. And then Redshift has a rich ecosystem and rich feature set to enable all uses cases (etl vendors, dashboard tools, etc.). Those users who have addressed the performance challenges that do come at scale for Redshift - they're happy users and never looked back.

On the pay-per-TB pricing for BigQuery and Snowflake. I think that's more of a marketing spin. Most companies we work with are compute-bound. So per-TB pricing helps them very little. They want to crunch their data as fast as possible. More CPUs give them more I/O. If you feel you're storage-bound - take a hard look at the data that really needs to be available in Redshift for analysis, and move everything else to S3.

For inspiration, watch the AWS Reinvent videos on S3 / Redshift / Athena from Netflix and NASDAQ. Like this one:

[https://www.youtube.com/watch?v=o52vMQ4Ey9I&t=256s](https://www.youtube.com/watch?v=o52vMQ4Ey9I&t=256s)

If you're entire data is already within AWS, I think Redshift is the way to go. But then again I'm biased :)

<comment><USER11> > On the pay-per-TB pricing for BigQuery and Snowflake. I think that's more of > a marketing spin. Most companies we work with are compute-bound. So per-TB > pricing helps them very little. They want to crunch their data as fast as > possible. More CPUs give them more I/O. If you feel you're storage-bound - > take a hard look at the data that really needs to be available in Redshift > for analysis, and move everything else to S3.

Do you mind elaborating on what you mean by that? I've worked a whole lot on BigQuery pricing, and am a little confused by your comment. Do you mean that separation of storage compute is an overrated concept?

(work on Google Cloud)

<comment><USER12> To clarify, Snowflake doesn't charge for processing per-TB like BigQuery, but for the (fully elastic) compute time.

(disclaimer: I work at Snowflake)

<comment><USER11> Thanks for the clarification!

BigQuery's on-demand model charges just for the resources consumed during the job execution (via a per-TB proxy), rather than resources provisioned. This is the highest order of cloud-native pricing models, and good on Athena for doing the same!

If one were to equate this to VM pricing, you're getting:

\- The ability to scale from 0 to thousands of cores in roughly 1 second (what it takes for Dremel to put together the execution plan)

\- Pipelined execution (meaning, each processing stage ONLY uses the resources required for that stage, rather than net resources. think of this as per- second autoscaling from query stage to query stage. This is part of the reason why BigQuery does very well in concurrency tests)

\- Equivalent of per-second billing

This in practice equates to 100% resource efficiency - never pay for resources you just deploy and not use.

I've written on this topic at [0] and at [1] and happy to chat further!

(And BigQuery did introduce a Flat Rate pricing model last Fall for the more enterprisey crowd)

[0] [https://cloud.google.com/blog/big- data/2016/02/understanding...](https://cloud.google.com/blog/big- data/2016/02/understanding-bigquerys-rapid-scaling-and-simple-pricing)

[1] [https://cloud.google.com/blog/big- data/2016/02/visualizing-t...](https://cloud.google.com/blog/big- data/2016/02/visualizing-the-mechanics-of-on-demand-pricing-in-big-data- technologies)

<comment><USER10> this is a very elaborate and insightful answer. appreciate the clarification.

</comment></comment></comment></comment></comment><comment><USER13> BigQuery is like a midrange luxury car, easy to do anything, looks great, and good performance. Redshift is like someone's turbo boosted Honda Civic. Athena is maybe like a Camaro, You dislike the concept, but they maybe made it powerful and easy enough that you want it. This is probably no help.

</comment><comment><USER14> I've used BQ, RS and Athena. Their benchmark results and links to how I conducted benchmarks on each can be found here: [http://tech.marksblogg.com/benchmarks.html](http://tech.marksblogg.com/benchmarks.html)

I could probably write a book on my experiences but I'll keep this short.

BigQuery requires the least amount of work to keep performant but this comes at a cost (this cost matches Athena).

Redshift seems to have support for just about every BI tool out there including Periscope which I've found to be popular with my customers. The downside is that you've really got to think about how you layout your data, how you write your queries and there is something of a weekly task of bringing up the console and hunting down the worst-offending queries for optimisation.

Keep in mind if you're already on Google Cloud or AWS you could incur bandwidth charges moving data to the other provider.</comment>


UX Problems Holding Back Crypto And Anti-Wiretapping Technology - <AUTHOR> http://falkvinge.net/2013/06/08/4-ux-problems-holding-back-crypto-and-anti-wiretapping-technology/

<comment><USER2> Good points by the OP.

I've railed about similar things. Most notably, VPNs are a UX disaster. And that is a shame because the fundamental technology behind VPNs would be a prerequisite to a model that I describe as "PAO" [1] wherein I host my own omnipresent applications and data. The failure of VPNs to deliver a user experience that casual users can manage is one of the many contributors to the rise of the traditional cloud. I contend that regular people would be perfectly fine buying and replacing hard drives in a home disk array if the experience were straight-forward (it's anything but). Similarly, regular people would be fine buying Internet connectivity with high upload speed if this was part of ensuring secure, (nearly) everywhere access to their data. Regular people would be fine joining a federated backup cartel with friends and family if it were easy to set up.

The savvy can do many of these things with today's mediocre UX, but the tremendous amount of money to be made by solving the problem using proprietary centralized servers explains why R&D has been directed toward the cloud as opposed to self-hosted application servers.

Diving into minutia for a moment, in light of this week's news, I also recently ranted that it's a real shame that GPG suffers from one of the worst user experiences across all modern software. PGP from 10 to 15 years ago was a better user experience than GPG in 2013. The Thunderbird+GPG interface is absolutely horrific and even when I know I should encrypt a particular communication, when faced with the reality of getting my recipient configured with GPG, I flinch at the certain pain and suffering. I concede, "well, it's probably not needed anyway--who is likely to be listening?"

[1] [http://tiamat.tsotech.com/pao](http://tiamat.tsotech.com/pao)

<comment><USER3> The GPG UI is ultimately a relatively minor issue when it comes to mass adoption. There have been several PGP-like products with decent UIs and good mail integration, none has enjoyed mass-market success. Creating another one wrapping an existing crypto library would be no epic undertaking.

The real problem is key management. Any system which depends on users to think about keys, protect their keys, back up their keys, is doomed.

The fix is going to have to involve cheap, idiot-proof hardware. When crypto key management is the same as house key management, people will start to use it, hopefully in a semi-correct manner.

<comment><USER4> > There have been several PGP-like products with decent UIs and good mail > integration, none has enjoyed mass-market success.

iMessage comes to mind. It is the most popular realtime messaging system in the world behind SMS, and is end-to-end encrypted with per-device keys.

Of course, it's centralized, as is the key management, so Apple can subvert it at the behest of the government. Crypto isn't always 100% of the answer.

<comment><USER3> iMessage is not PGP-like. Its centralized and Apple-device-specific nature is practically PGP's antithesis. It also fails several key feature checks, like the ability to encrypt or sign arbitrary files.

</comment></comment></comment></comment><comment><USER5> See also "Why Johnny Can't Encrypt: A Usability Study of PGP 5.0", Whitten et al [http://www.gaudior.net/alma/johnny.pdf](http://www.gaudior.net/alma/johnny.pdf)

and its followup, "Why Johnny Still Can't Encrypt: Evaluating the Usability of Email Encryption Software", Sheng et al [http://cups.cs.cmu.edu/soups/2006/posters/sheng- poster_abstr...](http://cups.cs.cmu.edu/soups/2006/posters/sheng- poster_abstract.pdf)

</comment><comment><USER6> We developers, especially those of us who consider ourselves activists, have been too slow to move this stuff forward. I'm fairly certain that is because most of us are "savvy enough" to use the tools that exist, etc. The movement has been growing to evolve these tools to catch up with the needs (well...) of modern users, but that evolution is really just beginning.

One among many such projects, I am a developer at LEAP ( [https://leap.se](https://leap.se) ). We are working on this very problem. We're getting ready for public beta of our Encrypted Internet Proxy ( VPN for now, Tor and more features to come) and will be rolling out truly end-to-end secure email, IM, SMS, and voice. Also calendar, contacts, and possibly password management. All client encrypted. All syncing across your devices. All in an Open Source, Trust No One, user friendly way.

There are many tools and services out there already, but the ones that the technology un-savvy can use happily mostly run in a centralized fashion, requiring that you trust your service provider. No different, except in mission statement, than what people use today with big mail or chat providers and social networks. Would it were not so, but we live in an era where that trust is a vulnerability that we are seeing exploited.

</comment><comment><USER7> Another thing: p2p is totally broken. You can work around it with central servers, but then [ _cough_ skype ] you have a dependence on a big server farm that costs a bunch of money. In the long run, they won't keep you safe.

As an addendum, at least with home connections the technically inclined could badger their routers into allowing incoming connections. Given that computers (that can run long-running background services) and land lines are going to be the exclusive domain of the rich or the programmers in a few years, there is essentially no hope at all of fixing this.

<comment><USER8> >Given that computers (that can run long-running background services) and land lines are going to be the exclusive domain of the rich or the programmers in a few years, there is essentially no hope at all of fixing this.

Not nessasarilly. I suspect that in home wifi will continue to be common, which means that homes would continue to have a wired connection to the internet. If a company develops and markets a product to be a 'personal cloud', then a typical consumer should be able to use it. The requirements for such a server (for a typicall user) are pretty low, and computers continue to become cheaper, so the price shouldn't be prohibitive for a large market. Even the UI for the server itself seems relativly simple. Essentially all you need is a good package management system, and for the individual p2p 'apps' to have easy to use configuration. Once IPv6 takes off, it would be trivial to these machines globally routable, then it is either a matter of allowing consumers to easilly get a DNS to their server, or creating some easy way to tell devices where to look (this is probably the hardest part).

The main problem in getting this adopted, I suspect, would be that it does not provide enough additional value over the centralized servers for mass adoption. Combine this with the network effects of lock-in, and you would need serious value add for this to work.

<comment><USER7> > Not nessasarilly. I suspect that in home wifi will continue to be common

Home WiFi has been widespread for at least a decade, and the UX for bypassing NAT is worse than ever. UPnP still comes disabled or broken, if at all.

You could build a better experience, but you will never, ever get Microsoft, Apple, Google, Cisco, or Belkin to cooperate, so we're still DOA- the curious need to learn a shitload of scary tech or buy new equipment.

<comment><USER9> You could build a fairly universal application for flashing DD-WRT or a similar firmware on an available router, and supply updates for new routers or flashing mechanisms through a web service.

Another concern is standalone routers becoming less and less common as ISPs supply routers.

<comment><USER10> > You could build a fairly universal application for flashing DD-WRT

That is definitely not the way to get mass adoption. Way too much friction.

It's the difference between installing Ubuntu from DVD and buying an Android device: you need to sell it preinstalled for mass adoption.

</comment></comment></comment></comment></comment><comment><USER11> Good article, but I'd like to disagree with the notion that what Google and bing are doing with long-term tracking is necessary to have good search results. blekko has its own crawl and index, doesn't do any per-user tracking, and doesn't even save unconnected and anonymized clicks if you have DNT set. We don't use super-long personal session data to pick whether we show you Fox News or the NYT article on Prism. Our results could certainly be improved, but having a bigger crawl and more unconnected and anonymous clicks is how we'd improve them, not by making a huge database which could easily be used to reconstruct our users' lives going back years.

</comment><comment><USER12> It is so easy to blame software, is it?

1\. If you are not able to manage your keys, which includes a simple unencrypted key-backup, you are doomed, even with a pretty interface.

2\. If you insist on other people to take away the responsibility from you, no service, hard- or software will provide security or privacy for you.

3\. You don't need a cloud, when decentralized or local services crater you much better and faster.

4\. Sharing data isn't really a problem, it's the lack of transparency of the data collecting entity that creates the problem.

The conclusion is, to protect your security/privacy and to successfully utilize any crypto-toolchain you have to accept that responsibility isn't comfy, and delegation will lead to compromise.

Did you really believe that any government, or parts thereof, in the world would not tap the vast resources that packet-switched networks, centralized services would offer them?

Zacqary knows nothing about security, secure procedures or even cryprography.

If you delegate your private, unsecured communications to an entity for a comfortable user experience, you did not want either security and privacy.

Sorry, to tell you so.

<comment><USER10> > If you are not able to manage your keys, which includes a simple unencrypted > key-backup, you are doomed, even with a pretty interface.

Too many hackers equate "good user experience" with "pretty interface". It's a mistake.

A good user experience really _would_ abstract the complexity out of key management, without compromising security. This is one of the most important outstanding problems you could solve: find a way to leverage people's existing meatspace notions of trust in a _concrete_ way to build a robust PKI.

If you can make the whole process by the side-effect of a game, so much the better.

And give them keys that are physical things -- they know how to protect valuable physical things, and they know who they can trust with a backup copy.

None of this is easy, but it's vitally important.

</comment></comment><comment><USER13> Check out what my friends at CozyCloud are working on: [https://www.cozycloud.cc/](https://www.cozycloud.cc/)

They want to be your user-friendly personal cloud. Easy as an an App Store. Open source. Host it wherever you want. Your data and your apps are yours.

<comment><USER14> "Host it wherever you want" \- unless it's physically self-hosted, it's not truly "yours" is it?

<comment><USER15> You can host it on your own machine. Even if you host it via an hosting provider, you can export all your data and your apps to any new servers like an home server for example!

</comment></comment></comment><comment><USER16> Space Monkey? [http://spacemonkey.com](http://spacemonkey.com)</comment>


What are Bloom filters? - <AUTHOR> https://medium.com/the-story/what-are-bloom-filters-1ec2a50c68ff <comment><USER2> Like everyone else, I too wrote a blog post on Bloom Filters - [http://prakhar.me/articles/bloom-filters-for- dummies/](http://prakhar.me/articles/bloom-filters-for-dummies/).

<comment><USER3> I preferred yours, but I'm not sure what you had for dinner on your last birthday.

<comment><USER2> Haha! What is that supposed to mean? :D

<comment><USER4> You didn't include non-important life details when explaining what a bloom- filter is.

 _> I sigh. I'm hungry and the main course has just arrived — venison glazed in honey, served with a sweet potato hash._

I learned more about what the author had for dinner on their last birthday than I did about bloom filters in the first paragraphs.

So while they prefer your blog - they aren't sure what you had for dinner on your last birthday...

</comment></comment></comment></comment><comment><USER5> If Bloom filters look interesting, you should probably also check out cuckoo filters.

[https://www.cs.cmu.edu/~dga/papers/cuckoo- conext2014.pdf](https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf)

The differences are somewhat subtle, but the use cases overlap a lot.

<comment><USER6> Would it be safe to say that Cuckoo filters are bloom filters with some additional overhead for the ability to add and remove?

<comment><USER5> Deletion is the big difference, but even without that I think there can be other significant differences as well. For any given number of items, it can be pretty unclear which one will give better performance (including the cost of false positives) for bounded size, or better space efficiency for bounded performance. Usually you'll need at least an accurate model for both, if not an actual implementation.

</comment></comment></comment><comment><USER7> I'm beginning to suspect that if I want to really drive a whack of traffic to my blog, I should write a post about how to implement Bloom filters using monads.

<comment><USER8> For maximum impact, you'll need to choose your programming language carefully. I would suggest Elm with a sprinkling of Coq, though I think using either K or MUMPS would earn you a lot of kudos in some quarters.

</comment></comment><comment><USER9> trying too hard.

Just go to [https://en.wikipedia.org/wiki/Bloom_filter](https://en.wikipedia.org/wiki/Bloom_filter)

It's clearer and more informative.

</comment><comment><USER10> _> To understand Bloom filters, you first have to understand hashing._

As pedagogy, I think this is the wrong approach.

The author _already knows_ Bloom filters and therefore, it seems like the most logical thing to first talk about is hash functions because _that 's how it's implemented_.

Unfortunately, that's not how a person brand new to the concept thinks about it. The first thing to talk about is _motivations_ and _scenarios_ and _use cases_.

For that, the first 2 paragraphs of wikipedia article[1] on Bloom filters is fairly straightforward. It explains _why_ it's an interesting technique. Imo, those paragraphs are a better introduction than the author's dive right into "hash functions" immediately after unrelated blurbs of " _I put my fork down_ " and " _my wife shakes her head with a rueful smile._ "

[1][https://en.wikipedia.org/wiki/Bloom_filter](https://en.wikipedia.org/wiki/Bloom_filter)

<comment><USER11> I'm fine with explaining the motivation first, but I struggle to see how you can explain bloom filters without eventually getting to hashing. Obviously you don't need to explain any actual real-world hash function, but you need to explain the concept of deterministically mapping X bits to Y bits where X > Y, which is essentially what a hash function does.

<comment><USER12> I don't think at the level of the article that it's inappropriate to assume that the reader is at least familiar with the term "hashing", and can wait on the technical details of the particular types of hashing functions needed for bloom filters until the second or third section.

I honestly just closed the article when he started in on hashing, because I want to know about bloom filters (which I don't know about), not the basics of hashing (which I learned as an undergrad, and need to know day-to-day as a working programmer).

</comment></comment></comment><comment><USER13> Anyone happen to have a great Bloom filter tutorial in Ruby geared towards early programmers?

I read the article, but am still not grasping the full construct and how it functions. I'm hoping a hands on tutorial might give me a better sense.

</comment><comment><USER14> Is anybody else irritated with the incredibly tryhard tone in this article? "venison glazed in honey, served with a sweet potato hash"? Seriously??? It's an article about bloom filters for god's sake, not Leveraged Sell Out for the tech industry...

<comment><USER15> Much like most TED talks, this is in line with the common Medium article theme. The article becomes more about ME ME ME, rather than the subject at hand. It's like some Moses syndrome.

When it comes to technical articles this starts to become extremely annoying to read blogs/papers as if it was written for BuzzFeed.

</comment></comment><comment><USER16> Can anyone give some cases of where Bloom filters are used? Has anyone used one in their job?

<comment><USER17> I've programmed bloomfilters for my job.

A certain product of ours keeps track of certain urls visited. We're talking millions of (unique) urls. We use bloomfilters to quickly check if a url was visited or not. If the bloomfilter search is positive a more expensive search inside a log file begins that gives a conclusive result (since bloomfilters have a (very) small false positive-rate, but we want to be perfectly sure).

<comment><USER18> > bloomfilters have a (very) small false positive-rate

One of the neat things about Bloom filters is that you can choose your own false positive rate, by tuning the number of hashes and the storage size.

<comment><USER19> Calculator: [http://hur.st/bloomfilter](http://hur.st/bloomfilter)</comment></comment></comment></comment>