Ad Nauseam - <AUTHOR> http://www.hackerfactor.com/blog/index.php?/archives/713-Ad-Nauseam.html <comment><USER2> Stop calling them ad blockers. They block surveillance features that advertisers put in their ads. I do not believe that ads would be blocked by surveillance blockers if they were just ads. Absent the surveillance, how would they recognize them? If ads were identical to the ads in analogue newspapers then surveillance blockers would let them through. When I read an analogue newspaper or magazine nobody is knowing if I read the ads or not. They only know whether they get results from advertising in that outlet. And they know that by tracking. They could run ads that don't spy on people. I always use surveillance blockers. I never shut them down for anybody. It is especially offensive when sites that are hardcopy outfits which have gone digital to keep up with the times complain about surveillance blockers. They certainly know how to run ads that are just ads. There is no technical difference between the ads and the rest of the page. The layout is the thing. They have experience with this and professional advertising people know about tracking results.

<comment><USER3> Ad blockers dont block ads. They arent even "ad" blockers but just filters - browser extensions that block anything on a page from loading if its coming from a blacklisted domain. Whether it's ads, trackers, social widgets or just an image - it gets blocked by canceling the network request.

Surveillance is not the leading cause for adblock, it's because people don't like ads and a 1-click install to remove them is incredibly easy.

Advertising online will always have some sort of tracking because that is the benefit of advertising online - to know the real metrics of who has seen and clicked and engaged with an ad. If you're worried about real privacy issues, you should focus on Facebook/Google and government agencies.

<comment><USER2> I don't use facebook or google. I use startpage a a google proxy on rare occasions, but mostly search with ixquick. My representatives know what I think about government surveillance and privacy issues, for what its worth. I agree and disagree about online ads always having interactive features. It makes sense, but the controversy around this does make for the possibility that some will go for straight ads. For me surveillance is the main reason I use blockers. History shows that corporations are more of a threat than government. Hitler could never have made it absent the Krupp family. They put him in power. I don't know about Stalin. He did have help along the way, but since he ruled till he died there is a lot less information available, and it is spotty at best. Of course in America the corporations run the government, and it isn't just from campaign contributions, though at this time that is the biggest thing. Europe to. Most of the world's governments are run by corporations behind the scenes. The big possibility to force ads to be non interactive is to seriously call newspapers and magazines that predate the internet on this. I do this pretty regularly. An ad that does not have any interactivity cannot be filtered without removing the article one is reading. That is easy to do. And the incredible amount of publicity waiting for the first newspaper or magazine to require advertisers to eliminate interactive ads should be really tempting. If their publicity department doesn't see this they need to hire me! Given the resources that publicity outfits have I could get the ball rolling and keep it going for long enough that it would be remembered for a while after the boom. The only thing to filter would be the graphics. Alt text would deal with this. If on has use of the advertised item(s), one will check out the pictures. However, I noticed when I "upgraded" my firefox that they don't have the option to block images in the new version, or at least they make it difficult by completely removing the tool bar that held that function. I've looked and looked, but not found. In any event, I utterly lack sympathy for interactive advertisers and the websites that allow them. I'll happily do without them as they go out of business, and I hope they do!

</comment></comment></comment><comment><USER4> and again Flash is the scapegoat

"By converting unsafe flash-based ads to safe HTML5 ads, they lower the risk of infection from a hostile ad." is laughable at best

An Ad Network is one of the fastest way to deliver a payload to a lot of users

Don't fool yourself, Operating Systems, Browsers and HTML5/JS also have a hell lot of CVE that can be exploited

It's funny how a company like Google making Billions from ads, having ton of smart engineers, have never figured out during the last decade how to "scan ads for malware".

It's not like anyone can upload an ad to those big network, or that they don't QA the ads before delivering them ...

Imagine this unlikely scenario: malware delivered by HTML5/JS

I guess we'll all have to run for the hills if that happen

<comment><USER5> "and again Flash is the scapegoat"

Truth hurts? Adobe Flash and Microsoft Silverlight are common exploit paths because they have new critical exploits every few days. Here's the CVE list for Flash -- notice how many critical exploits there are? It averages to about 1 every 3 days. [https://www.cvedetails.com/vulnerability- list/vendor_id-53/p...](https://www.cvedetails.com/vulnerability- list/vendor_id-53/product_id-6761/Adobe-Flash-Player.html)

In contrast, JavaScript itself has been pretty stable for years. I think the last vulnerability related to JavaScript ES5 impacted old Firefox browsers. [http://www.cvedetails.com/cve/CVE-2015-4516/](http://www.cvedetails.com/cve/CVE-2015-4516/) [https://www.cvedetails.com/vulnerability- list/vendor_id-452/...](https://www.cvedetails.com/vulnerability- list/vendor_id-452/product_id-3264/cvssscoremin-7/cvssscoremax-7.99/Mozilla- Firefox.html) (Two JavaScript exploits for Firefox in 2015, both low risk.)

And HTML5? Extremely stable. There may be specific plugins or specific browsers that are vulnerable, but the underlying HTML5 specifications are very safe and have been safe for years. [https://www.cvedetails.com/google-search- results.php?q=html5...](https://www.cvedetails.com/google-search- results.php?q=html5&sa=Search)

If you know otherwise, then please cite the specific CVEs. Otherwise, you're just spreading false information. You wrote, "Browsers and HTML5/JS also have a hell lot of CVE that can be exploited". I say: Prove it. Cite your sources.

Edit: Adding links to Firefox exploit CVEs.

<comment><USER4> OK, remember you asked for it

"If you know otherwise, then please cite the specific CVEs. Otherwise, you're just spreading false information"

man, you are so full of it

want proof ? no problemo

1\. CVE are organised by vendors and products

HTML and JS does not show as products, only browsers

see [http://www.cvedetails.com/top-50-products.php](http://www.cvedetails.com/top-50-products.php)

look #3 Firefox, #4 Chrome, #8 IE

that explains why you will never see a specific HTML and/or JS CVE, that does not mean they don't exists.

Also in term of volume, browsers have more CVE than Flash, it's all here in the numbers: Firefox 1320, Chrome 1216, but no let's ignore them and focus on Flash 713 CVE.

Just that it make your whole argument biased, the part "JavaScript itself has been pretty stable for years" is ridiculous, search for JS blackhole exploit, Rowhammer.js exploit, Heap Overflow exploit in JS, etc. you don't see them in CVE but they are here and exploitable.

It's better to think than JS is secure looking at that [http://www.cvedetails.com/vendor/10288/Javascript.html](http://www.cvedetails.com/vendor/10288/Javascript.html)

yeah no exploit in JS, none, we are all safe LOL

this for example [http://www.cvedetails.com/cve/CVE-2015-0817/](http://www.cvedetails.com/cve/CVE-2015-0817/) [http://cve.mitre.org/cgi- bin/cvename.cgi?name=CVE-2015-0817](http://cve.mitre.org/cgi- bin/cvename.cgi?name=CVE-2015-0817) [https://www.mozilla.org/en- US/security/advisories/mfsa2015-2...](https://www.mozilla.org/en- US/security/advisories/mfsa2015-29/)

you don't see it show up under the tag "JavaScript"

2\. Number of CVE listed do no equals CVE exploited in the wild

so you say "It averages to about 1 every 3 days", that's completely false

1 vendor patch for a particular product can close numerous CVE at the same time so it's more like "we squashed 50 CVE in 1 day"

look at [http://www.cvedetails.com/cve/CVE-2015-8449/](http://www.cvedetails.com/cve/CVE-2015-8449/)

follow up on [https://helpx.adobe.com/security/products/flash- player/apsb1...](https://helpx.adobe.com/security/products/flash- player/apsb15-32.html)

that's 1 patch, it does not indicate 1 CVE every 3 days, look at the details

"These updates resolve use-after-free vulnerabilities that could lead to code execution (CVE-2015-8050, ..." that's more than 50 CVE of the same type patched and closed at the same time

Also look the "Acknowledgments", numerous security team reported all those CVE for them to be patched, there is no indications they were exploited in the wild.

Saying such things as "oh 30 CVE discovered in 1 month, so that means there were 1 CVE per day" is totally misleading, even more misleading to assume all those CVE were exploited by default (eg. "could lead to").

At best it indicates that they (Adobe and other security team) are more serious about discovering and patching those CVE and so they close more of them more often.

</comment></comment></comment><comment><USER6> Or a third way, everybody hides content behind paywall, hail the new web 3.0. Maybe not a bad thing, subscription could bring the old qualified journalism back the in the print era.

If you don't think anti ad blocker is a problem, where is this article coming from? Hmmm, afraid that more websites would follow the trend so less content to read? The attitude that this is only websites and advertisers' problem is not as constructive as the author might presume.

<comment><USER7> Yet the news sites demonstrate how utterly clueless they are with the amount they set their online subscriptions to.

£1 a day for The Times - very nearly the cost of the actual paper. $1 daily to access Wired. Don't make me laugh. No one consumes all their news from a single source any more.

If my usage pattern is anything near representative, 2-5p a day for the Times and .5p a day for Wired, based on how often I visit equivalent sites and how many stories I read whilst there.

Seems like unless it's something very specialised (medical journal or similar), or the FT charging as though it was our sole news source just demonstrates how out of touch they are.

Sure, charge me £1-£2 a day for consumption, but that would have to be spread across 50-100 sites daily, some of which I've visited just once in the last year, for one article. AND, if I am going to be willing to be micro-charged I want a way to NOT pay a specific site (perhaps I visited and the content was poor). Make that happen I'll subscribe today.

Ask me for £1 for your shitty site daily and you'll wait forever, but good luck with your greed - that's what caused the adpocalypse in the first place.

<comment><USER6> Hmmm, so you now understand advertising is not a evil business really, right?

It is effectively a way to price the information, how much should be paid for your view. Note in print days, you still pay your subscription, yet you get shit loads of ads. And you have a variety choices of publishers.

So why this is the worst model ever?

The article is laughable that it gives no solution, but asks publishers to evolve into oblivion, which I think they won't.

Some people are so pissed that publisher got anti ad blocker in place, yet claim they won't pay to their shitty articles whatsoever. But then again, if you don't read those shitty articles that much, why are you so pissed in the first place?

After all you need to pay what you consume, and ad is one way of it. It is not perfect, nor evil. Your call then.

<comment><USER8> _> Some people are so pissed that publisher got anti ad blocker in place .... if you don't read those shitty articles that much, why are you so pissed in the first place?_

I'm not pissed about the anti ad-block. I'm pissed because the sites show up when I'm searching in the first place. I'm pissed because I go to the site thinking I can get the information I was teased with in my search only to find out I've been tricked. I'm forced to do something (unblock the ads), accept some fake implicit agreement (you agree to look at our ads), and be spied on (all the trackers) before I can get to the content I was lead to believe was there.

The fix is to remove all blocked content from the search so we won't even know it exists in the first place. We won't get upset, we won't get blocked, the sites won't get content "stolen" by those who won't view or click the ads to being with. Everybody's happy. Win Win.

<comment><USER6> Even paywall article get indexed and showed, I dont think anti ad block justify the cause.

</comment></comment></comment></comment></comment><comment><USER9> "I think that we need to hold the web sites accountable for the content that they display. If browsers get infected by ads at Forbes or people buy knock- off watches from ads at Yahoo, then we need people to sue Forbes and Yahoo. Remember: these web sites authorized the placement of the ad on their web page."

So effectively what you're saying is that we should eliminate ad networks. There is no reasonable way to screen every ad before it is shown when using an ad network. So in order to be safe from lawsuits, publishers would have to go back to directly contracting with advertisers for every ad. Certainly there would be some benefits to that in terms of reduced low quality ads. The problem is, the added overhead of doing so would put many small publishers out of business. Dealing with individual advertisers is a huge job, with massive economies of scale; it just doesn't make sense for websites that are orders of magnitude smaller than Forbes and Yahoo.

<comment><USER10> > So effectively what you're saying is that we should eliminate ad networks. > There is no reasonable way to screen every ad before it is shown when using > an ad network.

Or you could have ad networks that only circulate carefully vetted/curated ads.

Imagine if you had an ad network that was picky and only allowed ads that were clever/interesting, short, not annoying, and didn't lead to malicious/fake products!

<comment><USER4> "Or you could have ad networks that only circulate carefully vetted/curated ads."

No, you make it simpler than that

you simply forbid ads to be interactive or to contain any code

eg. you do only static ads like text, image, video

no code, no way to hide nasty stuff

<comment><USER5> Your proposed approach will stop direct risks to browsers, but does nothing for ads that link to web pages that are hostile. E.g., you click on an ad because you are interested in the product and get directed to a phishing site or a site offering counterfeit goods or a site that has malware and infects your browser.

It's not just the graphic used by the ad, it's also the ad's destination.

<comment><USER11> So disable linking.

The clickthrough rate on internet ads is execrable. Frequently in the fractions of a percent at best.

No other advertising space operates on the assumption that linking represents.

Eliminating linking and leaving pure visual ads would be in line with every other form of advertising in existence, and eliminate the "problem" of click fraud, link-bait, and actually fraudulent links.

Do we really need a business model that exists largely to enable ad networks to defraud each other and consumers? We have advertising standards bodies that are meant to prevent this kind of thing in every other form of advertising, but somehow the internet is "special"?

<comment><USER12> So you end up with ads that say: copy/paste this URL. What have you solved?

<comment><USER11> I think realistically, the friction against such a method is strong. People can scarcely be arsed to bother with QR-codes anymore. It could still happen, but this sounds an awful lot like a "perfect is the enemy of the good" sort of argument. Is not some X% of the problem better than the 100% that we have now?

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> The publishers aren't very innocent themselves. Clickbait articles are nowadays not a dirty strategy to get ad views, but to be expected. Then you have the endemic tracking going on, the tracking that has become intertwined with viewing an ad. Being tracked and seeing an ad... it's the same thing! Nowhere else in the real world but on Internet is this to be expected. And it's a debate that is frustratingly only discussed in organizations like the EFF, never lifted to the general public.

The mafia comparison feels much more like a stretch when talking of ad blockers than when talking of the bulk of the world's news sites secretly (unless inspecting network traffic or HTML code) using a common few advertisement agencies.

I think the recent cookie laws feel pretty useless, especially since cookies aren't nasty by themselves. "Hi! This site uses cookies! Click here to learn more." It doesn't tell me anything. It doesn't imply that the site is evil nor good. However, give me a law requiring web sites to say "Hi! We are part of a tracking network where your behavior on this site will be registered." Then we're talking. Where the link doesn't lead to an explanation by the publisher, but be required to lead to a link on an external part with an easily digestible, up front explanation of what an ad tracker does and can do. I'm honestly quite fed up that this offensive behavior can keep going on behind the scenes. All people see are photos of a new car model. A normal ad that is anything but normal.

For as long as there is this World Wild West on the publishers' sides, I'm not going to change my behavior on defending myself. Because I look at this as a form of defense. It's simply like running antivirus tools on Windows. I wouldn't want a trojan horse to be downloaded that uploads my browsing behavior to some server either. The difference from what these guys are doing seems razor-thin.

<comment><USER14> It actually depends on the publisher, the french news site nextinpact[1] listen to criticism of ads, looked into the issues and made their move: ads are limited to display format, no behavioral targeting, no animation, no mixing content and ads, no tracker on client-side (tracking is on server side with a locally installed piwik).

The mafia comparison is targeting adblock plus for their "do your ads as we say, give us a 30% cut of the money you make and we may whitelist your ads (only if you're big enough as in at least 10m ad impressions)" feature[3].

[1]: [http://www.nextinpact.com/blog/97835-pourquoi-next-inpact- ar...](http://www.nextinpact.com/blog/97835-pourquoi-next-inpact-arrete- publicite-classique-et-passe-au-https-pour-tous.htm) [2]: [http://www.nextinpact.com/publicite- partenariat](http://www.nextinpact.com/publicite-partenariat) [3]: [http://www.theguardian.com/technology/2016/feb/25/adblock- pl...](http://www.theguardian.com/technology/2016/feb/25/adblock-plus-opens- up-acceptable-ads-work)

</comment></comment><comment><USER9> "But keep in mind: not everyone is Google, not ever web site has a huge amount of traffic. With online ads, payment is usually tied to the click-through rate (CTR). The CTR is typically around 1% (actual percentage varies by web site). So if 100 people visit your web page, then 1 person will probably click on the ad, generating a fraction of a cent. If each click pays $0.001, then you need 1000 clicks to earn $1. And if 1% of uses click, then that's about 100,000 visitors."

I'm not sure where these numbers come from, but unless you are in fact running a spam site, and likely even then, revenue per click is going to be higher than a fraction of a cent. As a random data point, it looks like the combined revenue per click from Adsense on our sites is around 30 cents per click at the moment.

<comment><USER3> Yes, much of the info in this article seems completely made up. The cheapest clicks in the entire industry are probably around 2-3c and these are the worst of the worst. Many are far higher, probably somewhere between 20-70c and going up to $$ depending on ad quality and conversion ROI.

</comment></comment><comment><USER15> I admit it, I look at linkbait articles and sites... The problem is that it's crossed a line that there are so many intruisive ads, that the web doesn't work without blockers.

If I happen to click on an article from facebook on my phone, the resulting page shouldn't be something I can't even scroll/read because it's so riddled with ads.

Another part is an extension of what TFA says... they should be held responsible... current techniques are iframes, and when a timeout occurs or it bounces to another ad network, another layer of iframe and tracking scripts runs... if an average ad is 3 layers of iframes, and an average page has 5-8 ads, that'd 15-24 complete extra browser contexts just for ads...

</comment><comment><USER16> Just to throw out this idea I've been thinking about for a bit as a ad-golden- rule: The other end of an ad has to have an identifiable person attached (ideally a citizen or from a non-poor country): An ad-auditor.

So now the other end of the ad is not faceless/identiti-less. If the ad is found to serve malware, there's someone to ban/take action against (like banning from a good-paying ad-audit job for life). Ad-networks that require the golden rule can be white-listed by blockers, and become trusted. Networks that don't are considered malware haven.

Could this work? In the current ad-blocking war, the use of ad-blockers will only rise-and-rise, and something has got to give.

<comment><USER9> The problem right now is that it can often be difficult for anyone to tell where an ad ultimately came from. A user will complain to a publisher about a bad ad. So the publisher gets a description of the ad, maybe a url that it points to. The problem is the url goes through various redirects, so the one that the user ended up on isn't the one the ad points to, so it can't be searched. But a bigger problem is where to even search. The unit housing that ad will probably be served by multiple networks, with an algorithm deciding which to serve from based on how many ads that user has seen, where they're located, and various other criteria. Then if that network doesn't have a sufficiently valuable ad to serve, they will either farm it off to another network (and so on), or pass it back to the publisher's ad server, which will find another network to serve an ad there.

This all happens in real time. So the point is, when you get a report of a bad ad on your page, it's almost impossible to even know what network it came from. The networks themselves don't know if they ultimately served you the ad, because maybe they got it from someplace else. And no one can search for it based on the url anyway.

Now, none of those things is unsolvable, although it would take significant new regulation. For instance, when an ad is served through a network, there should be a standardized way to add metadata to the ad to state that it was served via that network. In cases where it is passed through several networks, it would carry each of their metadata, in order from the original source through the various levels until the network that actually serves the ad to the publisher. That would at least allow savvy users to make an informed report to a publisher when they get a bad ad. Something else to look at might be requiring that either 1) the target url of an ad points directly to the eventual landing page, or 2) if a redirect is made, the original url be encoded either in the new url (as a fragment id perhaps) or at least as metadata in the page. There are probably plenty of caveats there. But if a user clicks on an ad and finds themselves at some page, there should be _some_ way to figure out what ad took them there. That isn't currently the case.

Identifying the networks an ad has passed through would be the responsibility of those networks (with a standardized way of doing so). Avoiding or identifying redirects would be the responsibility of the advertisers, but networks would have to be required to periodically test ads for compliance.

<comment><USER16> With the ad-auditor idea, the model itself changes. The middlemen go away, and there's a person at the other end. Yes, the business of ads will change per this new model.

An analogy is financial-auditors -- a human has to be present and sign even if the report is for a company behind 10 shell companies.

</comment></comment></comment><comment><USER17> One point: by far most websites are on a pay by impression rather than pay by click. More predictable revenue.

</comment><comment><USER14> The article misses a major point, ads are a form a surveillance and as such more and more people prefer to block them out of privacy concerns.

<comment><USER5> Actually, I intentionally left that out. I thought 4000 words was long enough. Surveillance would add in another 4000 words.

</comment></comment><comment><USER18> One solution. We will watch ads. The revenue will be taken by micropayment site like Flattr and distribute to sites we visit.</comment>


Udemy's Marketing Department are letting them down - <AUTHOR> http://tosbourn.com/udemy-fail/

<comment><USER2> "I got back to Udemy and said I would check out the guide in detail, I pointed out the spelling mistake and mentioned that it makes me wary that their may be typos in the code."

Their, their, tsk, tsk!</comment>


Cookie Stuffing with Google Adsense - <AUTHOR> http://www.benedelman.org/news/050712-1.html

<comment><USER2> As the article points out, they must be making good money for this attack to be practical. The back of the envelope calculation seems reasonable: even paying Google reasonably high CPMs they're making it back via Amazon through even casual browsing.

One of the issues is that legitimate affiliates can't tell when their affiliate revenue has been stolen. The cookie lasts 24 hours and any Amazon transaction occurs offsite at a later time -- they have no idea what to expect re: compensation. For that reason Amazon may not be chasing it down as eagerly as Google would with Adwords. With Adwords users can far more easily see when they're being scammed.</comment>


JetBrains releases pre-beta Objective-C IDE - <AUTHOR> http://www.jetbrains.com/objc/

<comment><USER2> Why?

<comment><AUTHOR> One big, big reason for me that this is useful is I can now install IdeaVIM and have a real VIM environment for editing Objective-C with code completion.

The JetBrains guys are wizards and likely will disrupt XCode the way they did to Visual Studio.</comment></comment>


Why isn't there a good Scheme/Lisp on LLVM? - <AUTHOR> http://stackoverflow.com/questions/2143605/why-isnt-there-a-good-scheme-lisp-on-llvm

<comment><USER2> it's not on llvm, but have you tried clojure? it can call java code and runs on the jvm. it's pretty cool.</comment>


Who doesn't read books in America? - <AUTHOR> https://www.pewresearch.org/fact-tank/2019/09/26/who-doesnt-read-books-in-america/ <comment><USER2> Unfortunately, I count myself in that 27% due to not having the time/mental energy after putting in 12+ hours a day 5 days a week and then taking care of the house and family on the weekends.

</comment><comment><USER3> Tl;dr: Men, non-whites, old people, poor people, people with less formal education, and rural people.

In other words, the demographic that reads the most is young, urban, college- educated, economically comfortable, white women.</comment>


These Indian Fishermen ick plastic from Oceans to help build Roads - <AUTHOR> https://www.weforum.org/agenda/2018/06/these-indian-fishermen-take-plastic-out-of-the-sea-and-use-it-to-build-roads <comment><AUTHOR> [https://www.weforum.org/agenda/2018/06/these-indian- fisherme...](https://www.weforum.org/agenda/2018/06/these-indian-fishermen- take-plastic-out-of-the-sea-and-use-it-to-build-roads)</comment>


The rise of protein drinks for ordinary people - <AUTHOR> http://www.bbc.co.uk/news/magazine-22753620

<comment><USER2> you just have to be careful about traces of metals. Muscle Milk for example has a large amount of arcenic/cadmium/lead. research before you drink!</comment>


Why the Return Trip Feels Shorter Than the Way There - <AUTHOR> http://www.citylab.com/commute/2015/06/why-the-trip-back-always-feels-shorter/395714/ <comment><USER2> I wonder if the destination matters: Going to a fair or theme park, actually, even going to work, most people would be preoccupied with anticipation and planning, some sort of excitation, while going home, your mind is in a reflective/contemplative/relaxed state. In the former state, you're more acutely aware of not being there yet, while less so in the latter.

<comment><USER3> It's interesting that you mentioned theme parks, because the title reminded me of the illusion created on Disneyland's Main Street by varying the heights of the buildings. At the front of the park, the upper level is close to full size, but as they get closer to the castle the upper floor shrinks.

As a result, the street looks longer as you enter the park full of excitement, and shorter as you leave it exhausted at the end of the day.

</comment></comment><comment><USER4> I notice it very much when walking in my day to day life, although it also happens in car rides. As some others have expressed, it feels to me like my brain is more active and focused on the destination, whereas I'm more likely tired and docile on my way back, slipping into the rhythm of the trip. Specifically, I'm thinking of this in the context of a night out (alcohol or not), where incredibly long walks are feasible on the way back. Obviously, there are other psychological factors that influence perception at this point, but I notice the exact same effects all the time, just slightly less pronounced.

I also wonder about what I'll call the "Lifetime" effect. Consider how car rides in general or long wait times at a doctors office start to feel shorter as you grow up (infinitely long as a child, pleasantly short as an adult). Intuitively, this is because of your life experience (or lack there of) and the fraction of your life-to-date that any given wait takes up. The time lengths stay relatively constant, while they progressively eat up a shrinking percentage of your life-to-date. Perhaps this same effect could happen over the course of a day, or a couple weeks, or an hour. The initial trip consists of 100% of your experience thus far until you get there and subsequently turn around, but at that point any task necessarily takes a smaller proportion of the overall time than the initial trip felt like (and was) at the time.

Perhaps something like that would explain it as well.

</comment><comment><USER5> Reminds me of a proposed explanation for [https://en.wikipedia.org/wiki/Moon_illusion](https://en.wikipedia.org/wiki/Moon_illusion)

We think it's further away at the horizon, so should be smaller, but it's not smaller, so we think it's closer, and therefore think it's bigger.

</comment><comment><USER6> I've been flying between Brazil and Europe a lot and I can tell you both ways feel _very_ long.

Someone needs to resurrect the supersonic passenger carrier.

<comment><USER7> I've put some serious thought into going the opposite direction, and reviving dirigibles instead. Sort of a cruise ship experience, when you need to cross the Atlantic but you're not in a particular hurry.

Unfortunately, I don't think the economics really work out. Except maybe as an expensive novelty.

<comment><USER6> Dirigibles are not particularly sturdy. I'd be worried to travel great distances on one, specially in regions where bad weather is common.

</comment></comment></comment><comment><USER8> I used to make frequent 4.5 hour drives between Austin and S.E. Texas. The drive home always feels twice as long, not shorter. There's an excitement about the place I'm visiting and so the trip there feels quick. The drive home is after the fun has been had and now I just want to be home as quickly as possible. As a result that trip feels so much longer. My experience has always been the opposite than demonstrated in the article.

<comment><USER9> I think there is another effect at play during a return trip, at least for me. I used to regularly tackle a 900 mile (1,800 mile roundtrip) haul between home and college and it felt longer with each trip.

The trip felt longer as my familiarity with the route increased. I reached a point where I knew _exactly_ how far I still had to travel at any point and the close I got to my destination, the more frustrating it became. I'd feel like I was on the home stretch as I knew my surroundings intimately but I'd still have 1-2 hours left to go.

<comment><USER8> Yeah, driving into Austin at night after being on the road for hours was like this. We would see the lights of the city spread out across the horizon. It would feel so close, yet we still had an hour drive ahead of us.

</comment></comment></comment><comment><USER10> In dangerous situations, the brain works extra hard to process what's happening, perhaps so you can deal with a similar situation later. This can cause a "slow motion" effect, where your brain, by laying down memories more "densely", slows the subjective passage of time.

I think a less extreme form of this may account for the return trip phenomenon. I notice (very subjectively) that trips seem t take longer if: * I'm stressed about getting there on time. * I'm going somewhere unfamiliar (often the outbound part of a trip) * Either of the other 2, plus NOT using a nav system to do the thinking for me, so I have to look for signs, landmarks, etc.

In other words, the more actively your brain is involved in the drive, the longer it seems to take.

These days, I can just put on a podcast, put the destination in my phone, and feel like the trip barely took any time.

</comment><comment><USER11> A related effect I've noticed (perhaps a kind of "expectation" effect) is that it seems to take much longer when travelling over uninteresting terrain. I've recently moved from Toronto to a smaller city that seems to be one large suburb and even though Google tells me the distances I now travel are less, they _feel_ so much longer. I'm convinced this effect is due to visual stimulation, a kind of time compression through distraction. This is why I prefer travelling through a city, even if I'm not always travelling quickly.

</comment><comment><USER12> I've always wondered about this!

In my everyday experience I've always noticed a difference in this effect between:

1) my normal home<>work commute (15 minutes by bike) I don't feel a discernible difference in the time it takes (maybe because the trip to work is downhill and effectively shorter than the return home, which is uphill. the difference in speed compensating for the effect maybe?)

2) a holiday trip (100-200 Km by car) Here I definitely always feel the return trip to be much shorter

<comment><USER13> I wondered about my return trip while commuting on a bike and later used a device to track speed, location, etc while riding and found that it was actually about 6 minutes shorter because it was slightly downhill most of the way (but not noticeable) and I was travelling faster for the duration.

<comment><USER12> yeah definitely, non flat terrain, trips by bike are not the best to measure this effect =)

</comment></comment></comment><comment><USER14> Because the route is cached! :)

<comment><USER15> I was thinking the jet stream at your tail actually does accelerate the return trip quite a bit.

</comment></comment><comment><USER16> Funny that yesterday I went to see new apartment for rent, the broker was taking me to new places - the return trip did seem smaller.

Even as small as a staircase seemed way smaller when getting down. Analogy could be of writing code with so much hard work (forward trip) and then doing ctrl-a,delete (return trip!).

This could also be related to time passing fast when you are happy than you are sad or worried.

<comment><USER17> Also works in coding: `Spike` and `Redo`

</comment></comment><comment><USER18> The other thing I find peculiar, when driving, is why I can never remember any details about journeys from A to B that take around 30-60 minutes. When I arrive somewhere those last 60 minutes are just a complete blank.

</comment><comment><USER19> I never knew this was a thing. I haven't experienced it. Does it only happen on vacation trips? Or does it also happen with, for example, commuting?

<comment><USER20> Happens to me nearly everywhere I go. I experience it daily on my commute and even going to familiar places like family members houses.

The return always seems faster.

<comment><USER21> I have the opposite experience. I also didn't know it was a thing but I use to always say the opposite to people and they would agree with me. Hmmmm

<comment><USER22> Me too. Getting to a holiday destination is always fun, easy and seems short. Coming back home is boring and seems to take much longer.

</comment></comment></comment></comment><comment><USER23> I only seem to feel this going to and from new places.</comment>


Fun with NFL Stats, Bokeh, and Pandas - <AUTHOR> https://j253.github.io/blog/fun-with-nfl-stats.html <comment><USER2> > The small spikes at 5 yard increments is interesting and I don't really have > a good explanation other than to think that whoever recorded the yardage > data liked rounding to the nearest 5 if it was close. Anyone else have any > other ideas?

I'll go with the idea that the refs are biased with their ball placement and tend to put the ball on lines [0]. Also, players/teams practice, speak, think in 5 yard increments, so that's got to bias things a bit. "We've got to get to the 35 yard line for Morten to have a chance."

[0] [https://gutterstats.wordpress.com/2015/11/03/are-nfl- officia...](https://gutterstats.wordpress.com/2015/11/03/are-nfl-officials- biased-with-their-ball-placement/)

<comment><USER3> There are multiple other factors involved that aren't discussed fully in that article.

To start the ref bias is not necessarily unintentional. Measuring first downs and exact placement of the line of scrimmage becomes more difficult and takes more time when the ball is not initially spotted on an 5 yard increment or individual hash mark. The league offices might directly instruct refs to spot the ball on an exact yard line if there is any doubt of the spot in order to speed up the game. An obvious example of this intentional bias is when the ball is punted out of bounds. It is nearly impossible for a ref to get an exact spot in that situation and yet it is almost always spotted exactly on a hash mark.

Also lots of drives will start on a set yard line and penalties are often handed out in 5 yard increments. So while there are no rules that will start a drive on the 15, there are rules that will start a drive on the 20 or 25 and a standard 5 or 10 yard penalty would place the ball exactly at the 15 yard line.

Lastly, the spot is a continuous data point that is being recorded by humans as a discrete data point. That means the herding could be exaggerated in the recorded data and not necessarily as pronounced during the actual game. A ball might be spotted by the ref at 14.4 yards but the person responsible for data entry might eye the spot and record it as being at the 15 yard line.

</comment></comment><comment><USER4> As someone who's done a lot of numerical analysis and watches a lot of football, the analysis here is pretty rudimentary.

> On third down, pass attempts outnumber run attempts at almost a 4 to 1 clip. > This is likely out of increased desperation to make a first down.

Running is a low variance low yardage option, passing is a high variance high yardage option. Passing on third and medium to long is an obvious dominant strategy. Pulling the goaltender in hockey or bringing the keeper forward in soccer when trailing late in games/matches serves much the same effect.

</comment><comment><USER5> Does anyone have any insight about what kind of jobs are out there for people with the kind of skills demonstrated in this post?

I have a lot of data exploring, cleaning and visualizing skills, python/SQL skills and experience using it to make business decisions, but this type of thing falls short of what most people would consider "data science"

<comment><AUTHOR> Agreed. And I definitely make no claims about this being earth-shattering "data science". I just happened spend a few hours over the weekend making some plots and commenting about what I saw with some Python tools.

I'll also state that I am neither a data scientist nor a statistician. I'm a Python application engineer with a background in mechanical engineering, so that might help set the context a bit more.

<comment><USER6> Most companies don't need "earth-shattering 'data science'", they need a way convey a narrative with their information and maybe try to deduce something from it.

I work as a programmer at an architecture company and we do visualizations like this all the time for campus classroom usage for example. Is it groundbreaking? Of course not, but it helps the clients and designers a ton.

</comment></comment></comment><comment><USER7> I feel like this is a waste of bokeh's potential and matplotlib would have sufficed. I love bokeh for the interactive capability and controlling what you plot, zooming, and just overall more immersive feeling than a static 2d plot.

Still very interesting and insightful and lovely plots generated.

<comment><AUTHOR> Thanks. And I agree with Bokeh being overkill for this. My original intention was to make it fully interactive but I hit some snags on keeping the interactivity through Pelican SSG so I just kept 'em static.

</comment></comment><comment><USER8> I've always enjoyed this project for pulling nfl stats. [https://github.com/BurntSushi/nflgame](https://github.com/BurntSushi/nflgame)

<comment><USER9> That project is no longer maintained because I don't use it any more, but others have picked up the baton: [https://github.com/derek- adair/nflgame](https://github.com/derek-adair/nflgame)

Back in the day, I used nflgame along with

        https://github.com/BurntSushi/nfldb
        https://github.com/BurntSushi/nflvid
        https://github.com/BurntSushi/nflfan

to setup a simple local web UI that allowed me to quickly search through every play and _watch any single play I wanted_. Video footage was available as soon as the game was over, and play info was available live as the game was playing. It was amazing.

This worked because nflvid downloaded full HD NFL games from their CDN, which was unprotected at the time. (I paid for an NFL Game Pass subscription and never distributed the video footage.) They also had XML files that delineated the time at which each play started and its duration. Some ffmpeg slicing and dicing was all it took to cut up a full game and associate each clip with each play. That's all part of what nflvid does.

I hacked all of this together in my free time years ago, and I bet a lot of people would find it amazing. One wonders why the NFL doesn't build this and sell it themselves. When I used Game Pass a few years ago, you could search for plays with rudimentary criteria, but only over a single game at a time. It was artificially very limited.

<comment><USER10> We briefly spoke via GitHub about a month ago, and during that convo (it was in a ticket), you mentioned that the source has inaccuracies. Is there any elaboration there or do the NFL people use a different data source to do things like Fantasy and official stats?

<comment><USER9> I'm not an NFL insider. I don't know what they do internally. I only know that 1) the undocumented NFL GameCenter JSON is not 100% accurate and that 2) any user of a fantasy league would notice these inaccuracies. I did a test a while back by comparing GameCenter data with Yahoo's data. Kickers tend to have the most inaccuracies: [https://github.com/BurntSushi/nflgame/blob/master/test- data/...](https://github.com/BurntSushi/nflgame/blob/master/test-data/results- yahoo-2012-play/k.tsv) QB stats are more solid for example, but there are still minor problems: [https://github.com/BurntSushi/nflgame/blob/master/test- data/...](https://github.com/BurntSushi/nflgame/blob/master/test-data/results- yahoo-2012-play/qb.tsv)

From those observations, you can't really make any solid conclusions. But if you think about it for a bit, you might be able to reason your way to some guesses. For example, one possibility is that the GameCenter data is NFL's own construction that's only used for their GameCenter interfaces, where as places that "official" data is needed might be powered by Elias[1]. Why the discrepancy? Again, I don't know. It could be legacy software related. It could be contract/legal related. Or it could jus tbe plain old bugs. e.g., Maybe GameCenter hooks into an initial lossy but fast feed that is updated during the game, but never receives updates from a slower but more accurate feed later.

Or maybe the NFL purposely inserts data canaries because they know this JSON feed is unprotected, and they intend on using those data canaries to detect folks using their data in an unlicensed fashion. I'm pretty sure IMDb does this, for example. Or maybe they just insert errors purposely to make it too costly for anyone to use this data in situations that require 100% accuracy (like fantasy football leagues).

My guess is some innocuous blend of legal and legacy software reasons.

[1] - [http://www.esb.com/](http://www.esb.com/)

</comment></comment></comment></comment><comment><USER11> For a while I had a process to extract the top ten highest-WPA plays from my favorite team's (Broncos) most recent game. But then my data source dried up. I'm glad to find out about nflscrapR, it seems like that I might be able to figure out how to do that report again with recent play-by-play data.

Incidentally, that report was _really_ fun during the 2011 Broncos season. Normally when you are finding the plays with the largest WPA swings, you'd expect them to be distributed among both teams. But since Tim Tebow became starting quarterback, I searched for the top ten largest WPA swings for the rest of the season - and from what I recall, every single one of those dramatic plays was in the Broncos favor. Weird. :-)

</comment><comment><USER12> > Passing becomes more and more popular as you use up your downs.

Just in case author sees this, this is wrong, right? It should read Running becomes more and more popular as you use up your downs? (Regarding [https://j253.github.io/blog/images/article_01/01_play_by_dow...](https://j253.github.io/blog/images/article_01/01_play_by_down.png))

<comment><USER13> > Just in case author sees this, this is wrong, right? It should read Running > becomes more and more popular as you use up your downs?

Er, no, the author is right; the share of all plays that are passing plays goes up with down number (until dropping at 4), the graph shows that quite clearly.

</comment></comment><comment><USER14> Very cool! I would love to see heatmaps of play type with downs on one axis and yards-to-first-down on the other.

<comment><USER15> Something like this?

[http://www.yardsgained.com/#(passes_~_sacks)_~_first_down_at...](http://www.yardsgained.com/#\(passes_~_sacks\)_~_first_down_attempts_~_distance_~_down%2F%3A%2F%2B)

<comment><USER14> Oh wow, that site is excellent. I think this is the query I was thinking of before, the percentage of passes conditional on down and distance:

[http://www.yardsgained.com/#(passes_~_sacks)_~_(passes_~_sac...](http://www.yardsgained.com/#\(passes_~_sacks\)_~_\(passes_~_sacks_~_rushes\)_~__distance_~_down%2F%3A%2B%2B%2F%2B)

</comment></comment></comment><comment><USER16> this is super cool! well done.</comment>


Ask HN: How do you document a server? - <AUTHOR>

  - Do you describe all steps from OS installation to application setup?
  - What do you use to create the documentation?
  - Where do you store the documentation?
 Any information related to server documentation is relevant. <comment><USER2> Hmm, interesting question. I'd like to know how other people do it.

At work, we generally don't write narrative documentation for server setup. It requires quite the effort to make sure it doesn't go stale.

Instead, we keep Ansible playbooks on a Git repository. Every change we make to the server setup is done through the playbooks. Git commits provide a history of the changes made (we usually write the why in the long commit message).

Usually just having a README with information on how to run the playbooks and which distro the playbooks are based on is enough.

</comment><comment><USER3> Recently, I've been wanting to document how I setup my servers. I would start from a blank slate base linux install and go from there, listing each command I run and what it does. Then building off each command into the next leading the reader through the steps of configuration and installation of software. This is all for a blog, so take that into consideration.</comment>


Story of the acquisition of Ducksboard - <AUTHOR> http://novobrief.com/ducksboard-acquisition/

<comment><USER2> Great story! I was in Madrid in 2011/2012 and met the founders at one of the Betabeers events. Very driven and resourceful guys.. one of the very few founders I personally was impressed with in Madrid.

Congrats!

</comment><comment><USER3> Good for them.

At the same time, at the company I work for we make heavy use of Ducksboard (v1) and rather nervous about the future.

<comment><USER4> Please, don't be nervous. The v1 will be sold and maintained for a very long time.

Cannot talk about details, but New Relic doesn't want us to lose a single customer during the transition. Not only one.

So please, bear with us. No plans to discontinue anything, and with the upside that now we have a lot more of horsepower.

<comment><USER3> Ok, that's great to hear. From what I understand, we were one of your very first customers (CTIPath). I've only been here for 9 months or so though.</comment></comment></comment>


Show HN: Screen-light meditation for busy and stressed screen users - <AUTHOR> https://lumiate.co <comment><AUTHOR> Mindfulness meditation linked to a hue-cycle, like Night Mode for mindfulness. The cycle fades into the unconscious while you work (perceptual adaptation), but you can shift your attention to it for light-guided meditation.

Currently in beta, looking for feedback and happy to answer any questions.</comment>


Google Store - <AUTHOR> https://store.google.com

<comment><AUTHOR> I'm viewing from Australia. Can anyone in the US confirm whether there is or isn't Chromebooks on there? Seems a noteworthy omission, if not.

<comment><USER2> Yea there's definitely a chromebooks section:

[http://imgur.com/Ro5MQjf](http://imgur.com/Ro5MQjf)

Currently they only have the Acer Chromebook 13 and the Chromebook Pixel.

[https://store.google.com/product/acer_chromebook_13](https://store.google.com/product/acer_chromebook_13)

[https://store.google.com/product/chromebook_pixel_2015](https://store.google.com/product/chromebook_pixel_2015)</comment></comment>


Facebook Is Working on a Video Chat Device - <AUTHOR> https://www.bloomberg.com/news/articles/2017-08-01/facebook-is-said-to-work-on-video-chat-device-in-hardware-push <comment><USER2> So,I've got a cellphone with front and back cameras, a tablet with front and back cameras, and a laptop with a front camera, and a desktop device for which I could easily get a front camera for a far less than anything with it's own dedicated screen will cost, all of which have (or have available) both native and web-based video chat software (including versions specialized for group and one-on-one use.)

Whyever would I want or need a dedicated video chat device? Why would anyone?

Also, a laptop-sized (so, I guess, somewhere between 10" and 19") isn't going to make people feel like they are in the same room.

I suppose if you wedded a 3D camera to a HoloLens-like AR system, with some clever software, that might make people feel like they are in the same room.

<comment><USER3> This reminds me of that infamous "Show HN: Dropbox" thread.

It is 2017 and I don't know of any video conference system or software that "just works" everywhere, all the time, under all bandwidth conditions. If Facebook thinks they can pull this off by tight control on hardware, they will sell like hotcakes.

<comment><USER4> I feel like if I had to video conference with one of my friends running Linux we would just use ffmpeg and ffplay.

It probably wouldn't be anymore reliable than all the other conference software but at least it doesn't drag a mountain of bullshit and lockin with it.

We just use email and in person meetings though, I still don't quite get the point behind video conferencing.

</comment></comment></comment><comment><USER5> I'm all for this. We recently purchased some Echo Shows, one for us and one for my grandparents and the Amazon experience to setup communication between the two was absolutely painful.

First both need an Amazon account which, okay that's not so bad. But now both Amazon accounts need to log into the Amazon Alexa mobile app which my grandparents can not do as they do not have a mobile phone with apps on it so, instead, I have to use their login on my phone. Then I had to register their cell phone with their account on my mobile phone because they can't do the text message verification via their landline that they've had for 30 years and to top it all off now their Echo has all of the contacts from my phone because it doesn't let you manually manage contacts per Amazon account, no they have to be copied directly from your phone.

So if they can make it so all they need is a Facebook account and now they can video chat with us or any of their friends and family? That would be perfect for my grandparents.

<comment><USER6> Right, and for those of us who don't want to use a Facebook account? The world is lacking good, adopted implementations of open networking.

<comment><USER7> What's wrong with just making a disposable single use facebook account.

<comment><USER8> They generally ask for a phone number nowadays for new accounts, so you'd need a new number for each one (or reuse the same one, which defeats the purpose of the disposable accounts).

</comment></comment></comment></comment><comment><USER9> I worked in Videoconferencing in the 90's and the advent of better CPU and processing power come the turn of the century with netmeeting, many in the industry thought the days of dedicated hardware and expensive MCU (Multipoint control units) was soon to be usurped. Nearly 20 years on and yet to see that killer video conferencing feature.

Of note was also privy to early testing of Three (UK 3G mobile network) tests for video and whilst managed to stream the Matrix in the early days as a test, that never took of either and that was more due to the costs of data connections, an issue very much moot these days (least in the UK).

One area that often lets down videoconferencing at the consumer level are the camera's and the environment the user uses them in and gets down to lighting. But again, no killer, easy to use consumer solution has come and taken the market by storm.

Sure many point to point video solutions on offer and used but no universal standard multipoint as easy as sending an email to multiple people is jumping out.

With that, I wish Facebook good luck and they do have the user base to drive this, but I'm not sure it is a feature as high in demand for consumer users as many wish or feel it is.

I can only put this down again to the environments such consumer systems (your mobile say) are used in and whilst the processing power is there to alleviate much of this, that has still to come to fruition as a user friendly reliable experience.

<comment><USER10> Wow, I haven't run into refugee from that videoconferencing era in a while, I worked at VTel in the 90's in Austin.

<comment><USER9> I worked for a company called TopTel, managed the VC networks for ICL, DOH and few others, dealt with many Telco's and bridge companies who's names escape me at the time. international VC and dialing the extra 0 for ISDN lines to make sure they didn't route over satellite and then the fun of switched ISDN (North American ISDN standard with it's bit stealing 56k lines over the dedicated D channel 64k lines). Fun times.

One aspect of VC was you could tell if a network link (leased lines) was not upto scratch as you would see bit error, unlike network data, which would happily gloss over such odd bit errors. Other fun and games was ISDN circuits being stolen due to engineers repatching lines of little used circuits and then when the ISDN would lock up and that would often be that the bit error count hit a limit. You would call up, get the telco to test the line/circuit and low and behold it would be working again, this was due to the test resetting the error count. Had a neat way of testing lines using my 2G mobile on the Orange network, in that if I called an ISDN line the way it terminated the call would indicate if it was up or was errored, or busy.

But my funniest story would be spending 30 minutes trying to resolve an issue with a PicturTel setup with some French chap and why the audio from his end was not working, turned out he was sat upon the microphone.

<comment><USER10> My company and I could have used your knowledge! Got sent to Europe to debug some problems where we had an ISDN card from a third party installed in our systems and got to visit UK, Germany, Belgium, and France to look at different issues.

</comment></comment></comment></comment><comment><USER11> We've had the prospect of video chat for decades. 2001 A Space Odyssey showed us that it would be the de facto way to talk to our families. Tom Selleck promised us in the early 1990s ([https://www.youtube.com/watch?v=TZb0avfQme8](https://www.youtube.com/watch?v=TZb0avfQme8)) that AT&T would bring it to us. And when we finally got it, most people said, "meh". Almost all chat platforms have some sort of video integration, and for the most part, its acceptance is lukewarm. Perhaps it's because it doesn't work amazingly well? Perhaps its because people really don't want to have to get dressed to have a conversation? Perhaps I'm just not in the right crowd?

<comment><USER12> FaceTime seems to be way more popular than most realize. Anecdotally, my children use it daily to talk with grandparents and cousins and I see people on public transportation using it in place of telephone calls (makes more sense there because they're not driving). And, I use it when traveling to talk to my family.

The issue going forward is the lack of standardization across protocols. Despite Apple's mention in the keynote when FaceTime was announced that it would be open, FaceTime is iOS/Mac only for now. And, I don't see Facebook's playing nice with Amazon's, etc.

<comment><USER13> It's iOS/Mac only because it has to go through Apple server. This is because of this lawsuit:

[http://www.bbc.co.uk/news/technology-20236114](http://www.bbc.co.uk/news/technology-20236114)

<comment><USER14> Interestingly, it seems the company they lost to is effectively a publicly traded patent troll

<comment><USER15> And it has a market cap of just over 200 million dollars [1]. Makes me wonder why Apple wouldn't simply buy the company. They can more than easily afford it.

[1] [https://finance.yahoo.com/quote/VHC](https://finance.yahoo.com/quote/VHC)

<comment><USER16> If I were feeling vindictive I wouldn't want to reward a patent troll by making them multimillionaires. I'd rather sue them out of existence...

</comment></comment></comment></comment></comment></comment><comment><USER17> I must say the Video chat in Amazon Echo Show is very interesting from my experience using it over the past few weeks when talking to my family in other side of the world. The Mics are so good that they can hear them from pretty much anywhere in my 800 sqft home.

It would be very interesting to see how this device competes. The features i would love to have is a 360 Camera attached to a large screen. with Multiple Mics / Good Speakers. Ability to seamlessly which to multiple screens based on where I am.

<comment><USER18> _The Mics are so good that they can hear them from pretty much anywhere in my 800 sqft home._

So can The Others.

</comment></comment><comment><USER19> Do you trust a Facebook camera on your kitchen counter/night stand/living room table? I feel like they haven't done much for themselves(in terms of branding) as a privacy-friendly company to the point where technophobes are going to be comfortable setting these up in their homes.

<comment><USER20> No kidding.

My twitter feed is full of people pointing out their CEO (Zuckerberg) tapes over his webcams, but now wants users to put this in their homes.

</comment></comment><comment><USER21> Remember when we had a unified communication network?

I can use my phone for video chat over Facetime. Or Facebook Messenger. Or Skype. Or Whatsapp. Or Google Hangouts (haven't installed Duo. yet)

Signal and Line can both do audio calls but not video. Yet.

I _think_ with the union of all those networks I should be able to do a video chat with 95% of my contacts.

When's SIP taking over the world again?

<comment><USER22> The only way we get big dogs to play along to everyone's benefit is by force, sadly. If not then they will always attempt to carve up fiefs.

</comment></comment><comment><USER23> Oh sweat progress, now we can leave all the text based internet and chat-apps behind us, finally able to identify people again not by there words, but by there colour, there pattern of speach and the wealth of there surroundings.

If there ever was progress that should move humanity backwards this is how it looks.

</comment><comment><USER24> This move (along with the smart speaker), might help put 'M' at an equal footing with Alexa, Cortana, Siri, Google Assistant and the likes in terms of media recognition. Somehow, M is not brought up as often when AI assistants are mentioned.

</comment><comment><USER25> Instead of articles about what Facebook is working on, I'd be more interested in what they're not working on at this point. I can't think of an area they aren't trying to disrupt.

<comment><USER26> > I can't think of an area they aren't trying to disrupt.

Disrupt is a strange way of putting it when they are an incumbent copying and implementing exactly what others have been doing for a long time. To me disruption means a significant change from how things are currently done in a given market -- a change that also changes the paradigm for how things are done in the market.

</comment></comment><comment><USER27> My guess: it's something like a chrome stick with a connected webcam you place at the top of your tv.

</comment><comment><USER28> Hmm anyone else thinking of Pied Piper?</comment>


Your private Docker Registry - <AUTHOR> https://creekorful.me/harbor-private-docker-registry <comment><AUTHOR> Feedback is welcome !</comment>


A Roadmap for Rich Scientific Data Structures in Python - <AUTHOR> http://wesmckinney.com/blog/?p=77

<comment><USER2> Scientific computing is dominated by low-level or system languages like Fortran, C, and C++; and by domain-specific languages like Matlab and R.

There is no good high-level, GENERAL computing language that can do scientific computing well. This leads to a huge gap between "research" and "production" implementations.

Python stands to shine in this space if things are done right. pandas is a gem of a library.

</comment><comment><USER3> It's great that somebody is looking to R for inspiration. I've tried to like NumPy and SciPy on a couple occasions, but find it lacking. That said, there's a to of ways to improve R, mostly to do with cleaning up and homogenizing style.

I do hope that they avoid the BioConductor style of opaque objects and storing data in hidden attributes.

</comment><comment><USER4> I was happy to see "Hierarchical columns" on the features wish list. My app uses a R data frame like structure (table of heterogeneous columns with possible missing data). One "cute" thing I implemented was hierarchical columns. Super useful for what I'm doing, but hard to expose to other languages. The plan is to flatten the columns to 'parent.child' strings when using python.

I'm not quite at the point where I'll be wrapping this into python. When I do I'll take a close look at pandas. Any other recommendations? I took a quick look at pytables, and pandas looks better for my app.

<comment><AUTHOR> To be honest I haven't really broached the hierarchical columns issue inside pandas. If someone can look and suggest an implementation strategy that doesn't interfere with the rest of the API I would be all for it.

If you have heterogeneous columns with possibly missing data basically pandas is the only game in town (did you see my diagram?? :) ). It's possible to get a numpy MaskedArray with structured dtype to function like you want but it's relatively tricky to do.

</comment></comment><comment><USER5> I've been using the pandas library for a long time for financial applications. The data alignment and missing data handling features in pandas are far above and beyond anything else I've used in similar applications. I think from a data structure point of view it's already better than R (and FAR better than Matlab). R/Matlab should fear the day that the pandas statistical packages gains roughly equivalent features.

</comment><comment><USER6> Data frames (and the associated functionaly for reading and writing csv files) are one of the main features that make me use R over python. Whenever I need to manipulate, merge, slice, and dice table-like data, I turn to R. I would love to have something feature-equivalent in Python. Although maybe rpy2 and rnumpy are sufficient for now.

<comment><AUTHOR> It _is_ my project but I think pandas.DataFrame is already at 90+% feature- equivalency (especially if you're using the current git version...new release forthcoming). You don't have the integration with a million CRAN libraries. pandas.DataFrame actually does a lot more for you in many places than data.frame does-- for example data alignment is deeply intrinsic whereas it's very much a DIY affair in R.

<comment><USER6> Yes, I do intend to try out pandas at some point.

</comment></comment></comment><comment><USER7> Wow, fantastic work and what an important project. I can't believe I haven't used these libraries before.

</comment><comment><USER8> Without looking too closely, would h5/pytables suffice?

<comment><AUTHOR> HDF5/PyTables is fantastic as a binary IO format. This is really all about in- memory data manipulation / computation and how data and metadata get passed around to functions that can take advantage of it.

</comment></comment><comment><USER9> WebSense blocks this at work.

Is there a nice mirror somewhere? :-/</comment>


Do VC Dollars Equal Success? - <AUTHOR> http://www.pehub.com/51830/do-vc-dollars-equal-success/

<comment><USER2> That's such a weird thing to see coming up over and over again.

Finding an investor is like being in the casino and having a high roller give you money to bet in return for a share in the stakes because they believe you have better luck than they do themselves.

Success is when both you and the investor cash out with significantly more money than you put in or when you make the business so profitable that your accumulated dividends exceed the initial outlay.

To see VC money as 'success' is simply wrong, it may increase your chances of success, and it may cushion your losses a bit (because not all expenses come out of your pocket).

</comment><comment><USER3> No.</comment>


Ask HN: Resources for Learning about Databases - <AUTHOR> I recently got a Frontend Engineering offer from Facebook. I have dabbled in backend work earlier in my carrier but never did anything beyond simple API's on a single server. I set a goal for myself that every year I pick a computer science topic & study it thoroughly. I dont have a CS background but I am begining to love CS.
I started with Data Structures & Algorithms last year, studied & did over 250 leets, this is why I was able to land an offer with FAANG.
Next on my list is Databases, I want to know how they work internally, build a simple RDBS from scratch, learn SQL(I know simple CRUD operations) advanced concepts like procedures & the latest that is being used today.
I have googled yes, but I havent found any resource that meets my needs. I also plan to switch to backend soon.
Thanks in advance
Edit: I know I will not be building databases at Facebook, & I also know they probably have internal tools or ORM to access databases. My goal is not to become a database developer but to have a good knowledge of how they work just to satisfy my curiosity. <comment><USER2> It gets recommended all the time in these kind of threads, but it's so good I don't care. Bill Karwin's SQL Antipatterns. You need a decent understanding of the basics to get the most from it, but there's some excellent information and examples of what to (and what not to) do.

[https://www.oreilly.com/library/view/sql- antipatterns/978168...](https://www.oreilly.com/library/view/sql- antipatterns/9781680500073/)

</comment><comment><USER3> Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

[https://dataintensive.net/](https://dataintensive.net/)

<comment><USER4> How dry is this? I've had hit or miss success with technical books, and found that the writing style can help dramatically to feeling engaged and not dry or boring.

Thoughts?

<comment><USER5> Not very dry in my opinion. Good mix of explanations, information, and context.

I'm about 2/3rds through it.

</comment></comment></comment><comment><USER6> Great intro and overview: [http://coding-geek.com/how-databases- work/](http://coding-geek.com/how-databases-work/)

Great book: [https://dataintensive.net/](https://dataintensive.net/)

Also great read and overview: [http://www.redbook.io/](http://www.redbook.io/)

Great paper over-viewing the architecture of a DB: [https://perspectives.mvdirona.com/content/binary/Architectur...](https://perspectives.mvdirona.com/content/binary/ArchitectureOfDatabaseSystem.pdf)

If you're looking into building your own database, there are some great open source projects you can reference here: [https://github.com/danistefanovic/build-your-own-x#build- you...](https://github.com/danistefanovic/build-your-own-x#build-your-own- database)

If you want to actually dive into source code - SQLite is amazing. It has very clean and readable code, so I'd suggest using it as a reference as well: [https://github.com/mackyle/sqlite](https://github.com/mackyle/sqlite)

</comment><comment><USER7> I have three things for you

1\. Designing data intenstive applications

2\. Database internals [https://www.amazon.com/Database-Internals-deep-dive- distribu...](https://www.amazon.com/Database-Internals-deep-dive-distributed- systems/dp/1492040347)

3\. Andy Pavlo's database course videos at cmu and guest lecture series [https://www.youtube.com/channel/UCHnBsf2rH-K7pn09rb3qvkA](https://www.youtube.com/channel/UCHnBsf2rH-K7pn09rb3qvkA)

</comment><comment><USER8> Anything by Joe Celko: _SQL for Smarties_ , _Trees and Hierarchies in SQL for Smarties_ , Joe Celko

Also, the internals of Django ORM ([https://github.com/django/django/tree/2.2.5/django/db/models](https://github.com/django/django/tree/2.2.5/django/db/models)) and SQLAlchemy Core ([https://github.com/sqlalchemy/sqlalchemy/tree/rel_1_3_8/lib/...](https://github.com/sqlalchemy/sqlalchemy/tree/rel_1_3_8/lib/sqlalchemy/sql)) and its dialects ([https://github.com/sqlalchemy/sqlalchemy/tree/rel_1_3_8/lib/...](https://github.com/sqlalchemy/sqlalchemy/tree/rel_1_3_8/lib/sqlalchemy/dialects)) + ORM ([https://github.com/sqlalchemy/sqlalchemy/tree/rel_1_3_8/lib/...](https://github.com/sqlalchemy/sqlalchemy/tree/rel_1_3_8/lib/sqlalchemy/orm))

</comment><comment><USER9> I really suggest against building a database from scratch. It's just too annoying, and there's so much code to write (parser, storage, indexing, query planner, connections). If you're interested in internals, I'd say look at the sqlite codebase instead: [https://sqlite.org/src/doc/trunk/README.md](https://sqlite.org/src/doc/trunk/README.md) . If anything, reading code that works is probably more useful than writing code that almost certainly won't without months and possibly years of effort.

A lot of the more complex database things are only really learned by having a large database system. Performance, distributed databases, and complex schemas come to mind here. Most of the times with simple examples, you'll do something wrong performance wise, but you'll never know because of the scale (such as forgetting an index, or doing a bad join).

Many times, you don't need to know that much about database other than some basic SQL.

</comment><comment><USER10> These are more academic than practical (i.e. build a DB from scratch) but still interesting I think.

[https://github.com/rxin/db-readings](https://github.com/rxin/db-readings)

</comment><comment><USER11> I found this an enjoyable resource for learning about one of the fundamentals of RDBMS, indices: [https://use-the-index-luke.com/](https://use-the-index- luke.com/)

</comment><comment><USER12> anything on this channel - [https://m.youtube.com/channel/UCHnBsf2rH-K7pn09rb3qvkA](https://m.youtube.com/channel/UCHnBsf2rH-K7pn09rb3qvkA) \- CMU DATABASE GROUP. all thanks to - [https://mobile.twitter.com/andy_pavlo](https://mobile.twitter.com/andy_pavlo) \- Andy Pavlo - has a quote, something like "I only love two things, my wife and the databases". Follow his lectures and read his suggested papers.

</comment><comment><USER13> I would pick one RDBMS and try to dissect it, there is a lot to chose from nowadays, you can check out db-engines to get a general sense of what's out there:

[https://db-engines.com/en/ranking](https://db-engines.com/en/ranking)

From what I have seen most enterprises today will be using Oracle or Microsoft, however PostgreSQL seems to have gained popularity with the web developer and small business crowd (as well as with the HN community). I have been an Oracle database developer since 2015 and would definitely recommend going that route if it interests you, at the very least it might be a good starting point because of the fantastic documentation, here's a great guide I recommend to get you started with all the basic concepts:

[https://docs.oracle.com/en/database/oracle/oracle- database/1...](https://docs.oracle.com/en/database/oracle/oracle- database/19/cncpt/index.html)

</comment><comment><USER14> Readings in Database Systems [http://www.redbook.io/](http://www.redbook.io/)

</comment><comment><USER15> Alex's book: [https://www.databass.dev/](https://www.databass.dev/)

<comment><USER16> I just read a few chapters and it's good so far.

</comment></comment><comment><USER17> AWS re:invent 2018 talks:

[https://youtu.be/HaEPXoXVf2k](https://youtu.be/HaEPXoXVf2k)

He has a sequence of 2-3 great talks on DynamoDB, the history of relational databases and the rise of access-pattern oriented db design.

</comment><comment><USER18> I back up the previous hints for Designing data intensive applications and Database internals. I would suggest also to look at Jepsen tests, [https://aphyr.com/tags/jepsen](https://aphyr.com/tags/jepsen), and Adrian Colyer's blog, [https://blog.acolyer.org/](https://blog.acolyer.org/)

</comment><comment><USER19> You could port a database from one language to another as a learning exercise.

<comment><USER7> #easy

<comment><USER19> Well, what the OP asked was not easy. :)

</comment></comment></comment><comment><USER20> Anything with exercises and answers?

</comment><comment><USER21> Andy Pavlo courses on youtube.

</comment><comment><USER22> Yes

</comment><comment><USER23> Do you want to build a database or use one?

Also prolly at Facebook they use some API to access the database.</comment>


Android OS Geek News. Take a peak - <AUTHOR> http://androidflagship.com

<comment><AUTHOR> I would really appreciate an opinion about the website.</comment>


Facebook turns down $8 billion valuation, projects $550 million in 09 revenue - <AUTHOR> http://www.techcrunch.com/2009/05/19/facebook-turns-down-8-billion-valuation-term-sheet-claims-2009-revenues-to-be-550-million/ web 2.0 is alive and well... <comment><USER2> Contrary to what the title implies:

"Facebook declined the term sheet based on the requirement of a board of directors seat, says our source, and not the valuation."

<comment><USER3> True (and +1 for putting the actual detail into this discussion rather than the sensationalist headline).

But part of turning it down, at least in Arrington's seemingly informed opinion, is based on Facebook (read: Zuckerberg) believing he has other options which must be in the ballpark of that valuation or above.

</comment></comment><comment><USER4> What does it mean that Zuck has 3 seats -- one empty, one he holds, one by Andressen -- does Andressen get to vote as Andressen wishes, or does he have to vote Zuck wishes, or does Zuck effectively get to cast all three votes?

How do these board seats relate to the preferred boat seats?

</comment><comment><USER5> So, if Facebook is going to have between $400 million and $550 million revenue this year, does that put to rest once and for all the idea that they are unable to monetize their users?

<comment><USER6> I don't think that anyone has denied that they are able to monetize users in some fashion. What is hotly debated though is whether or not they can monetize users sufficiently enough to make those users profitable in the short and/or long term.

Facebook's overhead certainly appears to be quite substantial. They are literally building new datacenters and are running a ton of equipment. Even if you could theoretically fire all the sales, marketing and HR types, you'd still be left with a pretty big bill at the end of each month.

Various online sources say that Facebook has a little over 1,000 employees. Let's say the average employee is $80K fully loaded. That's $80M/year just in payroll. You haven't even talked about real-estate to sit those employees in, PCs for them to work on, or the servers and infrastructure to run the company.

Also, much of that revenue (for all practical purposes, maybe all of it) seems to be advertising related. There is a lot of talk about the declining values of online advertising, and no one seems to have cracked the system on how to do that advertising in a fully effective manner. So, Facebooks revenue is coming from a source that they don't have a strong ability to manage and control (vs., say, user subscription fees).

</comment></comment><comment><USER7> Revenue != profits. I wonder what their margins are. They might be losing money for all we know, even with $550m rev.

<comment><USER8> Agreed. 1,000+ employees, 10,000+ servers, plush offices etc don't come cheap.</comment></comment>


Android Wear smartwatches make Google Glass obsolete - <AUTHOR> http://arstechnica.com/gadgets/2014/06/android-wear-smartwatches-make-google-glass-obsolete/

<comment><USER2> Android Wear is for a mass market consumer products, it better have clear use cases and be embodied in devices that are affordable for millions of people.

Glass is a prototype that happens to be available to consumers. It's useful in giving one a chance to explore wh what a head mounted computer is like, in the same way that a Nokia 770 gave you a sense of what the smartphone world would be like a few years later.

That being said, Google ought to figure out and communicate what's next for Glass. Or end the experiment.</comment>


An overview of new featuers in Android 7.0 Nougat - <AUTHOR> http://www.thedroidsonroids.com/blog/android/whats-new-android-7-0-nougat/ <comment><USER2> "Starting from Android Nougat, some applications will be granted the permission to run before users enter the pin code"

I think that's bad. And what kind of 'some applications'?</comment>


Azure container service available today in public preview - <AUTHOR> https://azure.microsoft.com/en-us/services/container-service/ <comment><AUTHOR> OK, still trying to stand up a deployment.

First one failed silently and left the "Submitting Deployment for Template Deployment..." on my portal panel forever.

Second one failed within a minute with DNS naming conflict.

Maybe third time is a charm.

I'm currently using Dokku to run some small apps, and it seems to work well. However, all this magic control panel automation and deployment with weird DSL/JSON scripts causes quite a bit of pain.</comment>