Obama presents NSA reforms with plan to end government storage of call data - <AUTHOR> http://www.theguardian.com/world/2014/jan/17/obama-nsa-reforms-end-storage-americans-call-data <comment><USER2> Wow, this is a whole lot of nothing. Instead of recognizing the problem of having secret Star Chamber-ish courts in the first place, he just says "yeah we're just _really_ going to make sure the NSA goes to them first."

Mass storage of phone conversations doesn't appear to be ended, just shuffled off to some private contractor instead of being NSA-internal.

Phone records aren't even the biggest concern in the first place. The _real_ scary thing is their Internet firehose taps at places like AT&T room 641A and their Utah datacenter built to store exabytes upon exabytes of data. Put two and two together-- _that 's_ the worry. Not a peep about that.

This is literally nothing. None of the core issues have been addressed. He's just changed the window-dressing.

<comment><USER3> No, its something. Its saying the U.S. spymasters will not spy on Germany's spymasters without getting the approval of the secret U.S. kangaroo court. I dont know about you but I'm feeling more confident in the system already.

</comment></comment><comment><USER4> "… because nothing in that initial review, and nothing i have learned since, indicated that our intelligence community has sought to violate the law or be cavalier about the civil liberties of their fellow citizens." \- Obama

I had a very hard time getting past this line he fed at ~11 minutes in. If that is the presumed mindset, you can draw some very broad (and i fear accurate) assumptions about the changes going forward.

<comment><USER5> This isn't a line. Everything I've seen in the leaks indicates an intent by the NSA to try and stay within the contours of the law. They _are_ allowed to collect whatever foreign data they want, and they _are_ , at least based on reasonable interpretations of existing precedents, allowed to collect metadata about phone calls.

You can certainly draw assumptions from that: they aren't going to back away from these very reasonable interpretations unless the Supreme Court makes them untenable by, say, rejecting the Third Party Doctrine, but given the law that's on the books, it's accurate to say that what has been leaked is not consistent with the NSA actively seeking to ignore the law.

<comment><USER6> Everything I've seen has been a consistent effort by the NSA to eviscerate the intent of fundamental laws through clever loopholes of various implementations and rulings, often from completely different contexts.

They covered their asses to keep from going down for actual criminal activity, but I'm not willing to say that they seem to have placed any emphasis on the fundamental liberties or protections that the Constitution is supposed to afford.

Even their secret court has pointed out that they're crossing the line (likely, more than once).

I mean, it's at the point that they're having to lie to a secret, relatively unaccountable court because even that won't buy their bullshit.

<comment><USER5> Smith v. Maryland squarely says that your privacy interest in your phone calls ends to the extent that your phone call metadata is exposed to third parties. That's not a "loophole" it's a delineation of the boundary of the right. And leveraging Smith v. Maryland isn't taking advantage of a loophole in a "completely different context" but rather applying the reasoning of the case in the same context in which the case arose: is it a search to collect the meta-data of a phone call from the phone company? You may think Smith v. Maryland needs to be overturned, but for now, the NSA is entitled to rely on it.

As for the "intent of fundamental laws" the intent of the 4th amendment was to prevent customs officials from trespassing on private property. The founders never talked about the "right of privacy" as a coherent concept. The "right to privacy" was conjured from the "penumbras" of the Constitution (i.e. thin air) just a couple of decades before Smith v. Maryland was decided.

<comment><USER7> What exactly is the point you're trying to make here? That this is all legal? They have at least violated the SPIRIT of the law.

Also, you are ignoring all other reports that the NSA is storing CONTENT of every means of communication.

Just this week, it was revealed that they store hundreds of millions of text messages:

* [http://www.theguardian.com/world/2014/jan/16/nsa-collects-mi...](http://www.theguardian.com/world/2014/jan/16/nsa-collects-millions-text-messages-daily-untargeted-global-sweep)

They also store every single phone call made in the US:

* [http://www.pbs.org/newshour/bb/government_programs/july-dec1...](http://www.pbs.org/newshour/bb/government_programs/july-dec13/whistleblowers_08-01.html)

* [http://news.cnet.com/8301-13578_3-57589495-38/nsa-spying-fla...](http://news.cnet.com/8301-13578_3-57589495-38/nsa-spying-flap-extends-to-contents-of-u.s-phone-calls/)

Oh, and thinking about using Skype? Also tapped:

* [http://arstechnica.com/tech-policy/2013/07/nsa-taps-skype-ch...](http://arstechnica.com/tech-policy/2013/07/nsa-taps-skype-chats-newly-published-snowden-leaks-confirm/)

<comment><USER5> > What exactly is the point you're trying to make here? That this is all > legal? They have at least violated the SPIRIT of the law.

The "spirit" of the law is to keep government agents from trespassing on your property looking for contraband. The "right of privacy" wasn't conjured out of thin air by the Supreme Court until the 1960's. Collecting information under the Third Party Doctrine violates neither the letter nor the spirit of the 4th amendment.

> Just this week, it was revealed that they store hundreds of millions of text > messages:

Are they the text messages of Americans?

> They also store every single phone call made in the US:

If they store every phone call made in the U.S. that would definitely be illegal. But the evidence that they do this is much flimsier than for the other things we know they do.

</comment></comment></comment></comment></comment></comment><comment><USER8> > Mounting a forceful defence of the NSA, Obama said: "They're not abusing > authorities in order to listen to your private phone calls, or read your > emails."

Yes, they are. [http://www.theguardian.com/world/2013/sep/27/nsa-employee- sp...](http://www.theguardian.com/world/2013/sep/27/nsa-employee-spied- detection-internal-memo)

<comment><USER9> Just to play devil's advocate - I think he's talking about the NSA as a whole as a matter of policy, not these notably but otherwise isolated exceptions. It would simply be impractical and largely useless to listen to every average joe's conversations.

Also, is there any respite to be had considering the IG was aware of that particular case? Isn't that ideal?

<comment><USER10> > I think he's talking about the NSA as a whole as a matter of policy

The NSA lying to the Foreign Intelligence Surveillance Court -- as the FISC called them out for in a ruling which has become public -- about what they are actually doing in order to get things approved is _prima facie_ evidence that the intent to be unbound by law was not an exception but a matter of systematic policy.

<comment><USER9> So, follow through. The NSA is abusing its abilities on a systematic basis - to spy on the personal lives of average people? For political purposes?

</comment></comment></comment></comment><comment><USER11> "The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."

The data shouldn't be collected in the first place.

<comment><USER12> Really, not even when building, say, a RICO case against the mob?

I agree the NSA need to be reigned in, a lot, but there are grounds for listening in sometimes. Courts need evidence to convict.

EDIT: I agree that some kind of permission from a judge is needed for oversight.

<comment><USER13> Get a warrant. You can get evidence using one.

<comment><USER14> Warrant requires probable cause.

How do you get probable cause without any information? How do you even get alerted to the idea that you should be opening an investigation into something, without any information?

I mean, how does that work in court? "Yes Your Honor, we would like to obtain a warrant to search Abdi Zullah's Internet-based communications for the next 3 months since we think that he maybe might be planning to blow something up. Maybe. And oh, this is all just a hunch, we've got no evidence at all for this yet."

You'd be laughed out of court, and rightfully so.

<comment><USER15> Then so be it.

Plenty of criminals have been caught using traditional detective work. No issues, here. Many FEWER criminals have been caught using blanket surveillance, and the cost to democracy is high.

I'd rather have the occasional bomber than the government having this kind of access.

<comment><USER14> > Plenty of criminals have been caught using traditional detective work.

Yes, but that's on the other side of time's arrow. The whole point to _counter_ -terrorism is to detect it and stop it _before_ (not after) it has happened.

<comment><USER15> And that is the cost to living in a free society: occasionally you have to deal with violent assholes.

Again, I'd rather live in a society where the innocent are free and unmonitored than one which the NSA, etc., reign supreme. Counter-terrorism efforts are inherently anti-democratic and anti-freedom, despite protestations to the contrary.

Not interested.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> Right after they close gitmo.

He completely ignored all the recommendations of the panel he appointed to give him recommendations.

</comment><comment><USER17> He boldly took no action to confront a problem he didn't want to confront at a time not of his choosing because of a person he doesn't like.

</comment><comment><USER18> Prove it mr. president! How do we know the calls won't be stored? The stuff the NSA is doing is already illegal but I doubt anyone other than whistle blowers will go to prison.

</comment><comment><USER19> Why would the executive branch curtail its own power?

</comment><comment><USER20> If the executive really wanted this stuff to be closed, it would be closed.

<comment><USER21> Well, of course the executive doesn't want it closed (for whatever reasons), this is all grandstanding to placate the simple masses.

</comment></comment><comment><USER22> If data like this requires a bureaucracy to generate, you might be able to protect it with a bureaucracy. If it's collected automatically and stored in one place, it is considerably harder to protect.

</comment><comment><USER23> Oh no, I thought for sure he would fix everything after he found out about the spying. The NSA must be blackmailing him! Maybe the Supreme Court will swoop in and rescue us. Or maybe Hillary in 2016. Someone has to! I am on the verge of tears right now!!

Let's start a Facebook group - Gmail me!

</comment><comment><USER24> But Internet data is still fair game?

</comment><comment><USER25> So the data will be handed over to the likes of Booz Allen Hamilton, who as Snowden showed have a stellar record at keeping secrets.

Bullcock.

</comment><comment><USER25> Here is a quote from a NYTimes article from 2011 that sheds light on the targets of the military industrial complex at the end of 2010[1]. The industries various components don't always align to the governments side, more towards the interest of profits and profit interests:

'The e-mails include what appears to be an exchange on Nov. 9, 2010, between Aaron Barr, HBGary Federal's chief executive, and John W. Woods, a Hunton & Williams partner who focuses on corporate investigations. Mr. Barr recounted biographical tidbits about the family of a one-time employee of a union-backed group that had challenged the chamber's opposition to Obama administration initiatives like health care legislation.

"They go to a Jewish church in DC," Mr. Barr apparently wrote. "They have 2 kids, son and daughter."'

So we are told to believe that this sensitive national security data that is our collective communications will be safe in the same hands that not only had the tools they were given access to leaked, but who also had their correspondences plotting to attack dissidents leaked.

This is so fucked up on so many levels.

[1] [http://www.nytimes.com/2011/02/12/us/politics/12hackers.html...](http://www.nytimes.com/2011/02/12/us/politics/12hackers.html?_r=0)</comment>


The new Google product that could augur the death of Microsoft - <AUTHOR> http://www.slate.com/id/2168419/nav/tap3/ <comment><USER2> Or... not. This increases convenience for everyone, including those using Windows.

</comment><comment><USER3> When you're bringing some technology closer to Microsoft's platform, obviously you're not killing it.</comment>


New Spotify Web Player UI - <AUTHOR> https://play.spotify.com <comment><USER2> still uses flash player, so I can't even open it</comment>


Twitter and Yes, Inc - <AUTHOR> http://yesitsyes.com/joining-twitter/ <comment><USER2> WTF is "Yes, Inc"?

They're so insignificant, they don't even have an Alexa ranking. [http://www.alexa.com/siteinfo/yesitsyes.com](http://www.alexa.com/siteinfo/yesitsyes.com)

None of their products seem popular either: [http://www.alexa.com/siteinfo/getfrenzy.com](http://www.alexa.com/siteinfo/getfrenzy.com) and [http://www.alexa.com/siteinfo/itswyd.com](http://www.alexa.com/siteinfo/itswyd.com)

This has SV nepotism all over it. Thankfully, Twitter doesn't have a monopoly so they can't afford to keep doing this.

If I was a Twitter investor, I would be outraged (assuming that this article is true).

<comment><USER3> There's three kinds of hiring in the tech industry. The proles must take an agility course of whiteboard coding interviews that maintain a pretense of objectivity, but actually are loyalty filter tests that only measure how much time are you willing to spend on studying shit books like "Cracking the Code Interview" which have no application in the real world.

Then there's the networked hire who basically lets people know he/she is on the market, and can land a job without having to spend 6 hours at a whiteboard answering trick questions about the space-time complexity of the interviewer's socks.

Finally, the super-networked can get in through an acquihire, so they don't have to feel bad about having spent a couple millions of investor money on their startup sabbatical.

<comment><USER4> "Coding interviews are the devil" is my favorite HN meme. This post magically appears any time anybody mentions hiring of any kind.

</comment></comment></comment><comment><USER5> This whole thread is so bad I flagged the story as a form of psychic self- defense.

Like, apparently, the rest of you, I have no idea what "Yes" does, or why I should care. I don't know why 'coloneltcb submitted the story --- they don't work at Yes.

What I do know is that if you run a company --- really any company --- and it winds down, it is a normal thing to write a post like this. In this case, the company apparently has many dozens of users, so it makes even more sense to do it. †

Meanwhile, another perfectly normal thing to do in 2016 is to "acquire" a company as a means of importing a gelled dev team all at once. If the acquired company has no other real prospects, this isn't a bad thing; it's a soft landing. The availability of soft landings makes it easier for all of us to start companies. Since most companies fail anyways, it's hard to see how they hurt users.

Almost every comment on this thread seems to be taking Yes to task for putting a brave face on an acquihire. "Spot the duress acquihire" is one of our less appealing sports here on HN. I wish we'd do some more self examination about where the impulse to play that game comes from.

† _We just went through this ourselves, and the anticipation of a thread like this is a big part of why I haven 't gotten around to writing it yet._

</comment><comment><USER6> Each of their apps have 10-50K users. Based on the number of reviews on each, I'd say they're probably on the higher end of that. But let's say they have 100K combined users...how does a company like that even get on Twitter's acquisition radar? I doubt it's for very much - probably more of an acqui-hire situation - but even those can be in the low seven figures. It's interesting that they're doing acqui-hires in the wake of layoffs.

I guess it really is all in who you know.

<comment><USER7> Twitter seems to be learning best practices of Yahoo. Acquiring small/insignificant products and generate buzz (laugh?) in tech media.

<comment><USER5> Is there some other buzz I'm missing or is the only buzz we're talking about literally just the blog post the company they acquired wrote to announce they'd been acquired?

</comment></comment></comment><comment><USER8> I still have a hard time understanding why an early stage acquisition is something to be celebrated in this industry.

</comment><comment><USER9> Joining Twitter seems like a bad move for any company, hopefully those stock options paid out well.

</comment><comment><USER10> Lots of acquisition announcements today. 'Tis the season.

<comment><USER11> Maybe it's a case of 'use it or lose it' for their acquisition budgets with the year coming to a close.

</comment></comment><comment><USER12> Has there ever been an acquisition announcement that didn't include the word "excited"?

<comment><USER13> obligatory: [https://ourincrediblejourney.tumblr.com/](https://ourincrediblejourney.tumblr.com/)

<comment><USER12> Nice, thanks. Turns out the answer is "yes".

</comment></comment></comment><comment><USER14> Surprisingly devoid of information.

</comment><comment><USER15> What do they actually do/make?

</comment><comment><USER16> [https://ourincrediblejourney.tumblr.com/](https://ourincrediblejourney.tumblr.com/)

<comment><USER17> This is awesome, and sad, just like the Valley.

</comment></comment><comment><USER18> Acqui-hire?

</comment><comment><USER19> they joined the sinking ship...</comment>


Going Serverless: AWS and Compelling Science Fiction - <AUTHOR> http://compellingsciencefiction.com/blog/2016-11-10.html <comment><USER2> The biggest pain for me in managing a system is handling all of the service configurations; making sure the ACLs are in place, that the configuration files are all in the right place and in the correct format.

Serverless isn't saving me any of that pain. I still have to configure ACLs for _everything_ , web folders in S3, ensure that my backend isn't hitting any concurrency or timeout limits, ensure that all of my routes are set up properly in the API gateway, that my DB queries are properly tuned and that someone is watching that DB...

All it's really buying me is not having to write Ansible scripts, but instead I have to write CloudFormation templates. Sure, I have to think about maintenance and troubleshooting less, but when I do have to do troubleshooting, I'm for a long, frustrating day.

As easy as it is to create VMs anymore, the serverless story is nowhere near as compelling as it should be. It makes the easy things easier, and the hard things really f'ing hard.

Concrete example: I wanted to just store the contents of a github webhook post in S3, after verifying the hashed secret. Should be a simple case of wiring together the API gateway to S3, right?

First bug: You can't test an API Gateway -> S3 connection if they are both in the same region. Known issue from back in 2014.

First hurdle: You can't pass the contents of a post to an auth hook in API gateway, just a single header value. That means I can't use the API Gateway authentication hook for this purpose; github creates a HMAC hash of the post contents.

Second hurdle: Finding the proper Velocity Template Language function (and its location in Amazon's custom libraries) to escape the JSON from the webhook body so I could pass it on to a lambda function. It's '$util.escapeJavaScript($input.body)' by the way. You're welcome.

By this point, I was wishing I had just set up a t2.tiny server running flask.

</comment><comment><USER3> Let's talk about serverless/lambda/FaaS for a minute:

I'm currently working on a side project (which if successful can easily be a product in itself) to create a service that would be the Digital Ocean of FaaS (lots of talk - I understand).

What would be the core features that devs/users/managers are looking for? (something you downright expect for v1 that would compel you to put the service to use for development/production)

Currently some of the items top of my list (in no particular order) are:

* API development (like AWS's APIG)

* Support for more languages by default

* Ability for users to a custom defined language easily

* Creation of an opinionated framework / workflow for FaaS development (you don't need to do this if you don't want to...but FaaS first apps would be done better/easier this way)

* Rewriting common products (Wordpress, Ghost, Jekyll etc.) for to be served via FaaS.

Things I don't want to work on (or would prefer to not jump into):

* Storage service (S3 alternatives)

* Databases (there are so many out there...building a cloud is hard, building a database is even more so)

Who do I feel would pay for this?

* Front end & Mobile devs who don't want to complicate a backend

* Mobile devs who don't want to complicate a backend

* Enterprises maybe?

<comment><USER4> 1) Loop detection -- don't let me shoot myself in the foot an make an infinite loop of two functions calling each other

2) Multi-datacenter -- I want to deploy to two places at least so my risk goes down

The lack of database and/or storage would be a big showstopper. It would mean that if I want build anything with state, I need to store it in another service, which costs money to move data out of. Even a very basic key store would better than nothing. You could actually build one pretty easily on top of Postgres or even SQLLite (although you'd need one per customer w/ SQLLite).

<comment><USER5> Wouldn't loop detection be equivalent to solving the halting problem? Obviously a naive approach would be disallow recursion/ mutual recursion, but there are valid use cases for those that may not result in infinite execution

<comment><USER6> I helped design a large scale JS execution engine. We used Babel to rewrite code with loops that could be interrupted by our system. Eg. `for(...){mycode();}` would be recompiled to `for(...){if(!running()){halt();}mycode();}`. The technical reason is that tight loops wouldn't respond to SIGINT, if I recall correctly. So while it doesn't stop infinite loops, it can control the program. I wasn't involved in productionizing it.

<comment><USER7> Sorry--couldn't be interrupted by SIGINT? That's arguably the point of signals. Does the runtime just queue signals rather than respond in the signal handler itself?

</comment></comment></comment></comment></comment><comment><USER8> Doesn't AWS constitute a total vendor lock-in?

<comment><USER4> If you just use Lambda, S3, and Dynamo, no not really. It's pretty easy to move to Google. If you use more of the ecosystem, yes, but so what?

Are you worried about vendor lock in with your power company?

<comment><USER9> As you know, I've had to deal with a lot of these and certainly S3 is the easiest because GCS roughly supports the S3 XML API (we do resumable uploads differently). The sad thing is that even when you say "oh just rewrite this piece to be Datastore or BigTable instead of Dynamo" that can be a pretty tall order. I think "pretty easy" depends on the organization, and how tied to a single provider it was in the first place. That said, I think it's a mistake to "avoid lock-in" religiously, because otherwise you're really getting a lowest common denominator experience. You really should use BigQuery if you're on GCP, and if you need to leave it might be tough. But the alternative is worse (using an inferior service or hand-rolled system from the get go).

P.S. As we're both in the same region of California, we're both with PG&E and can't do anything about it. It's kind of okay because they're so regulated, but how do you know we're getting a good price per kWh?

Disclosure: I work on Google Cloud (and met jedberg in person once).

</comment></comment></comment><comment><USER10> Tangentially related question. What deployment tools are people using to manage systems with many lambda functions?

We use SWF to trigger over 50 separate lambda functions in processing. We've got some very nice internally developed tools to identify which functions are out of date and help with deployment. I'm just curious what else is available to handle DevOps tasks in a Serverless environment (i.e. deploying library updates, etc).

<comment><USER11> I'm not aware of a publicly available tool that does this (without bringing in a big clumsy framework), but internally we just use a script that idempotently redeploys all the lambdas by calling update_function_code on them. It ingests package dependency changes by running "python setup.py install" and grabbing the resulting site-packages contents. This script can be subscribed to an SNS- SQS bus that listens to GitHub notifications for changes on a branch, etc.

This will flush hot lambdas, so if you have a lot of traffic it can be a bit disruptive. I can imagine comparing deployment and commit timetamps make- style, or calling update_function_configuration(Description=...) to save and later compare version/commit metadata.

AWS recently came out with [https://github.com/awslabs/chalice](https://github.com/awslabs/chalice), which is pretty sweet if you're using Python (although it doesn't seem to directly address the concern of deployment orchestration).

<comment><USER10> I've looked at chalice very shortly, and it was really nice to easily set up an API Gateway -> Lambda connection, we haven't really looked at it for our non api-gateway uses.

</comment></comment></comment><comment><AUTHOR> Has anyone used the AWS API Gateway service with lambdas to set up a small backend for a web service? The next thing I'm thinking about setting up is a slush reader with a GUI, and I'm wondering how difficult it will be to pipe binary files from S3 -> Lambda -> API Gateway -> client. It looks like I'll probably need to encode the binaries to base64 on the lambda side, and will need a library for that.

<comment><USER12> Yes extensively.

API-Gateway => Lambda isn't fun to trouble shoot.

That said, once you get it working, it works really well, and it's ridiculously cheap.

<comment><AUTHOR> Have you written anything about this, or can you point me to any resources that might help me avoid pitfalls? I'd appreciate it!

<comment><USER12> Sadly, no I haven't written anything. (if my comments are any indication--I kinda suck as a writer).

The biggest pains are:

* Mapping parameters in API Gateway

* Setting the IAM Roles for API Gateway & Lambda (the latter is required to provide execution for API Gateway)

* Setting up custom domains for your API (it uses cloudfront, but must be done via the commandline, and the process kind of sucks)

I've done this work now 3 times. Feel free to email me any questions you have: email@example.com and I'll be happy to answer, and provide guidance.

<comment><AUTHOR> Thanks, I may take you up on that down the road!

</comment></comment></comment></comment></comment><comment><USER13> Any comments from people who are happy with testability of their Lambda code: how much testing were you able to achieve, and how did you achieve that? Indirection, mocks? Frameworks? I am thinking python, but any language would be interesting.

</comment><comment><USER14> Thanks for sharing. I have used SES previously but hadn't thought to use it to route INCOMING emails ... was only using it to send ... so that concept in and of itself made it worth the read!

<comment><USER15> It's a pain to setup, as most of these things are, but it works well after that. You can trigger lambda functions to read email stored in S3, which is quite convenient. And with a library parsing the email is easy as pie.

</comment></comment><comment><USER16> I'm curious regarding the motivation behind this. You are receiving and storing emails, looking at them, and optionally forwarding them for processing. Why not use an email client, and only forward your chosen stories into a linear ingest and publish lambda job? I ask because I'm considering building something similar and I'm thinking about the rationale.

<comment><AUTHOR> If I was the only person in the organization, that might work fine. However, there are two main drivers of this methodology:

1\. I need to allow other people to simultaneously evaluate stories, and a database works well to coordinate multiple people accessing the story queue.

2\. Having metadata and stories in dynamo/S3 allows for much more flexible querying of ALL stories, to look for patterns, model the data, etc.

<comment><USER17> Some friends of mine run a science fiction site. A reasonably serious one - they've been nominated for a Hugo several times. They have WordPress and some email accounts. They do grumble about the website, but it's nowhere near their biggest problem. You are definitely overthinking this.

<comment><AUTHOR> I'd love to talk shop with your friends, if they're interested please direct them to the site or have them email me at firstname.lastname@example.com!

</comment></comment></comment></comment><comment><USER18> Serverless? Err, yeah, right...

<comment><USER19> As a community we should be correcting this proliferation of useless, misleading, and blatantly wrong terminology.

</comment></comment><comment><USER12> Shameful plug: Our developer site is serverless: [http://developer.avalara.com/](http://developer.avalara.com/)

Content is stored in our public github repo, and we use TravisCI to build Jekyll pages on commit/merge, which are uploaded and served from S3.

Interactive functionality is provided via React in the front end, and API- Gateway/ AWS Lambda in the backend.

The API Gateway/Lambda provide various proxies to our various sandbox API environments, so Developers can quickly try our APIs directly on the site.</comment>


Show HN: Top Engineering Manager Must Do's - <AUTHOR> https://medium.com/@lifeinafolder/top-engineering-manager-must-do-s-756ffab49a61#.q1qt797oy <comment><USER2> Because it is a regular blog post and does not offer something for users to play with or try out, it does not meet the spirit of the "Show HN" guidelines.

Guidelines: [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

<comment><AUTHOR> Got it. Will keep in mind. I cant find a way to edit the title for this one but will ensure for next time.</comment></comment>


MBP vs. XPS - <AUTHOR> http://www.pcworld.com/article/3179677/computers/dell-xps-15-vs-macbook-pro-15-fight.html <comment><USER2> Not a dell guy, but this new xps starts to look really nice.

Too bad they are both so expensive. I could buy a powerful pc and a good 11" ultraportable for less.</comment>


We should be grateful for good news in Georgia - <AUTHOR> https://theweek.com/articles/915446/should-grateful-good-news-georgia <comment><USER2> I would rather look at the data than the narrative: [https://twitter.com/ScottGottliebMD/status/12638990598378987...](https://twitter.com/ScottGottliebMD/status/1263899059837898754)

Georgia seems to be flat for hospitalizations, at best. I know people want good news, but... if people are being more active and less concerned AND we are nowhere near saturation, why wouldn't the virus continue to spread?

I don't know why everyone expects things to explode exactly two weeks after opening things up and that we are fine if it doesn't. The flip side of exponential growth is a relatively slow ramp up in the beginning, we don't just linearly double every week but may still see 4x in Georgia a month from now.

</comment><comment> tom-thistime What are we actually seeing? Is there data?

(I'm discarding all the meta stuff. I think we've all had a trying couple of decades in the US of A. Snark is understandable or anyway universal.)</comment>


The Brexit delusion - <AUTHOR> http://www.economist.com/news/briefing/21693568-david-cameron-will-struggle-win-referendum-britains-eu-membership-if-he-loses <comment><USER2> Heh. Really good point about the lack of sovereignty in the new global order of international treaties. Globalisation is happening and it seems the only way forward is to grudgingly sit at the table and take a few hard punches. Damned if you do and damned if you don't.</comment>


Mellified man - <AUTHOR> https://en.wikipedia.org/wiki/Mellified_man <comment><USER2> For a similar practice that we actually know happened, Japanese self- mummification is fascinating and gruesome:

[https://www.damninteresting.com/sokushinbutsu-the-ancient- bu...](https://www.damninteresting.com/sokushinbutsu-the-ancient-buddhist- mummies-of-japan/)

</comment><comment><USER3> The first time I heard about this term and practice was in Ian McDonald's "the dervish house"[1]. It's definitely one of my favourite sci-fi books of the past couple of years. I'd suggest that you check it out if you're into this kind of stuff.

1\. [https://en.m.wikipedia.org/wiki/The_Dervish_House](https://en.m.wikipedia.org/wiki/The_Dervish_House)

</comment><comment><USER4> Related to the voluntary aspect: [https://en.wikipedia.org/wiki/Sokushinbutsu](https://en.wikipedia.org/wiki/Sokushinbutsu)

</comment><comment><USER5> Just an ancient urban legend. I flagged this.

<comment><USER6> If you think a post is inappropriate for HN, just flag and move on.

 _If you flag something, please don 't also comment that you did._

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)</comment></comment>


Stencila: An open source office suite for reproducible research - <AUTHOR> https://stenci.la/ <comment><USER2> > The calls for research to be transparent and reproducible have never been > louder. But today's tools for reproducible research can be intimidating - > especially if you're not a coder.

As someone (a software engineer) who has been trying (struggling) to reproduce biology research lately, I say amen. Hallelujah.

But. It's time to accept coding as a core skill. Science has more to learn from software engineering than it realizes. Software engineering (aka coding) eats reproducibility for breakfast, even when hundreds or thousands of "collaborators" are involved. These days, it's rare for a single biology researcher to produce (publish) code that is easily reproducible by an external researcher.

<comment><USER3> It's a bit more complex than that, as it's not just having access to the software, but to the whole environment. The move towards reproducible builds and configurations (Nix and the like) is a good thing for this, as well.

<comment><USER4> I've come across this problem before (as a third party - I don't do research) and I've considered writing software to help solve it. Maybe something like a Jupyter notebook with an attached mini-filesystem that can be easily shared with colleagues? What do you think would be a required to solve this?

<comment><USER5> If you're really interested in this, you should contact Ivo Jimenez and Felix Z. Hoffmann. Although there's not much information available, under the heading of " Reproducible Computational Research – a case study " at [0], they tentatively explored such an idea at a hackathon in Cambridge in May.

[0] [https://elifesciences.org/labs/bdd4c9aa/elife-innovation- spr...](https://elifesciences.org/labs/bdd4c9aa/elife-innovation- sprint-2018-project-roundup)

</comment></comment></comment></comment><comment><USER6> This looks interesting. I'm actually rather surprised at the lack of quality in most open source 'office' alternatives. I'm not sure why, but open/libreoffice is almost unusable for heavy duty office work. Microsoft products are the clear winners if you require productivity and stability. I really hope that changes soon, because I feel it's a major factor that holds back widespread Linux adoption.

<comment><USER7> There's a ton of mindnumbing testing and grinding work that honestly no one really wants to do, involving bit-level accuracy in many cases. We ([https://sheetjs.com/](https://sheetjs.com/)) are attacking the file compatibility problem (our major open source project is [https://github.com/sheetjs/js-xlsx](https://github.com/sheetjs/js-xlsx), used in stencila's spreadsheet converters) and it is painfully obvious that OO/LO cuts corners at certain places. For example, the mathematics of Calc is intentionally bugged:

> ignore the last two bits for many stuff to improve the user experience.

[https://bugs.documentfoundation.org/show_bug.cgi?id=83511](https://bugs.documentfoundation.org/show_bug.cgi?id=83511)

A successful alternative to Microsoft Office has to start from a level of compatibility that no current developer in existing solutions has expressed interest in attaining.

<comment><USER8> As you referenced that bug and there also gave an example involving fractions and =0.1+0.2 you would certainly also agree that the binary floating point representation of =0.1+0.2-0.3 is not 0.0, still Excel displays 0 as result (and so does Calc) instead of 5.551115123125783E-17 because users expect that.

Your bit-level accuracy approach isn't as simple when it comes to user experience and Excel compatibility.

<comment><USER8> Regarding your fraction example there:

=RAWSUBTRACT(0.1,-0.2,1/3) => -0.033333333333333

=RAWSUBTRACT(0.1,-0.2,2/7) => 0.014285714285714

So which one is closer to 0.3?

</comment></comment></comment></comment><comment><USER9> Not sure why that page does not include a link (that I can find anyway) to this intro video that the creator, Michael Aufreiter, put together:

[https://www.youtube.com/watch?v=EzrR96PDnO8](https://www.youtube.com/watch?v=EzrR96PDnO8)

</comment><comment><USER10> Interresting!

Does Stencilla offer any kind of author collaboration? Publication are usually not one person efforts and research teams oftentimes are not working in the same location.

In my experience the GDocs or Word comments and revise mode are heavily used in collaborations.

<comment><USER11> I was wondering that too. It seems like there might be collaborative editing, given dat project involvement?

<comment><USER12> Yes, we'd love to have real-time collaboration, preferably in a decentralised way using Dat!

</comment></comment></comment><comment><USER13> tl;dr - an interactive notebook, that accepts Jupyter files, for folks who usually use MS Office.

From their FAQ:

> Stencila allows you collaborate with colleagues who use other tools than > RMarkdown and Jupyter Notebook, without you having to give up your favourite > tool. Stencila Coverters make it possible to open documents in various > formats (R Markdown - Rmd, Jupyter Notebook - ipynb and so on) in Stencila. > The conversion is lossless for all interactive parts (such as code cells).

Nice to see dat part of the conversation.

<comment><USER14> Those coverters are really good for undercover work.

</comment></comment><comment><USER15> How's stencila supposed to produce reproducible research?

No one's fudging the formulae, they're fudging data. And stencila will digest whatever data you give it, real or fake

<comment><USER16> Hear, hear! Even if they're not fudging them as such, they may have cocked up the measurements. You figure that out from hard experience in the field (or relevant experience from another, which is often very helpful), not with some trendy piece of software.

</comment></comment><comment><USER17> Has anyone else noticed this trend of integrating data science tools into one package that installs and potentially monitors you? For example this seems like another `Anaconda Navigator` clone. Or am I missing something here?

</comment><comment><USER18> I'm surprised that there appears to be no link to the Github Repo on the homepage: [https://github.com/stencila/stencila](https://github.com/stencila/stencila)

<comment><USER19> There is a link in the "Contribute" section.

</comment></comment><comment><USER20> If any contributor sees this: The "Learn" link in the footer points to a 404 Github page.

Is this usable for daily usage? And can we output PDF and use LaTeX for publications?

<comment><USER12> Hi, Stencila dev here. In my opinion, Stencila is not ready for daily usage. But we are looking for beta testers who are willing to put up with bugs and crashes and help us shape the framework. We have converters, based on Pandoc, which are able to convert to both PDF and Latex - although they are also in preliminary development.

</comment></comment><comment><USER21> So this is a reproducible research application suite built on Node.JS, the epitome of non-reproducibility and characterised by fast pace, little care for compatibility, and an ecosystem of volatile libraries?

<comment><USER12> Hi, Stencila dev here, thanks for the feedback. There are several modular components that make up Stencila. The user interface modules are indeed built using Javascript, and Node.js is used in a number of places including the desktop and CLI apps and format converters. But for code execution, Stencila does not rely on Node, and users can use R, or Python or SQL.

> Reproducible research depends on reproducible execution, which depends on a > reproducible environment, which depends on a reproducible set of libraries > and frameworks.

Completely agree. We are trying to make it easy for people to use reproducible libraries and environments. To this end, we are developing Nix environments (a highly reproducible way of defining computing environments) which include Stencila "execution contexts" for R, Python etc with standard libraries included. These environments can be connected to the user interface. See [https://github.com/stencila/images/](https://github.com/stencila/images/) for more details.

<comment><USER21> Thanks for the explanations!

</comment></comment></comment><comment><USER22> So this is jupyter for "excel", right? Nice!</comment>


Shooting Yourself in the Head with Threads - <AUTHOR> https://www.stev.org/post/shootingyourselfintheheadwiththreads <comment><AUTHOR> Other than the clickbait-y title, the article is a decent one.

I only want to point out that the described issue with cond var signal/broad cast while not holding the mutex is not correct. You are definitely allowed to call signal/broadcast after releasing the lock, as long as the state was modified with the lock held. In fact this is a common performance optimization. Of course trying to be clever and attempting to modify the state outside the critical section (or not acquiring the mutex at all) will only lead to pains.

Assuming that waits are done correctly (i.e. checking the condition in a loop), what the author is probably experiencing is missed wakeups due to using signal (as opposed to broadcast) with multiple waiters waiting for slightly different conditions. This is not an issue with condvar per se, but with the client code itself.

Normally broadcast fixes many missed wakeup issues but not always.</comment>


Why digital privacy and security are important for development - <AUTHOR> http://www.guardian.co.uk/global-development/poverty-matters/2011/aug/04/digital-technology-development-tool

<comment><USER2> This is part of the reason I developed KEYBOX for iPhone ( <http://www.jayfuerstenberg.com/keybox/> ).

It gives iPhone users the tools to be more secure and to protect their own privacy. Sadly, most people won't try it until they become victims of security breaks and it's more of a preventative tool than a corrective one.</comment>


Indexing JSON logs with Parquet - <AUTHOR> http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-with-parquet.html <comment><USER2> parquet would be a lot more interesting if it could be freed from all the java/hadoop/spark baggage.

I don't want hadoop. I don't want spark. I don't want drill. I don't want presto.

I don't have big data. I do happen to have a few hundred gigabytes of compressed csv files which would likely be a lot faster to slice and dice if they were stored in a compact column format.

The other day I did something like

      $ zcat logs/*.gz | cut -f 3,5 | fgrep -w 23 | count_distinct

It took about 30 minutes on a single machine, mostly from the IO/decompression overhead.

All I want to do is something like

      $ zcat logs/*.json.gz | json2parquet parquet_logs
      $ parquet-filter --output-fields src dport=23 parquet_logs | count_distinct

<comment><USER3> I can see why you'd want to keep it non-Java and simple. I will admit that I am biased since I work with Spark all day, but I think you'll eventually want to write a query where you wish you had Spark SQL or could manipulate it as a Spark RDD.

Spark actually isn't too painful for this because you can run it locally in a single process. It is not exactly as elegant as the UNIX way you outline in your second example, but it isn't as horrifying as submitting a YARN job or spinning up a cluster.

In spark-shell:

        // Skip this conversion if you're just running one query. Do it if you're running many.
        spark.read.json(input paths).write.parquet("/target/path")
        spark.read.parquet("/target/path").createOrReplaceTempView("logs")
        spark.sql("select count(distinct src) from logs where dport = 23").show()

The incremental conversion of your JSON data set to Parquet will be a little bit more annoying to write in Scala than the above example, but is very much doable. There is also a small amount of overhead with the first spark.read.parquet, but it's faster on a local data source than it is against something like S3.

<comment><USER2> Yeah.. I agree. It's not really that I don't want to use spark, it's more that I don't want to have to use spark just to convert some data files.

If I can write a short tool that wraps the

      spark.read.json(input paths).write.parquet("/target/path")

As

      spark.read.json(argv[1]).write.parquet(argv[2])

and then use it like

      json_to_parquet /logs/json/2016-12-28.gz /logs/parquet/2016-12-28

or something, that would work.

I just wish there were some more cli friendly tools for the cases where the grep | cut | sort pipeline works, but you'd like to just make it a bit more efficient.

</comment></comment></comment><comment><USER4> I'm curious about where the threshold lies between "ELK Stack is good enough" and "We need Parquet".

<comment><USER5> My employer is doing a lot of this, mostly for cost reduction. Keeping live data in ElasticSearch (or some other horizontally scalable system) is expensive. Far cheaper to keep a small portion live and the rest in S3.

Also worth mentioning that managed ES in AWS has a hard limit of 20 data nodes with 512GB of EBS. We've outgrown that and are now looking hard at alternatives like this.</comment></comment>


Engineers Nine Times More Likely Than Expected to Become Terrorists - <AUTHOR> http://science.slashdot.org/story/15/11/25/1326242/engineers-nine-times-more-likely-than-expected-to-become-terrorists <comment><USER2> Terrorist ideologies are absolutist and fundamentalist. The mental habits that engineering cultivates are unfortunately compatible with this way of thinking in political and spiritual/religious realms. There is an absolute truth and an absolute right and wrong, etc.

I've never met a "terrorist" but I have known a surprising number of engineers who subscribe to totalitarian politics and/or religious fundamentalism. At first I found this shocking and confusing.

</comment><comment><USER3> One of the skills that engineering teaches and/or selects for is something like "taking ideas seriously."

If you take chemistry and conservation of linear momentum seriously, you get rocketry. If you take "America is the Great Satan" seriously, you get terrorism. There's a fundamental similarity between the processes.

<comment><USER4> _One of the skills that engineering teaches and /or selects for is something like "taking ideas seriously."_

I respect what you're saying but I think there's room for nuance. They tend to be competitive and driven. In a sense you can't become an engineer without those traits because engineering, or the application of scientific principles in design, is hard. You have to work hard to become an engineer.

Engineers are highly capable of implementing plans because of their method of thinking and acting. Daydreaming about something is different to actual implementation. It's why the phrase _Ideas are cheap, execution is everything_ is a common refrain in these parts.

</comment></comment><comment><USER5> What about the social components? It seems too obvious a stereotype not to at least investigate whether or not the "lonely nerds" are seeking social identity and belonging in groups like these.

</comment><comment><USER6> There will be harmony among the engineers as soon as we all agree on the best text editor.

</comment><comment><USER7> If anyone knows the API endpoint to consult membership details of terrorist organizations, including level of studies of their members, I'd like check myself that we are really "9 times" more likely, and not 8.4 or 11.3.</comment>


Good Quality DOSBox Video Capture and FFmpeg Options - <AUTHOR> https://susam.in/blog/good-quality-dosbox-video-capture/ <comment><USER2> There's a whole lot of additional subtlety to

> my contacts who might be on devices that do not support playing AVI files, > so I decided to convert this to MP4 format

– a file being an AVI tells you practically nothing about the actual codecs and streams contained within, and the same stands for MP4.

It seems DOSbox writes AVI files that contain video encoded in its own ZMBV codec (see [https://wiki.multimedia.cx/index.php/DosBox_Capture_Codec);](https://wiki.multimedia.cx/index.php/DosBox_Capture_Codec\);) even if OP's contacts had devices that could play AVIs, it's likely they couldn't play that AVI anyway...

        Input #0, avi, from 'dosbox_000.avi':
          Duration: 00:00:03.64, start: 0.000000, bitrate: 1532 kb/s
            Stream #0:0: Video: zmbv (ZMBV / 0x56424D5A), pal8, 640x400, 98 kb/s, 70.09 fps, 70.09 tbr, 70.09 tbn, 70.09 tbc
            Stream #0:1: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, 2 channels, s16, 1411 kb/s

<comment><USER3> At the time of me writing this, you missed a space between the ");" and the url for the dosbox codec.

For people on mobile/lazy: [https://wiki.multimedia.cx/index.php/DosBox_Capture_Codec](https://wiki.multimedia.cx/index.php/DosBox_Capture_Codec)

\-----

For those interested in an even deeper dive:

Decoder: [https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/zmbv...](https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/zmbv.c)

Encoder: [https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/zmbv...](https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/zmbvenc.c)

This implementation was written almost immediately after dosbox added the codec.

<comment><USER4> How hard would it be to let dosbox emit a newer container format? I remember AVI being quite bad re overhead.

<comment><USER2> I think AVI is being used because it's apparently _very_ easy to write a valid AVI file without extra libraries:

[https://github.com/svn2github/dosbox/blob/acd380bcde72db74f3...](https://github.com/svn2github/dosbox/blob/acd380bcde72db74f3b476253899016f686bc0ef/src/hardware/hardware.cpp#L132)

<comment><USER5> Old image and video formats are great if you want something quick and dirty.

</comment></comment></comment></comment></comment><comment><USER6> I went there, expecting some information how DOSBox was hacked to emulate some old-fashioned CRT-visuals and output 4K-content. Got instructions on how to convert AVI to another format with ffmpeg. Great!

<comment><USER7> It gets even pixel aspect ratio wrong, or rather, doesn't account for it in any way.

In some modes, like 320x200, pixels are 20% taller than "normal" square pixels. If you don't account for that, everything looks vertically squished.

<comment><USER8> Isn't that the right DOSBOX behavior? I think it should maintain the original aspect ratio as it looked in the old CRT monitors. I remember drawing circles in CRT monitors on IBM PCs and the circles did look vertically squished. Why should DOSBOX or FFMPEG alter that original aspect ratio? Altering that artificially would be like lying about what the real output looked like in the good old days.

<comment><USER9> If you use DOSBox with default settings, the pixel aspect ratio will be square. Old PCs, as you remember, had non-square aspect ratios, and so the graphics will look incorrect, they will look squashed horizontally.

<comment><AUTHOR> I am running DOSBox with default settings. I draw a square and then a circle with these lines of Logo code:

      REPEAT 4 [FD 1 RT 90]

      REPEAT 360 [FD 1 RT 1]

I find that both the square and the circle appear squashed. On measuring the pixel height and width of these figures as they appear on my desktop, I see that the square is 400 pixels in width and 320 pixels in height. That's an aspect ratio of 5:4. Similarly, the circle is 460 pixels in width and 364 pixels in height. That's an aspect ratio of 1.26:1.

I find the same aspect ratio in full screen (Alt + Enter) mode too. In the full screen mode, the square is 200 pixels in width and 160 pixels in height and the circle is 230 pixels in width and 182 pixels in height.

Here are the screenshots:

\- [https://imgur.com/l9YpTqp](https://imgur.com/l9YpTqp)

\- [https://imgur.com/WTwfSzq](https://imgur.com/WTwfSzq)

\- [https://imgur.com/CjMYJN6](https://imgur.com/CjMYJN6)

<comment><USER10> Edit the dosbox configuration file and change aspect=0 to aspect=1, it'll resize the output to be aspect correct but it might be a bit blurry (IIRC the scale3x scaler will do a pixel-scale and then _that_ will be bilinearly resized, giving mostly crispy pixels with a bit of in-between bleeding which is probably the best you can get with an LCD monitor)

<comment><AUTHOR> With aspect=1, the square is 400 pixels high and 384 pixels wide, not exactly an aspect ratio of 1:1 but quite close, 1.04:1.

The parent of my earlier comment, i.e., the one by klodolph, claims that the pixel aspect ratio would be square by default in DOSBox, so I thought I would verify it once. As per my experiments, that does not seem to be the case.

<comment><USER11> I think what he meant is that,

The pixels are square in DOSBox, but the graphics from these old programs are designed for non-square pixel ratio. So, if you display it with square pixel, the display aspect ratio will be wrong.

(Of course, this still contradicts with what neatcoder said, but I can't remember if these programs were "squished" on CRTs or not.)

<comment><AUTHOR> See these screenshots of SCREEN 2 in GW-BASIC running in DOSBox:

\- [https://imgur.com/NZUGL1e](https://imgur.com/NZUGL1e) (aspect=0)

\- [https://imgur.com/rnQOfA4](https://imgur.com/rnQOfA4) (aspect=1)

The one with aspect=1 looks very odd.

<comment><USER9> The one with aspect=1 looks incorrectly configured.

There are a number of different graphics modes, they have different pixel aspect ratios, you can also adjust the monitor, it is possible that back in school you were using a different mode or a different monitor. I think there are some other factors that can affect this.

<comment><AUTHOR> Back in school, I played with SCREEN 0, SCREEN 1, and SCREEN 2 documented in the GW-BASIC user manual. Here's a copy: [http://www.antonis.de/qbebooks/gwbasman/screens.html](http://www.antonis.de/qbebooks/gwbasman/screens.html) .

The reason why I stick to aspect=0 (the DOSBox default) is that this is the one that more closely reproduces the experience I had with the CRT monitors at school. I agree that the actual monitor or the way it is configured could affect the aspect ratio we observe. Perhaps ours was not adjusted to produce 1:1 square while working with IBM Logo. But it did produce 1:1 square with GW- BASIC SCREEN 1 mode.

<comment><USER9> CGA/EGA/VGA is supposed to have a pixel aspect ratio of 1:1.2 in 320x200 mode. If you adjusted the monitor to give 1:1 square pixels, then this was unusual.

<comment><AUTHOR> I think I know why the screenshot for SCREEN mode looks odd with aspect=1. The documentation at [https://www.dosbox.com/wiki/Dosbox.conf](https://www.dosbox.com/wiki/Dosbox.conf) mentions,

> Do aspect correction. It only affects non-square pixel modes like VGA Mode > 13h, which has a resolution of 320x200 pixels and is used by many DOS games > (DOOM, etc).

However SCREEN 2 mode of GW-BASIC has a resolution of 640x200 ( [http://www.antonis.de/qbebooks/gwbasman/screens.html](http://www.antonis.de/qbebooks/gwbasman/screens.html) ).

<comment><USER9> That mode is also non-square, with an aspect of 0.6:1. The IBM PC and its various video cards in the early days used a number of tricks to keep the cost down, one of those tricks is to keep the horizontal and vertical sync signals the same for different modes as much as possible. (Recently I've started to appreciate just how many tricks they used to keep the cost down, without sacrificing _hardware_ reliability in the process!)

For both 320x200 and 640x200, HSYNC ran at 15.75 kHz, VSYNC ran at 60 Hz. When you go from 320x200 to 640x200, all that happens is the pixel clock (the rate at which you read out from RAM) is doubled, so you get exactly two pixels horizontally packed in where there used to be one pixel. The older hardware, like EGA video cards, can only generate one other HSYNC speed: 21.8 kHz, for special 350-line modes. When VGA came out, it doubled the HSYNC frequency and, for these modes, would just read each row out twice.

[http://www.minuszerodegrees.net/video/bios_video_modes.htm](http://www.minuszerodegrees.net/video/bios_video_modes.htm)

So SCREEN 1 should have 1.2:1 ratio, and SCREEN 2 should have a 0.6:1 ratio.

One of the better ways to verify this kind of thing is to compare box art from DOS video games to screenshots taken from DOSBox or sprites extracted from the data files. You can see that the screenshots from DOSBox match the sprite files, if you leave aspect unadjusted, but if you look at the box art from the retail packaging, you will (often) see artwork and screenshots with the correct aspect ratio, just like you see on real hardware.

Here's a DOOM Wiki page discussing it, with a number of pictures and sources: [https://doom.fandom.com/wiki/Aspect_ratio](https://doom.fandom.com/wiki/Aspect_ratio)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> Good guides except for one things - Constrained Baseline isn't a thing for years now. Even Android 2.x devices supported at least Main profile, so I strongly recommend NEVER using Baseline H.264 profile. Using it will absolutely trash video quality at the same bitrate.

</comment><comment><USER13> FWIW, I've found that for screen captures, doing a completely lossless capture with FFMPEG often results in much smaller files. This is true mostly for situations where there is a lot of duplicate frames, not high action video games. And of course the quality can't be beat.

<comment><USER14> > doing a completely lossless capture with FFMPEG

How? Are the iframes losslessly compressed? For some reason I thought they were equivalent to JPEG.

<comment><USER15> If you configure the encoder for lossless compression (-crf 0) then it won't quantize the image data. There might be rounding errors in the DCT, but it's pretty close to lossless. And there might still be a loss of data in other stages of processing, like converting RGB to YUV 4:2:0 or something.

[https://dsp.stackexchange.com/questions/27752/is-dct- discret...](https://dsp.stackexchange.com/questions/27752/is-dct-discrete- cosine-transform-of-type-2-lossless-or-lossy)

<comment><USER11> So basically JPEG with 100% quality (a JPEG with all quantization table factors being being 1).

I personally won't call it lossless then, since no one will call 100% JPEG (even the 4:4:4 one) "lossless" (there is an entirely separate "true" lossless JPEG that no one uses).

But I'm not trying to argue if it's the term H264/video codec community uses.

</comment></comment></comment></comment><comment><USER16> I recommend "-tune animation" for content like this.

Also I second izacus' "don't use baseline". I would even use a higher profile, for quality's sake. For bad clients you would always convert content if needed. Unless of course, you are one of those users constrained by them in the first place.

</comment><comment><USER17> Anyone knows what theme is used in this blog?

<comment><USER18> Seems to be based on this repo : [https://github.com/susam/susam.in](https://github.com/susam/susam.in)</comment></comment>


How do users search your website? - <AUTHOR>

I'm  working on some search tools to enhance my start-up's existing infrastructure. Users will be searching highly-structured data, and I would like to provide them something better than SQL's LIKE operator. I've looked into some options, like the Google Appliance, but many commercial tools are currently too costly for our bootstrapped company. It may be better, in the end, to build my own.<p>Are there good tools or resources that the YC community would recommend? <comment><USER2> If you're looking to search an existing MYSQL DB I hear Sphinx is a good option. I personally use Ferret but everyone seems to think it sucks horribly so I'm sure you'll hear that from people. I have had no problems with Ferret at all however. Sphinx and Ferret are both Ruby based.

Solr is also an option but it tends to be slower than Ferret and require more resources.

Here is a recent post of mine that gives a quick comparison of Ferret and Solr. [http://www.embought.com/blog/show/10?t=SOLR-vs-Ferret-as- a-S...](http://www.embought.com/blog/show/10?t=SOLR-vs-Ferret-as-a-Search- Engine-Indexer)

<comment><AUTHOR> Thanks for the info, this is really useful. Currently I'm using mostly Python, but a parallel Ruby install is not out of the question.

<comment><USER3> +1 for sphinx.

It's fast and memory efficient, an in my experience rock solid. Comes with excellent python bindings, and if your python is django, there is also a really nice django-sphinx project on google code. Solr is also good, but it does require java and behaves slightly differently. I would recommend you give both of them a look.

</comment></comment></comment><comment><USER4> I've built custom search engines using Lucene before -- <http://lucene.apache.org/java/docs/index.html> \-- problem is that you have to build a lot of extra stuff to make it work for your scenarios.

Wikipedia lists a ton of sites that rely on Lucene: <http://en.wikipedia.org/wiki/Lucene>

You might also want to look at Solr -- I haven't tried it, but sounds like you have to make less special sauce to get it running. <http://en.wikipedia.org/wiki/Solr>

Ferret is actually a port of Lucene to Ruby.

If I were engineering something, I'd probably want to use Facebook Thrift to make my searches a service that can be called from anywhere -- just use the Java Thrift binding on top of a Lucene process, and then call into it from my web app.

</comment><comment><USER5> I will say it depend on how much data you have to search and how tightly link is the search to your business requirements.

If you are in search of some casual search mechanism with few thousand records to index, go for open source; if not you may have to implement your own search engine. I am in the business of search engine though I can't tell you much without violating my non-compete agreement.

Another option is to look at our hosted search solution not yet in beta. What will be interesting for you will be the data search service, a completely customizable search solution. Check it out here <http://www.intelliverb.com/PESS/>

</comment><comment><USER6> Take a look at our open-source tool, <http://hounder.org>

Among other sites, it powers wordpress.com (over 3M weblogs indexed).

We are looking for feedback and feature suggestions, you can also join our discussion group: <http://groups.google.com/group/hounder>

</comment><comment><USER7> I haven't played with it myself (though I intend to) but Lucene might be an option. <http://lucene.apache.org/java/docs/index.html>

It has also been ported to the zend framework if PHP is your thing.

</comment><comment><USER8> For a site search, see <http://news.ycombinator.com/item?id=184707></comment>


Show HN: Pacman Canvas in HTML5 - <AUTHOR> http://pacman.platzh1rsch.ch/ <comment><USER2> Cool stuff! The game feels really smooth.

If you're interested, this article has a cool explanation of the original Pac- Man's ghost logic: [http://gameinternals.com/post/2072558330/understanding- pac-m...](http://gameinternals.com/post/2072558330/understanding-pac-man- ghost-behavior)

<comment><AUTHOR> hey there, thanks for your feedback, glad you like it =) I took care of implementing the ghost logic from the original game (not 1:1 but close), I think I even based it on the article you just recommended, cool stuff!</comment></comment>


Unofficial Github Latest Commits Widget - <AUTHOR> https://github.com/TylerLH/github-latest-commits-widget

<comment><AUTHOR> planning to add a few more features shortly. if anyone has any requests/suggestions, i'm glad to hear them!</comment>


Ask HN: How to get back to life and improve? - <AUTHOR> Hey HN folks!
I could really use some wisdom and advice!
I am going through some rough phase of life. Although, I have a stable job but my personal life is not very stable. I have been in a relationship and then a marriage since last 7 years. Previously, I always worked on side projects apart from my job and was always continuously learning on the side. But since last 7 years I have not done anything except my day job. I don't have anything to show on my Github profile or any side project. I was too emotionally attached in the relationship and it sucked all my time. I wasted too much time in an abusive relationship and got depressed. All my evenings and nights were either spent drinking or slacking time on social feeds, reddit or HN. I lost my ability to focus on anything and it has been a long time I built anything from scratch myself. I never realized that I slowly killed my learning ability.
From the past few months, even though I am trying to get a side project started but I am unable to get anything started.
Has anyone been in a similar state like this before? How did you come out of it? <comment><USER2> As you're building yourself and your skill back up, think about how you're pacing the goals you set for yourself.

Any chance you enjoy video games like Skyrim or Zelda? Think about how the tasks a character gets start out really simple/short and increase in difficulty/duration over time. The pacing of those tasks is what makes the game fun and addictive to play! If you feel your skills have fallen behind, allow yourself to reset your expectations just like if you started Skyrim over with your character at level 1. Leveling might be frustrating at first, but you'll probably level up faster than you did the first time around and eventually you'll get back to where you were before.

<comment><AUTHOR> Thanks. I will use your advice!

</comment></comment><comment><USER2> What's your purpose for getting a side project going?

<comment><AUTHOR> mainly learning and doing something on own. It has been a long time, I haven't personally built anything. My job is not allowing me to expand my skillsets.</comment></comment>


Show HN: Backtick – A console for bookmarklets and scripts - <AUTHOR> http://backtick.io/

<comment><USER2> I tried creating an "Add to Pinboard" backtick command:

[https://gist.github.com/lox/7312183](https://gist.github.com/lox/7312183)

The script works fine in Chrome's console, but fails as a backtick script.

Aside from that, icons served from http show mixed content warnings on https pages. Would be good to require icons are either data uris or https.

<comment><AUTHOR> Ah, thanks for reporting. It seems that having comments in a command messes up the execution, that's obviously something I'm going to fix ASAP. For now, just remove the comments and it should work.

You're right about the mixed content thing, I should make sure to use HTTPS urls on all icons. Data URIs would be convenient, but it would make the amount of data that is fetched on first run (to get all default commands) unnecessarily big.

</comment></comment><comment><USER3> I seriously love this and am already making some commands for personal use :)

However is there currently a way to pass parameters to a command? Right now I'm using window.prompt() to get user input. It'd be nice if I could enter something like 'mycommand:parameter1,parameter2'. Maybe an excuse to fork the repo...

Great work though!

 __EDIT __I had only been using the extension on backtick.io before, but when I use it on other pages the console does not appear. This appears to be a problem with the CSS - when I inspect element on the page I can see that the HTML has rendered. By default opacity is set to 0 on the #console div - setting it to 1 fixes the problem.

<comment><AUTHOR> Adding parameters to commands is something I've got planned for a future release, just need to figure out an elegant way to do that.

Have you updated to the latest version of Chrome? I have never run into the problem you're describing.

<comment><USER4> I'm having a similar problem, except when I'm pressing backtick nothing happens at all, the __backtick__ div stays empty. similarly, it works fine on the backtick website Chrome 29.0.1547.57 on arch linux.

EDIT: in the interests of clarity, pressing the backtick the first time injects an empty __backtick__ div into the page nothing further happens.

EDIT: Fixed with update to latest chrome. Thanks for the extension, seems great.

</comment></comment></comment><comment><USER5> I thought this was for bookmark searching, that would have been far more useful.

<comment><USER6> I was hoping the same. I was nearly giddy with excitement, then I tried searching... no dice... re-read the description... ah.

I might try making this a reality. It'd be incredibly useful.

<comment><AUTHOR> I'm sorry to let you down, but what you're describing is not what I have visioned for this. But hey, if you're going to create this on your own, the source code for Backtick is available on GitHub ([https://github.com/JoelBesada/Backtick](https://github.com/JoelBesada/Backtick)) if there's anything you need to know!

<comment><USER5> Free software is never a let down.

</comment></comment></comment></comment><comment><USER7> This is a nice idea and the design is elegant. However, I already use Vimium's "b" key command for this, and it comes with so many other features too.

<comment><AUTHOR> Thanks! Vimium's "b" hotkey is not exactly the same though.

Backtick comes with a bunch of bookmarklets already available without you having to save them to your browser first. I'll also keep adding more commands to it as I get suggestions and find more to add.

The point is, Backtick allows you to discover new bookmarklets just by searching in the console while Vimium only has those that you've added yourself.

<comment><USER8> Please add pineapple bookmarklet: [http://pineapple.io/discussion/bookmarklet- released](http://pineapple.io/discussion/bookmarklet-released)

</comment></comment></comment><comment><USER9> Compatibility and UX qualms aside, this is a really fun little project.

It'd be fun to bake something like this into your app even, which I'm sure you've considered, but it'd make a nice little poweruser/admin tool, especially for those who prefer keyboard over mouse. For instance, I'm picturing Github with their command-bar backing it.

</comment><comment><USER4> Really like this, I think something that would be really helpful would be a repository website where you can collect any of these commands similar to will bonds package control. Automatic downloading doesn't have to be configured to start with, simply a central searchable place for people to share and find backtick commands. Thanks again.

</comment><comment> m-app Cool! Created a Pocket command from the bookmarklet, went really smooth:

[https://gist.github.com/michielappelman/7319013](https://gist.github.com/michielappelman/7319013)

</comment><comment><USER10> I use GleeBox to execute the hundreds of bookmarklets I've found and created over the years.

[http://thegleebox.com/](http://thegleebox.com/)

</comment><comment><USER11> I'm finding that it swallows the first keystroke after `. So `fontBomb results in ontBomb appearing in the popup.

Windows; Chrome 30.0.1599.101

<comment><AUTHOR> Are you using Vimium, or some other extension that listens for the ` key? If so, you should be able to fix that problem by either rebinding the Backtick hotkey to something else in the settings, or by unbinding the ` key from the given extension.

If you're using Vimium, add "unmap `" to the custom key mappings in the advanced options.

</comment></comment><comment><USER12> This is phenomenal but I feel a lot of people are not understanding its purpose. Perhaps you could better explain what it's really for.

</comment><comment><USER13> Fantastic. I would never have thought of that. Great execution.

Is there a repository of bookmarklets that this could hook into?

<comment><AUTHOR> The ones you see on the demo page is what comes with Backtick out of the box. I'm constantly adding more bookmarklets to the default list, just tweet at me at @JoelBesada if you want to suggest anything to be added.

</comment></comment><comment><USER14> Any chance of this coming to Safari too?

<comment><AUTHOR> Since my time is limited, I try to keep the scope of the project down to something I can handle. This means I'm currently only focusing on Chrome, but if anyone wants to port the extension to other browsers, the source code is available on GitHub for them to fork.

</comment></comment><comment><USER15> This is awesome. Quite possibly one of the best solutions I have seen so far to manage bookmarklets.

</comment><comment><USER16> for some reason,the only site that backtick is working for me on is the backtick site itself.

</comment><comment><USER17> not working on spanish iMac keyboard...

<comment><AUTHOR> You can change the hotkey from the extension settings.

</comment></comment><comment><USER18> upvote for fontBomb!</comment>


Show HN: A live map of sunlight on earth - <AUTHOR> https://sunlight.live <comment><USER2> You'll find this sort of visualization on a lot of ham radio websites, since skywave propagation between two points varies based on the amount of sunlight among other things. For instance, [https://www.pskreporter.info/pskmap.html](https://www.pskreporter.info/pskmap.html)

The Geochron ([https://www.geochron.com/about/what-is- it/](https://www.geochron.com/about/what-is-it/)) is a mechanical map+clock created in 1965 that shows it.

<comment><USER3> Does more sunlight help or hurt skywave propagation?

<comment><USER4> It's not quite as simple as that, because there are multiple ionospheric layers with different physical properties that affect RF propagation.

An over-simplified explanation is that the outer layers of the atmosphere persist continuously, and can be used for very long distance communication, but only reflect radio waves at relatively low frequencies (<10MHz). The lower layers that become ionized during the day absorb some of the radio waves, reducing overall efficiency and range, but also allow operation at higher frequencies which would be unusable at night.

And you can get very efficient long-distance propagation along the terminator at dawn and dusk: [https://www.qsl.net/w2vtm/grayline.html](https://www.qsl.net/w2vtm/grayline.html)

</comment></comment></comment><comment><USER5> This is a really useful visual if one has contacts or meetings in other world cities. The left edge of the light hump is roughly 6am, the center is noon, and the right edge is roughly 6pm. The center of the dark hump is midnight, give or take DST.

Gnome2 had a map like this in its calendar/time drop-down when you add cities to it: [https://help.gnome.org/misc/release- notes/2.22/figures/rnuse...](https://help.gnome.org/misc/release- notes/2.22/figures/rnusers-international-clock.png.fr)

It's one of the main reasons I still use Mate. No other desktop environment keeps it handy like this. I try them for a bit, then always go back to Mate.

<comment><USER6> Offtopic but MATE is also literally the only Linux desktop that gets multi- head right, especially when it comes to hot-adding and hot-removing displays. All others do weird things to the window layout or crash when you add or remove a monitor while the machine was asleep.

<comment><USER5> Interesting. Perhaps because it was one of the only ones to stick around long enough to mature, until it was abandoned CADT style.

</comment></comment></comment><comment><USER7> Interesting that these maps are always binary: light or dark, when the intensity of light varies based on how high the sun is in the sky. Is there a version of this map that takes that into account?

<comment><USER8> You would need to have the data of how high the sun is in the sky, plus to be accurate you would need to have cloud data as well.

<comment><USER9> This [https://www.timeanddate.com/astronomy/different-types- twilig...](https://www.timeanddate.com/astronomy/different-types- twilight.html) is probably a good approximation.

</comment></comment></comment><comment><USER10> Tangentially related plug for a small site I built to tell you when golden hour is at your location [https://whenisgoldenhour.com/](https://whenisgoldenhour.com/) [1]

Would love to figure out how to incorporate a world dot map like this, but showing the two current golden hours (morning and evening) instead.

[1] [https://github.com/ksho/whenisgoldenhour.com/](https://github.com/ksho/whenisgoldenhour.com/)

<comment><USER11> Out of interest - how are you getting location? I didn't get a consent warning for client side geolocation and it seems accurate so it doesn't appear to be IP based (my public IP appears to be a few hundred miles south of my actual location).

<comment><USER10> I'm using the google geolocation API [https://developers.google.com/maps/documentation/geolocation...](https://developers.google.com/maps/documentation/geolocation/intro) at [https://github.com/ksho/whenisgoldenhour.com/blob/master/pag...](https://github.com/ksho/whenisgoldenhour.com/blob/master/pages/index.tsx#L11)

I've gone back and forth on replacing this with a browser-based location request so the user is aware what's going on -- the tradeoff being the friction of clicking the allow button.

</comment></comment></comment><comment><USER12> Somewhat related- not quite live, but NASA's DSCOVR is a satellite that is always on the sunlit side of the earth:

[https://epic.gsfc.nasa.gov/](https://epic.gsfc.nasa.gov/)

</comment><comment><USER13> Here is another take at the same: [http://www.fourmilab.ch/cgi- bin/Earth](http://www.fourmilab.ch/cgi-bin/Earth)

</comment><comment><USER14> I was expecting a map showing the clouds!

</comment><comment><USER15> I like the one that has the sunlight map plus the live cloud cover feeds - Interestingly enough, this is the site that holds all the linux manpages.

[https://www.die.net/earth/](https://www.die.net/earth/)

You can actually just link the image to your desktop and depending on settings will auto update throughout the day. Sometimes you need to write a script

</comment><comment><USER16> Xsunclock.

[http://www.fourmilab.ch/xsunclock/](http://www.fourmilab.ch/xsunclock/)

</comment><comment><USER17> Nice!

Huh, I wanted to draw something exactly like this to put as hero image of my company page, except with ISS orbit instead of the terminator. I really like this "world map out of dots" style.

<comment><AUTHOR> Thanks! I was inspired by the world map that appears on my android's Clock application, and it was an opportunity to write some structured matplotlib code and learn about new stuff.

You might want to take a look at the Natural Earth [1] datasets and GeoPandas [2], they're great if you're dealing with any kind of geographical data.

[1] [https://www.naturalearthdata.com/](https://www.naturalearthdata.com/)

[2] [http://geopandas.org/](http://geopandas.org/)

</comment></comment><comment><USER18> Can't you use a static image on the server and compute the terminator on the client side? That way it would always be up-to-date even without a reload.

</comment><comment><USER19> This reminded me of a moving city on Mercury called Terminator from Kim Stanley Robinson's novel 2312. The city slides along metal tracks propelled by the expansion and contraction of the metal as the terminator of the sunlight shifts. There's even a sun cult that chases perpetual dawn. It's a pretty good science novel.

</comment><comment><USER20> Nice work.

Want another challenge? Try one with the azimuthal equidistant projection. Would be quite interesting to watch the comparison.

</comment><comment><USER21> In the old days I used use xearth ([https://xearth.org/](https://xearth.org/)) to set my X background to a view of our planet with the sunlight correctly applied.

</comment><comment><USER22> There used to be a page that showed the view from a webcam located wherever the sun was currently rising as you viewed the page.

Can't find it any more though - probably been removed.

</comment><comment><USER23> You can get a live map of where the sun is shining from Google maps. Just switch to satellite view and zoom out until you see the whole earth, then a bit more.

</comment><comment><USER24> I'd love to use this as a clock pane in my window manager. If you added the local time and removed some of the large text, it would be a great background.

</comment><comment><USER25> Very nice. It would also be cool to be able to input what datetime you want it to display sunlight for.

<comment><AUTHOR> Modifying the Python code for this would be trivial. I need to think about how to best serve each 'session' though, or transfer the image generation to the client-side. Nice plan for a weekend!

</comment></comment><comment><USER26> Poor Antarctica right now.

</comment><comment><USER27> I don't know off hand what this cliche wave is called(shown in like all SciFi space launch movies) the wider top part of the wave that's the Earth tilted in that direction? eg. northern part is exposed towards the sun more. At this time for me in US.

<comment><AUTHOR> As cosmic_quanta mentioned it's called the terminator, and it's generally the dividing line between the 'illuminated' and 'unilluminated' part of a celestial body.

You're right, the weird shape is due to the projection of "earth" in the X-Y plane, if you chose a different projection it would appear differently. The calculations make sure that the result is ±1 accurate all year long.

The truth is that even the definition of 'daylight' and 'night-time' is not straightforward. The most common definitions include the Civil, Nautical and Astronomical Twilights [1], depending on the sun's position, so it should actually be a gradient.

[1] [https://en.wikipedia.org/wiki/Twilight](https://en.wikipedia.org/wiki/Twilight)

<comment><USER27> thanks for the info, I just see that squiggly line all over that 'sine wave' looking thing on space-related scenes on a big screen.

</comment></comment></comment><comment><USER28> This would be pretty slick as a new tab chrome extension.

</comment><comment><USER29> related, and also useful: [https://xkcd.com/now/](https://xkcd.com/now/)

</comment><comment><USER30> simple and sweet - quite interesting and useful !</comment>


Ask HN: When do ISPs begin selling your data, and is a VPN all that's needed? - <AUTHOR> When does the "ISPs selling your data" begin, and is a VPN without logging all that's needed? <comment><USER2> IMHO the ideal situation to selectively play along. A router running Linux/BSD which can be configured have regular, VPN and Tor route is the ideal situation to avoid scrutiny from third parties.

Connections to Facebook, Google, Amazon, etc. should go un-tunneled. It's good to feed the beast with data it already owns anyway.

Connection to porn websites (say by your 16-year-old cousin who came to stay at your place for the weekend) and other _ethically debatable_ content should be routed via Tor. Connections to torrents should be routed via VPN[1].

I understand that some people here prefer their personal VPN against a VPN provider like TorGuard, etc. There's no _good_ and _poor_ solution here, everything depends on the use case. A VPN provider will be handling thousands of encrypted connections and gives you a dozen exit nodes. From each exit node thousands of different connections are routed. It's way harder to target and isolate a user, even for a medium state-level actor.

Conversely, if you route all your connections from, say a DO droplet, you're controlling the droplet, but you have one exit point for all your connections... It's extremely easy to target your connections for a state level actor.

Of course there are thousands of schemes one can choose. Everything depends on the use case.

</comment><comment><USER3> There are no clear answers; all the info about this is a bit fuzzy because there's no requirement for ISPs to disclose past or current activities surrounding their selling of data.

The bill was actually enacted to prevent privacy rules which hadn't even gone into effect yet, which means that technically, ISPs would have already been able to sell such data. However, the consensus seems to be that ISPs were only selling "anonymized" data, and this move will embolden them to push further into invasive practices.

<comment><USER4> Yea great point. A lot of headlines are portraying this as a sudden new practice rather than an attempt to formalize the rules governing existing practices.

</comment></comment><comment><USER5> Wherever you VPN to, your connection still comes out somewhere. If that happens to be in the US, datacenters still have ISPs. If you're visiting a site hosted in the US, that's on some ISP too. Basically, if Level3, CenturyLink, and Verizon decide that they want to collect and sell profiles based on browsing profiles, there's not a way around it. It'd be easier to build a profile on a direct subscriber (e.g. Comcast profile on a subscriber of theirs) but installing OpenVPN on a DO droplet won't magically save you from this if you're in the US.

Will they actually do it? Well, can they make money from it? Do you trust Comcast to be a good steward of your privacy in the absence of a legal requirement to do so? Comcast did a hard pull on my credit when switching my account to a new address because they were too incompetent to update it and ended up creating a second account as a new customer for me. Comcast is my only option of ISP, as it is for many many other apartment dwellers and many single-family homes as well.

<comment><USER4> There is not much useful data to gleaned from an encrypted VPN connection. If you're referring to the ISPs monitoring what the VPN proxies for you that data would be useless as well since it would be the aggregate over everyone using the VPN. It wouldn't be possible to build browsing profiles if every request originates from the VPNs IP address.

<comment><USER6> you only need to leak a few bits here and there once or twice and the connections can be linked together

</comment></comment></comment><comment><USER7> The recent law enacted by Congress countermanded an Obama-era regulation that had never gone into effect. Thus, the current situation is the same as it ever was - ISP's can sell anonymized and aggregated customer behavior data today, just like they could yesterday.

Also important to note -- existing regulations still in effect prevent selling un-anonymized data. Selling someone the browsing habits of a particular identifiable customer is not allowed, never has been.

<comment><USER8> I'm not sure where this 'nothing has changed, no need to panic' talking point has come from.

The 'Obama era regulation' was an attempt to maintain privacy regulations after internet businesses were found to be exempt from ftc oversight in late 2016. So now there is no privacy regulation from the ftc or fcc. Things are not the same as they always were.

</comment></comment><comment><USER9> VPNs aren't really private. They can see everything that an ISP could see if you weren't using one. It's just a level of indirection.

I trust ISP companies more than I trust VPN companies because ISP companies are in the USA and are much larger (so engage in less risky behavior), so they at least have to sell data in aggregate and scrub PII

<comment><USER10> > I trust ISP companies more than I trust VPN companies because ISP companies > are in the USA and are much larger

That fact only supports their practice of selling user data. If we did not have such a barrier of entry for new ISPs, the market would be able to react to this abuse. But instead, large ISPs like AT&T, Verison, Comcast, Time Warner, etc. hold the market so tightly that they can get away with abusing their customers without a serious reaction.

Because ISPs have so much control over the market, the only viable response without regulation are VPNs. A VPN can sell privacy to a customer who feels abused by their ISP. Since privacy is essentially the foundation of the VPN provider's business, VPN providers compete to prove to customers they can protect their users' privacy.

</comment></comment><comment><USER11> DNS and Certificates are the key:

You should remove all CA certificates installed by software that show like it were "installed by you".

Some AV software does MITM sending you a "trusted" certificate signed by their own CA whilst acting as a proxy between the actual site and the AV.

Theoretically anybody could do the same on the network side transparently.

Also if you don't trust your ISP, you shouldn't use their DNS servers. I don't know about commercial integration between DHCP and DNS requests to track people but it is feasible with some work.

For DNS just grab a raspberry pi and setup a dns resolver. You only need the right root zone seed file. Just don't make it available to the whole internet.

</comment><comment><USER12> There are a large number of VPNs, commercial or otherwise, that someone can use. Where I live, I have a very small number of ISPs to choose from and they would all have my billing information if I used them. Some of the VPNs offer connections in other countries, and some of them claim not to log. The VPN's ISP would see my data move around, but wouldn't have my billing information. The VPN would, but if it's commercial I can likely sue them if they do something egregious with it.

In my opinion, VPNs help more than hurt privacy, assuming you choose a reputable one to use.

If a person wants anonymity, then go for Tor or Freenet.

</comment><comment><USER13> VPNs only give you indirection. And you have to trust your VPN provider. You're just moving your trust. Now if you diversify well enough you are probably less likely to be effectively snooped on, but then you're just reinventing TOR[0].

Adopting HTTPS across the board is so much more important.

[0] Which too only gives you indirection, remember that!

</comment><comment><USER14> Will the VPN in Opera prevent this tracking? How long before Chrome and Firefox offer this?

</comment><comment><USER15> You could always setup your own VPN on a VPS with

[https://github.com/jlund/streisand](https://github.com/jlund/streisand)

<comment><USER16> FWIW I tried this on DO and had a bitch of a time.

I'm a developer and I run linux as my primary OS, so there weren't any new concepts, exactly. I had to do a lot of google searching of error messages.

In the end, it kept failing after droplet creation, I think when whitelisting my IP or something similar.

This happened 2 days ago.

Anyway, I couldn't even get it running on the droplet, let alone get all connected and ready to go. Might try again with AWS.

<comment><USER17> I did exactly this, using DO. It works beautifully, and my droplet is in a foreign country. Two things I'd recommend if you have trouble are:

a) Look up Digital Ocean's own instructions for setting up OpenVPN. Follow exactly.

or

b) look at HN threads from today, some people wrote Ansible playbooks and scripts to set it up.

<comment><USER18> There's a good chance I'm wrong here, but isn't directing your traffic overseas the exact thing that makes it fair game to state-level inspection?

<comment><USER17> Fair game legally or technically?

Legally maybe, probably, who knows. Personally I don't feel protected legally anymore with regards to privacy. It's a good thing I don't really do anything illegal, even if it both saddens and angers me that the world is letting this happen.

Technically I'm assuming without some serious cooperation with Digital Ocean it's going to be hard. Sure, if someone gives the authorities (or the hackers get) root, then my OpenVPN software will be more Open than "VPN."

</comment></comment></comment></comment></comment><comment><USER19> Answering the question of choosing a VPN is complex and varies by what's important to you. Like with security, there are few black and white answers.

The VPN comparison chart [1] is the best reference I've seen on the dozens of factors one might care about.

[1]: [https://thatoneprivacysite.net/vpn-comparison- chart/](https://thatoneprivacysite.net/vpn-comparison-chart/)

<comment><USER19> It's unclear to me why this got downvoted as it addresses OP's question with one of the most popular and detailed references for choosing a VPN which also explains the specific privacy implications to consider (which of course cannot be answered universally as they vary by person).

</comment></comment><comment><USER20> If you use a VPN, you are just trading one ISP for another.

<comment><USER21> Yeah, but I trust Digital Ocean or AWS (and whatever providers they rely on) more than I trust AT&T, Time Warner, and Comcast combined.

</comment></comment><comment><USER22> Context?</comment>


Docker Machine Guide (VirtualBox on Mac OS X) - <AUTHOR> http://waterlink.github.io/blog/2015/08/31/docker-machine-guide-virtualbox-mac-os-x/

<comment><USER2> For convenience I ended up with two aliases: "dm" for "docker-machine", and "dmenv" for "eval $(docker-machine env dev)".

But before all that, I simply downloaded and used the docker toolbox installer for OS X. I opted to skip the Kitemaric install because why? Otherwise it did the right thing with existing virtualbox install, found and imported my existing boot2docker vmdk, etc.

<comment><AUTHOR> Docker toolbox installer is convenient, indeed.

Installation instructions I have provided can be automated through a script though. Which can be distributed on the team as a part of `.dotfiles` repo.</comment></comment>


Startup Goals for 2009 - <AUTHOR> http://paulstamatiou.com/2009/01/23/startup-goals-for-2009

<comment><USER2> I am a big fan of Paul's blog and I've been reading it for quite some time. Congrats on making so much with your blog! It's definitely inspiring for a recent college grad to be making so much from a blog. Hope to see Skribit take off!

</comment><comment><USER3> Congratulations on Paul generating real cash flow! I'd love to see a essay written by Paul on how he started his blog and what he think made it popular. So many blogs, even good ones, fail to break critical mass. I do blog and day dream of being a professional blogger but realize it really is hard work.

<comment><USER4> I'm glad that Paul is making some cash from his blog, but $1600 a month is peanuts for a 3 year old site. A blog 1/10 as popular in a lucrative niche with some decent search traffic would make 10x as much.

<comment><USER5> Yeah I agree. I never tried to whore my site out with ads either though, so I can't really say I tried to make much money off of it. Now that I might be more reliant on it, that might change.

</comment></comment></comment><comment><USER6> Wow congrats on making enough to pay the bills from the blog alone. Although it could be rough when student loan repayment becomes required... would that be too much?

Anyway good luck, hopefully you can "build the hand glider before hitting the ground" as the saying goes

</comment><comment><USER5> twampss, who are you? You've (successfully, thanks!) submitted some of my posts in the past here and am wondering who you are.. you have a blog I can read too?

<comment><AUTHOR> Just another big fan of your blog! Given the content you write about and the quality of the HN community, I'm usually surprised if I'm the first one to submit a post from your site. It's a perfect fit!

No blog on my end currently - took it down a while ago and haven't found the time to get it started up again. Hopefully soon! My email is [this username] at gmail dot com. I am always up for a good chat! Keep up the great work!

</comment></comment><comment><USER7> I've only just recently released stable version of my service (like a day and half ago), but my startup goal at least for th early part of 2009 is gaining some traction. Particularly the valuable kind that Paul Buchheit talks of with finding that 100 people that really like your service. Though my service is aimed at bloggers so that might be a little hard I think.

</comment><comment><USER8> These goals sound very familiar to me - survival being the key one, although we hope to do a bit of growing too...</comment>


Dataset Search - <AUTHOR> https://datasetsearch.research.google.com/ <comment><AUTHOR> Here's the blog post announcing Google Dataset Search is officially out of beta:

[https://www.blog.google/products/search/discovering- millions...](https://www.blog.google/products/search/discovering-millions- datasets-web/)</comment>


Advice for entrepreneurs negotiating venture capital term sheets - <AUTHOR> http://asack.typepad.com/a_sack_of_seattle/2007/03/advice_for_entr.html <comment><USER2> Perfect timing- I really needed this article.

nah. just kidding.

<comment><USER2> btw immad- i was making fun of myself and the current state of my business, not your submission, which was definitely relevant. cheers :)

<comment><AUTHOR> :-)</comment></comment></comment>