The war inside Palantir: Data-mining firm ties to ICE under attack by employees - <AUTHOR> https://www.washingtonpost.com/business/2019/08/22/war-inside-palantir-data-mining-firms-ties-ice-under-attack-by-employees/ <comment><USER2> It's kinda interesting how people are just now realizing that all the big tech companies are government contractors and not this imaginary utopia of 'making the world a better place' they were sold on.

Google, Amazon, Microsoft are all US gov't contractors. It's hilariously ironic seeing the political views of the employees suddenly clashing with their employers views of making money as if it's a new thing.

<comment><USER3> This isn't a new debate though...

My grandfather started a company which was a smaller player in the military contracting space.

It was somewhat large and his argument was that he was always 'helping democracy' but he was willfully blind IMO.

It's always going to be a bit fuzzy but it's another thing entirely when the organization you're selling to is just flat out evil.

</comment></comment><comment><USER4> Maybe I misunderstand Palantir's mission but working with LEOs like ICE seems to be their core business. All credit to these employees for standing up for what they believe in but what exactly did they think they were signing up for?

I have a new coworker who wears his Palantir t-shirt and it makes me uncomfortable to even see the name in my office.

<comment><USER5> I am assuming ICE is "U.S. Immigration and Customs Enforcement". I am not from USA but why every comment here is implying ICE to be some evil thing.

<comment><USER6> Because they do stuff like this

[https://www.nbcnews.com/news/latino/border-patrol-agent- deta...](https://www.nbcnews.com/news/latino/border-patrol-agent-detains- women-speaking-spanish-montana-gas-station-n876096)

<comment><USER7> Note that the article you linked is about CBP, which is not part of ICE.

Also note that ICE doesn't just enforce immigration laws, they also do things like this: [https://www.ice.gov/news/releases/missouri-felon-indicted- se...](https://www.ice.gov/news/releases/missouri-felon-indicted-second-time- illegally-possessing-silencer-conversion-switches)

Officers executed a search warrant at Lee's residence and found numerous firearms, ammunition, firearm suppressors, suspected methamphetamine, marijuana, suspected stolen farm equipment, and a large amount of cash. Lee told officers that he received 10 to15 past deliveries of the selector switches, and that he converted and successfully fired a Glock pistol as a fully automatic firearm.

<comment><USER6> > The superseding indictment resulted from an investigation by the following > agencies: U.S. Immigration and Customs Enforcement's (ICE) Homeland Security > Investigations (HSI); Bureau of Alcohol, Tobacco, Firearms and Explosives; > U.S. Postal Inspection Service; and Newton (Missouri) Sheriff's Department.

I'd rather take the chance that those _several other law enforcement agencies_ couldn't have worked it out on their own than have thousands of innocent families torn apart and kept in cages.

<comment><USER7> Is this an either/or choice? Does HSI have to not exist in order to prevent reprehensible treatment of migrant families?

Conversely, would eliminating HSI prevent such reprehensible treatment of migrant families? Wouldn't the abuses just pop up elsewhere, given the agenda of the current administration?

<comment><USER6> > Does HSI have to not exist in order to prevent reprehensible treatment of > migrant families?

It sure seems that way. And people often forget that we didn't have these agencies at all before 9/11, and we did just fine. Metal detectors and stronger cockpit doors are well and good, but what we have are agencies that were created in a time of extreme fear, and have held on to fear as their guiding ethos.

> Wouldn't the abuses just pop up elsewhere, given the agenda of the current > administration?

Maybe, I don't know. But the problem right now is they have an excuse. We're giving people license to commit whatever thuggish, nationalistic fantasies they want, all sanctioned under some vague "rule of law" argument where the letter has far, far outstripped the spirit. And what we end up with are goon squads harassing and imprisoning innocents based on technicalities. I don't think that sort of thing could persist at this scale without the pretense that these agencies provide.

I'll put it this way: I live in southern Texas. I'm who they're allegedly protecting. And if ICE were abolished overnight, I would throw a party.

<comment><USER7> > And people often forget that we didn't have these agencies at all before > 9/11, and we did just fine.

ICE wasn't created out of thin air, it was formed by merging and reorganizing existing organizations including Customs and INS.

> We're giving people license to commit whatever thuggish, nationalistic > fantasies they want, all sanctioned under some vague "rule of law" argument > where the letter has far, far outstripped the spirit.

I agree this is happening and we need to do something to stop it. At the same time, I don't believe this is the only thing that's happening in ICE, as it's a large and complex organization.

> if ICE were abolished overnight, I would throw a party.

If that happened but the fundamental issue of nationalism and lack of concern for human rights was not addressed, I think we could end up in an even worse place than we are currently.

I'm not against abolishing ICE, but I don't think it's a solution in itself. The fundamental issue, in my opinion, is a lack of empathy and understanding for people who seem different, and it's happening on a large scale. I believe that is a much harder issue to tackle compared to restructuring our federal bureaucracy, but that's what we should be focusing on to make a real, lasting impact. Restructuring without addressing the underlying issue seems like a placebo that would do more harm than good.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> It's not just ICE but the entire government that is a problem. I personally stopped accepting any work from any government a few years ago. Once you realize what government is, there's no going back.

"The State […] is an anti-social institution, administered in the only way an anti-social institution can be administered, and by the kind of person who, in the nature of things, is best adapted to such service [a psychopath]. Taking the State wherever found, striking into its history at any point, one sees no way to differentiate the activities of its founders, administrators and beneficiaries from those of a professional-criminal class. As Dr. Sigmund Freud has observed, it can not even be said that the State has ever shown any disposition to suppress crime, but only to safeguard its own monopoly of crime." \- Albert J. Nock in "Our Enemy, The State"

<comment><USER9> It makes me sad that so many people are unwilling to engage with this position. Anarchism is unreasonably vilified imo.

<comment><USER10> A big problem with anarchy is that it's really complicated and is usually either misunderstood by the proponent or the listeners. Anarchy is about levelling the playing field and reducing the power that the most powerful structures have in our society. The problem of removing power structures in institutions, is that they will naturally get replaced by other ones, and the replacements are often as authoritarian as the originals.

In the US, it's popular to think that you can somehow replace elected government (where power is somewhat dependant on votes, and somewhat dependent on advertising budgets/capital) with business (where power is dependent entirely on capital), and that somehow this is more "anarchist". This is a naive idea.

</comment></comment></comment><comment><USER11> Palantir's entire business is providing data mining and surveillance software to government agencies. ICE is probably the least-morally-reprehensible thing they support. They are deeply involved with DHS, CIA, NSA, all manner of classified agencies. If any of their employees have a problem with ICE, they never belonged there in the first place.

<comment><USER7> > Palantir's entire business is providing data mining and surveillance > software to government agencies.

Palantir has a number of commercial customers, here's a few examples: [https://www.palantir.com/palantir- foundry/impact/](https://www.palantir.com/palantir-foundry/impact/)

<comment><USER12> >Its two main products are Palantir Gotham and Palantir Metropolis, more geeky winks from a company whose Tolkien namesake is a type of magical sphere used by the evil lord Sauron to surveil, trick, and threaten his enemies across Middle Earth. While Palantir Metropolis is pegged to quantitative analysis for Wall Street banks and hedge funds, Gotham (formerly Palantir Government) is designed for the needs of intelligence, law enforcement, and homeland security customers.

[https://theintercept.com/2017/02/22/how-peter-thiels- palanti...](https://theintercept.com/2017/02/22/how-peter-thiels-palantir- helped-the-nsa-spy-on-the-whole-world/)

</comment></comment></comment><comment><USER13> I'm maybe wrong, but I would like to share some ideas.

Where does this money is coming from ? I mean, does the whole question isn't a level "above" : why does such entity exists in a democratic gouvernement ? Isn't it showing a deep loophole in the system ? Same idea : how is this entity funded ? Isn't it with US-citizens's taxes ? So, aren't all US-citizens involved in this story as well ? Why would Palantir's employees be more involved than all the persons that allowed the money to be there for Palantir to be funded ?

I'm seeing more Palantir as a consequence of a system's failure than its premice.

Woild love to discuss about that however.

</comment><comment><USER14> Uhh, but unlike other tech companies palantir's core business model is violating the privacy of individuals to support the governments of many countries. Protesting your employer doing what it has always said it did is a bit rich.

</comment><comment><USER15> That seems... odd? They're fine with the domestic spying, pre-crime profiling based on flimsy evidence, but help with rounding up people for the concentration camps and people suddenly get cold feet.

</comment><comment><USER16> Palantir: hey dude want to work for a company whose job it is to help punish people, manipulate, and violate their privacy through big data?

Engineer: sure! Just as long as I'm not enforcing a national border that's where I draw the line!

<comment><USER17> If you replace "punish" with "control" I think that's an easily-extensible template, because isn't that ultimately tied to the personal ethics of the employee?

<Some Company>: hey dude want to work for a company whose job it is to help manipulate and control people by violating their privacy through big data?

Engineer: sure! Just as long as it doesn't violate this very specific aspect of my personal ethics!

That could easily describe Google, Facebook, Amazon, etc.

</comment></comment><comment><USER6> The company is literally named after the magic orb that Sauron uses to surveil Middle-Earth. I don't know what you expected.

<comment><USER18> It's not the Palantir that are bad, but Sauron and Saruman. They were created by elves for useful purposes.

Possibly the Palantir saved middle earth when Aragorn used it to scry events leading up to the final battles.

<comment><USER19> I guess this explains the "Defend the Shire" shirts I see Palantir employees wear sometimes. Always thought the idea they're on the side of the pastoral, mind-their-own-business Hobbits was one of the most absurd misconceptions of their purpose of any company in Silicon Valley, and that's _really_ saying something.

<comment><USER20> Oh. Wow. That is insane. Photo for reference:

[https://fm.cnbc.com/applications/cnbc.com/resources/img/edit...](https://fm.cnbc.com/applications/cnbc.com/resources/img/editorial/2016/01/11/103295500-Untitled-2.530x298.jpg?v=1452536513)

<comment><USER21> I have one of these shirts (different color) and get favorable comments from LOTR fans periodically who see me wearing it in public. The front just says "Save the Shire", and it only says "Palantir" in smaller lettering on the back.

It's a pretty good shirt, just going by people's reactions to it. I'm just talking about the shirt in isolation though, not in the larger context of what Palantir the company does for their business.

</comment></comment></comment></comment></comment><comment><USER22> I want to join Palantir just to disrupt their business because Peter Thiel is a scum sucker and deserves the absolute worst workforce. Palantir should be liquidated and the proceeds returned to tax payers.

<comment><USER23> Ok, but please don't post unsubstantive comments to HN, and especially not ragey unsubstantive comments. They just make this place worse, and we're hoping for better here.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

</comment></comment><comment><USER24> This is a big company profile for palantir with the "attackers" getting less than a paragraph. 60 people out of 2500 is not much and the article didn't mention who are they - so probably low level and easily replacable if they decide to quit or be fired.

Those types of articles under the umbrella of "some people on twitter are outraged about something" that give impression of society wide consensus when they are a tiny minority or widespread action are distorting the discourse quite a bit.

Also - they wanted Palantir to donate profits from ICE contracts, but they didn't donate their salaries.

I wouldn't go after Thiel. The guy has a habbit of winning.</comment>


Ask HN: The right way to learn how to program - <AUTHOR>

I've been programming on and off for the past 5 years or so. Online tutorials, physical books and looking at existing code have been my main ways of taking in information. While I do feel that I've learned a lot, I can't help but think that there's a better way to go about understanding programming languages and concepts. I feel like I'm stuck in some sort of limbo between being a complete beginner and having a solid grasp on whatever I'm trying to learn.
I'm familiar with the advice that goes something like this: "Try to solve a problem that's compelling to you and you'll learn fast", but I guess I just don't feel like I can even get to the point of being able to even attack whatever problem I think up.
Basically, I guess what I'm asking is: What's the most effective method for learning a programming language? <comment><USER2> Stock answer: It depends.

There are loads of different people, loads of different learning styles, loads of different methods, and we know nothing about you. You question presupposes there's a single good answer.

There isn't.

Some people pick up programming easily by reading some code, and then modifying it to do something slightly different. Lather, Rinse, Repeat.

Other people read through a textbook and do all the exercises.

Still others have a specific task in mind, then Google, read, pester various on-line groups, and gradually acquire knowledge.

You might actually be a guru and not know it. You might be a complete putz and not know it.

Perhaps the best way to assess where you really are and get a grip on the gaps is to find someone to partner with - even just remotely - and write some code together.

Here are a few random question:

\+ How fast can you write FizzBuzz?

\+ Did you have to look up what the question meant?

\+ Did you actually write it?

\+ What language did you use?

\+ How would you make it faster?

\+ How would you reduce the number of of division/modulus operations?

\+ How would you count the number of words in a file?

\+ Find another way.

\+ And another.

If you're brave enough to answer those question absolutely honestly here, people will give you their opinion.

<comment><AUTHOR> Thanks for your advice. I had to look up FizzBuzz, so that's one good direction to go in. I'm not looking for the be-all, end-all solution to learning programming, I want to read everyone's methods and possibly come upon the method that will deliver me to programming heaven.

</comment></comment><comment><USER3> Just on that advice: 1. try to find less ambitious tasks that still have some excitement for you - that is, the right level of challenge for you; and 2. try to learn how to break down tasks.

1\. I always find it exciting to see something happen because of my code. It's very basic, like a child playing with blocks. Graphics code is especially good for this; and there's also javascript that creates a webpage; or simply printing something on the screen (terminal/console). Combine a few steps together, and you can have something cool, even though it's simple.

2\. breaking tasks down is how you "get to the point of being able to attack something". This "divide-and-conquer" is crucial - absolutely crucial - throughout all computer science (and every other complex activity). This is not a skill you should expect to automatically have perfectly mastered, but it's one you already have the beginnings of and that you can practice to improve it. For a start, you could look at any task (of any kind), and ask yourself how it could be broken down into simpler steps - often, it will be obvious to you. Now, break down one of those subtasks again, and keep going until you have a task that you are 100% sure you can do. Then focus on that one task to the exclusion of the rest of it. This is difficult. Try to pretend that the original task and all the other subtasks and sub-subtasks etc don't exist; and when you have solved this one task, observe that you've made progress and congratulate yourself. Only after this little celebration of real progress should you go on to the next step.

        If you want to hit a bird on the wing, you must have all your will in
        focus, you must not be thinking about yourself, and equally, you
        must not be thinking about your neighbor: you must be living in your
        eye on that bird. Every achievement is a bird on the wing.

This might not suit your temperament, but it's _the_ way to accomplish things that are too complex/overwhelming for you. Once you're encountered one kind of problem a few times (or maybe even once), it can become intuitively obvious to you, and so you won't have to break it down so far next time. You have grown!

But be warned: there _always_ be things that are too complex for you, and that you'll have to break them down. The skill of breaking things down is essential.

</comment><comment><USER4> Can you clarify the question a bit more, please.

For me, at least, learning a language and solving a problem with it are two different things. For example, I can solve problems in quite a few languages that I would never claim to know.

Many languages share similar syntax and basic properties, so it's not particularly hard to "get things done" in them, provided you know one.

By breaking down a large problem into its components -- which will happen automatically if you follow a tdd-like process -- you can tackle each smaller problem in isolation, which might simply be googling for it and copy & pasting the code. (I'm not advocating this as the "right way", just a method to get things going.)

But, as I said, is it the language or the problem solving that you feel is the block?

</comment><comment><USER5> I'm not a programmer, but.....

I would gather if you pick something you find interesting, you could work backwards a bit and get ideas.

So for example, I like sound "stuff"

So what I would do first is just see if I can code something that plays an audio file when you click a button - easy enough.

then go from there.

If you don't have a tangible ideal to work backwards from then you will always be lost.

An analogy a friend of mine said was:

If you never opened an apple you would never know there was a seed inside. So working backwards seems to be embedded in the DNA of what people do when we learn.

</comment><comment><USER6> Just start using it. With the approach you're taking, you will continue to just read tutorials and never code. If you think you can't code, you can't.

Pick a problem. Something easy. And just code it. Yes, it's going to be hard. Yes, you'll have to look up a ton of things. Do it. Stick to it.

Here's another hint: Tutorials are not for reading. They're for doing. Reading tutorials is just about useless, even for experienced programmers. If you don't actually follow the steps, you won't learn anything.</comment>


Ad-free Outlook and Hotmail ($19.95) - <AUTHOR> http://windows.microsoft.com/en-US/hotmail/ad-free-hotmail-and-outlook <comment><USER2> The claim made about outlook.com is

"Google goes through every Gmail that's sent or received, looking for keywords so they can target Gmail users with paid ads. And there's no way to opt out of this invasion of your privacy. Outlook.com is different—we don't go through your email to sell ads."

from <http://www.scroogled.com>

How is outlook.com selecting which ads to show? Is there any transparency in that process, or are they showing personalized ads but just going through the title (or some other technicality that isn't a 100% email scan)?

<comment><USER3> They've said they use demographic data based on what you provide at signup for targeting ads in email.

It's enough to be used fairly effectively (if not as topically), for instance by correlating voter registration data with name/location data for highly targeted political ads:

 _For instance, surfers may be shown a shoe ad if they recently visited a shoe site. Most of this sort of targeting doesn't require your name. Political targeting does. Campaigns may want to reach only reliable party members, or independents who might swing their way...._

 _Finding voters online is difficult, since no public record connects voters to a particular Internet address. That's where Microsoft, Yahoo, AOL and other lesser-known companies come in. Their enormous stores of registration data can serve as the bridge between particular Internet users and their voter information...._

 _Microsoft and Yahoo's targeting service combines two crucial factors: their knowledge of users' personal information and their ability to add cookies to browsers. Over the years, Internet users have given these companies their name when they signed up for free programs like the Microsoft suite of services known as Windows Live, which includes Hotmail. (Microsoft said it does not sell campaigns access to information users provide when they register for Office or other Microsoft products they've bought.)_

 _Microsoft and Yahoo both said the cookies aren't connected directly to names or other personally identifying information. Instead, they use a complicated process to match coded voter information back to anonymous cookies on particular users' browsers._

[http://arstechnica.com/tech-policy/2012/06/how-microsoft- and...](http://arstechnica.com/tech-policy/2012/06/how-microsoft-and-yahoo- are-selling-politicians-access-to-you/)

</comment></comment><comment><USER4> I never understood why people were so against targeted ads. By definition, targeted ads have a higher chance of being something I'm interested in. Isn't that a good thing?

And yes, they scan your email... news flash - every online email service has access to your email by definition. Either send it encrypted from the client, or don't send things you don't want other people reading.

<comment><USER5> Every cloud-based email service "scans your email" and does it a lot. This is mandatory in order to:

\- Junk filtering, fishing protection and other security protections; all impossible to implement without deep scanning, analysis and correlation (this last item is important: some information extracted from your email is combined to information from other people's emails, in big statistical models and fancy machine learning jobs). \- Index content so you can search it (unless you want to only be able to search by subject and other headers... no, 1988 called, wants its Eudora back). Bing.com may not include results from email, but this is irrelevant, Outlook.com certainly has full-text search so there's indexing. \- Implement all kinds of cool stuff, e.g. Google Now right now shows that my last Amazon order has shipped, a single click on that shows the USPS tracking, how cool is that. Or more modestly, integration with other services from the same provider or from others such as cloud storage, office apps etc., or even simpler features like Outlook.com's ability to automatically sort email into categories, I doubt very much they can do it only looking at headers. (Even if they do, in many cases there's enough private info in headers that just not looking at the email body shouldn't reduce your paranoia.)

In the big picture, the extra scanning for ad targeting is really a very minor part of the whole. So it all boils down to discussing whether you prefer targeted ads or untargeted ads, and whether you trust the cloud service to protect the private information used for this. Notice that this same potential issue exists for _all_ functionality above, not just for ad targeting; so if you don't trust GMail as a matter of principle, then you shouldn't trust Outlook.com either.

</comment></comment><comment><USER6> This is something I wish more companies would do with their free products.

<comment><USER7> Until then you can just hide your ads in Gmail: <http://lifehacker.com/5879757/how-to-hide-ads-in-gmail>

<comment><USER8> The problem has never been about seeing the ad was it? It's mostly about what ad and what Google sees to show the ad.

<comment><USER9> is it a given in any email system that they see this though, i.e. would it even be possible to make it so that they couldn't see it?

<comment><USER10> You could use gpg and a non-browser email client to encrypt your email. But then you can do that with gmail too.

</comment></comment></comment></comment></comment><comment><USER11> Is there something sneaky in the wording of 'no graphical ads' ? Makes you wonder if they will display text only ads.

<comment><USER12> I had the same thought, Microsoft seems to keep saying deliberately, "No graphical adds".... why not no adds period?

</comment></comment><comment><USER13> For those with custom domain names... $20/year vs $60/year for Google Apps. Ads or not, thats a compelling savings for my many pet projects where I only need 1 or 2 @projectname.com mailboxes.

It's a shame that Google Apps got rid of the free tier. I've started 2 businesses on the free tier (< 10 accounts) and now pay around $200/month for each company as they have grown.

Now Microsoft will gain some ground in the email arena.

<comment><USER14> Yikes. If you're paying around $200/month (Not sure if this it total, or for each service, making it $400/month), would it not be feasible to invest in a VPS at linode and run your own mail server? You could even hire a decent system admin to look after it, and still save a few hundred dollars a month.

<comment><USER15> I've run a mail server for 5 years for a NGO. Running your own mail server is not as easy as running your own LAMP stack. The is SPAM, IMAP, bounces, ... and many more issues with running your own system. My Postfix is only redirecting to gmail accounts. If that works - fine, but that's probably not what a small non-tech-savy business needs.

</comment></comment></comment><comment><USER16> It always surprise me how willing people are to let other people read, use and handle ones private emails.

It commonly received as a surprise when I talk about it, but once you give away your email to be handled by someone else, the emails are no longer private. If you give it to google, google then owns it. If you give it to Microsoft, its Microsoft that's own it. Why are people acting so strange to the idea that if you give something away, its not longer in your control?

Running a email server is not hard. if its a company, running a email server + webbmail is as hard as installing two extra packages in debian and doing a single dns change. If you don't have a server, buy a old laptop/media station. Its email, its not going to demand processing power beyond simple spam filtering.

On that note, wonder if my phone can get postfix installed...

<comment><USER17> 99.9% uptime is nice ( where I'm at right now has had server outages more often than gmail has had).

Having a decent spam filter is nice (I've had loads of false positives in things I've tried elsewhere).

There is the idea that google/MS could just fall off the face of the earth, and then I look pretty stupid having loads of things connected to that account. That bothers me enough to where important things get thrown onto an email adress I'm pretty sure won't disappear for a while.

As to reading my mail... robots are reading my mail. This is a pretty valid "Chinese room" in my opinion. Nobody's actually understanding my mail, what do I care. The ads are usually pretty relevant to things I need though (I've clicked on more of the text ads than I would ilke to admit).

<comment><USER16> > 99.9% uptime is nice

Pick a random (not broken) piece of hardware. Install debian stable on it with base install + postfix + roundcube. Configure: grey listing, black listing, and auto-update (relative easy, and there is a bunch of tutorials if needed). Leave it alone until once every 2-3 year where it will need a aptitude update, aptitude upgrade. It will run as long it has power and network connectivity. If you are a company and either the power or network is down, email is unlikely to be high on the priority list.

Spam: ~0. Might get one every full moon or so. After 5-10 years it might get a bit worse, but adding SpamAssassin tend to get rid of it.

As for Chinese rooms, do they leak information? Do they give out mail if asked by someone with money/power? Sure, one can say "I got nothing to hide so what do I care", but then that's what all email then must be. You can never again say: "This email is private and mine. I and only I have control over it". If you are a doctor, lawyer or any person dealing with private information, then it mean that any email given to you are defacto public.

<comment><USER18> Roundcube is not gmail. The gmail interface is the killer feature that no one has been able to replicate, and it's the thing that pulled me back after 6 months of hosting my own email. I was in gmail withdrawal, and seriously considered not having an email account instead of dealing with roundcube, mutt and K9 anymore. Roundcube's interface is a great replica of Apple mail from 10 years ago, but it's seriously painful to use compared to the comfort and the features of the gmail interface.

</comment></comment></comment></comment><comment><USER19>
        No account expiration

Does that mean Outlook still has that _absurd_ system from Hotmail?!

<comment><USER20> You mean the one where if you stop using an account, it gets deleted?

<comment><USER21> Google doesn't do that. You can create a Gmail account for a project or event and they'll never delete it. Maybe you want a Gmail address for a yearly event, which you'll only use for 2 months out of the year - Gmail can do that for you, and for free.

</comment></comment></comment><comment><USER22> "No graphical ads"

Brace yourselves, text ads are coming.

</comment><comment><USER23> Does anyone know how outlook.com handles expiration?

In the terms it says you must log in every 270 days. Sounds like plenty of time but I have old gmail account which I use once or twice a year. It would probably have been deleted if it was with hotmail/outlook.

The worrying thing would be the email address going back into circulation with someone else. Wouldn't they start to get your mail? Couldn't they retrieve you password on websites with the associated email?

This scenario isn't too unlikely as the old gmail account I mention above is associated with login details for 50-100 websites. I simply moved to a new email address and only check it when I need to login into an old site and have lost the password.

</comment><comment><USER24> Hmm, but they still allow POP/IMAP for free? Most companies who want money for their premium mail service (e.g. mail.com/1&1) include that with removing ads, which seems a much bigger incentive.

I do kinda like the new outlook.com interface, but the way the conversation view works makes it basically unusable to me. Not that MS ever had big taste in that regard, as I still blame Outlook for basically establishing top quoting as standard.

</comment><comment><USER25> I'll stick with free e-mail.

<comment><USER2> What happens if google stops providing gmail? Suppose that, for example, google has some losing quarters and is pressured into making cuts and they decide to axe gmail. What happens then?

I am an unabashed fan of google voice, but there I know that it would be possible to get the number out (thanks to legislation), see [http://support.google.com/voice/bin/answer.py?hl=en&answ...](http://support.google.com/voice/bin/answer.py?hl=en&answer=1316844) . I don't have similar assurance with gmail.

<comment><USER3> what happens if Microsoft raises the outlook price per year to some level I'm unwilling to pay? At some point, it just comes down to a calculated risk.

<comment><USER26> That's why you pay for a domain. Then you can move your address wherever you want.

</comment></comment></comment></comment><comment><USER27> I find $19.95 a little too much, just for getting rid of ads.

<comment><USER28> How much do you think the ads are making them?

<comment><USER29> Not a lot when the second benefit they list is not needing to log in to keep your account active...

</comment></comment></comment><comment><USER30> adblock-plus is free

<comment><USER11> Unfortunately the servers and employees to run the service is not.

<comment><USER30> I understand that very well, I constantly thinker about possible ways to make revenue without ads. If you are really honest about it to yourself you will realize it's not a very solid business model. People can easily circumvent ads and/or just ignore them.

<comment><USER31> Ad block is only effective because not many people use it.

If it were even vaguely popular all major sites would have found a way to stop it working, they'd be a constant war going on.

<comment><USER32> I cannot imagine how could somebody prevent browser extensions injecting code to websites. As a happy user of ad-block I have seen some sites that force you to turn it off. Like iconmonstr.com But that can be easily circumvent.

<comment><USER33> For example, by making it difficult to tell what part of the website is an ad.

</comment></comment></comment></comment></comment></comment><comment><USER34> Since when does HN publish spam/ads?</comment>


Why OpenCV uses BGR color space (2015) - <AUTHOR> https://www.learnopencv.com/why-does-opencv-use-bgr-color-format/ <comment><USER3> > E.g. in Windows, when specifying color value using COLORREF they use the BGR > format 0x00bbggrr.

Wait a little bit... isn't Windows little-endian [1]? Following the link in that sentence:

> typedef DWORD COLORREF;

> The low-order byte contains a value for the relative intensity of red; the > second byte contains a value for green; and the third byte contains a value > for blue.

Given that Windows is and has always been little-endian, the byte order of a COLORREF is red first, then green, then blue, that is, the order is RGB. So the reason for BGR can't be Windows' COLORREF.

[1] "A decision was made VERY long ago that Windows would not be ported to a big-endian processor." [https://blogs.msdn.microsoft.com/larryosterman/2005/06/07/th...](https://blogs.msdn.microsoft.com/larryosterman/2005/06/07/the- endian-of-windows/)

<comment><USER4> That's precisely how little endian works: first memory location is LSB.

        union {
            uint32_t l;
            uint8_t b[4];
        } u = {
            .b = { 0x12, 0x34, 0x56, 0x78 }
        };
        assert(u.l == 0x78563412);

</comment></comment><comment><USER5> Well, not only it is quite long just to say "BGR was used a lot back then", even the back story is wrong : [https://www.snopes.com/fact-check/horses- pass/](https://www.snopes.com/fact-check/horses-pass/)

<comment><USER6> The article said that as well, so I'm unsure what your complaint is.

<comment><USER5> No, the post about OpenCV and BGR talks (for about 80% of it's length) about how everything we know in transportation, up to space shuttle, is linked to horse width.

Which, as pointed out in the link I provided, is simply false (or at the very least a gross oversimplification).

<comment><USER7> The author stated that the story is "not-so-true" and copied from Snopes, but he didn't include the debunking.

<comment><USER8> One of the key points of the "debunking" seems to itself make a logical error:

> The eventual standardization of railroad gauge in the U.S. was due far less > to a slavish devotion to a gauge inherited from England than to the simple > fact that the North won the Civil War and, in the process, rebuilt much of > the Southern railway system to match its own

But the article earlier states that the North won the Civil War _because_ of the logistical superiority of its rail lines that had settled on using the English standard gauge. Even if the South had settled on a single alternative gauge, they would have been unable to easily acquire rail equipment from abroad without added expense.

Snopes' attempt at debunking achieves the opposite in my reading.

</comment></comment></comment></comment></comment><comment><USER9> Before deep learning is a thing, most computer vision tasks are done either in greyscale, or in a more reasonable colorspace (HSL, YUV etc.). Thus, you most likely just call cvCvtColor the first thing when extracted the frame, and it doesn't matter BGR or RGB.

Only with deep learning, people start to use RGB directly, and this becomes something more interesting.

<comment><USER10> For DL the colorspace is irrelevant as long as it's consistent between training and inference. I.e. if you train your model in RGB, it better be receiving RGB and not something else from the video feed or input images. Interestingly, models can also be trained in YUV (including its downsampled variants like YUYV), which can make them a tiny bit more efficient, both because there's less data, and because most camera chips can output YUV raw data directly. The net will learn whatever transform it needs to learn to make sense of the data.

<comment><USER11> Interessting. Why is not the "raw" sensor format, for example Bayer BG8, used more in deeplearning? Would it not contain a little bit more correct information? and if used, we could skip the Bayer-> RGB conversion completly.

<comment><USER12> Public raw image datasets are rare, probably because most cameras, except DSLRs, don't support storing raw images.

Also, the nature of the inference may not require high fidelity raw images since they're going to be down-sampled anyway. Unless research suggests that raw images improve inference metrics, nobody's going to invest effort into building a raw image dataset. One notable recent exception to this is the "Learning to see in the dark" paper on low light image enhancement[1] - <USER2> found that training on raw images gives much better enhancement than JPEG.

The data availability problem may start getting solved in the near future because mobiles (atleast Android) support storing and processing raw images, and on-device learning and inference capabilities are getting better all the time on mobiles.

[1]: [https://www.youtube.com/watch?v=qWKUFK7MWvg&feature=youtu.be](https://www.youtube.com/watch?v=qWKUFK7MWvg&feature=youtu.be)

<comment><USER11> Interesting link! Yhea, I guess JPEG destroys a lot information low light images.

> probably because most cameras, except DSLRs, don't support storing raw > images

This may be true for consumer cameras but not for machine vision cameras. Most color machine vision cameras do support the Bayer format. Basler, FLIR (Point Grey), IDS and so on all support the Bayer format. If you roll your own camera you definitely have to deal with the Bayer format from the sensor you are using.

And if you do not want any compression on your data-set images the Bayer format also takes less space than the RGB format.

I'm not arguing it is a good idea, I'm just interested if it works :)

<comment><USER12> It's useful information about industrial machine vision cams, since I don't have any experience with them.

You are right about custom cameras - in fact, I have some camera modules for experimenting with surveillance, night vision and machine learning, and some of their datasheets mention registers that, when set, output Bayer format images. Machine learning in surveillance is still a rather under-serviced market, but it's potentially a rich source of raw image datasets.

Raw format training and inference definitely work. The obstacle is only data availability.

</comment></comment></comment></comment></comment></comment><comment><USER13> Somewhat off-topic, but I just hate that Snopes article. It's wrong, and rather off-base.

Functionally speaking, the main disconnect in the article is the difference between loading gauge and track gauge. 1435mm, or 8'4.5", may be the standard track gauge and uniform between most countries, but the actual standard loading gauges varies quite dramatically between different countries. The UK has a positively tiny loading gauge (about 2.5m), while the "standard" EU loading gauge is 3.1m, the US is slightly wider at 3.2m, and Sweden opts for a roomy 3.4m.

The original notion that the SRB has to be "slightly" larger than the 1435mm track gauge is ridiculous, since as you can see, the loading gauge is over twice the width the track itself. Of course, there's also rooms to play tricks here if you need to. Schnabel cars can laterally offset the loads they're carrying by up to a meter or so, which gives you some extra height to play with if you're dealing with a double-track tunnel with an arched roof.

The other point to mention that's completely omitted is there is a rather big difference between rail carriages and horse-drawn carriages. On rail carriages, the cars sit above the wheels, while horse-drawn carriages sit between the wheels. Something like this:

          bbbbbb    bbbbbbbbbbbbbbbnbb
          b    b          b   b
        w b    b w     w  b   b  w
        w bbbbbb w     w  b   b  w
        waaaaaaaaw     waaaaaaaaaw
        w        w     w         w
        w        w     w         w

(Note that this is one of the reasons that the standard gauge was chosen over a broader gauge).

As for why standard gauge was chosen, well, keep in mind that a) there was a lot of variance of track gauge in the 1800s, and b) the actual tolerance of track gauge was pretty damn massive. A track gauge differing by as much as two inches or so wouldn't have impeded the ability to transfer rolling stock. Stephenson built the first practical steam locomotive on a line that was 4'8", and so people who wanted to reuse his locomotives would have been drawn to a gauge around 4'8"-4'9". Gauges around 5" and slightly larger were selected in part to prevent interconnect between different railroads (basically, the same concept as vendor lockin).

Contrary to the Snopes article, the South mostly standardized on 5' gauge, but it didn't have as dense a network as the North did. The South eventually converted to 4'9" gauge (the Pennsylvania Railroad's gauge)--in 1886, 20 years after the Civil War. Conversion thence to 4'8.5" took much longer, but was much less dramatic because 4'9" and 4'8.5" are within gauge tolerance.

</comment><comment><USER14> So because of legacy _and_ unwillingness to change.

</comment><comment><USER15> The article makes it sound as if moving OpenCV to RGB is as difficult as moving the US to the ISO metric system. Neither is impossible. If Europe manages to abolish national currencies, the US surely can abolish imperial units (and drive faster ;).

But I guess that will only happen once under pressure of chinese overlords. I'm sure the day will come, so why wait?

</comment><comment><USER16> May be the "Roman horse" here is that BGR is RGB in little endian format. Also, it has lots of advantages: little endian pixel handling allows using similar code for 24 and 32bpp (rr.gg.bb.rr.gg.bb vs rr.gg.bb.xx.rr.gg.bb.xx), so you can loop over pixes with offset += pixel size.

</comment><comment><USER17> TL;DR because it used to be popular among camera engineers and other people who manipulated image data early on.

<comment><USER18> This begs the question: why was BGR popular among camera engineers while RGB seems to be common in software?

<comment><USER19> Big endian vs little endian hardware?

<comment><USER20> That's my understanding - a lot of the low power devices were big endian as for whatever reason big endian and risc seemed heavily correlated for decades, similarly the early mainframes and whatnot were big endian which is why a lot of network protocols use big endian formats.

<comment><USER21> Probably the popularity of the Motorola chips?

<comment><USER20> Seems plausible - I just couldn't recall who was dominating embedded chips at the time

</comment></comment></comment></comment></comment></comment><comment><USER22> Never mind the horses arse! Its 4 feet, 8.5 inches because Human's thumbs and foots! _chuckle_ (=1.4351 meters)</comment>


Microsoft working with Joyent and the Node community to bring Node.js to Windows - <AUTHOR> http://blogs.msdn.com/b/interoperability/archive/2011/06/23/microsoft-working-with-joyent-and-the-node-community-to-bring-node-js-to-windows.aspx

<comment><AUTHOR> More info on the official node.js blog [http://joyeur.com/2011/06/23/joyent- partners-with-ms-to-port...](http://joyeur.com/2011/06/23/joyent-partners- with-ms-to-port-node-js-to-windows/) And from Joyent [http://joyeur.com/2011/06/23/joyent-partners-with-ms-to- port...](http://joyeur.com/2011/06/23/joyent-partners-with-ms-to-port-node-js- to-windows/)

</comment><comment><USER2> There is an existing discussion regarding this topic here: <http://news.ycombinator.com/item?id=2688270>

</comment><comment><USER3> What has Microsoft been smoking lately? They're worryingly enthusiastic about JavaScript.</comment>


Ask HN: What did you learn in 2018? - <AUTHOR> What did you plan to learn in 2018 and what did you actually learn? <comment><USER2> \- Dove head-first into the fundamentals of distributed systems, applying all that I learned to solutions for scalability issues at work.

\- Realized that the secret to promotion is that it's all about making your superiors look good. This brought on even greater realizations about society in general and individuals in my life.

\- The importance of work/life balance and that the "life" portion must involve taking good care of yourself.

\- The value of being kinder to all those I see regularly. It's easy to drop some of the positive interactions with those we're comfortable with.

<comment><USER3> This sounds interesting. Please tell us more.

</comment></comment><comment><USER4> Improved my German considerably. Can definitely deal with any situation by now, but I still have much to improve.

Also learned a lot of Elixir.

Both are things which in the past year I've commited to learn. Learning is a constant process, next year I expect more of the same, but a higher mastery level!

Also learned a decent amount of management things, better organization skills etc, but that I didn't plan for.

<comment><USER5> Nice! I'm learning German as well. Can you tell me if you used any books/tutorials/apps/classes? Thanks in advance.

<comment><USER4> Duolingo, normal classes in the beginning. Nowadays mostly speaking to people and watching TV and every once in a while reviewing grammar.

Give it time.

</comment></comment></comment><comment><USER6> I planned to go from beginner Python to employed developer. Went a bit off track and learned the basics of lots and lots of languages. Kept switching the domains I wanted to work in.

I studied a fair bit of CS, but in a fairly shallow way (it's been a decade since my last maths lesson).

This year I proved to myself I could learn lots - next year is proving I can do the same, just in one or two things!

<comment> escape-effect It's harder to focus on learning a single thing perfectly with all the noise out there. Did you get employed as a python developer?

<comment><USER6> Nah, it turns out opportunities for Python + first job in the UK are somewhat rare, especially as I don't have a completed degree. If you don't live in London, C#/.NET or JS + a framework make up 90% of opportunities. Of course, just learning the right language isn't how it works, and it took me a while to take that on board.

> It's harder to focus on learning a single thing perfectly with all the noise > out there

Preach! I've definitely flipflopped languages and target positions based on a combination of reading posts here (and on Reddit), and a few damning/exciting articles or blog posts - not that I'd admit that at the time. I continue to seriously consider various levels of disconnection from the web, from blocking a couple of sites, to editing my hosts file, to using my routers child- blocking settings (block device by time, website etc)...

</comment></comment></comment><comment><USER7> I planned to learn React and modern JS/Web dev (had been doing iOS/Swift dev.) Instead, my dev. manager left and, since I was the most senior person on the team, I was given the job.

So, instead of learning React, I learned that I kind of like dev. management, or at least some parts of it. I also learned that I like traveling for work. I would never have thought that this would be so, but I was encouraged to try it and to my surprise it's working out well.

<comment><USER8> What kind of dev work do you do that requires traveling?

<comment><USER7> It's mostly because the dev team is geographically disperse (India/Boston/Seattle). I also have to meet with customers occasionally and of course they could be anywhere.

<comment><USER9> Don't want to turn this into a reddit AMA but .. would be nice to hear how you managed your transition to management. Did you read any books or blogs that helped? I was in a similar situation and have basically been emulating the style of previous managers - not sure how the team feels but I'm in a bit of a funk - feel like I'm letting myself down.

<comment><USER10> I am not the original poster, but you can do much worse than reading and really thinking about _The Phoenix Project_ and _High Output Management_.

<comment><USER11> Solid recommendations. I would also suggest books and content by Michael Lopp, aka Rands. He's gone from engineer to high profile manager and his work is solid. "Managing Humans" is a particular favourite and seems to be well received by engineers and leads making the transition.

</comment></comment></comment></comment></comment></comment><comment><USER12> Basics in Probability as well as a superficial introduction to Stochastic Processes. Decent foundation in mathematical analysis. Continued progress in calculus and basics in differential equations. Covered parts of the history of mathematics from 4000BC up to Newton/Leibniz. Did some mathematical modelisation, improved in R. Learned a basics in Mathematica and Matlab. Introduction to differential geometry. Had a solid introduction to group theory. Started an introduction to numerical analysis.

</comment><comment><USER13> Google cloud platform. Got certified as architect in exam beta. Kubernetes... ... and Salesforce (as a sales representative). I know, completely unrelated stuff; but worth every minute.

</comment><comment><USER14> That Django is neither the mental megalith nor archaic project I thought it was. It's actually a joy to program with and extremely flexible, without getting to e.g. Java levels of verbosity. I've not learned close to all of it yet, and have relied upon more glue code from blog posts than I'd like to admit, but it's actually pretty nice, and definitely Pythonic!

Which reminds me, the Java I knew from 6 or so years ago is apparently nothing like the Java of today..!

<comment><USER15> It's worth checking out Two Scoops of Django for some best practices and just understanding how to use the framework well.

<comment><USER14> Cool, thanks for the rec

</comment></comment></comment><comment><USER16> Learned Vuejs and oh my god it is the best thing that has happened for web development (at least to me) in a very long time.

Before this I used to like Angular 1.x but it was quirky with all the boilerplate code, DI stuff, etc and then I was completely feeling lost and upset with v2 upwards. A friend suggested I try Vue and it turned out to be everything that was missing in Angular 1 and so much better. I feel so happy working with it everyday now!

<comment><USER17> Same here. Started off by trying to pick up React. Found React to be really complicated for a non-JS developer (I've done mostly server-side stuff). Someone suggested I try VueJS, tried it and had a much easier time.

</comment></comment><comment><USER18> I started studying intensely math since May (~2-3h/day) to fill some huge gaps. Calculus up to multivariable, linear algebra and discrete math.

After those, some basic ML and I hopped straight to DL.

For CS related: Kotlin, Algorithms Design, Operating Systems & basic Networks. I'm preparing for OMSCS and I want to strength my application by earning as much MOOCs as I can. It was a very difficult -but enlightening- year.

</comment><comment><USER19> Finally had the opportunity to use Rails after not liking fat frameworks for over 10 years and kept using my own framework instead and learnt I never actually needed it.

It attracts non decent programmers and code is usually not great when you inherit a project which causes performance issues later on and things in Rails are just odd.

Variables and methods are predefined with unnecessary English plural/singular rules which you can't tell if it's a variable/method defined by the local developer or done by rails and had to google so much to know how to use it when that knowledge is completely useless once I leave Rails.

Always better to learn the actual underlying tech like SQL.

<comment><USER20> I have to say I had the exact opposite (with the exception of performance).

After 10 years as a Ruby developer I had resisted working w/ Rails (always opting to use Sinatra).

This year I embraced Rails for a large project and I'm kicking myself for not doing it sooner. The documentation is great and the speed at which you can knock out features is second to none.

</comment></comment><comment><USER21> Took a French course on a whim, and ended up learning moving to France for a month, getting myself to comfortably conversational in the space of 5 months in all. Had my first insightful conversation on Blockchain tech entirely in French!

note: French shares many long and technical words with English, thus more complicated topics are often easier to discuss.

</comment><comment><USER22> I didn't plan to learn anything new, but I did (re)learn some about c, mostly a good refresher on syntax, pointer and strings, plus the PKCS#11 API to communicate with HSMs. And along to this, a few crypto fundamentals around DES, AES, CBC/ECB, and KDF functions.

Also I used Installshield and VBScript for the first time to work on the Installer of our current project.

</comment><comment><USER23> Did not plan it as such, but trying to convert Perl one-liners to Ruby one- liners led to learning Ruby more than I expected. I tried my hand at translating "Think Python" to "Think Ruby" and by November, I even published a book on Ruby regular expressions.

For next year, I hope to continue learning various tools for text processing

<comment><USER24> Ah, another translation of "Think Python"! Laurent Rosenfeld originally wanted to translate "Think Python" to French, but then decided to translate it to Perl 6: Think Perl 6 ([https://greenteapress.com/wp/think- perl-6/](https://greenteapress.com/wp/think-perl-6/)). Which in turn got translated to Spanish by Luis F. Uceta: Piensa en Perl 6 ([https://uzluisf.gitlab.io/piensaperl6/](https://uzluisf.gitlab.io/piensaperl6/)). Where will it stop? :-)

<comment><USER23> that's because Think Python is simply too awesome ;)

I did about 60-70% before stopping mine though, hopefully will finish next year or ask for ruby community to pitch in.

</comment></comment></comment><comment><USER25> I decided to try my hand at hardware design. Just for fun, not profit. I expected to make modest progress because it has always seemed like some sort of dark art to me.

I spent a small amount of money on some Chinese FPGA boards and I have been having a great time building things and doing experiments with logic gates instead of software.

<comment><USER3> Could you tell more. I get same "dark art" feeling when I think about hardware design. And simultaneously I want to play with them.

<comment><USER25> Honestly the hardest part I have encountered so far has been learning the lingo. So much of what I was reading early on went over my head because i didn't fully understand the terminology and I don't really have an electronics background. But with enough patience and reading it starts to become clearer. And there is so much material to learn from online.

The cheapo Chinese FPGA boards don't have much onboard except an LED or two, so you will need some components and a breadboard or two to start making stuff happen.

If you have any experience with micro-controllers like Arduino (IE: knowing how to connect to components on a breadboard), and are already familiar with Boolean Algebra and finite state machines, then you won't find the initial learning stages to be too challenging either.

</comment></comment></comment><comment><USER26> A lot of Python/Django from work, Japanese by myself, and a lot about art and history from reading.

<comment><USER27> Can you share the library or packages you used in django projects

</comment></comment><comment> nevi-me Learnt Flutter and built an Android app.

Learn{t|ing} Rust, ported a lot of stuff from Java/Kotlin and JS to Rust.

Started my third degree, Maths and Stats. First semester went well, but I had to bail out of 2nd because of life issues.

I hadn't planned on learning Kotlin, everything else was planned.

</comment><comment><USER28> I learnt how to code in Swift targeting Linux. That, in turn, taught me how to deal with the availability of just a subset of the frameworks/library with which I am familiar.

Next year, I'm looking into if I can port some of my old .NET code to .NET Core.

</comment><comment><USER29> I planned to learn French, but je ne speakez it bien. Hopefully 2019 is the year.

<comment><USER22> As a French native speaker, good luck.

<comment><USER29> Thank you very much. 2019 is my year!

</comment></comment></comment><comment><USER30> \- That humans are rather primitive creatures. Monkeys in disguise.

\- The average person is quite selfish.

\- The average person is poorly educated.

Much of history/politics now makes sense to me.

</comment><comment><USER31> I didn't plan anything, but I learned enough about dozenalism to invent my own timekeeping (i.e. breaking the day down into 12 units ad infinitum). Currently it's 3,9,6 est.

</comment><comment><USER32> I've started learning Rust, and I have to say, it is the most satisfaction I ever felt from a programming language. Hoping I could do something useful with it someday :D

</comment><comment><USER33> I didn't really plan to learn about anything specific. However, I did learn that I enjoy keeping journals/notes about everything I'm doing/want to do.

</comment><comment><USER34> Ended up learning kotlin, advanced my python skills, acquired some basic ML knowledge, and journeyed further into the rabbit hole that is brazilian jiu jitsu.

</comment><comment><USER35> Mini robotics combining mechatronics, computer vision and deep or reinforcement learning at the cost of an average lunch at restaurant has really been a dream come true.

</comment><comment><USER36> Seriously upgraded my python skills. Switched almost everything I was using matlab for to python/C++ and it has been working out very well for me.

</comment><comment><USER37> I learned a lot about unix systems, server administration, Windows internals, some more C++, graph theory and distributed systems theory.

</comment><comment><USER38> Learned Elixir and Italian :D. Both are very rewarding experiences and somewhat similar in concepts of how to approach them.

What was the hardest ? Elixir.

What was more enjoyable ? Italian.

</comment><comment><USER39> Took a great Bayesian analysis/PyMC3 class [http://am207.info/](http://am207.info/)

</comment><comment><USER40> Swift - Wanted to get into iOS apps but ran out of steam.

React - No option, work required it. Starting to enjoy it finally.

</comment><comment><USER41> \- Flutter: yes I did

\- First project fully in javascript (as a backend developer this was a hard one): yes I did

</comment><comment><USER42> * factorization machines * word2vec * WARP loss for Matrix Factorization

</comment><comment><USER43> Meta question: What do people expect to get from this thread?

<comment><USER33> I want to know about what people pursued this year. I'm also the sort of person that enjoys reading resumes, and making new years resolutions. I find seeing what other people do can inform my own actions.

</comment></comment><comment><USER44> Vue.js and EF Core. Publishing packages on NPM and Maven.

</comment><comment><USER45> Coding a social network with Django and Neo4j.

</comment><comment><USER46> its not too late to keep learning in 2018!

</comment><comment><USER46> Sanskrit

</comment><comment><USER47> Learned Vue -> React

Go, Rails, Lambda/serverless, docker, java, python

Learned how to use dynamodb, google cloud datastore, bigquery, redshifts

I did this while working for startups which seen to use many different technology.

I work in ad tech because that's the only place i get to handle billion of clicks and challenges to deal with the traffic volume.</comment>


Ask HN: What are the monitoring tools you use? - <AUTHOR>

I am a newbie, and wanted to add monitoring services. I read about nagios, then later read more and found sensu|sonian was better at it http://www.sonian.com/cloud-monitoring-sensu/
Then there are things like new relic(which appears too complex to me)
I am a new sysadmin, and do not have much experience, though I have googled and read stuff, and done the homework.
I just wanted to ask folks over here, as to what services they use for monitoring their different servers, and what led them to their respective choices?
Thanks for reading the question. <comment><USER2> I guess I need to know more about what you want to monitor. We use nagios (host/service up/down notification), ganglia (host metrics like ram/disk/etc), and MRTG for bandwidth etc, and rsyslog with tenshi for log alerting. If you are a sysadmin then you probably want nagios. I would be suspect of new shiny things, nagios is rock solid and is used as an industry standard for monitoring in many open source shops (think universities, HPC clusers, etc).

p.s. I run a sysadmin screencast site @ sysadmincasts.com. I'll try and do a nagios episode in the next couple weeks.

</comment><comment><USER3> Nagios is great if you manage bare metal.

If you have a very dynamic environment (a.k.a. virtualisation, cloud, etc) usually you have to go one level furter and integrate your provisioning tools with the monitoring (and backups) system.

I've never used sensu, but I've write my own monitoring and reporting tools many times. I can recommend you to try to get that skill, learn howto script a protocol and report, etc... probably you will need it to adapt your monitoring tool, whatever you choose.

<comment><USER3> Also, as you ask for experiences:

In my current and previous job, nagios, with nrpe.

Before that, in a shop I did have only Debian servers, I did write my own dashboard.

The reason has been always pragmatism.

</comment></comment><comment><USER4> We use:

* SPM for monitoring and alerting for our ElasticSearch, Solr, Hadoop, and HBase clusters, as well as Java (web)apps + the usual server metrics + custom business and performance metrics

* Sensu for server metrics monitoring + alerting

URLs:

* [http://sematext.com/spm](http://sematext.com/spm)

* [https://github.com/sensu](https://github.com/sensu)

</comment><comment><USER5> New Relic is primarily a tool for monitoring and diagnosing on the application lauer, although they have recently expanded their offerings.

For monitoring servers, I'm currently using munin and pingdom.

</comment><comment><USER6> Zabbix!</comment>


PostgreSQL, Rails, and why you should care - <AUTHOR> http://awesomeful.net/posts/45-postgresql-rails-and-why-you-should-care <comment><USER2> Plus, FWIW, with MySQL I've had lots of table corruption issues over the years (although it's been a couple of years since then, so it might be more stable now).

Never had any issue with PostgreSQL.

<comment><USER3> I have heard about the MySQL table corruption thing before, but frankly I'm not exactly a database expert and never really worried about it. Then one of the critical tables backing Bingo Card Creator died, hard. (Thank God it was on the staging server, not on production.) I still have no clue why, and was unable to recover anything from it -- I ended up having to restore from backup.

This was on... crud, I don't really know what version off the top of my head, but it is on whatever Ubuntu ships with in summer 2009.

P.S. Have you checked _your_ backups recently?

</comment></comment><comment><USER4> One thing that PostgreSQL can do that mysql cannot (so far as I know) is lets you define your own triggers and even your own data types. In pretty much whatever language you choose.

PL/Ruby <http://moulon.inra.fr/ruby/plruby.html> for instance, could be used to define an insert trigger that would check the values being inserted into a table and perform arbitrary actions based on them; say, an alert if a user on a watchlist (another table in the db) changes certain values in their profile entry, for example.

It can lead to a fairly clean coding style where data consistency and manipulation code lives in the database.

</comment><comment><USER5> I think some folks sense blood in the water around MySQL, and are trying to grow marketshare.

<comment><USER6> Oh, I don't know - this is the same kind of "Hey everybody, PostgreSQL exists and is really neat" article that people have been writing for years. It's only poignant now that some of the MySQL issues have become harder to ignore.

</comment></comment><comment><USER7> What in particular does this have to do with Rails?

<comment><USER8> It wasn't much of an argument for Postgres either. He said it had lots of features, kinda had speed, but oh wait those things don't matter because it works with Rails and MySQL got bought by Oracle.

<comment><USER9> Actually he also said that PostgreSQL has a permissive license with a whole community of contributers where MySQL has Oracle and GPL and 1 corporate contributor. That's a significant qualitative difference in their nature and very well worth weighing up if you're deciding which way to go.</comment></comment></comment>


Fire-damaged Brazilian tortoise receives new 3D shell - <AUTHOR> http://www.9news.com.au/national/2016/05/22/12/07/fire-damaged-brazilian-tortoise-receives-new-3d-printed-shell <comment><USER2> Would have been cooler if he received a new 2D shell in my opinion.

<comment><USER3> Technically they are all at least 4d

</comment></comment><comment><USER4> Here's is cooler one. 3d printed prosthetic leg for a turtle [https://www.youtube.com/watch?v=kkHHuVAaORE](https://www.youtube.com/watch?v=kkHHuVAaORE)

</comment><comment><USER5> Do natural shells grow continuously? I'm not remembering whether that's the case. I'm curious to know how this affects the tortoise's ability to continue proper shell function. Unless, of course, the poor thing was damaged to the point that new shell growth will not be impossible.

</comment><comment><USER6> Aww they missed an opportunity to print a red shell, or a blue one. Wonder how an animal like a tortoise living with others would react to if it had a different colour shell.

<comment><USER7> I don't think their point was to experiment with reactions on unusual shells.

</comment></comment><comment><USER8> I was expecting a z shell...

</comment><comment><USER9> I actually think the right thing would have been to shoot the tortoise right where they found it. These kinds of things are more done for the selfish joy of experimentation then for the actual benefit of the animal.

<comment><USER10> It's Australia mate. They don't have much in the way of gun ownership. Once a group of children were massacred there, see the 1996 Port Author massacre for reference, they collectively decided that gun control was needed. I doubt it would even occur to most Australians to kill anything with a firearm.

Your reaction freaks me out actually.

<comment><USER9> > Your reaction freaks me out actually.

This is kind of an odd reaction. In most countries hunting is a way to control animal population and prevent the spread of diseases amongst certain populations. I also live in a country where arms are heavily regulated, but you can still call some form of animal protections services who will take care of the animal and even kill it if deemed necessary.

<comment><USER11> _In most countries hunting is a way to control animal population_

After hunting, in a dictionary definition of irony, killed off all the natural predators of the animals that need "control". So I don't think parent was an odd reaction in light of "we created a problem by killing animals, so the obvious solution is more killing!", the outgrowth of which is "got a problem with an _individual_ animal? Just kill it!"

Now you might not agree, but can you at least understand how someone might have an "odd reaction" when someone's first suggestion to the problem is to pick up a gun?</comment></comment></comment></comment>


Ask HN: How would you rank 500M rows of data in 5 seconds? - <AUTHOR> We have a database (SQL Server) which has a table filled with stats. The table currently has around 10 million entries in it. The table is organized "horizontally" so that one user has one row, and we have about 50 stats in the table.
We have a leaderboard for every single one of these stats, and we want to have a "live" leaderboard - that is, the ranks are fresh on each request.
Right now we use an indexed RANK() query which can rank the top 100 in like 15ms. Top 10K is 100ms. Top 1M (or, offset 1M and get the next 100) is around 20 seconds.
At the very least, we'd like to rank 1M within a couple seconds (10x improvement) but ideally we'd be able to serve any number (up to say.. 10 million) of ranks in under a second. This is probably unreasonable, but it's our goal.
Is there any advantage to using a "vertical" stable structure where each stat is a row? That's where the >100M figure comes from - if we convert these stats into rows it's (number of users) x (number of stats).
Is there any specialized way of accomplishing this? Is the database slowing us down? Maybe there's a better way to store all of this stuff than in a DB? I was thinking perhaps storing it in the DB and an in-memory store at the same time, and repopulating the in-memory store when the server starts. But that wouldn't be much use if the in-memory solution doesn't give us a massive performance boost.
Right now we use dedicated servers for our application and database needs. We'd like to switch to cloud architecture but we want to both accomplish what is mentioned above while also reducing costs (we pay quite a bit for our current stack.) <comment><USER2> So to be clear:

        - You have 500M rows x 50 columns of stats for each row.
        - For any stat, you want the ability to display the top 10M.
        - In under one second.

Is that correct?

If so, I think the problem is not clearly described. What does it mean to display a result row? Are you displaying the stat value and a primary key? The entire row? What exactly does it mean to display 10M rows? Are they all presented on one web page? Or is it N at a time with prev/next links? Or is there random access within those 10M (e.g. find me in the ranking for stat #27). Do you overlap retrieval with the display process, (showing the first rows when available)? When someone clicks prev or next does it recompute the entire result?

<comment><AUTHOR> Pretty close. 10M rows for each stat -- this number grows linearly with the number of users. There are two cases for showing stats. The first is a leaderboard: random access (because of how we allow users to scroll through the leaderboards) where you'd be fetching 100 consecutive records from ranking 0 of that stat to rank N where N is the total number of rows for that stat. The second case is where you want to get the rankings for all stats for a particular user.

As for how the data is returned - whatever is fastest but ideally the response is sent back as JSON so it should all be available by the time it is sent back.

As for caching, I think it would be ok to cache the data for a short period of time, though I don't know how efficient it would be to cache 10 million rows at once (per stat.)

<comment><USER2> Scrolling is not random access. Scrolling means like start at the beginning, and keep going, stopping eventually. Rows are provided in a fixed order. Random access means jumping immediately to any row at any time.

All rankings of one user is an additional requirement, not mentioned in your first post.

I wouldn't necessarily rule out caching, but it isn't yet obvious that it is needed.

This is probably not the best way to discuss your requirements and possible solutions, going back and forth on HN. I consult if you'd like to talk further.

<comment><AUTHOR> Yeah I suppose I haven't done a great job outlining all of the requirements. Is there a way we can get in touch? Not sure we're looking to hire a consultant yet but it would be good to have as an option if we decide to go that route.

<comment><USER2> Sure, I'm jao at geophile dot com.

</comment></comment></comment></comment></comment><comment><USER3> The first thing _I_ would try would be to put each variable in its own table. If that's not enough you can go all the way to each stat on its own server.

With What you have right now, you are going to do 50 sorts on the same data, jumbling it all up unless all the stats are highly correlated. The entity- value structure you propose would probably be worse.

But, you have the data, you can measure these things.</comment>


Amazon Launches GSM-based International Kindle Device - <AUTHOR> http://www.amazon.com/gp/product/B0015T963C

<comment><USER2> Looking at amazon.com's store, many books are not available in Europe .e.g. "The Angel's Game" or even Dan Brown's latest. Comparing the top sellers list (assuming you buy a Kindle hoping to get new books for $10 kindle prize rather than $25 local imported paperback price), only 23 out of top 59 books are available in UK (this I found by sorting by bestseller in the store with region set to Europe and US accordingly -- the order matches up, except for the gaps).

I'm going to wait until the International Kindle is up on Amazon.co.uk -- buying anything from outside of EU for me personally is probably more expensive than waiting for it to arrive in UK, even though the price will likely be higher. You've got VAT, you've got customs and you get a $25 customs fee and VAT on top of customs fee. And of course, you're paying VAT and customs on the shipping charge as well.

I'm still happy it's finally here although I wonder about bookstores. Do you Kindle owners still go in, browser a bookstore and buy books at random? Even though I buy plenty of stuff from Amazon it's mostly professional books or hard-to-find paperbacks. I still go into bookstores several times a month and just browse at random at highlights or new editions.

<comment><USER3> "many books are not available in Europe (...) even Dan Brown's latest."

I call this a feature

</comment></comment><comment><USER4> funny, they will ship to Uganda, but not Canada

<comment><USER5> yup, looks like Canadians are screwed by licensing again.

<comment><USER4> is it because of licensing? my first thought was they weren't able to get a deal with Rogers.

<comment><USER6> Bell/Telus have HSPA as of November, so maybe something's coming later?

<comment><USER7> Should be. For Canada they say they are working to bring us the Kindle. They don't say that for all countries.

</comment></comment></comment></comment></comment><comment><USER8> this is the 6 inch version. Hopefully they'll launch the international dx version soon. It will be HUGE in India if it works as well as advertised.

<comment><USER9> The price point is _way_ too high for mainstream usage in India. The actual price in India is $435 ($289 base price, $42 for shipping, and $104 for import duty!!).

I wonder if there will be a hack to buy a US Kindle and then use it locally here in India.

<comment><USER3> It would be a lot smarter to unbundle it from the unlimited wireless connectivity forever plan. The device would be cheaper and users could decide how much (or how little) they want to spend on connectivity. I also bet slashing the cost of the perpetual wireless could allow Amazon to include wifi, making moot my whole point here.

</comment></comment></comment><comment><USER10> They also dropped the price of the US Kindle 2 from $299 to $259.</comment>


Woman registers period late on app, is starts seeing baby related ads - <AUTHOR> https://twitter.com/TaliaShadwell/status/1191054331119837185 <comment><USER2> Reminds me of a story maybe ten years ago where a daughter had her pregnancy outed to her parents due to all the "you're pregnant!" spam being sent to everyone in the house all of a sudden.

<comment><USER3> Wasn't that caused by Target physical mail spam? At least that's what I remember.

<comment><USER4> Yes, I remember reading this in the book about building habits(?) I believe.

</comment></comment></comment><comment><USER5> Doesn't Facebook allow you to see why an ad targets you?

[https://newsroom.fb.com/news/2019/07/understand-why-youre- se...](https://newsroom.fb.com/news/2019/07/understand-why-youre-seeing-ads/)

[https://www.cnbc.com/2019/04/01/facebook-new-tool- explains-w...](https://www.cnbc.com/2019/04/01/facebook-new-tool-explains-why- am-i-seeing-this-post-on-news-feed.html)

</comment><comment><USER6> Moms-to-be leads are the most expensive consumer leads.

</comment><comment><USER7> That's a big privacy issue if you start doing IP address based targeting on a residential address....

</comment><comment><USER8> This looks like confirmation bias to me.

</comment><comment><USER9> Surveillance app is to personal data as woodchipper is to arm.

It's a gross failure of professional ethics to allow these attractive nuisances to exist and trip people up.

</comment><comment><USER10> Was the app's name mentioned?

<comment><USER11> No, she's a reporter, and wasn't confident enough to out the app without proof.

<comment><USER12> I am not convinced that an app like that would share potential pregnancy data with ad networks, if there's even such a channel to begin with. It would be good to have the app's name, and investigate if this is reproducible.

Without that we're just looking at very complex ad networks/algorithms and trying to reason about them without seeing thousands of signals they might be seeing. For example, age and other interests based on browsing history would've triggered this, too.

<comment><USER11> 1) she claims the ads abruptly stopped as soon as she corrected/updated the app to show she wasn't late.

2) There is huge demand for finding out about a consumers pregnancy.

It's one of the "big events" that marketers sometimes pay $10s-$100s for a jump on competitors, along with weddings, buying a house, & going to college.

It's one of the few windows in a person's life where they 1) are about to spend a bunch of money on products they have no/little prior opinion on, and 2) are willing to substantially change previous habitual behaviors, including where to shop.</comment></comment></comment></comment>


How do the rest of you feel about Quora.com? - <AUTHOR> https://www.monosnap.com/image/I2Q29GVCDfHcYtgi7DAzLjw3P

<comment><USER2> I receive the Quora Weekly Digest emails and will tend to open them up and have a skim when I'm looking for something to read. I'd say that roughly 50% of the time they include a topic / response which grabs my attention enough to click through.

I don't engage with the site beyond this, so personally it fills a similar role as scoopinion and other "content recommendation" sites I occasionally hit up to find something to kill a few minutes reading.

</comment><comment><USER3> Am I the only one who thinks this is unnecessarilly rude and abhorent?

The lady was just doing her job for duck sake. For someone whose main point of argument is that they show humanity, you seem to have very little of your own.

<comment><USER4> Was he rude? Yes. No excuse for that. But what he dead on right? Absolutely.

<comment><USER3> Are you saying there was no other way to pass on the exact same message with the exact same effect without being a total cunt about it?

</comment></comment></comment><comment><USER5> I received the same email, but I deleted it instead of responding. I guess Tom's response is more likely to get the point across to them that they suck. Nothing makes me click away from a page faster than seeing the word "Quora".

On second thought, they probably don't care what anyone thinks of them given the way they so obviously try to suck you in to exploit you.

</comment><comment><USER6> Why would I ever go there? If I go to their home page, all I get is a "sign up to read Quora". Umm, no, you don't get my email just so I can see what you are about. Okay, I've come across quora plenty of times in google searches, so I know what you are about, but I've never felt that I am terribly missing out by not being able to read the link.

</comment><comment><USER7> Tom's reply here is awesome and it would be funny to find out if he ever received a reply to his message.

Personally I think Quora had a lot of potential (and still could radically improve) but I stopped bothering with it when I couldn't actually read it without logging in.

</comment><comment><USER8> I'm a polymath here's my answer...

Seriously? Somehow this only happens on Quora.

</comment><comment><USER9> I'm never going to register (or login) just to see if any responses to questions are relevant to my current problem at hand.

I'll visit quora if it came up in a google search. If the answer to the question is not right there, that tab's being closed. (This is pretty much the reason that experts-exchange.com died as much as did)

</comment><comment><USER10> About sums it up for me. I don't like being herded, so I don't visit Quora anymore.

</comment><comment><USER11> Never heard of it.

Oh, wait. That's the site that tries to be like [http://stackexchange.com](http://stackexchange.com), but is behind a login, and they steal your data, violating your privacy, tracking you through their mobile app, and is run by former Facebook engineers?

Not interested.</comment>