Who said what inside the Trump tech meeting - <AUTHOR> http://www.recode.net/2016/12/15/13976806/immigration-maternity-leave-grid-software-president-trump-tech-meeting <comment><USER2> > Immigration and how the government can help tech with things like H-1B visas > to keep and bring in more talent. Nadella pointed out that much of the > company's spending on research and development was in the U.S., even if 50 > percent of the sales were elsewhere, so that immigration would benefit those > here.

If the H1B program remains, they need to give foreign workers the ability to switch firms while in the US. Current policy forces them to remain with their current firm, which reduces their bargaining power and puts them at risk of being overworked and exploited. Further, it reduces competition between firms, lowering wages. (e.g. it is difficult for a H1B worker to get another job with a pay raise, so they are stuck with whatever salary they have.)

A study by U Notre Dame (which I cannot find on mobile) showed that H1B visas did _not_ increase innovation measured by patent grants, but did 1) increase company profits, and 2) decrease all salaries at the firm, citizen and non- citizen.

This suggests the true incentive of H1B visas is to increase profits by lowering wages, not to find workers that firms are unable to find as citizens.

<comment><USER3> They should start by solving the Indian Consultancy H1B problem.

If you look at the H1B statistics, you'll see that all the top employers are indian consulting firms. They consume a huge percentage of the very limited H1B visa pool, only to take cheap H1B workers from abroad.

The tech companies are far behind.

[http://h1bdata.info/topcompanies.php](http://h1bdata.info/topcompanies.php)

<comment><USER4> At the risk of being crucified here, Trump did say he plans to make H1Bs more expensive for companies; in hopes that the program starts being used for extraordinary talent (as meant to) rathan than cheap talent. If he keeps his promise, it's most likely to end that problem.

</comment></comment></comment><comment><USER5> > Alphabet CEO Larry Page, Tesla and SpaceX CEO Elon Musk, Apple CEO Tim Cook, > Facebook COO Sheryl Sandberg, Microsoft CEO Satya Nadella, Cisco CEO Chuck > Robbins, IBM CEO Ginni Rometty, Intel CEO Brian Krzanich and Oracle CEO > Safra Catz.

3 women out of 9 people that's not too bad for tech (or CEOs)!

</comment><comment><USER6> Way to go Larry, direct DC power for everyone!

[https://en.wikipedia.org/wiki/High- voltage_direct_current](https://en.wikipedia.org/wiki/High- voltage_direct_current)

<comment><USER7> Yeah, too bad that people were apparently confused about it. I think an HVDC grid will be key as renewables take up a larger share of electricity supply. It might not be windy or sunny here, but it probably is somewhere else.

</comment></comment><comment><USER8> > Also brought up ... the repatriation of tech company profits

...and‽ We want to know not just the agenda but what took place. Even if the administration was noncommittal/unresponsive, stating that would make me feel better. Or if you couldn't get anyone to go on record, maybe at least let us know that you asked and no one offered comment.

> Musk also ... brought up ... climate change, in other meetings at Trump > Tower.

...and what did Trump say‽ Jeez, you did the drum roll and left me totally hanging. Seems like a controversial topic for sure and no mention of whether Trump dismissed this or debated it or just sat quietly.

</comment><comment><USER9> Trump campaigned on a stance of abolishing the H1B scam - he even had displaced workers giving speeches on stage about it - so I don't know how Nadella can turn around and say "we need to increase H1B issuance".

<comment><USER10> Nadella didn't campaign on it, and it benefits tech companies to have a ready supply of cheap workers who can be kicked out of the country if they quit.

So I imagine it was pretty easy for him to say that.

<comment><USER9> Oh well yeah, I understand that he would say it because it's in his interest. I just think it's odd that he doesn't seem to be aware of Trump's (ostensible) position.

<comment><USER8> Odder still is Trump's response:

> Surprisingly to the group, Trump apparently responded favorably, "Let's fix > that," he said, without a specific promise, and then asked, "What can I do > to make it better?"

It's almost as if he doesn't have ideals and is very easily influenced by others who can offer him money and/or power.

<comment><USER11> Or his "let's fix that" was just a way of keeping the conversation going and getting more information. It sounds like that's what this meeting was--a chance for the attendees to voice their perspective on what's important.

</comment></comment></comment></comment></comment><comment><USER12> .

<comment><USER13> The media tells us what to think about. The billionaires want to control media so they can sway public opinion and get what they want.

1\. The media creates an "issue"

2\. The "issue" riles people up

3\. Calls are made to congress critters

4\. Facebook posts are posted

5\. Twitter hash tags are create

6\. Billionaire suddenly has a plan to correct\fix\control issue</comment></comment>


San Francisco flattened the curve early. Now, coronavirus cases are surging - <AUTHOR> https://www.washingtonpost.com/technology/2020/08/02/san-francisco-coronavirus-surge/ <comment><USER2> People stayed inside for 3 months to buy the feds time to do something. And the state ran out of money.

Clearly, nothing is going to change at the federal level so people have quarantine fatigue.

<comment><USER3> the feds provided large quantities of ventilators and are funding multiple vaccine research. What else should they do? Every state has enough power to mandate mask wearing, quarantine, school closings without federal intervention

<comment><USER4> The debt black hole pushing everyone to reopen is a direct product of the Federal Reserve. A real shut down would involve a moratorium on rent denominated in USD.

(Never mind all of the little bits of discouragement the federal government has been feeding down to states, eg threatening to withhold school funding)

<comment><USER3> 1\. I'm pretty sure that states can decide to not evict tenants (thereby severely hurting landlords who possibly have multiple mortgages to cover the rentals.

2\. Did any other country uniformly cancel all rent?

<comment><USER5> Some like Germany safeguarded salaries instead by ensuring that workers remained employed. That's to say that they recompensed firms for salaries on condition workers remained employed during the lockdowns.

The US should have more leeway because it has the world's reserve currency.

I haven't worked it out but I'd like to hear why the Fed or Treasury (let's just say federal govt) couldn't simply replace lost real estate loan payments by adding to banks' balance sheets.

<comment><USER3> how is Germany's solution any different than giving unemployment benefits for furloughed employees? Regarding the Fed, they _could_ just print gobs of money, and probably nothing bad will happen in the near future. But if you want examples of why printing gobs of money is bad, just looks at the many other countries that tried that. Also, I would think that if the fed decides to print gobs of money, it would be bad for them to allocate all of it to repaying bank mortgages. I would much prefer the money gets distributed to the people (eg. $1,200 per person) or as loans to keep small businesses afloat until after the pandemic

</comment></comment></comment></comment></comment></comment><comment><USER6> I'm really not being an apologist here but how did Europe avoid this? I mean let's say Paris did everything right. Isn't this still where you end up once people venture back out?

Clearly not but I don't understand how.

<comment><USER7> I found this data [0, 1] from the economist which does suggest that Western European countries really did mitigate the virus and continue to do so. Excess deaths are a more robust measure of the effects of the virus because they're unaffected by testing differences and idiosyncrasies between countries.

I have no idea if there are any travel restrictions in place internally to the Schengen area right now but I wouldn't be surprised if the free movement in the US between areas with very good control over the virus and area where it went haywire was a factor in the current spikes.

Hispanic people are greatly over represented in current coronavirus cases in SF [2], supposedly because they are more likely to be essential first-line workers, but I think also having to live in very cramped living situations (e.g. 5 people in a 2 bed apartment) is another factor. So in that case wealth inequality could be another contributing factor as to why cases are higher in the US than the rest of the world.

[0] [https://ourworldindata.org/excess-mortality- covid](https://ourworldindata.org/excess-mortality-covid) [1] [https://www.economist.com/graphic- detail/2020/07/15/tracking...](https://www.economist.com/graphic- detail/2020/07/15/tracking-covid-19-excess-deaths-across-countries) [2] [https://www.cnn.com/2020/06/18/health/coronavirus- hispanic-s...](https://www.cnn.com/2020/06/18/health/coronavirus-hispanic-sf- spread/index.html)

</comment></comment><comment><USER8> [http://archive.is/5kSVB](http://archive.is/5kSVB)

</comment><comment><USER9> I definitely worry about this here in the Midwest, where we originally slayed the curve. But the key difference on these later spikes is testing infrastructure is better developed, so we _should_ be able to isolate people more effectively.</comment>


Student Charged $14k on Stolen Google Cloud Credentials - <AUTHOR> Hi,
In 2017, I made a Google Cloud Account to use Google Maps API for a Computer Science student group project and put my debit card in. I naively put a $5 account notification in, thinking it was a cap. This project was defunct after 2017 and I should have just closed the Cloud account.
All was fine up until January 2019 when the Google Cloud Credentials were somehow stolen and over the course of two days on Google Maps API, racked up enough API calls to generate over $14k invoice. I disabled the Google Cloud Account a day after I noticed an email from Google Cloud. Google Cloud did try to use debit card to deduct from checking account, but I don't leave thousands sitting around in it, so charge was declined.
I talked to Google Cloud Billing and they have not been helpful, telling me to contact my bank. Today, I got a scary email from a collections agency demanding I login to my Google Cloud account and pay the bill! Worst part is, this API used to be free, until Google started charging exorbitant amounts for it.
I know I did not make these API calls -- if you looked at the call volume history, there was nothing for well over a year, until those two days in 2019, it started going crazy (and the project is not running on any server or being used in any way). I suspect a group member might have accidentally leaked the credentials.
I know AWS has waived costs[1] like this in the past, but Google is not known for customer support. I should have been more proactive in setting up a cap.
Appreciate any advice or Google contacts to talk to an actual human. Should I see if Google is willing to actually verify this was unauthorized usage or just lower the bill? I'll eat a few thousand just to make this go away.
To say GCP has left a sour taste in my mouth is an understatement!
Thanks for reading.
[1] https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn <comment><USER2> Sorry this happened to you! Feel free to send me your case number (email in profile), and I'll escalate it.

The Support personnel have hopefully been helping out, as all Billing Issues are covered regardless of support tier. I obviously don't know the ins and outs of payment instrument refunds / do debit cards mean that you actually do have to contact your bank, but I'm sure people in Support do.

<comment><USER3> Thanks for responding to this and offering to escalate boulos.

I certainly don't think a student just learning the ins and outs of a cloud provider's services should be able to spend 10k+ without warnings/thresholds that require configuration to exceed. It would be positive for platform adoption to make that process better.

<comment><USER4> This is hilarious. Student doesn't understand security in depth model, gets owned. Has a sour taste with said cloud provider. At what level do you accept responsibility for shoddy security practices. If the project was truly defunct then you should have closed the project or removed everyone's access who isn't project owner. Hindsight is 20/20.

<comment><USER5> Nice victim shaming you got there. The fault of all of this is 100% on whoever stole the credentials and made those calls. OP could maybe have been more careful but that doesn't mean it's all his fault or that we should be shaming in oblivion. And Google can still try to help rather than just take advantage of the situation. Life is easier when we are not dicks to each other, a little empathy can go a long way.

<comment><USER6> >>>Life is easier when we are not dicks to each other, a little empathy can go a long way.

I found my new email signature.

<comment><USER7> I am seriously wondering whether it's a good idea to put it in my business email or not :D

</comment></comment></comment></comment></comment></comment><comment><USER8> I've heard so many stories of something similar happening on AWS and after an email to support, all of the charges were dropped.

This isn't exactly helping Google to fight the narrative that it isn't good with customer support and they can't be trusted as a platform for business.

So if you were a person deciding who to choose as your cloud provider, who are you going to choose?

AWS - "No one ever got fired for choosing AWS"

Microsoft - well known for their enterprise support and there are plenty of MS Shops out there.

Or

Google?

<comment><AUTHOR> I agree with you 100%. This has certainly left a sour taste in my mouth. Unfortunately Google Maps API is much better than the competition. AWS has none here.

The google customer experience is just horrific.

</comment></comment><comment><USER9> While you're figuring this out, backup all your data on Google. Google is crazy and could possibly delete all your accounts and data.

<comment><AUTHOR> Backing up now, thanks. Even worse I'm an Android user. I don't have a good solution for the people that mail me every few months. Oh hey, Google might lock my account because I didn't fork over 14k in fraudulent charges, can you start emailing me from my Outlook email instead?

<comment><USER10> The idea email is your own domain name. I'm yet to take my own advice though!

</comment></comment></comment><comment><USER11> Before disputing the charge, be sure to back up all data and contact info from your Google accounts. Fighting charges has been known to trigger account lockouts with no appeal.

<comment><AUTHOR> This is even more terrible as an Android user. So much for "don't be evil". Thankful for the HN forum.

<comment><USER11> Hope it works out with a minimum of hassle!

</comment></comment></comment><comment><USER12> Did you check your Github repos and associated commit history for accidental push of secret files? There's an article on the HN front page describing secret leakage in Github repos (the most common is Google API keys, go figure)[1]. I imagine somebody out there has a bot to monitor pushes in realtime to extract secrets. You or a team member might have leaked keys in a similar manner.

[1]: [https://blog.acolyer.org/2019/04/08/how-bad-can-it-git- chara...](https://blog.acolyer.org/2019/04/08/how-bad-can-it-git- characterizing-secret-leakage-in-public-github-repositories/)

</comment><comment><USER13> Google will typically waive charges in cases like this.

The only time they won't is if (by looking at the logs) they decide you were probably scraping and storing all their data.

</comment><comment><USER14> Make them prove you used it to generate the charges. Make them provide IPs etc.

You need to say it was used fraudulently and you don't agree to the charges.

<comment><AUTHOR> Will do, thanks for the advice. The only thing they did was validate the API count, but I was not disputing this, only that I made the calls.

The call volume itself looks suspicious. No calls for well over a year and then suddenly an incredible amount? You would think Google would have some algorithm to detect this?

<comment><USER14> Yeah exactly. But the key is not to say "lower my bill", but rather "these charges aren't mine".

If they send it to collections you can dispute it. A lot of laws and things around that.

It sounds like it hasn't been charged off yet though, since they want you to pay Google, not them.

I wish you luck. :)

</comment></comment></comment><comment><USER15> Next use a credit card. Basically thanks to credit card laws the bank will go tell google to f off and give you your money back. Debit cards don't have the same protection, but just call your bank or OCS ([https://www.occ.treas.gov/](https://www.occ.treas.gov/)). They have a little more bite.

<comment><USER16> His debit card ended up declining it anyway, but that's not the point. The point is Google thinks he owes them thousands of dollars, and a collection agency is hounding him for the money.

Disputing a charge with your credit card company doesn't change that. The credit card company might side in your favor and reverse the charges, but the original company can still pursue payment with you directly. Their claim can even show up on your credit report. I remember working with a payment processor in Europe that even automated sending a bill to collections if it was charged back by the consumer's credit card company.

<comment><USER4> playing devils advocate, but taking peoples word prima facie is dangerous. Especially when it deals with financial issues. Pretty sure you can any contempt person could game the system if people found out "students" automatically get refunds for bs charges. I think AWS is different as they have their educate program and distinguish between students and free trial customers. AFAIK gcp doesn't have an education system for students, so it's harder to differentiate a legitimate request

<comment><USER10> What will these system gamers do with computer power? Mining crypto will make them pennies, and running a business on stolen accounts isn't sustainable.

<comment><USER17> there are entire dark market businesses built out of finding secrets and deploying a crypto miner as fast as possible and mining as much as possible.

that's just one use case, DDOSing an API/website through seemingly legit requests from the G3's cloud infrastructure is another.

Lots of things you can do with massive amounts of computing power. It doesn't have to be sustainable in the short term, you just keep repeating the process and the value adds up fast

</comment></comment></comment></comment></comment><comment><USER18> Keep trying to talk to them and explain -- so far every instance like this I've heard about was refunded. Good luck.

</comment><comment><USER17> I dunno if google lets you do this but amazon/azure will pretty reliably let you create new free tier accounts with fake emails and access them from the same IP. i just create a new debit/credit card every 6 months(it's pretty hassle free in india).

i do pay for production instances, i just don't want to mess around on those production instances</comment>


China's surveillance state has created at least four billionaires - <AUTHOR> http://www.bloomberg.com/news/articles/2019-02-21/big-brother-billionaires-get-rich-as-china-watches-everyone <comment><USER2> Writing this from China- it's hard to really understand how little the Chinese seem to care about any of this. The systems that so obsess western writers seem to have absolutely no effect on Chinese society, and the majority of these new systems seem to be replacing similar but more intrusive systems of human-to-human checks. The concerns are warranted, but the interest approaches zero in the culture here.

<comment><USER3> You have a completely different worldview, a completely different education, a completely different history, and a completely different upbringing.

For example, China never had the Magna Carta event that Western civilization had centuries ago, enabling a foundationally disparate view of government and society eons ago.

This is not a criticism of you. It's an explanation why you can't understand. You need to transplant yourself into Chinese history to understand its people. That in turn will help you understand its domestic policy, economic policy, foreign policy, and more.

Have gone through that process myself, lived in China for almost a decade now. I also didn't get it at the beginning. I'm an overall China bear, think its final destination is to be like Japan, but mainly due to secular trends beyond China's control. That doesn't stop me from figuring out things that I didn't know before. It's weirdly educational, as I didn't realize how intellectually arrogant, ignorant, and unwilling to consider other perspectives I was before. Perhaps still am. :)

<comment><USER4> I think you do a great disservice to the Chinese people by assuming that they don't care or that they support it. The fact is that they're helpless. They have nothing to gain by expressing their true beliefs and everything to lose. I have many Chinese friends and they all either post government propaganda (that they don't believe in or support) on weixin or stay quiet. Why? Because it will be seen by their families, friends, classmates, professors, coworkers, bosses, etc. When they go travel, want to get a loan, a promotion, etc. then it will play a major role.

<comment><USER3> What can I say? My Chinese friends who like the job their government is doing outweigh my Chinese friends who don't. Want to compare numbers and demographics? I've accumulated a fair number of Chinese friends from all walks of life in multiple cities in multiple regions across the country, both very rich and very poor. When you speak of me doing a great disservice, it speaks of a moral superiority in your mind. I used to think like that. Now I feel like I understand Chinese society better. I don't agree with much of it, it still doesn't match my values. But I do understand it better than before, even if I disagree with it.

<comment><USER5> I have a fair few Chinese friends and have found something similar. Sure, I know a few Chinese who are desparately trying to do whatever they can to emigrate from China, but I also know quite a few who studied/worked overseas and then returned to China because they prefer the conditions there. They recognise some problems with how things are in China, but view the problems overseas as greater.

America for instance is perceived to be a more dangerous place to live. By official numbers (which is what people see), you're much more likely to be raped or murdered in a big American city than in a Chinese one. The streets aren't covered with homeless people, used needles and human faeces in China. You don't get $x0,000 healthcare bills for simple procedures in China. Big Chinese cities don't have "no-go zones". Your kids won't be bullied for academic success, they'll be bullied for academic failure. Your kids are much less likely to be exposed to hard drugs and drug users in high school. Sure, this may not actually be due to the efforts of the authoritarian government, but that doesn't stop people crediting the government for it.

Europe is also seen as dangerous. There is no political correctness in China, and many middle class Chinese fall relatively far to the right of the political spectrum by western standards, viewing Europe as a failing state ravaged by Muslim immigrants and undermined by its coddling welfare policies. A Chinese friend, who grew up in a relatively poor part of China but is now stably middle-class, recently went on holiday to Italy, and one of the first things they said when they returned was that they were shocked by how poor everybody seemed there.

For ambitious Chinese, many feel they have a bigger chance of making it big in China. Count the number of ethnically Chinese tech company CEOs in Chinese companies vs US companies, for instance. The "bamboo ceiling" is not present in China, and due to the population the sheer number of job opportunities for professionals is bigger than most other countries can offer.

<comment><USER3> _> > For ambitious Chinese, many feel they have a bigger chance of making it big in China. Count the number of ethnically Chinese tech company CEOs in Chinese companies vs US companies, for instance._

You were expecting Chinese tech companies to mostly have non-Chinese CEOs? ;)</comment></comment></comment></comment></comment></comment>


The domain name htt.ps is for sale - <AUTHOR> https://htt.ps/ <comment><AUTHOR> This could be weaponized for phishing</comment>


Are ML and Statistics Complementary? [pdf] - <AUTHOR> http://www.ics.uci.edu/~welling/publications/papers/WhyMLneedsStatistics.pdf <comment><USER2> They definitely are as far as their roles at (most) startups are concerned.

Unless your startup's core strategy involves machine learning, statistics tends to come handier than machine learning in the early days. Most likely, what moves your company is not a data product built atop machine learning models but the ability to draw less wrong conclusions from your data, which is the very definition of statistics. Also, in the early days of a startup, you experience small/missing data problems: You have very few customers, very incomplete datasets with a lot of gotchas. Interpreting such bad data is no small feat, but it's definitely different from training your Random Forest model against millions of observations.

</comment><comment><USER3> LeCun has a comment on this paper here: [https://www.facebook.com/yann.lecun/posts/10153293764562143](https://www.facebook.com/yann.lecun/posts/10153293764562143)

<comment><USER4> Thanks for sharing that, he's got some interesting stuff to say about this topic.

</comment></comment><comment><USER5> Here is a link to the paper referenced in the beginning: [http://courses.csail.mit.edu/18.337/2015/docs/50YearsDataSci...](http://courses.csail.mit.edu/18.337/2015/docs/50YearsDataScience.pdf)

Great read for anyone interested in the debate.

</comment><comment><USER6> I think they will eventually converge.

Probabilistic programming is already a hint of this. The most general class of probability distributions is that of non-deterministic programs. ML is just a quick and dirty way to write these programs.

<comment><USER7> It's not just a way to write them, it's a way to do inference. Probabilistic programming is extremely powerful in terms of representation but inference is, in general, intractable. Yes, you can express all those ML models as probabilistic programs, but the sampler isn't going to perform nearly as well as the original algorithm.

</comment></comment><comment><USER8> What is commonly understood as 'statistics' is just a specialized subset of machine learning. Machine learning generalizes statistics.

The correct complement to machine learning is cryptography -- trying to intentionally build things that are provably intractable to reverse engineer.

<comment><USER9> Working with both statisticians and pure machine learners on the same task, I did notice some tendencies, presuppositions and modus operandi that were different (beyond being a specialized subset). Like said in this position paper, machine learners like to throw computation and parameters at the problem, where statisticians are more careful and sober. As an analogy, a statistician will approach a cliff very carefully, stomping the ground to make sure it is sturdy enough to carry a human. They'll approach the edge of the cliff 'till they have their p-measures and that is their model. Machine learners will jump head-first off the cliff and when you listen you can hear them yell: Cross-validatioooohhhh... as they plummet down.

I like the complement with cryptography. I would add another coding method: compression - Approximating the simplest model with explanatory power.

<comment><USER8> I have had the exact opposite experience with machine learning and statistics. In my experience, those who come from the 'statistics' side tend to use constructs, like null hypothesis significance testing, which are not consistent even from a theoretical point of view. And further, when they use them, they do awful things like p hacking, or using a direct comparison of t-stats as a model selection criterion, which are further rife with theoretical problems, not to mention lots of statistical biases and so forth.

I find the machine learning approach is far more humble. It starts out by saying that I, as a domain expert or a statistician, probably don't know any better than a lay person what is going to work for prediction or how to best attribute efficacy for explanation. Instead of coming at the problem from a position of hubris, that me and my stats background know what to do, I will instead try to arrive at an algorithmic solution that has provable inference properties, and then allow it to work and commit to it.

Either side can lead to failings if you just try to throw an off-the-shelf method at a problem without thinking, but there's a difference between criticizing the naivety with which a given practitioner uses the method versus criticizing _the method itself._

When we look at _the methods themselves_ I see much more care, humility, and carefulness to avoid statistical fallacies in the machine learning world. I see a lot of sloppy hacks and from-first-principles-invalid (like NHST) approaches in the 'statistics' side. And even when we consider how practioners use them, both sides are pretty much equally as guilty of trying to just throw methods at a problem like a black box. Machine learning is no more of a black box than a garbage-can regression from which t-stats will be used for model selection. However, all of the notorious misuses of p-values and conflation over policy questions (questions for which a conditional posterior is necessarily required, but for which likelihood functions are substituted as a proxy for the posterior) seem very uniquely problematic for only the 'statistics' side.

Three papers that I recommend for this sort of discussion are:

[1] "Bayesian estimation supersedes the t-test" by Kruschke, [http://www.indiana.edu/~kruschke/BEST/BEST.pdf](http://www.indiana.edu/~kruschke/BEST/BEST.pdf)

[2] "Statistical Modeling: The Two Cultures" by Breiman, [https://projecteuclid.org/euclid.ss/1009213726](https://projecteuclid.org/euclid.ss/1009213726)

[3] "Let's put the garbage-can regressions and garbage-can probits where they belong" by Achen, [http://www.columbia.edu/~gjw10/achen04.pdf](http://www.columbia.edu/~gjw10/achen04.pdf)

<comment><USER9> Thanks for the links to interesting papers. I really liked the Breiman paper. I did not try to qualify either machine learners and statisticians as bad or good, just pointing out a difference in their approaches to problems.

I do not know enough about statistics to make a (negative) quality statement about it. I know a bit more about machine learning though, and there I also see things like: Picking the most favorable cross-validation evaluation metric, comparing to "state-of-the-art" while ignoring the real SotA, generating your own data sets instead of using real-life data, improving performance by "reverse engineering" the data sets, reporting only on problems where your algo works, and other such tricks. I believe you when you say much the same is happening for statisticians.

Maybe it was my choice of words (careful, sober). I think its fair to say that (especially applied) machine learners care more about the result, and less about how they got to that result. Cowboys, in the most positive sense of the word. I retraced where I got the cliff analogy. It's from Caruana in his video "Intelligible Machine Learning Models for Health Care" [https://vimeo.com/125940125](https://vimeo.com/125940125) @37:30.

"We are going too far. I think that our models are a little more complicated and higher variance than they should be. And what we really want to do is to be somewhere in the middle. We want this guy to stop and we want that statistician to get there, together we will find an optimal point, but we are not there yet."

<comment><USER10> Thx for linking to the Caruana video, very interesting.

</comment></comment></comment></comment></comment><comment><USER11> This is a great summary of the field.

</comment><comment><USER12> I think feasibility to get an explanation for the results of modern machine learning is wishful thinking. I personally cannot explain my gut feelings. So why should we expect an explanation when machine deals with the same class of problems?

Besides, it is easy to get wrong explanation and, as Vladimir Vapnik in his 3 metaphors for complex world observed, [http://www.lancaster.ac.uk/users/esqn/windsor04/handouts/vap...](http://www.lancaster.ac.uk/users/esqn/windsor04/handouts/vapnik.pdf) , "actions based on your understanding of God's thoughts can bring you to catastrophe".

<comment><USER9> As we start to use AI/ML for more tasks, the need for model interpretability rises. We expect doctors to explain their gut feelings, much like we expect computer vision models that detect disease to explain their findings and have a (theoretically sound) estimate of confidence.

SVM's were so popular, pretty much because they had a firm theoretical basis on which they were designed (or "cute math" as deep learners may call it). As Patrick Winston would ask his students (paraphrasing): "Did God really meant it this way, or did humans create it, because it was useful to them?". Except maybe for the LSTM, deep learning models are not God-given. We use them because, in practice, they beat other modeling techniques. Now we need to find the theoretical grounding to explain why they work so well, and allow for better model interpretability, so these models can more readily be deployed in health care and under regulation.

<comment><USER12> The article calls for an explanation not why some ML method works, but for an explanation of a particular ML result, like why a car drives this way or why a patient got a cancer. While I hopeful for the former, I just do not see the basis for the latter.

If some regulations shall require such explanation, the end result will be fake stories like parents tell to the children that Moon do not fall because it is nailed to the sky.

<comment><USER9> I don't think the paper asked for that. Relevant quote:

> machine learning is more concerned with making predictions, even if the > prediction can not be explained very well (a.k.a. "a blackbox prediction")

So in your example: an algo may explain that a car slows down, before taking a turn, because else it would likely crash. It may even get to a threshold ("under these weather conditions, anything over 55Mph is unsafe when taking a turn of such and such degree"). Statistics can help with that.

Welling is not asking for deep learning models to explain why a person got a cancer, but to explain its reasoning when it diagnoses a person with cancer ("I am confident, because in a random population of a 1000 other patients, these variables are within ..."). Statistics can help with that. It aligns with their mind set and tool set.

Regulations are cheated even with these kind of explanations, but that is for another story (black box models may provide some plausible deniability).

<comment><USER12> I am referring to this fragment:

> Thus, for many applications, in order to successfully interact with humans, > machines will need to explain their reasoning, including some quantification > of confidence, to humans.

No doubtful there are cases when an explanation is easy. Often this is because we have a very solid model like physics of a car. In fact since we know the model, we do not need an explanation, we must demand that the algorithm follows the model or declare it unfit.

But how can we expect an explanation for a behavior in a critical situation on a road that was not explicitly programmed and when the algorithm decided to turn to a particular degree bases on a non-trivial inference? Similarly, when an algorithm decides if a patient needs an emergency operation or if they can wait, why can we expect an simple explanation especially for the patient with rare conditions when algorithm again must perform an inference, not a deduction from 1000 very similar cases?

<comment><USER4> Maybe you couldn't always expect an explanation, but having one would certainly be useful. Methods that could get the job done while also providing enough information to explain or at least shed some kind of light on why the solution was chosen would likely be preferable.</comment></comment></comment></comment></comment></comment>


Developing Provably-Correct Software Using Formal Methods - <AUTHOR> http://www.infoq.com/news/2015/05/provably-correct-software <comment><USER2> _some tools such as Verum Dezyne and Quantum Leaps can generate 100% correct source code in C, C++, Java ..._

The input into these tools, from which they generate source code, is therefore, the real program and the real source code. How do they prove that this real source code, from which they generate C,C++, or Java, is "correct"?

In fact, this input just constitutes a new programming language -- a DSL -- and the tool is just another compiler or scripting engine. If existing compilers are unprovable, this new compiler is for the same reasons unprovable too. In other words, they did not solve the problem.

<comment><USER3> Because the code will literally not compile if it isn't correct. As opposed to weaker languages where bugs (run-time problems) can exist.

I'm still something of a layman on this, and I know less about formal specification languages like TLA+, but languages like Coq work by using things like dependent types.

Here's an example: if you have a method that takes (through specifying type parameters) an Int and returns an Array of Integers, and it compiles, then you have effectively _proven_ that it is possible to take an Int and return an Array of Integers.

This is not particularly useful to the method author that is only using that method signature to write some more specific business logic. Dependent types allow you to spruce up that method signature until it actually reflects your business logic. So then you know that if it compiles, the actual intent of your method/function is bug free.

Yes, it is true that there is a "turtles all the way down" element to this kind of thing - "how do they prove that this real source code is correct?" The answer is that languages like Coq rely on a small kernel of code that cannot be proven. It's like their axiom from which everything flows. But the kernel is small enough that people verify it with eyeballs. Bugs can still happen there, and I think they even found one a few years ago, but it didn't have any impact on all the mathematical proofs that have been written using Coq.

I'm sure some experts can jump all over what I just wrote but that's the gist as I understand it so far.

<comment><USER4> Is the Coq bug you're thinking of this one, from earlier this year?

[https://github.com/clarus/falso](https://github.com/clarus/falso)

<comment><USER5> That wasn't really in the core verifier exactly. It was more like an optimization.

</comment></comment></comment></comment><comment><USER6> It's all definitely worth looking into. Anyone trying to start will find Wheeler's page [1] much more helpful. I also included a book [2] on using dependent types for certified programming. There's also a few examples after of verifications with [3] being pretty cutting-edge in automation & thoroughness. I think we need people to put together a single resource where newcomers can see the various approaches, tools, and practical examples of how to use them. Further, we need to research more on which tools give you the most bug hunting benefit with the least time and expertise. There's some research done already but everything in this field is scattered.

Another point came up in a similar discussion: we need to focus most of our energy on raising the baseline of correctness for the average programmer. This includes things such as automated test generation, design by contract, and better type systems. The developer should be able to specify his or her intention with the code while the type system or runtime catches most of the errors. Maybe we just need a version of BASIC/Pascal with Ada-like safety built-in or similar restrictions on certain constructs. Not sure of the specifics but this is an important problem to solve.

[1] [http://www.dwheeler.com/essays/high-assurance- floss.html](http://www.dwheeler.com/essays/high-assurance-floss.html)

[2] [http://adam.chlipala.net/cpdt/](http://adam.chlipala.net/cpdt/)

[3] [https://www.umsec.umn.edu/sites/www.umsec.umn.edu/files/hard...](https://www.umsec.umn.edu/sites/www.umsec.umn.edu/files/hardin- icfem09-proof.pdf)

[4] [http://se.inf.ethz.ch/people/morandi/publications/prototypin...](http://se.inf.ethz.ch/people/morandi/publications/prototyping.pdf)

[5] [http://compcert.inria.fr/compcert-C.html](http://compcert.inria.fr/compcert-C.html)

[6] [http://repository.readscheme.org/ftp/papers/vlisp/guide.pdf](http://repository.readscheme.org/ftp/papers/vlisp/guide.pdf)

[7] [http://research.microsoft.com/pubs/122884/pldi117-yang.pdf](http://research.microsoft.com/pubs/122884/pldi117-yang.pdf)

</comment><comment><USER7> (I'm the author of the original post.) Model-checkers don't prove that a _program_ is correct, only that the logic in the program's abstracted and simplified communications/state _model_ has no inherent race conditions, deadlocks and so forth, i.e. no typical concurrency flaws. Model-checkers don't "prove" correctness in the mathematical sense (and maybe I should have written "verifiably-correct" instead of "provably-correct"). Rather, model- checkers search for counterexamples to the hypothesis "there are no concurrency flaws in this model", by exhaustively tracing through all possible distinct execution paths through the model. Amazon calls their models "exhaustively testable pseudo-code". If no counterexample is found, then we can be sure the model has none of the standard concurrency flaws, and we can be sure that the communications "housekeeping" code auto-generated from the model will have none either. This is a limited yet extremely powerful and valuable aspect of correctness, the hardest kind of correctness for humans to achieve in concurrent software.

I believe software engineers can and should start routinely using model- checking on concurrent logic, taking advantage of the machine's ability to race relentlessly through enormous numbers of execution paths to uncover subtle flaws in logic. Even a small change to concurrent logic can have far- reaching consequences that are very hard for humans to envision. I believe competitiveness and sound engineering practice demand use of model-checkers on concurrent logic.

</comment><comment><USER8> Okay, so the software is now provably correct.. now what were the costs in doing so, both in additional time pre-writing definitions and other abstractions for the software itself? Also, how does one prove that said software actually meets the business needs behind said software?

In an academic environment, or something tied to physical devices, this approach isn't a bad one at all... These are settings where software is much more akin to an engineering discipline than it is a craft.

In _most_ software development, which are in any number of business environments building/adapting or bridging custom software that will run in any number of environments... the additional time that this will take over current approaches will be simply unacceptable. Most people don't care if a software is provably correct.. and for that matter, most bugs aren't critical in nature, and don't lead to end of the world scenarios... since most systems don't actually touch personal medical information or financial data.

I think on an intellectual level that approaches like this for some systems make a lot of sense... The problem is rarely do you have well defined interfaces/algorithms for well defined problems... so, you build your system with provably correct software, then the proverbial moose's neck breaks, because you didn't know about the moose.

<comment><USER7> In the context of physical systems, companies in the Netherlands including ASML, FEI, Philips Healthcare and Ericsson, have found in the past 10 years that applying model-checking to concurrency logic in control systems _reduces_ both costs and time-to-market while increasing reliability of the built software. Why reductions? Because concurrency bugs are removed at the earliest possible moment, hence they cost orders of magnitude less than if they were found during testing or (worse) at customer sites.</comment></comment>


When is a nation not a nation? Somaliland's dream of independence - <AUTHOR> https://www.theguardian.com/news/2018/jul/20/when-is-a-nation-not-a-nation-somalilands-dream-of-independence <comment><USER2> The blue Somali flag has a star that with each of the points of the star representating a Somali inhabited region: French Somaliland which is Djibouti; British Somaliland, which is the focus of the article; Italian Somaliland, the Somalia with Mogadishu as its capital; the Ogaden in Ethiopia and Northern Frontier District NFD in Kenya.I am an ethnic Somali from NFD though the name has since changed. Last time it was called North Eastern Province before Kenya abolished the provincial system. The Ogaden as a name is also frowned upon. Ironically, I am also from the Ogadeni clan. Ethiopian Somalis, and Ethiopians in general, prefer calling the same as Somali Regional State.

Anyway, Somalia, before the union and merger of the British and Italian Somalilands, claimed sovereignty over all these other Somalilands. This dream was known as Greater Somalia. But, the same is know probably dead. The collapse of the Somali government didn't help matters.

Interestingly, rather than Somalis ultimately uniting in one country, after the war, the numerous other Somalilands came to the aid of the fleeing Somali regions.

My home county in Kenya has been host to one of the largest refugee camps in the world - Dadaab. Well, that was before Syria happened.

Somaliland, the one in the article, seems to be functioning much better than Somalia - an entity it is tied to in a marriage that wasn't supposed to work from the beginning. Somaliland should have negotiated for a better deal before unconditionally joining Somalia.

Also, the Africa Union is probably not going to recognise Somaliland as an independent state. This, for fear of opening a Pandora's box. There are many other African communities not happy of being part of the countries they have been forced to be a part of.

Anyway, I do support the independence of Somaliland though the border issues would need to be sorted out. Some clans are happy with being in Somalia. I support their independence because of the potential for their prosperity. They are currently being forced for Somalia to sort out its issues which I think is unfair.

<comment><USER3> Very interesting comment, thanks.

 _Also, the Africa Union is probably not going to recognise Somaliland as an independent state. This, for fear of opening a Pandora 's box. There are many other African communities not happy of being part of the countries they have been forced to be a part of._

This kind of surprises me. I've sometimes thought it might be in Africa's long term interest to have this happen. The current countries were created arbitrarily by Europeans based on what they wanted to control, without regard for the underlying peoples with their languages, cultures, and histories. Perhaps a rethinking of borders would be positive in the long run.

<comment><USER4> Ceding territory and authority doesn't benefit the constituents of the African Union, which are governments of states. Is it actually that surprising?

<comment><USER5> Prolonged guerilla insurgencies etc typically don't benefit the constituents either (unless you take the viewpoint of them being the crooked type).

<comment><USER6> Pre-colonial borders weren't all static and stable. Redrawing the borders to the status quo ante wouldn't magically erase border conflict.

Even if the status quo ante was stable and conflict-free, there have been hundreds of intervening years of shifting alliances.

</comment></comment></comment></comment></comment><comment><USER7> Reminds me of the confusing situation of Québec, Canada:

"Our position is clear. Do the Québécois form a nation within Canada? The answer is yes. Do the Québécois form an independent nation? The answer is no and the answer will always be no," Harper told the House on Wednesday.

[https://www.cbc.ca/news/canada/quebecers-form-a-nation- withi...](https://www.cbc.ca/news/canada/quebecers-form-a-nation-within- canada-pm-1.624141)

<comment><USER8> Countries within nation states are not uncommon. For example, the Kingdom of The Netherlands consists of multiple countries: The Netherlands, Curacao, Sint Maarten and Aruba. The same can be said for example for the United Kingdom.

<comment><USER9> > Countries within nation states are not uncommon.

If it has more than one distinct nation within it, it is a multinational state, not a nation-state. Nation-states have been the dominant (but not universal!) norm for so long that people use "nation-state" when they just mean "state".

<comment><USER10> Unfortunately, "state" can't be used unambiguously to mean a sovereign political entity, because there are so many states that aren't sovereign (e.g. New York). I would guess "nation-state" has acquired it's current popular meaning because it is one of the few terms that unambiguously implies sovereignty, which is what the speaker is usually trying to emphasize.

<comment><USER11> When the United States were created, they were intended to be sovereign (Articles of Confederation)

<comment><USER10> Sure, that's how they were more or less envisioned. But they are not, leaving us with at least two meanings for the term "state".

<comment><USER12> The US isn't the only place to use "states". Austria, Germany, Argentina, Mexico, etc are all based around states.

There's an important distinction between "states" and "provinces" that people tend to forget. Provinces are subdivisions of a larger entity, usually a unitary State, and inherit their powers from it. States, on the other hand, grant a central arbiter powers via cession; almost exclusively in the form of a federation or confederation.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> Fairly safe compared to its neighbor in the South. I know a lot of the Somali refugees in Minnesota who have gotten back on their feet financially have been investing in Somaliland and some are moving back.

<comment> barry-cotter You could say the same regarding investing and moving back about Mogadishu, where the UN puppet government has its capital and it is not safe. People just have real affection for their home.

<comment><USER14> Except if you read the article you would see it's a fairly safe place, compared to the rest of Somalia.

<comment> barry-cotter If all you know about Somaliland comes from reading that article look at the Wikipedia article on Somaliland. More detail, less colour. Ditto for Puntland, which is also safer than any region settled by ethnic Somalis bar Dijibouti.

Mogadishu is safe enough that one of my friends spent two months there trying to find business opportunities and her most memorable story was of an argument with a "government" soldier. It's safer than Congo.

<comment><USER15> > It's safer than Congo.

Uh ... OK

</comment></comment></comment></comment></comment><comment><USER16> Fun fact: Somaliland recognizes Liberland[1] as a country.

[1] [https://en.m.wikipedia.org/wiki/Liberland](https://en.m.wikipedia.org/wiki/Liberland)

<comment><USER17> Fun fact extension: most unrecognized countries recognize each other. One of my favourite moments in Tiraspol (Transnistria, de facto independent of Moldova) was finding the joint embassy of South Ossetia and Abkhazia.

<comment><USER18> All these are simply russian puppet quasi-states, so the "embassies" there are no wonder

</comment></comment></comment><comment><USER19> I was discussing this with my Catalan friend, that from the outside world their declaration of independence looks comical, because Catalonia doesn't act like a real country in any shape or form, unlike Somaliland here.

Now I get it that Catalonia is inside a powerful state and you can't just do "country things", but still.

<comment><USER20> Can you elaborate a little what you mean by "doesn't act like a real country"? My understanding is that provinces in Spain actually enjoy quite a lot of autonomy in regards to how they organize their operations/policies. Catalonia promotes its own language in schools, they even have some form of foreign affairs with other governments. What do you think is missing?

And just to be clear, I also thought the declaration of independence was really comical :)

<comment><USER21> My understanding is that it's a bit like Texas or California or New York dreaming that they could be their own separate countries. If you live in one of these places it's easy to put on rose colored glasses and dream. But are they really going to put up a border wall, close off economically, form their own military, _et al, et al?_

Their prosperity is based in large part on integration into the union. They will always be dependent on it in some form, even if they achieved political independence.

Unlike Somaliland, which really has no use for it's neighbor and the realities on the ground reflect that.

<comment><USER22> As a determined supporter of Scottish independence, I obviously disagree with that assessment :)

Independence doesn't mean that countries become completely detached from their neighbours. In Catalonia for example the assumption is that both the new country and the remainder of Spain would remain members of the EU and the associated intergovernmental machinery that brings. That means no border guards or economic walls!

In the Scottish case, for example, it's worth bearing in mind that as a country it already has an entirely separate government, legal system, healthcare and education systems. Is it such a big leap to imagine that a separate foreign policy and military organisation might be feasible?

<comment><USER19> But what do you think would have happened if Scotland declared independence against the will of the UK government? Unless you were willing to fight for it, I don't see how the outcome would have been different than the Catalan one.

<comment><USER23> I think you're really underestimating how tenuous the UK's claim on Scotland really is. Frankly, there is no model in which the UK deploys military assets to forcefully retain Scotland. This would almost certainly delegitamise the entire "Great Britain" concept.

Catalonia is different because it, frankly, was never really an independent nation. And while Spain recognizes a certain autonomy it does not grant anything like an independent legal system.

That said the case of Catalonia is interesting. There's nothing quite like it in France. France has never even entertained autonomy, instead emphasizing at every opportunity the "supremacy" of the Parisian government. There's probably a lesson to be learned from Spain and the UK which may have unwittingly given "hope" to these separatist movements.

<comment><USER19> I don't know much about UK politics, but I thought UK did use the military to prevent Northern Ireland from splitting off.

But I agree that in $CURRENT_YEAR military intervention against Scotland would be highly unpopular even in England.

<comment><USER24> > I thought UK did use the military to prevent Northern Ireland from splitting > off.

You may be thinking of the UK military being in Northern Ireland to try to keep the peace from a population that were attempting to kill each other in the 70s?

However if you're talking about the 1916-1922 period:

Some people in Dublin declared independence in the "Easter Uprising" (in the middle of a war) over 100 years ago, it was dealt with by the UK military. The second attempt in 1918 was far better -- there was a massive democratic will for independence expressed through the ballot box. The UK government (100 years ago) resisted this independence initially, which was wrong, just as wrong as Spain resiting Catalan independence.

Eventually the UK government relented, and granted independence to Ireland after free and democratic elections in 1921, when 98% of the voters in what's now called "Ireland" voted for the pro-indepedence Sinn Fein (it was unopposed in almost every constituency)

However in the north, many residents did not want to become independent from the UK. The election there returned 67% in favour of the unionist party. As such independence was offered to the entire island, but the north could opt out by having it's parliament ask to (through it's democratically elected parliament). They did - the day after the Irish Free state came into being.

So people who lived in each area got to choose where they wanted to be, and all was well. That's how it should be for Scotland, and Catalonia.

The messy business of unpicking a country began (including a brief 'civil' war), but by 1937 the Irish government in Dublin ratified a constitution. This constitution claimed sovereignty over the north as well, despite the population in the North not wanting that, which wasn't exactly a good way to make friends and influence people.

It was only in 1998 that Dublin agreed to remove this claim from their constitution, and agree that whether the north should be moved from Britain to Ireland was a matter for people who live in the north (and indeed one for those in the south, who would have to agree to take them)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER25> Somaliland is a customer of mine; I build national software and Somaliland is my second oldest customer. It's cool to hear about their continued success on Hacker News. I like to think I played a small part in it. I hope that one day they achieve their dream of independence.

<comment><USER26> How can I reach you to talk about this? From the area

</comment></comment><comment><USER27> I had literally no idea that Somaliland existed, and I consider myself to be pretty up to date with the political world map. It's fascinating to read about a country you knew nothing of, even an unrecognised one. No, _especially_ an unrecognised one

<comment><USER28> I was fascinated to learn about Western Sahara when I drove through.

It was at one point a fully-fledged country, now it's listed by the UN as an "occupied territory". i.e. Morocco invaded and have taken over. They're adamant it's simply Morocco, and they get _really_ mean to anyone that says otherwise, or even if you have a map that shows it as a separate country.

Driving through it's just like you never left Morocco, except there are military roadblocks every 100km, there is a lot of new construction, and gas is noticeably cheaper than the rest of Morocco (i.e. Morocco attempting to spur development and help "claim" it)

It's fascinating to me a country can be invaded and occupied by their neighbor and the world does nothing about it.

<comment><USER29> > a country can be invaded and occupied by their neighbor and the world does > nothing about it.

Western Sahara was never a sovereign, recognized state. Spain relinquished it's colonial claim over it and Morocco and Mauritania rushed to claim parts of the territory (Mauritania has since renounced too).

The closest thing to a nationally representative organization was the Polisario Front, a military-political structure based on ethnic Sahrawi affiliation. It didn't help that the area is so sparsely populated and could barely sustain an independent state in the 70s, let alone a state capable of withstanding the territorial advances of such a powerful neighbor like Morocco.

<comment><USER30> > Spain relinquished it's colonial claim over it and Morocco and Mauritania > rushed to claim parts of the territory

Is this a case where the colonial power should have been more firm when drawing arbitrary lines in the sand? The opposite of what they usually get criticised for.

</comment></comment></comment></comment><comment><USER31> In political theory and International Relations studies, there is a reasonably well established set of theories regarding the difference between "A nation", "A state" and a "Nation-State" the three are not synonymous and it irks me somewhat how even in an article like this the three seem to be conflated as I think it adds t everyone's confusion about things.

Having said that, there is no "hegemon" (arbiter/judge/police/enforcer-of- rules) at international level so what is, or is not, a state is somewhat arbitrary and contingent on whatever power structures are at play at the time.

</comment><comment><USER32> Perhaps allow a major Western military power to come and establish a huge permanent base, and pray for some natural resources to be found might shorten the long road to independence.

That and paying huge wads of cash to celebrity-turned-activists ($$$ not Somaliland cash bricks!!) to come and create heartbleed documentaries could potentially help their cause.

Just sayin...

</comment><comment><USER33> If other countries recognize your independence you are there. The US gained recognition from the Netherlands and France day one. Nobody gives a shit about Somaliland nor does anyone gain anything for supporting it.

</comment><comment><USER34> When no country important recognizes you as a country? eg. Taiwan

<comment><USER35> Taiwan (ROC) is a bit of an odd duck because so many places treat them as different but don't do it 'officially' because China demands that they don't as part of maintaining relations with the PRC. For example the US sells them arms after all, it's not like the US completely refuses to admit they exist. Many countries have quasi-embassies in the form of Taipei Economic and Cultural Representative Offices (TECRO) which are definitely totally not embassies just act like them in pretty much every way.</comment></comment>


Why I stopped contributing to Vundle - <AUTHOR> http://gmarik.info/blog/2014/02/04/why-i-stopped-contributing-to-vundle

<comment><USER2> Was really hoping for "I switched to emacs".

Interesting the effects of unapproved "copies" on the OSS community though.</comment>


Average .NET developer salary in U.S.: $92,000 - <AUTHOR> http://visualstudiomagazine.com/articles/2012/01/01/developer-salary-survey.aspx

<comment><USER2> This seems low for me for .NET. But then I live in the bay area, where everything is way too expensive.

</comment><comment><USER3> median!

</comment><comment><USER4> I wonder how this compares to the average salary for Ruby or node.js devs?</comment>


C++/Python producer consumer question - <AUTHOR>

Anyone here have experience storing incoming data to ram as files (through C++) and accessing said files through python?  Trying to get a gauge on whether I can get away with processing the files without committing them to disk (which is may be too slow).  Native C++ processing is possible but I hate making/testing changes in that language. <comment><USER2> not sure I understand your question fully, but python does have support for memory-mapped files.

>>> import mmap >>> help(mmap)</comment>


Dell hints it may follow Google out of China  - <AUTHOR> http://www.itpro.co.uk/621826/dell-hints-it-may-follow-google-out-of-china

<comment><USER2> The only problem with Google and GoDaddy's movement out of China is that we're tipping the economic scale in the wrong way. We're taking away the products and services that make us money from the Chinese, but what we ought to be doing is taking away the things that give them money, such as manufacturing. Dell definitely has it right by pulling in the other direction. The trade gap is enormous and we need to fix that.

<comment><USER3> China may make the most money by doing the manufacturing, but _we_ also make the most money by letting China do the manufacturing. This is a basic tenet of economics called _Comparative Advantage_ (<http://en.wikipedia.org/wiki/Comparative_advantage> ).

(I note that in the presence of market-distorting factors like tariffs, comparative advantage becomes much muddier)

</comment></comment><comment><USER4> It's moving to India!

That's a freer country, right? If not, please enlighten me.</comment>


Ask HN: Why don't tech companies collaborate to fix the H1B and Startup Visa? - <AUTHOR> The big tech companies and organizations(including YC) have the legal,financial and political resources to bring about changes to the current H1B visa situation(which prevents the smartest minds from abroad entering the USA), and the Startup Visa which forever seems lost in the political process.<p>Why is it that even though all these companies and organizations have a common unambiguous goal, they are not actively working together to fix the mentioned issues? Why can't they form some sort of platform to collaborate on these issues to speed up the introduction of the proposed legislation? <comment><USER2> I'd say that they have in fact been collaborating on this, and reform failure shows that all in all, they haven't got that much political weight.</comment>


Lyft invests $100M in its drivers - <AUTHOR> https://techcrunch.com/2018/05/23/lyft-invests-100-million-in-its-drivers/ <comment><USER2> Lyft is still outsourcing risk to those drivers.

Lyft is certainly well aware of how much money it saves by ensuring that when expensive tragedies happen, they are borne by individuals. As to how well the drivers are prepared, well it's going to vary by individual, but it's safe to say that _none_ of them understand the risks they are taking on as well as Lyft does, and in the aggregate most are unprepared, not even having proper rideshare insurance.

<comment><USER3> This was my first thought as well - it seems like a thought out risk mitigation strategy that plays well from a PR perspective.

It's a savvy move, and to your point likely escapes the typical drivers awareness. So at what point does regulation get in the mix as Lyft or other ride sharing companies take direct action that signals risk acknowledgement or indemnification?

</comment></comment><comment><USER4> Alternative headline: "Lyft invests $100 million in self-driving car maintenance hubs." By centralizing cleaning and routine maintenance to it's centers Lyft is laying the ground work to provide a place for self-driving vehicles to return to between revenue service.

<comment><USER5> People keep saying the Lyft and Uber endgame is self driving cars, but there are not going to be fully autonomous self driving cars for at least 10 years. It's not a realistic business plan and I very much doubt that's actually the strategy at these companies.

<comment><USER6> Where does the number "10 years" come from? 5 seems plausible to this non- domain expert

<comment><USER7> 10 is optimistic IMHO. The popular 80/20 rule is at work here. It's going to take a lot of time to get the rest 20% right. For example, how long it's going to take to solve the driving on the snowy roads?

<comment><USER8> I, for one, have never taken a Lyft from San Francisco to Tahoe.

"On the roads" doesn't mean "Every single road in every single market around the globe"

</comment></comment></comment></comment></comment><comment><USER9> The headline could stand alone as a succinct master class in the value of the "relations" part of PR. Imagine Uber getting a headline like this from TC in 2018, _assuming it did exactly the same substantive thing._

</comment><comment><USER10> 1.4M drivers now, they are doubling to 2.8M over 5 years.

100M / 2.8M = $35 each. Or about one oil change.

</comment><comment><USER11> Interesting the last bit, this will technically subsidize Uber's work. Most of the drivers are on both unless they add a minimum driven hours per week to get this perk.

</comment><comment><USER12> This is no better than a press release. How does this absolute crap make it to the HN front page? Is it hatred of Uber? I don't get why such a fluff piece can get so many upvotes.

Even the title "Uber invests in its drivers" sounds like a press release.

</comment><comment><USER13> The only item in this that strikes me as "investing in its drivers" is the tax education. Everything else would be better described as cost subsidies intended to incentivize drivers to continue to work for Lyft as independent contractors.

</comment><comment><USER14> This sounds like yet more cost on an unsustainable business model. Just like Uber announced today (iirc) that the are paying a bunch more quasi-benefits to their drivers in the EU.</comment>


Cryptocurrency - <AUTHOR> Hi,cryptocurrency is going to dissapear or must be the money of future? <comment><USER2> depends.

Crypto is the biggest video game ever existed.

<comment><AUTHOR> Jajaja video Game? Yes maybe You are right, i pass everyday to get crypto...</comment></comment>


When your IP traffic in AWS disappears into a black hole - <AUTHOR> http://engineering.clever.com/2014/12/10/when-your-ip-traffic-in-aws-disappears-into-a-black-hole/

<comment><USER2> We saw this earlier this year after upgrading to a new Linux kernel.

The solution for us was to set this in sysctl.conf:

net.ipv4.neigh.default.gc_thresh1=0

[https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150...](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150/) [https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150...](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150/comments/12)

</comment><comment><USER3> Am I weird because I actually muttered 'ARP caching issue' halfway through your article? :-)

Love the technical write-up -- thanks!

<comment><AUTHOR> I don't think you're weird, I think you're probably a great candidate to work as a sysadmin / SRE, or at Clever, we're hiring: [https://clever.com/about/jobs#engineer-distributed- systems](https://clever.com/about/jobs#engineer-distributed-systems) !

<comment><USER4> Your job descriptions list a lot of technologies but dont mention anything to do with operating systems or networking. It seems odd to explicitly call out Node and redis but not anything down the stack.

<comment><AUTHOR> It's a good point, but right now we don't do anything that relies specifically on one technology at the very low levels (for instance, kernel modules or very specific networking optimizations we _don 't_ need and would be over- optimization).

Instead we try to maintain a SOA which is agnostic to what underlying tools back our services, something that we've found Docker is useful for.

That being said, we generally just use Ubuntu, and have some other AWS- specific setups around networking. If you like hacking on Cisco routers, I think it's perhaps not the right position :)

</comment></comment></comment></comment><comment><USER5> Interesting trace down to stale ARP entries. It gets worse when the switches are running mac address filtering and _they_ get out of date. We had that issue with some Blade G8052 top of rack switches with their upstream 10G ports. They sometimes "forget" which upstream port has the MAC address that they are switching too, and those packets just spew out messily into the data center leaving a mess. The "fix" it to force the switch to ping up through a specific upstream port periodically to the center switch's management IP address. Sigh.

<comment><USER6> I've seen the exact same issue in some Dell PowerConnect switches.

We only saw the issue when using the "iSCSI Auto-Config" mode. Manually configuring the switches with the same config but entered by hand resolved the issue.

</comment></comment><comment><USER7> This reminds me of a time at a previous company years ago, where we experienced an issue that felt similar, although the root cause was quite different.

Basically, we had multiple teams all launching/terminating web servers. Unfortunately, they were all in the same EC2 deployment, and more often than not our load balancers from one team would send traffic to the web servers of another team. Furthermore, our setups were similar enough that this would sometimes cause bad results for users. We fixed it by making sure that our web servers on every team spoke on different ports. Not elegant, but effective (until two teams accidentally picked the same ports).

These days we have good enough infrastructure tools that this problem should never happen. But in 2009, at a company that was overwhelmed with growth, those sort of things happen.

<comment><USER8> Would a better fix have been to have a load balancer per team, and then have the EC2 instance bind to the correct load balancer?

<comment><USER7> That's what we had, but because the IPs were often recycled, traffic sometimes went to the wrong place if the load balancers has stale data.

</comment></comment></comment><comment><USER9> Try an arping from the new workers on first startup? Ran into this quite a bit when using VIPs for DB failover, and an arping fixed the caching issue in most cases.

</comment><comment><AUTHOR> I think the biggest thing I was surprised by with this investigation was the lack of documentation about data-layer tools. At one point I was looking through the source of the 'ip' command to try to find out exactly which conditions caused a 'STALE' entry in the ARP table...

</comment><comment><USER10> I wonder if this is a problem for any cloud provider, I also wonder if ipv6 could help mitigate? Then the IP collusions would be rarer.

<comment><AUTHOR> I think you are definitely right that IPv6 would mitigate, but I think that the real way to avoid this issue is for the kernel to make the tradeoff for you - it's so cheap to do an arp ping, you might as well on maybe every 100th packet or so. Or similar: timing out and clearing the arp entries more frequently.

As for other cloud providers, I'd be really interested to hear from Digital Ocean, Rackspace, and Google!

<comment><USER11> Answered parallel to you in this thread: [https://news.ycombinator.com/item?id=8732356](https://news.ycombinator.com/item?id=8732356)

</comment></comment></comment><comment><USER12> FYI, Clever: I click "Engineering Blog" at the top, and all links to blog posts on that page 404.

<comment><USER13> sorry about that. It's fixed now!

</comment></comment><comment><USER14> We had a fascinating bug on EC2 -- we could connect _to_ the instance, but no network traffic made it out. It wasn't security group problems, it was literally a really weird bug in EC2's network that we somehow triggered, the engineer over at Amazon that looked at it was really excited when he came across our case as it was so weird, heh. They fixed it, I can't remember exactly what was done on their end, but it was one of the weirder problems I've attempted to debug. Nothing I tried worked!

</comment><comment><USER15> Wouldn't it be a better solution to not reuse IP addresses quickly? If I understood it correctly, they are in a private network anyway, so they could afford it.

<comment><USER16> I thought there was a mechanism that would allow some sort of failure notification packet to make it back to the origin, so that it would then perform a proper ARP lookup _WHO HAS 22.214.171.124? TELL 126.96.36.199!_.

Perhaps AWS is blocking some class of packet?

<comment> X-Istence The issue is that if it is on-network the only real "failure" notification is a time-out. No ICMP packets, nothing are sent back to indicate you are sending to a non-existent IP address.

</comment></comment></comment><comment><USER17> I just experienced this early this week. Very frustrating. I also posted to AWS forums and got zero assistance; am currently not paying for AWS support plan. This article came at an opportune moment -- it makes sense and removes the shroud of mystery around why it "works sometimes" which leaves me with an uneasy feeling for a production setup.

<comment><AUTHOR> awesome, I was hoping it could help out someone else :)

</comment></comment><comment><USER18> Looking at the port number, it looks like Clever is a MongoDB user =)

<comment><USER19> You, uh, don't even have to just recognize the port number, reading the article is enough...

> [...] These workers all connect to a MongoDB replicaset to update and read > data [...] > 27017 = port mongod is listening on

</comment></comment><comment><USER20> I was going to respond with my little story, but I see your article already linked it! ;)</comment>


Google upgrades physical authentication technology for iPhone and iPad - <AUTHOR> https://www.imore.com/google-upgrades-physical-authentication-technology-iphone-and-ipad <comment><USER2> Great! Now everyone who already has an NFC U2F/WebAuthn key (e.g. YubiKey with NFC) can use it for the "advanced protection" program (which increases security on your Google account by enforcing 2FA, among other things). Before this you needed to get a separate Bluetooth dongle that at least I have not found other uses for.</comment>


Apache Hadoop: Best Practices and Anti-Patterns - <AUTHOR> http://developer.yahoo.net/blogs/hadoop/2010/08/apache_hadoop_best_practices_a.html

<comment><USER2> Interesting collection. Unfortunately, as per my experience, these aren't that useful -- they pretty much come down to understanding how hadoop works, making sure mappers and reducers have reasonable work and jobs are balanced, etc. Essentially things you would easily see from the jobtracker page. Others stem from defects in hadoop design -- why should users have to care about file compression? Why should multiple output files from reducers be a problem? What idiot decided to make the bloody jobtracker keep all counters and statistics IN RAM (I've heard of these things called databases that are useful for offline storage of structured data)... For the interested, we actually bounce the jobtrackers on bigger lanes daily to increase reliability and avoid long gc times with tons of accumulated in-ram stats.

More interesting would be how to solve hard problems -- one that took me quite a while was how to do a distributed balanced exact rank, particularly when the field being ranked is not uniformly distributed so you can't simply partition it.</comment>


Visual Studio Code November 2017 - <AUTHOR> https://code.visualstudio.com/updates/v1_19 <comment><USER2> Glad to see the continued focus on performance. The last few updates have had noticeably faster startup times IMO.</comment>