The STEM Crisis Is a Myth (2013) - <AUTHOR> http://spectrum.ieee.org/at-work/education/the-stem-crisis-is-a-myth <comment><USER2> There is not a shortage of STEM graduates. However, from an employer's point of view, there is a shortage of cheap STEM graduates. Hence the recent tech company uproar over the skilled visa changes.

<comment><USER3> Those are the same thing. You may as well say that there is no shortage of food in Venezuela, just a shortage of cheap food.

The reason companies can't get STEM grads at the prices they're willing to pay is because those STEM grads can simply work for a company willing to pay more. That's not a problem for the grads (you bet your ass I'm not complaining too much, haha), but it is for everyone who wants to hire them.

<comment><USER4> It's a problem with a pretty clear and easy solution: Pay more.

</comment></comment></comment><comment><USER5> In relation to programming specifically, I once saw someone on Quora put it perfectly. It effecticly stated that we don't need more programmers. We need more senior developers with 10+ years of experience.

<comment><USER6> Chimes with anecdotal experience hiring in London. Glut of really well educated, keen and passionate juniors but tumbleweed when trying to hire seniors, even at higher salaries.

<comment><USER7> Have you tried offering internationally competitive salaries?

<comment><USER8> Then there is also the issue of the ease or difficulty of getting someone from a foreign country to work in the UK. Are work permits required? How much paperwork? etc

If I were an American, what would it take to work in London?

<comment><USER9> > If I were an American, what would it take to work in London?

A good internet connection and some perseverance in getting remote jobs. Also strong work habits.

I have many friends back home in Slovenia who stay in Slovenia but work for San Francisco. Sure they don't make 100k, but they don't have to. At 80k they're socking away 40k+ per year. And the SF startup is super stoked because they get good engineers for peanuts.

</comment></comment></comment></comment></comment><comment><USER10> Lumping 'IT' in with STEM is a historical mistake from the days when IT was not a mass-market phenomenon, but firmly the territory of academia, government ventures, defense contractors, safety-critical systems, and research.

Today its placement increasingly serves to muddy facts and justify various views and policies on H1-Bs and the like, either in favor or against. In truth, IT should be split into its own category with distinct analyses and approaches, addressed independently from S, E, M, and the leftovers of T.

</comment><comment><USER11> Even as an immigrant to the US, I agree with this. A large part of the shortage is self-inflicted.

Did we see a shortage of American workers who could do the job right away?

YES. Not many people work in the same area of technology. There really aren't many people working in some areas of tech.

Did the employer try to compromise on their requirement of "rockstar coder with 1000 wpm typing speed, PhD in app programming, merge k-sorted arrays in 15 mins with good syntax and no compiler?"

NO. And this is why there is no shortage. It's their demand for rockstars when reality is most companies are mediocre (at best). It's not shortage, your company sucks, interviews suck and nobody wants to work there.

<comment><USER12> > Did the employer try to compromise on their requirement of "rockstar coder > with 1000 wpm typing speed, PhD in app programming, merge k-sorted arrays in > 15 mins with good syntax and no compiler?"

Or when they want years of experience with a specific framework, library, or other system.

I remember looking at Java positions last year (before I got my current job), and a lot of them would sound nice until I got to the item about "Must have X years of experience with Spring".

Yeah, I have a whole bunch of core Java under my belt, and I made the requirements for everything else, but because I don't have experience with that _one_ library, I'm shut out.

<comment><USER13> In my experience, almost all of those "must have X years of experience with Y" requirements are written up by HR people who are loosely interpreting poorly understood requirements told to them by either the hiring manager, or other HR people. In many or most cases, the people writing up those job postings are not the same people who would be interviewing or hiring you, and they're more aspirational requirements than anything else. They hope that someone with X years of experience with Spring will magically come out of the woodwork and send them a resume, but they're just as likely to hire a fresh grad who doesn't know anything about Spring other than what they skimmed on Wikipedia right before going to the interview. And the people doing the interviewing and hiring might actually be fine with that.

Basically if something looks perfect except for some random "must have X years of experience in Y" requirement, just apply anyway. Most likely it won't be a dealbreaker for them (the hiring manager at a decent-sized company may not even know that that was in the job posting), but the worst case scenario is that you won't get the job but will get some good interview experience.

<comment><USER4> Why would I want to apply to a company that has demonstrated they don't know much about technology?

<comment><USER14> Because (taking the comment in its context) you need to get a _first job_ so you haven't so many options and you have nothing to lose. If you're some genius maybe Google had already recruited you right from uni.

</comment></comment></comment></comment></comment><comment><USER15> From my experience job searching, it seems that companies are looking for someone to fit the job description exactly instead of being willing to hire and to train someone. And when they can't find anyone, they complain about a shortage of eligible workers.

It's really difficult to find a job the traditional way (submit resume/cover letter, interview, etc.) in this economy. The article makes a good point that if there was a shortage of STEM talent, companies would be drooling over everyone with a B.S. degree, but I haven't really seen that.

I do agree with the article that there is a STEM literacy problem...I have friends in grad school constantly complaining about people gaming the peer review system, students getting by on shoddy research, etc. I've had co- workers that I've been really disappointed with in terms of having solid engineering fundamentals.

If the article is right, and there's not a STEM shortage, what would you tell a high schooler applying to college? What field should they study so that there's a better chance of them having jobs?

</comment><comment><USER16> On a related topic, most of the unfilled "STEM" jobs can't be filled by pushing college students into STEM fields.

Usually a scientist, programmer, engineer, or mathematician is made in elementary school.

<comment><USER17> The approach Harvey Mudd took to get more women into Computer Science could (and should) probably be applied across all STEM fields, and to all student types.

[https://qz.com/730290/harvey-mudd-college-took-on-gender- bia...](https://qz.com/730290/harvey-mudd-college-took-on-gender-bias-and-now- more-than-half-its-computer-science-majors-are-women/)

 _The overhaul of the computer science curriculum to make it more inclusive began the year before Klawe arrived from Princeton, where she was the dean of engineering. The school had redesigned its introductory course, required for all first-year students, to emphasize practical uses for programming and team based-projects, and switched from the Java programming language to Python, which more closely mimics the way humans communicate. The course is broken into three tracks, including one for students with no programming background.

To make everyone feel at ease, professors urge know-it-all students who always have their hand in the air to talk during office hours, instead of in class."Too often, people with experience are taking up all the air time," Klawe says.

As a result of the changes, women who take the introductory course are more likely to leave with a positive impression of programming, and often sign up for the second class in the sequence. Many go on to internships or research projects in the field after their first year, and by then, they're hooked._

<comment><USER18> >Many go on to internships or research projects in the field after their first year, and by then, they're hooked. //

Are these the same people who are leaving the industry after a few years? Seems like if you hoodwink people in to something that's not really their vocation then that's the sort of effect you'd expect?

<comment><USER19> I don't think people really have "vocations" built-in at some biological or personal level. Lots of people train for one thing, retrain for another, and work in a third.

<comment><USER18> I've the feeling it's more common to move industries nowadays but some stats that show "lots" of people move between sectors would be good support for your assertion.

Anyone know of pertinent info?

</comment></comment></comment></comment></comment><comment><USER20> I think it's likely that both sides of the argument are correct. There are a large amount of STEM jobs that are not being filled, while at the same time there is a surplus of STEM graduates.

The takeaway is that businesses don't just need people that have STEM degrees, they need people that will add value to their company. Just because you have a STEM degree does not mean that you'll be a net positive for the company.

It seems to me that incentives for colleges right now are way out of wack. They're not doing a good enough job at preparing students for actual industry roles (elite institutions being the exception), yet their enrollment rates and the cost of their tuition keeps going up.

<comment><USER21> > The takeaway is that businesses don't just need people that have STEM > degrees, they need people that will add value to their company. Just because > you have a STEM degree does not mean that you'll be a net positive for the > company.

I agree. I graduated a long time ago with a CS degree and almost a minor in business. The CS foundation helped with all the learning I have done since then, but if I think about the most valuable classes I had in college they were economics, accounting, finance and management. Of course some jobs need PhD level CS without any other skill, but the majority of positions need someone who knows how to program plus <insert skill here>.

</comment></comment><comment><USER22> So is the engineering crisis and the talent shortage and the diversity gap. You think Facebook wants more H1Bs because it wants to promote diversity? You think it's because of a shortage? That's not how these things work. It just so happens that what's currently en vogue aligns with the business interests of the largest tech companies in the world.

<comment><USER23> Specifically, Mark Zuckerberg is a co-founder of FWD.us which "is mobilizing the tech community in support of policies that keep the American Dream achievable in the 21st century, starting with commonsense immigration reform" according to their website [1].

The major issues they lobby for are (1) increase in H-1Bs (2) illegal immigrants. The one thing they _don 't_ talk about is legal immigration paths for people on H-1Bs. People on H-1B from India and China have long backlogs for Green Cards, which causes them to stick with one employer for long, with reduced negotiation leverage, and there by depressing wages for the entire industry. But FWD.us does not have a care for that aspect of immigration - presumably because this helps the bottom line of everyone involved in FWD.us. Meanwhile they don't miss any opportunity to shed crocodile tears for the plight of DREAMers. Sigh!

[1]: [https://www.fwd.us/](https://www.fwd.us/)

<comment><USER24> Do you know any Indian or Chinese people in the Bay? They switch all the time. Getting your H-1B transferred isn't hard.

<comment><USER23> I am one.

When you switch jobs on H-1B, your GC application has to be redone from the first step. This is an uncertain process that can get stuck in red tape for indeterminate amounts of time. Your place in line stays intact in many cases though. So, switching jobs is possible, but is no way easy.

While you wait decade+ for your GC, you freedoms are severely restricted as an alien.

<comment><USER24> But once you've got your i140 (which is fairly quick after your PERM application), you're going to hold onto your priority date.

You can switch at low cost at that point. I know so many people who've switched jobs like this and retained their place.

</comment></comment></comment></comment></comment><comment><USER4> Every time someone mentions that they have a hard time finding qualified workers, or saying that qualified people don't apply to their company, they tend to ignore a very important question: Why would those qualified people want to apply at your company in the first place? A lot of people like to say they're paying "market wages", but most qualified people are already making that or more.

<comment><USER25> Because the money for salaries is not generated out of thin air. It gets determined by the profit-margin provided by the market.

Your competitors are paying market wages in China to "qualified people", or have depreciated the high salaries in employees via higher-productivity and thus pricing you out.

<comment><USER4> "Because the money for salaries is not generated out of thin air. It gets determined by the profit-margin provided by the market."

While this is true, it's also very much not my problem. If your company is offering X, and another company is offering 1.25X, then most of the good people are going to go to the other company first. And it would be quite dishonest of you to say that there's a "shortage" when it turns out you just don't want to pay as much as the competition for your talent.

</comment></comment></comment><comment><USER26> So theoretically, without pricing controls, there's no such thing as a shortage because prices will rise until demand == supply. So the arguments that there's no shortage of STEM workers (only a shortage of below-market- price STEM workers) makes sense.

But let me pitch a different angle: there are a lot of people who could do their job better if they had certain specific STEM skills (e.g. business analysts who were better at statistics, spreadsheets, and maybe SQL).

It's true that you can get these skills if you pay more. However, the cost of hiring someone with these skills might be >= the business value, even though the business value might be >= the cost of the education for these skills.

So I would argue that we're underinvesting in certain STEM skills.

<comment><USER24> Wait a minute, your premise is easily susceptible to a counterexample. Take a 2-employer world with 1 candidate. Let's say both employers have the same desire for the candidate, i.e. no matter what salary one employer offers, the other will offer the same and if one employer believes a salary is too high so will the other.

This system will perpetually be in a shortage. No matter what rate is offered, at least one employer will go without an employee.

<comment><USER26> Wikipedia: "a shortage occurs when for some reason the price does not rise to reach equilibrium"

In your example, the price is at equilibrium, even though one buyer cannot buy at market price. Thus there is no shortage.

If we define shortage as one buyer not being able to buy at market price, then you are technically correct.

<comment><USER24> That's just a problem with Wikipedia being Wikipedia. That's not a definition. If you read the actual source that's used in that section§, you'll see that what he uses as a definition is the following:

> A shortage is a market condition existing at any price where the quantity > supplied is less than the quantity demanded.

He then goes on to talk about how the shortage provides a signal to sellers to move the price closer to equilibrium (upwards, in this case) but that doesn't make it not a shortage.

In questions of definitions, it's also informative to try an alternative source. I propose the website of the Federal Reserve Bank of St. Louis (simply because they have a convenient online glossary): [https://www.stlouisfed.org/education/glossary](https://www.stlouisfed.org/education/glossary)

> Shortage: When the quantity demanded of a good or service exceeds the > quantity supplied at a particular price.

I'm not going to update the Wikipedia page because the Definition section needs quite some rewording and citation to make it authoritative and if I just add this in, it will make that part incoherent.

§ Tucker's _Economics for Today_ , which is also wrongly named in that section as _Economics Today_. In the 7th edition, this is on Page 80

</comment></comment></comment></comment><comment><USER27> SV Tech Companies remind me of contractors who want to build a giant mansion on a hilltop.

But instead of hiring workers with years of experience building houses they hire the best athletes from US university sports programs. Fantastically talented athletes.

Then they give these great athletes a bunch of shovels, cement mixers, hammers and nails and say: "Build the best mansion the world has ever seen."

Somehow it doesn't work out. They get a large house, but it isn't really any better than other large houses. Some of the workers decide to leave.

I wonder if management really knew what it was doing in the first place. Maybe they figured they could just hire "the best" and everything would take care of itself?

</comment><comment><USER28> There is a nasty little issue that is not very well communicated: cheating.

There is a non-zero amount of grads from prestigious to lousy universities and boot-camps that cheated. Some cheated a little bit, some cheated the whole way. But this may be a factor in the STEM issue. Some non-zero amount of prospective hires may have the cred, but the company can identify them quickly as hacks.

I'm going to go out on a long limb here and say that _most_ grads from large research universities cheated their way through and know squat all.

</comment><comment><USER29> I feel like the industry treats software development as a race to the bottom. The goal seems to be to pump out as many STEM grads as possible so they can pay them barista wages.

Are other industries like this? Why do I not see bootcamps for electricians and plumbers?

<comment><USER30> Electricians and plumbers have the same kind of issues.

I have a relative who worked for a company that did the electrical contracting for a huge amount of [huge retail company] stores. Ultimately, [huge retail company] started bringing in large groups of cheap Mexican labor who were mostly inexperienced in the trade. These groups would screw up the job, so they'd bump the opening date a bit and bring in experienced crew to patch everything up.

The end result was a much worse job, but [huge retail company] didn't care because the overall cost was a lot lower.

The places where this does not happen are places where the electrical union is strong and laws make sure workers are certified journeymen/masters (or in an apprenticeship).

I've thought quite often that the world would view programmers much differently if there were a viable programmer's union.

</comment></comment><comment><USER31> This article is over 3 years old, is there something more recent? Not that I discount what it's saying but what made this article relevant again?

</comment><comment><USER32> If you run a small business, it's a real thing. On wages, it's nearly impossible to compete with the big corporations.

That leaves you with one choice - outsource your engineering to China or India. Or they could just come here (wherever here is) and contribute to the local economy by buying stuff, driving down the per capita crime rate (immigrants have a far lower crime rate), etc.

<comment><USER33> This means the business isnt feasible.

<comment><USER34> This is a very important point, let me expand a bit. If your business/product requires a supply of sub-market waged talent, then you have not found true product-market fit or your product just does not provide sufficient value to sustain a real business (real meaning one that can operate on market wages.)

<comment><USER32> Why do you say sub-market? When Disney or any other huge corporation outsources their IT dept to India / China, that's the MARKET. Good Luck putting an import tax on virtual goods / services.

<comment><USER34> It is market wage in India/China. If they can export jobs to India/China and make things work, good for them, that is a good business.

If the company is complaining of a shortage of US workers however, they are paying sub-market US wages. There is no such thing as a shortage in economics.

</comment></comment></comment></comment></comment><comment><USER35> Well, it's a shortage in the sense that there is a shortage of brand-new Ferraris for $15,000.

<comment><USER36> But looking at individual salaries is just not the right idea though: to measure shortages in an industry, we have to look at total supply.

Yes, I can get all the engineers I want if I pay a million dollars a year. But if they are also employed as engineers somewhere else, the lack of engineering talent just moves to those other companies.

Say there is only a car in the entire world. I could buy it for more no more money, but ultimately there is still a shortage, in the sense that even if we had 5 people willing to pY 5 billion for the only car I. The world, they would all not be able to get it. In cars, this is fixed by the fact that we can build cars easily: It doesn't take that long to make another car, and eventually you can build thousands a day.

So if we look at the system overall, we have to consider unemployment in the profession, and how many people just don't want to go I to stem because it doesn't pay enough.

They say I see it, pay in CS specifically is pretty darned good, compared to alternatives in the US. Any lack of developers we have today doesn't come from them choosing other occupations because programming doesn't pay the bills: We just have way too many people getting paid a lot less, or unemployed altogether, that either can't afford the retraining to be programmers, or just. can't do it altogether.

So it's not, IMO, as if the shortage of good pay is the problem: Remember the dot com boom, when a lot of borderline incompetent people were getting a ton of money and achieving very little? That,s the best we'd get if we just multiplied all tech salaries by 10x. Our real problems are far more complicated.

<comment><USER37> _> But if they are also employed as engineers somewhere else, the lack of engineering talent just moves to those other companies._

That, of course, is the entire purpose of high prices. You want some of the demand to go away because they find it to be too expensive so that the supply eventually matches up to the demand. There would be no purpose in having different prices for different jobs otherwise.

A shortage happens when price cannot rise for some reason, such as the government implementing a wage ceiling. I'm not quite sure what could be preventing price from rising in this specific case. It seems like a pretty wide open market.

 _> Any lack of developers we have today doesn't come from them choosing other occupations because programming doesn't pay the bills_

Or they are simply doing better elsewhere. Paying the bills isn't necessarily enough incentive. Even in my own situation, while I still keep my foot in the developer door a bit, I am spending more and more time in my other careers. I _could_ go back to programming full time, but I haven't found an offer incentivizing enough yet.

<comment><USER38> That is an Economists definition of a shortage. Not a normal persons definition of a shortage.

Let's say a loaf of bread went up in price to 100$, and the majority of people in the USA were literally starving to death.

Yes or no, is the US suffering from a food shortage?

I would argue, yes.

<comment><USER37> It is, of course, an economist term. Although I am interested to learn more about your perspective. How do you quantify a 'normal person' shortage?

* Does every single person need to be able to afford bread for it to not be a food shortage? Or is it okay if some still go without as long as the price goes below a certain point? What is that point?

* Is there a shortage of Ferraris because they cost hundreds of thousands of dollars, which is more than I can spend on car? Or are non-essential items not included in shortages?

* If non-essential items are excluded, what is essential about developers? We lived for hundreds of thousands of years without without software just fine.

* If developers are essential, are they always going to be in a shortage situation until they all work for free? I can imagine several developer jobs I would have done for $1/hr, but not for $10/hr, as it is no longer worth the cost to me at that point.

I may agree with you, but I'm not quite sure I understand what a 'normal person' shortage even is. It does not seem to be clearly defined anywhere.

<comment><USER38> Labor markets are inherently inelastic in the short term.

I'd define a shortage by comparing short and long term movements in labor.

As in, if price goes way way up for certain positions, it takes a long time for labor to react and produce more supply. This is bad.

So in the short term, there is a shortage of developers, while labor catches up and tries to fulfill this demand.

To put it another way, imagine if workers could see the future, 10 years ago, and could know how high salaries for devs were going to be.

If workers were armed with this knowledge, a whole lot more of them would have chosen to go into tech.

The Ferrari example doesn't work, because if Ferrari 10 years ago, knew exactly how much demand was going to be, I don't think it would have changed their supply or price at all.

I am not sure of a better word for this "shortage", but I am thinking of any situation where expectations for the future do not match up with the reality of the future, in short term inelastic markets.

And the solution to this market information failure is to either get better information about the future or work towards making the market more elastic.

<comment><USER37> So how does this apply to developers? I remember in the 90s it was already considered the job to have if you want to make a lot of money. That's a good 20 years to prepare the workforce, at least (I start to become too young to have really paid attention to the job market before that).

I got my first full-time programming job soon after the dot-com bust and even at that time it paid better than the offers I'm seeing for new hires these days. And that's without even factoring in inflation. They are making quite a bit less than I was in real dollars.

By that definition I think it is very safe to say that there isn't any sign of a developer shortage. It is a high paying profession, for sure, but it always has been. It's not like it was a minimum wage job 10 years ago. It was still a six figure job even then.

</comment></comment></comment></comment></comment></comment></comment><comment><USER39> Useful question: how many unfilled STEM jobs could be filled by hiring an available applicant and putting them through three months of training? If you want narrow expertise, create it. HR organizations need to be thinking that way.

 _" Lack of planning on your part does not constitute and emergency on my part."_

<comment><USER4> I think part of it is, there doesn't appear to be any penalty for sitting and waiting for that Unicorn candidate that checks all the boxes and works for peanuts. Most employers already feel empowered to have their existing employees do the work of 2 or 3 people, so not hiring someone for a couple months doesn't affect them.

</comment></comment><comment><USER38> If the stem crisis is a myth, then why do I get paid so much?

I'm not the top of my class. I'm just an average web developer doing easy stuff.

<comment><USER40> Probably cause you are in a good area? My last programming job was $38k a year, and considered great pay.

<comment><USER41> Well if you considered it great pay that would mean that it was a high salary for the area, right?

<comment><USER40> Nah, it was barely enough to cover cost of living, and I got a 136% raise by switching to another field. I didn't think it was great pay, that was the employer when they refused to give me a raise after 2 years...

</comment></comment></comment></comment><comment><USER42> I can speak from my own anecdotal experience. I am an Indian immigrant with 2.5 yrs of industry experience and a masters degree from a decent school. Please note that I do not support the H1b abuse by the outsourcing firms. They are pure evil and I honestly hope they go bankrupt. I also believe H1b needs to be reformed to remove the loopholes but that is a topic for another day.

I was recently in the job market and got three offers within a month. One from a big four tech firm Amazon/Facebook/MS/Google, one from a big four consulting firm Deloitte/E&Y/PWC/KPMG and the final one from a very promising startup in North Carolina. The first two paid above market rate but the startup paid 105k + equity because they cannot afford to pay more than that. Thing is, my specific skill set is in shortage ( Big Data and Analytics) and I am sure there are a few other specialized fields where there is a shortage.

There may be a number of kick-ass local software engineers in the market but it is the specific skill set the companies need. Please do not tell me that companies can just hire local candidates and train them on the skill set they need. There simply isn't that much time and experience trumps(yea I know its ironic) training. When a company pays you N amount, they expect X amount of value from you where X is a multiple of N. Also, these companies are often competing with each other to get a product out to the market as fast as possible. Loosing precious time in training is not very optimal as your competitor might beat you.

To be honest, I do not think I can be considered the best in my field and a good american developer can do my job if s/he has the necessary experience. Right now, demand for data engineers with big data skills far outstrips the supply. This won't be the case in say 4-5 years down the line.

STEM in itself is a very broad term, Software development has many disciplines. Are you a Sr. Software Developer who wants to get paid that 200k salary in your local market outside of Silicon Valley? Make sure to cultivate a skill set that is in shortage. Simple macro economics.

For those who are curious, I took up the consulting position and am getting paid 120k base + 10k signing bonus + 10-15% year end bonus + other benefits. Not bad for a person with 2.5 years of industry experience who lives in North Carolina. Is it?

<comment><USER34> By the way, $120k is nice for NC, but the banks in NC pay $180k for the same talent. There is no shortage in NC for Big Data and Analytics talent, the company cannot find the talent because they are simply not paying as much as Wells Fargo next door. If they pay $180k, they will find all the talent they need. They might even convince talent from NY/NJ to move down for that sum. AIG Investments recently convinced a dozen+ group of analytics workers to move down to NC for that salary range. The people would not have moved down for $120k.

Also, $120k is pretty nice, until you have some kids and loads of medical co- pays, dental fees, school fees, and then it is simply too little. Dont believe me? Several years ago, my medical co-pays post insurance were over $20,000 for the year, which is more like $32,000 pre-tax.

What happens is people move into non technical roles which pay commensurate to skillsets. There is no shortage of skillsets, there is simply a shortage of experienced people willing to take sub-market wages for strong resumes.

<comment><USER42> I know three people (2 immigrants on H1b) of the top of my head that are getting paid 200k + inclusive of year end bonus in NC and they have 10 + years of experience in the industry. My old company tried to find a Big Data and Analytics lead for 6 months before giving up. They simply could not find anyone good. This was back in 2014-15. They were willing to go upto 200k to pay for it. To pay anything more than that would not be worth it for them as the ROI just does not make sense at that point.

My role is of a Sr.Associate which requires 2-4 years of experience and I am a 27 year old guy. Would you really pay 200k plus for a person with my level of experience? Hiring a person with more experience for my position and paying 200k plus does not make sense because they would be over qualified for what I do. Also, there are not enough local developers with 2-4 years of exp with my skill set. At my old firm, I personally mentored 2 american fresh grads because they could not find Jr.developers with Big Data skills. I am pretty sure that I can make more than 180k in NC by the time I turn 32-33 but I will need to upgrade my skills at that point. I am already looking at Blockchain instead of waiting for my company to train me. I believe that it will be in demand 2-3 years down the line.

I support the increase in wages but you also need to consider what ROI you are bringing to the company if you are commanding a 200k + salary in a place like NC. Are you brining a 3-4x annual return? Then yes, command that salary by all means. If not, you do not deserve such a hefty pay check.

<comment><USER34> I'm running a startup right now, and you are absolutely correct -- I would not pay $200k for an analytics person. Our company's unit economics cannot support that type of wage.

 _However_ , i'll be honest and say we cannot afford the market wage, so we make development decisions accordingly. It would be _dishonest_ for me to claim a shortage when I'm clearly not paying market wages.

Not wanting to pay market wages is not evidence of a shortage. You get what you pay for. I drive a Hyundai because I am not willing to pay $80,000 for a car. I'll be honest and say I'm unwilling to purchase a Corvette. There is no shortage of Corvettes, i'm just not willing to pay for one. If I try to pay $27,000 for a Corvette, that is not a shortage, that is just nonsensical.

<comment><USER42> Interesting perspective. But what determines market wage? If my company is going to make 500-700k on me per annum, I think a 150k salary inclusive of bonuses + health care + very generous PTO + 5% 401k matching is a very good pay. Which is what I am getting right now. In the end, my company will spend about 200k on me per annum not including my visa expenses. So would that mean the market wage for this position is some where close to what I am getting paid? I think yes. However, if the company is making 2-3 million on me but only pays me my current salary, there is a lot of room for market wage increase.

In the end, I think market wage is also based on the ROI you bring to the company. In my case, if a local appears and commands a 200k base, why should the company pay him/her? In the local's eyes, s/he is worth 200k + but the company does not generate enough value of of them to pay that salary. So, I do not think I am displacing anyone here. In fact, as I said, I helped mentor two americans and I would gladly teach my skills to the locals if the govt had certain programs where immigrants can teach the locals and get some sort of benefit from it( May be a faster track for a green card? ). I don't know, just a thought :)

<comment><USER34> Dear @capRidiculous -- regarding your other question, "what determines market wage" \-- market wages are whatever wages attract sufficient people to meet demand. That might be $15/hr in the case of flipping burgers, or $275k in the case of machine learning. It has nothing to do with the ROI of the company. It is the salary that will convince someone from giving up a competing job elsewhere and join an unfilled position.

There are some things that can reduce market wages -- you can make the job more attractive such that people are willing to fill a spot for less salary. It happens all the time, here are some ways to do so:

\- Federal government jobs pay much less yet attract people because of long- term job security and tax-free earnings in many cases

\- Municipal government jobs pay much less yet attract people because of easy hours

\- Some startups pay much less yet attract people because of exciting products or growth opportunity

\- Hedge funds can pay much less yet attract people with the promise of a share of investment returns

Workers are smart though. If there is a position going unfilled, it is because it is not an attractive position. It either needs to pay market wage, or offer some of the above attributes to help reduce market wages for the position.

<comment><USER42> You make very valid points, but how do you expect someone to pay you 300k if they are not making more money off of you?

The market is willing to pay machine learning experts 250k + because they bring in multiple of 250k in value. If not, no one would be paying such an amount just because machine learning is hard. If that is the case, astro physicists should be among the highest earners in the world. Much more than your NFL stars.

<comment><USER34> By the way, our startup is a machine learning startup. We are pre-profit because we're still in the midst of a parallel medical trial of our diagnostic classifier. We are on venture funding so do not pay market wages. Instead I tend to personally hire new grads and train them. I hire pricey ML experts only as short-term consultants when we need.

Our startup economics do not support wages above ~140k/yr which is why I go through all this trouble. But i'm honest -- it is because we cannot afford market wages, not because there is a shortage. Just because our budget cannot hire a market rate experienced hire.

I've enjoyed our discussion very much. I'd love to offer you a job for sub- market wages :-)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER43> I think another factor contributing to this is a need to answer the question of how we can replace all of the jobs that we are losing due, primarily, to automation.

Because our society is built on the idea that work is the means to justify one's existence, we depend on the possibility of full employment. Therefore, when jobs go away, we have to make them up somewhere. As manufacturing shrinks and automation starts to chip at the large base of service jobs once thought safe, it is necessary that there be something to replace it.

Enter STEM. Instead of assembly-line workers, customer service reps, and bank tellers, we will soon all be the scientists and engineers designing and servicing the grandiose machinery that now does most of the actual work.

<comment><USER19> >Enter STEM. Instead of assembly-line workers, customer service reps, and bank tellers, we will soon all be the scientists and engineers designing and servicing the grandiose machinery that now does most of the actual work.

With the problem that the "automated" infrastructure has _incredibly_ high initial capital costs and _incredibly_ specific skills requirements. Just think how many software frameworks are used for a typical complex product that really does something useful: call that N.

Now, where do you find 100 engineers trained and experienced in _all_ of N different things?

</comment></comment><comment><USER44> The question is not how many engineers are there, but how many _good engineers_ are there.

</comment><comment><USER45> When will the curve of demand start to drop precipitously as we replace STEM workers with AI? (Right now, for instance, we have no QA team, as we can machine test changes and use program flow analysis to test)

<comment> a-saleh Bit off topic, but how did you manage that?

I currently work in QE, but our team is miniscule compared to the devs, so we would rather just automate ourselves away and then hop over to the other side :-)

<comment><USER45> We still QA, but since we have a pretty sturdy code review process where we check out changes and double check them, alongside requiring debts to write test plans and then using Flowtype to limit side effecty changes.

</comment></comment></comment><comment><USER46> These days its easy for companies to remain mediocre and stay in business, thanks to FOSS and a competitive market lowering their tooling price. Some time ago I realized that the only way to get the real worth out of value I create (as a highly skilled developer) I need to be building products / services and selling them to these mediocre corporations, I could not see a way I could survive in these companies in the long term. (weather they provided me with a growth path or not)

</comment><comment><USER47> I don't buy this. In my university, we constantly have companies trying to get our computing graduates, offer them internships, try to persuade them to come for interview. We need to teach our students how to negotiate well.

In most other departments, staff instead have to teach students how to go out and find jobs, put lots of work into applying, etc.

There is certainly a shortage of good graduates, compared to most degree programs.

<comment><USER48> What university do you go to?

<comment><USER47> I teach at St Andrews University, in Scotland.

</comment></comment></comment><comment><USER49> A friend of mine studied bioengineering , but she is working out of the field. Low salaries and temporal jobs moved here to another different field. She doesn't repents her choice, as it was an incredible experience. And she would like to work in her field of choice. But the market pays more her time and capabilities elsewhere.

<comment><USER50> > But the market pays more her time and capabilities elsewhere.

or the market has somehow succeeded in depressing wages to the lowest possible rate. I've heard you have to have a minimum of a BS degree to break into the field of bio/chem. And then you're just a lab monkey following orders from a superior. yet "biotech firms" generate enormous profits, whether its in genetic engineering, healthcare, pharma, etc. It makes me sick.

</comment></comment><comment><USER51> But it is also very difficult to hire a qualified candidate. There is also data suggests that SDE's salary has been hiking for several years in a row and beyond inflation rate. If there is no shortage, how so?

[https://www.fastcompany.com/3051405/the-future-of- work/the-2...](https://www.fastcompany.com/3051405/the-future-of- work/the-20-jobs-with-the-biggest-salary-increases-this-year)

I don't think paying 110k+ in Bay Area for a entry level is unfair, but from what I can tell, my company has a hard time finding one candidate and often the ones we like also get multiple offers and prefer big companies like Google/Facebook, etc.

Yes, 'qualified' is very subjective standard, however, if you ask 5 experienced people, their opinion towards the same candidate barely changes dramatically from one and another, it is either unanimously yes/no, very rarely in between.

So, I think yes, there is a lot of STEM students, but how many of them actually have actual engineering skills is question left for debate.

</comment><comment><USER52> add a (2013) ?

</comment><comment><USER25> All these articles start with an implicit assumption that talent in other countries is somehow automatically inferior and that USA will always remain a leader.

The reality could not be further from the truth. Just take a look at what haappened to the EE / Chip-making industry. The cost difference has shifted all but very high end (intel processors) manufacturing and along with them jobs out of USA to Taiwan, China, Korea and Japan.

While as an employee you might think you can ignore competition from other countries via protectionism, sooner or later the entire industry ends up suffering the effects. This is also what happened to Detroit car makers. Unionization & Protectionism guarantees salaries for half a decade until the market effect trickles down.

The real existential threat to SV is not from Indians or Chinese immigrants as much as the post-Trump Anti H-1B wave might have you believe. Its from failing to remain competitive in an age where privacy corncerns and a hostile US government are causing a rethink worldwide.</comment>


Appserver.io – A Production-Ready PHP-Based Server - <AUTHOR> http://www.sitepoint.com/appserver-a-production-ready-php-based-server/

<comment><USER2> PHP lives on.....

</comment><comment><AUTHOR> another great news for php community to know what appserver is. Love the finding.</comment>


Open source tutorial for Hackintosh installation - <AUTHOR> https://github.com/huangyz0918/Hackintosh-Installer-University <comment><USER2> I just built a Hackintosh on what perhaps might be the closest modern equivalent to an old-style MacBook Pro not made by Apple, the HP Spectre x360, and while the process of getting it up and running requires some technical knowledge, it was something I was able to do, start to finish, in less than a week. (Even the touchscreen works!)

I've used Macs for 15 years, and I wanted to keep using Macs, but I had a real problem with the fact that Apple has let its upgrade cycle become so lax, has fostered a lack of upgradability, and hasn't done enough to support an ecosystem of creatives that have been very passionate about their products over the years. That speaks to a company not listening to customer needs—one that is trying to translate a mobile business model to desktop users. It speaks to a company that walked away from its users, not the other way around.

If Apple's not going to make its machines more upgradeable, it should at least, for the love of God, upgrade its processor architectures without having to have a keynote every single time. They do so few keynotes in 2018 that it's messing up their ability to release products.

<comment><USER3> So because you think that Apple isn't listening to your needs, you can take the OS that they created and use it however you want to?

I am honestly surprised at how many people in this thread don't seem to care about Apple's right to decide how people use the software that they licensed from Apple.

<comment><USER2> This site is called Hacker News, right? Certainly, taking a piece of hardware I own and making it do something it was not designed to do properly fits the description of "hacking," right?

I appreciate Apple's value as a company as much as the next guy, but this was not a decision made lightly. If they want $100 from me over the right to run their OS on my computer (as I believe they used to charge during the PowerPC days), fine. But their failure to give consumers the option, in this and so many other cases, highlights a big frustration longtime users have with the company nowadays.

I'm sure Apple knows about the Hackintosh community, and I'm sure they follow it. I also know they have done little to stop it. As a user, I want them to care about my needs as much as they do about their bottom line. They used to.

<comment><USER3> I understand that people are usually driven to build a hackintosh out of frustration with Apple's hardware offerings.

That doesn't change the fact that it's against their licensing agreement. Anyone who wants to be able to sell their software in the future (or write software for a company that sells software) should care about that and have good reasons for violating it if they choose to do so. Many people in this thread seem to accept that it's a morally okay thing to do without thinking about it at all.

The site may be called Hacker News but it's full of mainstream software developers, many of whom work for companies like Apple. It's also (as far as I know) always been more focused on the startup scene than the hacking scene. You would think that people who's goal is to build their own business would care about this sort of thing.

<comment><USER2> I'm not a software developer, I'm a technology enthusiast.

<comment> xkcd-sucks a "disgruntled customer"

</comment></comment></comment></comment></comment></comment><comment><USER4> The fact that Apple actively takes measures to prevent MacOS from running on foreign hardware (including virtual machines) really annoys me. If I want to publish a cross-platform app for iOS, I literally have to buy a device just for compiling and uploading the archive. That's not the case for any other mobile platform I've ever seen.

This is even more ironic considering the good reputation of being open that Apple has for some people, and tries to promote through marketing.

<comment><USER5> _The fact that Apple actively takes measures to prevent MacOS from running on foreign hardware (including virtual machines) really annoys me_

I tried a Hackintosh VM in VirtualBox recently, just to play around, and it was surprisingly easy. AFAIK all that distinguishes "non-Apple" from Apple hardware is an "SMCDeviceKey", a short identifier in the BIOS, and a few other (mostly cosmetic) version strings. With a VM, all you need to do is set those and boot from an ISO --- there are instructions around to create one from the appstore package. No modification of the OS itself was needed; it thinks it's running on a real Mac.

<comment><USER6> > I tried a Hackintosh VM in VirtualBox recently, just to play around, and it > was surprisingly easy.

I'm sure people (I) will appreciate some links. Thanks.

<comment><USER7> Here's one that I use: [https://github.com/AndrewDryga/vagrant-box- osx/blob/master/R...](https://github.com/AndrewDryga/vagrant-box- osx/blob/master/README.md)

</comment></comment></comment></comment><comment><USER8> My experience running a hack for 6 years:

Pro:

\- best desktop OS there is reg security and privacy

\- my hardware, BD-RE and 4TB drives, full size GeForce

\- 1/4th the price

Cons:

\- learning curve is a bitch, but done once

\- updates can be tricky, thanks to Nvidia :)

\- for IT pros / hackers only

Result is indistinguishable from real iMac, sleep and all works.

Once you'd settle your clover.config, it is mostly painless. I even have FileVault on my volumes :))

<comment><USER9> How can you possibly claim a proprietary operating system is better with regards to security and privacy than Linux distros, FreeBSD and OpenBSD?

<comment><USER8> Nothing beats openbsd, lol. I was talking desktop OSes, and Linux is not one, imho, apps wise.

</comment></comment></comment><comment><AUTHOR> There are a lot of people have posted their configurations and successful installation guides in GitHub, but they are hidden very deeply so that many newbies cannot find the configuration the same with their machine. It's a great pity:( So I came up an idea that why not to build an index which can collect some basic knowledge about operation system and successful Hackintosh configurations together? Maybe we can add some systematic guides in it too. It's just a new repo, but I will do this step by step.

<comment><USER10> Since your repo is not software but only documents I would suggest that instead of the Apache license you use the Creative Commons CC 4.0 BY license.

[https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/)

<comment><AUTHOR> Thank you,it sounds good !

</comment></comment></comment><comment> cup-of-tea I remember when PearPC was made and people started to get excited about making a Hackintosh. I didn't have a fast enough internet connexion back then to download the required software to try it, but I heard that it was generally not a good experience. But I certainly had dreams of running OSX on my PC hardware.

These days I use GNU/Linux and have no desire whatsoever to try MacOS.

</comment><comment><USER11> The more Apple is failing to deliver hardware to the high standards they used to do, the more Hackingtosh's popularity is going to grow.

<comment><USER12> As a Mac user disappointed with Apple's current direction with its hardware line, there are some risks with using Hackintoshes:

1\. It requires considerable more work to set up and maintain a Hackintosh system than using Linux or *BSD, and it's completely unsupported. When I researched this last time, there were three sore points: (a) Upgrades between even minor point versions could sometimes mess up installations, (b) finding compatible hardware can be difficult (especially in the case of laptops), and (c) using Messages and iCloud on a Hackintosh could risk getting locked out of those accounts.

2\. Installing macOS on a non-Apple product is a violation of the macOS EULA. Because of this, a Hackintosh may be a liability in professional environments.

3\. Apple could make moves, intentional or unintentional, that could disrupt or even end the Hackintosh ecosystem. In the early days of Hackintoshes Intel Atom processors were supported, which resulted in people installing Hackintoshes on netbooks such as the Dell Mini 9. This ended when Apple stopped compiling support for Atom processors. A more serious showstopper would occur if Apple abandons the x86-64 in its Mac lineup. Then the Hackintosh community would be stuck with whatever is the latest version for the x86-64 architecture.

<comment><USER13> 4\. Installing a Hackintosh often requires installing binary kexts of questionable origins. It is beyond me why people give some random person on the net ring-0 access.

<comment><USER14> People do that all the time, though, for software like drivers or network extensions.

</comment></comment></comment></comment><comment><USER3> I am surprise that almost no one seems to be discussing whether it's right or wrong to be installing MacOS on non-Apple hardware.

I'm pretty sure it's against apples EULA. Shouldn't we as software developers care about that kind of thing?

<comment><USER15> Oh no what will the poor _most valuable company in the world_ do if we violate their made-up rules? It's an existential crisis, isn't it?

Why should software have a EULA and not a screwdriver or a tractor or a pair of socks? "I bought it and I'll do what I want with it", why is software the only thing that isn't sold?

Just because rich software dudes made up some rules doesn't mean they can make us follow them. lol.

Isn't this place called _hacker news_ , don't hackers bend and break rules all the time, in the pursuit of knowledge and better technology? Even the founders of YCombinator (one of em) made the most infamous worm in computer history, that's hardly "following the rules".

And then Apple decided you couldn't install any software on your own computer (aka phone) unless they both approved it and you give them a 30% tithe on top of what the author makes. Fuck 'em. Build more Hackintoshes; do whatever you want. I'll keep on using open source since it's no-effort, free, and non- patronizing w/r/t my software sources.

The internet only stops being the wild west when you turn in your guns and buy a plow (that can only be used on Apple-approved sharecropping projects).

<comment><USER16> So when people violate GPL or other open source licenses, it's ok to ignore that too right? This is morally equivalent: one can't expect "made up rules" of open source to apply to anyone if "made up rules" can just be arbitrarily ignored.

<comment><USER17> GPL is about distribution, completely different kind of license than EULAs.

</comment></comment></comment></comment><comment><USER18> Does anyone know a good virtualbox virtual machine version of a hackintosh ?

<comment><USER19> VMWare Fusion can install MacOS images with a few tweaks. I don't know about virtualbox though.

<comment><USER20> Isn't VMWare Fusion a Mac app anyways?

<comment><USER21> Yes, and the only legal way to run macOS in a VM is when you run it on Mac hardware running macOS.

<comment><USER19> Yes it's not legal, same as Hackintosh.

<comment><USER21> What makes you think so? You're only allowed to run macOS on Mac hardware. You're doing that if you run it in a VM.

</comment></comment></comment></comment></comment></comment><comment><USER22> I would have been more happy if PureDarwin had taken off or a true uKernel had incorporated Apple OSS in their code base.

For people in need of hackintosh this is a solution but why not have the real thing or a community version (like good ol Gnu-Darwin).

It is apple to blame here, the could have taken the FOSS direction and do a Darwin-EL (Like RHEL) for FOSS community, even with a fee. MacOS is definitely not for me though they have done an interesting work in the kernel and this is what I would like to try instead of BSD or Linux.

The GUI? Far from my tastes.

</comment><comment><AUTHOR> Hackintosh is good but I think most people just want a macOS system, they don't really care about how much they can learn from hackintosh installations.(e.g., bootloader, dsdt, kexts adaptions). This repo may be more suitable for those geeks and those who want to understand the computer OS better. So it will focus more on the technologies and theories (if one just want to install macOS and get no interests in hackintosh, he should turn to tonymacx86 or some other forums)

</comment><comment><USER23> For those who run Hackintoshes - do you use a custom boot loader or kext to bypass dsmos.kext? Have you reviewed it to make sure it doesn't install backdoors, key loggers, or introduce other undefined or unexpected behavior?

</comment><comment><USER24> I'd be much more interested in a tutorial for setting up PureDarwin on generic x86 hardware (and qemu if needed) as eg. a build and CI server for Mac OS/Mach-O command-line and other non-GUI apps.

</comment><comment><USER25> Can hackintosh machines develop iOS apps and sign apps using XCode? I don't think macbooks or the apple desktops are the best machines at all, but the MacOS is too valuable to leave for Windows.

<comment><AUTHOR> Yes many Hackintosh users are iOS developer, I built my first Hackintosh in order to develop my iPhone apps too.

</comment></comment><comment><USER10> If you want a low cost, relatively low effort Hackintosh, look into running macOS on one of the lower-end ThinkPad X220 or X220 Tablet laptops that you can buy second-hand, following this guide:

[http://x220.mcdonnelltech.com/](http://x220.mcdonnelltech.com/)

Granted, the lower-end x220 and x220 Tablet machines are nowhere near as strong as the MBP you'd typically buy.

Perhaps even an higher-end (stronger CPU, more RAM) version of the x220 or x220 Tablet might be worth it?

Another thing about this vs MBP is battery life. Apple did a really good job of optimizing MBP for battery life. You won't get the same kind of battery life on the x220 or x220 Tablet. Also, speaking of battery, when you buy secondhand you risk getting a bad battery, so be prepared to shell out for a replacement battery.

I bought an x220 Tablet with an i5 CPU and 8 GB of RAM second-hand and put macOS on it. I also bought a compatible mini-PCIe WiFi card like the page I mentioned above suggests and switched the one that was inside for that. Switching out the WiFi card in the machine is a nice opportunity to clean the insides of the laptop of dust anyway. ThinkPads are very easy to disassemble, service and put back together, that is a big part of their appeal for me and many others.

However, I've only booted the system and tested it a little bit to confirm it works, because I haven't had time to play around with it. I did have some problems with the mouse pointer or keyboard when I tested it. That problem might be because I haven't yet installed the suggested BIOS.

So before you decide you decide to go with x220 or x220 Tablet, talk or read about what others that are using it are saying about the performance and check if anyone has experienced input problems with the keyboard and mouse and if that is simply solved by installing that BIOS.

There are some people on /r/ThinkPad that run macOS on x220 or x220 Tablet. Likewise, there are some posts about these computers on /r/hackintosh.

[https://www.reddit.com/r/thinkpad/comments/8chtpk/just_bough...](https://www.reddit.com/r/thinkpad/comments/8chtpk/just_bought_a_new_x220_and_installed_high_sierra/)

[https://www.reddit.com/r/thinkpad/comments/839544/is_bios_v1...](https://www.reddit.com/r/thinkpad/comments/839544/is_bios_v143_needed_for_high_sierra_on_x220/)

[https://www.reddit.com/r/thinkpad/search?q=x220%20macOS](https://www.reddit.com/r/thinkpad/search?q=x220%20macOS)

[https://www.reddit.com/r/thinkpad/search?q=mcdonnelltech](https://www.reddit.com/r/thinkpad/search?q=mcdonnelltech)

[https://www.reddit.com/r/hackintosh/search?q=x220](https://www.reddit.com/r/hackintosh/search?q=x220)

Also perhaps someone else on HN do too and would like to chime in here as well?

<comment><USER26> A hackintosh running on a tablet with touch screen is actually a better mac than the ones made by apple. I wonder for how long apple will cling on to only producing laptops without touchscreens.

<comment><USER27> I'm using macOS on an x230t for precisely that reason. Pen works great, even has force sensitivity and served me well in the tutor groups I've held.

</comment></comment></comment><comment><USER28> I don't understand why Apple don't release macOS for non-Apple devices. Sure, it's a huge undertaking support-wise but it's not like they don't have the resources. I'm a full time desktop Linux user but I'd buy macOS for my laptop in a heartbeat to get a Unix environment with proper commercial apps like Office and Photoshop. I just don't want to pay the Apple premium for the hardware.

<comment><USER29> Do you think the untapped market of people who would buy macOS only if it means not buying Apple hardware is a bigger incentive than what they would lose if people who get cornered into buying Apple hardware became able to get cheaper non-Apple hardware?

<comment><USER30> And then they buy shitty hardware, everything lags and they blame it on macOS. Hello Windows problems.

</comment></comment></comment><comment><USER31> Looking at the effort and cost, why don't people just buy a mac? (other then of course the fun of actually doing this)

<comment><USER32> Once Apple moves to arm you will have no choice, hackingtosh won't have anything but an old version of macosx. The only hardware that will be able to run macosx in 2 years would be in house at Apple. For the same reason that while you can build your own Android phone from parts you would never be able to DIY an iPhone.

<comment><USER33> "Once Apple moves to ARM" is still pure speculation. Steve Jobs also famously sneered at Motorola about the day he "wont need them anymore". Look how well that panned out with the G5

You can run Android on just about anything with minimal effort. X86, ARM, MIPS

Hell. Even the Raspberry Pi supports Android fairly well

[https://developer.android.com/things/hardware/raspberrypi?au...](https://developer.android.com/things/hardware/raspberrypi?authuser=0)</comment></comment></comment>


Reading bits in far too many ways - <AUTHOR> https://fgiesen.wordpress.com/2018/02/19/reading-bits-in-far-too-many-ways-part-1/ <comment><USER2> _The first major decision to make is whether fields are packed MSB-first or LSB-first_

Unfortunately in many cases like image and video codecs, that decision has been made for you already --- and not in the direction that's most efficient for a CPU (but of course is pretty much immaterial for hardware.) I suspect "It's easy to fool yourself into thinking that one variant is better than another because it looks prettier" was what drove those decisions.

The big-endian/MSB-first order requires extra "bit movement" if you are maintaining a bit buffer in a register, since the bits of interest are always going to reside in the topmost bits, but you often need them in the lowest bits to use them. With little-endian/LSB-first, the bits of interest are already in the lowest position, and naturally "fall off the end" as you shift right and replenish into the upper bits. IMHO another reason to prefer little- endian: it's very logically consistent.

<comment><USER3> Your reasoning about which end has interesting bits has no basis in how CPUs actually do any of their work The byte-ordering does not fundamentally affect how operations are implemented.

Us humans often find little-endian confusing because we write numbers big- endian. If you have the number 258 (0x102) in memory in little endian (which most computers use) 32-bit, it is hex 02 01 00 00, binary 00000010 00000001 00000000 00000000. If you bit-shift that one to the right without care, you end up with 00000001 00000000 10000000 00000000, hex 01 00 80 00, which is 8,388,609.

To fix it, write your digits also in little-endian order, so the first number is 10000000 010000000 00000000 00000000. Then the shift operations match expectations.

<comment><USER2> _Your reasoning about which end has interesting bits has no basis in how CPUs actually do any of their work The byte-ordering does not fundamentally affect how operations are implemented._

Yes it does. Consider an MSB-first ordering, and you want to extract 4 bits to use as an integer or whatever: in the bit buffer, that would be "aaaaxxxx...", where "a" are the bits you're interested in. You'll need to copy from the bit buffer to the "work" register, then shift right in order to put them in the right place. Furthermore, to "eject" those bits, you need to shift _left_ and insert from the lower bits, i.e. the bit buffer becomes "xxxx..." in its most significant bits.

With LSB-first, "...xxxxaaaa", the bits do not need any shifting --- they're already in the right place.

<comment><USER3> Just no. CPUs don't have an endianness on internal registers and operations - it only applies to memory access (and occasionally some simple conversion operations).

<comment><USER4> They are talking about _bit-endianess_ here not the usual _byte-endianess_ when they mention MSB-first and LSB-first.

The bit-endianesss you chose greatly impacts how your bit-reading loop will be implemented. In particular, if you want to _use_ some bits, you want them in the low bits: if you write the value 42 into a 7-bit field, and then later I give you back (42 << 56) or something, you are going to be very confused: you expect to get back 42 (i.e., the return value equals 42, or to be pedantic the 7-bit field should be right-aligned in the uint32_t or uint64_t return value).

<comment><USER3> Oh my. You're right, I entirely misunderstood. How embarrassing. I wish I could delete my comments.

</comment></comment></comment></comment></comment></comment><comment><USER5> This article is awesome, I don't work in this kind of thing at all, and I feel like it's a good exersize for my brain.

</comment><comment><USER6> Part 2: [https://news.ycombinator.com/item?id=16419774](https://news.ycombinator.com/item?id=16419774)

</comment><comment><USER7> Fun small educational project: If you want to learn how to work with units smaller than a byte, implement Base64 from scratch. It's really fun!

<comment><USER8> Better would be the Huffman coding; one has to deal with variable bitfield widths.

<comment><USER6> In fact, that appears to be where the author is coming from. Part 2 mentions techniques coming from the Kraken Huffman decoder ([http://www.radgametools.com/oodlecompressors.htm](http://www.radgametools.com/oodlecompressors.htm))

</comment></comment></comment><comment><USER9> Almost everything on this guy's blog is amazing if you are interested in hardware, optimization and that kind of thing.</comment>


Ask HN: An Alternative to Google News Alerts? - <AUTHOR> Im not sure if its supported or not anymore but rarely if ever works for me. Paid or Free alternatives would be welcome <comment><USER2> You probably forgot RSS.

<comment><USER3> Maybe they forgot RSs, or maybe not. I used to get Google alerts as feeds, until they stopped capturing most things. RSS alerts like Google alerts, when they used to work would be pretty cool.</comment></comment>


NeverSSL – helping you get online - <AUTHOR> http://neverssl.com/ <comment><USER2> Originator/Maintainer of NeverSSL here. AMA. Funnily enough, I'm on a plane to Fresno right now, and I used it to get online.

FAQ:

1\. Why does neverssl.com use Javascript to redirect to a random subdomain?

Over the last year users reported that some networks aggressively cache the fake DNS and pages they use for wifi capture. Neverssl.com now works around this generating a request to a random subdomain - this will bust any DNS cache, and any HTTP cache. It also means that if your browser or ISP caches the Neverssl.com page itself, that's fine.

2\. Is NeverSSL.com tracking you or anything like that?

Normal S3 access logging is enabled, so I have a record of every IP address that accessed NeverSSL.com. I aggregate this information by AS number (ISP basically) to "rank" the networks that get the most usage ... this helps me figure out which networks are most broken. I occasionally forward this aggregated data to those network operators to encourage them to fix wifi capture. There's no other tracking, but you are trusting me on that. I am a founding director of Digital Rights Ireland, we sue governments and win for better user privacy.

3\. Can I put ads on NeverSSL.com?

I thought about this, but I think it'd degrade the experience too much. I am thinking of maybe putting some rotating art, or inspiring poems, something to add a bit of soul to our day. Ideas welcome!

4\. Shouldn't I use example.com?

I predict that example.com/org/net will all go HTTPS, as they are intended to service as canonical examples of internet standards.

5\. What happens if NeverSSL.com gets pwned?

I don't understand this risk; it's not setting cookies, there's no passwords, or any personal information.

<comment><USER3> > I don't understand this risk; it's not setting cookies, there's no > passwords, or any personal information.

I agree with you technically, but in practical terms imagine if you changed the web page to look like a Google-style search engine window.

You can expect all the less-sophisticated users to try typing site names and even URLs into that text widget.

And if you give them back what looks like a link to the place they wanted, they absolutely will click it, and most of them will happily trust that having typed "My Bank" and clicked a link labelled "My Bank: A secure bank" in the resulting "search results" the site they've reached, fake.mybank.neverssl.com must obviously be their bank, and they will cheerfully give you their bank credentials.

I'm not suggesting this as a criticism, it's purely an observation, that in practice NeverSSL getting pwned would be a real problem, the same exact way it's a problem when some Hollywood actress (who doesn't know the first thing about medicine) tells women not to vaccinate their kids. People are dumb, and we have to allow for that.

</comment></comment><comment><USER4> I always just use [http://example.com](http://example.com) since it's managed by ICANN and if it ends up pwned then we have bigger problems. I don't know who is behind this one, or whether it will stick around or be sold off later. Example.com is (basically) forever.

<comment><USER5> I'm being hyper-pedantic here. Wouldn't the relevant kind of pwning happen to you rather than to the site? I'm thinking something like a MITM at your local coffee shop's wifi or an evil employee at your work. If someone took over the machine serving example.com, HTTPS wouldn't have stopped it or helped alert anyone to it.

<comment><USER4> The point is to generate a 302 redirect, so I guess it doesn't matter what example.com is actually serving if you never see it. But the captive portal may forward you along to it after you authenticate.

This doesn't do anything to prevent a MITM at all, just that if neverssl.com starts serving crap you have a nonzero chance of loading it when trying to use it to generate a redirect. If there's a local MITM you're in trouble either way.

<comment><USER5> That makes sense. My one change to the workflow is that I tend to visit example.com again, one last time, after jumping through the captive-portal hoops to make sure it's all really working. Thus I do see the content.

</comment></comment></comment></comment><comment><USER6> +1 for example.com here. IMO "SSL" is not great to use in the domain name as non-tech folk have no idea what it is and might struggle to remember it. "NeverHTTPS" might be better, but even then how many people know their connection isn't working because of HTTPS? I'd wager not many.

I'm kind of amazed no one has implemented a better system for this yet. Public WiFi is very common, and relying on intercepting HTTP requests to show users a login screen seems very hacky to me.

What if a device pinged the gateway IP on a certain port whenever it connected, and received a small payload with details on connectivity authorisation URL, etc. etc.? For backwards compatibility if that ping fails it could just assume the network is open. It would take years for all the public routers out there to be updated, but you kind of need to start somewhere.

<comment><USER7> Perhaps the folks running these networks lack the incentive to care about technical correctness? OTOH, one would think a better system would lead to a higher conversion rate (i.e. more people are actually able to pay you).

<comment><USER8> They seem to have an incentive mostly to say that they "have public wifi" but not take responsibility for delivering a working system.

<comment><USER7> Yeah, for "free wifi", the incentive is definitely to _not_ connect you. You don't consume bandwidth, so the handful of people who _do_ connect won't complain about shitty speeds.

For paid wifi, on the other hand, wouldn't they want to ensure you _can_ connect? Most airlines charge for wifi. Yet I find myself typing "example.com" on almost every single flight.

<comment><USER9> There is incentive to connect you to the wifi, but no to the internet. Having you trapped behind a captive portal is perfect for this. You are connected to their wifi so they can more accurately track you through the store, but you are not using their internet bandwidth.

</comment></comment></comment></comment></comment><comment><USER10> I believe [http://detectportal.firefox.com](http://detectportal.firefox.com) is also useful for this purpose.

</comment><comment><USER11> I see my iPhone and Mac use [http://captive.apple.com](http://captive.apple.com)

</comment><comment><USER12> I use [http://4chan.org/](http://4chan.org/)

The homepage is fairly innocuous, it's always http, and if it loads I also know the WiFi doesn't have some crappy web filtering software running on it.

</comment><comment><USER13> Previous post (2017): [https://news.ycombinator.com/item?id=13369038](https://news.ycombinator.com/item?id=13369038)

</comment><comment><USER7> "example.com" has been my go-to. Impossible to forget.

On the other hand, they don't guarantee that they won't implement an HTTPS redirect someday. Seems like that's the major selling point here.

I wonder how this site is funded. They have the ability to gather (presumably valuable) statistics on WiFi hotspot usage...

<comment><USER14> Looking at DNS and the resolved Its, it's hosted via CloudFront and (I'd imagine) an S3 bucket. On the other hand, the creator is a fairly prominent employee of AWS, so that's hardly surprising.

The only tracker on the page is from Twitter (for the follow button). I don't know what metrics they give you, but other than that it looks to be whatever can be grabbed from the server side logs.

I've been using this for several years now both to test internet connectivity and to force captive portals to load - I didn't know about example.com until today though!

</comment></comment><comment><USER8> The MITM to make you agree to a pointless EULA for using wifi is one of the most obnoxious things going. I know this trick (I've usually used [http://fark.com](http://fark.com)) but non-technical people don't know it and as such public wifi is largely useless to them because of the MITM they use. Also, plenty of public wifi is flaky and as such by the time you've loaded the authentication page, you've lost your signal and need to re-auth!

</comment><comment><USER5> Related: browser is confused and caches some of the captive portal content for the site you want to visit, so it remains broken even after the portal lets you through. Add a cache-buster to the end of the URL -- ?a=1 for example -- and most webservers will handle the modified URL without complaints.

</comment><comment><USER15> I often find my self having to use Firefox to get online on public WiFi. Firefox have frequent requests to their own servers using only http to circumvent this problem.

Thumbs up for this website.

</comment><comment><USER16> What's with these generated subdomains like "kmbsznxflchdtrwv.neverssl.com"?

Can anyone here explain the idea behind those?

I have used NeverSSL in the past, but I haven't seen those...

<comment><USER17> It prevents caching: [https://twitter.com/NeverSSL/status/1136488879106666496](https://twitter.com/NeverSSL/status/1136488879106666496)

</comment></comment><comment><USER18> I usually open 192.168.0.1 which works quite good;)

</comment><comment><USER19> I've been using this website for the last 2 years or so. I can't not recommend this site enough. Simple and effective

</comment><comment><USER20> Using this site? No thanks, don't want to be injected with bad JavaScript code or other kind of trackers.

<comment><USER5> Please give examples of sites that pass your threat assessment.</comment></comment>


China blacklists millions on booking flights as social credit system introduced - <AUTHOR> https://www.independent.co.uk/news/world/asia/china-social-credit-system-flight-booking-blacklisted-beijing-points-a8646316.html <comment><USER2> In the video they say the system is based on the principle of "once untrustworthy, always restricted." I can't see how that's a good idea. The less stake in society you give people, the more instability you're bound to have. Why put people in a position where they've got nothing to lose?

<comment><USER3> Sounds like a great way to generate domestic terrorists. IIRC there was some Secret Service report after Columbine that basically found that being goth, listening to metal etc weren't pre-cursors to school shootings... BUT that heavy handed suspensions/expulsions _could_ contribute to someone being violent.

</comment></comment><comment><AUTHOR> > According to the state-run news outlet Global Times, as of May this year, > the government had blocked 11.14 million people from flights and 4.25 > million from taking high-speed train trips.

> Punishments are not clearly detailed in the government plan, but beyond > making travel difficult, are also believed to include slowing internet > speeds, reducing access to good schools for individuals or their children, > banning people from certain jobs, preventing booking at certain hotels and > losing the right to own pets.

<comment><USER4> Is there a concept of due process? I mean, in the US, you could draw various parallels to several of those things, but we do expect or at least aspire to a certain amount of transparency and due process.

</comment></comment><comment><USER5> What's the cultural history behind the motivation for all this?</comment>


Show HN: Tillid-Cast – A virtual browser in the cloud - <AUTHOR> https://cast-tillid.tillid.io/demo/ <comment><AUTHOR> Hey op here.

I create Tillid-Cast in order to provide a virtual browser in the cloud. With this the demo, you can browse any website without any tracks in your history. You will also get a different IP than yours.

With the full version of Tillid-Cast, you are able to save your browser state, cast your screen, and share your browser with somebody else, which is convenient when you need to share temporarly an access to a website and don't want change the password.

</comment><comment><USER2> Hey, excellent project. I think there are a lot of applications fur such remote browser service. I really like it. More in depth detail on security features and history tracking would be ideal.

<comment><AUTHOR> Hey amir6. Thanks for the feedback.

Regarding Security, you can count on two level of isolation, one being provided by the browser sandbox, the second one by a docker layer. Regarding the trafic, everything is https. Access is protected by a passwrd provided in the url. this password in gereted randomly when starting your own browser.

Regarding history tracking, there is no build in mechanisme to keep track of history. In this demo, the image is destroyed after 15 minutes.

Of course, if you want use a remote browser for long period, you might want to get history. For now, this is only build in mechanismes. I did not do anything particular in order to keep track of it, or in the opposite hide it.

One more point, which application do you have in mind for it ?</comment></comment>


Groovy Project Lead to Focus on API Dev Tools - <AUTHOR> http://restlet.com/blog/2015/03/02/lets-make-apis-groovy-er/

<comment><USER2> There's a second blog post published at basically the same time by the CEO of Restlet with more info about the connection between Groovy and their API platform and language development: [http://restlet.com/blog/2015/03/02/head- of-groovy-project-jo...](http://restlet.com/blog/2015/03/02/head-of-groovy- project-joins-restlet-to-lead-api-development-tools/)

<comment><USER3> > In 2014, Groovy was downloaded 4.5 million times

Almost 1 million of those downloads were made from a proxy server in China from Bintray during May 2014, see [http://groovy.codeplex.com/wikipage?title=Blog07#2](http://groovy.codeplex.com/wikipage?title=Blog07#2)

<comment><USER4> It's 4.5M excluding _all_ downloads from China. There were probably a lot of legitimate downloads from China but given the impressive number of downloads from bots there we had to exclude all of them in the total.</comment></comment></comment>


My experience with alternative Android markets - <AUTHOR> http://martingryner.com/on-alternative-android-app-stores/ Overview of alternative Markets for Android performance based on real world application example. <comment><USER2> More than half of the total downloads/day of my six Android Apps are from alternative Android markets. If you remove one of my applications from consideration, almost 80% are from alternative stores. (~400k downloads in the past year with most of my apps released last fall)

The distribution isn't consistent across Market stores. My most popular application on the Play store is my least popular application on Slide Me (yet still the most popular app of its type there). Slide Me is really not a productivity store.

Slide Me is the goto store for decent apps that have been banned but are still good apps (i.e. GameBoy Emulators). Slide Me also featured two of my apps with "above the fold" exposure. That could not happen on the Play Store. Also the Slide Me submission process is relatively easy and quick.

One of my apps is really only for young children, and it gets horrible reviews on the Amazon App Store by what appears to be 13 year olds. Yet they keep downloading it and downloads exceed what I get on the Play Store. To top it off they also pay $ for the ad-free version! The Amazon submission process is slow but not all that painful if you're patient. Also stagger deployment of your latest APK and do the Amazon app store last. After you know for sure that there are no last minute FUBAR bugs feel safe to submit the APK.

I've also found my apps stripped on Chinese App stores with the the certificate stripped off. One of them has more pirate downloads than all legit Android market stores combined. Also if you want to keep your ad revenue, don't place the advertising ID in the manifest file. Embed it in your code in a non-obvious way. You won't make as much because of the regions requesting ads but this turns at least turns piracy into something positive.

Android Pit and AppBrain were pretty good alternative app stores, but the Play Store has caught up to them in terms of functionality.

The AppsLib store is pretty sad as well. (appslib.com) Previously it wasn't that bad. But since [KC]or?by has jumped to GetJar, the # of downloads has stagnated.

GetJar requires you to be vigilant. The site can be a bit buggy and the submission process was a tad more painful than the Play Store. Approval took forever as well and I ended up having some of my apps duplicated on the store. Also please note that GetJar has been actively improving.

There are also carrier based stores that are so incredibly painful to use that they may give you an aneurysm. You have to submit your app on a _per device_ basis. The nice thing about that is some of the carriers will send you a video clip of a bug on a device. (This was very helpful when I had no way to know that a device 'broke' implementation of the accelerometer. Input was off by 90 degrees in spite of checking default orientation and current rotation.)

</comment><comment><USER3> I think I'm a fairly well-versed Android user -- there are now five devices in regular use in my house, and I've been using it since the original Droid came out. All except one is rooted, and most are running unofficial ROMs.

I've only heard of two of the alternative markets: GetJar and Amazon. I have them both installed on one device, largely to see if anything interesting comes along. So far, my opinion is that they are smaller and even less well laid out than Google Play.

I suspect doing an alternative market right -- with good search and categorization, suggestions made on some basis other than "current promotion", friendly to developers -- could be a lucrative proposition. I'm not seeing that now from any current vendor.

</comment><comment><USER4> What's going on with the blog post? Completely unreadable from my device, every time I zoom in that annoying black bar on the left grows to epic proportions and covers most of the content!

<comment><AUTHOR> I'm sorry about that. The theme is not optimized for mobile devices. At least now I know I have to push mobile theme for the blog higher on my todo list.

<comment><USER5> Maybe also tweak the cyan link color to be a bit darker - it is difficult to read on both of my screens

</comment></comment></comment><comment><USER6> A piece of (semi off-topic) advice for those indie developers that want to publish their apps in alternative Android markets: get a VA to do it for you.

It's a great way to get started into outsourcing, given that the task is so straightforward. We're currently having 6 of our applications published into 10 alternative Android markets at a very competitive rate by a freelancer we've contacted through oDesk.

</comment><comment><USER7> This is an incredibly well written article. Thank you for contributing it. Even though I'm an iOS developer exclusively, I like seeing other perspectives as well. As a side note, I apologize for not writing this on your site, but I don't like to mix my Internet comments with Facebook.

</comment><comment><USER8> Thanks for sharing, especially on Amazon App Store.

</comment><comment><USER9> I'm an iOS developer and as much trouble as iTunesConnect is to deal with, that sounds _terrible_.

How do you get any work done, ever? Deciding on which app stores to deal with and then having to learn all of their idiosyncrasies based on trial-and-error? Jesus.

<comment><USER10> You can publish to a single app store as well if you want. You just don't have to.</comment></comment>