Secret Court: FBI "Backdoor Searches" of Americans Violated the 4th Amendment - <AUTHOR> https://www.eff.org/deeplinks/2019/10/secret-court-rules-fbis-backdoor-searches-americans-violated-fourth-amendment <comment><USER2> Here's the thing,someone violating a constitutional right of a citizen does notx carry a penalty. They basically get told "don't do thid anymore" ,worst case they get a court order to cease the practice and if they don't a contempt charge.

Perhaps deliberate removal of a citizens rights should carry a criminal penalty? I mean,the highest law of the land does not carry a penalty against officials that willfully ignore it?

</comment><comment><USER3> Does it seem like the FBI largely goes unregulated? Since the nineties it seems to have recessed into the dark, and operated without much oversight.

<comment><USER4> A quick thought on regulation:

1\. It will generate huge bureaucracy that will most likely be a huge overhead both on money and time.

2\. The bureaucracy still have no idea what's going on.

<comment><USER5> Is status quo desirable? What would be desirable, and what would be necessary to get there?

<comment><USER6> The rules change all the time and are subject to layers of oversight. See [https://icontherecord.tumblr.com/](https://icontherecord.tumblr.com/)

\- FISA court (judicial)

\- House / Senate intel committee (legislative)

\- IC Inspector General (executive, they police the agency itself)

</comment></comment></comment></comment><comment><USER7> FISA courts, fusion centers, stellarwind etc.

</comment><comment><USER6> The FISA court ruling proves the system works. This is good news.

To those outside USA, FISA is like an intelligence related court.

From an earlier gov brief:

> Our government is tasked with protecting an interest of utmost significance > to the nation - the safety and security of its people.

By definition, a government targeting its own citizens, except very exigent circumstances, is not protecting its people. It's targeting them, what are they? Are they a saboteur? A patsy?

I think the FISA court is finding the circumstances being put through under these orders aren't very pressing issues.

A country's intelligence wants to stop spies, terrorists, and gain positive foreign intelligence, they don't want psychopaths who feign ignorance of memes getting their jollies off on destroying the lives of its own citizens.

Also, judging by the comments I read, I think users want a GDPR-like law to protect their personal information when dealing a consumer setting. For stuff related to OP [https://icontherecord.tumblr.com](https://icontherecord.tumblr.com) and its post relating to IC IG and FISA are US ways of oversight in that area.</comment>


Ask HN: How to set and achieve goals in small startup? - <AUTHOR> I'm founder of startup with small team of people (4 myself included). I want to ask HN, how you set up goals for the future? How far you plan in future and how detail. How you meter and achieve your goals, employers goals, team goals? <comment><USER2> Focus on getting users and keeping them happy. Pass every decision through that filter. Right now is the only time in the potential timelines of your company where all goals and incentives are so neatly aligned (between users, employees, founders, and investors).

You don't need any complicated goal maintenance frameworks right now. You need to get users, keep them happy, and repeat. Otherwise, you will die. You will never reach the point where you have divergent goals between entities. So right now you have a simple goal, you don't need to think about setting others. It's a simple filter that facilitates all possible legitimate goals of a tiny startup.

If you have any short or medium term goals right now aside from getting users, keeping them happy, and acquiring more resources to fuel that positive feedback loop; why are you even doing a startup?

</comment><comment><USER3> Have you ever read about the "Scrum methodology"? I think it is a good place to start...

[http://en.wikipedia.org/wiki/Scrum_%28software_development%2...](http://en.wikipedia.org/wiki/Scrum_%28software_development%29)

[https://www.scrum.org/Portals/0/Documents/Scrum%20Guides/Scr...](https://www.scrum.org/Portals/0/Documents/Scrum%20Guides/Scrum_Guide.pdf)

</comment><comment><USER4> Hi, is this a business development question or a software development question?

<comment><AUTHOR> Hi, Its business development question.

<comment><USER4> Okay well there isn't any right answer here and honestly what you are asking is extremely broad. Any advice is only valuable if it is tailored to your situation.

My biggest question to you is, "do you have a business plan?" Sure the thing gets obsolete the moment it hits the printer but I've found the discipline of writing one and doing the financials a very good thing.

If you don't have a business plan I would honestly start there. New Venture Creation: Entrepreneurship for the 21st Century by Timmons has an excellent Business Plan Template. In my younger years I won a business plan competition at my university with it. The book also has a good opportunity screening framework which is really helpful. Buy a second hand copy of the book and you don't need the latest edition. I have the 5th edition of the book and it is used time and time again when needed to do business plan stuff.

Once the full business plan is done, I have found that condensing it all down into a one page business plan for the start-up team members keeps everyone in the loop about the big picture moving forward. You will probably need to keep revising the one page business plan depending on what is actually going on. I wouldn't change the full business plan unless you are doing a major pivot or if any significant amount of time has passed (like 1 year).

Having worked through the full business plan you will have figured out where you want to go and what you will need to get there. How you move forward really depends on what type of business you are in, and what your business model is. You should have also done a bit of project planning in the business plan and I would have paid close attention to the critical path of your start- up (and maybe done a Gantt Chart with who is doing what).

Having said that I would pay close attention to your burn rate—or how quickly you are using up your cash. This is sometimes called your 'runway' for launch but I like to call it your 'death clock'. You must get to a point as quickly as possible where you are break even including a living wage for founders and staff. I think PG from HN has a blog post somewhere about being 'Ramen Profitable' which might be worth a read.

Anyhow having a 'death clock' can motivate you and your team to hit break even before you run out of cash and die. Fear is an excellent motivator in the short run. If you have a great team I don't think you need to really push them too hard beyond that because they will know what they need to do to get to break even before you die (if they don't what the heck are they doing in your start-up team?). Having said that you'll need good team communication and coordination and I would say weekly meetings with a sort of whip-around where everyone gets a chance to tell it how it is... something like:

A) This is what I did last week, this is what I'm doing this week B) This is what I worked, this is what didn't work C) I need help/I'm at full capacity/I have spare capacity.

Also try to get someone to summarize the team's progress against your critical path. Are you early, on track or behind? If you are behind what can the team do to get back on track?

If you have already hit the break even milestone, congratulations... you have a new type of problem set on your hands. You'll need to do the same sort of process but motivate with carrots rather than fear. This really starts to get business specific. If you give more info maybe I can offer more ideas.

Good luck!</comment></comment></comment>


How Good Are UW Students in Math? - <AUTHOR> http://cliffmass.blogspot.com/2010/01/how-good-are-uw-students-in-math.html

<comment><USER2> As shown on the scanned answer key, this was an anonymous quiz. College students have a tendency to put in pathetically minimal effort on things they're not being directly graded on. Also, the quiz's stated purpose was to help the teacher calibrate the assignments; maybe some students purposely answered questions wrong in hopes of dumbing down the class. (Though I don't dispute the author's central point.)

</comment><comment><USER3> The author says that UW students taking Atmospheric Sciences s101 "should be the creme of the crop of our high school graduates with high GPAs".

1) Atmospheric Sciences 101 sounds like a "blowoff" class that liberal arts students would take only to fulfill a science requirement. In other words, the students are likely self-selected for lack of math/science ability.

2) UW is a public university. It may very well be the best public university in Washington, but I'd bet the best students have a nontrivial rate of private school attendance. (So as to dispel any illusions of elitism, I attended the University of Michigan, which is also a public research university.)

<comment><USER4> _"UW is a public university. It may very well be the best public university in Washington, but I'd bet the best students have a nontrivial rate of private school attendance."_

Don't kid yourself. This is an epidemic problem at _all_ schools, public and private. I have friends who teach at both public and private universities, and they all have the same horror stories. My own personal experience (private undergrad / PhD at UW) bears out their anecdotes. With certain institutional exceptions, private school kids are richer, not smarter.

Also, the GPA and SAT scores of incoming UW freshman are _astonishingly_ high right now (IQR of 3.60-3.91 GPA, 570-680 SAT math, according to <http://admit.washington.edu/Numbers>). Admittedly, those aren't Harvard numbers, but one would still expect a 600 math SAT student to be able to do well on a basic high-school algebra quiz.

<comment><USER3> I did not disagree with the author's main point, merely the assertion I singled out.

<comment><USER4> Perhaps, but your argument was a challenge to the statement that the students represent the best and brightest of the state, and that's what I was addressing. When the test is as easy as this one, the argument that these students aren't elite isn't important.

</comment></comment></comment></comment><comment><USER5> I'm just as upset with the state of math education in the US as anybody, but if UW is anything like my alma mater, then "Atmospheric Science 101" is a class that liberal arts majors take to satisfy their math & science general education requirements. A lot of these students (my wife, for example) have serious math anxiety. She could probably get a 70% on this test, even today, so many years after school, but throw it at her without warning on the first day of a non-math class and she'd literally cry.

Now there's a case to be made that a good high school math curriculum should not leave anyone anxious about math, but I still think it's a bit of a leap to take this test and extrapolate to all students.

<comment><USER6> Same experience with my girlfriend. She has a good analytical mind (she graduated near the top of her law class) but she's just terrified of math.

The weirdest aspects of this phenomenon are that: a) there's a very real fear of math unlike any other subject and b) there's no stigma against being mathematically illiterate as there is for other subjects (most likely because so many people just accept that "math is hard" and not for them).

<comment><USER7> Is it any surprise, though? Math is taught in a manner that focuses very much on the _how_ but rarely ever the _why_. And while Discovery Math/Reform Math/etc are lambasted for not teaching kids to memorize certain algorithms, they do something else that's very important: encourage students to use creative problem solving skills to develop their own solution algorithms.

I'm not going to advocate reform math in its current form, but I have to question traditional "drill and kill" teaching methods in today's modern world. If I can't remember how to find the area of a triangle, I just Google it. Drilling it into my brain would have made sense in the 1920's when there was no internet, no assurances my peers had the appropriate level of education to know such a thing, and possibly no ability to reach an "expert" via telephone to tell me the algorithm. I'd be completely reliant on my memory and (possibly) a math book if I was fortunate enough to have one with me. For all practical purposes, memorizing one half times base time height would have been the way to go in a pre-Internet era. Today, it seems like it's far more worthwhile to develop students who are solid problem solvers with strong information retrieval skills, as opposed to wasting precious time memorizing trivial algorithms that are readily available.

<comment><USER8> _Is it any surprise, though? Math is taught in a manner that focuses very much on the how but rarely ever the why. And while Discovery Math/Reform Math/etc are lambasted for not teaching kids to memorize certain algorithms, they do something else that's very important: encourage students to use creative problem solving skills to develop their own solution algorithms._

The idea sounds really great, but everything i've ever seen is that the students just end up terrible at everything. I even tried to teach it some, and the material seemed great to me as a teacher, but the students just never got it. I feel like the discovery idea makes a lot more sense in hindsight, but when learning that way, doesn't work out like hoped.

However, I study applied math and I would say i learn in a more discovery way... But thats at the graduate level.

<comment><USER7> If students are ending up terrible at everything, then the problem is deeper than just tradition vs reform. What's likely occurring is we are much better at teaching plug-and-chug than we are at developing problem solving. That's to be expected, as it's significantly harder to develop problem solving skills than it is to simply memorize steps, grade, correct, and repeat until perfect. This is especially true when you consider most math teachers really aren't trained in the art of problem solving, and were likely taught in a traditional manner themselves. As such they have no real frame of reference to go off of when attempting to foster problem solving skills in their students.

</comment></comment></comment></comment></comment><comment><USER9> And I remember in France how my university professors used to complain that high school students became weak in set theory...

I didn't know that the level in the US had fallen that far... but it's true that I remember helping a girl in the nearby community college on her math classes and the level of the exercises she had was about the french 7-8th grade math.

But of course, I think the emphasis on math in europeans countries is maybe different than in the us. In France, for example, ability in math is the main selection criteria and students who took the scientific curriculum in highschool had 6-8 hours/week of mathematics and for calculating the gpa, mathematics had a coefficient of 7-9 (compared to 3 for english for example).

And because the scientific curriculum is considered the most prestigious a lot of students who intend to later study non-scientific specialities like law or business take it.

</comment><comment><USER10> Interesting premise, questionable conclusion. Kids can't do simple concept problems, so they need more drilling? Failure to calculate 1/.1 is not due to lack of practice, it is due to fundamental lack of understanding of fraction division. This is not a problem you fix via repeated long division of 4 digit numbers. This is a problem stemming from the fact that these students were never given a good idea of what the symbols they are being forced to move around really mean.

And who cares that they don't know the definition of cosine, or the formula for the area of a circle? These are not fundamental ideas for non engineers.

<comment><USER11> In my opinion, those are fundamental ideas for human beings in the year 2010; obviously we have to lower our expectations to match reality, but we shouldn't pretend they are unreasonable. Really, how difficult is the idea of a cosine? That's not exactly an unapproachable level of abstraction.

Remember, these are people who have had _12 years_ of full-time mandatory schooling, and have signed up for a few more years on top of that. If anyone with an average IQ cared even a tiny bit about such things, they could learn all of it in half that time; and if they don't care at all, parents and teachers didn't do their job.

Of course, I agree with the main thrust of your post. Somehow, you have to actually encourage (and allow) the students to understand what they're dealing with, instead of giving up, moving numbers and letters on paper, and passing to the next class. If that doesn't happen, there's no surprise that they've forgotten it six weeks after the course.

<comment><USER12> I agree that just drilling on the mechanics of long division doesn't help. But, drilling the concepts can help. The point of the article is that instead of letting kids discover everything on their own and then feeling happy, teachers should direct their learning. In doing so, I would say they should drill the same concept repeatedly and ask questions that test both the ability to do the procedure and understand the concept.

Anecdotally, this is how I taught my sister multiplication and how my friend tutored someone in calculus.

</comment></comment></comment><comment><USER13> In UW's defense, "Weather 101" is pretty commonly known as an easy class to fill up General Education requirements.

</comment><comment><USER14> Another lament of what's wrong with mathematics, by mathematician Paul Lockhart:

<http://www.maa.org/devlin/devlin_03_08.html>

He doesn't quite agree with this article.

</comment><comment><USER15> Actually, I'm pretty sure this problem is exclusive to Washington. Go here

<http://www.wheresthemath.com/>

and scroll down to 'Washington State Facts'. Anecdote: I attend a community college in California, found the problems laughably easy, and have met tons of people in my classes who would feel much the same way.

</comment><comment><USER16> I would bet these students got excellent grades in math.

EDIT: As in, I'm sure that during high school they got high marks in math class without actually understanding the fundamentals of what they were 'learning'.</comment>


We're in the Middle of a Data Engineering Talent Shortage - <AUTHOR> https://blog.stitchdata.com/new-research-were-in-the-middle-of-a-data-engineering-talent-shortage-bdd59673608c <comment><USER2> Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay."

I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where we start to part ways. If you are really serious about a shortage, you should be really serious about making offers that can be competitive, but I keep seeing the same $150k offers. That isn't a "shortage" kind of offer.

<comment><USER3> Although your statement is technically true, it is basically meaningless.

Yes, you can always always always find somebody to do a job is your a willing to pay 10 million dollars. That means that "shortages" are impossible. It means that you can never have a shortage in any situation, because you can always pay 10 million dollars for a single visit to the doctor.

But this line of logic isn't very useful when talking about "shortages".

If you had to pay a million dollars for a loaf of bread, is there a shortage of bread? IE, billions of people will starve to death by next week, because they can't afford to buy food.

Most people would say "Yes, there is a shortage of bread".

When people talk about shortages, they are obviously talking about a shortage at a certain price point. There is no other definition of the word shortage that makes sense.

A good definition that I use for the term shortage is "If the government could snap its fingers and instantly produce large amounts of X overnight, would the world be a better place"?

If the answer is "Yes, the world would be in a much much better place", then that means there is a shortage of X. If the answer is "No, the world would only be a little better". Then that means that there is NOT a shortage of X.

<comment><USER4> Haven't you just done a 180? I mean, I'm pretty sure the world would be in a much much better place if the government could snap its fingers and instantly produce large amounts of almost anything. Therefore there is a shortage of almost everything.

<comment><USER5> Given the millions of unemployed Americans, it seems this is not true for at least some occupations.

Wal-Mart greeters can be wonderful people and I'm not saying they aren't valuable as humans. But in labor market terms, there is clearly not a shortage of them.

</comment></comment></comment></comment><comment><USER6> We should rename this job position to Data Sanity Engineers.

I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually takes three months, because you have to go back and forth making sure the data is right, and there is always 10 tricky edge cases that you have to work some magic on. Not only that but you need to have smart people cleaning the data, so that you don't make some big mistake down the line or your REST API is super slow, and you have to add another couple weeks or month to rework the data again. So that one month becomes three months, and most likely a year, because somebody will say that looks great but can we also add this, and it goes on and on. It's literally a mind-numbing job that most nobody wants. I have found that products like Tableau are the best for this, you still have to clean the data, but it helps speed up the process.

Data cleaning is a super golden problem to solve.

<comment><USER7> As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing.

Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder, and it doesn't really do anyone service to look down on the work others find enjoyable. If doing D3 makes you happy, that is awesome, and I can only congratulate you for your passion and your ability to look forward to work I don't "get," and I wish the feelings would be mutual.

<comment><USER6> So I guess you are a data engineer? What makes it fun for you? How do work with your customers to give them what they need in a timely matter? I would be interested to know what stack you use to go from dirty data to customer consumption.

<comment><USER7> Closer to an aspiring data engineer, though I've done my fair share of ETL, cleaning, database building / rebuilding, admin. Prior jobs have been database engineer, probably closer to DBA.

I just enjoy working with raw data and raw code more than I enjoy writing something that launches a graphic. I enjoy writing a script that finds a bad piece of data, or a script that fixes up everything, or writing something that was once unable to run at all get converted to something that runs in 500ms. Perhaps it is that journey of constant discovery, and seeing that every situation is a unique little puzzle. It is seeing the world _as it is_ with no one reinterpreting what the data means for me. I can explore it and discover what it really means. It is hollow truth, a mess of ideas converted to sets of ideas layered on sets of ideas, and when it is finally drawn down, converted, and passing all tests, it is self-evident and self-reflecting, and true. Hard to explain, but I suppose I like all the things people hate about it.

The tools matter about as much as it matters what CSS framework you are using. You have the ability to logic through UI and UX, whereas I do not. I have zero hope of ever doing well at what you do, since I simply don't have the foundation, but if it matters, I know most jobs I've applied to and worked at tend to be more ad hoc, using PL, Python, Ruby, etc.

<comment><USER6> I'm not comparing frontend to backend. I also think data is fun and I don't mean to be little the job, but in a real world scenario its detail intensive, under appreciated, tons of edge cases and extremely complex if you plan to make it scalable and fast. So if you are an aspiring data engineer be aware of these pitfalls, because the first couple times you do it you will think its fun to try something new and create some fun useful analytics, but customers will often complain at how long it takes and want more. It starts to wear away at ones drive and passion for data. Its not the data aspect its the job/deadline aspect.

<comment><USER8> You're getting very close to the root cause - customers and even colleagues don't really care about the work that goes into the data. They care about the end deliverable, because that's what creates value for them, and fairly so. That gets at why data engineering as a discipline isn't (IMHO) very well respected.

I know this isn't reddit, so I'll point you to reddit. Check out /r/datascience where those folks talk about what it takes to be a data scientist. Some folks are honest about data engineering, but most handwave past it, or talk about it like it's beneath them. Their role would not be possible without solid data engineering, rather than a complementary and equally important discipline. Good luck doing "data science" or "analytics" or "machine learning" or every other buzzword without clean data, and for us data engineers, good luck ever demonstrating value without the analytics folks working with us.

</comment></comment></comment></comment></comment></comment><comment><USER9> From the article: "Data engineers are the janitors who keep your data clean and flowing."

Hm, I wonder why he's having problems hiring janitors.

<comment><USER10> Bizarrely, I remember a recent HN discussion where a poster was arguing that any software developer who is not working in machine learning is like a plumber.

I guess this means that the entire profession consists of janitors and plumbers.

<comment><USER11> Considering that plumbers and janitors have likely, in the entire history of human civilization, done more for health and longevity than doctors and scientists...I'm kind of ok with this analogy.

<comment><USER12> Doctors, maybe, but it was the scientists who told them about the germ theory of disease, for instance.

I've read, but not confirmed for myself, that in the US the biggest gains in health came in the post-Civil War period, when "plumbers and janitors" made the difference. Of course, that's really starting with, after the science, the civil engineers who designed the public works systems that supplied clean water and took away sewage, and let's not forget that politicians and like who found it worthwhile to buy votes that way (now, they take our infrastructure for granted and buy votes more directly...).

<comment><USER13> Confirmation: [https://ello.co/dredmorbius/post/MsQfdPAn_0XUdZUoReBfbg](https://ello.co/dredmorbius/post/MsQfdPAn_0XUdZUoReBfbg)

<comment><USER12> Thanks! There was a delay after the Civil War as you'd expect from all the chaos and disruption that caused (e.g. MIT got its charter before the outbreak, but wasn't able to start up until after), but it's pretty clear, and gets really dramatic the further you go forward.

<comment><USER13> I've been studying the period (mostly the Industrial Revolution and onward, though the accelleration of the late 19th / early 20th century is staggering), and it's pretty phenomenal.

There was a lot going on. Germ theory, of course, was part of it. But public health measures, especially sewerage systems, clean drinking water, and municipal waste removal, were all massive contributors. Note that the decline in mortality occurs _well in advance of_ antibiotics and even most vaccinations.

For all the recent debate on vaccinations, it's interesting to note that the peak period of their impace (roughly 1930 - 1960) saw relatively little reduction in _mortality_ , though there was a _large_ decrease in disease _incidence_. It turns out that with septic control, antibiotics, food quality, and nutrition, many viral diseases weren't killers, but _did_ present quality- of-life issues. And yes, often quite severe -- polio was no joke, and I know people who've suffered lameness from it myself. Measles and smallpox are similarly scarring and have long-term impacts.

But the major impacts of virtually _all_ medicine are front-loaded to the period _before_ 1950, with much the gains _since_ attributable to either greater access (especially for the disadvantaged) and removal of environmental agonists (lead, tobacco, alcohol, asbestos, miscellaneous poisons, safety hazards).

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Ignoring the breathless nature of the article, this is a buzzword label for a commodity skill set that pays a commodity salary in tech. It is also the commodity skill set that my employers have all paid me for.

There has been for a _long_ time hype around new technology and labels for business intelligence, data warehousing, big data, and now data engineering/science. I'm not saying there are not some roles in this space that return huge value to organizations, but that these opportunities are much rarer than the buzz indicates.

I wonder if the perceived shortage is mainly hype as the shift to new cloud technologies makes many of the older ideas a little less useful - if you are plowing data into BigQuery, you probably aren't so worried about your star schema data model for reporting.

I would strongly advise people that look at these types of articles to look at the roles in question and ask "Is this role on the critical path to customers paying us?" My experience has been that the answer is often "No." This is bad. I have also seen situations where businesses that do rely on smart data integration can show that they are selling dollar bills for ten cents that _still_ have trouble getting customers on board with spending that ten cents. Business is weird.

</comment><comment><USER15> I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

<comment><USER16> My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment.

Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data team and very useless if it does not.

That said, you may want to move closer to my role. There's actually a shortage of data-savvy people who can also write production software, and you would nicely complement a more research-inclined data scientist or analyst -- someone with far more experience with research/analysis than development.

<comment><USER17> > There's actually a shortage of data-savvy people who can also write > production software, and you would nicely complement a more research- > inclined data scientist or analyst -- someone with far more experience with > research/analysis than development.

I experience the same problem with shortage-at-price-X in the field you describe. I'm a machine learning engineer with experience in MCMC methods, but I also have a lot of low-level Python and Cython experience, some intermediate experience with database internals, and lots of experience writing well- crafted code for production systems.

There are basically zero companies willing to pay what I'm seeking (which is a salary based on my previous job and a few offers I got around the time I took that job). In fact, in some of the more expensive cities, the real wage offered is far _lower_ than other markets.

I've seen reputable, multi-billion dollar companies offering in the $140k range for this type of role in New York. That's wildly below anything reasonable for this sort of thing in New York. I've seen companies in Minneapolis offering $130k for the same kind of job -- and even _that_ is still too low for Minneapolis! The same has been true in San Francisco as well.

Because these companies value you more for simply looking good on paper and looking good as a piece of office ornamentation when investors stroll through, and they view you as an arbitrary work receptacle closer to a software janitor than a statistical specialist, their whole mindset is about how to drive wage down.

Frankly, given the stresses of the job and the risk of burnout, I think it's actually a terrible time to be in the machine learning / computational stats employment field, despite all of the interesting new work and advances being made. The intellectual side is good, but the quality of jobs is through the floor.

<comment><USER18> "I've seen reputable, multi-billion dollar companies offering in the $140k range for this type of role in New York. That's wildly below anything reasonable for this sort of thing [in NY/SF"]

Man, do I ever agree. This is where the "shortage" argument falls apart.

This is why I'm so uninterested in the abstract arguments happening elsewhere on this topic about whether markets are failing and basic laws of supply and demand no longer apply at theoretical salary levels (10 million was offered as an example).

Why are we bothering with this debate, when it's so far from reality? I'd say that if you're trying to hire a very high skilled and critical tech worker in SF, and you just can't find one no matter how hard you try, and then I find out that you're only offering 140k a year?

In San Francisco and New York (and anywhere else in the US, really), that's nowhere close to the kind of pay where we should start scratching our heads about a shortage and start wondering why the usual laws of supply and demand aren't working anymore.

</comment></comment></comment></comment><comment><USER19> Every time something comes up on HN about a talent shortage in a field related to software engineering, it hurts. I have been unsuccessfully looking for a full time position since my last start up (I was not a founder) folded six months ago. I have been on over 25 in person interviews and gone through untold degrading whiteboard interviews, code tests, trick questions, and take home projects; all have ended in rejection. This industry has a need to torture candidates because we are all considered to be liars by default. Much is said about combating impostor syndrome in ourselves but we are too eager to engender it in others.

It seems people in this industry refuse to understand that some people are not perfect. I never graduated college because I hated it with the very fiber of my being, so I am not particularly great at white boarding answers to algorithm questions off the top of my head in a high pressure environment. If I need them during my job, I look up answers and learn from people who are much smarter than I am.

My personal identity has been shattered, as I thought my ~5-10 year history of success in the industry indicated I was in demand and talented. I saw posts like this and thought that if the worst happened I'd still be able to find a job. The idea that there is a talent shortage is a lie, or candidates like me wouldn't be treated as I have been. I'm not asking for a free job, or a handout. I have had a successful career so far and am capable of doing good work. But I'm not a specialist in Big Data Machine Learning Neural Networks.

I have struggled with bipolar disorder and suicidal ideation most of my life. I've dealt with the death of my beloved grandmother and my father who was instrumental in my choosing to be an engineer with only minor lapses in control. Nothing has caused me to consider taking my own life as much as the past 6 months. It seems there is no future for me in the only career I have any skill in and which is a huge part of my identity. And to constantly be told that there is such a shortage of engineers only salts the wound.

<comment><USER20> " I have been on over 25 in person interviews and gone through untold degrading whiteboard interviews, code tests, trick questions, and take home projects; all have ended in rejection."

The fact that you pulled through 25 of them is already commendable. Unfortunately as a labor provider you'll be subjected to all kinds of crap for the privilege of working.

Every single person on here needs to have a secondary business going on right now. Doesn't have to be a highly skilled industry either, selling hand made stuff on Etsy can be a lifeline in these situations.

</comment></comment><comment><USER21> I've heard more than one CTO/Sr. Engineer refer to people in these roles as 'data grunts' or something similarly dismissive. Then they're mystified as to why solid engineers are so quick to move up or out, year after year.

</comment><comment><USER22> anything and everything is marketed as "data science" and "data engineering" these days becasue this is the buzzword of the day.

I've been dealing with large data even before "big data" was a word but i dont call myself "data scientist" or "data engineer". I am still a software engineer working on what benefits my organization.

"Serial Entrepreneur" is the same these days, claimed by anyone who had a lemonade stand as a kid.

<comment><USER23> > I am still a software engineer working on what benefits my organization

But if you saw a nearby local maximum that's higher than your current local maximum, wouldn't you change what you call yourself, if it means being paid more but doing the same work?

This is similar to how the average "software engineer" makes about $30k/year more than the average "programmer".

</comment></comment><comment><USER24> It's digital Charlie Work [0], that's why.

I really enjoy that kind of work but it is difficult to articulate your business value in that environment. The best thing is working closely with a data scientist/front-end dev who can deliver products to the analysts and executives that need the data and make sure that you get the credit for enabling new streams of data. But most of the time you are putting out someone else's dumpster fire.

One advantage of data engineering: unlike front-end work, there are few non- technical people who will have an opinion on how you are doing things and burden you with bikeshedding.

[0] - [http://www.avclub.com/tvclub/its-always-sunny- philadelphia-c...](http://www.avclub.com/tvclub/its-always-sunny-philadelphia- charlie-work-214628)

</comment><comment><USER25> There are 6600 jobs listed and 6500 individuals on LinkedIn with that particular title, and therefore there's a shortage? Seriously?

* How many aren't on LinkedIn?

* Since the whole article is about how the job title is poorly defined and growing in prevalence, why would you assume that people who don't already have such a job would use the term?

* The "growth" charts on the full study are just as bad - how much of that is just from renaming existing generic developer positions, since "data engineer" is clearly a relatively new term?

<comment><USER26> 6500 data engineers on all of Linkedin, but 6600 job openings in the bay area. so there are more job openings in one area than all data engineers on linkedin

</comment></comment><comment><USER27> The fact that the original, unmodified article referred to data engineers as "janitors" pretty much says it all.

It's very analogous to front-office and back-office work in Investment Banking. "Data Scientist" are the front-office, with all the prestige, and "Data Engineers" are the back-office, doing a lot of the heavy lifting without nearly as much recognition.

In my opinion there shouldn't be a delineation. You shouldn't be a data scientist if you can't gather, process, and clean up your own data.

<comment><USER28> Ideally you'd have a symbiosis, and each side would recognize the importance of the other.

Even if you require your data scientists to be able to do engineering work, it's probably way more efficient to have some good generalist Software Engineers doing all the "pre-math" work and freeing your statisticians up for what they're (hopefully) good at.

Plus as a side effect, your software will probably be better.

</comment></comment><comment><USER29> Data engineering sounds much better than "data plumbing", but in my experience the latter is a more accurate description of the work of a data engineer: Building -and often unclogging- pipes that transport data from A to B, and putting in filters to clean it and extract the useful bits.

So why not change your LinkedIn job title to "data plumber", which is sure to get you some serious recruiter attention ;)

</comment><comment><USER30> Ahh the ol' write a post about a not well understood distinction and then proceed to not explain the distinction.

Looks like we need more English engineers too.

</comment><comment><USER31> I'm puzzled at the omission of Scala and Spark in this report.

</comment><comment><USER32> I worked for about 10 years doing exactly what they want, but I ended up having to write a lot of the tools which means I'm not able to check the boxes on some tool you require which gets me punted by HR.

I'm starting to think that the message is if HR is going to do checklists then developers should really make sure they work mostly with contracts that use popular checklist items.

<comment><USER33> As a data person I would really like to put some numbers on how much the typical HR hiring process costs a business. I don't know anybody that says they are happy with how hiring works in he tech industry but I've also never seen an HR person try and improve the process.

<comment><USER34> That's because the system is already optimised for the needs of HR people.

</comment></comment></comment><comment><USER35> Quick sidenote, anyone know where the databases / distributed systems engineering jobs are at? E.g. if one wanted to not use these tools but also go help build these tools?

I can think of Facebook, Google, Microsoft, IBM (which locations and groups within these companies / where?). I can also think of Confluent, CitusDB, Databricks, etc.

<comment><USER36> Market Research is a $40B industry that depends almost completely on these concepts. I'm not sure how prevalent distributed systems are with MR companies, but that's an implementation detail anyway.

<comment><USER37> > that's an implementation detail anyway.

Which is what the poster was asking for.

</comment></comment></comment><comment><USER38> Weirdly the problem is most hires have it backwards.

Before going out to the market and discovering what talent exists and consequently what salary it will take to get them to join (ie negotiate) _most_ organisations decide on a salary range, usually reflecting the current internal structure not the current external market.

The longer an organisation has existed the more out of whack with the market its internal set up is.

As such companies decide on their price point first, then go looking. Which is of course backwards.

</comment><comment><USER39> Am I the only one who thinks there will be a ton of people changing their job title on LinkedIn to "Data Engineer" as a result of this article?

<comment><USER40> I am thinking about it. Actually a friend recommended that I change my title to Data Engineer a few months back.

</comment></comment><comment><USER41> We surely need data mechanics.

</comment><comment><USER42> These "shortage" stories always make me roll my eyes, because they're usually about money more than anything. And money is usually about cost of living more than anything.

If you choose to locate your company in one of the highest cost of living regions in the world, then you are complicit in the "shortage". Supply and demand - pay up. Or don't.

</comment><comment><USER43> I am a data engineer working on a machine learning team with models actively used as part of our product(s).

From my experiences working in various contexts (applied machine learning, analytics, policy research, academics, etc...), there are several of factors that contribute to this shortage: (1) "data engineering" often requires a lot of breadth and knowledge, (2) "data engineering" is often (derisively and naively) referred to as the "janitorial work" of data science, (3) the spectrum of roles and requirements within the "data engineering" domain, in terms of job descriptions, can range from database systems administration, to ETL, to data warehousing, curation of data services / APIs, business intelligence, to the design/deployment/operation of pipelines and distributed data processing and storage systems (these aren't mutually exclusive, but often job descriptions fall into one of these stovepipes).

Some of my quick thoughts and anecdata:

Companies have made large investments in creating 'data science' teams, and many of those companies have trouble realizing value from those investments.

A part of this stems from investments and teams with no tangible vision of how that team will generate value. And there are several other contributing factors…

"Dirty work." People haven't learned how to, and more often don't want to do it. There's a vast number of tutorials and boot camps out there that teach newcomers how to "learn data science" with clean datasets -- this is ideal for learning those basics, but the real world usually does not have clean or ideal datasets -- the dataset may not even exist -- and there are a number of non- ideal constraints.

There are people that wish to call themselves "data scientists" that "don't want to write code" and would "prefer to do the analysis and storytelling"

Engineering as the application of science with real world constraints: there are a number of factors that we take into account, often acquired through painful experience, that aren't part of these tutorials, bootcamps, or academic environments.

Many "data scientists" I've met have a hard time adapting to and working with these constraints (e.g. we believe that the application of data science would solve/address __ problem, but: how do we know and show that it works and is useful? what are the dependencies, and costs of developing and applying that solution? is it a one-time solution, or is it going to be a recurring application? does the solution require people? who will use it? what are the assumptions or expectations of those operators and users? is it suitable? is it maintainable? is it sustainable? how long will it take? what are the risks involved and how do we manage them? is it re-usable, and can we amortize its costs over time? is it worth doing? This is part of a methodology that comes from experience, versus what is taught in data science)

Larger teams with more people/financial/political resources can specialize and take advantage of these divisions of labor, which helps recognize the process aspects of applying data science and address some of the above

Short story: if you view data engineering as "janitorial work" you're missing the big picture

Anyone else notice that the attributes of a 'unicorn' data scientist include the traits of a 'data engineer?'

<comment><USER44> How does one get started with this? I suppose a lot of people who hang out at HN are competent devs good in programming and databases, but probably beginners in math, ML, AI etc. How does such a person get started and find a job in this field?

</comment></comment><comment><USER45> It was only 20 years ago that companies hired a "web master" or a generalist to do everything. But pieces of those jobs became specialized. Now we need UX, UI programmer, general engineers, dev ops, data engineers, a data scientist, etc.

And how many companies are still interviewing with fizzbuzz?

</comment><comment><USER40> So I know SQL, Python, Django, Java (though its been a while), Javascrit, Linux, some cloud computing and a bit of devops. Am I a data engineer? Software engineer, with a lot of database background? What makes a data engineer different from a software engineer?

<comment><USER46> \- The challenge for an organization is to recognize that there is a significant difference between the 'data engineer' working on a vertical project and the 'data engineer' responsible for integrating data across the enterprise.

\- The project 'data engineer', in today's world, most likely will be a software developer responsible for ETL, etc. The data design will be more or less up to the software developer.

\- An enterprise 'data engineer' is more concerned with data that affects the enterprise. This typically involves some sort of data integration. For example, how to integrate relevant data from N projects (e.g. A,B,C .. Z) where each project has its own idea of how to represent similar concepts (e.g. person, user, customer), with different provenance, truth assertions, access rules, data retention periods, granularity of metadata (e.g. at the attribute level vs entity level), etc. The enterprise is interested in questions like 'What did we know and when did we know it?", etc. The enterprise 'data engineer' will probably levy requirements on the project 'data engineer' to meet the enterprise's needs.

</comment></comment><comment><USER47> Just checked, the # of data engineers rose to 9,246 (42%) in the last six months. So, the shortage is at least being addressed by people changing their job titles on LinkedIn.

</comment><comment><USER48> What I've learned from the comments: If something is valuable, there is a shortage of it.

I'm not even sure if I'm being sarcastic.

</comment><comment><USER49> We hire only the best! We only hire the top 1% of candidates.

But only 1 out of 100 are qualified :(</comment>


RxGo: Reactive Extensions for the Go Language - <AUTHOR> https://github.com/jochasinga/RxGo <comment><USER2> I'm a fan of the Rx programming model, but is it really worth it to introduce such a library and model into a language with such an opinionated and built-in model for streaming/pipelining and concurrency?

<comment><USER3> If you don't want to use this nobody will force you. The overwhelming and unfortunately predictable heap of negativity that comes bearing down on people sharing something cool they made is really awful.

Well someone has made this, and now you can decide if it's worth it for you as an individual to decide whether you like it or not and then if you don't, then don't use it.

<comment><USER2> I find it incredibly ironic and hypocritical that you accuse me of creating a negative environment... in a negative reply to a fairly neutral question.

I don't use Go, but am a contributor to a Rx project for a different language. As such I am not interested in using this but genuinely curious about the potential (non)benefits of Rx in Go, especially given that they both promise to solve similar problems in very different ways. From experience HN is the prime place for such discussions.

I think HN would be a much worse place if we would only be allowed to cheerlead projects here.

</comment></comment></comment><comment><USER4> On the one hand this seems like a strange decision given that Go purposefully avoids asynchronous code. On the other hand, I think this project is trying to get at something that is (IMHO) a blind-spot for the Go community. Namely: I find it difficult to design abstract data-flow patterns and encapsulate behind a nice API.

Go is great for dealing with the nitty-gritty of concurrency, but I find it hard to express a high-level vision of what's going on in my app/lib. This is why this project _almost_ feels good. There's a clear view of what we're manipulating and how we're positioned relative to the data.

Has anybody else struggled with this?

The closest thing I've found to a solution has been go-mangos[0], a native go implementation of the scalable protocols. This is extremely useful for designing the data-flow of an application when I'm shuffling bytes around, but what I find myself wanting is an equivalent system for raw (i.e.: unserialized) Go data-structures.

I want to be able to do something like this (assume a hypothetical library to this effect called `portal`):

        p0, _ := portal.NewPortal(iface, portal.REP)  // iface is a types.Interface
        p0.Bind("foo")

        p1, _ := portal.NewPortal(iface, portal.REP)
        p1.Dial("foo")

        go func() {
            for {
                i, _ := p0.Recv()  // `i` guaranteed to satisfy the interface represented by `iface`
                i.DoSomething()
                p0.Send(i)
            }
        }()

        p1.Send(iRequest)
        iResult, _ := p1.Recv()

I'm very curious to get some feedback on this idea. It's been gnawing away at my brain for a while now.

[0] [https://github.com/go-mangos/mangos](https://github.com/go-mangos/mangos)

</comment><comment><USER5> Rx not encoding side effects in the type system, as ReactiveSwift does, is very unfortunate.

</comment><comment><USER6> Isn't this what channels are for? Doesn't Go have this already built in?

<comment><USER7> Not really. Channels have a single function, to transport data to the first available consumer.

With observables, there's a whole world on top of that. The simplest observable, when subscribed to, will cause the producer of content to start the production. If 3 consumers subscribe to an observable, then the producer will do it's work three times, once per subscription. Compare that to a channel, where a producer has no way of knowing if a consuer is available. Generally, they start the work immediately and try to stuff the result in a channel, regardless of whether there's a consumer on the other end. Moreover, when another consumer uses the same channel, you have two consumers fighting for the same result.

And since observables are transformable, there's a plethora of operators available (at least in rxjava). If you take the previous example, you might want to share the observable, replaying the last result, so that regardless of how many consumers subscribe to it, the producer will only do the work once, and the consumers will receive the same results. It will be quite a bit harder, though not impossible, to implement the same thing with channels.

</comment></comment><comment><USER8> FWIW not exactly related, but check out Go-LINQ as well. [https://github.com/ahmetalpbalkan/go- linq](https://github.com/ahmetalpbalkan/go-linq) (disclaimer: my project). I will take a look at combining RxGo and Go-LINQ and probably will write a blog post soon.

</comment><comment><USER9> The examples lack the elegance of RxJS. I'm gonna stick with the Channels for a further while.

</comment><comment><USER10> Why is Rx so huge? What does it give you that Promises dont?

<comment><USER2> Promises are to Streams what single values are to arrays/collections.

Promises are all fine and dandy for "do this, wait for that, do that", but they are not powerful enough to express "every time this happens, do that".

I guess you're coming from a JS background as you are using the word "Promises". Imagine instead of registering a callback for mouse click events, you have a stream of mouse events where you can listen to. This is exactly how it works in Angular 2 [1].

[1] [http://blog.rangle.io/observables-and-reactive- programming-i...](http://blog.rangle.io/observables-and-reactive-programming- in-angular-2/)</comment></comment>


Code Review: Easier Better Faster Stronger - <AUTHOR> https://ckl.io/blog/code-review-easier-better-faster-stronger <comment><USER2> Code review is indeed a powerful tool to both learn and share your experience</comment>


"Nerd" and "Geek" Should Be Banned, Professor Says - <AUTHOR> http://bits.blogs.nytimes.com/2009/12/21/nerd-and-geek-should-be-banned-professor-says/?partner=rss&emc=rss <comment><USER2> I think the last sentence says it all:

He is an expert on the subject, and the reason I called him for the piece was that I had noticed the praise for his book, "Nerds: Who They Are and Why We Need More of Them."

So... he has no interest in "banning" these words, he wants attention and for more people to buy copies of his book.

<comment><AUTHOR> Full disclosure: I took this guy's class in college.

I don't think that wanting to symbolically "ban" a word, and writing a book about its effects on our culture, are mutually exclusive.

</comment></comment><comment><USER3> Ah, Bennington. Pick up a copy of Randall Jarrell's novel _Pictures from an Institution_, where it figures as "Benton" College and see whether the place has changed.</comment>


From Sentiment Analysis to Emotion Recognition: A NLP Story - <AUTHOR> https://medium.com/neuronio/from-sentiment-analysis-to-emotion-recognition-a-nlp-story-bcc9d6ff61ae <comment><USER2> The expectations that we had 15 years ago, about the leaps and bounds in research and progress in this area, has not happened, maybe it's a good thing...</comment>


Playing with Keybase.io (2016) - <AUTHOR> http://nishanttotla.com/blog/playing-with-keybase-io/ <comment><AUTHOR> Keybase seems like a really good implementation of the idea of identity verification, but I feel that it needs a lot more users to make it more useful for everyone.</comment>


Whistleblower claims LDS Church stockpiled $100B in donations, dodged taxes - <AUTHOR> https://www.sltrib.com/news/2019/12/17/whistleblower-claims-that/ <comment><USER2> You'd very likely find that a Church with only a hundred billion stashed away is merely scratching the surface.</comment>


1.6 million Catalans form human chain 250 miles long for independence from Spain - <AUTHOR> http://www.washingtonpost.com/world/europe/catalans-in-favor-of-breaking-away-from-spain-form-huge-human-chain-to-advertise-their-cause/2013/09/11/0ce45f06-1afa-11e3-80ac-96205cacb45a_story.html

<comment><USER2> Official sources from Spain are particularly well known for bogus counting when it comes to protests, etc. I suggest to take this with a pinch of salt

<comment><USER3> German news also showed smaller numbers. For what it's worth, 1.6e6 people on 250 miles means everybody had only about 25cm (<1 foot) of space. A realistic number for a human chain would be about 1m (approx. 3 feet).

Seems bogus.</comment></comment>


Ask HN: Hosting a Java Spring web application - <AUTHOR> A self-taught newbie here who would like to know where to host his Spring web application. This is more than a hobby application. Do I go for the cloud and, in that case, is Pivotal Web Services the best? Or, do I go with a VPS host? In the latter case, which host would be good for a non hobby app for a newbie? Last time I was here, someone suggested Webfaction. At the moment, I am not sure whether that would actually be a good idea. <comment><USER2> Because it's more than a hobby, I'd suggest looking into Amazon Web Services (AWS), in particular, Elastic Beanstalk - checkout the links here for more details:

[http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create...](http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Java.html)

</comment><comment><USER3> For me, the deciding factor is what data services you need. If you need a database, then on a VPS, you'll have to install and manage it yourself, whereas on AWS you can use RDS, and on a PaaS you can use whatever managed database the PaaS provides.

If you don't need any data services, then a VPS is as good a bet as AWS; deployment is much the same, and not substantially harder than on a PaaS.

<comment><AUTHOR> The application is essentially going to be a front-end for a database. Thanks, PaaS it is going to be. I have since been looking at Elastic Beanstalk as well as Pivotal Web Services and OpenShift. I will now have to make a decision along those lines.</comment></comment>


Room 641A - <AUTHOR> http://en.wikipedia.org/wiki/Room_641A <comment><USER2> > [The class action lawsuit against AT&T] was dismissed on December 29, 2011 > based on a retroactive grant of immunity by Congress for telecommunications > companies that cooperated with the government.

What an overreach of power on the part of Congress. Something goes to court to discuss its legality, and then Congress retroactively declares it to be legal? Yikes.

<comment><USER3> Well no: Congress is well capable of granting retroactive immunity. The constitution restrains it from retroactively criminalizing things, so that you can be prosecuted for something that wasn't illegal when you did it; but retroactive decriminalization happens fairly frequently and is an essentially liberal action. Why should a private actor be burdened for the perceived misdoings of government? It's not up to corporations or individuals to regulate the government, from a legal standpoint.

<comment><USER4> > Why should a private actor be burdened for the perceived misdoings of > government?

But why does this have to be ensured through the action of the Congress? If the misdoing was the Government's, it is up to the courts of law to absolve the private actor of guilt.

Granting immunity of private actors for government illegalities is a very dangerous precedent. Imagine if the same was to be done if a private security company fires on civilians on behalf of state orders, and is then absolved of responsibility through an act like this one.

When a private authority does something on behalf of state authority, the state effectively gives away some of its responsibility; when it decriminalizes it later, the part of responsibility that it has given away is now vanished. In some cases, of course, the state can still be sued over the initiative, but such a measure ensures that companies are sheltered in case the whole thing goes awry, so there's little incentive in not collaborating.

What does suck about this in terms of ethics is that it's hard to draw a line between when retroactive decriminalization is beneficial and when it isn't. After abolishing slavery, it obviously makes sense to decriminalize slaves' attempted escapes. But does it also make sense to decriminalize racist behaviour?

Edit: just to be sure, I'm arguing over principles here, and all I have to bring is historical arguments, not judiciary ones, since I neither am a lawyer, nor do I reside in the US. If I missed something due to unfamiliarity with the US law, I apologize in advance :-).

<comment><USER5> "But why does this have to be ensured through the action of the Congress? If the misdoing was the Government's, it is up to the courts of law to absolve the private actor of guilt."

Courts don't absolve anyone of guilt. They merely determine if a law has been broken. It is congress that makes law, so it is up to congress to construe the law such that the private actor has not broken anything.

<comment><USER4> But does this not open the way for arbitrariness? At this rate, given enough support in the congress, the executive power can pretty much do anything -- the congress will just decriminalize it afterwards.

Edit: What I meant in previous post was that a court of law has the possibility of absolving private actors or responsibility, the same way they do with someone who has committed an illegal action under blackmail: they are still guilty of having broken a law, but there are obvious reasons why they should not suffer the full penalty of it (or any penalty at all), and legal action moves against the blackmailer.

</comment></comment></comment></comment></comment><comment><USER6> I've been saying this for years. Pretty much all I've said over the last few days is "I told you so".

My biggest concern is come a week everyone will go back to not caring and I'll go back to pestering people to give a shit.

<comment><USER7> What do you propose people do?

</comment></comment><comment><USER8> I had never been to SF before last summer. This building was right across from the bus I took home every day. And every day I wondered: what was in that windowless building? Never was it not an imposing sight.

Even after I found out about this, it freaked me out; especially that it's right in the heart of downtown.

<comment><USER9> Is this the building?

[http://www.mccullagh.org/db9/1ds-8/611-folsom-street- nsa.jpg](http://www.mccullagh.org/db9/1ds-8/611-folsom-street-nsa.jpg)

Is that just very horrid architecture, or some kind of shielding? Perhaps both?

<comment><USER10> It's a bit a lot of things. I can't find when the building was built, but I'm guessing the 60s/70s, so pretty much the height of brutalist architecture. Basically, very hard geometric forms, heavy on repetition, lots of concrete. There's a lot of that type of stuff sitting around, especially on university campuses.

It appears to be have been actually designed for use as a telephone switching center, for which windows actually are mostly useless. Should take a look at [http://en.wikipedia.org/wiki/33_Thomas_Street](http://en.wikipedia.org/wiki/33_Thomas_Street), another telephone exchange in NYC. It is literally a window-less skyscraper.

<comment><USER11> Most big cities have a communications building like this.

My favorite one is in NYC: 33 Thomas Street. It looks like it was designed to be the capitol of a dystopian megalopolis.

[https://upload.wikimedia.org/wikipedia/commons/a/a8/33_Thoma...](https://upload.wikimedia.org/wikipedia/commons/a/a8/33_Thomas_Sidewalk_View.JPG)

[https://upload.wikimedia.org/wikipedia/commons/7/74/33_Thoma...](https://upload.wikimedia.org/wikipedia/commons/7/74/33_Thomas_Street_December_1991.jpg)

Speaking of dystopia... I bet it houses a spy room just like the one in SF!

<comment><USER12> Reminds me of the film Equilibrium [http://en.wikipedia.org/wiki/Equilibrium_(film)](http://en.wikipedia.org/wiki/Equilibrium_\(film\)) very imposing building!

</comment></comment></comment></comment></comment><comment><USER13> The talk page for the PRISM article[1] is also very interesting.

> Why is it all caps? Some other reason? It should be noted in the article > somewhere...

And:

> Everyone is losing their minds over this thing, but as far as I can tell, > data is only provided with a "legally binding order or subpoena". This isn't > the NSA reading through every inbox in the world. Or am I wrong?

Once more:

> The denials by the participating companies are not surprising and may, in > fact, be a requirement under the National Security Letters, which the NSA > and FBI use to give quasi-legal cover to their signals collection.

[1] [http://en.wikipedia.org/wiki/Talk:PRISM_(surveillance_progra...](http://en.wikipedia.org/wiki/Talk:PRISM_\(surveillance_program\))

<comment><USER14> The page itself notes, curiously:

> A related program, a big data or data mining system _based on cloud > computing and Free and Open Source Software (FOSS)_ technology known as > "Boundless Informant", was disclosed in top secret documents leaked to the > Guardian and reported on June 8, 2013.

(my emphasis)

So, after we've had licenses claim (in one way or another) "do not use this software to wage war", will we see a rise in "do not use this software for secret intelligence gathering or processing" licenses?

Still, both are rather pointless when the entity you're trying to subject to the terms of your license has a standing army and heaps of nukes. Seriously. This is why we can't have nice things.

</comment></comment><comment><USER15> From Matt Bors:

"There's a line I've seen trotted a few times now that goes, "Ah, we knew all this was happening, you idiots!" We didn't know. If you knew about PRISM you should have scooped The Guardian and made a name for yourself. The fact is, this story is mammoth and is bringing an incredible amount of attention to a program that's been operating in total secrecy for a decade. But, yes, you're smart and get a pat on the head for not being a "sheeple." "

[http://www.mattbors.com/blog/2013/06/07/prism- thoughts/](http://www.mattbors.com/blog/2013/06/07/prism-thoughts/)

<comment><USER16> PRISM is dramatically less interesting than this.

<comment><USER15> Care to enlighten us as to how? And by "us" I mean President Obama (who was pretty much forced to comment on it and certainly didn't say anything like you did), those in the administration going after the leaker(s) (for if this is uninteresting, the leakers aren't in the wrong at all), and those people who marked the slides as TOP SECRET. Oh and the German data protection and freedom of information commissioner [1]

What of the others programs in the slides? Two even had to be redacted out because of the potential of the fallout against NY-based Guardian US

[1] [http://www.reuters.com/article/2013/06/07/europe- surveillanc...](http://www.reuters.com/article/2013/06/07/europe-surveillance- prism-idUSL5N0EJ31S20130607)

<comment><USER16> This thread concerns unlimited access to the entire internet.

PRISM concerns limited access to a few massive internet companies.

</comment></comment></comment></comment><comment><USER17> The discovery of Room 641A is more profound than PRISM. Every web request, text message, phone call... it goes through these hubs where the data is copied and saved. PRISM is just about getting access to data sent over secured HTTPS connections, from what I can tell. It's not (currently) economical for them to use computing power to break (all of) those codes, and it's easier to apply pressure to tech companies to just hand them the keys.

<comment><USER18> Yes, that must be exactly what they're doing. Siphon all the unencrypted data at the ISP level, and gain access to the rest of the data that is encrypted from the biggest companies - and then expand the "program" to more companies in the future.

And to think that they rejected the idea of UN doing the same thing (the ITU debacle last year). It was only so _they_ are the only ones doing it, all along.

</comment></comment><comment><USER19> "...The room measures about 24 by 48 feet (7.3 by 15 m) and contains several racks of equipment, including a Narus STA 6400, a device designed to intercept and analyze Internet communications at very high speeds.

Author James Bamford raises the suspicion question of an Israeli backdoor to the Israeli Narus in The Shadow Factory 2008.

</comment><comment><USER20> From that link, the guy who leaked it was a "computer technician" for twenty years. Is that real, or was there a effort to discredit him or something. Twenty years is a long time in the same role.

<comment><USER21> Yes it's real. It's amazing to me how quickly this room, ECHELON, Carnivore and all the rest have been forgotten. There seemed to be a small but vocal technical uproar over this rooms existence when it was discovered but the public at large ignored it. Then Congress retroactively made it legal so there was nothing we could even do about it at that point.

<comment><USER22> I think it is basically a form of semantic satiation. Mention something controversial enough times and eventually the public stops understanding that it was real. I wager that if you took a poll of Americans, most would believe that ECHELON is a fictional plot device in spy movies that "conspiracy theorists" think is real because they are dumb.

</comment></comment></comment><comment><USER23> [http://en.wikipedia.org/w/index.php?title=File:SER_marcus_de...](http://en.wikipedia.org/w/index.php?title=File:SER_marcus_decl.djvu&page=17)

</comment><comment><USER24> This suite was just below Twitters main HQ for years

<comment><USER25> Different building. Twitter was at 795 Folsom (at 4th) which was also an AT&T building. This building is at 2nd and has very few windows and does not appear to be occupied by any office tenants (I walk by it every day).

</comment></comment><comment><USER26> It's surreal to watch all of this old stuff get drudged up here and elsewhere.

I almost wish I watched mainstream news to see how much of this depth is covered or if it's just PRISM and not anything about how bullshit FISA warrants are or how crazy insane National Security Letters are (and how many hundreds of thousands of them are issued)... OR that they regularly wiretap the internet.

Between:

\- this (which we've known about)

\- the new leaked slide that indicates they do massively, widely wiretap fiber

\- PRISM (which I suspect were simply the dates that the companies brought up infrastructure to be able to quickly and easily comply with FISA warrants. Which means "direct access" is true, but it also means that "we were forced to make it easy to give the government what they want as soon as they need it")

They basically have real time access to HTTP data (and HTTPS if a CA is compromised or controlled by them [is it really even that crazy of a theory at this stage]). They also can get organized, categorized access to my data with a FISA warrant (please, read up, they are as rubber-stamp as they're referenced in these articles).

Sigh.

<comment><USER26> [joke]Now the slide that needs to leak is the one that lists compromised root CAs.[/joke,sorta]

</comment></comment><comment><USER27> The EFF is protecting us more than the NSF. Donation sent.

<comment><USER28> Strictly speaking it's not really the job of the National Science Foundation to protect us.</comment></comment>


Ask HN: anyone interested in going to Thailand to work on a startup? - <AUTHOR>

First of all: not looking for a co-founder, but if you'd like good company - read on.
I'm working on my own stuff now (quit my job) and I figured going to Thailand to live for 6 months would actually save me some money. I already lived there for 6 months and it was great. I had a studio in a modern building with a gym and a pool on the roof with a beautiful seaview, which was perfect because I didn't have to actually waste my time to get to the gym. And it all costed me around $400/mo. In my opinion Thailand may be the ultimate location for a startup's stealth mode, allowing to fully concentrate and relax when needed.
What I'm looking for is good company, a technical person who's bootstrapping a startup and with whom for that reason I potentially have a lot in common. I think it's important to have at least one person to talk to about startup and technical problems. We could rent two rooms in the same or nearby buildings and... well... work! Occasional beers and discussions as well as island trips are also on the list. Like I said, I lived there for 6 months, so I know the place.
My contacts and bio are in my info. Please ask me anything here, be happy to answer. <comment><USER2> Why Thailand though? From my opinion, Malaysia is a much better place for you to find talent, and enjoy your life a lil bit. If you willing to spend a lil bit more then try Singapore.

For good company during travelling, try couchsurfing. It's big everywhere in Asia.

<comment><AUTHOR> Screw couchserfing, people there have a radically different set of values from mine, as I found.

Malaysia is wonderful, I've been there once. However it's a bit more expensive and I don't think I can afford it currently. Then there are visa problems and my friend also told me, that agents are very reluctant to rent an apartment for less than a year there, which is not really acceptable for me.

</comment></comment><comment><USER3> _if you'd like a good company_

 _What I'm looking for is a good company_

I think you mean "good company", not "a good company". Just in case anyone was confused.

<comment><AUTHOR> Yes, thanks for the correction.

</comment></comment><comment><USER4> By coincidence I am also in South-East Asia (Indonesia but building a presence in Malaysia). I do not know exactly what you are interested in doing and I am more of a Perl/Python guy than a Ruby guy but would be interested in talking.

<comment><USER2> Where you're at?

</comment></comment><comment><USER5> Was always tempted to do the same. Heading there next week.

How'd you sort out the visa? Extending each month, or doing a visa run every 30 days?

<comment><AUTHOR> I used to go to Cambodia & Malaysia to get 2+1 months visas. What place in Thailand are you going?

</comment></comment><comment><USER6> I'm in a similar situation (quit my job, building a startup), but I choose Taiwan.

</comment><comment><USER7> I always wanted to work/travel to Thailand.. but am afraid of the ladyboys.</comment>


GitHub after Microsoft: How it has changed - <AUTHOR> https://www.infoworld.com/article/3335256/github-platform/github-after-microsoft-how-it-has-changed.html#tk.rss_applicationdevelopment <comment><USER2> Interesting perspective. I'm tempted to move back to GitHub following their allowing free private repos, but I'm also hesitant to trust Microsoft. Might just stay with Bitbucket ..

</comment><comment><USER3> Moving away from github "cause Microsoft" strikes me as an odd move fueled by 90s and early 2000s spite. I'm sure there are legitimate technical reasons to choose gitlab over github. But for a mature product and development team I saw no need to make the switch in our organization. The sky still hasn't fallen.

<comment><USER4> The reason to ditch a specialized product after it is acquired by a large, old user hostile firm is because they have the ability to turn the heat up slowly. They can wait you out.

<comment><USER5> And if/when they do, you leave once the temperature is too hot.

Doing it before is just premature optimization.

<comment><USER4> You'll be too entrenched by then and your org will be too dependant. At that point, you'll use fancy mental gymnastics to justify the, for example, impossible to disable telemetry. Get out while you can.

</comment></comment></comment></comment><comment><USER6> Maybe Microsoft has left Github doing its own thing, but they're still using their operating systems to massively spy on users. I cannot help but think Github is not longer neutral: its business used to be about code repository and only that, but now it's also about all the things Microsoft does and wants to protect. Actual and potential competitors legitimately can wonder how private their code is.

</comment><comment><USER7> At least they're not Oracle ...

</comment><comment><USER8> [https://outline.com/VWkneB](https://outline.com/VWkneB)

<comment><USER9> That doesn't show the full article either. Cuts off where the real article cuts off too.</comment></comment>


Kilroy Is Still Here - <AUTHOR> http://www.theparisreview.org/blog/2015/11/12/kilroy-is-still-here/ <comment><USER2> Fun read. I am of course biased, as I have a collection of Kilroy pictures: [http://woodruffw.us/media/kilroy/](http://woodruffw.us/media/kilroy/)

</comment><comment><USER3>
      You're the one who
      must decide who's
      to live and who's to die.
      You're the one who gives his
      body as a weapon of the
      war—and without you all
      this killing can't go on.

That's from the Donovan song "Universal Soldier" (original by Buffy Sainte- Marie). Looks like some historian published it as a "mystery poem" in the Smithsonian magazine[1], not realizing it was just quoting song lyrics.

¹ [http://buffysainte-marie.com/?p=809](http://buffysainte-marie.com/?p=809)

</comment><comment><USER4> Nice read ! The author could just as well used code comments as one of the many examples ;-)</comment>


Greek crisis stokes Bitcoin prices higher - <AUTHOR> http://www.cnbc.com/id/102796314

<comment><AUTHOR> Coinbase is apparently seeing a 300% increase in euro/btc exchanges from users across europe in the last few weeks, but not much from Greece itself.

[https://twitter.com/coinbase/status/615949087829946368](https://twitter.com/coinbase/status/615949087829946368)</comment>


Ask HN: Share your interview / hiring practices? - <AUTHOR>

What does your interview process look like? How do you decide if someone is the right fit?
We do a phone interview to cover the basics, a tech screen + tech interview and a "do at home" coding exercise. This combination ends up cutting out more than 90% of the people we talk to, but bad seeds are still making it through. In the future, we're going to change our process to have an "in-office" coding exercise, but I would like to be considerate of people's time.
How do you assess the people that apply with you in order to improve the chances of hiring well? <comment><USER2> I wrote about the other side of it (the interviewee) here [http://rawsyntax.com/post/6112439871/interviewing-for- progra...](http://rawsyntax.com/post/6112439871/interviewing-for-programmers)

FWIW, I don't think coding exercises help. I think the employer should require and example project that runs, and the source code to it. Many good developers have a side project that they can show off.

<comment><USER3> I've interviewed many people who looked great on the resume and failed to do even a trivial coding question. The good developers quickly wade through such a question without a winch which leaves plenty of time to discuss interesting projects and see how they think.

</comment></comment><comment><USER4> I think that a) people need to make time when you're hiring. Like 20% of their time. and b) firing quick.

Hiring isn't foolproof, and if it's a priority, then yes, it'll eat in people's time.

</comment><comment><USER5> I'm also interested in this, but as an interviewee. Personally, I like in person interviews better than phone interviews, which I've found to be very formal and unpleasant.</comment>


Threatening Lawsuit, HBGary Federal Intimidates Aaron Barr Out of DEFCON Panel - <AUTHOR> http://threatpost.com/en_us/blogs/lawsuit-threat-pushes-former-hbgary-federal-ceo-out-defcon-072711

<comment><USER2> I find this one the most worrying:

"In 2008, the Massachusetts Bay Transportation Authority (MBTA) obtained a temporary restraining order to prevent a talk at DEFCON by three MIT students who had uncovered physical and logical security holes in MBTA infrastructure."

I mean, the IOS guy worked for IOS, so he was divulging information that Cisco paid him to obtain. HBGary paid Aaron Barr to research anonymous, so they have a case for him doing the same thing. However, what kind of case could MBTA possibly have against some kids who found problems with their infrastructure? I fail how to see how such an injunction was anything but unconstitutional.

<comment><USER3> _However, what kind of case could MBTA possibly have against some kids who found problems with their infrastructure?_

The flaws they found allowed them to obtain, according to their talk's description, "free subway rides for life", which would deprive MBTA of an unknown amount of revenue which was rightfully theirs. They wouldn't disclose the vulnerability when MBTA approached them, although they changed the description of the talk to no longer imply they would explain how to get free rides.

You can read the MBTA complaint yourself: <http://www.universalhub.com/images/2008/mbtamit.pdf>

<comment><USER4> Really? I thumbed through the complaint (thanks for the link), but I'm still really disturbed by this. It's presumably illegal to help yourself to free subway rides. The law protects the MBTA. How is it a crime to talk about how it could be done? The talk wouldn't deprive the MBTA of any revenue.. the application of information in the talk by third parties committing illegal acts would. It's akin to giving a talk on lock-picking, no?

</comment></comment></comment><comment><USER5> I believe this falls into "doing it wrong".

Backing out because they _threaten_ to file an injunction is to capitulate too soon. You're supposed to keep going until/unless they actually get an injunction, and then challenge it (plus, civil lawsuit for loss of income due to the injunction once it's rescinded).

BlackHat/BSidesLV/DEFCON is going to be really interesting this year, for a change.

<comment><USER6> _You're supposed to keep going until/unless they actually get an injunction_

I do not know if you have ever been involved in a lawsuit, but they can be enormously time consuming and ver expensive. In short, even if you are in the right they can play absolute havoc with your life.

<comment><USER5> I think he's already past the point of deciding where he'll be involved in a lawsuit or not -- holding out to force them to get a court order to bar the talk wouldn't make that materially worse.

I agree in a lot of cases about (especially individuals, especially people who aren't very wealthy) backing down from legal challenges, even if they have a good chance to ultimately prevail, due to the cost.

For a startup, getting involved in a major lawsuit can be a huge distraction too -- so even if it the startup wins the lawsuit, the company still is stunted and dies.

</comment></comment></comment><comment><USER7> Why would DEFCON want Aaron Barr on any sort of panel? The guy is at best a hack of a security professional and at worst a complete idiot of a slime ball who doesn't understand the first thing about technology.

<comment><USER8> It's sort of like staring at a car wreck as you pass by. Doesn't mean you approve of car wrecks or are glad they happen, but you can't help it.

<comment><USER9> I'm not at all a computer security professional(considering getting into the field, though) but I can't help but wonder if this mars DEFCON's image by having this guy there.

I see your car accident analogy, but Aaron's probably accepting money to be on the panel. Why should they pay him when they know his reputation is tarnished?

</comment></comment></comment><comment><USER10> Kinda reminds me of this:

[http://news.infracritical.com/pipermail/scadasec/2011-May/01...](http://news.infracritical.com/pipermail/scadasec/2011-May/019934.html)

the more I see in the security world, the more I think full disclosure is the only responsible way to work towards a more secure system.

<comment><USER11> What geeks think of as "full disclosure" is thought of as "cyber crime" or "cyber terror" by some companies and agencies. Things have changed a lot in the last five or ten years.

So, be cautious and get legal advice if you do decide to disclose anything, and understand that while geeks mainly just want to share what they found and help fix it, that others may accuse you of being a criminal, or worse a terrorist if you do that.

<comment><USER10> _How_ is information a crime if you're not in the government service?

No, seriously!</comment></comment></comment>


Zombie (Programming Language): Programming Up to Congruence [pdf] - <AUTHOR> http://www.cs.yale.edu/homes/vilhelm/papers/popl15congruence.pdf <comment><AUTHOR> > The ZOMBIE language aims to provide a smooth path from ordinary functional > programming in a language like Haskell to dependently typed programming in a > language like Agda. However, one significant difference between Haskell and > Agda is that in the latter, programmers must show that every function > terminates. Such proofs often require delicate reasoning, especially when > they must be done in conjunction with the function definition. In contrast, > ZOMBIE includes arbitrary nontermination, relying on the type system to > track whether an expression has been typechecked in the normalizing fragment > of the language.</comment>