Why NSA Surveillance Scares Me - <AUTHOR> http://bentilly.blogspot.com/2016/01/why-nsa-surveillance-scares-me.html <comment><USER2> I agree with the author wholeheartedly, except for the mathematical analysis at the end.

I believe one of the existential threats to our country is becoming a dictatorship. I'm always shocked when I share this point of view with people around me and their reaction is basically, "that's so unlikely it's not worth worrying about". There are many examples of this throughout history, e.g. the Roman Republic, the Weimar Republic, those that the author cites, etc. We should not be so arrogant to think that we are somehow exceptional and immune to this.

My personal belief is that there are certain constants in the dynamics of human nature, and one of them is seek power. Broad, unchecked domestic surveillance is a dangerous tool to give to a potential dictator. We should not set up a framework where a power-seeking individual can circumvent our democracy through coercion.

<comment><AUTHOR> I deliberately made my analysis overly conservative because I didn't want to come off as a paranoid nut. I could have easily come to much higher figures.

Here is an example. Let's suppose that we're observing the USA at a random point in its history. Then there is a 10% chance that we're looking at it within the last 10% of its history. The USA turns 240 this year, so that means that there is a 10% chance of it collapsing in the next 24 years.

That figure is much, much higher than the 1% odds I used of collapse in the next 30 years. It is also likely to be more realistic. "Stable governments" survive in periods measured in centuries, not millennia. However that number is so high that it would have left me more easily dismissed as a paranoid nut, so I avoided it.

History demonstrates that humans tend to over-estimate the stability of whatever exists now. And are surprised when the present turns out not to be permanent. You should try to compensate for such perceptual biases when you find them in yourself. And should try not to trip such biases in others.

<comment><USER3> Can you explain the math as to how you know there is a 10% chance that we're seeing the US in the last 10% of its history? I don't understand that part.

<comment><AUTHOR> If we assume that we're observing at a random point in our history, we are equally likely to be observing at a point in the first 10%, second 10%, third 10%, and so on. That's what the assumption means. Which means there is a 10% chance that we're observing in the last 10%.

See [https://en.wikipedia.org/wiki/Doomsday_argument](https://en.wikipedia.org/wiki/Doomsday_argument) for the most famous use of this line of argument. (It concludes that the human species is likely not long for this universe.)

<comment><USER4> What's stopping us from applying this math to other things? Like, every baby at the 1 minute mark of their life. Shouldn't that mean that ~10% of all babies born dies within a minute and a half?

We can take this to extremes. At the 1 minute mark we have a 99% change of being in the latter 99% of their life (and thus a 99% chance of dying 99 minutes later). It's amazing how many of them beat these odds.

<comment><USER5> It's an uninformative prior ([https://en.wikipedia.org/wiki/Prior_probability#Uninformativ...](https://en.wikipedia.org/wiki/Prior_probability#Uninformative_priors)). Meaning it's the best estimate you can make, if you don't have any additional information about the problem. Obviously we have a lot of additional information on human lifespans, which allows us to build a more accurate probability distribution.

Also see my comment above.

</comment></comment></comment></comment></comment></comment><comment><USER6> The problem with this analysis is that the problem set the NSA exists to address is not limited to terrorism. Most of the work they do is spying on foreign governments in order to inform U.S. policy. Building a cost/benefit analysis of the loss of privacy effected by NSA operations on the number of lives lost to terrorism does not provide an accurate measure of the value delivered by the national-level SIGINT enterprise.

With regard to the author's general point: we should indeed be very concerned with the potential for widespread surveillance to abet the violation of civil liberties or even a totalitarian takeover.

<comment><USER7> > Most of the work they do is spying on foreign governments in order to inform > U.S. policy.

Really? I'm not disputing this, but I'd love to see some evidence backing it up. What I remember from the Snowden disclosures is the vast majority of the leaked information actually did relate to violent threats to national security. Am I remembering wrong, or did I miss something?

<comment><USER8> The NSA was founded in the early 1950s to be the primary signals intelligence (SigInt) branch of the US Government, largely to gather information which would be of strategic importance in the Cold War. You might look at the NSA as a SigInt counterpart to the National Reconnaissance Office (NRO), which runs most of the spy satellites and other strategic imagery. The Wikipedia article on the NSA is quite good.[1]

[1] [https://en.wikipedia.org/wiki/National_Security_Agency](https://en.wikipedia.org/wiki/National_Security_Agency)

</comment></comment></comment><comment><USER9> On the off chance that it might clarify the thread a bit:

Is there a single non-throwaway commenter that disagrees with 'btilly on this?

Not, like, disagrees about whether it's the _most important_ NSA concern, but disagrees that the gravity of this concern is outweighed by the importance of the work that NSA is doing.

To be sure: there are _tons_ of people who disagree with him around the US. But I'm surprised if many of them comment here.

<comment><USER10> The issue is that we do not have the full data set. We only see those cases where the NSA fails and select cases of success. Most successes, if there are any, are hidden to maintain secrecy in operations. I'm not saying there are any successes. What successes we do see are ones that are 'scrubbed' of intelligence or were made with intel that is now known to be useless and only then years after the fact. Again, if there are any successes to begin with. To make an educated conclusion, we need the facts, something we cannot have. Things could be worse or better, we do not know.

<comment><USER11> This is a fatally utilitarian response. You want to be able to calculate whether an equilibrium lies between good and bad outcomes of a process. That is a somewhat defeatist analysis to make because it implicitly denies the gaping moral and ethical risks of the system.

<comment><USER10> Yes, completely. But to make an assessment of moral risks, you have to have the data. Something we are unlikely to get. Everything else is just conjecture, perhaps well founded and correct, but still conjecture. Perhaps then the issue is with over classification of intel, not the people in charge. We may never know as a consequence.

edit: In the end, it is reasonable to assume we do not have adequate data on the successes of these programs. 2 paths then arise. 1) Then you wait for more data. 2) You examine those people that you reasonably know to have access and take a look at their moral history and actions to determine the ethical issues; it's the only somewhat good data you can add to the set to maybe get a conclusion.

<comment><USER11> No, you're missing my point by saying "you have to have the data," and it appears that you are missing some fundamental concepts in normativity.

You do not need data to make a moral argument based upon principle. There is no conjecture involved. (You could read some Kant on this, if you like.) You need data only to determine a particular consequentialist ethics that involves a balance of good and bad outcomes often understood in lay terms as "effectiveness." I think most commenters here would agree that effectiveness is the wrong way to evaluate this system, since epistemologically we cannot know its effectiveness. (... Although many doubt it could ever be truly effective in the presence of creative adversaries.) We can only know that is it categorically wrong.

</comment></comment></comment></comment></comment><comment><USER12> I agree with the dangers of an executive branch that has access to nearly unlimited information on everyone, especially when you consider the surveillance of Congress, governors, or Supreme Court justices.

The math doesn't sit quite as well with me. As someone recently mentioned, the projected threat of terrorism cannot be effectively measured by deaths to-date because those numbers can suddenly grow exponentially in the event of an attack. The probability of extra surveillance is baseless, etc. The points are valid, the statistical argument is weak and unnecessary.

<comment><USER5> I disagree. There is a lot of evidence that terrorist groups are incredibly ineffective. Not just recent attacks in the US, but statistics world wide and through all of history. Gwern wrote a detailed essay on how incompetent terrorist groups are: [http://www.gwern.net/Terrorism%20is%20not%20Effective](http://www.gwern.net/Terrorism%20is%20not%20Effective)

>9/11, the crowning incident of terrorism in those centuries, was equaled by just 29 days of car accidents in the US - and 9/11 was only _accidentally_ that successful! 9/11 is also a sterling example of the availability bias: besides it, how many attacks could the best informed Western citizen name? Perhaps a score, on a good day, if they have a good memory; inasmuch as the MIPT database records >19,000 just 1968-2004, it's clear that terrifyingly exceptional terrorist attacks are just that. Remarkably, it seems that it is _unusual_ for terrorist attacks to injure even a single person; the MIPT database puts the number of such attacks at _35% of all attacks_. Certainly the post-9/11 record would seem to indicate it was a fluke...

>Many terrorist organizations keep very detailed financial records (consider the troves of data seized from Al-Qaeda-in-Iraq, from Bin Laden's safehouse, or Al Qaeda's insistence on receipts), with little trust of underlings, suggesting far less ideological devotion than commonly believed & serious principal-agent problems. Stories about terrorist incompetence are legion and the topic is now played for laughs (eg. the 2010 movie Four Lions), prompting columnists to tell us to ignore all the incompetence and continue to be afraid.

Also see his essay _Terrorism is not about Terror_ : [http://www.gwern.net/Terrorism%20is%20not%20about%20Terror](http://www.gwern.net/Terrorism%20is%20not%20about%20Terror)

<comment><USER13> Some more examples of clueless morons being 'terrorists' are the 'underpants bomber' from a few years ago, and the idiots in Glasgow that crashed a car containing some propane cannisters into the airport terminal.

Not so long ago such ludicrous incidents would have been written off as being the misguided actions of 'nutters'. Now of course they are 'attacks' by 'terrorists', prompting hyperbole about 'heightened terror threat levels' and Al-Qaida cells.

<comment><USER12> >Some more examples of clueless morons being 'terrorists'

Don't forget the Four Lions

</comment></comment></comment></comment><comment><USER14> To summarize:

Give me six lines written by the most honest man in the world, and I will find enough in them to hang him.

— Cardinal Richelieu

<comment><USER15> If that's your position, then does it matter if they surveil you? There is doubtless already far more than six lines of your script publicly available to any who might desire it, at any time. Logging your activity, then, does not increase your risk.

My point is they already have all they need, publicly available any time they desire it. _You_ write it for them and publish it for them! It's fanciful to imagine privacy protects you, IMO.

<comment><USER14> The point of the quote is that those who claim that you don't need privacy if you have "nothing to hide" are wrong, because the power always can abuse information it gets, as long as it wants to target someone.

 _> My point is they already have all they need, publicly available any time they desire it_

Which doesn't mean you need to be lax and give them even more, or even accept such situation as normal. Don't feed the machine. To put it differently, the argument that "privacy is dead, so ignore the issue" is invalid and actually only facilitates the problem.

<comment><USER16> The point of the quote that many miss though is that Cardinal Richelieu wasn't a clairvoyant, and by "six lines written" he _didn 't_ literally mean Twitter. It's true that, as you say, "the power always can abuse information it gets, as long as it wants to target someone". If it wants to get you, it will abuse _any_ information available. If you protect your Facebook from them, they'll break your e-mail. If you encrypt your mail traffic, they'll tap your phone. Or pull out census and employment data. Or simply ask around.

I agree that you may be marginally safer if you're not "feeding the machine". But let's not delude ourselves - if the government turns evil, we're fucked either way, and no amount of Internet privacy will help much. There's too much data out there, and even more in the physical, material world.

<comment><USER14> _> If it wants to get you, it will abuse any information available_

That's exactly the point. Therefore helping that abuse (by either being lax with privacy, or by assuming it's pointless) is not something that you should be doing if you care about this issue.

 _> But let's not delude ourselves - if the government turns evil, we're fucked either way, and no amount of Internet privacy will help much._

It's not an argument to ignore this. That's the whole point of pushing for FOSS, privacy respectful technology, no DRM and so on. The more pervasive the technology is in society (and it's surely becoming so), the higher is the potential of that abuse if said technology is prone to it. So taking care to prevent it on all levels is essential if you care about it as an individual.

<comment><USER16> I agree about FOSS/no DRM and abuse potential. I think that the biggest problem here is not surveillance itself, but the power asymmetry it creates. There are two possible options for an (idealized) solution - either no one gets that power, or everyone does. The first one may be more desirable, but I think it'll turn out that the latter is more practical / achievable.

</comment></comment></comment></comment></comment></comment><comment><USER17> As the author notes, we've tread this ground before when J. Edgar Hoover essentially ran the country via intimidation/blackmail for decades. It always shocks me that Hoover's reign is mentioned so lightly. Democracy wasn't functioning during those decades because one unelected individual had hugely outsized power.

NSA surveillance is a backdoor to democracy because it enables blackmailing of politicians or inconvenient people. The CIA does this all the time abroad using information the NSA gathered.

</comment><comment><USER18> I have began to change, or at least think more deeply about survelience and the role of gov't.

I still am pretty much the same mind of the author, and blanket survelience is intrusive, ineffective and anti-anerican; both literally and in sentiment.

However, and I still not sure what role I trust the gov't to play here, the need for defense is pretty strong.

Let's take this red test for example:

While $100k is a lot of money to me, we will assume many people could have access to this amount.

There were > 1 million people in times square on NYE.

The strike cabanilities of an inclined individual are higher now than ever. This is the negative sode effect of the otherwise positive shift in individual autonomy and lowering of power distance between entrenched heirarchies and individuals.

A flock of drones can be purchased and controlled remotely and outfitted with bombs or guns and used to conduct an attack previously reserved for the u.s militaries indiscriminate bombing raids on other countries.

Analytical tooling and conputation can be purchased cheaply. Trend analytics, macroshift evaluation and near real time satelite imagery can be had for little money.

Biohacking is quite new but even after the overhype of the synthesis of (i think it was) smallpox from discrete labs, this is still semi-possible.

So, I don't think it is up to the government to fix this, especially with blanket survelience, but we do need to come up with a reasonable way to mitigate threats.

5 people and 100k could basically outfit a safehouse, a drone fleet, real time intelligience and a small bio labratory giving those who would not use that amazing luxury for good, the ability to create a high impact negative change event.

<comment><USER19> You set a scenario for a potential disaster but what has surveillance done to protect those in times square on new year. This question has no answer and the answer if it exists is NOT something a democracy has access to.

So if surveillance is an answer to a potential terror plot then what needs to be known is whether it can actually be effective. Terror plots are not the norm of human life. They are feared more because we humans are designed by nature to be afraid first when we see a lion, run away and think later. If we stood there contemplating the possibility of of having a deep conversation with that lion that lion would have managed to have a meal deep in its stomach. Thought is very low on mother nature's agenda. Even other great apes don't comment here! Basically we are more afraid of the danger because we are primed to be so. I guess what I'm trying to say is that while fear is a danger is a logical thing it shouldn't be the only thing to consider when deciding on surveillance. Currently it is the only reason presented to the public and it is not good enough on its own.

<comment><USER18> I agree and, except for the "terror plots aren'the norm" conclusion, im with you.

What I am driving at is that while the blanket "people" who watch fox news and are scared of brown people label terroists as myslims, i am proposing a different idea of "distuptive event".

It had nothing to do with edgecases like 9/11\. Look at it like gun violence. I think people should be able to own guns, but the more people own guns, the more ahootings there are. If 0 people had guns, there would be 0 shootings.

So, suppose you were (i am) willing to accept x amount of ahootings to keep this right and we know some people will legally pr illegally obtain guns and do ill with them.

My point, is that eventually, people will have the same access to the strike capabilities of small governments in the 90s for very little money. I believe we should retain these rightts, we will have casualties to such freedom, but we also need to expect that as peoplehave the ability to affect global change positively, they do negatively as well.

Whena group of small people have the power to go to mars for example that is great, however similarly many peoplewill have access to satellites in > 10 years. So we need to be pretty cognizant of how we handle this.

</comment></comment></comment><comment><USER20> Either you stand by your principles when they're tested, or you don't have principles.

</comment><comment><USER5> This is the best argument I've read on the subject. Before I really didn't worry about the NSA. I mean I didn't think they were doing much good, since I think terrorism isn't a significant threat. But I'm not that cynical or conspiracy-theory-ish to worry that they would abuse their power significantly. This is the first time someone's tried to show that's a real threat with examples from US history.

<comment><AUTHOR> Thanks. Reaching people like this was exactly my hope.

It is easy to preach to the choir. It is much harder to produce an argument that can reach people who don't alread agree with you.

</comment></comment><comment><USER21> I think it's important to differentiate between countries of our present era and countries that "fell" in past eras, when evaluating or speculating on the potential for longevity...

The technologies of today, especially the ease and rapidity with which information can be exchanged by private citizens, seems to me at least, a hindrance to an "overthrow" of our government...a freely functioning "Fourth Estate", journalism--the press, also acts as a safeguard...

At the point where either of those are verifiably co-opted I'd become very concerned...

Surveillance, per se, doesn't worry me so much...tens of thousands of covert operations are in play every day around the globe...those in play to guard against terrorism on U.S. soil are likely just the tip of a vast iceberg...

</comment><comment><USER22> I hope I am wrong, but I can't imagine a modern day Hoover, with better surveillance, to not have already placed blackmailed politicians in all key political positions and is effectively ruling the United States.

</comment><comment><USER23> Governments and more specifically democracies evolve over time. They benefit from all the ones that have come before them and failed. It could be that the inevitable conclusion of any democratic society is a last ditch power grab made by the unlucky person standing in office when if all falls apart. It's happened quite a few times before, and I don't see any reason why this time around would be different. I for one welcome our new overlord Trump and look forward to the new government that forms in his wake.

</comment><comment><USER24> > Surveillance sliding into dictatorship is one of the most common ways that > democracies fall.

What? Give me one example where surveillance was the _reason_ and main tool for ascending dictator.

<comment><USER25> In Nazi Germany the Stasi was quite effective in monitoring people, suppressing the people and keeping them in constant fear.

Maybe not the main tool for Hitler to reach his position as Fürer but it was certainly one of the main tools to keep him in power.

<comment><USER26> The Stasi was East Germany's post WWII security apparatus. You're confusing it with the Gestapo.

I understand and agree with what you were probably trying to say ;)

</comment></comment></comment><comment><USER16> A HNer, whose identity shall remain undisclosed, posted an insightful extension of the statistic games that later got deleted, probably because of the downvotes. I believe this is an important thing to consider, so let me take the karma hit and restate the problem.

The poster pointed out that if we're taking a 3000 years view, it's worth considering the great filter hypothesis[0]. If one is to believe it's ahead of us (as an explanation for why we haven't seen any signs of aliens yet, and mind you, this is not a joke), then it stands to reason we'll likely fuck ourselves over with some future technology. Given that, mass surveillance may be a way to avoid this fate.

Also, the poster pointed out that "the debate is silly. This level of surveillance is inevitable."

\--

The comment I was about to post in response:

Actually, I do agree, for two reasons.

1/ SIGINT is basically inevitable, barring total collapse of technological civilization. We're only starting to be good at data gathering and processing, and there is no reason to suspect we won't be getting much better very quickly. Look - 100 years ago we could barely record a person's voice. Today, we use sound to acquire pictures. We use light to acquire sound. Between cheap microphones being good enough to pin-point the position of a firearm discharge in a big city and your mobile phone about to be equipped with a 3D geometry sensor and _a fucking radar on a chip_ , with NASA being able to map ocean floor using _gravity sensors_ , that by the way happen to be Cold War-era technology, does anyone really believe we can stop mass surveillance? It's a side effect of useful high-tech tools. You basically can't have one without the other.

2/ Progress of technology also keeps increasing the amount of power a single person can wield. Throughout almost all history, the biggest damage an ordinary individual could do to other people is to set something on fire. Burn down a building, maybe. 100 years ago, an ordinary citizen could at best drive someone over, or blow something up. In the last few decades, bombs had become easy, and people were worried about terrorists with crude nukes. Today, a teenage hacker could do anything from blowing up a gas station to shutting down power in a large area. _Tomorrow_ , your high-school friend may be playing with potentially dangerous viruses in their basement.

This is not a joke. We're entering an era where playing with self-replicating machines is getting very easy very quickly. Today it's not just big biotech companies doing that - your local hackerspace is most likely doing baby steps with reprogramming bacteria too.

Now tell me, honestly, _how are we going to defend against that_? And keep in mind - it's not the angry that are the problem. It's the crazy ones.

Surveillance will probably have to be a part of the answer, but I fear it will be far from enough.

[0] - [https://en.wikipedia.org/wiki/Great_Filter](https://en.wikipedia.org/wiki/Great_Filter)

<comment><USER19> On the 2 points you raise:

1/ I agree with you on the premise that it is somehow inevitable but it is a lawless arena today.

2/ Also the single person does end up having more capability but also protection from them is also more possible.

But the problem with it is that in its current form surveillance is secret and the decision of action based on the surveillance is done in secret. There is an exclusion of democracy in any country when it comes to surveillance. Laws governing response to surveilled information are non-existent. This means we don't as a people have known effective means of using surveillance nor can we come to know unwanted side effects. Also citizens of a democracy are treated as unqualified individuals to know about surveillance. There used to be a caste system in the past of people who were worthy and those who are not. Kingdoms, dictatorships and communist states are the classical examples. Today the caste system is whether you belong to the surveillance community or not. Yes we need it but we need it to be governed by rules. Surveillance is a product of wanting to have an upper hand on the international landscape however we are discovering that it is slipping into domestic situations like the parallel construction cases exposed by Snowden. So while surveillance is inevitable and necessary it really is the wild wild west of government.

<comment><USER16> > _2 / Also the single person does end up having more capability but also > protection from them is also more possible._

I would like to believe that, but for now if it comes to biohazard events, I'm not reassured. If we had good enough mitigation strategies today, people generally wouldn't catch communicable diseases so much as they do. And I haven't heard of any potential research on something that could tip things in favour of defense. Again we encounter the same dynamics that make airport security theater ineffective - the assymetry between attack and defense. The defender has to handle _all_ possible attack variations, while the (either human or random in nature) attacker can just focus on one strategy, a strategy that will not be known to the defender beforehand.

For now, the best short-term strategy is infrastructural - basically what the health services are doing today, but even more of it.

I totally agree with you about present surveillance situation being a total Wild West. I have no clue what to do with it, and while I don't really mind data gathering _per se_ , I'm very much against the shenanigans governments and private companies are pulling on us with said data. We desperately need to figure out a good solution.

I think that one obstacle on the road to that solution will be attempting to protect the idea of privacy at all costs. I think we'll end up having to give up a big part of it, or even the whole concept entirely - in exchange for reducing the power it gives to the governments and corporations. For instance, you can't really blackmail people with stuff everyone knows everyone does, because nobody will care.

<comment><USER19> From a digital point protective measures such hack proof system can be _expensively_ engineered [https://www.phoronix.com/scan.php?page=news_item&px=MTc1MTg](https://www.phoronix.com/scan.php?page=news_item&px=MTc1MTg) but biology is a little bit trickier. I can think of biosuits of the future being like what we see in SciFi which is currently unrealistic but still plausible. I don't know where future protective technologies can go but there has always been some form of measure and countermeasure being developed in parallel one as a byproduct of the other.

As for privacy, I think we don't have as much privacy as we think we do. The thing that technology introduced is the recording of our already public moments. Now they can be played back at someone else's discretion and that someone else can control the narrative of our lives portraying us against our will in whatever way that someone wants. We're stuck here.

</comment></comment></comment></comment><comment><USER27> > Even then, organized crime was prioritized below his private anti-Civil > Rights Movement vendetta. Even though this put his policies at odds with > several Presidents.

Not many people realize the FBI started as a domestic intelligence agency first and only became primarily a federal law enforcement arm later on. This would make suppressing foreign-backed or (what they deem) dangerous political dissent their priority early on.

</comment><comment><USER28> While I don't agree with the way he is pricing in government failure...because it can be argued and its ultimately weakening to the argument.

The fact J Edgar Hoover controlled the FBI for more than half of its existence. He is also 1/12th of the people who have held the position.

I'd say that is a strong enough argument in and of itself. You just need one guy at the top of the surveillance pile pulling the same tricks and 1 in 12 odds is pretty high.

[ Yes, I'm aware this isn't technically accurate but its close enough for rhetoric purposes. ]

<comment> rm_-rf_slash I think the author's two points were that by mixing an unscrupulous individual with a lot of power and access to immense amounts of information you have the recipe for totalitarian control, and those ingredients are much closer than we would like to think.

</comment></comment><comment><USER29> surveillance is a power grab, whoever controls the people's private lives, controls the people.</comment>


Introduction to OS Abstractions Using Plan 9 from Bell Labs (2007) [pdf] - <AUTHOR> https://lsub.org/who/nemo/9.intro.pdf <comment><USER2> It should be noted that Plan 9 is far from dead or outdated. The 9front developers have continued to make many improvements to the OS while retaining backwards compatibility. 9legacy offers a set of patches which try to stay closer to the original Bell Labs distribution. There are also other ongoing developments, such as my own Advanced Namespace Tools and a new grid of public 9p services.

<comment><USER3> Is there a contingent of 9fans doing work on using Plan 9 for DevOps? The fact that doesn't have popular graphical software (or even the same conventions) wouldn't hold it back there. If it's really as well designed and coherent as the praise always makes it seem, it _should_ be an attractive option on the server. So developers can stick with Windows/Mac/Linux on personal machines, and should be deploying to Plan 9 in the cloud, right?

<comment><USER2> It might make sense in theory, but in practice there are very very few Plan 9 users and developers. Fundamental design quality is never a guarantee of "success" as measured by size of the userbase. If you look at the culture of Plan 9 users, there is also often a somewhat adversarial perspective on standard software. It's a complicated topic, with a lot of weird history involving things like Plan 9 not having a decent open source license for a long time. Once the patterns were established, sociocultural issues outweighed any technical issues. You can imagine an alternate history where things all worked out differently, but at present, Plan 9 is hard to view as a realistic option for any kind of mainstream corporate software work. A lot of us think that is probably more of a good thing than a bad thing, economic incentives and software design quality are often misaligned.

<comment><USER4> If not for economic impact, why do you work on Plan 9? Do you view it as a research project? A hobby?

<comment><USER2> I guess you could call it a hobby as much as anything. It feels more serious than that to me, based on the amount of time and effort I put into it. In addition to developing os-level projects for Plan 9 itself, I write interactive fiction games, which feels to me like a serious artistic pursuit, but most people would put interactive fiction text adventures in the "hobby" category. My brain doesn't really follow the dichotomy of using whether or not something is profit-making for how "important" it is. I understand that is an attitude that is somewhat a measure of personal privilege to not focus that hard on economic issues.

<comment> sahil-kang Would you mind sharing a few good intro sources for Plan 9? I'm a small-time Linux contributor, but haven't delved into any other operating systems. Interactive fiction games are always appreciated too :)

<comment><AUTHOR> HN discussion of a 'What makes Plan 9 unique?' discussion in the 9fans mailing list: [https://news.ycombinator.com/item?id=11882797](https://news.ycombinator.com/item?id=11882797)

For another short overview, there's also Eric S. Raymond's take on Plan 9 in "The Art of Unix Programming": [http://www.catb.org/esr/writings/taoup/html/plan9.html](http://www.catb.org/esr/writings/taoup/html/plan9.html)

<comment><AUTHOR> Direct links to the relevant introductory papers: [http://9p.io/wiki/plan9/Recommended_Readings/](http://9p.io/wiki/plan9/Recommended_Readings/) [http://9p.io/sys/doc/](http://9p.io/sys/doc/)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> If anyone here hasn't played with plan9, I _highly_ recommend it. It's no overstatement to say it's the best operating system ever designed.

<comment><USER6> I kind of disagree, at least in what concerns the Plan 9 lineage, Inferno was way better.

Sadly it appears most people just stop at the middle, instead of the last version of Plan 9's evolution.

<comment><USER7> I've always enjoyed your opinion on alternative operating systems and languages. I love plan9, I wore my thesis in it, but that was based on the knowledge that other distributed operating systems existed at the time, amoeba being the most fun to learn about. Now it's a wasteland... And I really would prefer to explore.

I think better or worse isn't the important theme here. It's that we've lost variety...

<comment><USER8> Distributed computing is being reborn in the crypto currency sphere.

Perhaps in 20 or 30 years we will have something akin to plan9 or inferno composed from nodes on these networks?

After all, at some point everything old becomes new again when it comes to computer technology.

<comment><USER4> What does it mean for an operating system to run on the blockchain?

<comment><USER9> Operating System: \- the software that supports a computer's basic functions, such as scheduling tasks, executing applications, and controlling peripherals.

Blockchain: \- A decentralized and distributed digital ledger that is used to record transactions across many computers so that the record cannot be altered retroactively without the alteration of all subsequent blocks and the collusion of the network.

<comment><USER4> Is this supposed to answer my question?

<comment><USER9> No, just preface into their intended functions.

Edit: One might question what the ledger tracks in terms of operating toward user needs as opposed to financial activity.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> Anyone else feel that a hierarchical 'bytearray' file system isn't really a great base for organizing an operating system? Some limitations are that you have to share everything as 'sequence of bytes', requiring every reader to re- implement the parsing logic, and not everything fits well into a file metaphor, for instance socket style stream IO.

<comment><USER11> Other organisations (isam, sparse, ...) can be laid on top of sequences. Not so much the away around.

Almost every criticism of the Unix fundamentals* puts me in awe of the original design. It is _so_ hard to maintain simplicity.

Regarding parsing, yes. That would be nice but I believe you'd some isolation to ensure that. Re sockets -- shrug.

* (rather than say the user space / shell)

<comment><USER10> I wasn't thinking about another data storage system but more about what core abstractions should be provided by an OS?

For example, could the Erlang model be an OS? Say the OS provides a namespace for a constellation of communicating processes. You still have freedom to implement various languages and configuration systems but you submit your code to a process rather than store it in a 'file system'. Storage is accessed via a process and can be switched out with another process with a different implementation, etc.

</comment></comment></comment><comment><USER12> Worth pointing out here is harvey ([http://harvey-os.org](http://harvey- os.org)). Which is a descendent of Plan 9 but with updates for toolchains & architecture.

<comment><USER13> One of the things I like a lot about plan9 is that it doesn't use GCC.

<comment><USER14> You can build Harvey with clang if you prefer, or possibly even pcc. Getting Harvey compiling with GCC/clang meant translating Plan 9's C dialect into ISO C. The translation process means there wasn't a lot of opportunity to introduce GCCisms, so any standards-compliant C compiler __should __be able to build Harvey. I say should because you may run into one or two small nits that have to be fixed before it 'll work, and of course you have to find an up-to- date C compiler that's not GCC or Clang (I guess it would be good to try icc?)

</comment></comment></comment><comment><USER15> It's nice to see Plan 9 getting some love in the limelight.

There's a 9fans Discord server lying around for anyone interested: [https://discord.gg/6daut5T](https://discord.gg/6daut5T)

</comment><comment><USER16> Great resource!</comment>


Teen-created Muziic likely to irk YouTube - <AUTHOR> http://www.cnn.com/2009/TECH/03/09/youtube.itunes.muziic.music/index.html <comment><USER2> Reminds me of Songza, though they are browser based. They prominently display the video now, probably because YouTube was not too pleased. I think it's pretty awesome that a 15 year old made this and is getting coverage for it, but the website is a way too noisy for me.

</comment><comment><USER3> sounds like what playlist.com has been doing, quite successfully too, but I don't think they use youtube vids. I was wondering if they would start.</comment>


JSIL - .NET bytecode to JS compiler - <AUTHOR> http://jsil.org/ <comment><USER2> There are a handful of projects translating C# to JavaScript:

\- [http://sharpkit.net/](http://sharpkit.net/) (commercial)

\- [http://scriptsharp.com/](http://scriptsharp.com/) (closed compiler)

\- [http://www.saltarelle-compiler.com/](http://www.saltarelle-compiler.com/) (open source)

<comment><USER3> I am very partial to saltarelle. It supports c# 5.0 constructs like async, c# 2.0 constructs like generics, and everything in between. It has a full suite of tests and relatively active. I have a lot of full projects implemented in it from 60fps canvas games to just normal sites. If anyone is interested in seeing some stuff, or just chatting about it, let me know.

<comment><USER4> Yes, please! Post some links to samples. How easy is it to port an existing XNA/MonoGame project?

<comment><USER3> To be fair, I've never done a port, always a custom implementation. [http://oursonic.org/](http://oursonic.org/) is my favorite personal project. You can see the source at [https://github.com/OurSonic/OurSonicSharp](https://github.com/OurSonic/OurSonicSharp)

[https://github.com/OurSonic/OurSonicSharp/blob/master/OurSon...](https://github.com/OurSonic/OurSonicSharp/blob/master/OurSonic/Sonic/Sonic.cs#L524)

maps to

[https://github.com/OurSonic/OurSonicSharp/blob/master/Output...](https://github.com/OurSonic/OurSonicSharp/blob/master/Output/OurSonic.js#L6923)

As a bit of self promotion, if anyone is interested in helping on this project, or any other web based c# -> js game, shoot me an email dested@gmail :-)

</comment></comment></comment></comment><comment><USER5> So, if one were to use this or something like it, how would you handle DOM interaction? Write JavaScript by hand for that, and have it call out to JSIL- generated code?

<comment><USER6> Like gecko said, you can use C#'s 'dynamic' feature to grab references to JS objects from the DOM and do late-bound manipulation of them.

Functions can also have their body entirely replaced by a JavaScript expression, and it's done using an attribute so that it doesn't affect how your code runs in native .NET. This is used in parts of the runtime library and in some of the examples, like so:

            [JSReplacement("document.getElementById('speed').innerHTML = $text")]
            static void WriteSpeedText (string text) {
              Debug.WriteLine(text);
            }

You can also directly embed raw JavaScript into function bodies, like this:

            var a = 2;
            var b = 5;
            Verbatim.Expression("print($0 + $1)", a, b);

That allows you to embed particular JavaScript constructs that the compiler won't normally generate, call out to things like jQuery directly, etc. The downside to this is that you've now written something that won't run in any other .NET environment, but you can always wrap it in a JavaScript-only conditional.

<comment><USER5> Cool, thanks for the explanation.

</comment></comment></comment><comment><AUTHOR> [http://www.playescapegoat.com/](http://www.playescapegoat.com/)

I thought this was a pretty cute little example game.

<comment><USER4> Makes you think, doesn't it: MonoGame+XNA => Windows, Web, iOS, Mac OS, Android, Linux, PS3/4, XBOX.

</comment></comment><comment><USER7> Good approach using bytecode so that any .NET langugage can be supported.

<comment><USER8> Honestly the JavaScript it generates isn't very nice to read. I'd rather use a language like TypeScript which tries to generate idiomatic JavaScript.

<comment><USER9> I'd rather it'd go the other way: please generate hopelessly unreadable but optimal javascript. However, include the original source as comments in some debug mode.

In my experience with coffeescript, the idea that the source is really readable is only somewhat true. Sure, you can read what's going on. But fancier features aren't always trivially translated; and even where a translation is trivial the small differences in syntax mean that in a large enough codebase it can take some time to find the corresponding source line.

Furthermore, because the code is supposedly "easy to read" that means it _must_ be fairly close to javascript native, and _that_ means (for instance) either a heavy abstraction (JSIL) that's probably slow, or exposing lots of unfortunate javascript quirks (coffeescript) such as 0 vs. null vs. undefined vs. false and/or => vs. ->. Basically, to reason about coffeescript you often enough need to understand in detail how it's translated to javascript; at that point you've lost any productivity gains you were hoping for but are still paying the costs in terms of poor tooling and browser integration.

If you're using javascript like assembly, treat it like that: please don't waste my time being sorta-almost-but-not-quite readable. Just make fast, robust, no-leaky-abstraction javascript, please.

</comment></comment></comment><comment><USER10> What happens if your program uses classes from the .NET Framework, like List or Dictionary? Does the framework IL get translated to JavaScript and sent to the browser too?

</comment><comment><USER11> To add my 2c: We have a great esxperience using JSIL for executing some C# business logic on the clinet to make the page more responsible. By chaninig single c# file we get updated functionality both on client and on the server.

</comment><comment><USER12> Is there anything like this that emits JVM bytecodes? It would be interesting to write C# code to run on the JVM.

<comment><USER13> Search and you shall find: [http://dev.mainsoft.com/Default.aspx?tabid=267](http://dev.mainsoft.com/Default.aspx?tabid=267)</comment></comment>


Bookshop.org earns more than $1M for indie bookstores - <AUTHOR> archive.is: http://archive.is/wip/5AAAS<p>archive.org: https://web.archive.org/web/20200510140635/https://www.latimes.com/entertainment-arts/books/story/2020-04-29/coronavirus-bookshop-org-earns-1-million <comment><USER2> Love Bookshop. the UX is great, shipping is fast, and indie bookstores are benefiting. much better than Amazon all around.</comment>


Apple Prohibits Android Lessons in New Treehouse App - <AUTHOR> http://tech.co/treehouse-ipad-app-2013-07

<comment><USER2> I teach Android and iOS at Treehouse, and I was shocked to find out about this! As disappointing as it is, we are still excited to offer students this app for everything else, and are hopeful that Apple will change their stance on this. As a side note, if we release an Android version of this app, we will take the high road and include the iOS content! :-p

</comment><comment><USER3> Searched for the android app, haven't found anything yet, hope it comes out soon!

(difference is, not really interested in the iOS part of the app :P)

Amazing business, i have seen you evolve from the blog i usually followed: Carsonified, if i'm not mistaking.

Btw. to bad a douchebag company decides how you should run your business...

</comment><comment><USER4> Is there a precident of Apple taking this type of action?

<comment><AUTHOR> > "There's some really weird behavior going on," says Carson, claiming that > the iPad apps for Lynda.com and Udemy have Android content. "We don't know > if it's just a random oversight by Apple or if there's something worse going > on."

</comment></comment><comment><USER5> Douches.</comment>


Bulk Carrier Completes Fully Carbon Offset Voyage from South Africa to Europe - <AUTHOR> https://gcaptain.com/bulker-carbon-offset-voyage/ <comment><USER2> And the added cost is.. .???</comment>


The Most Secure Messengers 2018 - <AUTHOR> https://zangi.com/news/en/secure-messengers-2018/ <comment><USER2> Seems to me the most secure messengers are probably those that don't make a big deal about it. One of the most intriguing secure messengers, in my opinion, is Semaphor -- [https://spideroak.com/semaphor/](https://spideroak.com/semaphor/) \-- and they publish the encryption code for inspection and analysis (though not the keys, obviously).</comment>


Deep Reinforcement Learning for Robotic Manipulation - <AUTHOR> https://arxiv.org/abs/1610.00633 <comment><AUTHOR> video: [https://sites.google.com/site/deeproboticmanipulation/](https://sites.google.com/site/deeproboticmanipulation/)

blog: [https://research.googleblog.com/2016/10/how-robots-can- acqui...](https://research.googleblog.com/2016/10/how-robots-can-acquire-new- skills-from.html) "How Robots Can Acquire New Skills from Their Shared Experience"</comment>


Quantum Random Number Generator Created Using A Smartphone Camera - <AUTHOR> https://medium.com/the-physics-arxiv-blog/602f88552b64

<comment><USER2> This article defines a "quantum random number generator" (QRNG) as "a device that exploits the probabilistic nature of quantum mechanics to produce a sequence of entirely random digits" and claims that QRNGs are "complex, expensive devices".

Firstly, people have been building RNGs from web cams and radioactive sources cannibalized from smoke detectors for years, at a total cost of less than $100. Radioactive decay is a quantum process so it seems fair to describe these as QRNGs.

Secondly, the smartphone camera-based random number generator (SCRNG) described in the article/paper does not appear to generate its random numbers from quantum-level events (i.e. the emission of individual photons). The classic model for quantum cryptography uses a half-silvered mirror to create a quantum state by introducing uncertainty over whether an individual photon has been deflected or not. The SCRNG is not observing individual photons. Instead, it's using the variation between the observed number of electrons that are generated as a result of exposing the camera to a light source, and the number of electrons that the light source is _expected_ to generate. Because the variation is random (i.e. the entropy is high enough), this generates random numbers but I question whether it's really a QRNG.

Disclaimer: I am not a physicist so I could be wrong. In fact, the question of whether I'm right or wrong is currently in a quantum superposition, pending "observation" by someone who is suitably qualified to determine whether I'm right or wrong. ;-)

<comment><USER3> For the purpose of this discussion there are two types of noise in electronics: shot noise and thermal noise.

Shot noise is discrete noise that happens when single electrons are emitted and adsorbed. Shot noise is quantum noise and can't be reduced by cooling and don't vary with frequency.

Thermal noise is noise that depends on the temperature of the circuit. Temperature increases the energy of material and this energy allows electrons to bounce around more creating noise.

For the purpose of QRNG, you don't have to separate shot noise from thermal noise if you know the shot noise to thermal noise ratio. We are not interested of knowing actual bits generated by shot noise. We just want to collect enough random bits so that we can guarantee that there is enough quantum randomness to make the quality equal to just quantum bits. For example, if the ratio is 1/100, we collect more than 100 bits and mix them together to get one bit of quantum randomness. Shot noise is amplified and has easily detectable effects in in p-n junctions (diodes, photodectectors).

<comment><USER4> That sounds reasonable. For a camera sensor, the noise budget is made up of four primary terms:

1\. So called "fixed pattern noise," i.e., a pixel by pixel pattern in the measured value that isn't reduced by averaging. This pattern would be a "fingerprint" of the camera, probably long term if not forever. You can make a well averaged measurement of the fixed pattern (somewhat temperature dependent) and subtract it from subsequent exposures.

2\. Thermal noise of the amplifier that converts charge into a signal that can be read by the analog-to-digital converter. This is referred to as "readout noise."

3\. Shot noise in "good" photocurrent, i.e., photons from the scene being converted into charge. This becomes dominant at higher light levels, as it's proportional to the square root of photocurrent.

4\. Shot noise in "bad" photocurrent, i.e., a constant leakage current in the light sensitive elements, whose magnitude is temperature dependent.

</comment></comment></comment><comment><USER5> The article is a little breathless about how "...physicists have long searched for other ways to make genuinely random numbers based on physical processes that produce random outcomes. "

My understanding is it's pretty straightforward to do so - here's a nice little device available on tindie:

[https://www.tindie.com/products/ubldit/truerng-hardware- rand...](https://www.tindie.com/products/ubldit/truerng-hardware-random- number-generator/)

And here is the physical process it's based on from a (5 year old) paper.

[http://cockrum.net/Implementation_of_ECC_on_an_8-bit_microco...](http://cockrum.net/Implementation_of_ECC_on_an_8-bit_microcontroller.pdf)

And, not to be pedantic, but you really only need a small amount of entropy to seed your PRNG - for all practical purposes, the rest of the random bits can be computationally generated.

<comment><USER6> This article seems to be talking about using quantum effects for random number generation.

The devices you point to use the semiconductor avalanche effect ([http://en.wikipedia.org/wiki/Avalanche_breakdown](http://en.wikipedia.org/wiki/Avalanche_breakdown)) which is a thermal effect.

<comment><USER3> Avalanche effect from shot noise is quantum noise.

</comment></comment></comment><comment><USER7> Doesn't anybody else think the real intersting observation is that commodity cameras are approaching unitary quantum efficiency?

In short: we're mass producing quantum-scale-detectors and deploying them to billions of humans. That in itself is pretty extraordinary.

</comment><comment><USER8> No thanks.

My banking app does not need access to my phone's camera "for security" purposes.

My banking app will more than likely be snapping photos of me at the time of the transaction, encrypting them, and then transmitting the images back to themselves as opaque binary blobs (claiming that they are part of the normal transaction data, even though they add 3MB to the bandwidth, because security), and retaining them for audit, in case there's a security breach, and securing their unfortunate scenario for their own purposes with ordinary photographic information, and never actually generating "quantum" random seed information (their database fell into the wrong hands, but they have upwards of 20,000 distinct, recognizable faces as a starting point for a possible ID during the subsequent postmortem and investigation). All while receiving kickbacks from the NSA for for sharing geotagged facial recognition images for their world domination scheme.

In this situation, cameras become a dual-use technology. Maybe they're used for a QRNG or maybe something else?

How would I honestly ever know whether my bank was lying to me about what it's really using my camera for?

<comment><USER9> The obvious solution for this would be for Android to expose "derive random numbers from image frame" as a permission. But this is unnecessary, because they can just seed /dev/random from this source at boot (or if the device is unseeded).

That said, mobile devices really aren't lacking in entropy sources. With all the radios and sensors in a modern smartphone, why do they need additional methods to generate random numbers?

For information security purposes, a cryptographically secure PRNG is typically at least as secure as the encryption algorithms that it protects.

</comment></comment><comment><USER10> It's unfortunate that this requires an LED shone at the camera. I hope something similar can be done using just the variation between frames w/o needing controlled illumination.

<comment><USER11> that's the beauty of it: the randomness is "bigger" for objects illuminated "just right" (half way through completely dark and 100% bright), but it still works for pictures taken in less than ideal conditions. And it's all because of the imperfections in all smartphone cameras. Out of 100.000 pictures taken in the exact same position with the same camera, do you think you can get two 100% identical ones? Even if you could get the exact same conditions (illumination, position, etc.), there are always factors like that tiny little bit of degradation that happens each second you expose the sensor to light.

</comment></comment><comment><USER12> Two uses of quantum effects are 1. Generating random numbers, 2. Entangling bits to detect "man in the middle".

The point of this discovery is to decide whether it improves the security of smartphone communications. If yes, it will upset governments, if no, it is an interesting observation.

</comment><comment><USER13> Glad to see the N9 do one more good deed during the rough path it has been on. Simply love how open that phone and OS is. Yeah, they could have tested this on any phone probably, but I still felt a little tingling inside when I saw it mentioned :)

</comment><comment><USER14> A RNG with 10^118 secure bits isn't really that great at all. All I need is a 49 character entirely random password or a 36 word long xkcd-936 complaint password. I can then use them as my key/seed for secure cryptosystem (RSA or Rabin's) and then generate on the order of 10^118 bits before my system is breakable. QRNG's are only useful when you need large amounts of random data are needed for a (relatively) slow device. The average person has little need of a QRNG. That said, the work itself is an interesting method to create one, its just that the article makes it sound like something its not. (I'm not a cryptologist, so feel free to correct me.)

</comment><comment><USER15> Can you mess it up? Take your friends phone, point it at a known pattern and send tons of insecure data?

<comment><USER16> _Can you mess it up?_

Lay the phone's camera facedown on computer desk. It'll probably be a black picture.

There's still noise in the picture even if it's entirely black, but Whether it's possible to generate secure random numbers via nothing but that noise is an open question.

<comment><USER15> Also saturate the sensor, then you don't get noise.

<comment><USER17> In my experience even a "saturated" sensor (with light bright enough to cause all pixels to read full-white, but not bright enough to cause damage) will still generate some random noise, although far less of it.

(Ironically, the cheaper the camera the more noise it is likely to have, and thus be better for this application...)

</comment></comment></comment></comment><comment><USER18> If someone could build this into a usb key, I'd buy one for each of my servers in a heartbeat.

<comment><USER9> [https://www.tindie.com/products/ubldit/truerng-hardware- rand...](https://www.tindie.com/products/ubldit/truerng-hardware-random- number-generator/)

But I'll give two cautions:

1\. This device has received less auditing than linux's software crypto. And some HWRNGs are quite bad: [https://news.ycombinator.com/item?id=6060636](https://news.ycombinator.com/item?id=6060636)

2\. You don't really need that much randomness. After your machine has been on for a while and has seeded correctly, /dev/urandom is just as secure as /dev/random. Entropy is not gasoline - it does not disappear as you use it.

</comment></comment><comment><USER19> I'll put in my usual -1 rant for medium disabling zoom on tablets</comment>


How  To Ensure You Get Actionable Feedback From Your Customers - <AUTHOR> http://blog.getvero.com/feedback-from-customers/

<comment><USER2> Ask your customers to give you a review and keep it simple... you might be interested in looking at www.testifor.com for more details</comment>


Curiosity images strange metallic-looking object on Mars - <AUTHOR> http://www.universetoday.com/99750/another-weird-shiny-thing-on-mars-2/ <comment><USER2> The landing created an extensive debris field [1]. For example, the descent stage energetically disassembled on impact with 100+ kg of unused propellant still on board, as planned [2]. Bits of plastic and metal are strewn across a 30km-long strip. Curiosity has imaged several of these pieces already [3]. On the other hand, some 'bright material' that was thought to be spacecraft fragments turned out to be soil material. These fragments are important: they're a science hazard. A big goal for the mission is to find Martian organic matter, so the project is keen to avoid accidentally ingesting Earth plastic into the exquisitely precise onboard laboratory, SAM [4].

[1] <http://photojournal.jpl.nasa.gov/jpeg/PIA15696.jpg> (debris from the last few minutes of flight),

<http://www.lpi.usra.edu/meetings/lpsc2013/pdf/2800.pdf> (debris from interplanetary cruise and atmospheric entry).

[2] [http://www.spaceflight101.com/msl-descent-stage-impact- analy...](http://www.spaceflight101.com/msl-descent-stage-impact- analysis.html) (crash movie).

[3] [http://www.nasa.gov/mission_pages/msl/multimedia/pia16230.ht...](http://www.nasa.gov/mission_pages/msl/multimedia/pia16230.htm)

[4] <http://link.springer.com/article/10.1007%2Fs11214-012-9879-z> (free full text)

<comment><USER3> [3] is a 404 page.

Love the phrase "energetically disassembled".

<comment><USER2> [3] [http://www.nasa.gov/mission_pages/msl/multimedia/pia16230.ht...](http://www.nasa.gov/mission_pages/msl/multimedia/pia16230.html)

</comment></comment></comment><comment><USER4> Does anyone know NASA's policy on further observation of crowd-sourced findings?

BTW, you absolutely have to love the tack NASA has taken with making images virtually immediately available for the public to consume and research. Imagine if the Curiosity photo archives were only made available months after the fact - it'd be too late to do anything based off of community findings!

Way to go team!

<comment><USER5> Heh. I don't really consume images. I just view them.

</comment></comment><comment><USER6> I'm certainly not an expert on this, but I'd say the most important number to think about when looking at rover images is the _scale_.

 _"the little protuberance is probably about 0.5 cm tall, or even smaller."_

Five mm is obviously tiny. Keep this in mind before making up hypotheses as to what the thing is. =) We have to wait until better data is available.

What's fascinating in itself though is that no person, however sceptical, can immediately rule out the _possibility_ that stuff like this is some form of life or even evidence of non-human technology. Of course, such hypotheses have to work extremely hard before they should be accepted. Our standards of evidence for stuff like this must always remain as high as possible, given the emotions and mass misunderstandings involved. But the very fact that we've come this far and still cannot rule out the existence of life on Mars is quite remarkable.

<comment><USER7> One of my dearest wishes is that humanity will find solid evidence for other lifeforms in the universe (whether it is through remains or actual contact).

This moment, if/when it happens, will be so mindblowing on so many levels and affect us deeply as a society/species/etc. that I just WANT to see it.

<comment><USER8> So you just want to see all religions on earth, dead.

<comment><USER9> Religion will survive. It survived learning the sun didn't revolve around the Earth. It survived learning disease wasn't caused by demons and that Zeus wasn't hurling lightning from Olympus. It survived evolution and the Big Bang. People will say the mysteries of God* were just greater than they realized, and their mythologies will start to seem a bit more like Star Wars.

* or what have you

<comment><USER10> You're correct. Religious belief is an evolved memetic/genetic survival mechanism.

Those parts of the human brain were never about logic and accurate observation of the material world - so they are safe from new facts and logic for now.

Probably at some point (assuming we survie), our own genetic/cybernetic modifications will so increase our intelligence, that religion will fall away completely... but that's beyond the Singularity.

<comment><USER9> Well, assuming the Singularity itself is not a doctrine of religious faith, sometimes it sounds like one.

</comment></comment></comment></comment></comment></comment><comment><USER11> I find these things fun, of course some folks want to find alien technology, but I remind people that we've lost roughly 2/3rds of all the probes and ships that we've sent to Mars [1]. Since there isn't a lot of stuff covering them up, eventually we may find a bit of antenna from the Mars Observer sticking up where the spacecraft attempted to fly through the planet.

[1] [http://en.wikipedia.org/wiki/Exploration_of_Mars#Past_missio...](http://en.wikipedia.org/wiki/Exploration_of_Mars#Past_missions)

<comment><USER12> That seems extremely unlikely. If a few dozen planes crashed on Earth (on land), how likely do you reckon you'd be to ever run across significant debris from any of them just by randomly driving around?

<comment><USER13> If Earth were devoid of vegetation and instead uniformly rocky, running into a sliver of a downed plane might've been far more probable. Especially with volunteers scouring every taken image for anything that doesn't look like a rock.

<comment><USER14> > Especially with volunteers scouring every taken image for anything that > doesn't look like a rock.

I don't think anybody is confusing the likelihood of finding a piece of a plane due to not looking at the picture well enough. It's a question of if there would be anything in that picture to begin with.

</comment></comment></comment></comment><comment><USER15> Usually I prefer not to speculate on this sort of thing, because if we just wait we'll have a ton more information soon. However, I wonder if it could be a fulgerite: <http://en.wikipedia.org/wiki/Fulgurite>

<comment><USER16> Interesting theory. I had to check, and yes, Mars does have dry lightning (in dust storms).

If not that, I wonder if it's something produced volcanically like onyx.

<comment><USER17> I was thinking an ore vein that was harder than the material around it, and said outer material had been weathered away faster to expose it.

Either that or a door handle! :P

</comment></comment></comment><comment><USER18> If the object really is 0.5 cm tall, it could be similar to this calcium vein in a rock which is comparable in size: <http://mars.jpl.nasa.gov/multimedia/images/?ImageID=5017>

<comment><USER19> Yep, it has a distinctly 'mineral vein' appearance to it.

</comment></comment><comment><USER20> The human brain is a remarkable pattern recognition machine with a keen penchant for extracting meaning from noise.

This, of course, will turn out to be nothing more than an unremarkable optical illusion; an urban legend is born.

<comment><USER21> This is called Pareidolia (<http://en.wikipedia.org/wiki/Pareidolia>).

 _> Pareidolia ... is a psychological phenomenon involving a vague and random stimulus (often an image or sound) being perceived as significant._

<comment><USER22> When you're driving in a snow storm (sometimes unavoidable) one of the hardest things is to maintain a feeling of heading and _not_ to respond to every impulse when your brain thinks it has made meaning out of the falling snow just ahead of the car. And even harder to realize when it actually has picked up something of significance.

I got caught in snow storms several times and what surprised me was how much physical effort it takes to do this for any length of time.

</comment></comment></comment><comment><USER23> So that's where R2D2 ended up.

It looks almost like molten metal poured into a sand cavity, and then with the sand blown away. I guess you wouldn't get oxidation or similar activity on Mars (?), so it remains nice and shiny. I wonder how far the shape goes down, and what could have caused it?

<comment><USER24> > _I guess you wouldn't get oxidation or similar activity on Mars_

Mars is principally deeply covered in iron III oxide. That's why it's red.

</comment></comment><comment><AUTHOR> I'm hoping that we can just drive the rover over to take a closer look before very long. I can't guess whether this is the result of past erosion or the result of a past meteorite impact, or even some offthrow of the Curiosity landing that just _appears_ to be embedded in the rock but really isn't. In any case, this seems like it will be educational.

</comment><comment><USER25> "Images"? Really? What is happening to language? It took a photo! Imagine someone saying: "I went to Paris and imaged the Eiffel Tower".

<comment><USER26> Photographs are formed by light through a lens falling onto film (or a sensor).

NASA has images like that. They also have images formed after extensive post- processing, or merges of several images, or from frequencies other than light, or computer rendered from map data, or etc etc.

</comment></comment><comment><USER27> "The chances of anything coming from mars are a million to one", he said.

</comment><comment><USER28> Surely there is a very reasonable possibility of a crashed asteroid/comet... at least much more so than some of the other wild speculation (non-human intelligent life, etc.)

</comment><comment><USER29> As a reminder, if a human explorer had seen this, we'd already know much, much more about it. They'd naturally take a few steps over for a closer look as soon as they saw it.

Robots are a poor substitute for humans when it comes to planetary exploration.

<comment><USER30> A human explorer probably wouldn't have seen it though; it's really tiny. The reason this was seen at all is that the high resolution images are being looked over by a large number of people on Earth. NASA themselves didn't even spot it.

(High resolution imagery + many many human eyes) > Human eyes

Never mind that Curiosity has now been operating for 6 months without food, water, or sleep, and is expected to continue doing so for years.

<comment><USER29> It's shiny, the odds are very high that a human would indeed have seen it. For one thing, a human's eyes are constantly processing the world as they move through it. Curiosity's eyes are not.

What Curiosity has done in 6 months could have been done in a couple of weeks by a human, sleep time included. Robots are vastly inferior to humans for this work, and will remain so for some time to come.

</comment></comment></comment><comment><USER31> Resembles <http://en.wikipedia.org/wiki/Loch_Ness_Monster>

</comment><comment><USER32> Kind of looks like a drill bit to me

</comment><comment><USER33> Perspective switch: "It's an alien weird shiny object much larger than us, and it's mobile! It has arms and lets Hmm...we won't shoot it down, let's wait and see what it does next. I think it's looking at us directly now! On the side it reads, 'C U R I O S I T Y'"

</comment><comment><USER34> I'll go straight out and say it: if there is intelligence elsewhere in the universe, we have no guarantees to say that that form of cognition would in anyway resemble our own since the only models we have presume terran conditions.

<comment><USER25> Mathematics appears to be universal.

</comment></comment><comment><USER35> Good info... i want to say thanks for share with us...

</comment><comment><USER36> Its a UFO from Venus</comment>


Show HN: Breef Daily – Customizable Morning News from Your Favorite Sources - <AUTHOR> http://www.breefdaily.com <comment><USER2> Not even a preview?

Also I like how the payment forms work but literally nothing else on this page.

</comment><comment><AUTHOR> I started working on Breef Daily this Thursday while bored out of my mind on a tour bus with my University's Symphonic Band. It's a customizable mailing list which sends you the top content from the sites you select every morning.

Right now everything is hand curated, but I'm working on some scraping scripts which should automate the process quite a bit and I can do some fancy AI tricks to make the emails nice to read. I'm also working on adding more and more sources every day.

I already have about a half dozen subscribers from techy friends and family, and they are giving me some valuable feedback but seem to really enjoy the content that I'm producing so far.

I would love some feedback on what you think, how I can expand on the business model, or any thoughts you think are worth sharing. Thanks in advance!

</comment><comment><USER3> Doesn't limiting one's media menu to the readers favorite sources exacerbate the "media bubble" problem?

<comment><AUTHOR> Possibly, though you can see that most of the sources I first added were non- political news sources. More traditional news sources were added as a request from the first people who used it. Now I'm thinking about a "balanced news" genre or source which compiles articles from a balanced spectrum of political journalists. What do you think?

</comment></comment><comment><USER4> How does this compare with an RSS reader?</comment>


Why It's Good to Be Wrong - <AUTHOR> http://nautil.us/issue/20/creativity/why-its-good-to-be-wrong-rp

<comment><USER2> "The fact is, there's nothing infallible about "direct experience" either. Indeed, experience is never direct. It is a sort of virtual reality, created by our brains using sketchy and flawed sensory clues, given substance only by fallible expectations, explanations, and interpretations. Those can easily be more mistaken than the testimony of the passing hobo. If you doubt this, look at the work of psychologists Christopher Chabris and Daniel Simons, and verify by direct experience the fallibility of your own direct experience. Furthermore, the idea that your reminiscences are infallible is also heresy by the very doctrine that you are faithful to."

This is skepticism raised to the nth degree -- brain-in-a-vat-ism. I'm not even sure I read the article now.

<comment><USER3> [http://en.wikipedia.org/wiki/Reality_tunnel](http://en.wikipedia.org/wiki/Reality_tunnel)

>The idea does not necessarily imply that there is no objective truth; rather that our access to it is mediated through our senses, experience, conditioning, prior beliefs, and other non-objective factors. The implied individual world each person occupies is said to be their reality tunnel.

<comment><USER4> And after that, kick things up a notch with the Boltzmann Brain paradox:

[http://en.wikipedia.org/wiki/Boltzmann_brain](http://en.wikipedia.org/wiki/Boltzmann_brain)

> _The paradox states that if one considers the probability of our current > situation as self-aware entities embedded in an organized environment, > versus the probability of stand-alone self-aware entities existing in a > featureless thermodynamic "soup", then the latter should be vastly more > probable than the former._

</comment></comment></comment><comment><USER5> And at the conclusion, David disappears in a puff of logic.

</comment><comment><USER6> First off, if the Pope said that, it wouldn't count.

Therefore, faithfully adhering to the tradition received from the beginning of the Christian faith, to the glory of God our savior, for the exaltation of the Catholic religion and for the salvation of the Christian people, with the approval of the Sacred Council, we teach and define as a divinely revealed dogma that when the Roman Pontiff speaks EX CATHEDRA, that is, when, in the exercise of his office as shepherd and teacher of all Christians, in virtue of his supreme apostolic authority, he defines a doctrine _concerning faith or morals_ to be held by the whole Church, he possesses, by the divine assistance promised to him in blessed Peter, that infallibility which the divine Redeemer willed his Church to enjoy in defining doctrine concerning faith or morals. Therefore, such definitions of the Roman Pontiff are of themselves, and not by the consent of the Church, irreformable. [http://www.ewtn.com/library/COUNCILS/V1.htm#6](http://www.ewtn.com/library/COUNCILS/V1.htm#6)

Second, we don't have time to become experts in every subject, so we defer some decisions to existing experts.</comment>


Tamil Nadu village lived in darkness for over 35 days to help a sparrow - <AUTHOR> https://www.indiatoday.in/india/story/this-tamil-nadu-village-lived-in-darkness-for-over-35-days-to-help-a-sparrow-1704532-2020-07-26 <comment><USER2> This behavior is rooted in a world view taught by religious/cultural teachings that all living beings have equal right to exist. They suffer just like humans and they cherish their lives just as much as humans.

</comment><comment><USER3> Love that story!</comment>


Linux System Call Table - <AUTHOR> http://thevivekpandey.github.io/posts/2017-09-25-linux-system-calls.html <comment><USER2> So the issues noticed so far:

* Missing syscalls

* Wrong syscall numbers

* Wrong calling convention

* Links to source are to wrong version

Does the table get actually anything right? I mean this is pretty spectacular cascade of failures.

<comment><USER3> At least for x86, you can get this same information fairly easily directly from the source. The table is located at arch/x86/entry/syscalls/syscall_64.tbl, from there you can grep for the function with git grep. For example, git grep 'SYSCALL_DEFINE.*read'.

<comment><USER4> why bother with git grep vs. just vanilla grep. i could see the use if you're working with an older binary, but you didn't mention.

<comment><USER3> If you ran something like grep -r SYSCALL_DEFINE. _read from the top level of the linux source it would search through not just your source code, but also all of the artifacts of building the kernel. Basically, git grep is faster in this case because it filters the searched files down to only ones that are checked in. You could achieve a similar effect with standard tools like this: find -type f -regex '._\\.[hc]' | xargs grep 'SYSCALL_DEFINE.*read'

<comment><USER5>
        grep -r --include='*.[hc]' 'SYSCALL_DEFINE.*read'

<comment><USER3> Nice. I hadn't used --include before.

</comment></comment></comment></comment></comment></comment><comment><USER6> Actually, the syscall numbers are wrong! This reference seems better: [http://blog.rchapman.org/posts/Linux_System_Call_Table_for_x...](http://blog.rchapman.org/posts/Linux_System_Call_Table_for_x86_64/)

Consider simple C program:

        #define _GNU_SOURCE
        #include <unistd.h>

        int main(){
            syscall(276);
        }

Strace'ing it shows the syscall used is tee, just as the reference I linked shows, and not pwritev as in OP's table.

<comment><USER7> It's odd. The owner of this Git repo has Issues turned off so I can't post a question/issue, and it appears to have been auto-generated - [https://github.com/thevivekpandey/syscalls- table-64bit](https://github.com/thevivekpandey/syscalls-table-64bit) is a "fork" of [https://github.com/paolostivanin/syscalls- table-64bit](https://github.com/paolostivanin/syscalls-table-64bit)

</comment></comment><comment><USER8> If man pages were up to date, this should be the index of chapter 2. I have discover unix with sun in the 90s and I am very nostalgic of the quality of man pages. At that time, man pages were complete and up to date. My latest frustration was with the option -m of df command. Chapter 2 should be updated each time a new version of kernel is installed.

<comment><USER9> It's very strange that adding/updating documentation isn't treated as a basic requirement for a patch that adds to or modifies Linux's public interfaces.

<comment><USER10> For the BSDs, incorrect or missing man pages are considered a serious bug.

</comment></comment></comment><comment><USER11> Nice, I'm curious how it maps the system call to the source code line number dynamically? (Edit: seems like ctags + [http://elixir.free- electrons.com/linux/latest/source](http://elixir.free- electrons.com/linux/latest/source) [1]) It supports every kernel version.

The linked source code browser seems like a useful way to check the history of system calls for research...

[1] [https://github.com/thevivekpandey/syscalls- table-64bit/blob/...](https://github.com/thevivekpandey/syscalls- table-64bit/blob/master/gen_syscalls.py)

</comment><comment><USER6> That's all cool and everything, but the registers are wrong... Not only are they 32-bit (eax vs. rax), but their order is wrong too - the first argument in x86-64 ABI is rdi, for example.

<comment><USER12> The registers look correct for the i386 ABI. eax for the system call number, then ebx, ecx, edx, esi, edi, ebp for the next 6 arguments.

I skimmed a couple files in the code. And it seems like it might be parsing this information out of some other sources, and maybe getting confused about the info it's grabbing?

[https://github.com/thevivekpandey/syscalls- table-64bit](https://github.com/thevivekpandey/syscalls-table-64bit)

</comment></comment><comment><USER13> There are tables in the kernel git repo if you want a good reference for their values; however, the register definitions aren't provided.

x86: [https://github.com/torvalds/linux/blob/master/arch/x86/entry...](https://github.com/torvalds/linux/blob/master/arch/x86/entry/syscalls/syscall_32.tbl)

x64: [https://github.com/torvalds/linux/blob/master/arch/x86/entry...](https://github.com/torvalds/linux/blob/master/arch/x86/entry/syscalls/syscall_64.tbl)

</comment><comment><USER14> This man page describes the syscall ABI for all architectures. [http://man7.org/linux/man- pages/man2/syscall.2.html](http://man7.org/linux/man- pages/man2/syscall.2.html)

<comment><USER15> ...and the syscalls(2) man page lists them: [http://man7.org/linux/man- pages/man2/syscalls.2.html](http://man7.org/linux/man- pages/man2/syscalls.2.html)

</comment></comment><comment><USER16> I put out a syscall table back in the day for Linux 2.2 (up to %eax 190). Someone copied it (I'm glad.): [https://www.cs.utexas.edu/~bismith/test/syscalls/syscalls32....](https://www.cs.utexas.edu/~bismith/test/syscalls/syscalls32.html) They didn't attribute it to me, but I remember a professor did for his class. There were better tables after that I admit, though I liked my version because it linked into the source code.

</comment><comment><USER17> I'm a bit puzzled. The code is at "syscalls-table-64bit", yet the regs are eax, etc. This makes very little sense.

In any event, I think the args should just be labeled arg0..arg5.

<comment><USER18> In several cases, the order of the args for a syscall varies between architectures--writing a general "arg0" doesn't make a lot of sense.

That said, I don't know what's up with it using 32-bit register names.

</comment></comment><comment><USER19> Latest complete list is at [http://elixir.free- electrons.com/linux/latest/source/include...](http://elixir.free- electrons.com/linux/latest/source/include/linux/syscalls.h)

</comment><comment><USER20> This is very handy with the asm registers mapped to arguments. Thanks!

</comment><comment><USER21> Nice work. the table has been generated for 4.10 and hence the link to the source code files should also have this kernel version in the path of the url for direct access

</comment><comment><USER22> Ah, this is neat! Would be nice to have a script for this that you could just point at a local copy of the source tree too!

</comment><comment><USER23> What is the use case for this? Is it for someone trying to write their own syscall wrappers?

<comment><USER24> You might need that when you want to reimplement Linux, the Joyent team did that on their OS (derived from solaris) so that user can run linux binaries on a solaris kernel (so thay have dtrace, zfs, mdb, ...) Bryan Cantrill did a bunch of conferences on that (one here: [https://youtu.be/TrfD3pC0VSs](https://youtu.be/TrfD3pC0VSs))

The idea behind is that Linux is only a list of syscalls, if you are able to reimplement them, you reimplement linux, you don't need anything else. On the contrary if you want to reimplement a BSD you need to reimplement their libc (and perhaps some other libraries)

<comment><USER25> > On the contrary if you want to reimplement a BSD you need to reimplement > their libc (and perhaps some other libraries)

To clarify, what you're saying is that in BSD land, the syscall API is not considered stable, but libc is?

<comment><USER24> I'm not ultra familiar with the topic so if someone wants to correct me please do but :

\- Linux has always been described as just a kernel, which translates as just a syscall table. The fact that this table is stable or not is not relevant here.

\- *BSD on the other hand are shipping a kernel plus a lot of libraries/binaries, if you want to simulate a BSD system, you have to expose those libraries/binaries.

It's not so much a technical difference, it's more of a different approach to OS development (kernel space vs kernel/user space).

<comment><USER25> Thing is, if syscalls in BSD are considered stable the way they are in Linux, then you could just ship your own kernel with BSD's libc. But if they consider it an internal API between kernel and libc, and apps are only ever supposed to depend on libc, then of course that doesn't work.

So stability of syscall API is the de facto differentiating factor here. It sounds like Microsoft couldn't do "Windows Subsystem for BSD" the way it did WSL, for example.

</comment></comment></comment></comment></comment><comment><USER26> Do system calls put their return value on the calling threads stack or in a register?

<comment><USER27> AFAIK, A process isn't required to have a stack.

</comment></comment><comment><USER28> Someone should put together a list of which ones are irredeemably broken (and as such, humanity is stuck with a broken ABI in perpetuity), e.g. epoll.

</comment><comment><USER29> What about non-x86/64 platforms?

</comment><comment><USER30> The mere fact that we are debating over the correctness of this table confirms the quality of the documentation of the OS we base our entire civilization upon is pretty poor.

</comment><comment><USER31> This is great! It would be even more useful to have this for Mac OSX too. A lot of the projects I do ends up being on both Mac and Linux. It's always a pain to find the corresponding number for the system call on Mac.

<comment><USER32> System calls have not stability guarantee on macOS. You should use libc instead. In general the use of syscalls directly is fairly limited.

Edit: for instance go broke once for macOS Sierra, when Apple changed the gettimeofday system call: [https://github.com/golang/go/issues/16570](https://github.com/golang/go/issues/16570)

<comment><USER33> Linux doesn't guarantee syscall stability either. Just make sure your wrappers can use a syscall table chosen at runtime, depending on which kernel you are running.

<comment><USER34> Yes it does, at least in the sense that syscalls which become officially public will never be removed from Linus' tree except in rare circumstances (i.e. proof that nobody is using it), nor will the arguments change. This is Linus' famous "never break user space" ABI mantra. While distributions may deprecate and remove them (e.g. sysctl(2)) they certainly won't be assigned new IDs. A table won't help in such cases.

<comment><USER15> Exactly. This is why, for example, the original 'mmap' system call entry point on x86 still exists, even though it is overwhelmingly likely that every program on your machine is actually going to use the 'mmap2' entry point.</comment></comment></comment></comment></comment>


Things You Should Know About WebAssembly - grey-area https://auth0.com/blog/2015/10/14/7-things-you-should-know-about-web-assembly/ <comment> grey-area WebAssembly is being developed by the teams behind asm.js and (P)NaCl</comment>


Use Slack as your professional diary - <AUTHOR> https://medium.com/@thinkindie/use-slack-as-your-professional-diary-f2da7b632026 <comment><USER2> Considering all of the malfeasance we've seen on the part of companies providing data-hungry web apps, I'm going to pass on using Slack for anything I'm not required to do in order to keep getting a paycheck. My professional diary is too important to me to trust to Slack, or to any cloud application.

<comment><AUTHOR> there are open source alternatives too that can be used instead of slack if that is your concern, but the concept will stay the same.</comment></comment>


Ask HN: Best way to learn the CS background I missed by not going to school? - <AUTHOR>

I'm not sure if this is the right place to ask if it's not I apologize. This is my first post to HN.
Background: Basically I've been a developer for almost four years now. I started as a front-end developer writing HTML/CSS/JS, but quickly moved to the back-end writing PHP. Before now I've considered myself a decent developer, but lately I've been feeling burnt out and have been looking to move to a position that would make me happier. I've been looking at Ruby/Rails positions on the various job boards and I really feel like I don't have enough CS knowledge to land any of these positions. I'm feeling more and more like a copy and paste PHP developer. I mean, how can I get a job that is asking for engineers to apply? I really got a punch in the face when I read through a post on Stack Overflow about lesser known data structures like Ropes, and Bloom Filters, and Skip Lists. I have ZERO understanding of this kind of stuff and I think it's going to hold me back in my career.
What's the best way to learn all the stuff I missed by not going to school? Are there CS books  or technical papers that are pretty much required reading for CS students? Should I take some open courses on the web from places like MIT or Stanford?
I can't live my life as a jr. PHP programmer... I'm feeling burnt out as it is, working on the same enterprise app for an employer that won't embrace open source or even new technology!
Help me hacker news, you're my only hope. <comment><USER2> Structure and Interpretation of Computer Programs <http://mitpress.mit.edu/sicp/full-text/book/book.html>

It is one of the classic CS textbooks still used by Berkeley (CS61A) and MIT (not sure) intro CS courses. Its not trivial stuff. While it wont teach you obscure data structures like ropes, it will expose you to a wide variety of topics including but not limited to: functional programming, lambda calculus, OOP, logic programming, client/server programming, non deterministic programming, streams as data, the meta-circular evaluator, lazy evaluation, and concurrency. I had a fairly strong CS background before taking CS61A at Berkeley, but this book (thanks to Scheme) taught me how beautiful computer science can be. Now working in Java is a complete turn off ;)

<comment><AUTHOR> Thanks choochoo! That's something like I was looking for! How's the book for someone with no formal CS background? Is it something I can just into now, or should I use another book as a stepping stone.

</comment></comment><comment><USER3> From next time write the code yourself and minimize the copy/pasting habit as much as you can. Using a good IDE helps control the copy/pasting habit to some extent.

Start here, follow the topics you have not tried yet: <http://www.tuxradar.com/practicalphp>

_What's the best way to learn all the stuff I missed by not going to school?_

 _how can I get a job that is asking for engineers to apply?_

You need skills that those engineers have.

You don't need to go to school to master the data structures and algorithms, for reference there are bunch of university resources available, and there are lot of implemented code for us to checkout. Refer links others have suggested here. Give some time to learn these advance topics.

Edit: added few lines

<comment><AUTHOR> Thanks for your reply. When I said copy and paste PHP developer, I didn't mean I literally copy and paste. I don't think I've copy/pasted code in my work for a few years, except for a few times I just needed to get working code into a prototype.

I meant it more as, trying to get work in Ruby, or Objective-C makes me feel lesser of a developer, as IF I was a copy and paste coder.

</comment></comment><comment><USER4> Others already recommended great books, but I think it might be helpful to clarify one thing. I'm extremely pleased with the breadth and depth of my undergrad education; I'd never encountered ropes, bloom lists, or skip lists. These are all concepts for very specific subsets of problems, so if you were just worried that you hadn't heard of them, don't be. If on the other hand you looked up their implementations and still didn't understand how they worked and couldn't compare them with more mundane structures then most definitely pursue the resources listed in this topic.

</comment><comment><USER5> When I did this I looked at the books necessary for a CS degree from the University of Washington (but it could have been anywhere) and I read them. Nowadays it is easier. A lots of CS classes are online now. Berkeley and Stanford both have full classes in podcast format. MIT has OpenCourseWare which contains the notes and homeworks.

The short of it though is to study the same stuff. There is a lot that a CS background will fill in for you which you won't tend to learn on the job.

</comment><comment><USER6> Why not trying googleing around for information on the topics. You don't necessarily need school to learn.

</comment><comment><USER7> [http://www.quora.com/What-are-the-most-learner-friendly- reso...](http://www.quora.com/What-are-the-most-learner-friendly-resources- for-learning-about-algorithms)

</comment><comment><USER8> Shoot me an email and we can talk about it. I'm working on a related problem right now, so we might be able to help each other out. Address is in my profile.

</comment><comment><USER9> <http://cslibrary.stanford.edu/>

</comment><comment><USER10> If you have experience as a PHP programmer and only as a PHP programmer, AND you feel like you are a copy-paste PHP programmer... you need to program more. Plain and simple. Take on different projects. Use new different tools, just for the sake of having to learn something new or use a new framework or a new data structure or architecture, etc.

<comment><AUTHOR> I only have professional experience in PHP and Perl. I've learning/using in my personal projects Ruby, Python, Objective-C/Cocoa, and Java.

I'm trying to learn the basics of a new language every week for 6 months. Then I'm going to focus on the ones I like best and dedicate a few months to learning the ins and outs of each language.

Thanks for your reply!</comment></comment>