Why Privacy Matters Even if You Have 'Nothing to Hide' (2011) - <AUTHOR> http://chronicle.com/article/Why-Privacy-Matters-Even-if/127461/ <comment><USER3> The "nothing to hide -> nothing to fear" sophistry is about shifting the basis of judgement from "innocent until proven guilty" to "guilty until proven innocent," replacing reason with superstition. It's McCarthyism; it's an inquisition. An individual has no rights over the hysterical "consensus" of the mob.

Unfortunately, there is no piece of evidence that can exonerate a person in such a context; one cannot "establish innocence," the best one can hope for is to plead tirelessley in an endless popularity contest.

The right to privacy is nothing less than the right to keep an individual mind... To own a body. If you value your own life, the right to privacy - <USER2> be left alone - is your beginning and your end, if not kept.

(Edited an error/reversal.)

<comment><USER4> That's very poignant, are these your words or some quote of some such.

<comment><USER3> No quote, just thinking out loud (so, as another comment asked, I can't really offer any recommended reading... Other than maybe this classic episode of TNG: [http://www.youtube.com/watch?v=j8WpjLGh3wg&list=PLB579CFA383...](http://www.youtube.com/watch?v=j8WpjLGh3wg&list=PLB579CFA383430858) ).

</comment></comment></comment><comment><USER5> One fundamental problem with the "nothing to hide" argument is that it makes several casual assumptions which should not be made. An alternate formulation of the saying is "why fear the truth if you have done nothing wrong", which is generally the sentiment of the "nothing to hide" sayings as well. However, the proper form of that saying should be "why fear the truth if you have done nothing the government dislikes". And this is on a much different stance than the others. The issue is not right or wrong, the government is not an absolute arbiter of morality, the issue is government power.

Consider how often and how recently in our own history there have been activities which have been illegal and yet not "morally wrong" as considered today. Aiding escaped slaves. Homosexuality. Inter-racial marriage. Abortion. And so forth.

Also note how I said "nothing the government dislikes" rather than "nothing illegal". And that's because when a government has broad sweeping powers, especially of surveillance, government agents can easily punish people and ruin people's lives regardless of whether their activities are illegal.

Our system of governance has been designed at its core to limit the powers of government. This is very much intentional because it is designed to allow government to enforce the laws only with the cooperation with the public at large. A government that can enforce laws independent of the will of the people is a government poised for the transition to tyranny.

There is a reason why the term "police state" is so reviled even though in itself it contains merely mechanical descriptions. And that is because even if a "benign police state" could exist the danger is far too great that the reins of power would be usurped by those seeking their own ends and their own advantages. And the most forceful way to avoid such a catastrophe is again to limit the power of the state.

<comment><USER6> Indeed. Consider someone is planning on going to an Occupy Wall Street protest. Say that the government is monitoring that persons emails and finds that the person wrote to a fellow protester: "man, fuck corporations, I can't wait to try and bring them down on xday." Now the people reading that private email decide that this message constitutes enough of a threat to act on. They send the police over to that guy's place on the day of the protest and have the police detain him for the rest of the day. They let him go once the protests wind down without incident.

Now I don't think anyone actually committed a crime in that scenario, but it is easy to see how such powers could be used to seriously hinder the democratic process.

</comment></comment><comment><USER7> Read /u/161719's amazing post on reddit on this topic: [http://www.reddit.com/r/changemyview/comments/1fv4r6/i_belie...](http://www.reddit.com/r/changemyview/comments/1fv4r6/i_believe_the_government_should_be_allowed_to/caeb3pl?context=3)

<comment><USER8> I just finished writing a post about pretty much about the same thing ([http://maxmackie.com/2013/06/08/We-have-a-very-important- dec...](http://maxmackie.com/2013/06/08/We-have-a-very-important-decision- ahead-of-us/)).

(I tried submitting my link but I'm getting a "Please try again" error every time, so I gave up. Feel free to post it yourself.)

Thing is, we have the ability to take action right now. The author of the post on reddit is right: things like this already happen in other parts of the world. We have ways of protecting ourselves at our disposal. They're even accessible to non-techy people. People just need to recognize the events of the past couple days as an actual threat.

<comment><USER9> What was the point of linking that article, it is misleading- he didn't provide solutions.

<comment><USER10> Not everything in life is as simple as checking Stack Exchange.

</comment></comment></comment></comment><comment><USER11> Do you fully support the current party in control of the government? If not, you have something to hide.

Do you fully support the party that will be in control of the government in five years? If not, you have something to hide.

You think I'm being silly? Six months ago you might have been able to float that argument, but with an openly partisan IRS, go ahead, tell me why that's wrong.

<comment><USER12> I agree with your message, however was the IRS really being partisan or is it that tea party groups in particular(and those that seek to fund them) are more actively exploiting the "social welfare organization" loophole?

<comment><USER11> The IRS was being partisan. I consider this fairly well established at this point. The only remaining question is _exactly_ where the idea to be aggressively partisan came from.

<comment><USER12> Yet there is evidence they investigated "liberal" groups for the same sort of thing.

<comment><USER11> Applying correct amounts of scrutiny to one set of people, and abusive amounts of scrutiny to another, is still being partisan. Don't let them hide behind that.

Seriously. It's extraordinarily dangerous to our way of life to try to defend the IRS on this one because, I presume, you don't like the Tea Party. This isn't bad news for Tea Party or conservative groups. This is bad news for anyone subject to IRS jurisdiction. Who's going to be in control of the IRS in four years? (We're a ways out, but certainly decent odds it ain't gonna be someone with a (D) after their name.)

</comment></comment></comment></comment></comment><comment><USER13> this article also misses 'falsification'. If the government is collecting data, there is an implicit level of truthfulness in this data, which someone malicious working in the government could use to frame you for a crime or cause other less severe harm to you (financial, social, whatever).

This person need not be accountable, say, to the voter (it could be a low- level bureaucrat with a petty axe to grind), and although presumably there is 'accountability', what if the harm done to you is significant enough to make your life miserable, but not significant enough to the monolithic agency (which remember is only accountable indirectly, to elected officials who may care about stupid stuff like political wedge issues) to justify the expenditure and effort to root out the malicious actor.

</comment><comment><USER14> Most of those articles are missing one important point.

"I have nothing to hide because I don't do anything wrong."

What does "wrong" mean in this context? Against the law.

So - how many torts or crimes are there on the books? How may ways of interpreting them are there - both already writted down and possible in the future? Are they interpreted by flawless machines or emotional humans? Etc. etc.

<comment><USER15> Another point I was not seeing is reciprocity. Even if I really do not have anything to hide--and I might even not mind the naked picture thing, because will anyone actually care?--then I should be able to see everybody else as well, especially the rich and powerful including those who are running the surveillance system.

</comment></comment><comment><USER16> Why would you need to have something to hide to want privacy? What bothers me about being watched or spied on is everyone having their nose in my business. I also see it as an insult, as you are essentially degrading me and treating me as though I _am_ hiding something, when I'm not. My business is none of your business; keep out of it.

</comment><comment><USER17> "Retorts to the nothing-to-hide argument about exposing people's naked bodies ... are relevant only if the government is likely to gather this kind of information"

Yes, because it is crazy to suggest that the US government might be collecting images of people's naked bodies...

(To be clear, I do agree with the article. I just couldn't help but laugh a bit at the idea that the government taking nude photographs of people is depicted as "extreme.")

</comment><comment><USER18> If I have nothing to hide, i.e. am doing nothing illegal, you don't have any reason to be watching me.

</comment><comment><USER19> A good and very comprehensive article on the subject. The only area I wish the article would also include is how society is today built on top of the assumption that privacy exist, and what happens when that assumption is found to be wrong.

To take an example, the court system assumes that judge and jury members private life is not known. When one side in a trial knows the judge or juries dreams, aspirations, deep secrets or just plain biases, that party has suddenly gained an huge unfair advantage. Since we can now start to grasp the amount of information gathered by the state, Google or Facebook, can a trail involving either one of them still be claimed to be fair and just if the other party do not has similar inside knowledge of the judges or juries lives?

</comment><comment><USER20> The real question is: When do you have a reasonable expectation of privacy, either online or IRL?

Yes, I have curtains, but unless I close them, I don't expect privacy. If I'm using mainstream social media sites, I don't expect privacy either. The problem is that most people don't think about the "curtains" being open by default when they're online.

I think we do have a right to privacy and obviously most people have something to hide, but that privacy should require intentional effort to enable (eg closing the curtains, TOR + VPN).

<comment><USER21> The problem comes when "they" start to peek in everywhere. They being the guys in uniform who has guns and the authority to put you in jail.

"First reasonable suspicion, then limited surveillance" have served us well in the past. I don't see how that changes if the communication channel is semi- public.

You never had any privacy in a downtown coffee shop, but if they wanted to listen they had to send someone to follow you around, which kind of limited how much surveillance we had to put up with.

Electronic communication is a game changer, now we have the option to read everyone's mail, all the time.

Unlimited surveillance will lead to unreasonable suspicion - how many honest people may be flagged and get on the no-fly list, perhaps getting their job/visa applications rejected for no obvious reasons so you can _feel_ secure?.

I don't fancy having to watch my words so they don't get misinterpreted by the guys with guns in uniform.

We are the people, we make the laws. And IMHO the law should be very clear ... Gentlemen don't read each others mail - unless there is a very valid reason to do so.

<comment><USER20> I agree that the law should be more clear, but I also think a lot of people are overreacting to this case and not applying a consistant standard of privacy outrage.

Google constantly reads my email to display ads in GMail. That's a tradeoff I'm willing to make for free email service. There are many Google advertisers that I trust less than the NSA, and yet I continue to use GMail.

The big difference is that I didn't have a chance to evaluate the security/privacy tradeoff in the case of PRISM, but if I had, I think I would still choose to use the services of the companies involved with the program.

<comment><USER21> When google are watching and you are careless with your words you may get an inappropriate ad displayed.

When the government is listening you may get on the no-fly list, or worse ...

Not a small difference IMHO.

</comment></comment></comment></comment><comment><USER22> "nothing to hide -> nothing to fear" is a misdirection. It gets the enemy to waste time arguing against something that is obvioulsy stupid.

</comment><comment><USER23> There is no such thing as 'Nothing to Hide' unless you have an extremely naive view of the world. If you feel that way, you just need to remember that out there, somewhere, are those who'll judge you harshly for being you.

Sure, you may not be ashamed of it. You're probably not be breaking any laws that you're aware of, at least none that really matter. You're pretty sure you live in a country that you're free from persecution. All should be OK, right?

Sadly, whether that's true or not, that only works for today. Tomorrow, well it would seem those with enough bile and hate to judge you, seem to be one of the few sets of people who actually want to seek out power and privledge over you.

You sure you're really that free today? You sure you'll really be that free tomorrow. Your sure it'll be the same for your kids? If there is any hesitation, any hesitation at all, then it's probably not a brilliant idea to sit idly by and accept Government creating profiles on you and the future you's.

Dead god, won't anyone think of the children? :P

</comment><comment><USER24> > "What if the government mistakenly determines that based on your pattern of > activities, you're likely to engage in a criminal act? What if it denies you > the right to fly? What if the government thinks your financial transactions > look odd—even if you've done nothing wrong—and freezes your accounts?"

That's a lot of ifs. I don't think it really has anything to do about privacy issues. Privacy issues created those "ifs". That's a paranoid state, not a state of law and rights. Those are issues created by society unable to sort its things. I doubt a government can really forbid things based on information, and if so, it already existed before.

Of privacy issues make things worse, but those issues were the mix of a digital age and terrorism. Nobody can escape the fact congress will vote laws and agencies will want to work on things.

</comment><comment><USER25> In pre-urban times with smaller houses people used to have far less privacy from their families and their neighbours than they do today, but of course back then the lack of privacy was symmetric. I think I would like to live in a society with less privacy, but only if it was something that everybody had to endure. I certainly don't want some person unknown to me spying on me without being able to spy back, that creates a very dangerous power differential.

And of course a big part of the problem nowadays is that everybody does things that are illegal regularly, there are so many laws that how could we not? "Ignorance of the law is no excuse", just the fact that you're not aware of having committed any felonies doesn't in any way guarantee you haven't committed one.

</comment><comment><USER26> Pretty sure I've read with this exact or a very similarly titled article.

Unfortunately, I can't help but find it very unconvincing, despite that I'd like to be convinced. There are lots of words but few strong arguments (or any kind of arguments).

<comment><USER27> Agreed. I read it hoping to be convinced (wanting to be convinced), but it was too wordy and lacked a compelling argument. I do believe privacy is important and we should all care about it. But I also want to read a strong well-written argument to support that position.

</comment></comment><comment><USER28> The fundamental problem with that saying is that it for it to be formed, for it to even exist, as a thought, the reality in which you are currently cohabiting must be terribly skewed off the truth axis. Black must be white, up must be down, and good must be evil.

I mean, look at our society...just look at it. So much of its energy is devoted to ruling people. Judges, courts, bailiffs, clerks, kings, police, lawyers, etc. It's madness. Most people can't even imagine any other way of living...when they write about alien societies, the aliens have laws as well, and clerks as well, and laws as well.

</comment><comment><USER29> Of course privacy matters, even if you are doing nothing wrong. Without it, we'd continually have to explain our actions and prove our innocence to people who don't understand our situation.

And that's a lot of work. And they really aren't motivated to understand.

And its a lot of work trying to understand everyone else's situation. I certainly don't want to.

So nature gave us a need for privacy and the means to respect the privacy of others. And even the sense to often pretend that we don't see many things.

<comment><USER3> I have often thought that any successful attempt at AI would have to reproduce the behaviors associated with fear/paranoia and _lying_. (Or maybe I'm just taking _The Hitchhiker 's Guide to the Galaxy_ too literally.)

<comment><USER30> I would say that there is a difference between artificial intelligence and artificial _human_ intelligence.

Then again, I also prefer the terminology "synthetic" to "artificial".

</comment></comment></comment><comment><USER31> "So the government is spying on me. So what?"

[http://pastebin.com/7SRmFpFH](http://pastebin.com/7SRmFpFH)

<comment><USER32> Anyone without a Facebook account of `median person' content and traffic will be flagged ORANGE for further investigation. Of course this includes the homeless and off-grid.

New phone app opportunity:

Chaff-bot of expected `median person' content and traffic into their's and your own Facebook account.

</comment></comment><comment><USER33> Regarding the original Veriz court order.. Answer me this: Does your phone number, or the fact that your particular string of numbers is a phone number, somehow exist in the universe with some sort of innate privacy apart from you? Nope. What exactly does a phone number, absent a name, address, billing info, or other customer info, mean to anyone, even the NSA.

<comment><USER34> The fact is that if you have someone's phone number, you generally have access to all that other information because linking it to a name and thus everything else is fairly trivial, even for ordinary people not affiliated with the NSA. Claiming it's somehow okay and protecting privacy because they have the phone numbers without names is ridiculous.

</comment></comment><comment><USER35> Basically, knowledge is power. The more knowledge the government have the more power they have over you. We all do something wrong, do drugs, be unfaithful, avoid taxes, bend rules/laws, say things against friends/family/employer, etc, etc, all this can be used to blackmail and control us. We all have something we want to hide from someone.

</comment><comment><USER36> Although I agree the "nothing to hide" argument is fundamentally flawed, I have yet to read a solid rebuttal to it. This article has not broken that trend. While it has given me some food for thought, I feel it resorts to a weak slippery slope argument, not quite capturing what feels truly wrong about universal monitoring.

\--garbled by my mobile

</comment><comment><USER37> Very related: [https://news.ycombinator.com/item?id=5848540](https://news.ycombinator.com/item?id=5848540)

</comment><comment><USER38> I think there's a big difference between privacy of information about you and privacy of your physical self, which is a bit jumbled in this article.

</comment><comment><USER39> For once, an article that defends privacy while recognizing the non- obviousness of the issue. That makes it unusually convincing, but there is still one part I find weak:

 _Investigating the nothing-to-hide argument a little more deeply, we find that it looks for a singular and visceral kind of injury. Ironically, this underlying conception of injury is sometimes shared by those advocating for greater privacy protections. For example, the University of South Carolina law professor Ann Bartow argues that in order to have a real resonance, privacy problems must "negatively impact the lives of living, breathing human beings beyond simply provoking feelings of unease." She says that privacy needs more "dead bodies," and that privacy's "lack of blood and death, or at least of broken bones and buckets of money, distances privacy harms from other [types of harm]."_

 _Bartow 's objection is actually consistent with the nothing-to-hide argument. Those advancing the nothing-to-hide argument have in mind a particular kind of appalling privacy harm, one in which privacy is violated only when something deeply embarrassing or discrediting is revealed. Like Bartow, proponents of the nothing-to-hide argument demand a dead-bodies type of harm._

 _Bartow is certainly right that people respond much more strongly to blood and death than to more-abstract concerns. But if this is the standard to recognize a problem, then few privacy problems will be recognized. Privacy is not a horror movie, most privacy problems don 't result in dead bodies, and demanding evidence of palpable harms will be difficult in many cases._

The author dismisses the "need for blood" as too extreme, but that is not convincing. For example, consider the most compelling argument in the article, which is that government data collection creates a Kafkaesque world in which the opacity of the procedures renders the individuals powerless:

 _Government information-gathering programs are problematic even if no information that people want to hide is uncovered. In The Trial, the problem is not inhibited behavior but rather a suffocating powerlessness and vulnerability created by the court system 's use of personal data and its denial to the protagonist of any knowledge of or participation in the process. The harms are bureaucratic ones—indifference, error, abuse, frustration, and lack of transparency and accountability._

This is a powerful argument, but it is not going to convince a majority of people unless you can show real harm. Not blood or broken bones, but actual cases where good guys (as perceived by the majority) were harmed by government surveillance, e.g. harassed or put to jail. Why would people feel a "suffocating powerlessness and vulnerability" if the Kafkaesque bureaucracy does not eventually bring actual harm? (The case of Aaron Swartz is not an example: the problem lied with the CFAA rather than government surveillance).

I'm still optimistic: the process will work as usual. Government powers will be abused. There will be scandals. In the long term, laws will be improved and transparency will increase. What reasons do I have to think otherwise?

</comment><comment><USER40> The whole argument is premised on the idea that the laws are all moral and just and government is some noble protector, watching over us. The reality is that the government is a monopolistic protection racquet and I pretty much want to hide everything from them. Surveillance is a mechanism by which they control their subjects.

The more they know about me the more they can steal off me and the more liable I am to be attacked for disobeying one of their edicts. If I want to do something that the government arbitrarily decides it "wrong", then I've got something to hide.

So yeah, I have everything to hide from a criminal gang.

<comment><USER12> So the government does nothing worthwhile? Public roads? The FAA? The park service? The EPA? GPS?

Should all of that be privatized, where there is absolutely no control available to the public? These are services that are by nature a monopoly, we can't really vote with our money.</comment></comment>


Explosives planted on man to test airport security - <AUTHOR> http://news.bbc.co.uk/2/hi/europe/8441891.stm

<comment><USER2> "Slovak authorities were reportedly trying to test screening procedures for checked-in luggage by placing items with unwitting passengers."

Red-Team screening is a fairly important element of the TSA's approach to testing network security, but you run your own people (with appropriate ID and escalation contacts) through these security checkpoints. At the _most_ you grab some civilian security contractors.

Something about "unwitting passengers" makes me want to ask WTF? If the Slovakian minister for the interior had not apologized, I wouldn't have believe the article had their facts straight.

Bizarre.

</comment><comment><USER3> There are some inaccuracies in BBC report. Here is what happened according to Slovak news [1]:

Slovak police was training explosive sniffing dogs at Poprad airport. They planted several samples of explosives in passengers' luggage (90 grams of RDX / Hexogen [2]).

Here versions differ - Slovak police claims they planted two, Irish media tell they planted eight samples. One of these was missed by dogs and subsequently police screwed up, forgot about it and let it pass to the airplane.

The explosive sample was in a baggage of Slovak citizen living and working in Ireland.

Once the police realized their mistake, they let know the pilot (still during the flight), and he was supposedly ok with it.

They also let know destination airport in Dublin. Again here versions differ - Slovak police says they did it already on Saturday, when the incident happened, Dublin airport claims they got the message only on Tuesday.

Three days after the flight, once Irish authorities got to know there is a missing explosive (from Slovak official sources), they sent a bomb disposal squad to poor unsuspecting passenger, cleaned two streets around him, and arrested him.

\----

BTW This was not the first time such incident happened. Five years ago similar thing happened at Paris Charles de Gaulle airport:

<http://news.bbc.co.uk/2/hi/europe/4069785.stm>

Also, not explosives, but similarly Japanese police lost planted drugs in an unknowing traveler's luggage:

[http://www.boingboing.net/2008/05/26/japanese-customs- pla.ht...](http://www.boingboing.net/2008/05/26/japanese-customs-pla.html)

\----

[1] [http://www.sme.sk/c/5179110/policajti-na-letisku-schovali- vy...](http://www.sme.sk/c/5179110/policajti-na-letisku-schovali-vybusniny-do- batozin-jedna-odletela-do-dublinu.html)

[2] <http://en.wikipedia.org/wiki/RDX>

</comment><comment><USER4> I never travel with anything more than cabin approved luggage, no matter how long I go away from home.

There is the odd weird look when I pass customs with nothing but a shoulder bag for a 2 month stay (hey, they do have washing machines!) but usually not much trouble because of that.

If you let your bags out of your sight then anything could happen to them, you never know who could mess with your bags, some thrower or other person behind the scenes using you as a mule.

It happens, and it happens more often than you think. This is the first time though that a government is on the record as having done this.

What an absolutely unbelievable breach of trust. These people ought to be fired.

<comment><USER5> I do the same, just a backpack. Have customs agents ever given you crap for not having much luggage?

<comment><USER6> I also do this whenever possible. I've been told by the person handling checking in that I may face an enhanced screen or words to that effect but have always breezed through without any issue.

Edit: My experience is travelling from the UK to US and Europe and back.

</comment></comment></comment><comment><AUTHOR> Absolutely unbelievable. A massive abuse of trust on the part of the police forces.

</comment><comment><USER7> Shouldn't we have agents trying to get through security with guns, explosives and anything else that's restricted, multiple times per day so we can measure how effective airport security is and improve it? Ideally these agents would not get on the plane, or what they are given to carry through could be disabled in some way.

Otherwise how do we know it works at all? How do you check that the screeners are doing their jobs properly?

<comment><USER8> _"Otherwise how do we know it works at all?"_

This may be cynical of me, but I suspect this is precisely why we _don't_ do penetration testing regularly...

Funny scene a couple of days ago - I was taking the train from Canada to the USA, pre-clearing US border security before getting on the train. The gentleman in front of me had a pair of nail clippers in his carry-on, and was told he needed to put them in checked luggage to get on the train.

First of all, nail clippers. Really? _sigh_

Secondly, passengers carried their checked luggage past the security checkpoint themselves, out of sight of the security guys, and voluntarily placed their luggage on the checked luggage cart sitting essentially unattended. That gentleman, if he meant any harm, could have easily taken his nail clippers and gotten on the train with it and... well, I dunno what you would _actually_ do with nail clippers on a train that's really so bad.

<comment><AUTHOR> He might decide to fly the train into a building.

</comment></comment></comment><comment><USER9> Wow, I am surprised this guy didn't get tortured or anything. How is he supposed to convince a foreign country that the bombs discovered in his bag are not his?

<comment><AUTHOR> "Slovak authorities admitted planting them in his luggage" - First they told the Irish police he had a bomb, then they told them that they were at fault. It is a good thing that the innocent traveller didn't get severely injured in the raid.

</comment></comment><comment><USER10> Why would slovac authorities want to test the security of Irish airport security?

<comment><USER11> I believe they were testing their _own_ airport security, and it failed, resulting in the man getting through to Ireland with the planted explosives.

</comment></comment><comment><USER12> Finally a good reason not to leave your luggage unattended at the airport.

</comment><comment><USER13> Stolen from another forum: "Is there not an 'Eicar' equivalent for testing airport security?"</comment>


Tokyo Cabinet DB: Beyond Key-Value Store - <AUTHOR> http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/ <comment><USER2> I have built a distributed key-value store on top of Tokyo Tyrant (which is a network interface on top of Tokyo Cabinet). I have also tried to use memcachedb (that's based on Berkley DB) and Tokyo Tyrant outperforms it - especially with large amounts of items. memcachedb has also problems with log (it grows furiously large and db checkpoints stalls the database for several seconds).

One can also script Tyrant via Lua (which is pretty amazing) and there's master-master replication. Here's how to implement incr via Lua: <http://paste.pocoo.org/show/103848/> I have currently extended it with a fixed list and Mikio (the dev of Tokyo *) has implemented an inverted index for search.

Currently I have my key-value store in production (it has around 1 million key-values) and the performance is pretty good.

I will release it open-source, it's only Python now, but the code is around 200 lines + a consistent hash library implementation ( which I implemented in <50 lines of code <http://amix.dk/blog/viewEntry/19367> ) so it should be possible to port it over.

<comment><USER3> Is there something like CAS of memcached? It's fine to live without locking but low-level atomic functionality is essential.

<comment><USER2> There's global locking and key locking. I currently use key-locking when updating.

You should be able to implement CAS in Lua I guess, but it isn't supported from the start.

<comment><USER3> Key-locking makes sense, I'll give it a try. Thanks for the info!

</comment></comment></comment></comment><comment><USER4> Sounds impressive and cool. I wonder about Table engine performance though.

My experiences with Berkeley DB (using the native C API) were positive, from a performance point of view, largely _because_ BDB has no built-in query support. When I wanted to look something up, and I had no index on the desired value (BDB calls indices "secondary databases"), I always knew I was doing a linear table scan, because I had to write the loop myself. No SQL meant that inefficient queries could not hide in plain sight.

Berkeley DB trades off the flexibility of SQL queries for (1) excellent lookup performance in simpler cases, and (2) the ability to mix relational and object-based data storage, reducing the famous impedance mismatch of relational data modeling. For some applications, this trade-off works beautifully. I daresay that it holds true for most web apps which really don't need a relational data store, and either outgrew pure in-memory storage or just sensibly want to persist their state to disk. (Hint: if you use an ORM, then you probably don't want a relational data store.)

For some other apps, the lack of SQL queries hurts. I once worked on a trading system implemented on top of a proprietary engine which did not support arbitrary queries. It made writing even trivial reports ("list all transactions done two days ago by trader X using instrument Y") unnecessarily painful. (Of course, on a typical RDB with a non-trivial schema and a large dataset, these arbitrary queries would take 20 minutes to run, so catch-22.)

The Table engine for Tokyo Cabinet seems to do everything implicitly. Fast and arbitrary queries. If it works on the kind of schema-less free-form data the author refers to, and has working master-master replication, that sounds almost too good to be true.

<comment><AUTHOR> Some performance benchmarks: <http://gist.github.com/64391>

They're not conclusive, but give you an idea of relative performance.

</comment></comment><comment><USER2> For Tokyo Tyrant there's a Python library: <http://code.google.com/p/pytyrant/>

It uses the binary protocol and it's written in a very simple and very clean way - and it should be production ready.

I would not recommend using the memcached protocol with Tyrant (it offers less operations and it isn't binary).

<comment><USER5> I'm currently using Tokyo Tyrant with Rails and the Memcached client (in production). It works great for my needs though I don't need to story binary data. Overall, TT has been extremely fast and stable. The dual master replication is a nice bonus.

</comment></comment><comment><USER6> The article is mostly about the Ruby bindings, but Tokyo Cabinet is written in C99 and has bindings for most of the usual suspects. English language specifications are here: <http://tokyocabinet.sourceforge.net/spex-en.html>

<comment><USER7> Just as an aside, using Ruby to access the db server via Net::HTTP should limit the performance to at max a few dozen http requests per second per (Ruby) process.

I think the more http is used to bridge web architectures, the more critical this aspect will become for Ruby.

<comment><USER8> It's safe to say there's some reasonable replacements already, and there are some smart guys working on better ones.

[http://www.pauldix.net/2009/01/ruby-http-client-library- perf...](http://www.pauldix.net/2009/01/ruby-http-client-library- performance.html)

But, if you didn't know better, yes, Net:HTTP is going to negate the amazing speed of Tokyo *.

</comment></comment></comment><comment><USER9> Haven't used it, but Erlang bindings: [http://dukesoferl.blogspot.com/2008/06/tokyocabinet-and- mnes...](http://dukesoferl.blogspot.com/2008/06/tokyocabinet-and-mnesia.html)

<comment><USER10> Those erlang bindings are very immature. I wouldn't use them in production.

<comment><USER9> Good to know, thanks.

</comment></comment></comment><comment><USER11> So, this is a BDB clone without the years of testing of BDB? That's exactly what I want for my critical data.

I am really confused as to why one would implement this in C. One pointer math problem, and _boom_ , gigs of data are corrupted. If you are going to reimplement BDB, you might as well do it in Haskell or OCaml and buy yourself a little type safety. (Incidentally, this is on my TODO list, but BDB works well enough for us at this point.)

<comment><USER12> The table DB type and Tokyo Tyrant for network access seems like a pretty big deal to me. The full text search seems pretty damn awesome too. With regular backups it doesn't seem that risky for certain things.

<comment><USER11> This is really easy to add on top of BDB, though, that's why I ask.

(As an example, we implemented KiokuDB on top of BDB: <http://www.iinteractive.com/kiokudb/>. We haven't added full-text search yet, but we plan to do it soon.)

</comment></comment></comment><comment><USER13> There is also a full-text search system for Tokyo Cabinet called Tokyo Dystopia (<http://tokyocabinet.sourceforge.net/dystopiadoc/>)..Has anyone written Ruby API for this?

Was wondering whether it be a superior option for Rails search ahead of Xapian and Sphinx.

<comment><USER14> rufus-tokyo has the ruby interface: [http://github.com/cheapRoc/rufus- tokyo/blob/7e524947825b6f7f...](http://github.com/cheapRoc/rufus- tokyo/blob/7e524947825b6f7f4ad8801325f970a655c8439d/lib/rufus/tokyo/dys_words.rb)

</comment></comment><comment><USER15> any PHP API??</comment>


Show HN: Learn React JavaScript in Browser - <AUTHOR> https://www.educative.io/collection/5638830484881408/70001 <comment><USER2> I would love to see a next & previous arrow for navigation after i complete each section.

<comment><USER3> I'm one of the co-founders of Educative. We are working in it.</comment></comment>


Backing up elasticsearch indices with curator and minio - <AUTHOR> https://aboullaite.me/elasticsearch-curator-minio/ <comment><USER2> AWS specific. Elastic is on gcp too y'know...

<comment><AUTHOR> This has nothing to do with AWS! I'm aware that elastic is available on other cloud providers :) this is an on premise alternative to S3.

<comment><USER2> The scripts embed Amazon dependencies don't they? If I mis label this as AWS depending, can you explain how I use this to do a dump on a gcp hosted es instance?

<comment><AUTHOR> I think you just missed the purpose of the article! It simply explains how to backup your es indices on minions, which is a S3 alternative and that's why the use of s3 plugin! There are different plugins for different cloud providers that's above the scope of the post!</comment></comment></comment></comment>


Mall of Cosmetics - <AUTHOR> https://www.mallofcosmetics.com/ <comment><AUTHOR> The use of essential oils is traced back over 6000 years ago. The Ancient Chinese, Egyptians, Greeks, and Romans used these essential oils in aromatherapy for a healthy mind and body. You can try aromatherapy at home too! Buy your essential oils at Mall of Cosmetics now.</comment>


Ask HN: Why are comp engineers being hired as software devs? - <AUTHOR>

Hi HN, I recently started a new position in a software development firm and most of the devs there have a computer engineering background rather than a computer science background.  I actually feel like the odd person out with my comp sci degree.  Is this a growing trend in the field?  What does this mean for the newer disciples of computer science coming out of school also having to compete with computer engineers for the same job and why do engineers have a leg up on them? <comment><USER3> Computer Engineers and Computer Scientists take essentially the same software design related courses. The difference is that Computer Engineers don't have to write an Operating System, and instead dig deeper into Electrical Engineering. I had to do Software Engineering, software theory, electrical design, hardware/chip design, embedded systems design, web design, all of it.

Being a Computer Engineer myself, and choosing Computer Engineering over Computer Science, I'd say the degree is tougher to earn, and you learn a lot more random things (there are a lot of quality algorithms and design approaches that Electrical Engineers learn, and it's very convenient to have been exposed to them, to have my mind open to the possibilities, and to have them in mind as I work) that give a slight edge everyday. Also, to generalize, Computer Engineers tend to be more versatile, open-ended, jack-of-all-trades types (fits with the Entrepreneurial thing), versus the specializing Computer Scientist. They also tend to keep up with new developments and look for new and different ways to do things. Of course, a crappy Engineer is a crappy Engineer, and many that make it through the Computer Engineering course end up specializing, because they can't handle it all.

I don't think it's a growing trend, though, and my experience had always been similar to that of a designer's: not enough credit/appreciation. Maybe that's changing, I don't know. Also, all these random things I know, and the difference in my approach to problems has always been continually suppressed, watered down, and stomped on by management, and after a few months/years, I realized the only way to make full of my potential, my approach to things, and what I know was to branch out on my own.

<comment><USER4> I thought a CE degree tends to learn less theory than a CS degree. For example, my favorite CS course was language and automata theory. I looked at the ECE courses for UIUC (<http://www.ece.illinois.edu/courses/> ) and they don't offer an automata course, or data structures. For that you need to take a CS course [http://cs.illinois.edu/undergraduates/academics?quicktabs_1=...](http://cs.illinois.edu/undergraduates/academics?quicktabs_1=3#quicktabs-1)

In my career so far developing software for chemistry I've made heavy use of graph theory, formal language theory, and data structures that I learned while an undergraduate.

I don't have the same assessment as you in the advantage of a CE degree over a CS degree. I suspect there's a bias error, as you did not enter a field where a CS degree would have been more useful. Nor do I think that a CS degree is specializing, given the courses I took in operating systems, programming language theory, computer graphics, databases, etc. Of course, from my perspective "electrical design, hardware/chip design, embedded systems design" are all variations of the same thing, so it's a definite matter of perspective.

In any case, after 5 years or so in a career, the specific advantages of one background over the other get pretty weak.

<comment><USER3> Yes, I didn't take the Automata Theory course, but I learned essentially the same thing in Discrete Mathematics, Control Systems, and Digital Design (albeit with an eye towards Electrical Engineering applications). I remember looking through the courses, as you did, and the only gap I remember noticing was Operating Systems (and maybe graph theory, now that you mention it). It would have been nice to be introduced to machine learning while in school, but that's fine, I ended up learning it on my own.

There could easily be a bias error, but I said what I said simply because everything covered in the CS curriculum seems to also be present in the CE curriculum, but in the CE curriculum more is added. So I see the CE degree as a CS degree plus the essentials of an Electrical Engineering degree.

The advantages, if there are any, can definitely fade away after a few years of real experience, assuming everyone continues to better their knowledge, and those years aren't spent, as is typically the case, repeatedly re-implementing the same solution.

<comment><USER4> Now I'm curious - which of those courses covered the pumping lemma or LR(1) grammars? I'm surprised to hear that they are relevant to a Discrete Mathematics, Control Systems, or Digital Design course.

Or was those covered in your compiler course?

We covered graph theory in discrete math - it wasn't an independent topic - <USER2> I'm surprised that you didn't cover it in your discrete math courses. As I recall, first semester was more combinatorics and second was more graph theory, but I don't recall at this point.

Also, in reading the ACM curricula guidelines, I came across the comment "As is typical of most programs in engineering, this program has a smaller general education component than Curriculum A described in this appendix, with more hours devoted to computer engineering topics."

That is (and using your earlier terminology), a computer engineering degree is more specialized, while a computer science degree will include more non-CS courses and give a 'more versatile, open-ended' education. :)

</comment></comment></comment></comment><comment><USER5> Likely it's coincidental. The firm probably just hired whomever they thought could best do the job, largely ignoring the degrees. Maybe the people with a computer engineering background demonstrated better software development skills or aptitude.

Suppose the company wants experience in Obscure and one of those computer engineers spent the weekend reading Obscure's manual and playing with the software before the interview. That could be the leg up.

</comment><comment><USER6> Comp engineers may have an initial leg up because their field of study is more practical than computer science which is more theoretical.

Don't worry about it. You were hired for a reason and you should have complimentary strengths to the engineers

</comment><comment><USER6> Also worth mentioning. Many schools don't have engineering and some engineering schools don't have comp sci.

For example in chemistry companies will teach new hires the engineering or the science depending on which degree they have.</comment>


CTM author Peter Van Roy: Programming Paradigms for Dummies [pdf] - <AUTHOR> http://www.info.ucl.ac.be/~pvr/VanRoyChapter.pdf

<comment><USER2> Author's page

<http://www.info.ucl.ac.be/~pvr/cvvanroy.html>

I'm guessing the PDF chapter is from this textbook, described on the author's page:

Programming textbook: Concepts, techniques, and models of computer programming

It's published by MIT Press.

EDIT: Page for English version of book: <http://www.info.ucl.ac.be/~pvr/book.html>

<comment><USER3> No, that chapter is not from Concepts, Techniques, and Models but most of the programming models mentioned in that PDF are covered in more depth in CTM.

<comment><USER2> Oh, my bad. Thanks for the correction. Since I took the time to find a little context for the orignal linked PDF, I thought I'd share for others' convenience.</comment></comment></comment>


Nars2000: An Experimental(open source) APL Interpreter - <AUTHOR> http://www.nars2000.org/

<comment><USER3> "J is a modern APL dialect with ASCII characters."

It is a great language, but not Open Source. Hard to commit to a closed source language these days.

EDIT: If anyone knows of an open source equivalent to J (apl like , ascii characters) , please submit a URL here. I'd love to poke around its innards to see how it worked.

<comment><AUTHOR> Take a look at famous "J prototype" source inspired by Arthur Whitney.

<http://www.jsoftware.com/jwiki/Essays/Incunabulum>

amuse yourself :)

<comment><USER3> "Take a look at famous "J prototype" source"

thanks! Anyone know where I can find "An Implementation of J" from which the source fragment is extracted?

</comment></comment></comment><comment><USER4> J is a modern APL dialect with ASCII characters.

<http://www.jsoftware.com/>

<comment><AUTHOR> I prefer q/k ASCII APL dialect from kx systems[kx.com], it's not open source, but they released kdb+ free personal edition to play with.

<comment><USER5> _kdb+ free personal edition to play with_

which drops state and restarts every two hours.

<comment><AUTHOR> then there is kdb+ terminal on <https://code.kx.com> But I admit - <USER2> are annoying.

<comment><USER5> I wonder, but am too lazy to check, if this can be circumvented by changing kernel jiffy definition. K has to get the time from somewhere.

<comment><AUTHOR> Honestly I have no idea...</comment></comment></comment></comment></comment></comment>


How the FDA process is biased against new technology - <AUTHOR> http://mobihealthnews.com/46693/is-the-fda-process-biased-against-new-technology/print/ <comment><USER2> Hmm... the MelaFind thing doesn't sound risk-free to me. If it's approved, there are two ways that could go wrong:

1) Doesn't identify a real cancer (risk is obvious)

2) Falsely identifies a non-cancerous mole as cancerous (exposes patients to unnecessary surgery, which always carries some risk, even with something as minor as a typical mole removal).

In both of those cases there's going to be a specific, identifiable dead or injured patient (or family of same) raising hell, creating a PR black eye for the FDA.

If the software _isn 't_ approved, people will probably also die, but they _won 't be as identifiable_, at least not nearly as easily. They likely won't even know.

I'm not saying that this is a good situation ('cause it isn't), but that's how things are. I don't think they're biased against technology, I think they're biased against risk (specifically, risk that might reflect badly on the FDA).</comment>


Dell Starts Taking Pre-Orders For The XPS 12 Convertible Tablet / Ultrabook - <AUTHOR> http://www.maximumpc.com/article/news/dell_starts_taking_pre-orders_xps_12_convertible_tablet_ultrabook_starts_119999 Few would argue that Windows 8 is a radical departure from a software perspective, however this generation it won't just be Microsoft looking to set new trends. Redmond's hardware partners are gearing up fast for the October 26th release date, and Dell has started taking pre-orders for its new XPS 12 convertible Windows 8 tablet/Ultrabook. <comment><USER2> I am really looking forward to this generation of laptops. Years ago I had a 12.1" HP TX1000 which had a built in wacom and the screen rotated / swiveled, and it was awesome.</comment>


I found the Holy Grail of backups (2014) - <AUTHOR> https://www.stavros.io/posts/holy-grail-backups/ <comment><USER2> To what extent do companies like Spideroak mirror your backups? Sure you can rent a VPS for a few dollars a month, but setting up a reliable infrastructure will take more than one VPS, no?

</comment><comment><USER3> I'd recommend Borg, a fork of attic with better compression options and better filesystem support (i.e. NFS).

</comment><comment><USER4> Seems a bit stale: "Conceived on 10 Apr 2014"...maybe note the year, in the title (?)

<comment><AUTHOR> I've added the year to the title, but the content is far from stale.

I'd never heard of Attic before, and just like the author my reaction was "how did it take so long to find this?!" It's easy to use, simple yet configurable, featureful, and has the slickest way to restore snapshots I've seen. And the blogpost linked gives a much better overview of that awesomeness than the Attic site directly ( [https://attic-backup.org/](https://attic-backup.org/) )</comment></comment>


Chinese companies rally around Huawei after CFO arrest - <AUTHOR> https://asia.nikkei.com/Economy/Trade-War/Chinese-companies-rally-around-Huawei-after-CFO-arrest <comment><USER2> It would be great if instead of rallying for boycotts, they could rally for the release of the two Canadians that disappeared.

<comment><USER3> I'm sure that it's retaliation over the Huawei arrest. But those two aren't exactly innocent tourists.[0] And I'm sure that the Chinese could make passable arguments for their detention. Espionage is a difficult charge to disprove.

0) [https://abcnews.go.com/International/wireStory/biographical-...](https://abcnews.go.com/International/wireStory/biographical- sketches-canadians-missing-china-59790435)

<comment><USER4> That's a viewpoint of a bullshit legal system of thugs, based on the accused being guilty until proven innocent.

<comment><USER3> Actually, that's a pretty common standard. It's more or less a part of legal systems based on the Napoleonic Code. Defendants are basically treated as guilty until proven innocent. Don't get arrested in Mexico, for example, unless you can pay good bribes.

And actually, now in the UK you can be jailed for refusing to provide decryption keys or passphrases. We just saw the new policy in India. And things are heading that way in the US. Also, those charged with politically incorrect offenses are more and more treated as guilty until proven innocent.

So anyway, claiming moral superiority is an iffy proposition.

</comment></comment></comment></comment><comment><USER5> The US obsession with huawei is at best, a corporate witch-hunt. Its a case of wanting to have your cake, and eat it too.

You cannot spend 40 years outsourcing manufacturing for near-term profit, and expect to maintain longer term revenue as a market leader. Sooner or later, your cheap manufacturers will outpace you in quality, price, and features.

Just ask Harley Davidson. Their solution to high quality competition from Yamaha and Honda was, albeit less aggressive than this, protectionist legislation.

<comment><USER6> It is not a witch hunt-All Five Eyes countries have independently come to the same conclusion.

There is very little reason to trust corporate China, they have a horrible track record of breaking rules, cutting corners, cheating, hell even the regulators don't trust their own numbers.

It's not far fetched that a country which steals IP because they don't have the means to produce people that come up with these valuable IPs, would do everything in their power to subvert, infiltrate and manipulate to achieve their objective.

It's like this. China has really bad street cred. People buy their product because its cheap, but that's about as far as the trust from West goes.

Don't act surprised when China comes under extra scrutiny and fails the test. It's already happened to steel, ship building where companies no longer trust Chinese quality, and the trade war has largely benefitted the Korea and Japanese as a result.

</comment></comment><comment><USER7> The us involved Canada in this trade war on purpose by issuing this arrest warrant that we had to execute because of our legal agreements with the us. Now the Chinese is retailating against us and we will have to respond. Great.

It would be one thing if we had willingly decided to get involved but this is us being forced to get involved.

<comment><USER8> Canada can choose to walk away, which would be smart. But apparently there is some miscalculation, now they stuck in between two superpowers.

<comment><USER9> >Canada can choose to walk away

What is "walking away" in this context? Releasing Meng, and in the process, reneging on the extradition treaty?

<comment><USER8> Your claim makes it sound like Canada has to do whatever US told it to do under any circumstance. I had hard time to believe that is the actual situation here.

<comment><USER9> >Your claim makes it sound like Canada has to do whatever US told it to do under any circumstance

No, Canada has to do whatever the US-Canada extradition treaty says, not whatever the current US administration tells it to do.

>I had hard time to believe that is the actual situation here.

How so?

</comment></comment></comment></comment></comment><comment><USER10> > Other companies are boycotting Apple, which is battling with Huawei for > second place among the world's smartphone producers.

> A machinery maker in Shenzhen, where Huawei is based, threatened to > confiscate Apple devices from employees and fire those who did not comply. > Menpad, a Shenzhen-based tech company, said it would punish employees who > buy Apple products. Finally, Shenzhen Yidaheng Technology said it would fine > staffers who bought iPhones the equivalent amount of their device, while > other companies threaten to withhold bonuses.

</comment><comment><USER11> They are boycotting Canadian brands.

<comment><USER4> The boycotts are hollow when Chinese airlines continue to buy Boeing and Bombardier jets.

</comment></comment><comment><USER12> It's become apparent that in the world of technology you cannot trust China. Rightfully earned or not, instead of arresting people, I think the Chinese communist state should do more to instill trust in people.</comment>


Even Tetris is hard to test - <AUTHOR> http://blog.jwhitham.org/2014/10/its-hard-to-test-software-even-simple.html

<comment><USER2> The interesting bit that isn't mentioned is that all the given examples are actually part of the game specification, i.e. those behaviours are there on purpose. It means that if you have an accurate list of the desired features, you could probably also achieve 100% coverage. It is also possible that testing some features would not increase the code coverage.

"Hard to test" in the submission title didn't mean what I thought: I though it meant it was hard to write tests for Tetris, not that it was hard to recover a complete specification of the game while playing it.

<comment><USER3> The corrolary is that you must have a specification in order to test comprehensively. Either you use some form of TDD and your tests are your specification (and any behavior not under test is undefined), or you have exact and complete specifications that the as-built software can be compared to, and any hehavior not in the specficiation is undefined.

</comment></comment><comment><USER4> This is amusingly naive. I've seen the spec for Tetris, and it's surprisingly big - much larger than the smallest known fully conforming code, which is still surprisingly large. There are also a non tiny number of people knocking around for whom it's their entire job to test it. "Even" Tetris? No, sorry - that's nonsense.

The big problem here is that code coverage tests don't help you cover what you should have explicitly defined or tested but didn't. As a result a lot of things end up still defined by implementation and not specification, as all sorts of important details only got defined during implementation.

</comment><comment><USER5> Nitpick: As an avid Tetris player, I have to say that the examples given for "extremely rare" events actually happen quite often during normal play. In particular, clearing 4 lines at once, two times in row will generally happen within the first few minutes of game play and is sort of the whole goal (if you're trying to maximize points). Wall kicks are less common but certainly not rare enough that their processing code would be left uncovered after playing a few games.

<comment><USER6> Are there more rare scenarios that you can think of?

<comment><AUTHOR> There's a lot of special conditions around having almost the entire playing field filled in.

</comment></comment></comment><comment><USER7> One way to reduce the number of possible paths is to reduce the number of distinct cases, often via refactoring or a simpler yet more general algorithm; to use an extremely contrived example, it's like the difference between

        int addOne(int x) {
          if(x == 0)
            return 1;
          else if(x == 1)
            return 2;
          ...
        }

and

        int addOne(int x) {
          return x + 1;
        }

I always keep in mind the famous Dijkstra quotes about testing and program complexity:

"Program testing can be used to show the presence of bugs, but never to show their absence!"

"Simplicity is prerequisite for reliability."

<comment><USER8> This is a perfect example of why line/function coverage is a silly measurement. It doesn't take into account global state and function input. Take the second function above, you could get 100% line coverage in just one run. But the function does _exactly_ the same thing as the one where you only got 50% coverage, already things smell fishy. You can also test the function with millions of different inputs, all of them giving the same 100% coverage, and it will work perfectly fine, until suddenly you try addOne(int.max) and it will fail.

What you want to be really sure is state coverage, or input range coverage assuming your functions are pure. Now testing every function from int.min- int.max might seem unrealistic but what you have to do then is constrain the possible range of input or divide into ranges with special cases that you can somehow group together. Say for example int.min, negative numbers, zero, positive numbers and int.max.

Also, just because you covered a line doesn't mean it's correct, the only thing you've really tested is that the program doesn't crash. For the test to be really useful you also need a correct result of the output, added by a human. You can't just randomize input to increase the coverage.

<comment><USER9> I came to this thread hoping somebody would say pretty much exactly your first paragraph. The author appears to be selling a code coverage tool so of course he falls into the trap. In real life you have to remember that hitting a line once is not the same as showing it to be correct. People who buy into code coverage tools make this mistake a lot.

</comment></comment></comment><comment><USER10> Bah one line of BBC basic. One line of _incomprehensible_ BBC basic more like. When I was at school I implemented a game in three lines of spectrum basic. 5 if you wanted scoring. And it was readable. And it was the most popular computer game in the school by virtue of it being quicker to type than loading a tape, and being moderately entertaining. Just bound the cursor keys to the values in the line() function. The game crashed if the line went out of bounds :).

</comment><comment><USER11> This is a great example of the value of test plans. This is basically technology to reconstruct missing test plans via the code. But of course someone already knew about the subtle "wall kick" feature since she or he wrote that code. It shouldn't be this hard, with some effective communication.

And actually, especially in games, test plans are still poorly communicated. In the old days, it was awful -- you would have the publisher doing all the QA, and barely speaking to the development team apart from bug reports. QA still often doesn't get involved until the last half of the project, before which time nobody has been thinking much about testing.

As studios improve their production, this is getting better. As a programmer, I've had more collaboration with QA as I work, at its best including having our QA liaison talk out a test plan with me while I'm working the feature. With enough communication, hopefully this kind of detective work to figure out what to test can be avoided.

<comment><USER12> Agreed. Dev's see the specs, and QA sees the holes. I get to participate from the initial planning into release. As a QA person I feel very lucky.

In regards to the article, i would wager the better scores will be found by QA than developers :-P

</comment></comment><comment><USER13> Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

<comment><USER14> One of the things I've found as a side effect of people using code coverage tools is that instead of testing the behaviour of a method they end up testing the implementation. I think this is because they initially test the behaviour, but then see that one path is missing, so add a test to ensure that path is ran - instead of just testing the behaviour that calls that path and checking the code coverage tool. This ends up causing trouble if you ever want to change the implementation as you end up having to throw away half the tests, which means a lot of effort you spent to get 100% code coverage is now gone.

<comment><USER15> I've faced this problem in my own tests. I want to achieve total coverage so that I know that I've got all the cases covered, but then I end up testing the implementation rather than the contract. I'm not sure what to do about it.

<comment><USER16> You cant have it both ways in my opinion. High test coverage == testing every possible paths == looking at implementation details. If you are testing an algorithm(and games are full of these)you want it to be a 100% accurate therefore you dont have much choice.

<comment><USER17> Tests should be based on the specification. If I want to change some internal implementation detail I should only have to verify that the current tests pass.

If a e.g game contains a sorting in some place in the renderer, I can replace the quicksort with a mergesort as long as the renderer interface is still testing ok. The new sort algorithm may have new special case paths (even number of items vs odd for example) but it's not a concern of the renderer public interface. I may however have introduced a bug with an odd number of items here and the old code was 100% covered and now it isn't. So there is a potential problem and the 99% has actually helped spot it.

If the sorting is a private implementation detail of the renderer then there is no other place to test it than to add a new test to the renderer component only because the sorting algo requires it for a code path. This is BAD.

The proper action here is NOT to add tests to the renderer component to test the sorting code path, but instead to make the sorting visible and testable in isolation via its own public interface.

So one of the positive things about requiring coverage is that if you do it right, it will lead to smaller and more decoupled modules of code.

The bad thing is that if you do it wrong you will have your God classes and a bunch of tests coupled tightly to them.

</comment></comment></comment></comment></comment><comment><USER6> Nice to see a context and link to code specifications in a high-risk and highly-regulated industry. 'Work slowly and don't break anything' would make a good poster.

([http://en.wikipedia.org/wiki/DO-178B](http://en.wikipedia.org/wiki/DO-178B))

</comment><comment><USER18> Meanwhile BBC Basic one-line version has 100% code coverage the moment you start it :)

<comment><USER19> well, if your code coverage scope is one line that's true, but it would make more sense to scope on statements.

</comment></comment><comment><USER20> TL;DR: Use a coverage tool for tests.</comment>


Facebook accepts slightly mis-typed passwords - <AUTHOR> https://pbs.twimg.com/media/C2dGvgvWEAAienm.jpg <comment><USER2> Note the title is wrong but the linked response from Facebook describes what they _actually_ do quite succinctly. So if you're coming straight to the comments, do take the 15 seconds to click through and read it!

You don't have to wonder if it "seems like going to far" because you can calculate almost exactly the reduction in the search space / entropy.

But don't just go based on the theoretical key space reduction. You have to look at the kinds of passwords that people actually choose and see if an online attack which gets on the order of 10 tries against a password before an account is locked would actually be marginally more effective because each submission is technically trying 4 passwords instead of 1?

The reality is the 4 additional tries that Facebook is giving you for free are not passwords that would be in the attacker dictionary in an attack which is allowed only 10-100 tries.

Whereas the 4 tries are probably at around 1% of the time, in other words billions of logins per year, result in a valid login for a user who knows the right password but simply failed to type it correctly.

Another key point to consider is if you give users these 4 extra tries for free on each submit, you cut down drastically (power law on the types of errors that users make) on the number of failed login attempts overall. When this happens you can now _reduce_ the account locking threshold without catching a larger percentage of your real users in the "your account has been locked" trap. This results in a significant net improvement for security against online attack.

<comment><USER3> I'm pretty sure the title is accurate.

Facebook isn't giving an additional "try" by storing the password with caps lock. It instead reduces the search space by 26 characters.

Take for an example a password someone might have -

'AaaAaAaaAAaaaaaAaaAaAaaaAAaAaAaAAaaAAaAaAaAaaAaa'

If an attacker knew it consisted of only [A, a] at random it would take 2^50 guesses. But now it only takes a single guess, just all A's. It's a toy example, but Facebook definitely isn't just giving four extra guesses for free.

<comment><USER2> I'm sorry, this is incorrect. Facebook does not store additional hashes. If the provided string does not match, they also try the following 3 variations;

      1) String with case inverted -- effectively undo capslock
      2) String with first character lowercased, if it was uppercase
      3) String with last character removed

So your example, a single guess of all 'A's would not match. You can try it yourself if you like!

EDIT: So it's actually 3 extra tries, not 4 extra tries.

<comment><USER3> > String with case inverted -- effectively undo capslock

Well don't I feel stupid. Didn't realize undo capslock meant inverting the casing. Doh.

</comment></comment></comment></comment><comment><USER4> Here's a recent research paper discussing the security implications of typo tolerant authentication schemes: [http://www.arijuels.com/wp- content/uploads/2016/07/CAAJ16.pd...](http://www.arijuels.com/wp- content/uploads/2016/07/CAAJ16.pdf)

</comment><comment><USER5> It's not case-insensitive. Try it. I knew they had the case-flipped version but I didn't know about the others. Why would a user's password have an extra character on the end anyway?

<comment><USER6> >Why would a user's password have an extra character on the end anyway?

That seems the strangest of them all: "If the login fails, try stripping a character off the end and see if that matches". I can understand the "First character inadvertently capitalized one" somewhat, but the CAPS LOCK thing seems unforgivable. I just did that yesterday with Apple and it simple suggested to me caps lock might be on. If your test for that is (password didn't match AND all alpha characters in the submission are capital letters) you don't even need to store a second hash for it.

I know there will always be trade-offs between security and UX but, if this image is accurate, that seems like going too far in favor of usability. No matter how small, multiplying the numerator in the probability of matching a password is a dangerous decision.

<comment><USER5> They don't store separate hashes. If the password is wrong, they just flip the case of the input and try again. They do a similar thing with changing passwords, see this comment [https://security.stackexchange.com/questions/53481/does- face...](https://security.stackexchange.com/questions/53481/does-facebook- store-plain-text-passwords#comment84577_53483)

<comment><USER7> How can they get away with not storing at least a second hash? I don't see how an all-caps password can be matched to an any cased password, unless you only store the hash to the all-caps password.

<comment><USER8> It's not all-caps, it's typed with caps-lock down (i.e., case _inverted_ ).

<comment><USER9> Note that this behavior is OS specific. Windows will invert case (shift gives lowercase), while macOS will be all-caps (shift doesn't do anything).

Frankly I can't remember the Windows approach ever being useful to me, and more often than not it bites me in the ass because I habitually hit something like shift-i to type "I" and end up with a lowercase i instead.

<comment><USER10> Drafters (the guys who draw engineering drawings) love it because you very rarely but _just enough_ have to type lowercase letters (such as units, like 3mm) whilst all other text must be in capitals. Thus, you just use shift to shift it to lowercase for those rare moments and no other times.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> One wonders how they discovered these common mistakes. The only way to do a systematic study would be to keep some un-hashed passwords around and compare.

<comment> semi-extrinsic No, you could think up 20 different plausible "common mistakes", add fixes for those and roll out to some subset of users, and then just count the improvement rate of each fix to get the best four.

</comment></comment><comment><USER12> Do they hash and store these possible 'other' passwords?

<comment><USER13> All you have to do is to store a canonical version of all of these.

</comment></comment><comment><USER14> CapitalOne.com also authenticates without case sensitivity.

<comment><USER15> Capital One, at least at some point in the past, was storing in plaintext - they had a "what's the fourth character in your password" check.

At least in this thread, people seem to be giving FB the benefit of the doubt on that particular anti-pattern.

Can anyone rule out plaintext storage?

<comment><USER16> They've been pretty open about how they store passwords and it's definitely not plaintext, here's a talk about it: [https://video.adm.ntnu.no/pres/54b660049af94](https://video.adm.ntnu.no/pres/54b660049af94)

My outside impression is that security is taken seriously enough at Facebook to hire highly competent people to run it. They pay out bug bounties, and run internal red team exercises to test their own ability to detect and react to a compromise. [https://threatpost.com/how-facebook-prepared-be- hacked-03081...](https://threatpost.com/how-facebook-prepared-be- hacked-030813/77602/)

I think we can extend them enough benefit of doubt to rule out storing user passwords in plaintext.

</comment></comment></comment><comment><USER17> password is not case insensitive, it looks to me that they store 4 password version (one for each variant) so they check both 4 in parallel to see if there is a match, it is not bad, it will be worst if they have the password in clear and they see if the inputted one match the stored one in one of the 4 variants (calculated on the fly) that will be bad because means that the password are in clear text somewhere :(

<comment><USER18> Why store 4 versions? They just store the hash of the correct one and try hashing up to 4 versions of whatever you enter. As soon as one of those corresponds to the hash, you're in.

<comment><USER19> Facebook has been doing this for years and I assumed that this is exactly what they do. It saves on db space. Three extra hashes for each user adds up with that many users. And if they decide they need a new variant, they just need to release a new auth module, not recalculate hashes next time the user logs in. Or if they deem that one of these variants is no long worthwhile (for security or ux reason), again, they are not modifying the db.</comment></comment></comment>


Ask HN: Does anyone know why Hacker News was down today? - <AUTHOR>

<comment><USER2> Not sure if it's the case this time, but from previous comments I've read on here HN is run on minimal hardware and when something does go wrong they reboot it and go from there. I've seen it happen a few times now, depends what time of day you check in.

</comment><comment><USER3> Nope, it did happen though: <http://wheresitup.com/results/50b4da27d331167b28000030></comment>


Ask HN: Which professions are impossible to get because the oldies never retire? - <AUTHOR> <comment><USER2> Tenure in higher education is probably one of the hardest ones... some legal and medical programs are pretty locked down as well.

<comment><USER3> Came here to say this. I am fond of saying that when I was in grad school, I was preparing to enter a profession where I'd literally have to wait for someone to die to get a job.

</comment></comment><comment><USER3> Scientific glassblower: [http://www.latimes.com/local/education/la-me-caltech- glassbl...](http://www.latimes.com/local/education/la-me-caltech- glassblower-20160613-snap-story.html)

</comment><comment><USER4> Oddly enough, I think there are a lot of positions in the maritime industry like that. Harbor pilot comes to mind.

</comment><comment><USER5> The ATF (Alcohol Tobacco Firearms enforcement agency in the US) was on a hiring freeze for a long time because no one retired and the size of the agency was fairly fixed. They're expecting a huge wave of retirements in the near future and are now hiring like crazy (relatively speaking, anyway).

<comment><USER2> In the mid-80's my dad had an offer from the ATF that he would have accepted, had the USPS not lost/damage/delayed the delivery of said offer... It arrived in a plastic bag with a postmark from 3+ months prior.

</comment></comment><comment><USER6> Supreme Court Justice.

<comment><USER7> At the risk of looking ignorant (because I am) have there been any exploratory works on imposing term limits on SCOTUS justices?

EDIT: Is 'exploratory' the correct term, at least in the PoliSci crowd?

<comment><USER8> There's the time Roosevelt was on a mission to get the New Deal passed, and threatened to pack the courts if they kept ruling his legislation unconstitutional. The Chief Justice changed his position very quickly when that threat came up.

Packing the courts involved creating 6 additional spots on the court(chosen by the President), if existing justices didn't retire by 70.

[https://en.wikipedia.org/wiki/Judicial_Procedures_Reform_Bil...](https://en.wikipedia.org/wiki/Judicial_Procedures_Reform_Bill_of_1937)

</comment></comment></comment><comment><USER9> Monarch of Great Britain

</comment><comment><USER10> Congress

<comment><USER2> It's funny, but I should have mentioned that one as well... Higher level politics is pretty locked in too, though that may change drastically in the next decade in the US.

</comment></comment><comment><USER11> Librarians. For years our professional associations cried there would be a shortage and convinced people to go to library school (real librarians have a Masters degree). Now people aren't retired, and when they do retired their position is either not filled or filled with a part timer.

</comment><comment><USER12> Some branches of engineering. Civil and corrosion are mostly locked up by established firms, led by founders.

</comment><comment><USER13> Oligarch</comment>


Team Improvement Techniques - <AUTHOR> http://www.dcaulfield.com/team-improvement-techniques/ <comment><USER2> > how do we gather improvements that are worthwhile and specific enough to > solve our problems...?

My teams have tried many ways, and we like using a lightweight template thanks to medical professionals: [https://github.com/joelparkerhenderson/issues/blob/master/TE...](https://github.com/joelparkerhenderson/issues/blob/master/TEMPLATE.md)

<comment><AUTHOR> Excellent! This is very similar to the 'Preventative Actions' practice that we have in our team. Your template provides an added level of detail though, so I might steal it and give it a go.

</comment></comment><comment><USER3> I've seen different teams and sometimes for some reason it's quite challenging to get folks participate in retrospectives in a meaningful way. E.g. it's hard to come up with a few things that went bad for engineers. And it's not a matter of trust, it's something else. Does it come with practice? What are the techniques to get people involved?

<comment><USER4> In my experience, everyone on every team has ideas that could help improve the team. Everyone's experiences shape their perceptions of things. Getting your team to talk about those ideas is the hard part. Small teams can grow in to high trust teams, but big teams pretty much can't. It's _much_ easier to get meaningful feedback/retrospectives from high trust teams.

I think part of the lead engineer's job is to foster trust. But garnering this feedback by asking the right questions is just as important. Even if it means asking difficult questions until answered.

If a team struggles with meaningful retrospectives, it might be because they don't see the value in them. When that's the case people can end up thinking "it's just another meeting", "just let me code". Or it might be because the trust isn't there, even though it might seem like it is. Exposing problems (of whatever nature), tends to end up involving management at some point, and I've noticed over the years that many engineers have a strange fear of management. You can't have a meaningful discussion about problems if you don't trust everyone involved, including management.</comment></comment>


Ask HN: Which functional programming language should I learn? - <AUTHOR> I have experience with most of the top languages in the TIOBE Index (OO and imperative, including the functional aspects a lot of these languages provide). I also have experience with native, JVM, CLR and several JS engines, so I'm not partial to a specific runtime.<p>I don't have a particular side project in mind, but I would like to strengthen my understanding of functional programming concepts through practice. Which language would be a good starting point for this? Thanks in advance :) <comment><USER2> I went through the same question a few months ago. I settled on OCaml.

Pros: it's reasonably straightforward to pick up, and it shines as a functional programming language. There are some good learning resources out there, e.g. _Real World OCaml_. I like the syntax much more so than, say, Haskell, but YMMV.

Cons: a very small std library, meaning you need to go with one of several 'battery' libraries. Feels less polished than mainstream languages in terms of user-friendliness (to be fair my reference point on both subjects is Python). A whiny post on these shortcomings: [https://email@example.com/advent-of- code-2019-its-...](https://firstname.lastname@example.com/advent-of- code-2019-its-always-day-1-f57312336695) (hopefully this gets me motivated enough to finish the follow-up post)

</comment><comment><USER3> I personally adore Elixir and am always quick to recommend it, if not that I would say check out a LISP like Clojure or Racket, they offer a very different experience than Elixir, but I attribute how quickly I picked up on Elixir and FP in general to my previous (yet very very short) experience with Clojure.

</comment><comment><USER4> "Haskell

This is the whole new universe. It is very special, mind blowing, and completely different experience. I think learning Haskell is a very good idea. The core language is very elegant, and it brings on the table a lot of interesting concepts. If you know Haskell, it is not a problem to pick up any language. Haskell is the king. It is in the front line of computer science." [0]

I recommend going through [1] and once you get to monads switch pace and look for more resources.

[0] [https://www.quora.com/When-do-I-use-Haskell-OCaml-Scala- and-...](https://www.quora.com/When-do-I-use-Haskell-OCaml-Scala-and-Clojure)

[1] [https://www.goodreads.com/book/show/912217.Programming_in_Ha...](https://www.goodreads.com/book/show/912217.Programming_in_Haskell)

</comment><comment><USER5> What have you already considered? For each one you have considered, what makes you hesitate? That will help us understand what sort of thing you're looking for so we can make informed recommendations, rather than everyone just naming their favourite.</comment>


Dead simple web crawling (at scale) with Zillabyte - <AUTHOR> http://blog.zillabyte.com/2014/08/04/crawl-entire-domain-one-line/?

<comment><USER2> So just two hours ago the other post by OP ([https://news.ycombinator.com/item?id=8143000](https://news.ycombinator.com/item?id=8143000)) was flagged to death. Presumably because quickly after posting, two eerily generic and positive comments showed up (one from a new account).

That being said, as far as the service goes, some comments:

1\. The popup "join our newsletter" after a minute was a little annoying.

2\. Without guarantees on capacity and utilization, pricing by "machine hour" is useless to me. I'd prefer something more concrete.

</comment><comment><USER3> Why hasn't someone built this before?

Maybe I'm missing something. I figured you could do this already using a few mashable API's.

<comment><USER4> There have been efforts. Looking at the product, I see that the component builds on an open source project.

Doesn't passing data through multiple APIs raise concerns of data polution/cost overhead? At least this uses a flat,pay as you go cost.

</comment></comment><comment><USER5> useful tool</comment>


CU-SeeMe - <AUTHOR> https://en.wikipedia.org/wiki/CU-SeeMe <comment><USER2> It's not HD by any means, but look at the hardware this ran on: A few MB of RAM and a few dozen MHz of CPU.

Meanwhile, a certain "modern" text-only (no, emoticons don't count as video) chat client manages to take upwards of 1GB of RAM and lag severely on a barely-1-year-old quad-core i7...</comment>


Martin Gardner dies, aged 95. - <AUTHOR> http://www.jameslclark.com/martin-gardner-my-friend/

<comment><AUTHOR> I grew up with Martin Gardner's books (of which there are many!) of puzzles, math and magic. From them I learned so many simple and elegant tricks and techniques that I use to this day.

It's largely due to Gardner's writings that I do what I do today, and I love what I do. I had an afternoon with him in March and it was one of the truly special events in my life. I haven't the words to express how I felt then.

Or how I feel now. My world has grown a little darker knowing he's no longer in it.

<comment><USER2> Martin's books changed my life. In one particularly introverted and dark period of my adolescence I poured through them at the library. Thinking through hexaflexagons, the game of life, multiple dimensions, and many more ideas kept me from brooding over my troubles.

Good-bye, Martin. Thank you.

</comment></comment><comment><USER3> for those like me who asked who?

Martin Gardner (October 21, 1914 – May 22, 2010)[1] was an American mathematics and science writer specializing in recreational mathematics, but with interests encompassing micromagic, stage magic, pseudoscience, literature (especially the writings of Lewis Carroll), philosophy, scientific skepticism, and religion. He wrote the Mathematical Games column in Scientific American from 1956 to 1981, and published over 70 books.[2].

From wikipedia

<comment><USER4> A lot of people first learned about Lisp from him. In fact, I bet more people learned about Lisp from him than anyone else.

<comment><USER5> _A lot of people first learned about Lisp from [Martin Gardner]. In fact, I bet more people learned about Lisp from him than anyone else._

Your comment prompted me to do a Google search on Gardner and Lisp. Are you sure you don't mean Douglas Hofstadter, Gardner's successor as Scientific American columnist?

[http://www.nytimes.com/books/97/07/20/reviews/hofstadter- the...](http://www.nytimes.com/books/97/07/20/reviews/hofstadter-themas.html)

No doubt about it, Gardner is credited by a lot of mathematicians in roughly my generation as their inspiration for studying mathematics.

<comment><USER4> Hmm, I know I'm thinking about the famous _Scientific American_ column. I thought it was before Hoftstadter took over, but it's hard to be certain.

<comment><USER6> The column that introduced Lisp is from Hofstadter's era. It's collected in his _Metamagical Themas_.

</comment></comment></comment></comment></comment><comment><AUTHOR> In case anybody knows, cares or was wondering, the magic words are Squeamish Ossifrage.

[http://en.wikipedia.org/wiki/The_Magic_Words_are_Squeamish_O...](http://en.wikipedia.org/wiki/The_Magic_Words_are_Squeamish_Ossifrage)

</comment><comment><USER7> Earlier discussion here: <http://news.ycombinator.com/item?id=1371335>

<comment><AUTHOR> Thank you for that. It's a different article with different information. The comments are complimentary.

This is a case where I wish HN has some sort of correlation or coalescence system.

<comment><USER8> I could live with 3 or 4 Martin Gardner threads that link to each other, he certainly deserved it.

</comment></comment></comment><comment><USER9> As others have already said, just thank you Mr Gardner. I will use his books to (try to ) introduce my son in the magic kingdom of maths.</comment>


A Python wrapper (written in C) for using the high-precision QD library - <AUTHOR> https://github.com/baruchel/qd

<comment><AUTHOR> A wrapper for the old and famous high-precision library written by D.H. Bailey. See: [http://crd-legacy.lbl.gov/~dhbailey/mpdist/](http://crd- legacy.lbl.gov/~dhbailey/mpdist/)</comment>


Adobe Creative Cloud Offers CS6, Lightroom 4, 20GB Storage for $49.99 Per Month - <AUTHOR> http://www.one37.net/blog/2012/2/7/adobe-creative-cloud-offers-cs6-lightroom-4-20gb-storage-for.html <comment><USER2> Great news, they are finally starting to listen!

Make it affordable and people wont have an incentive to steal.</comment>