Should the Apache Foundation stop hosting NSA-sponsored code? - <AUTHOR> http://tomslee.net/2013/06/should-the-apache-foundation-stop-hosting-accumulo.html <comment><USER2> The picture this article is painting is almost completely wrong on every point that matters.

First of all, the NSA has not sponsored development of Accumulo for more than a year. The gist of the story is that companies that charge to support other implementations of Big Table-like systems argued that the government support of Accumulo directly interfered with their ability to compete. Their main point was that the government's job was not to compete in an already-thriving industry, and their lobbying was successful: in summer 2012, the NSA was formally ordered to halt production on Accumulo. Some of the employees left to start their own company, last I heard.

Second, as other users have noted, this is not the only implementation, and dropping it will certainly not make it impossible to do what they do. Remember that Accumulo was started when no OSS technology could do what it did. Now is a different time, and arguably it is actually behind.

I agree that it feels good to stick it to "the man", but let's be realistic here. This move is accomplishing nothing.

</comment><comment><USER3> Accumulo is designed deeply with authorization and access control in mind. Every time Ive spoken with [edit: people at the] NSA, I wind up hearing about how much time they spend just jumping through hoops to avoid looking at information they aren't supposed to.

Accumulo is also a suitable data store for education and medical data, where there's also lots of regulation over who can see what.

The linked article is reactionary and has no substance.

<comment><USER4> "I wind up hearing about how much time they spend just jumping through hoops to avoid looking at information they aren't supposed to."

If they didn't collect so much, it'd be easier to avoid the info they shouldn't be looking at.

<comment><USER5> Even if they "didn't collect that much" they still need to protect sensitive data.

<comment><USER3> Exactly. Medical records are a great example. If you get a new doctor, you have to give them written authorization so that they can get your old medical records.

They can't just say "look I'm a MD, give me the datas", partly because that would be a felony!

Same sort of regulation, albeit more complicated, applies to NSA.

</comment></comment></comment></comment><comment><USER6> This is a political approach--"I don't like one aspect of what an organization does, so I will harass anyone who associates with them in any way."

It's pointless in this case because the NSA is not subject to such grassroots political pressure. Their sigintel teams do not give a shit if you use Accumulo or if Apache hosts it. Harassing Apache will have zero effect on Prism and related programs, but will harm Apache, which is a known-good organization.

If you want to change things at the NSA, the political pressure points are in Congress and the White House. Not Apache.

<comment><USER7> It's exactly Apache's known-good status that makes it a point of leverage. Surely the pragmatic thing to do is to put pressure on organizations that you think may respond, rather than organizations that you know will not. I really don't understand why everyone is saying that Apache's hosting of this project is irrelevant and useless to the NSA. Would you say the same about other projects that Apache hosts?

<comment><USER6> The NSA is one of the most well-funded tech organizations in the world. Apache does nothing for them and has no leverage on them whatsoever.

</comment></comment></comment><comment><USER8> As some of the comments on the linked article point out, Apache Accumulo is one of several implementations of BigTable; another one, also hosted by Apache, is HBase. As far as the fundamental technology goes, this particular cat (key-value stores scalable to very large data sets) is out of the bag, and used by many people besides the NSA.

Would Apache dropping support for one particular BigTable implementation meaningfully hinder the NSA?

</comment><comment><USER9> "Open to all contributions, as long as we agree with you" is a pretty slippery ethos.

</comment><comment><USER10> So we want them to be more open but we don't want to host their open source code for all to see? Believe me, I'm sharpening my pitchfork and lining up my torches over this thing, but that just doesn't make any sense.

</comment><comment><USER7> mpyne. I expressed it badly - I did not mean to say that open source developers are supplying the NSA with Accumulo, just that Apache is supplying the NSA with a useful service by hosting Accumulo as a project (and that for some reason open source developers are OK with this in a way that they would not be for private corporations selling closed source software to the NSA).

The general picture seems to be what antics says in the first comment: NSA adapted Big Data technologies to suit their own access control and other requirements, and then decided that the best way forward was collaborative. A group of employees set up sqrrl to provide a commercial distribution. But it's an ecosystem that the NSA gains from, all the same, and the broad membership of Apache seems like it makes for a place where people may have leverage.

</comment><comment><USER7> As the poster of the linked article, I'd like to respond.

antics' "First of all" paragraph is completely consistent with what I wrote. The "Second" paragraph is an "If I don't do it someone else will" argument: it's not acceptable for arms companies and I don't see why it's more acceptable for open source foundations.

cartershonwald's first paragraph is also consistent with what I wrote. I am sure that the NSA takes more concern over privacy than, say, Google or Yahoo. But that's not really relevant here is it? And the fact that the technology is also useful for other things - well, we can say the same for internet filtering technologies and deep packet inspection, but we still criticize Blue Coat and others when the sell that to authoritarian regimes. I believe what I'm saying is consistent with criticising those companies.

_delirium's comment is also "If not Apache, someone else will do it" as above. Not good enough for arms manufacturers, and not good enough for the Apache Foundation.

I don't understand showerst's comment.

Pherdnut - I don't think the main argument with NSA is that they are being secret, it's that they are engaging in massive and illegitimate surveillance.

<comment><USER11> For showerst's comment, he's basically saying that the idea of e.g. having a hardware store refuse to stock a screwdriver just because it was invented by a nuclear weapons development lab is a pretty slippery slope ethically.

After that we won't stock screwdrivers invented by the government (incl. EPA).

Then we won't stock screwdrivers invented by $UNDESIRABLE_GROUP.

That line of reasoning is why both the FSF and DFSG explicitly say that Free software cannot have terms-of-use restrictions. If it's free for one to use it should be free for all to use, even those you hate.

Accumulo is a tool. It's a tool that happened to be useful for NSA, but NSA is _not_ the only group that has "Big Data" problems and so it makes no sense for Apache to punish the rest of us because Accumulo has a source that you personally don't like.

<comment><USER7> mpyne - thanks for clarifying: I see it now. I've never been a fan of "slippery slope" arguments because once you start on them, you never know where they finish.

I know why FSF takes the line it does, I'm just saying it is a line that has taken us to a strange place where open source advocates are simultaneously supplying the NSA with services to carry out surveillance, defending the practice, but complaining about the surveillance.

And as for "Accumulo is a tool". That's kind of a "guns don't kill people" argument. Would you apply the same argument to private companies supplying authoritarian regimes with deep packet inspection software? I see little difference in the cases.

<comment><USER11> I'm not a super big fan of slippery slope arguments either. Then you get people opposing programs not on their own merits, but what might happen if that program were in the hands of the "other party", and then later in the hands of some dystopian totalitarian state.

Either way, I'm not sure why you say open source developers are _supplying the NSA_ with Accumulo. You have it completely the wrong way around; NSA supplied _us_ with Accumulo.</comment></comment></comment></comment>


Ask HN: How do you rake production? - <AUTHOR> Ruby on Rails has rake tasks baked in. It let you run a piece of code meant to run only once. How do you do it in other environments/languages?
How do you fix some database items that got corrupted due to a bug? How do you ensure code review and time logs with the person triggering the execution? Considering that manual database updates in production are not a good thing.
I faced this problem with a couple of companies. There was never a good solution.
Does anything exist that I am not aware of? <comment><USER2> We use a change management process. Pretty much a jira ticket that outlines the change that needs to happen. Affected systems need to have someone review the plan. It has a rollout and rollback process and instructions should be written in a way that anyone on the team could run it.

Ideally, all steps are scripts that have been code reviewed and tested in a staging or dev environment.

When it comes to DB migrations, the code is updated to work with the old and the new. The db migration happens and is ran by a script that is start/stoppable and has logs. The code is then updated to not need the old db stuffs.

<comment><AUTHOR> Nice! This is the best process I can think of. I've worked with this model.

But then everybody in the team has access to run the scripts, right?

In my case a few people had, and with scale they became bottlenecks

</comment></comment><comment><USER3> Rake is just a task runner, it sounds like maybe you had some rake tasks that were constantly ran to fix bad data or take care of recurring problems. This is the wrong approach. I've seen this done by DBAs who had stored procs to fix issues with data, without ever bothering to check with the dev team and have them fix the integration that was contributing to the faulty data entering the db.

Bad data integrations and poor UI validation bleed garbage data into the data store. That's just the reality of life. If you're constantly fixing records and cleaning data, you need to fix it at the source. Assuming there's a pattern, you can fix the data integration, automated process, or poor UI that's producing those records.

Generally speaking, you want good constraints at the data model layer, in order to prevent bad data from being persisted to the data store by the application layer. It takes a good understanding of database design and data modeling to engineer a good model with constraints. A well structured and properly normalized schema will outlast multiple application layer backends and front-ends, for decades perhaps. Making sure you add proper unique constraints, do not accept NULL values, add required validation to fields, check constraints for various logic, and proper relationship management and normalization of the model.

Once you have the proper constraints on the database table, the app layer will no longer allow poor data to be written that violates those constraints, and the errors will propagate back out to the end user, without ever being persisted in the data store.

Of course constraints can't catch all bad data, the rest really relies on having a good support system and reproducing problems and fixing the processes and integrations that produce poor/unwanted data records.

<comment><AUTHOR> Completely agree. These problems should not happen twice, and if they are frequent there is a problem somewhere else.

But as you mention, even with the best validations, it can happen sometimes. What processes/tools have you seen work to fix them without Manual work or untraceable scripts?</comment></comment>


A New Idea About How to Stop School Shootings - <AUTHOR> https://slate.com/news-and-politics/2018/02/a-nationwide-teacher-walkout-could-shake-us-out-of-our-mass-shooting-stupor.html <comment><USER2> Consider using big data to figure out who is going to go off the rails next. Is this a taboo use of big data? Guns are just tools and can be replaced with equally dangerous tools.

It's the mentally unstable that must be detected before they cause mass harm using unique means. Imagine a kid that gets bullied and has the skills to mass hack self driving cars. Or perhaps they have access to hazardous chemicals and modulating pumps that can bypass check valves on water mains at a school. There are infinite bad things a person can do. Can we not engineer solutions to these problems?

<comment><USER3> Sounds like Minority Report at least the TV show that quit using psychics and use an AI ML instead.

</comment></comment><comment><USER4> Are they talking about taking a day off in protest, or are they thinking of going on strike until gun laws change, that would be something truly amazing!</comment>


Returning to the Original Social Network - <AUTHOR> https://begriffs.com/posts/2016-07-08-returning-original-social-network.html?hn=1 <comment><USER2> Very cool article!

</comment><comment><USER3> I thought it was going to be FidoNet or Compuserve.

My first email was via Bluewave, propagated BBS to BBS

</comment><comment><USER4> OK, this goes from DNS, to email, to... something which is not Usenet, which surprised me, because Usenet fulfills most of the goals of being a social network much better than either DNS or email can.

Usenet can, in fact, be built on top of an email delivery mechanism, such as UUCP, which is the store-and-forward dial-up network both Usenet and Unix email ran on top of before Unix was allowed to sit at the big-kids table of networking (Arpanet). We have NNTP now, of course, and a number of NNTP servers you can run on a laptop because it somehow fails to be 1995 anymore and most of the people reading this have better network connections and more disk space than most of the servers which ran NNTP in the heyday of Usenet.</comment>


Stream_libarchive: workaround various types of locale braindeath - <AUTHOR> https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f027338b0fab0f5078971fbe <comment><USER2> I can't believe it took this long for somebody to say it, to be honest. It's not exactly the best-kept secret that locale is one of the most horrifically broken things in modern operating systems.

Windows has its own problems. As pretty much any modern Windows coder will notice, all the APIs that take locale-dependent characters have an 'ANSI' and 'UNICODE/Wide' version. As far as I know this distinction was added with Windows NT and patched over with horrible macros (#define UNICODE anyone?) When Windows NT was developed originally, 'UNICODE' was virtually synonymous with UCS-2.. or at least Microsoft and Sun Microsystems seemed to think so.

ANSI variants actually accept plenty of multi-byte encodings, like Shift-JIS. And hey, there's even a codepage entry for UTF-8!... and it doesn't work at all. Why has always been a mystery, because for a lot of English programs, a UTF-8 codepage would've magically made everything work with UNICODE seamlessly. Initially I wondered if I was just missing something, but I made a small hook that did the translations automatically for a few APIs and yep, it worked. So Microsoft created this horrid system of 'non-Unicode' locales, and 'wide' APIs, but figured UTF-8 was a waste of time I guess.

You can, of course, change Windows locales per-thread, but that doesn't stop Microsoft from making it an absolute PITA to switch your so-called non-Unicode locale, which is not only considered global per-app, but global on the system and can only be changed by rebooting. I'll be honest, I'm not really sure why this is necessary to any degree, but it's exactly the same in Windows 10.

Even though it's considered 'legacy', it still affects all kinds of stuff you wouldn't expect. The most noticeable thing is that in the Japanese legacy locale, your backslashes turn into... yen! This is an extremely legacy behavior all the way back to the first Japanese versions of MS-DOS.

Instead of fixing this to any degree, like perhaps adding a compatibility option per-program, Microsoft released a program called AppLocale that awkwardly lets you set the locale per app. It's clunky.

</comment><comment><USER3> I am so conflicted about the morality of this pull request, it's a work of art.

He insults a bunch of unknown people in a terrible way, but kindly points that out and then proceeds to tell you he'll now justify the statement.

And to sum it all up, it's attached to a commit that actually fixes the problem (its not just a rant without a solution).

I've never been so conflicted how I feel about something.

</comment><comment><USER4> I'm a novice Linux user, and locales seem useless to me. Everytime I install Ubuntu locally, I have to reset the system locale to UTF8.en_US, because the default it picks (UTF8.en_HK) crashes a lot of programs due to "unsupported locale" errors.

</comment><comment><USER5> I have always been amused by the "C" locale. It always seemed like some standards writer punting on a hard problem.

As for the article I agree 100%. The way Unix Locales work always seemed like a horrible hack that did nothing but break programs to me. Granted, as an English speaker I was never the target audience, but even for the people they were supposed to help it seemed to make more problems than solutions.

</comment><comment><USER6> So true. Can't be put any more eloquently.</comment>


Camcorder Brings Zen to the Shoot - <AUTHOR> http://www.nytimes.com/2008/03/20/technology/personaltech/20pogue.html?pagewanted=all <comment><AUTHOR> Some product-design insights buried in there.</comment>


Sorry, Blue Apron. The Joys of Cooking Can't Fit in a Box. (2015) - <AUTHOR> https://newrepublic.com/article/122959/sorry-blue-apron-joys-cooking-cant-fit-box <comment><USER2> Blue Apron is the cooking version of a LEGO kit or a coloring book, the former of which has been popular for decades and the latter of which is apparently experiencing a moment as people look for ways to combat the lack of direction and clear goals in their careers and lives.

A lot of people aren't creative, or don't have the energy or confidence to be creative, or cooking just isn't "their thing" but they want to try it (or maybe they just need a gentle introduction). Following clear directions towards a well-specified goal when you have all the tools and ingredients you need right in front of you can be fun, relaxing, satisfying and confidence- boosting. See just about every video game ever made.

Of course someone who loves cooking isn't going to see the appeal of Blue Apron. For a lot of other people, cooking looks like this: [http://theoatmeal.com/comics/cook_home](http://theoatmeal.com/comics/cook_home).

</comment><comment><USER3> I can see the appeal of these services though I don't use them (don't even know if we have them in the UK).

I'm actually fortunate since my office is about a third of a mile from an old fashioned style in-door covered market that has a lovely sandwich place, so I go pick up my lunch and all the ingredients I need to cook (there is a butchers, fishmongers, veg stall) that night, it's _hugely_ cheaper than the supermarkets, the food is generally better as well.

I tend to cook simple meals during the week (lots of steamed veg and fish or stews) since they are fast, easily cooked and in the case of stews eaten the next day or frozen.

The above was a huge part of my losing 60lbs last year, no fancy diets or anything I just stopped eating _anything_ I hadn't cooked or prepared.

It's actually quite hard to eat 2500 calories of fish and veg :)

</comment><comment><USER4> We subscribe to both Blue Apron and Home Chef. I've always fancied getting into "cooking", but never knew where to start. With these services I've explored new techniques, equipment, and ingredients that I probably never would have otherwise. The recycling problem is improving, as Blue Apron will let you send back packing and refrigeration materials postage paid.</comment>