Programmer problem solving sequence: should it be like this? - <AUTHOR> http://www.johndcook.com/blog/2010/10/22/programmer-problem-solving-sequence/ <comment><USER2> Why does using google and stackoverflow get a negative connotation?

There's a collection of information on a topic that's larger than any other collection in the history of mankind, and we can search it in minutes. Why shouldn't that be the first option?

And those solutions certainly don't preclude the "think" option, they augment it (Also, I've already RTFM).

<comment><USER3> tl;dr of my post: Thinking through a problem and not going to the web first makes you a better programmer, instead of relying on the better programming of others.

The negative connotation depends on what you're using google and stackoverflow for. If you need to understand something that can't be directly reasoned through to come to an answer (random example: you need a description for an API call in some public web service) then there's nothing wrong with google, stackoverflow, etc being your first choice in search for your solution.

However, if it's a problem that -can- be directly reasoned through (random examples: any application logic issue, determining a regex to match a specific pattern, tree traversal issues, etc, etc) then your first choice should be to try to figure it out yourself before turning to the web. The rationale is twofold. 1) With enough practice, you'll often find you can solve your own issues faster than you can dig through websites trying to find someone else who has done the same. 2) Even if you ultimately can't find a solution on your own, attempting will at least exercise your brain and give you more experience, which feeds back into #1.

Another way of stating the rationale is that it helps to prevent cargo cult programming (see: <http://en.wikipedia.org/wiki/Cargo_cult_programming>). Even if you can't come up with a solution, or even if your solution is less than optimal, attempting to come up with your own answer first gives you a much better chance of understanding the underlying principles involved in the solution that google, stackoverflow, etc give you. That will minimize the chances that you apply the solution in a half-assed way that just causes you more problems down the road because you didn't fully understand its implications. It will help you in the long run.

</comment></comment><comment><USER4> This is nonsense: 90% of the time the problem exists in your code and no amount of Googling or Stackoverflow questions are going to help with that!

<comment><USER5> True, but you often don't realize that until you've exhausted Google and StackOverflow.

</comment></comment><comment><USER6> I certainly see a significant number of people on IRC asking questions that could be typed verbatim into google to get a better answer.

I guess some people just want/need to avoid as much thought as possible.

<comment><USER7> Okay, so, sometimes I do this. Not with IRC specifically, but in person or over IM with other humans. Sometimes I want to talk to a someone, and I feel like an active listener will be much better at explaining the whole picture and answering my follow-up questions. Or I want to start a discussion around the issue.

Hopefully you don't do this, but I think that linking people to <http://justfuckinggoogleit.com/> is anti-social and is bad for any community.

<comment><USER6> I don't think anyone is going to think less of you for going to real people with your problems. It's specifically when people go into a channel full of strangers and ask things like "how can I do a find with a many to many in rails" that I'm talking about.

I guess it's the sense of entitlement that's annoying. Why would you expect strangers to help you when you can't even be bothered to state the question in a way that would make things easy for them? I realise english isn't everyone's first language, but I have seen people who speak very broken english take great pains to explain their problems clearly as best they can, it isn't an excuse.

And no, I don't ever get snarky. People with stupid questions should get ignored at worst. In these cases, the questions are usually so trivial that being convincingly rude would take more effort than just solving the problem.

This is a better link to post in these situations: <http://catb.org/~esr/faqs/smart-questions.html>

In fact, the bot in #rubyonrails cites this link when prompted with !ask

</comment></comment></comment><comment><USER8> Whether or not this is good or bad depends on how experienced you are -- On a platform I have been using for years, in which I have expertise... this would be a tragedy.

But, as an example, I just picked up rails 2 weeks ago, and have written a fairly complex app with it. I google quite a bit, not as a crutch to thinking, but to gain insights from people who know the platform better than I. Each new component I need for my app instigates some research to see how other people have done it. I do not copy and paste the results... I analyze and critique them for my own purposes. The next time I need a similar function, I think for myself. Over time, I become an expert and no longer need Google to get a project done.

</comment><comment><USER9> Most of the time, I'd rather hit StackOverflow than interrupt a coworker.

Apart from that this solving sequence is on the spot, even though the author is apparently critical of it. For the record, I started programming in the early 90s, before I had access to the internet.

Here's a message that I got while doing some Haskell code. If I were to RTFM first, i.e. read the Haskell Report, it would take me perhaps more than 1 hour to understand the problem. If I google it I will be able to understand the general idea and decide between the compiler's suggestions in a couple of minutes. I can always go back to the specification later to acquire a more solid understanding of the language.

        Ambiguous type variable `a' in the constraint:
          `Show a' arising from a use of `show' at ParseQuote.hs:123:11-16
        Possible cause: the monomorphism restriction applied to the following:
          f2 :: a -> String (bound at Foo.hs:123:0)
        Probable fix: give these definition(s) an explicit type signature
                      or use -XNoMonomorphismRestriction

</comment><comment><USER10> This is too vague to be meaningful. You shouldn't hesitate to consult your coworkers for advice on a problem. If the problem is external, they may have wisdom and feedback on, for example, what library works best. If the problem is internal, they may spot your coding error in five minutes that you've been blind to for two days.

Should you make your coworkers and Stackoverflow users solve all of your problems for you? No. Should we repeatedly categorically demonize these excellent resources, thereby instilling an irrational fear of using them in new developers? No!

I've seen more problems caused by prideful programmers who refuse to consult with coworkers and consider their input. Furthermore, the people who have complained the most about being pestered by others were the ones who wrote the convoluted, undocumented mess that everyone else was stuck maintaining.</comment>


China Considers Ban on Cryptocurrency Mining Because It's a Waste of Energy - <AUTHOR> https://gizmodo.com/china-considers-ban-on-cryptocurrency-mining-because-it-1833904745 <comment><USER2> Dupe: [https://news.ycombinator.com/item?id=19611848](https://news.ycombinator.com/item?id=19611848)

</comment><comment><USER3> > One of the founding myths of cryptocurrencies is that fake money like > Bitcoin isn't dependent on society at large to operate.

What could this possibly mean?

<comment><USER4> Money just have value because society (be it government or just a group of people) need to accept it so it has value.

Either government makes it mandatory ("everybody must accept dollar bills as payment"), or people accept them at free will ("I will give you this horse if you give me 3 sheeps").</comment></comment>


The Scarlet B: A Brief History of Buggery in Colonial America - <AUTHOR> http://priceonomics.com/the-scarlet-b-a-brief-history-of-buggery-in/ <comment><USER2> An unconventional article and quite humorous in it's peculiarity. Reading the whole article really builds well for the final sentence and I gave a hearty guffaw.

Never has the saying rung so true: "don't be ugly".</comment>


Blogger: The Rodney Dangerfield of Blogging  - <AUTHOR> http://www.markevanstech.com/2009/09/06/blogger-the-rodney-dangerfield-of-blogging/ <comment><USER2> The reason I use blogger rather than wordpress is because I don't really want to host my blog myself (seeing as to how its updated about once a year, self- hosting makes no sense). However, i still like fiddling around with the look of my blog and in that respect blogger lets me customize to the hilt whereas with wordpress (.com), I only have access to one set of themes and plugins (or atleast that's how it was when I last looked, which was a couple of years back).

Also, I don't really see how Blogger is inferior to Wordpress.

</comment><comment><USER3> >>> Below is a chart comparing unique visitors (U.S.) for Blogger to Wordpress.com. Blogger is leading, although Wordpress' numbers don't include the millions of people who have self-hosted Wordpress blogs. <<<

But that "although" is quite substantial, isn't it? The point of wordpress is that you self-host the installation, so not including those into the statistic is quite misleading.

But I agree, blogspot is quite good, and you can even map your blog to unique domain name.

<comment><USER4> But can't you do the mapping for every service? Even Posterous lets you do that - for free.</comment></comment>


Paul Graham To Hold Y Combinator "Office Hours" At TechCrunch Disrupt - <AUTHOR> http://techcrunch.com/2011/05/17/paul-graham-to-hold-y-combinator-office-hours-at-techcrunch-disrupt-apply-now/ <comment><USER2> I hope everyone realizes that these are going to be held on stage, and that if you do this, a lot of stuff of the type that companies usually like to keep secret is going to get discussed in front of a roomful of people.

I also don't know if it's going to work. I compressed the time slots about 3x to make it interesting for the audience. Since I won't know the startups already and the time slots are so short, the result will probably be more like YC interviews than office hours.

<comment><USER3> Would you consider doing this as a teleconference on a regular basis (ex. once a quarter)? It would be valuable access to a much wider audience.

<comment><USER4> With a family, and N Y Combinator companies to follow, what would be the benefit for PG in doing so?

</comment></comment></comment><comment><USER5> What is the top comment (from Robin Wolaner) talking about?

"Does he allow women founders to come to office hours? Although given his admittance that he prefers not to invest in women of child-bearing age, perhaps none will bother."

[https://www.facebook.com/l.php?u=http%3A%2F%2Ftechcrunch.com...](https://www.facebook.com/l.php?u=http%3A%2F%2Ftechcrunch.com%2F2011%2F05%2F17%2Fpaul- graham-to-hold-y-combinator-office-hours-at-techcrunch-disrupt-apply- now%2F%3Ffb_comment_id%3Dfbc_10150200958421225_16526152_10150201067986225&h=f55f3)

<comment><USER6> An article called "The False Choice Between Babies and Startups" was on the front page a few days ago.

From the article:

"When I got to the bottom of [Paul Graham's] post I found the following footnote that read:

[2] One advantage startups have over established companies is that there are no discrimination laws about starting businesses. For example, I would be reluctant to start a startup with a woman who had small children, or was likely to have them soon. [emphasis added] But you're not allowed to ask prospective employees if they plan to have kids soon…Whereas when you're starting a company, you can discriminate on any basis you want about who you start it with.

While its common knowledge among career savvy women that we often have to prove that we are "in it to win it," so to speak, I was surprised to see this advice given so matter-of-factly without noting any further mitigating factors a "reluctant" founder might consider when contemplating starting a company with a woman who was, or desired to become, a mother."

The article goes on to explain this is a false choice.

Link to the article: [http://blogs.forbes.com/85broads/2011/05/16/the-false- choice...](http://blogs.forbes.com/85broads/2011/05/16/the-false-choice- between-babies-and-startups/)

Link to the HN thread on the article: <http://news.ycombinator.com/item?id=2554807>

</comment></comment><comment><USER7> I don't see how this would be like true OH (not that I've ever been to one), which only truly works if it's about one or a couple specific/isolated topics.

I suspect it would be more like a pitch-your-company-and-Paul-will-give-you- his-opinion-about-it session.

</comment><comment><USER8> It really would be nice to have open office hours where any team can come into chat in private. When I move down to the valley I would like to chat with people I think would be interested and not have to chase people around or feel they are too busy. As an entrepreneur I think we should be able to score a meeting with anyone, too bad its not the way it goes. In reality some of us sacrifice so much and for a meeting to never be setup at times is such a waste for entrepreneurs and then again I understand not everyone would fit the "YC" type. I do love YC and many others, but its time for the entrepreneur to be on stage interviewing the Angels and VCs. Without us they aren't much :)

I guess if were that good they will find us or at least hear about us later.</comment>


Ask HN: Where can I find the number of people tested for Covid-19 per country? - <AUTHOR> I'm looking for the number testing that is happening not the number of infected... <comment><AUTHOR> Found data for the US [https://covidtracking.com/api](https://covidtracking.com/api)</comment>


How I Stole a User's Siacoin - <AUTHOR> https://mtlynch.io/stole-siacoins/ <comment><USER2> I often wish that password entry for things fully under your control (i.e. when there are no retry limits aside from brute computational power) would come with limited brute forcing support.

Such password dialogs could just let you type your best effort, and they could use the things you type to inform the guessing process; you could fat-finger a character or two, and it would just take a moment longer to log in as it uses the accurate parts of the data to make educated guesses of the password. For old encrypted files, for example, I often don't remember which password or which combination of passwords I might have used, but I can provide all the important bits and a smart program could easily guess the right combination.

<comment><USER3> (offtopic - misunderstood comment. Let's assume the parent comment said 'we should prevent bruteforcing')

As the other commenter said, there's nothing you can do to prevent brute forcing. What you _can_ do, is have a _very_ expensive KDF. So for every password you enter the wallet will take a very long time to 'unlock', which is basically the process of deriving the key from the input.

'Expensive KDF' sounds cryptic, but often just having some memory/CPU requirements for an instance of the KDF should suffice.

Fun fact: There is a reason why when you enter a wrong password for `su` or `sudo` it seems to take longer to throw the wrong password dialogue than to log you in. That's because the password authentication module (called PAM) artificially delays you to prevent brute forcing. You can go and change it if you want. (I would discourage it)

Of course, this won't stop everyone. One can just put the harddrive in a different computer to get the hash of the password and crack it within a day with proper resources. This is the problem with trying to 'stop' bruteforcing at the input level. You must already asssume the attacker has the hash, then the difficulty must be determined. That's the point of people arguing about password hashes (for fun, of course)

<comment><USER4> The post you responded to was asking for an easier way to support multiple attempts without having to sit there typing all the nearby variants of a mostly-known passphrase. I'm not sure what part of this post addressed the parent post.

<comment><USER3> >I often wish that password entry for things fully under your control (i.e. when there are no retry limits aside from brute computational power) would come with limited brute forcing support.

This part. I misunderstood the comment. I thought by 'limited brute forcing support' they meant to limit the brute forcing process. The sibling comment also thought the same thing so I didn't doubt it.

</comment></comment></comment></comment><comment><USER5> I used to mine Bitcoin back in 2011 and I lost my wallet.dat file (through several stupid moves on my part). It's got approx 103 BTC in it, anyone is welcome to it, I've given up trying.

[https://blockchain.info/address/166BuLPWHUjqoqiYp5rGE3B5r5Am...](https://blockchain.info/address/166BuLPWHUjqoqiYp5rGE3B5r5AmqgHpoL)

<comment><USER6> My brother accidentally deleted his wallet.dat from Dropbox a few years ago - he had given it a random filename and encrypted it with GPG so it was unrecognizable to hackers (and apparently him as well).

It had 1,000 BTC in it! He had received them from a generous Bitcoin contributor in the early days who said "here you go, hold on to it, it will be worth something someday."

I still give him a hard time about his $3 million USD mistake...

<comment><USER7> Did he try Dropbox support to see if they have a backup?

<comment><USER6> Oh yeah, completely exhausted that route - it was over a year before he realized it and way past their retention period...

<comment><USER8> "Hi Dropbox support! You get $1m USD if you figure this out."

</comment></comment></comment></comment></comment><comment><USER9> So what are both "ionic" and "tonic" in the same dictionary for a _human readable_ entropy library?

<comment><USER10> EFF has a new wordlist which can be used for things like this. It focuses on phonetic and spelling differences across each word so that this doesn't happen, plus it prevents words from "duplicating" when you combine them (ie the two words `in put` and `input` being the same).

[https://www.eff.org/deeplinks/2016/07/new-wordlists- random-p...](https://www.eff.org/deeplinks/2016/07/new-wordlists-random- passphrases)

<comment><USER11> This is awesome. I had no idea my EFF money was doing great things like this in addition to fighting for net freedom :)

</comment></comment></comment><comment><USER12>
        A wise wife tagged and jagged and nagged
        Her aptitude had altitude to push the lush
        He bore the brunt with a grunt and tonic
        His music was ionic sonic
        Their topic too toxic to adapt
        And they, too adept to adopt

</comment><comment><USER13> This was an amazing story, but there are LOT more take-aways here!!!

First of all, let's look at something: the burden of memorizing 29 words was SO great, that despite carefully writing it down and double-checking it, the user failed to memorize it or even come close: after trying _500_ times, they could not tell that ionic was a different word from tonic. No doubt they had looked at each handwritten word very carefully during the 500 attempts, but just could not do it. By the way, if you write the word ionic down in your own handwriting, you could easily see that it might look exactly like your own handwritten tonic.

There is something else about these 29 words. You can find the number of bits of entropy in a dictionary you'd pick one word from at random by taking the log2 of the number of entries. (In a pinch you do log 2 by taking the log and dividing by the log of 2). That shows that 1626 words (the number of entries in the dictionary) have 10 bits of entropy.[1]

So by making the user "remember" (write down) 29 such words, you are making them memorize (write down) 290 bits of entropy.

2^290 is 1.9892929e+87. There are about 10^80 atoms in the ENTIRE universe (a hundred billion galaxies with a hundred billion stars each). You'd have to get every atom in our entire universe -- every planet's every atom, every sun's, every black hole's, every one of the atoms anywhere in the world, to try 10,000,000 operations each, before you got an answer.

That is _WAY_ too much.

But despite having such an incredible amount of extra information in there (base-64 encoding 290 bits would take 48 characters - six bits per character), it does not contain enough of a checksum to correct against a single transcription error.

So this is a great example of a solution that is very user-hostile: so long that the user is forced to write it down, but despite its length so fragile that it does not contain any help against any amount of corruption. And very clearly, the longer it is, the greater the possibility of user error: could you hand-write an entire Dickens novel without a single error anywhere for example? What about a 12-character alphanumeric password? So the latter is _stronger_ than the former! The latter is a better password.

I am not sure what kind of passwords would have redundancy built-in (so that a slightly wrong version would be corrected and accepted) but this would be a good time to find out.

One last thing. Does anyone know how long it takes to try a combination? I'm surprised that the blog poster went through the trouble of finding Levenshtein distance, since I would think from a coding standpoint it would be faster to code trying all 1625 other possibilities for the 1st word (leaving the rest unchanged), trying the other 1625 possibilities for the 2nd word, and so forth. Since there are 29 words this is just 47125 possibilities in total which doesn't seem like it's that many. (Then again, some 'treasure hunter' the blog poster was "competing with" might have had that script running already when the blog poster got there first!)

[1] [https://www.google.com/search?q=(log+1626)+%2F+(log+2)](https://www.google.com/search?q=\(log+1626\)+%2F+\(log+2\))

<comment><USER14> The need for embedded some form of error correcting codes into readable keys like these is a really good point.

While not the same, I'm reminded of the issue with etherium addresses where they've (after initially having no extra checking) started using mixed case to provide a checksum to detect incorrect entries. Otherwise, it's really easy to send coins to a very slightly different address due to a typo.

With Saicoin, it seems like just adding 1 more word could allow correction of mistakes like these. (And you'd end up with a round 30 words :).

<comment><USER15> It does have a checksum. That means error-correcting works just like in the article, by picking the nearest valid code. This could be built-in to the software.

<comment><USER16> It would take about 0.5 seconds of brute forcing for the library to figure out if you had gotten a word wrong, so that's actually reasonable.

</comment></comment></comment></comment><comment><USER17> I'd pay to discover the error I made :)

</comment><comment><USER18> So I can overlook the misdemeanor pocketing of a few bucks with the intent on giving it back, but you basically admit and _brag_ about breaking the Computer Fraud and Abuse Act as some kind of exercise of how clever you are for doing a dictionary attack against a weak and exposed key?

Good luck sir.

<comment><USER19> Would CFAA really apply here? He's not accessing any computer illegitimately, the blockchain is public record, the key was posted to a public website. He's accessing the public siacoin network, posting transactions that anyone has permission to.

<comment><USER20> I'm not convinced that "posting to a public network" gets you out of the sketch. Sections 5-7 seem at least partially relevant, but then I'm neither in the US nor am I a lawyer anywhere else.

Posting data to a remote system, with the clear intent of taking a thing of value from another person without permission. Perhaps it falls between the cracks, but I'd be reasonably surprised if it doesn't come under this or another similar act.

[https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act](https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act)

<comment><USER21> He didn't just post the passphrase, he also posted this:

"If someone figures it out, I will send you free sias"

I'd call that a clear invitation/authorization for anyone to try to crack his passphrase.

<comment><USER20> Reasonable, but not an invitation to transfer the entire amount then setup an automated process to transfer any remaining amount to your own address.

<comment> kyle-rb I don't know, I think transferring it was the reasonable thing to do, rather than leaving it in the compromised wallet.

It's like if you find someone's (real) wallet, and you pick it up and contact the owner to ask where to drop it off. Rather than leaving it there and just telling the owner what street corner it's on.

<comment><USER22> I recently heard about a guy who was arrested and charged for exactly that. He took a phone home that he found in a carpark, the owner texted it saying "please return to Subway" He went there the next day to hand it in and the police were waiting! Apparently he was supposed to have turned it in to any nearby business at the time he found it. Not tried to find out who owned it the next day.

<comment><USER23> Any nearby business? I thought lost property was to be turned into the police station. Otherwise the owner could call the cops on the business employees "I was never at Dunkin Donuts yesterday."</comment></comment></comment></comment></comment></comment></comment></comment>


India considering jail term for mining, holding, buying or selling crypto - <AUTHOR> http://moneycontrol.com/news/business/cryptocurrency/10-year-jail-for-buying-selling-cryptocurrency-in-india-report-4073451.html <comment><USER2> Interesting... On the same day Facebook announces plans to launch their own cryptocurrency.

</comment><comment><USER3> Just another reason for the highly educated of India to leave.</comment>


Median of Medians - <AUTHOR> http://functionspace.org/articles/19

<comment><AUTHOR> Related discussion here - [http://functionspace.org/topic/32/On-application- of-Median-o...](http://functionspace.org/topic/32/On-application-of-Median-of- Medians)</comment>


Wikileaks in Iceland court victory over payments blockade - <AUTHOR> https://www.bbc.com/news/business-22294108 <comment><USER2> From 25 April 2013. Not a new court win. See also [https://news.ycombinator.com/item?id=4247220](https://news.ycombinator.com/item?id=4247220) with 50 comments, from about that time.

<comment><AUTHOR> You are right, I posted the wrong link. There's actual news in that case: [https://grapevine.is/news/2019/04/25/valitor-sentenced-to- pa...](https://grapevine.is/news/2019/04/25/valitor-sentenced-to- pay-1-2-billion-isk-over-wikileaks-payment-block/)</comment></comment>


How to Be a Good Office Slave - <AUTHOR> http://www.yegor256.com/2015/10/06/how-to-be-good-office-slave.html?2015-40 <comment><USER2> While this article rises some important points regarding company culture that each of us co-create, I have to say that overall aggressive tone and provocative, unfair in many cases, insinuation at the end, made it quite unpleasant read.

But maybe that's a way to start a discussion?

<comment><AUTHOR> Yes, indeed, that's one of the ways :) Just post your comment under the article and we'll discuss</comment></comment>


Zoltan: Parallel Partitioning, Load Balancing and Data-Management Services - <AUTHOR> http://www.cs.sandia.gov/zoltan/ <comment><USER2> It took a few clicks but the Zoltan user guide explains the project's raison d'être:

 _The Zoltan Library contains a number of tools that simplify the development and improve the performance of parallel, unstructured and adaptive applications [for application developers]._ [1]

[1] [http://www.cs.sandia.gov/zoltan/ug_html/ug_intro.html#Motiva...](http://www.cs.sandia.gov/zoltan/ug_html/ug_intro.html#Motivation)</comment>


NSA director heckled on stage at Black Hat security conference - <AUTHOR> http://www.forbes.com/sites/andygreenberg/2013/07/31/nsa-director-heckled-at-conference-as-he-asks-for-security-communitys-understanding/ <comment><USER2> This entire event was a staged press op. Keith Alexander is a ~30 year veteran of SIGINT, electronic warfare, and intelligence, and a Four-Star US Army General --- which is a bigger deal than you probably think it is. He's a spy chief in the truest sense _and_ a master politician. Anyone who thinks he walked into that conference hall in Caesars without a near perfect forecast of the outcome of the speech is kidding themselves.

Heckling Alexander played right into the strategy. It gave him an opportunity to look reasonable compared to his detractors, and, more generally (and alarmingly), to have the NSA look more reasonable compared to opponents of NSA surveillance. It allowed him to "split the vote" with audience reactions, getting people who probably have serious misgivings about NSA programs to applaud his calm and graceful handling of shouted insults; many of those people probably applauded simply to protest the hecklers, who after all were making it harder for them to follow what Alexander was trying to say.

There was no serious Q&A on offer at the keynote. The questions were pre- screened; all attendees could do was vote on them. There was no possibility that anything would come of this speech other than an effectively unchallenged full-throated defense of the NSA's programs.

Even the premise of the keynote was calculated to wrong-foot NSA opponents. However much you might want to hear Alexander account for the activities of the NSA, the NSA itself is not the real oversight mechanism for the NSA! My guess is that no pol with meaningful oversight over NSA would have consented to address a room full of technology professionals about NSA's programs; they were happy to send NSA's own supremely well-trained figurehead to do that for them.

I think a walkout might have been effective, had it been organized well enough in advance (perhaps with some of the same aplomb as the [I think misguided] opposition to CISPA); at least you'd get some stinging photos.

<comment><USER3> Agreed; his extensive infosec background and smooth-talking politician talents meant that he probably had full knowledge of how this was going to go down. What's interesting is why he attempted PR reconciliation at a relatively niche conference. Black Hat doesn't have much connection to the mass public, and trying to make yourself look good in front of a mass of angry hackers is pointless anyhow because hackers tend to stay angry about topics like privacy.

Now, if he had a full blown press conference with civilian attendees, then, well...

<comment><USER4> I think he went to Blackhat because the feds were 'banned' from Defcon.

<comment><USER2> I think he went to Black Hat because unlike Defcon, Black Hat is run by a large multinational media company with a PR department trained and motivated to secure keynote speakers who will attract the most press attention to the conference.

<comment><USER3> Alexander actually spoke at Defcon last year and garnered quite a bit of attention regardless, so I don't quite think this is the case.

<comment><USER5> Exactly! Actually, Alexander was asked the Question of NSA datacollection after his talk, basically it went down like this:

Q: "[..]Does the NSA really keep a file on anyone? [...]"

A: "[..] Frist, no, we don't [...]"

Absolutely astonishing!

listen/watch here: [http://youtu.be/tz0ejKersnM?t=33m2s](http://youtu.be/tz0ejKersnM?t=33m2s)

<comment><USER6> What is astonishing by that? They can easily be splicing fiber cables taking a copy of most Internet traffic and recording either the actual call or just the from/to/length info and his answer is totally correct. It is actually a really terrible question.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> > Alexander also noted the 6,000 NSA cryptologists who have deployed to > Afghanistan and Iraq, 20 of whom were killed in the line of duty according > to Alexander. "Think about people willing to go forward to Iraq and > Afghanistan, to make sure our soldiers, airmen and marines get the > intelligence they need," he said. "I believe these are the most noble people > we have in this country."

Having lost a brother who deployed as a signals operator in Afghanistan (to an IED), it always makes me cringe whenever they use forward-deployed soldiers as a defense of the higher-level states "nobility". There is nothing noble about mass surveillance or the invasion of privacy of non-enemy combatant nations/citizens.

The fact citizens signed up to risk their lives in combat-zones at the bottom end of the chain does not legitimize the actions of those at the top.

<comment><USER8> Hear, hear! When the top brass evoke these deaths, it's like they're saying "if our sainted dead soldiers followed these orders, how can _you_ possibly disagree?" Frankly, the last general worthy of making that sort of comparison was Eisenhower, and he would have been the _last_ man in the military to do so. He was already sickened at this leadership culture when he was President. If he were alive today he would cross the street to avoid a conversation with Alexander.

</comment></comment><comment><USER9> Hackers _applauded_ Alexander? He really has played them like pipes.

You can argue all you want, but in the end, some boundaries have to be set. What the NSA did (does actually) is way past that boundary for most people, and I think with reason, but that's another debate. He hasn't supplied any argument that would make us reconsider the boundaries. All this terrorist talk is bullshit.

<comment><USER10> Despite the vaguely militant-underground name "Black Hat", it's a conference of mainstream security researchers, run by a large media firm (CMP Media, which also runs the Game Developer's Conference, and owns a bunch of magazines like InformationWeek). Many of the attendees are themselves in government positions, or serving as contractors. So I'm not sure I'd expect a particularly strong backlash. Heck, some attendees could well be in a position to have already known some of the things that were leaked.

<comment><USER11> Well, a militant-defensive-underground would call themselves "Grey Hat" or something. Black Hat implies exploiting flaws for personal gain, doesn't it?

I imagine a bunch of these Black Hat people fancy their big payout using their skills for evil.

</comment></comment></comment><comment><USER12> Funny how Alexander says he's read the Constitution, and implores the heckler to do the same. If one can read the Constitution and square the NSA's mass surveillance with the 4th amendment then one can square a circle. There really is no point in debating people like this on the technicalities of the Constitution. What they understand is power. He has it and we don't. He knows how to pull the levers that matter, and which Congressman and executive branch bureaucrats to lean on to make sure his bailiwick is not reduced by a single inch. The rest of us, sadly, are left flailing around hoping that pointing to a piece of paper is an effective check on evil.

<comment><USER13> The NSA is filtering a public utility. The users of this utility choose what information they want to broadcast, and should do so with the full knowledge that this information is being bounced between dozens of routers every time it goes out, any one of which may be monitored by its owners. This information ultimately ends up at the destination IP address, whose owner is able to use this data in whichever manner he sees fit, including submittal of said data to a major governmental intelligence organization as occurs with PRISM.

The problem is that people foolishly assume that the plaintext packets they send online are private by default. As in real life, good privacy can only be assured by significant effort on the part of the communicants. Is it evil to observe something occurring on a public street corner? It is not different to observe something occuring on a public IP router.

<comment><USER14> >"The microphones in every nook and cranny are merely filtering a public good, the air. You should know that when you vibrate the air with your vocal chords to make the sound of words, you're willingly broadcasting information patterns in quadrillions of particles in the air. It's not our fault you're so stupid about your security."

Don't be dense. Letters are easily intercepted and read, but we still expect that the government is not doing so because the constitution forbids that sort of behavior and the judicial branch is supposed to protect us from it.

You're conflating privacy with security. Security requires encryption, etc... privacy exists in the absence of monitoring devices.

<comment><USER13> The proper analog is a letter v. a postcard. There is no reasonable expectation that the contents of a postcard will remain private -- the message is right there, and people will see it even accidentally. A typical plaintext packet is the same way. Stuffing your papers in a sealed envelope is analogous to passing your plaintext through an encryption algorithm.

If you are "vibrating the air" in public, then yes, you should realize that there could be a microphone nearby that records what you're saying (and in fact, in modern practice, there is a microphone on pretty much every person you see, any of which could be recording at any time).

<comment><USER15> The problem is people don't necessarily know know they're sending a postcard when they think they're sending a sealed envelope.

<comment><USER13> This is a great opportunity for education, then, correct? Tools have existed that would ameliorate this entire issue for a long time. We need to get people to start using them!

</comment></comment></comment></comment></comment></comment><comment><USER16> >the four-star general presented a timeline of terrorist attacks around the world, from the 1993 World Trade Center bombing to the _Boston Marathon attack_.

Can anyone speak as to why, with the NSA's systems, they were not able to thwart an attack by the ass clowns in Boston? Russia even warned us about them and they made frequent contact with foreigners.

I'm sorry, but that terrorist event seems like low hanging fruit if their system really works to protect us.

<comment><USER17> The question is: Did they _want_ to thwart the attack?

One obvious possibility is that the system that has been erected simply needs _fear_ to be able to continue to exist.

Let some attacks happen (and some Taliban prisons break) from time to time and people can be and will be manipulated via fear, so the powerful stay powerful and the rich become richer. Same old, same old...

Next stop: Middle Class Gone.

Then, people will have nothing to lose _again_ and wake up. Next, we'll have riots and one day, the old system is dead and history will repeat itself once more. So much for the long term.

<comment><USER18> In wartime, it's not uncommon to not act on intel to stop an attack, so as to preserve its strategic value and save more lives later. For any insiders who truly believe in the NSA's mission (which is being generous), "let a few die today to preserve a system that will save thousands tomorrow" is not a big leap.

</comment></comment></comment><comment><USER19> I don't care whether the director of the NSA is a bad guy, it just makes _you_ look bad if you heckle someone on stage. There are probably few more scrutinizing audiences for the director of the NSA than a professional security conference. I would rather have him say what he has to say and be analysed by the many bright minds at the conference than for him to be childishly interrupted and waste time that could be spent talking about the actual role of the NSA in the security of the nation.

<comment><USER20> General Alexander doesn't have to act rude when he has the power of the NSA behind him. The only power we have in a situation like that is to be disruptive and to be rude. Why should we treat him with respect given all the disrespectful things that he's instrumental in?

Those in power keep trying to present this as a "conversation" or a "dialog," but that's absolute bullshit. They're not actually trying to have a conversation, so we shouldn't behave as if that's what's happening. The most powerful thing we could have done would have been to boo him off the stage.

<comment><USER19> How about civil disobedience rather than childish disobedience?

<comment><USER20> We shouldn't be criticizing the heckler for not being a good enough or different enough heckler, we should be criticizing everyone else who did nothing at all.

<comment><USER21> Something along the lines of those who remain silent when powers that be discuss Freedom and Security deserve neither?

</comment></comment></comment></comment></comment><comment><USER22> I find particularly hard to believe the "6,000 NSA cryptologists who have deployed to Afghanistan and Iraq" part. Why would anyone want to deploy cryptologists into a war zone?

I also found disheartening the applause given to the general for his clever answers. "I have. So should you"?! I expected more from the Black Hat crowd.

And, finally, I applaud Jon McCoy for his sacrifice. His willingness to endure all the cavity searches he'll be subjected to before and after every flight inspires us all.

<comment><USER23> Military here.

They aren't really the same cryptologists that we think of, and they aren't the same guys who are writing programs to go through our phone records.

The majority of cryptologists who are forward deployed are there to ensure that the cryptography is used correctly. Despite many years of attempts to make it easier to use, cryptography. especially for portable radios, is a pain in the ass to use. People have to be trained. Areas have to be made where crypto equipment is stored and secured. Deficiencies in use have to be identified. That's what those people are for.

With the fact that we've had more than two million troops deployed during the War on Terror, I'm actually surprised that the number of NSA cryptologists is at 6,000. They must be very senior people.

</comment></comment><comment><USER24> " _Ninety-eight percent of society has issues with this_ "

"The national survey by the Pew Research Center, conducted July 17-21 among 1,480 adults, finds that 50% approve of the government's collection of telephone and internet data as part of anti-terrorism efforts, while 44% disapprove. These views are little changed from a month ago, when 48% approved and 47% disapproved."

[http://www.people-press.org/2013/07/26/few-see-adequate- limi...](http://www.people-press.org/2013/07/26/few-see-adequate-limits-on- nsa-surveillance-program/)

<comment><USER7> I highly recommend reading "The Myth of the Rational Voter: Why Democracies Choose Bad Policies" [1].

Even if polling shows a lack of discontent, it does not mean they were voting rationally. The emotional fear machine of terrorism will always sway towards totalitarianism policies. But if the majority of those citizens were honestly questioned about having their phones (and their entire families) monitored, I doubt they would be for it.

Irrationality and logical fallacies are flourishing on both ends of the spectrum (citizens and politicians), and there are few vocal voices in between correcting the bullshit.

What's missing are leaders (aka media) evaluating and promoting policies based on rationality reasoning rather than what emotional ploys that sell.

[1] [http://www.amazon.com/Myth-Rational-Voter-Democracies- ebook/...](http://www.amazon.com/Myth-Rational-Voter-Democracies- ebook/dp/B007AIXLDI/)

<comment><USER25> > What's missing are leaders (aka media) evaluating and promoting policies > based on rationality reasoning rather than what emotional ploys that sell.

Leaders in media have no external incentive to promote rationality. They make money by appealing to their audience, to increase the size and loyalty of that audience.

There is no solution.

<comment><USER26> Don't make the news beholden to ratings.

<comment><USER27> News is a commercial product. Ratings represent the value of the program to advertisers as a measure of viewership, and the news exists as a way to generate those ratings. No ratings, no advertisers, no news. As with so many other forms of media, you are not the consumer, you're the product (or more specifically, the likelihood of your attention in those all important spaces between the segments.)

Unless you want the state to control everything, how do news organizations make the money they need to put on the broadcasts, if not with ratings? They have to sell _something_ to _somebody_ right? Should the US have a licensing system like they have in Britain?

There are newscasts on PBS but almost nobody watches them (I do, but then I used to work at a small PBS station) and tote bags can only get you so far.

</comment></comment></comment></comment></comment><comment><USER17> > "No, I'm saying I don't trust you!" shouted McCoy.

I believe that's pretty central here. I don't see how we can ever be able again to trust an organism such as the NSA - or even the government.

Transparency is the only solution here. Will we get it?

If we don't, the only _other_ solution would be to cut budgets so drastically that such an enterprise will simply not be possible financially, anymore.

And maybe, as a general improvement: Decentralize the government and give the States back their autonomy (and here you have your link between technology and politics - see the recent cry for less "political posts" on HN).

<comment><USER28> > I believe that's pretty central here. I don't see how we can ever be able > again to trust an organism such as the NSA - or even the government.

We won't. This cancer is a one-way street. It will kill the organism and kill itself with it. And they know it. That's why they buy billions of ammunition rounds and dispatch lightweight tanks throught the states borders. They know that history repeat itself and they don't want another fourth of July to be celebrated in 500 years now (this time US 2.0 that bunch of clever people used GIT to form a better government). NSA systems are not in place to save you and your family from terrorist -- terrorism needs to be wisely guided; some need to leak through so that you and your family keep being scared, submit to the State and pay more taxes for more "safety". Terrorism is not on their agenda. Had they not let shoe bomber and Boston bombers do what they did, me and you would protest against more surveillance (with Boston bombers officials were inconveniently kept notified by russian government) because we would have been felt safe.

> Transparency is the only solution here. Will we get it?

We won't and we can't. They will oppose it saying that if we tell you what we do, bad guys will use it. Nothing will change this approach. Manning did a huge leak of gov docs and military officials said many lives will be lost as a result; other than Arab spring that did not happen on US soil, nothing happened at all. It was a bluff that they will continue using.

> If we don't, the only other solution would be to cut budgets so drastically > that such an enterprise will simply not be possible financially, anymore.

Its not that simple. Even President of United State does not know how many black-ops agencies exist within the goverment. Some entities of the gov sponsor themselves God only knows doing what. If you cut their air supply, they will go to CIA and say: "here are the tapes when you and other officials are negotiating deals with drug cartels in Afganistan, here is a conversation where smuggling is being organized to move tons of heroine into US on a military aircraft, sponsor our venture or we will leak the info". And there you have it. Most likely dirty branches of gov do not need public money but of course they will take it anyways.

<comment><USER17> > That's why they buy billions of ammunition rounds and dispatch lightweight > tanks throught the states borders.

I didn't know this, but I was thinking along the same lines when I read about the "anti-missile blimps" being deployed over DC. I mean it's as if they "prepared for what's coming".

So you seem to think that there is actually no real solution apart from letting things fall apart and then "start from scratch"?

I'd be really interested to hear what other people see as potential solutions for all this.

</comment></comment></comment><comment><USER17> > "I haven't lied to Congress," Alexander responded

And WTF was that, now?

EDIT: I guess this is true because actually, he lied to the chairs in the room. Or some coffee mugs that were also present.

Is that how one evades a lie detector, btw?

It really can't get any sicker. We have now reached the bottom.

<comment><USER29> Last I checked he wasn't James Clapper (they guy who lied to Congress). But maybe I'm wrong.

<comment><USER30> [http://www.youtube.com/watch?v=oYNXVgYhPOc#at=61](http://www.youtube.com/watch?v=oYNXVgYhPOc#at=61)

<comment><USER17> Just watched it again - it's so instructive to see how Keith Alexander is well equipped to function and lie like probably only psychopaths can.

And then, is Obama part of the same category? The contrast between how he's been selling to us and what's really happening is just too much.

</comment></comment></comment></comment><comment><USER31> Looks like I'm a little late for this comment party, but I'd just like to highlight one comment from the Forbes website that I thought was good:

"In general I agree with McCoy in his ad-hoc debate with Gen Alexander as well as his post presentation remarks about the distorted perspective of national security. At the same time, I do conceed that the NSA is acting in good faith and 'within the law' as presented to them via the Patriot Act. I have similar feeling about the value of TSA, but whereas the TSA is a publicly disclosed program, my most pressing concern with the NSA is the secrecy of it. First, the secrecy presents an extremely high degree of risk (in terms of both the probabilty of occurance as well as the dangers) of mission creep. I doubt Congressional nor FISC oversight are adequate to mitigate this risk. Second, secrecy is not really a valid tool for deterence. Anti-terrorism programs are like a doomsday devices: it's a good deterence only if it is widely known about (see Dr Strangelove). Moreover, while secret programs might be good for support of kill/capture programs, it is far from clear that such efforts reduce terrorism.

For all I know, the government spy/drone program is to terrorists as a hammer is to an ant colony. If you have a spilled pot of honey in the kitchen, it doesn't matter how good you are at spotting and smashing ants. I want to know these things, so that I can help shape public policy through the ballot box." \--uspatriot2001

I think that's a good assessment of the high-level picture, and it's much more informative than "NSA BAD! Freedom good!" (not to belittle anyone's comments here) in that it reflects a road forward.

</comment><comment><USER32> Audio here: [https://soundcloud.com/larrymagid/nsa-director-general- keith](https://soundcloud.com/larrymagid/nsa-director-general-keith)

<comment> UVB-76 The exchange in question starts around 33:40

<comment><USER33> I listened starting a couple minutes before that and felt like I was going to fall asleep. I'm sure it loses something not being there in person but Mr. Alexander's delivery sure seems ponderous.

</comment></comment></comment><comment><USER34> All things considered, it seems like Keith Alexander did a pretty good job of handling the situation. Personally I think the heckler sounded like a douche ("read the Constitution" sounds like a dumb thing to say to a 4 star general who probably has better knowledge of it than your average hacker), even if he (the heckler) had some good points.

It would have been easy for the situation to devolve into something much uglier, but fortunately it seems like things stayed pretty calm!

<comment><USER35> I'm curious why you claim that a 4-star general would have a better knowledge of the Constitution than an average hacker?

Especially one who seems to be running an illegal and unconstitutional program like Gen. Alexander.

<comment><USER36> For the same reason that an average hacker would have a much better knowledge of the code for various daemons than the sysadmins who administer them - they're reading critically and looking for vulnerabilities where the actual semantics differ from the purported ones. I've no doubt that what the NSA does actually _is_ "constitutional" under the common corrupted interpretations of it - government programs are structured to avoid mortal conflicts with the constitution, sympathetic or even just probabilistic courts then assert that specific facets aren't actually over the line, and severable agents acting "independently" informally make up the difference.

Law is akin to code, and suffers from the same complexity-induced limitations that end up as the halting problem. With enough indirection, meanings and definitions divert wildly from what they purport. This divergence starts as soon as abstraction enters the picture - for example the widespread tendency to see rights as "primitives" that are acceded when forming higher-level emergent structures. Participation in the higher-level structure becomes de facto mandatory, and those purported rights end up nowhere to be seen. And once this erosion process starts, it becomes harder and harder for anybody to see that their rights should apply universally, rather than just in a nondescript wood shack on unincorporated land in Wyoming.

<comment><USER35> Certainly I agree with your perception of how the NSA and General Alexander believe that they are "conforming" to a proper interpretation of the Constitution, though I disagree with that particular interpretation they hew to.

I guess I feel that the average hacker has a better understanding of the platonic ideal or "most correct" interpretation of the Constitution than Gen. Alexander.

</comment></comment></comment></comment><comment><USER5> Wow, I am really, really surprised. Especially as Gen. Alexander gave the keynote speech to DEFCON last year, and explicitly said, when asked after his speech:

Q: "[..]Does the NSA really keep a file on anyone? [...]"

A: "[..] Frist, no, we don't [...]"

Absolutely astonishing!

listen/watch here: [http://youtu.be/tz0ejKersnM?t=33m2s](http://youtu.be/tz0ejKersnM?t=33m2s)

</comment><comment><USER37> No idea how accurate a depiction of the speech this is, but the article is -- contrary to the title -- portraying the speech as having been warmly received, applauded, with one lone heckler. Now, whether or not one agrees with General Alexander, there are more courteous and productive ways to express yourself than yelling "bullshit".

<comment><USER38> > there are more courteous and productive ways to express yourself than > yelling "bullshit".

I disagree. When chatting with Faust, I think it's better to avoid the drawn out conversation and just tell him to fuck off. This Alexander guy is a liar and a felon (lying under oath to congress) and has earned our collective disrespect. He deserves a lot more than "bullshit!" He's a domestic terrorist that's infiltrated the government as far as I'm concerned.

<comment><USER39> What has he done that makes him a terrorist? Or are you just inferring that mass spying is a form of terrorism?

<comment><USER38> It was kind of a flip thing to say; I was on a bit of a roll.

The argument, such as it is, would go something like this: Instead of physical terror, the policies Alexander pursues create a psychological terror that jeopardizes "our way of life". Instead of a fear of being blown up, we have a fear of persecution for what we say.

A terrorist thinks he is beyond the law and uses his limited power to create fear in a population in order to achieve his ends, which the majority does not support. That's a description that seems to apply to Alexander.

I know he's different, but he's definitely not as different as he thinks.

</comment></comment></comment></comment><comment><USER40> Keith Alexander's way of bending words, and his facility with swinging a crowd and lying skillfully under pressure once again reminds me disturbingly of accounts I've read about psychopaths, and of probable psychopaths I've encountered. Those patterns could of course be stuff you pick up anyway as a top player for power, but I find the parallels highly worrisome. For those who have never read anything about the subject, may I suggest browsing e.g. "Without conscience" by Robert D. Hare. The sections with court proceedings in particular illustrate well what I mean.

</comment><comment><USER41> I think this underscores Defcon's decision to exclude the Feds this year.

<comment><USER42> Defcon and Blackhat were founded by the same dude. Why didn't they exclude feds from BlackHat as well? Guessing it was probably because that is their big money maker with all the corp/fed support.

<comment><USER43> BlackHat is a more "professional" event for the security industry. People that ostensibly work in the security industry go there just like people go to o'reilly conferences for various technical areas.

DEFCON is a community event for anybody and is totally anonymous (entry fee is only payable by cash in person) and has a more freewheeling, community feel.

</comment></comment></comment><comment><USER44> I hate Forbes - interstitial ads and persistent headers are super annoying. Can we stop posting Forbes links?

Boing Boing has the same coverage with a much more reader-friendly site design: [http://boingboing.net/2013/07/31/nsa-capo-heckled-at- black-h...](http://boingboing.net/2013/07/31/nsa-capo-heckled-at-black- hat.html)

<comment><USER44> Grrr, the Boing Boing article just links to Forbes.

</comment></comment><comment><USER45> Video seems to be up at [https://www.blackhat.com/us-13/archives.html#Alexander](https://www.blackhat.com/us-13/archives.html#Alexander)

<comment><USER46> *Note the above links to an MP4.

The 2013 Keynote is now on Youtube: [https://www.youtube.com/watch?v=xvVIZ4OyGnQ&t=5m](https://www.youtube.com/watch?v=xvVIZ4OyGnQ&t=5m)

\-- Alexander begins at 5 minutes in

\-- no slides visible in video

\-- 'heckling' @ 39minutes in)

\-- TLDR: One cropped screenshot of 'telephone metadata', assurances of internal audits, a thumbup from a Senate sub-committee.

Slides: [https://media.blackhat.com/us-13/us-13-Alexander- keynote.pdf](https://media.blackhat.com/us-13/us-13-Alexander-keynote.pdf)

</comment></comment><comment><USER47> In fairness, the title could just as well be "NSA director _applauded_ on stage at Black Hat security conference".

The crowd's reaction was definitely mixed.

</comment><comment><USER48> From the article: "Everyone's thinking this, but no one's saying it public, so everyone thinks they're alone," he said. "Ninety-eight percent of society has issues with this…But no one speaks up."

While I think some people don't how to speak up, beside calling/writing my congress person, many don't care either way. In fact, many older people I have talked to (I am in my twenties) dont have a problem with what the NSA is doing. They dont really mind if their data is being sifted through as long as its being used to "Find Terrorists or what have you."

My problem is the lack of oversight and the fact court that has approved 1,789 out of 1,748 requests (97.7%!!!!). Not only that but they have legal justifications for what they are doing, therefore since its legal its allowed, forget the morals/ethics involved.

Who's to say they won't turn this around on the public at large because it will prevent smaller crimes instead of terrorists. Also whats to stop the government from calling people who oppose their views terrorists and then just spy directly at American people.

</comment><comment><USER49> "And if you disagree with what we're doing, you should help us twice as much."

Right, so they end up like Snowden...

</comment><comment><USER50> All this bad press is a great incentive for people to develop encryption skills. I'm now looking into javascript client to client encryption of certain social sites and services using greasemonkey to organise the contacts and keys database.</comment>


Apache Geode: Distributed, in-memory database - <AUTHOR> http://geode.incubator.apache.org/ <comment><USER2> Didn't understand much from the linked page, but I found this website (from Pivotal, the commercial entity behind Geode) quite informative. Perhaps it's useful to others.

[https://pivotal.io/big-data/pivotal-gemfire](https://pivotal.io/big- data/pivotal-gemfire)

===

I found this interesting deployment:

        China National Railways use Geode to run railway ticketing for the entire
        country with a 10 node cluster, managing 2 TB of "hot data" in memory,
        and 10 backup nodes for high availability and elastic scale.

        Holiday travel periods [Chinese New Year's] create peaks of 15,000 tickets
        sold per minute, 1.4 billion page views per day and 40,000 visits per second.

[http://pivotal.io/big-data/case-study/scaling-online- sales-f...](http://pivotal.io/big-data/case-study/scaling-online-sales-for- the-largest-railway-in-the-world-china-railway-corporation)

<comment><USER3> to understand where it comes historically and ideologically/architecturally from - the keyword here is Smalltalk [https://en.wikipedia.org/wiki/Gemstone_%28database%29](https://en.wikipedia.org/wiki/Gemstone_%28database%29)

The guys have been for 3 decades sitting on the forested Oregon river bank (the place is called Beavertown for reason :) thinking straight and clear in Smalltalk... :)

<comment><USER4> Geode and its predecessor GemFire are pure Java.

The original engineering expertise came from the team that built the Gemstone OODB.

Beaverton isn't so forested anymore, but lots of indigenous object oriented expertise for sure.

<comment><USER3> >Geode and its predecessor GemFire are pure Java.

i know (was talking at some point about joining, already here in SV), and this is why i wrote only "3 decades" as i don't know precise dates of GemFire only suppose it to be beginning of 200x. My simple arithmetics mistake though - 2002 - 1982 is 2 decades, sorry, i see where confusion comes from :)

</comment></comment></comment></comment><comment><USER5> Landing page and documentation should be reconsidered. lots of unanswered questions,

\- _distributed, in-memory database_ \- how does it compare to SAP HANA? or Redis? if "... database" where is the querying language? if you have own querying language other than SQL please show us.

\- _Performance is key_ \- tell us about some benchmarks

\- _Consistency is a must_ \- CA or CP in CAP theorem?

\- Can I fully drop RDBMS in favor of Geode?

\- web site built as Landing page/PR for product, then all of a sudden starts Community/Contributors/Getting started columns, with loooooong list of mailing lists, smaller content for contributors and very important part(getting started) presented as tiny small link :(, please put there some useful info about product, you already have Community/Contribute menus.

<comment><USER6> > \- Consistency is a must - CA or CP in CAP theorem?

There is no CA. So, they must mean CP.

Remember, kids: if they promise CA, run away.

<comment><USER7> CA is possible if its read only, essentially, with all updates synchronized when a partition is not present [which is the majority of the time in the real world].

<comment><USER8> It's semantics, but I'd argue if you're read only than you're changing the definition of available.

<comment><USER4> Geode is write-intensive, and generally optimizes for consistency over availability. That being said - there's a lot built into Geode to ensure availability as well.

</comment></comment></comment></comment></comment><comment><USER9> Feedback on the documentation:

It's not immediately obvious to me how Geode is different from Redis. When would I want to use Geode over Redis, and vice versa?

<comment><USER4> Redis and in-memory data grids are pretty different animals. I would characterize IMDG's like Geode to be concurrent write intensive, and have flexible data models. It also scales out better than Redis in a more automated fashion.

Redis is a great read-intensive cache. It also has a powerful data model, but you have to use their data models. Example: If you want to run calculations on lists or sets, they have powerful operations you can call.

IMDG's such as Geode were built with the rise of automated trading in the finance industry.

</comment></comment><comment><AUTHOR> Two Videos:

1\. "Open Sourced GemFire In-Memory Distributed Database and Apache Contributors" ([https://www.youtube.com/playlist?list=PL62pIycqXx- TTMXsq09BE...](https://www.youtube.com/playlist?list=PL62pIycqXx- TTMXsq09BEGfacLjeHZM28))

2\. "Creating a Highly Scalable Stock Prediction System with R, Geode & Spring XD" ([https://www.youtube.com/playlist?list=PL62pIycqXx- Rzd_HcjU7Y...](https://www.youtube.com/playlist?list=PL62pIycqXx- Rzd_HcjU7YXq1kIOa28M8X))

</comment><comment><USER10> I'm starting to get lost in the new Apache animals.

How does Apache Goede compares to Apache Ignite (advertised as "in-memory data fabric"?

<comment><USER4> Apache Geode and Apache Ignite are more similar than they are different.

Apache Ignite, based off the commercial distribution Grid Gain is newer to market.

Apache Geode, based off the commercial distribution GemFire, has a long history in the market.

<comment><USER10> I wonder why Apache would need to have two "more similar than different" products.

<comment><USER11> Apache accepts anything that companies donate. (They say they don't, but it's hard to find anything they've rejected.)

<comment><USER12> That's not quite the right emphasis.

Apache is happy to provide a home for any community that is willing to adhere to our governance rules and traditions. Competing projects are OK.

Projects are almost never rejected because preparing a proposal for incubation is rigorous and many projects who would be a poor fit self-select out.

Source: former VP Apache Incubator, who has both helped prepare successful proposals and privately counseled projects who decided not to come to Apache.

</comment></comment></comment></comment></comment><comment><USER13> Can anyone comment on Geode's non-Java support?

I'm asking because a lot of Java "big data" stuff tend to prioritize Java clients (ZooKeeper, Kafka, Hadoop HDFS, Storm, VoltDB and HBase come to mind), and while there are sometimes clients in other languages, they tend to be second-class citizens that take years to reach feature/performance parity with the Java stuff.

For example, last I checked there still wasn't a mature, feature-complete Kafka client (consumer and producer with built in offset management) for Go.

<comment><USER14> Gemfire (on which geode is based) has a fully featured c, c++, c# client which has feature parity with the Java client. I don't know if pivotal is going to open source these clients too.

There however is a REST api and a python client [https://github.com/gemfire/py-gemfire-rest](https://github.com/gemfire/py- gemfire-rest)

</comment></comment><comment><USER15> Sorry for the stupid question: how do in-memory DBs deal with power failures? e.g. someone walking in the server room and trip power cable.

I can understand for read-only in-memory, but what about writes?

<comment><USER16> In the case of Geode, you can make a "Region" (think table) persistent on disk and using the concept of shared nothing architecture [1] to avoid SPOFs.

What's also interesting is that we offer a very efficient way to recover data from disk as well[2] in the case of a crash of a single node or the entire cluster.

[1] [https://en.wikipedia.org/wiki/Shared_nothing_architecture](https://en.wikipedia.org/wiki/Shared_nothing_architecture) [2] [http://gemfire.docs.pivotal.io/docs- gemfire/latest/managing/...](http://gemfire.docs.pivotal.io/docs- gemfire/latest/managing/disk_storage/how_startup_works_in_system_with_disk_stores.html)

</comment></comment><comment><USER17> FWIW, Pivotal is hiring in our Big Data team, largely based in Palo Alto. Geode (incubating), HAWQ (incubating), Greenplum, Pivotal HD, MADlib etc are all mostly developed with engineering effort that we donate.

Hit me up with an email (email@example.com) or visit pivotal.io/careers if you're interested.

</comment><comment><USER18> > Gemcached > Geode servers can be configured to talk memcached protocol.

hey this is very interesting, it could work as a persistent acid memcached drop-in replacement!

<comment><USER19> There's already memcacheDB for that [https://github.com/LMDB/memcachedb](https://github.com/LMDB/memcachedb)

<comment><USER18> That one doesn't seem distributed. But I just glanced at the readme.

<comment><USER19> Distribution is usually handled by the memcache client libraries.

</comment></comment></comment></comment><comment><USER20> Source here:

[https://github.com/apache/incubator- geode](https://github.com/apache/incubator-geode)

</comment><comment><USER21> I believe this competes with another well known open-source In-Memory Data Grid, Hazelcast. worth checking out.

<comment><USER22> Last time I checked Hazelcast couldn't be run standalone. Also I wouldn't use it as a database for anything more than a few MB of data.

This is probably closer to Apache Ignite aka Gridgain.

<comment><USER21> Can you explain why ? I remember reading a ton about off-heap memory work they were doing

</comment></comment></comment><comment><USER16> Geode FAQ Wiki page [https://cwiki.apache.org/confluence/display/GEODE/Technology...](https://cwiki.apache.org/confluence/display/GEODE/Technology+FAQ)

</comment><comment><USER4> If you are interested to join the developer community around Apache Geode, subscribe via: email@example.com

</comment><comment><USER23> It looks similar to Hazelcast / Oracle Coherence?

<comment><USER24> Yes, both Geocode, Coherence, Ignite and Hazelcast are quit similar.Grid computation.

</comment></comment><comment><USER25> Would you choose it over Datomic?

<comment><AUTHOR> Datomic's immutable storage and time-travel query capabilities are awesome, and I often miss them in other DBs. But Datomic currently isn't designed for write-intensive workloads. And while you can shard Datomic's transactor and then combine multiple DBs in a query ([http://nosql.mypopescu.com/post/19310504456/thoughts- about-d...](http://nosql.mypopescu.com/post/19310504456/thoughts-about- datomic#comment-465580077)), that's only going to get you so far.

However, Apache Geode lets you add custom indexes so it might not be too hard to add Clojure's persistent data structures as a custom index scheme and hook in Apache Geode as a backend to Clojure Datalog:

Clojure Datalog: [https://github.com/fogus/bacwn](https://github.com/fogus/bacwn)

Datscript: [https://github.com/tonsky/datascript](https://github.com/tonsky/datascript)

Clojure's Persistent Data Structures for Java: [https://github.com/grignaak/clj-ds](https://github.com/grignaak/clj-ds)

</comment></comment><comment><USER26> Let me guess: it's written in Java?

<comment><USER26> Yes.

Why are all Apache projects are written in Java?

<comment><USER27> Apache isn't written in Java :)

[https://en.wikipedia.org/wiki/Apache_HTTP_Server](https://en.wikipedia.org/wiki/Apache_HTTP_Server)

<comment><USER28> That's Apache httpd. Apache is a software organization.</comment></comment></comment></comment>


Oracle v. Google - Oracle Loses on Motion Regarding Leonard and Cox Reports - <AUTHOR> http://www.groklaw.net/article.php?story=20111129080612204

<comment><USER2> Too bad the recent EU decision about copyright on programming languages probably won't influence this case by the time it's over.

<comment><USER3> It's highly unlikely that the EU decision would alter the outcome of Oracle v. Google in any way regardless of timing. This is a US case involving completely different parties and a significantly different set of facts.

Decisions of non-US courts usually have little influence unless statutory or case law is severely lacking or highly ambiguous, and even then they are treated much like one might treat a book by a legal scholar. Decisions by courts that do not share the English common law traditions are of even more attenuated influence.</comment></comment>


How to Manage Connections Efficiently in Postgres, or Any Database - <AUTHOR> https://brandur.org/postgres-connections <comment><USER2> Good pre-reading to this article is Brett Wooldridge's (of HikariCP) article on connection pool sizing: [https://github.com/brettwooldridge/HikariCP/wiki/About- Pool-...](https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing)

An important consideration is that increasing the pool size many times above the number of cores is very often counterproductive in the face of decreasing IO latency. Also, there may often not be justification for anything other than a constant-sized pool.

<comment><USER3> Yes this is a great article, thanks for reminding me of it.

</comment></comment><comment><USER4> PgBouncer is basically a requirement of any production Postgres system, but I've always wondered why it simply wasn't part of the Postgres package itself? For instance, it means a dedicated EC2 instances is necessary for getting the most out of AWS RDS - even the whole point of RDS is to avoid managing that server..

<comment><USER5> We were big proponents of pgbouncer for years, but having one extra, sometimes capricious actor in our architecture ended up not being worth it. Especially if you're IO-heavy like we are, and that Web App -> DB pipe is the hottest path in the entire system.

Removing it altogether made monitoring and debugging performance issues easier, and client-side pools were essentially "good enough" for our purposes.

<comment><USER4> Were you managing your own DB server instance? I don't think client side pools are possible with RDS.

<comment><USER5> We're on RDS right now. We use connection pools from [https://www.yesodweb.com/book/persistent](https://www.yesodweb.com/book/persistent)

<comment><USER4> Ohhh, I understand. Long day. I got that backwards, I thought there was a server-side option I wasn't aware of. So you don't have a dedicated server with PgBouncer on it, you just move the pooler to where the requests originates from. I think this might be problematic in a highly elastic/distributed architecture but would work if you have a beefy dedicated application instance.

<comment><USER5> Correct, you'd have to instruct your pools to resize as machines are added and removed. We haven't had to do that so far, as we're not that dramatically elastic.

</comment></comment></comment></comment></comment></comment><comment><USER6> "Or any database", except it's really driven by postgres' process model, and definitely doesn't apply to most distributed databases

</comment><comment><USER7> This is a fine basic intro to the topic, but the whole argument is hung on a very sketchy benchmark (as noted by the article itself in a footnote).

Some things that'd help improve the argument:

1\. Account for table/row locking in the benchmark: is it a factor or not?

2\. Re-run the benchmark with for some of the proposed solutions, e.g. local connection pooling and PgBouncer.

<comment><USER8> I didn't know about this global locking but I'm assuming the row / table locking is taken into account in the testing. Presumably just inserting into a simple table with explicit unique ids each time should avoid issues with locking (I guess you'd want to avoid indexes and sequences).

</comment></comment><comment><USER9> This trips up a lot of newbies who have no concept of connection management or scope. I routinely get questions about it in regards to the orm library I wrote, which includes a connection pool of course.

Another thing to be aware of with postgres is that connections in an error state (for example, due to trying to insert a row that violates a constraint) must be rolled back before they can be recycled.

</comment><comment><USER10> I wonder how alfresco ends up with such a huge recommended default pool size of 275 [1]. Is this bad architecture on their part or are different principles at play?

[1] [https://docs.alfresco.com/5.0/tasks/postgresql- config.html](https://docs.alfresco.com/5.0/tasks/postgresql-config.html)

</comment><comment><USER11> Nice article, but doesn't actually demonstrate that connections setup and tear down is the bottleneck. If you really have 500 active connections all "shouting" at the database simultaneously, the fact that those connections were obtained from a pool or not isn't going to change the throughput of the database.

Pooling introduces numerous other issues as well. For example, clients probably shouldn't all be using the same entitlements to connect, right? Surely there are some segregation of duties and roles that clients will be using alternative entitlements into the system?

Finally, you have to show some before and after results. This article simply asserts that connection pooling would improve the situation, but does nothing to demonstrate the point. Even the details of the server matter...how many cpus, how many drives, HDD or flash, what file system is being used?

<comment><USER12> I don't have the numbers but I do know we are facing this problem at work. When you have microservices that are supposed to respond in <30 ms connection setup time becomes noteworthy.

Ironically we exacerbated the symptoms by reducing average calls per page below 1.0. The burstiness of the traffic made it clear that the low water mark for the connection pool was not working out.

</comment></comment><comment><USER13> note that while the MySQL connection-pool equation is pretty different from that of Postgresql's, at least in my experience MySQL connections use less memory (sorry, no measurements handy, feel free to correct me, but they generally are handled as a thread rather than an entire subprocess and we regularly see apps that spin up thousands of them fairly cheaply) and also have a lower startup time (also anecdotal, sorry), the MySQL world does finally have middleware connection pools now such as proxysql: [https://proxysql.com/](https://proxysql.com/)

<comment><USER14> There isn't much practical difference. MySQL's memory cost is lower, largely due to using a thread per connection rather than a process, so you can have a larger number of connections before running out of memory. But this really doesn't matter since by the time you run out of memory with PG or MySQL you'll probably have run into a situation where performance has tanked due to the amount of context switching going on. In both cases you likely want something like `num_cpu_cores * 2..3` connections to the database for optimal performance.

<comment><USER13> it is different, because if you have 100 processes all talking to the database, they can safely all have an application-local connection pool holding onto a dozen connections in a pool that are probably largely idle, but are ready in a flash. context switches aren't necessarily a problem because the database probably sees 5-10% of these connection receiving queries at a given moment. Also, mariadb has a thread-pool feature ([https://mariadb.com/kb/en/library/thread-pool-in- mariadb/](https://mariadb.com/kb/en/library/thread-pool-in-mariadb/)) that allows a significant reduction in threads for a given number of connections.

Contrast to Postgresql, where none of the above is an option, every connection is a process and you don't want thousands of those piled up.

</comment></comment></comment><comment><USER15> Does anyone have an understanding of if and how much this applies to AWS's Aurora service, when operating as a PostgreSQL database?

<comment><USER16> The Aurora handbook strongly recommends using these techniques, or using a connection proxy if connection pools won't work for your situation. Having too many connections can cause your Aurora instances to scale up as well, limiting database performance, and leaving you with hundreds more instances than you want to pay for.

[https://d1.awsstatic.com/whitepapers/RDS/amazon-aurora- conne...](https://d1.awsstatic.com/whitepapers/RDS/amazon-aurora-connection- management-handbook.pdf)

<comment><USER15> Thanks. Your link is for MySQL, but I found a little related info for PostgreSQL: [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Managing.html)

</comment></comment></comment><comment><USER17> Those diagrams look really nice. Does anyone know how they were made?

<comment><USER18> The diagrams were done with monodraw - [https://monodraw.helftone.com/](https://monodraw.helftone.com/)

<comment><USER19> Second that these diagrams look really nice and clean- does anyone happen to know if there's an alternative that isn't Mac only?

<comment><USER20> Have a look at draw.io ([https://about.draw.io/features/](https://about.draw.io/features/)) which can make a lot of fantastic diagrams. Be sure to look at the "More shapes..." options and choose the libraries that are of interest to you. Free and open- source.

</comment></comment></comment></comment><comment><USER21> You need to use non-blocking async. clients.

And you should probably use HTTP too.

All database clients are blocking which kills the CPU with IO-wait and most of them use some binary protocol.

Here is the worlds first and only async. parallel HTTP app server and HTTP/JSON database:

[https://github.com/tinspin/rupy](https://github.com/tinspin/rupy)</comment>