Ethereum Price Climbs to $360; Crypto Market Cap Pierces $160B - <AUTHOR> https://www.cryptocoinsnews.com/ethereum-price-climbs-360-crypto-market-cap-pierces-160-billion/ <comment><USER2> It seems really disingenuous to talk about market caps in cryptocurrencies. The US dollar doesn't have a market cap. No currency does. Market cap is the markets valuation of a company. The term is used to contrast it against their actual valuation. The use in crypto is just a mechanism to dress them up like stocks.

If you call these currencies, then you need to recognize that they are extremely deflationary and will never be used as fiat currencies are. If you call them speculative investments, you need to realize they're not backed by any actual value.

I haven't done a deep dive on Ethereum yet, so this is mostly addressing Bitcoin and all its derivatives.

<comment><USER3> > If you call these currencies, then you need to recognize > that they are > extremely deflationary and will never be used as > fiat currencies are.

For me, one of the best things about bitcoin is that it is actually going allow a lot of arrogant economic dogma to finally be put to the test, including that:

\- currencies must be inflationary or the universe will explode and people will save so aggressively that they will die of hunger

\- printing free money and giving it to rich banks is a good thing and doesn't totally fuck the little guy

\- monetary stimulus smooths out economic cycles, and totally isn't part of the cause of the ridiculous boom-bust cycles we are having now

<comment><USER2> You need some semblance of stability though. No one is going to agree to pay me 1 BTC per week if they're implicitly giving me a 30% raise in week 2. Just the same, I would not be happy to get paid in an extremely inflationary currency that meant I was taking a pay cut week over week.

It has not shattered any belief in traditional currencies or governments for me. In fact, it has made me realize their value. People are just using it for speculation which makes it feel a lot more like a multi-level marketing scheme than a currency.

<comment><USER4> How does speculation make something a "multi-level marketing scheme"?

<comment><USER2> First, I should clear up some terms. I say speculation as a contrast to investing. Investing is when you have an understanding of something underlying value (or expected value) based on some financial fact (e.g. past profits). Speculation is when you purchase something and hope for it to go up in value.

I see cryptocurrency prices largely driven by speculation. There is no underlying value that justifies the price. This means that the people purchasing cryptocurrencies today are relying on someone else purchasing it at a higher price tomorrow in order to make a profit. I might have used the term MLM incorrectly when what I meant was pyramid scheme.

<comment><USER4> > I might have used the term MLM incorrectly when what I meant was pyramid > scheme.

You are now using the term 'pyramid scheme' incorrectly.

Out of curiosity, do you consider other objects of speculation, including stocks, bonds, futures, derivatives, commodities and real estate, to be 'pyramid schemes'?

</comment></comment></comment></comment></comment></comment><comment><USER5> One thing that I rarely hear about crypto-currencies is the environmental impact. I am a bit surprised to see the environment-friendly silicon valley getting excited by such an energy-wasting protocol. If we have to burn CPUs reversing hashes every time someone makes a payment, we should start building new coal and nuclear power plants now.

<comment><USER6> Ethereum is planning a switch from Proof of Work (burning CPU/GPU/ASIC cycles to secure the chain), to Delegated (?) Proof of Stake - people lock away some of their ETH holdings, some of those people are random selected to validate a block. They attest which transactions are valid, and put them in a block. If they're judged to be honest/correct they're rewarded with ETH in proportion to their stake. If they're found to have broken the rules their stake is not returned, or only partially returned.

PoS has detractors/claimed flaws, but it is definitely much less energy intensive than PoW.

<comment><USER7> Ethereum's moving to PBFT (Practical Byzantine Fault Tolerant) Proof of Stake.

See this FAQ for more details, it has the most up to date details on Ethereum's PoS progress:

[https://github.com/ethereum/wiki/wiki/Proof-of-Stake- FAQ](https://github.com/ethereum/wiki/wiki/Proof-of-Stake-FAQ)

</comment></comment></comment><comment><USER8> Is there some meta-miner project you can use to mine a few of every not- obviously-garbage new coin, then stop, just in case they turn out to be worth something?

<comment><USER9> This is developing along those lines: [https://www.nicehash.com/?p=gstarted](https://www.nicehash.com/?p=gstarted)

</comment></comment><comment><USER10> I invested a small (hundreds) into Ethereum and a few other coins earlier this year. They've since doubled in value. I can imagine if you were a very high risk investor and believed in crypto's right now you could be making a lot of money.

<comment><USER11> I am up 6,200% ytd.

Some like NEO and Waves did that for me. It took me cycling through 25+ altcoins to reach that level.

<comment><USER12> NEO is interesting if for no reason other than it doesn't even exist yet, and the market price has risen so quickly. One can easily imagine a scheme with any of the premined networks where the price is manipulated by only releasing a small percentage of the total supply and than buying up what's available on the exchanges with BTC to pump the price and than as rubes buy in, slowly selling off at the inflated prices.

The 50,000,000 units were arbitrary distributed to unknown parties within China.

</comment></comment></comment><comment><USER13> I wonder how much of that is is lost wallets ect as it's very easy to have unrecoverable issues.

<comment><USER14> Here's an example … [https://etherscan.io/address/0x4f35f119145b8d599d2b70b37c730...](https://etherscan.io/address/0x4f35f119145b8d599d2b70b37c73086f71cd416b)

I mined these ~1500 ether in the first few days of their going live. Then I killed all my AWS instances, inadvertently making this account inaccessible as a result. It's "fun" to watch it go up in value. It was under $1K at one point.

I don't feel too bad, though, as I would have sold at 2x rather than 500x anyway …

<comment><USER15> someone would have to work 30+ years to make as much after-tax income as someone made just mining some coins ..just mindblowing

</comment></comment></comment><comment><USER16> I don't mean to say that all coins are valuable. But it does ressemble another gold rush. Do some remember the domain name craze 20 years ago? Any kid smart enough to have bought generic domain names back then made lots of money.</comment>


What's New in Ruby 2.1 [pdf] - <AUTHOR> http://www.atdot.net/~ko1/activities/toruby05-ko1.pdf <comment><USER2> Can someone explain the "write barrier" stuff? It's hard to glean what the problem is that they're solving here... And what they're saying about Arrays, Strings, Hash, etc, in relation to WB protected vs. WB unprotected.

<comment><USER3> I've not read the code, so someone may have to correct me, but I think this is the gist of it:

Ruby 2.1 introduces a generational garbage collector, this divides all objects into young and old generations. A regular GC run will only look at the young generation, with the old being collected less frequently. An object is promoted to the old generation when it survives a young generation run.

If you have objects in the old generation referring objects in the young generation, but you're only looking at the young generation it may seem like an object doesn't have any references, and you might incorrectly GC an in-use object. Write barriers prevent this by adding old generation objects to a 'remember set' when they are modified to refer to a young generation object (eg old_array.push(young_string)). This 'remember set' is then taken in to account when collecting the young generation.

Most generational garbage collectors need these write barriers on all objects, but with the many 3rd party C extensions available for Ruby this isn't possible, so a workaround was devised whereby objects that aren't write barrier protected won't ever be promoted to the old generation. This isn't ideal as you won't get the full benefit of the generational GC, but it does maximise backwards compatibility.

</comment></comment><comment><USER4> Everytime I read a Ruby version announcement/explanation, it reminds me that I've been a coder working and thinking exclusively in English, but my current programming language of choice is prominently steered by non-English speakers...and somehow things work pretty peachy. But it always makes me wonder how much more in-depth and richer the discussion of Ruby amongst its Japanese maintainers, before they translate it for us in English?

<comment><USER5> You can subscribe to ruby-dev and find out. ;) It's actually much more low- traffic than ruby-core, and someone has volunteered to translate any email if you ask.

</comment></comment><comment><USER6> I really like the ObjectSpace.trace_object_allocations addition. I had a quick look at the 2.1 docs for ObjectSpace and there seem to be all sorts of interesting methods in there reachable_objects_from(obj) and memsize_of(obj) for example. It looks like understanding memory usage and tracking down memory leaks will get a lot easier with 2.1.

[http://ruby-doc.org/stdlib-2.1.0/libdoc/objspace/rdoc/Object...](http://ruby- doc.org/stdlib-2.1.0/libdoc/objspace/rdoc/ObjectSpace.html#method-c- trace_object_allocations)

<comment><USER3> #reachable_objects_from was added in 2.0, and I think #memsize_of was introduced in one of the 1.9 releases. The documentation is new though.

I wrote a little about them here: [http://globaldev.co.uk/2013/03/ruby-2-0-0-in- detail/#id6](http://globaldev.co.uk/2013/03/ruby-2-0-0-in-detail/#id6)

<comment><USER6> Ah nice, thanks, missed the part about requiring "objspace" to mix them in and assumed they would be on the already existing ObjectSpace module.

</comment></comment></comment><comment><USER7> Koichi is gives some of the best presentations. It is hard enough to be funny in one language, yet to be funny and understandable in two is quite an accomplishment. If you're ever at a Rubyconf, go to his talks. Also love the speed work he's been doing with generational GC. For a primer, Pat Shaughnessy has a great talk you should check out, also Ruby Under a Microscope is a good dive into Ruby.

</comment><comment><USER8> I'm uncertain why they've added an `f` suffix for frozen string literals/instantiation.

I can understand needing to freeze an existing String, but you can do that with Object#freeze, but a literal syntax seems to overlap with the usage of Symbols. The only benefit I can think of is not needing to use Symbol#to_s when working alongside strings.

I'm not saying frozen strings are not useful, because they can be after taking in some input/params to work with/store, but a literal syntax seems extremely edge-case-y. Now instead of "foo".freeze it's "foo"f, but how frequently will that save you time/trouble?

The bigger implication is that slides 23/24 shows that immutable strings and symbols will share their heap locations, if I'm reading the diagrams correctly, but have different object_ids.

This seems a bit bewildering and focused on micro-optimization, which is not the mental model I have in mind when I'm coding in a very high level language. It's also blurring the semantic difference between Symbols and Strings.

<comment><USER9> _Now instead of "foo".freeze it's "foo"f, but how frequently will that save you time/trouble?_

Worse, it seems to break the model of "interact with objects by sending messages", where you send a message using

        <object>.<message>

Is "foo"f _not_ sending a message? If so, then what is it doing? If yes, then why new syntax?

<comment><USER10> It's not sending a message - that would defeat the purpose. It's constructing a frozen string, reusing an existing one when possible.

Prior to 2.1:

        def foo
          "bar".freeze
        end

does these things every time `foo` is called:

1\. copies the characters 'b', 'a', and 'r' into a mutable string

2\. sends the message `freeze` to the new string, which...

3\. marks the string as frozen.

In 2.1 `"bar".freeze` is equivalent to `"bar"f`, which will not make a copy every time `foo` is called. See [https://bugs.ruby- lang.org/issues/8579](https://bugs.ruby-lang.org/issues/8579) for more discussion.

<comment><USER9> I see. Thank you.

</comment></comment></comment></comment><comment><USER11> I'm looking forward to `def` returning a symbol (page 11).

        private static void def main(args) ...; end

<comment><USER12> That slide was a bit confusing for my underslept mind. Is that meant as an example of calling methods which work over the result of defining main (in this case static and void)?

<comment><USER13> If you say "private def methodname(args); end",

the "private" keyword already takes a symbol for a method name to privatize, so in this case the return of "def methodname(args); end" is ":methodname" which then gets passed to "private" and everything magically works.

</comment></comment></comment><comment><USER14> I wish they dedicate 2.2 and 2.3 only to performance improvements of the core and std lib.

<comment><USER15> Yes, JIT to Native Code, Better GC. We have yet to see any "substantial" improvement.

</comment></comment><comment><USER16> It's unclear from the intro summary, but I'm assuming from the lack of further explanation that refinements are just GONE, and not just "no longer experimental"?

<comment><USER17> Nope, by "no longer experimental" they are now officially in the core language. More information here: [http://rkh.im/ruby-2.1](http://rkh.im/ruby-2.1)

<comment><USER16> Ah, well that's something then. Thanks for the clarification.

</comment></comment></comment><comment><USER18> The most important feature for me is mandatory named args:

      def foo(bar:)

Named args with defaults solved one half of the problem we'd been solving with the "options hash"; now that's the second half. Can't wait!

</comment><comment><USER19> Is it possible to add optional typing to Ruby? Something similar to what Dart has.

Optional typing would be very useful for big Ruby apps.

<comment><USER20> I'd be intrigued to see how a ruby type checker would handle boolean expressions.

</comment></comment><comment><USER21> I've upgraded one of my Rails4 apps to 2.1.0-preview2. Everything seems to be working fine except pry-debugger.

What is it with the ruby debugger? It has always been problematic for me.

<comment><USER22> I believe the debugger is broken in Ruby-2.0, and you should use byebug instead.

There are pry wrappers for byebug - [https://github.com/deivid-rodriguez/pry- byebug](https://github.com/deivid-rodriguez/pry-byebug) \- maybe that'll work?

<comment><USER21> Yesss - that worked. Tip of the day - thanks a ton !!</comment></comment></comment>


Computer Scientists Are Astir After Baidu Team Is Barred from A.I. Competition - T-A http://www.nytimes.com/2015/06/04/technology/computer-scientists-are-astir-after-baidu-team-is-barred-from-ai-competition.html <comment><USER2> Some background:

This competition (the "Large Scale Visual Recognition Challenge" aka "ImageNet") isn't just some random competition. This is _the_ competition that gave rise to the recent explosion in interest in Neural Networks.

In recent years Google, Microsoft and Baidu have been one-upping each other to the point now where they are getting very close to better-than-human performance (ie, humans disagree with other humans more often than their systems disagree with the average human rating)[0].

Andrew Ng went to Baidu to start their team. I don't believe he is still involved in this challenge.

Baidu has been getting close and close to Google's performance. Every year so far Google has topped it at the right time, but Baidu has later passed the Google benchmark that year.

There were reasons to think that this could be the year they finally beat them. [1] is a story from January about the system that Baidu had built then.

Now this.

[0] [http://karpathy.github.io/2014/09/02/what-i-learned-from- com...](http://karpathy.github.io/2014/09/02/what-i-learned-from-competing- against-a-convnet-on-imagenet/)

[1] [https://gigaom.com/2015/01/14/baidu-has-built-a- supercompute...](https://gigaom.com/2015/01/14/baidu-has-built-a- supercomputer-for-deep-learning/)

<comment><USER3> They were just caught blatantly cheating.

This brings quite reasonable questions about their past performance.

<comment><USER4> > This brings quite reasonable questions about their past performance.

I don't think this second claim is necessarily derivable from your first. Yes, they were caught cheating... By making multiple submissions of their own systems in order to faster derive which is the most promising. IMO, the performance they have achieved is still theirs to claim, because in the end they did create a system that reaches that performance.

<comment><USER3> [http://en.m.wikipedia.org/wiki/Overfitting](http://en.m.wikipedia.org/wiki/Overfitting)

Ideally, your final test data set should be completely separate from training data. Because even limited exposure quickly renders that test data meaningless.

PS: In the end it's much like taking the same test with word for word identical questions a second time. Yes, it's a test of something, just not the intended subject matter.

<comment><USER5> It is hard to emphasize how bad overfitting on the test set is. Your cheating analogy is accurate, and cheating in science is a serious matter. If it is unintentional, you should have your work rejected or retracted. If it is intentional, you should have your work rejected or retracted and, frankly speaking, you should most likely leave the field.

Science is based on a huge deal of trust and violating that trust for your own short term gains is inexcusable. Even worse, any honest scientist will have find it more difficult to improve upon results that can be explained by overfitting.

A good short description on several issues that plague the field is "Clever Methods of Overfitting".

[http://hunch.net/?p=22](http://hunch.net/?p=22)

</comment></comment></comment></comment></comment><comment><USER6> Cheating is pretty entrenched in Chinese culture. Sometimes I even wonder if they consider it's cheating at all or if it's just cleverness and they should be rewarded for it. I believe they feel that way about infringing copyrights too: why should they follow someone else's rules that puts them at a disadvantage when they can instead demonstrate cleverness and break those rules?

There was an interesting story recently about how a disproportionate amount of Chinese students are expelled from US universities for cheating.

[http://blogs.wsj.com/chinarealtime/2015/05/29/u-s-schools- ex...](http://blogs.wsj.com/chinarealtime/2015/05/29/u-s-schools- expelled-8000-chinese-students-for-poor-grades-cheating/)

The problem is so entrenched that some students _rioted_ when they were prevented from cheating, because they felt that this put them at an unfair disadvantage compared to other schools where cheating was tolerated:

[http://www.telegraph.co.uk/news/worldnews/asia/china/1013239...](http://www.telegraph.co.uk/news/worldnews/asia/china/10132391/Riot- after-Chinese-teachers-try-to-stop-pupils-cheating.html)

<comment><USER7> If you look at it another way, it's not so bizarre. The person is retrieving their information from another's head. They're using a social storage mechanism. This works fine in the classroom, it works OK in the workplace. It works well in most situations. The only place it breaks down is during testing.

Because standard testing is only supposed to verify that you're storing the data locally. Standard testing is designed in a way that only helps "local storage" people, and alienates these users who get their information from their environment.

Open-book, open-note, and open-internet testing removes this barrier and levels the playing field for these "network storage" learners.

Or am I completely out of touch?

<comment><USER8> Why is it even bizarre? Chinese people cheat for the same reasons anyone would cheat. Being chinese and born in America, I actually found it really surprising the first time I met software engineers who refuse to download even one copy of pirated software.

Culturally we just feel significantly less guilty about it. Doesn't mean we aren't aware that it's wrong. Chinese people are fully aware of what's right and wrong and we choose to deliberately cheat. Whether that's a cultural tendency or genetic one is another story.

<comment><USER9> > Chinese people are fully aware of what's right and wrong and we choose to > deliberately cheat. Whether that's cultural tendency or genetic one is > another story.

What?! Genetic?! There is _very_ little, if _any_ evidence to support the idea that genetics has anything to do with the Chinese propensity to cheat...

<comment><USER8> There's no evidence at all. Even when attempting to do research on this, the political backlash could destroy any scientist's career. No official evidence will ever be collected because of this.

I only speak from personal experience. I know many chinese people both born here and born abroad. I am also chinese and born in the united states. I am telling you, honestly, from a purely anecdotal standpoint: I think there's a chance it's genetic.

Edit: Just to keep things from getting out of hand, and more balanced I want to state this fact: Statistically, it is far more likely for a serial killer to be a white caucasian male then it is for a serial killer to be of any other race. Do I think this is a cultural thing? No. I'm leaning towards genetics. But that's a purely anecdotal opinion as there's no evidence pointing in either direction.

<comment><USER10> Why would those things be genetic? If there were a "serial killer gene" or a "cheater gene", those traits should manifest themselves in all kinds of obvious differences in behavior - which we do not see. It's hard to imagine a protein causing such complex differences in behavior while affecting nothing else.

<comment><USER8> What's the other explanation then? Culture? Why don't we see higher rates of serial killers in other races born in the United States?

It's very possible for many differences in behavior between races to be genetic in origin, in fact it's the more logical hypothesis versus the alternative which states genetics doesn't influence behavioral differences between races.

Think about it. If genetics influences physical traits from height, skin color, facial features, and even athleticism, what black magic in this world makes it so that genetics doesn't even touch behavior or intelligence?

>It's hard to imagine a protein causing such complex differences in behavior while affecting nothing else.

It's impossible to logically deduce a conclusion from the bottom up. We simply currently don't have enough knowledge to know how proteins scaffold the entire human neural network. With highly limited knowledge, we can only look at the problem from the top down. That being: genetics is known to influence physical traits, therefore it is logical to conclude that it also influences mental traits.

<comment><USER9> > If genetics influences physical traits from height, skin color, facial > features, and even athleticism, what black magic in this world makes it so > that genetics doesn't even touch behavior or intelligence?

Because we have no evidence that such is the case on a culturally grouping level.

There is no genetic concept of "Chinese". It literally doesn't exist.

You're making this about the possibility of genetics impacting behavior, when the real issue is you thinking cultural boundaries exist in genetics. They don't.

<comment><USER8> There's no concept of humanity on the atomic level. It literally doesn't exist. One configuration or mishmash of atoms we call rocks are no different then the mishmash we call humans. Try, without using any high level concepts or groupings, to define what configuration of atoms signifies a rock and what configuration signifies a human.

If you go low enough on any topic the boundaries between categories become vague and the definitions become extremely complex. It's very hard to define what a human is in terms of atoms. The same goes for race, it's very hard to define, at the genetic level what is chinese, and what is not, but the category and boundary exists at all levels, and we can't ignore it.

I've heard of your argument before. They say that the delta in genetic differences between two people of different races is the same as the delta of two people, of the same race, therefore race doesn't exist. This argument is flawed. I believe the "genetic" definition of race is immensely more complex than simply the delta of genetic differences. Here's a more accurate definition: People of the same race have a higher probability of sharing certain genetic traits.

So let me redefine my argument in way you can understand. The people who we label as "chinese" who share similar physical/genetic traits, I believe will be more likely to also share a behavioral genetic trait that makes them more likely to cheat.

<comment><USER9> I'm sorry, you misunderstand -- the people we label as "Chinese" do _not_ share similar physical/genetic traits.

Common misconception that they do, but there is very little genetic consistency across cultural boundaries, and when such a thing does exist, it's quite noteworthy.

<comment><USER8> > the people we label as "Chinese" do not share similar physical/genetic > traits.

This statement is utterly and completely incorrect. It is a common myth in the social sciences.

Please read: [http://en.wikipedia.org/wiki/Race_and_genetics#Population_ge...](http://en.wikipedia.org/wiki/Race_and_genetics#Population_genetics)

first paragraph from above page: "The relationship between race and genetics is relevant to the controversy concerning race. In everyday life many societies classify populations into groups based on phenotypical traits and impressions of probable geographic ancestry and socio-economic status - <AUTHOR> are the groups we tend to call "races". Because the patterns of variation of human genetic traits are clinal, with a gradual change in trait frequency between population clusters, it is possible to statistically correlate clusters of physical traits with individual geographic ancestry. The frequencies of alleles tend to form clusters where populations live closely together and interact over periods of time. This is due to endogamy within kin groups and lineages or national, cultural or linguistic boundaries. This causes genetic clusters to correlate statistically with population groups when a number of alleles are evaluated. Different clines align around the different centers, resulting in more complex variations than those observed comparing continental groups."

In short it's saying genetic traits can be statistically correlated with population groups (race) but variations of traits that are different within population groups can actually be more complex than those observed when compared with people outside of their race.

This is literally exactly my argument. Supported by wikipedia at the very least.

<comment><USER9> I do not accept the given definition of race from this page, as it presumes the term "race" is in any way scientific or rigorously defined when in actuality it is not.

What we "tend to call" race is _not_ defined, despite this wiki page's attempt to do so.

<comment><USER8> This wiki page is the reflection of the general opinions of the scientific community. You can redefine any word to have any definition that fits your universe, but when communicating with other people, we must go with general consensus.

<comment><USER9> > This wiki page is the reflection of the general opinions of the scientific > community.

It isn't. The concept of "race" is not rigorously defined.

<comment><USER8> A word not having a rigorous definition does not make the concept non-existent among scientists. "Life" is not rigorously defined.

<comment><USER9> Life is very rigorously defined, however it's not unequivocal:

[http://en.wikipedia.org/wiki/Life#Definitions](http://en.wikipedia.org/wiki/Life#Definitions)

A word not having a rigorous definition means it cannot be discussed scientifically. Hence the actual problem of studying the existence of life, e.g. is a virus alive?

<comment><USER8> please note. Unequivocal and rigorous are synonyms.

[http://www.thesaurus.com/browse/unequivocal/4](http://www.thesaurus.com/browse/unequivocal/4)

>A word not having a rigorous definition means it cannot be discussed scientifically.

Life is discussed scientifically in many contexts yet it is not unequivocally or rigorously defined. In fact there's an entire field based on the study of life. It's called biology, aka the study of life. If a scientific field can stem from a word that does not have a rigorous or unequivocal definition, then it can be discussed scientifically.

<comment><USER8> @dimino

I'm getting pretty tired too. You choose not to accept the facts even when a scientific description proving my point is thrown in your face. Ideas need evidence for support, you have presented me with ideas, but no evidence.

The folks in the field are in agreement with me, (see the old wikipedia link I sent you). You got nothing, only empty claims.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> In a globalized world, institutions need to tighten up their safeguards against cheating, fraud and dishonesty. In much of the world, cultural attitudes toward cheating are a lot more relaxed than some of us presume.

There was a story about Indian students' rampant cheating at US colleges, that was presumably flag-killed off HN despite the horror stories that were emerging from academics. I've read about Australian universities basically selling degrees to foreigners who can't speak English and get admitted fraudulently. There is a currrent story about fraudulent admissions to US colleges by Chinese students as a result of massive SAT cheating. I'm not picking on Asians: Switzerland seems to have its fair share of scoundrels, as the FIFA scandal reminds us.

We need to acknowledge that gloablization sometimes brings unwanted side- effects and deal with these head-on.

<comment><USER12> In my opinion, tightening up the safeguards isn't the right approach. Rather, institutions, particularly educational institutions, need to fulfill their obligation to inculcate the right values in people. Cheating and petty corruption culture is an existential threat and the solution isn't to catch the cheaters, its acculturate them to follow the rules.

Educational institutions, however, have totally abandoned this obligation. Shocking cheating behavior will merit just a note in someone's record, if administrators are even willing to take it that far. Cheating isn't publicized, shamed, and punished in the way necessary to have any impact on the cheaters' values. Instead, everyone gets to save face.

<comment><USER13> I totally agree with your opinion here, and especially that "cheating and petty corruption" \- and the breakdown of public trust that follows - is an _actual_ existential threat, something that may ultimately lead to a collapse of technological civilization. The current trend of trying to figure out trustless solutions for everything actually worries me.

<comment><USER14> _The current trend of trying to figure out trustless solutions for everything actually worries me._

If you're worried about breakdown in the face of loss of trust, then these sorts of solutions seem like very important things to be looking at. Do we have any good ideas about how to create trust at the institutional level?

I agree with your concerns. I think they are representative of a broader theme - the acceleration of technical change seems like it will eventually (if it hasn't already) bring us to a point where the rate of cultural change is too slow to catch up to properly adapt to what is now possible.

</comment></comment></comment></comment><comment><USER15> Who wrote that headline. It makes it sound like they were barred for no reason and the community is mad. A more accurate headline would be "Baidu Team barred from A.I. Competition for cheating". Its like they are trying not to get censored or something.

</comment><comment><USER16> As a former glider pilot I had serious trouble parsing the headline because of the all caps and this: [http://en.wikipedia.org/wiki/Grob_G102_Astir](http://en.wikipedia.org/wiki/Grob_G102_Astir)

<comment><USER17> Astir (αστήρ) means "star" in greek. I imagine that's where the G102 got its name from!

</comment></comment><comment><USER18> One important thing to recognize here is that this isn't just unfair play (in the sense of trying to win a competition by submitting multiple entries). It's a loss for all players. The point of having distinct test and training sets, and not training on the test set, is to ensure that your system can generalize (IE, work on things it wasn't trained on). If you don't do that, you're not making progress, just memorizing examples. Example memorization is fine on its own, but it's not really a true improvement.

</comment><comment><USER19> Just some anectdotal evidence, but at work I had to implement a clone of our Google Maps application in Baidu Maps for our customers in China. The only reason I was able to use their API was because they copied Google's interfaces, in some cases, word for word.

<comment><USER20> Evidence for what? Sounds like a good thing that they used the same API as Google for their maps to me. Like how Google used Sun's API for Java.

<comment><USER21> Google uses the APIs from Java 6 and back for the "boring" bits. Android Java is the de facto client Java today because Google abandoned the UI APIs in Java 6 and created their own. Sun's, then Oracle's argument is that Google is "harming" Java by breaking the standard.

<comment><USER22> I think there is subtile difference. Java was, is and shall be a standard. To say you had a Java implementation meant that you supported the Java APIs. It was a legal definition. Google had Java-like runtime that they promoted as a Java runtime. This led to the question of API protections and reasonableness.

Using Google's API for your own competitive product API is different. Google has not set a standard for map APIs. At least not officially. It might be THE standard for maps, but that is neither here nor there. At this point you could call it cheating. They took intellectual property and appropriated it for a clone. Plus side, we now have an implied standard. Downside, theft of intellectual property.

<comment><USER21> You are mixing a number of issues here:

1\. Google Play Services APIs: These are optional, proprietary APIs. You don't have to use them, and you have to treat them as optional if you want to operate across Google, Amazon, and other AOSP-derived Androids. What's the issue here? They're like any other proprietary API.

2\. "Breaking" the "Java standard:" Google "appropriated," via a permissive open source license, an open source implementation of some Java base classes, and the (not protectable) syntax of the Java language. They added support for their own remote API feature and a bunch of APIs, mainly to make a usable UI system. The result runs on a runtime Google devised, using a bytecode Google devised. Where is the intellectual property theft?

</comment></comment></comment></comment></comment><comment><USER23> It is odd that there is such a significant cultural difference. On the other hand, the difference in culture may be more subtle than we realize.

For this test, they should have had an automated way to enforce that rule, if possible.

Look at the actual levels of piracy in the US versus China. In the US, you will get some people who say they are worried it might be wrong or that they will get caught. But most people will say something like 'watching this free stream doesn't hurt anyone'.

In the US, we have no problem with high end shoes that look similar to one another, or low-end similar to high-end. But there is a subtle distinction -- there must be some way to claim that this is not copying the other. But maybe it more often comes down to taking a different attitude towards the same thing. E.g. "cheap Chinese knock-offs" versus "inexpensive sensible alternative to overpriced designer brand".

Its also perfectly fine in general for companies to copy a successful business model. But we insist that there must be some distinction. However the difference between these companies may only be slightly greater or mainly surface-level, and so when you get down to the fine analysis, I thinj the real difference between the cultures is smaller than people want to admit.

</comment><comment><USER24> Why is the competition designed in such a way that it is an advantage for a competitor to "run test versions of their programs twice weekly ahead of a final submission"?

How does it work? You submit your classifier to some server and it is run against what? The data set that determines your final score - hopefully not.

<comment><USER25> > Why is the competition designed in such a way that it is an advantage

See my comment & links at the bottom of this page. For some reason HN always puts my comments near the bottom, even when they're getting many upvotes. (I'm guessing that HN has a comment-editing penalty from editing my comments too many times, but I'm not sure, or maybe it's from times I must have insulted the mods here.)

Edit:

I constantly get "submitting too fast", so I'll reply here.

Thanks for the info kefka. Yeah, I'll happily agree with that linked comment, that the childish crude moderation techniques used here have been hurting a once great community. It was the exact same scenario for me as for that guy. Seems I got rank-banned immediately after confronting dang about the trends of excessive downvoting and increasing side-project criticizing on HN.

<comment><USER24> Is it correctly understood that in "Large Scale Visual Recognition Challenge" the competitor's biweekly runs ahead of the final submission is against the set that actually determines the final result of the contest?!

<comment><USER26> That's what I was wondering. If so, submitting many test runs makes it trivial to provide a submission that works perfectly _against the test data_ specifically.

<comment><USER27> But the test set has 50,000 images (across 1000 categories). It's not that easy to just "try many times" to get it right.

<comment><USER26> ...for some definitions of "trivial" and "perfect". At this level, I suspect even a small advantage would result in winning the contest, which is the point here.

</comment></comment></comment></comment></comment></comment><comment><USER25> Background:

The technique is actually pretty fascinating. This is something that's been well understood by the cryptography community for decades, but is somehow just recently being fully appreciated by the ML community. See here:

[http://blog.mrtz.org/2015/03/09/competition.html](http://blog.mrtz.org/2015/03/09/competition.html)

[https://www.kaggle.com/c/restaurant-revenue- prediction/forum...](https://www.kaggle.com/c/restaurant-revenue- prediction/forums/t/13950/our-perfect-submission)

Summary -

Submitting guesses to a system that gives you back scores for your guesses, will quickly leak out enough information that you can reverse engineer a huge number of hidden numbers/labels in surprisingly few iterations, e.g. 700 iterations to covertly extract 10,000+ real numbers with high precision. This surprisingly rapid convergence is a bit reminiscent of the birthday paradox.

Further, this not only lets you win the against the "test" dataset, as apposed to the final "validation" set, but this allows you to significantly increase the data available to you to train your model on, since now you can train your model against both the "test" and "training" datasets.

Layman summary -

ML breaks datasets into 3 partitions "test", "train", and "validation". In cases where they're evenly split, this technique can double the training data you have access to, which is a massive advantage in ML competitions where scores differ by tiny amounts.

Moral judgement -

My opinion, this moral argument is misdirecting the attention from where it needs to be. Yes, it's bad what occurred here. But at this point, in 2015, and with tools readily available to crack this problem effortlessly, it's inexcusable for contests to allow so many scoring reports against their validation sets anymore. It's no longer a question of whether contestants will do it, but how many of them will. We'd might as well just let people self- report their scores on an honor system, if we're going to be this overly trusting.

Try creating a contest system like this in the cryptography field any time in the past 3 decades and you'd be insulted and laughed out. Allowing so many scoring reports against the validation set is fundamentally flawed. The only solution is to _globally_ limit calls to the scoring api.

Another proposed solution -

Allowing everyone to see everyone else's guesses & resulting scores against the "test" set, so that everyone is on equal ground for reverse engineering the "test" set, and then globally limiting the number of scoring attempts so that the test set isn't reverse engineered too significantly.

Overfitting the "validation" set actually is not a problem either way, because none of these contests are dumb enough to let anyone score against the validation set at all until the contest submission deadline is over.

<comment><USER28> Academia is basically "self-reporting on the honor system". It works generally but there are lots of holes. Ultimately, "trust but verify" is necessary to avoid getting caught in a wave of hype, or at least having someone in your own "circle of trust" say it works. This system leads naturally to elitism and a bunch of other problems which are seen in academia, but it seems better than the current alternatives to me given the current rabid focus on exact percentage score instead of quality/utility of an idea.

The "right way" to do it is test once only _per model /paper_. If you are interested there are a huge number of sneaky ways overfitting can happen in ML [1]. Also interesting that you too see crypto and ML as related - I see them as opposites of the same coin. One tries to pull signal out of noise, the other tries to bury the signal _in_ noise... but special noise.

[1] [http://hunch.net/?p=22](http://hunch.net/?p=22)

</comment></comment><comment><USER29> Is training of the Google AI the reason that reCAPTCHA is now showing pictures and asking users to select all images of a certain kind?

</comment><comment><USER30> This is clearly cheating.

HOWEVER, the goal of these contests should be to promote the most accurate and powerful image recognition algorithms that will transform the world as we know it. Limiting access to training data makes it more difficult to test changes to an algorithm. These rules do not make sense to me, and I would advocate against them.

<comment><USER31> Two things - firstly you need to understand the concept of overfitting : [http://en.wikipedia.org/wiki/Overfitting](http://en.wikipedia.org/wiki/Overfitting) . If teams were allowed to train on the full dataset, it would be possible to get a 100% score, yet still not have a model that was useful on any images that were not in the training set. Furthermore, if you allowed infinite submissions, teams could just train a million models with slightly different hyperparameters, and submit the one that did best (which may be what Baidu was trying to do here). This is a problem because now there is the possibility that you are overfitting the test data - there would be no way to tell if the accuracy generalised to other images without coming up with more labelled data, i.e. making a new test set.

Second of all, cross validation : [http://en.wikipedia.org/wiki/Cross- validation_%28statistics%...](http://en.wikipedia.org/wiki/Cross- validation_%28statistics%29). You don't HAVE to submit to the test set to get a feel of how well your model is performing. On datasets this large, cross validation should be effective, if more time consuming method (unless your model is extremely unstable).

<comment><USER30> Was there a training set made available and distributed? I got the impression that there was not.

<comment><USER31> Yeah the training set is called Imagenet, it's widely used in research.

</comment></comment></comment></comment><comment><USER32> Funny for a moment I thought some one submitted a headline with an Indian word in it. Asthir in Hindi/Bengali means unstable, not stationary or worked up (remotely connected to Astir?)

<comment><USER33> My thought is that the "a-" prefix in this sense probably comes from the Germanic use "to show a state, condition, or manner"; [http://en.wiktionary.org/wiki/a-#Etymology_2](http://en.wiktionary.org/wiki/a-#Etymology_2) . It's no longer used to make new words, but the "a-" form remains in many words, including abloom, aflame, and abuzz.

The first known use of the word "astir" is from 1765, says Merriam-Webster at [http://www.merriam-webster.com/dictionary/astir](http://www.merriam- webster.com/dictionary/astir) . This is what I would expect if "stir" was a word, "a-" was a possible mechanism to create new words, and eventually people started to use "astir.

Etymonline gives a first known use as 1823: [http://www.etymonline.com/index.php?allowed_in_frame=0&searc...](http://www.etymonline.com/index.php?allowed_in_frame=0&search=astir&searchmode=none) :

> "up and about," 1823, from phrase on the stir, or from Scottish asteer; from > stir. Old English had astyrian, which yielded Middle English ben astired "be > stirred up, excited, aroused."

The root "stir" has a longer heritage. Etymonline at [http://www.etymonline.com/index.php?term=stir&allowed_in_fra...](http://www.etymonline.com/index.php?term=stir&allowed_in_frame=0) says:

> Old English styrian "to stir, move; rouse, agitate, incite, urge" > (transitive and intransitive), from Proto-Germanic * sturjan (cognates: > Middle Dutch stoeren, Dutch storen "to disturb," Old High German storan "to > scatter, destroy," German stören "to disturb"), from PIE * (s)twer- (1) "to > turn, whirl" (see storm (n.)).

Hindi is another descendant of Proto Indo-European, so that may be where there's a connection to "asthir". However, do bear in mind that the surface similarity is false - astyrian would be linguistically closer to the Hindi than astir.

<comment><USER32> Thanks for the informative post! However I think asthir may be a false friend in this case. The a- prefix is used to negate sthir in asthir. Sthir in Indian languages means stable, stationary, motionless etc. Kind of like the a- prefix in English sometimes (social, asocial etc.)

But it is not uncommon for word meanings to change in such a way that they take on a meaning exactly opposite of what they used to mean (see for example the meaning of the word nice: [http://www.etymonline.com/index.php?term=nice&allowed_in_fra...](http://www.etymonline.com/index.php?term=nice&allowed_in_frame=0)). I wonder if there is a name for this phenomenon.

<comment><USER33> Awesome! [http://etymonline.com/index.php?term=awe&allowed_in_frame=0](http://etymonline.com/index.php?term=awe&allowed_in_frame=0) :)

</comment></comment></comment></comment><comment><USER34> In science and technology cheating is often self-correcting. Your result is not reproducible or your product defective if you cheat too much. Everyone will know.

</comment><comment><USER35> The PLA appears to have taken down the NYT story.

<comment><USER36> Huh? Link works fine for me.

<comment><USER37> PLA = [https://en.wikipedia.org/wiki/People%27s_Liberation_Army](https://en.wikipedia.org/wiki/People%27s_Liberation_Army)

I think that was intended to humorously imply that the link appears down when accessed from China.

<comment><USER24> NY Times has been blocked for years in China.

<comment><USER38> For a couple of years. WSJ has been blocked for less than a year. CNN is still up, though I guess its just a matter of time.

</comment></comment></comment></comment></comment><comment><USER39> I'm not an expert, so apologies if my question is stupid.

Why can't the competition have the same test data each week across all participants? So that no matter how many accounts you create, you will train with the same images everyone else gets to train with.

</comment><comment><USER40> I began watching Andrew Ng's course just this morning. I had also watched some part of the courses by Pedro Domingo at the Washington University a few months ago.

Having never been employed but having coded for 8, I wish I could get myself educated in this field.

</comment><comment><USER41> A major company spending so much, even risk their reputation, on a competition in a field where computer already achieved better than human performance. I start to feel that maybe the industry should start looking more on other new application of deep neural network (e.g. reinforce learning by deepmind). There is still a long way to go before we can achieve a thinking machine.

<comment><USER36> We only recently (and barely) passed the performance level of a human. There's still a lot to be gained from competitions like this.

</comment></comment><comment><USER42> Another solution would have been to allow other researchers unlimited access to the test server as well.

<comment><USER43> That would turn the competition into an exercise in overfitting.

<comment><USER25> It wouldn't, because final scores are only evaluated against the "validation" set.

As for turning current contests into an exercise in overfitting the "test" set - we already reached that point long ago. Test vs validation scores often diverge wildly in these contests.

Edit - Replying to arnsholt:

Completely true. The huge problem I see, is that all the classic NLP tagging corpuses are created from the very narrow domain of news articles, and a few good corpuses now appearing for biology texts, and that's about it. Want to do, e.g. NER for product reviews or chat logs? - Incredibly bad results. There's a huge corpus problem in NLP today.

<comment><USER44> Not to mention out-of-domain performance.

I'm not familiar with computer vision, but in NLP taggers are hovering around human-level performance, and parsers are quickly approaching that level. But if you take a state-of-the-art system and test it on a slightly different corpus (even something as simple as text from the same newspaper, but a year later!) performance drops by a lot.

</comment></comment></comment></comment><comment><USER45> What's more important, winning a competition or improving AI for the entire world? Do the ends justify the means? I Don't think we should be too hard on Baidu, considering they are attempting to improve their algorithm for the interest of humanity (or evil AI that will bring an end to mankind, depending on how you look at it).

<comment><USER46> > I Don't think we should be too hard on Baidu, considering they are > attempting to improve their algorithm for the interest of humanity

They are not improving the algorithm for the interest of humanity. They submitted and obtained test results much more frequently than allowed. With that information, they can tune parameters to more closely fit the test data.

Basically it is like scoring higher on an exam where some of the questions have been leaked. It does not suggest you now have a better understanding of the subject.

<comment><USER30> If they are building an algorithm which overfits to training data, their algorithm will lose performance on in the actual validation test.

It is NOT like looking at leaked answers. It's like re-reading the textbook more than everyone else. Definitely an unfair advantage however.

<comment><USER47> You misunderstand. They're using repeated attempts on the test set to improve their network. That is equivalent to training on the test set and that is unequivocably cheating.</comment></comment></comment></comment>


How to write cross-platform code - <AUTHOR> http://blog.backblaze.com/2008/12/15/10-rules-for-how-to-write-cross-platform-code <comment><USER2> I would say building for all platforms to verify you didn't break anything is the _perfect_ job for a continuous build and integration server.

Simply don't accept the changes if the code doesn't build cleanly and passes the test suite on all platforms.

Why bother the programmer to remember to do this?

</comment><comment><USER3> _By this I mean that once a new feature is implemented and tested on one platform (let's say Windows), the programmer checks it into Subversion, then IMMEDIATELY checks it out on both Macintosh and Linux and compiles it._

Yuck. Here's how we do it:

      $ git commit
      $ git push origin HEAD:refs/for/master

The commit shows up in gerrit for code review. Immediately the buildbots (one for each platform) notice the change and attempt to compile it. The results are reported back to gerrit and a failed compile blocks the change from being submitted.

Assuming the change compiles cleanly and passes code review, it can then merged to the master branch. Importantly, broken code doesn't get checked-in in the first place.

Now, we happen to use git and gerrit and buildbot, but any decent system should have code review and a build farm that allow changes to be tested and reviewed before they are checked in.

If your developers aren't following some procedure (e.g., test compile on every platform), consider it may be because there is too much friction. Instead of firing the developers, ask why they aren't following it. Maybe other developers are impacted too and just have a higher tolerance for menial tasks, but those tasks are still costing the organization.

<http://code.google.com/p/gerrit/>

<http://buildbot.net/trac>

[http://github.com/jaysoffian/buildbot/commit/e4c0d458374b9a0...](http://github.com/jaysoffian/buildbot/commit/e4c0d458374b9a0ce34c1e9e1c9813015b2e6add)

And if you happen to use hudson:

<http://wiki.hudson-ci.org/display/HUDSON/Gerrit+Plugin>

<http://wiki.hudson-ci.org/display/HUDSON/Gerrit+Trigger>

<comment><USER4> Agreed. Using git or mercurial would get around this. I worked for nearly a decade on cross-platform code that followed a similar methodology.

Our rule was the code most compile w/o warnings on 3 dissimilar platforms _before_ you could commit it. We were able to do this easily because we had network-mounted home directories, so when you logged into another machine, our development tree was automatically mounted. This made it easy to compile without checking in, then checking out.

</comment></comment><comment><USER5> The thing that I most favor in cross-platform programming is the code quality. Heck, I might almost say that even if you only intend to write a program for Linux, make sure it compiles and works on Windows and OS X too. (Well, _almost_.)

Somehow, almost magically, if you have two or preferably three platforms the relevant code abstracts itself nicely out of what is the low-end mess composed of platform specific concepts. I don't mean merely thinking in terms of FooHandles and BarStreams instead of "file descriptors" but the whole process of thinking that happens when you have to separate, in your brain, that what is "platform level code" and what is "problem level code".

</comment><comment><USER6> It's worth pointing out that even 15 years after the Netscape IPO, _it's not possible to write cross-platform networking code for Windows and Unix using C or C++ without using a 3rd party library_!

On top of basic socket support, you then have to add support for the common protocols (HTTP, SMTP etc.). I think this alone explains the popularity of networking "batteries included" dynamic languages like Python, and Java's status as the de-facto enterprise standard for network programming. Standardized languages like C and C++ move too slowly for the internet world.

</comment><comment><USER7> Those are all v sensible rules. You could give everyone an OSX desktop with Windows and Linux in VirtualBox, using a shared folder, then you could do the test build locally without having to bounce it off version control in the middle.

<comment><USER8> I had a job with exactly the dev environment you're describing (plus an HPUX box and a Solaris box), and it worked quite well. Sadly we were stuck using Visual Studio 6 in Parallels for development, but otherwise it's not a bad idea.

</comment></comment><comment><USER9> In my work, I end up writing code that compiles on a lot more platforms than that. These are my thoughts, I suspect I work at a much lower level than the author, so the disagreements I have are likely just because of that:

1) Don't use C++. Not all platforms have a c++ compiler.

2) Get your build system right. Don't let it get in your way.

3) Avoid #ifdefs. These don't scale particularly well as you increase the number of platforms. In the example in the article with the file size function, there would be a completely separate file with this function in it for each different version. The build system will compile the correct one.

4) Use plain boring ANSI C.

5) Developers don't have to compile on every platform (doesn't scale when you support bazillions of platforms and toolchains). If something is broken on your platform you fix it. If someone breaks something, you politely let them know what the problem was.

6) Avoid undefined behaviour. This is harder than you might think. Different compilers will be able to do different optimisations, and they will take advantage of different bits of undefined behaviour. Classic problems include overflowing a signed integer, right shifts on negative numbers, modulo on a negative number, type punning for endian checks, etc.

7) I agree with the point about standard 'C' types. However, there are a few extra bits. First, know what the types actually mean (char >= 8-bit, short/int >= 16-bit, long => 32-bit). Don't make any assumptions that aren't in the C spec. You might then want to put typedefs on top of these types to give them a bit more meaning.

8) Don't assume there is a FPU. Be prepared to do fractional maths in fixed point. There will always be float/double types, but they will be so dog slow on some platforms, and the memory cost of bringing in the library for them will be bad.

9) I disagree with UTF-8 for all APIs. Define a string abstraction, so on platforms (or rather, "on windows") you can have UTF-16 strings passed in and operated on without a conversion, and everywhere else you can use UTF-8. Avoid paying the conversion penalty on every operation.

10) Think about memory management. When memory is allocated, the malloc-like thing that is doing it should know what the memory is actually for. On some platforms, you will want to put different allocations into different spots in memory (this is also a nice place for another abstraction, since on some machines there won't be any different types of memory).

11) Don't avoid undefined behaviour by writing a single wrapper function (e.g. "safe signed integer add") and using it everywhere. The behaviour is typically undefined for a good reason (since different platforms will natively want to do it different ways). In the integer add example, you might want to assert that it never overflows, wrap around on overflow, truncate on overflow, return an error if it overflows, etc. Each of these things will have their spot, so don't bunch them all together.

12) Code that might want to be implemented in assembly should live in a small .c files. Then when you go to write it, you can just do it bit by bit (and again, use the build system to tell it how to build it, don't hack it with the preprocessor).

<comment><USER10> what sort of programs are you working on that must run on platforms that lack a C++ compiler and need to be portable? (I am legitimately curious)

<comment><USER9> Tiny embedded things from people you've never heard of.

Often, I won't even get to see the compiler for the platform, I'll just write the code, send it to a completely external person, and say "this should compile and run on your platform, I can help if it doesn't".

The core of what I do is audio DSP, so it has to be memory/cycle efficient too, otherwise no one wants to use it.

But as soon as you start developing for these platforms, you then need to build testing tools which also work there, and all the other fun stuff that goes along with it, so it ends up being a lot more than just the core DSP stuff (which is where most of the platform specific stuff comes for me actually, as the core DSP stuff is just memory in memory out type stuff with no side effects).

<comment><USER11> Check this out: [http://www.parasoft.com/jsp/products/article.jsp?articleId=3...](http://www.parasoft.com/jsp/products/article.jsp?articleId=3083)

</comment></comment></comment></comment><comment><USER12> They aren't entirely bad suggestions, but it can be improved somewhat, depending on taste:

>Rule #4: Use only built in #ifdef compiler flags, do not invent your own

Actually you can separate the source code into src/linux src/w32 src/mac -- but that is just my preference.

>Rule #9: Require all programmers to compile on all platforms

No no no -- build has to take no more than a make command. Anything else is just domed to failure.

<comment><USER13> > _Actually you can separate the source code into src/linux src/w32 src/mac -- > but that is just my preference._

IMO the author suggests to develop in a single source tree which compiles on every platform for very good reasons, in particular to minimize duplication and being able to update a single method in a single file to work on all platforms. Your suggested multi-directory approach requires to duplicate most os-specific methods and classes. Depending on the situation one or the other approach might be better, but I think the author has a point in many of them.

> _No no no -- build has to take no more than a make command. Anything else is > just domed to failure._

Building with just a "make" should be fine on Linux, but for anyone who downloads the source code to a Windows/VS or OSX/XCode machine, it's great being able to just open the project in the standard editor and compile it.

<comment><USER14> Our 'make' vs Visual Studio issue is: to add a new module in Visual Studio you add it to the project. VS automatically getting included in many makefile scenarios. Which causes a common bug: a _WIN32-only module gets built on the wrong platform because the developer forgets to put the module in the makefile exclusion list.

Small matter, easily detected and fixed in the first build, but would be nice to automate somehow.

<comment><USER15> CMake (or perhaps Scons, but I have no experience with that) might be useful for you. Although going through the motions of writing build scripts for you build system might not be something you want to endure.

In any case, I used CMake in one shop I worked in and it was nice to have the build files all managed in one place. We were able to manage builds for Windows, Linux, Mac, and Solaris.

</comment></comment></comment></comment><comment><USER16> An #ifdef _WIN32 is a platform ifdef. Especially to cross-compile Unix flavors, it can help to use capability ifdefs, such as #ifdef HAVE_VSNPRINTF. This approach replaces if-then-else logic on platforms in the source with tests in the configure (or CMake) script.

Nice article. Is the GoF Abstract Factory pattern not popular for #5, "developing cross-platform base libraries?"

</comment><comment><USER17> Care to elaborate on why you chose C/C++ instead of Java for the backup client?

<comment><USER18> He doesn't have big enough patent warchest?

</comment></comment><comment><USER19> As console/pc video game programmer I can just say:

Make it compilable on your target platforms: PC, Wii, PS3, Xbox, what other is needed.

Don't care about other platforms, until they arrive.

Instead of #ifdef, try to find the pattern in your code, and use select macroses such as:

PC_WIIXBOXPS3(pcValue,consoleValue) or WII_XBOXPS3_PC(wiiValue,xboxOrPs3value,pcValue)

wrap align, threadlocal, etc. as macro-define.

The UNICODE advise is wrong. These system do not have even working locales, so don't trust strupr(), strlwr(), toupper() - etc. to work correctly for your localized game.

</comment><comment><USER20> Check out LiveCode (formerly known as Runtime Revolution), it is a cross platform language + IDE. It is a modern descendant of Apple HyperCard that is able to deploy Desktop Apps for Macs, Linux, Windows. Has an engine like PHP for Web Application development and soon will be available for iOS.

I've been using it for the past six years to do most of my development and am quite happy. I am able to develop on my macs and deploy anywhere.

<comment><USER20> sorry forgot the url for LiveCode <http://www.runrev.com>

</comment></comment><comment><USER21> Lots of good advice, but watch out for Unicode. IIRC, you can have illegal surrogate pairs in NTFS filenames (which are a kind of naive UTF-16), which renders them unconvertible to UTF-8. Certainly an edge case, but that's where bugs come from...

(BTW, what really shocked me about this was that it means some file paths can't be turned into file: URLs! I had assumed you could always do that.)

</comment><comment><USER22> >Rule #9: Require all programmers to compile on all platforms

... and have a build server do this automatically & email the results. Manually building for each platform if there are more than two seems like quite a hassle and you know programmers will skip over it for minor changes.

</comment><comment><USER23> The hardest part of cross-platform stuff is the GUI, and the solution given here is "don't make it cross-platform."

Which is really the right answer. But it makes the actual cross-platform stuff trivial in many apps.

</comment><comment><USER24> this doesn't sound realistic

<comment><USER25> The down-votes on parent are misplaced. Why not enlighten with comments about how it -is- realistic?

<comment><AUTHOR> Because he/she doesn't explain why it is not realistic, his/her comment doesn't add any value to the conversation.</comment></comment></comment>


Show HN: Firewall – Extreme Robocall Blocker - <AUTHOR> https://apps.apple.com/us/app/firewall-robo-call-blocker/id1447702878?ls=1 <comment><AUTHOR> From the team that made Burner, an iOS app that screens out all calls, allowing only a user-designated whitelist to ring through.

</comment><comment><USER2> Looks fantastic! Could this be made available in the Canadian App Store by chance?

<comment><USER3> I'm the CTO at ad-hoc labs. Thanks for the feedback! We do plan to make Firewall available in the Canadian app store soon. We have to go through the Canadian wireless carriers and build out the support for them one at a time.</comment></comment>


A message from George Orwell – your digital rights - <AUTHOR> http://www.youtube.com/watch?v=c4EEa0HAqzQ <comment><AUTHOR> The words of George Torwell:

Tor is peace. Prism is slavery. Internet is strength.

This is what a Tor supporter looks like :)</comment>


Async Rails 3 stack - <AUTHOR> http://github.com/igrigorik/async-rails

<comment><AUTHOR> Since this has come up a few times already: ab -c 5 -n 10 <http://127.0.0.1:3000/widgets/http>

That's not a benchmark, it's a proof of concept: [http://github.com/igrigorik/async- rails/blob/master/app/cont...](http://github.com/igrigorik/async- rails/blob/master/app/controllers/widgets_controller.rb#L3)

/widgets/http makes an http request to /widgets, which does a 1s sleep query inside of mysql. All of this is happening on the same reactor loop, and finishes with ~2s, aka, the sleeps are not blocking the reactor and hence the "async stack".</comment>


Mysterious Markings May Hold Clues to Origin of Writing - <AUTHOR> http://news.nationalgeographic.com/2016/05/cave-art-ice-age-paleolithic-writing-first-signs/ <comment><USER2> I was fortunate in 2013 to visit cave painting sites in SW France and NE Spain. It is a moving experience to see images made tens of thousands of years ago, especially the "negative hand" pictures made by holding a hand up to a wall and blowing pigment across it, leaving a hand-shaped hole in the stain on the wall. One cave had dozens, from adults and children, in one spot.

The Nat'l Geo article doesn't mention a glyph that is common in the caves: the "tectiform", which looks a bit like a sign for a picnic shelter in a state park. It shows up in many caves and some people think that it may have been a tribal identifer.</comment>


Mozilla Server Side TLS – Recommended Configurations - <AUTHOR> https://wiki.mozilla.org/Security/Server_Side_TLS#Recommended_configurations <comment><USER2> Re-posting my comment from my submission of this, here's a summary of the major changes from the last update (which was published 2.5 years ago):

\- TLS 1.3 has been recommended across the board.

\- The "Modern" configuration is TLS 1.3-only.

\- The "Intermediate" configuration is TLS 1.3 and 1.2-only.

\- The "Intermediate" configuration now only uses AEAD ciphersuites with PFS. No more 3DES, CBC, SHA-1, or static RSA key exchange.

\- ECDSA certificates (using NIST P-256) are now recommended for the "Intermediate" configuration (in addition to the "Modern" configuration).

\- Servers now respect the client's cipher preferences with the "Modern" and "Intermediate" configurations as all enabled ciphers should provide sufficient security. This allows clients without AES hardware acceleration to prefer AES-128 or ChaCha20 over AES-256 and vice-versa.

\- The "Old" configuration drops SSLv3 and some uncommon ciphers (CAMELLIA, SEED, DSS). This loses support for IE 6 on Windows XP, bumping the minimum up to IE 8.

\- X25519 is now the recommended curve for key exchange (followed by NIST P-256 and P-384). NIST P-521 is no longer recommended as it doesn't provide any major security benefit over the other curves, has less widespread support, and is slower.

</comment><comment><USER3> Despite the title of this submission, the wiki page actually is recommending the "Intermediate" configuration by default, which also includes TLS 1.2

In practice, how much of a compatibility hit is there between the Intermediate and the Modern configurations?

<comment><USER4> Indeed the title is wrong though I can see how someone could get confused. These are all "recommend" configurations depending on how bleeding edge (or not) you want to be. But intermediate is the recommended of the recommended configurations.

<comment><USER4> The submission title has now been changed to better reflect the content so my comment no longer applies.

</comment></comment></comment><comment><USER4> On the client side I've been living with my browser set to accepting a minimum of v1.2 without issue for a long while, aside from the odd random page found in search but that's no loss.

I just recently tried bumping it up to the next version but so many sites don't support v1.3 yet, including HN.

<comment><USER5> > so many sites don't support v1.3 yet, including HN.

That's not surprising. Distributions like Debian and Ubuntu do not have a recent enough version of OpenSSL in their stable package repository yet.

<comment><USER6> Ubuntu 18.04, the newest LTS, recently backported OpenSSL 1.1.1 to its stable package repository.

Because of this my personal webserver, running nginx on Ubuntu 18.04, started offering TLS 1.3 without any manual action on my part, because the server is configured to auto-apply updates from these repositories.

</comment></comment></comment><comment><USER7> A small nit about Curve25519 vs X25519. The name of the curve is Curve25519, while the name of Diffie Hellman function utilizing the curve is X25519. Same with Curve448 and X448.

That's why the line `TLS curves: X25519, prime256v1, secp384r1` should be `TLS curves: Curve25519, [...] `

I'd impose my pedantry on the Mozilla wiki but it's not accepting account registrations, I guess understandably.

<comment><USER8> X25519 is the mechanism for using Curve25519. Software that uses OpenSSL all uses "X25519" and so it would be needlessly confusing to use other verbiage despite being more technically correct.

</comment></comment><comment><USER9> What is the advantage of dropping support for TLS 1.0 and 1.1 (especially if the server has TLS_FALLBACK_SCSV)?

Anyone who visits with a client that supports TLS 1.2 will just use that, while any client that can only deal with <1.2 will still be supported. There are various reasons why someone may not be able to use 1.2†, so what use is abandoning them?

† I'm personally still using NetNewsWire 3 because I like how it works compared to anything else out there that I've tried (since the last time I did a comparison), but it is compiled against OpenSSL 0.9.8, and so no TLS 1.2 for me.

<comment><USER10> It's prohibited from usage by various regulations, like PCI-DSS, so you simple have to turn it off.

There are a few vulnerabilities like CRIME or BEAST with variable impacts, some can be patched. It's very poorly explained so good luck understanding any of that.

There are a few old algorithms that can be decrypted with current computing resources, negating the point of encryption. You have to disable DES, RC4 and NULL.

I personally think the whole TLS 1.0 depreciation is a mess that's very representative of why cyber security is a mess and will never improve. A huge cargo cult telling to turn it off because it's broken, ignoring that it's breaking a lot of clients, and without a single explanation of what's the problems or how to mitigate them.

<comment><USER11> Funny that you should mention cargo-culting and regulations.

In my experience, much of the cargo-culting is _driven_ by regulations. And a good part of that has been caused by the industries being regulated, as an unfortunate consequence.

Now... I think that warrants an explanation.

Regulations that make sense are often not descriptive - capturing the intent and scope of a rule often requires technical expertise. More than that, it's the type of expertise most organisations do not have. And instead of improving themselves, these companies, who may form the _grand majority_ of the industry, petition the regulators to provide a safe checklist of technical mitigations that can be implemented to remain compliant.

You can probably see where this is going. (I am going to make a wild guess and say that you likely work in the banking industry.) Instead of providing these mitigation checklists as external annexes, they are instead incorporated directly into the regulations themselves.

Regulations _always_ trail reality by a few years -- and in some cases, by decades.[0] The very same checklists that were originally meant to help the smaller players from caving under the regulatory burden are now a drag anchor on entire industries. Instead of doing the right thing and meeting the _planned intent_ , companies are instead ticking nonsensical boxes that the regulators and their auditors demand.

Blindly.

Mindlessly.

Divorced from reality.

And worst of all, unable to improve. Because if you made an improvement that deviates from the historically old, untouched checklist, you would no longer be compliant.

0: Don't ask. I have given a couple of talks on the subject, but refuse to go on record, let alone state the details of my beliefs publicly.

<comment><USER10> Maybe I should rephrase. I don't mean cargo culting like that. My grip is with (most) security people whose only handling of vulnerabilities is to spread fear mongering and stupid action ASAP, without any explanation of the issue or solution. Long story short, cyber security is a circus.

Turning off TLS 1.0 in 2015 would have cut at least 30% of your user base (internet explorer) and was simply not an option for anybody. If there was an explanation of what is the issue, the impact and how to mitigate, this could have been evaluated and implemented (or not), but it's never been given.

We worked together. You should recognize me really ;)

Regulations are largely fine. You should known from doing the audits. They are usually worded to say that something should follow the generally accepted standards or good practices, without specifying what they are. Either way, it's always fairly easy to delay or argue that whatever you're doing is the right thing.

<comment><USER11> Thank you for confirming, I was quite certain it was you.

And I do agree with you, within reason: _financial industry_ tech regulation is largely fine. ("Sane") The regulations tend to state the intent of a rule, and may even specify something that clearly is not fine.

More recently I have come across different sets of rules. While it is obvious that these particular regulations have been written with the best intentions in mind, they end up making things worse. Instead of specifying what the _goal and the reason_ for any rule is, they state only The One True Way. Reason has to be inferred, because it is unstated.

Doesn't matter if the truth in The Way may have been induced by bad 'shrooms. Once written down, it's gospel.

</comment></comment></comment></comment></comment><comment><USER12> Given the ubiquity of Java 8 in enterprise IT, I think we'll all be stuck supporting TLS1.2 for at least another 5 years

<comment><USER10> Java 8 is end of life in 2023. The OpenJDK one maintained by RedHat that's the de facto standard on all Linux.

Java 7 is end of life in 2020. It only support TLS 1.2 if you have a recent JVM release. Don't expect people on java 7 to ever update their JVM though.

</comment></comment><comment><USER13> Can someone points me to the commit when OpenSSL changed from TLS13-AES-256-GCM-SHA384 to TLS_AES_256_GCM_SHA384?

<comment><USER13> Also, idk why OpenSSL have used underscore this time when all the rest of ciphers use dash!? [https://testssl.sh/openssl- iana.mapping.html](https://testssl.sh/openssl-iana.mapping.html)

<comment><USER2> The "official" names of the ciphersuites have always been in the underscore format, the dash format was something OpenSSL invented. I guess with TLS 1.3 changing all the cipher names anyways they decided to just use the standard names to avoid the confusion that comes from their own separate naming scheme.

</comment></comment></comment><comment><USER14> That's impossible to do, at least on MS IIS directly since SCHANNEL doesn't support TLS 1.3 yet

<comment><USER15> The generator doesn't support IIS anyways but there are some in it's list that don't support the Modern configuration (gives a comment saying it's unsupported if you try): PostgreSQL, Oracle HTTP Server, and awselb

<comment><USER16> I wonder what feature is missing from PostgreSQL. It supports TLSv1.3 and supports letting the client pick cipher.

EDIT: NVM, I forgot that ssl_min_protocol_version was added in PostgreSQL 12 which is not released yet.</comment></comment></comment>


The city is a big brain that can solve big problems - <AUTHOR> http://nautil.us/issue/18/genius/san-francisco-is-smarter-than-you-are-rd <comment><USER2> Moment by moment, the writing is quite decent, but I had to look hard to find anything of substance here. The fundamental 'conceit' [technical term] of the essay is incredibly well-worn, and it is surprising to see it reproduced here so uncritically. There is one substantive and interesting claim, based on research by two scientists named Wayne Gray and Wai-Tat Fu: "externalized knowledge is accessed like any other memory, and that we treat 'memories' on a computer screen just like memories in our head." This is joined to the suggestion from Edwin Hutchins that the 'cognitive task' of navigation at sea was socially distributed across "several people, charts, papers, clocks, and a variety of physical instruments." As ships in truth use GPS, and used radio beacons before then, and as in truth a single man with a map, a compass and sextant can _roughly_ determine the position of a ship at sea, this sounds suspiciously like a thesis that is picking and choosing its own Scotsmen. What exactly makes a distributed task be _cognitive_ rather than, say, a computation? I do not think that this is answered, and it seems that the entire idea never gets beyond an "A implies B, C implies B, therefore A is C" fallacy.

Anyway, what excites the author is the idea that this external cognitive process extends seamlessly into the internal cognitive process by way of the aforementioned findings about externalized knowledge. There is no real notion beyond that. He believes that as the city population grows larger and denser, the "genius" of the metaphorical city mind will increase; he cites strong evidence that this is true, but he attributes the solved logistics and communications of a large city to that genius, no matter that a mind is capable of no similar thing. If "people are to cities as neurons are to brains," and based on the power-law scaling he mentions, he logically ought to end with the fear that all large cities will be eventually destroyed by massive epileptic cascades. Instead, he does the usual soft-shoe across different scales and concepts to freshen up an old and familiar imagination of science.

I have no idea why this is about San Francisco, although the golden gate bridge is quite obviously a corpus callosum.

<comment><USER3> Interesting thoughts--thanks for sharing. If one wished to read further on this topic, are there any resources you would recommend?

Also, your comment about epileptic cascades made me think... Could it be that political and socioeconomic struggle and headbutting within a city is a sign of that?

<comment><USER4> [http://www.amazon.com/Supersizing-Mind-Embodiment- Cognitive-...](http://www.amazon.com/Supersizing-Mind-Embodiment-Cognitive- Philosophy/dp/0199773688)

</comment></comment></comment><comment><USER5> "While some of us can name the inventor of the light bulb, does anybody know who invented the iPhone?"

The iPhone is not an invention. Its an iteration.

<comment><USER6> Yeah, the smartphone was not such a big leap as it's made out to be. It was mainly UI. I used to have a Sony Ericsson dumbphone and I had Opera Mini and Google Maps on it. I ordered train tickets online with that phone.

<comment><USER7> It was as big a leap in UI as it's made out to be.

</comment></comment></comment><comment><USER8> Strange article. The article is trying to say something about collaboration and then somehow leaps to cities being good for collaboration.

This claim is made despite abundant evidence that the greatest tech innovations in Silicon Valley did not occur in the city, but the much more mundane suburbs and universities.

This trend has largely continued because cities are poor environments for "innovation" \- higher cost of living, less space for tinkering and equipment, and lower "neighborly" sentiment which is required for collaboration.

[http://en.wikipedia.org/wiki/Bowling_Alone](http://en.wikipedia.org/wiki/Bowling_Alone)

<comment><USER9> > … cities … lower "neighborly" sentiment which is required for collaboration

I've found this to be the opposite in comparing cities to suburbs.

If you're referring to the tech startups that created silicon valley, and why they started where they started, you really can't ignore the subsidized land granted through the university system.

<comment><USER10> The point Jane Jacobs makes about innovation and cities is that old buildings are essential. Basically, you need clusters of places with cheap rent. If all the real estate is new and shiny and impressive and nice, it's only going to be available to businesses that are already firing on all cylinders. And that's great for them, for a while. But the other point she makes is that nothing lasts forever. The conditions that make a business work will eventually change, and in most cases, the kinds of business that worked well at one time won't be able to adapt to another. They'll just go away.

If the city in which this constant drama is playing out has a decent stock of old buildings, there's a good chance that there will be another generation of businesses that are better attuned to the times who are ready to move into the high-rent districts. But if a city falls prey to short-term thinking and allows all its cheap neighborhoods to be gobbled up by rapacious developers, it's going to find itself driving off a cliff in terms of long range economic viability.

In agricultural terms, getting rid of cheap rents is like eating your seed stock. In more vivid terms, it's like setting your house on fire to stay warm in the winter.

</comment></comment></comment><comment><USER11> This whole idea is similar to the one Robert Pirsig explored in his book Lila, the less-well-known sequel to his famous "Zen and the Art of Motorcycle Maintenance."

He explored the idea of evolution at many levels for a whole chapter, pondering about how the concept of evolution has itself evolved. From pure energy, atoms "evolved" in a sense. Those atoms accumulated in a sort of physical evolution into larger bodies; stars, asteroids, and planets. Stars fused atoms into larger, more complex atoms. Planets congealed and evolved on a geologic scale, the Earth specifically forming mountains, rivers, oceans. In that form, molecules evolved further and increased in complexity and organization into amino acids, and life itself, which evolved in the sense with which we're more familiar, from single cells, to multicellular organisms, to a myriad of forms, and finally into sentient humans.

The evolution, however, has not stopped as we normally assume. Instead, humans are and have been only atomic parts of the phase of evolution in which we're currently taking part, where we surpass the limits of our biology and evolve in a social way, and then a technological way, to form a being far greater than any single one of us. We are networked like a giant brain would be, we have information from the far reaches of the world at our fingertips, the ability to coordinate and execute incredible feats, and surely the ability to grow and spread and prosper as never before.

This is the evolution of the universe. Each phase happens at a more and more rapid pace than the one before, and each increases in the complexity and capability of the organism as a whole. For some reason, despite entropy and the supposed tendency of the universe to randomize, it turns out it has a surprising tendency to organize in spite of itself.

So yes, the city is in essence a big brain that can solve big problems. The whole world is, and all of history has been leading up to this. What will we do with it?

Here are two appropriate images.

1\. Neurons as compared to cities: [http://laughingsquid.com/wp- content/uploads/cityneuron3.jpg](http://laughingsquid.com/wp- content/uploads/cityneuron3.jpg) 2\. Neurons as compared to the distribution of galaxies in the universe: [http://3rdeyevisionblog.files.wordpress.com/2013/02/neuron-g...](http://3rdeyevisionblog.files.wordpress.com/2013/02/neuron- galaxy.jpg)

Take them as you will. Is it a perfect analogue? Of course not, but it is a striking fractal self-similarity that illustrates the types of structures that tend to form in the universe. Who is to say that a city is not a form of brain; or the globe, networked in much the same way? Or the universe for that matter, massive clusters connected by vast filaments of matter held together by gravity and energy itself?

Each part, at every scale, centers around the transfer and storage of information. What is this thing, this universe, that has formed these patterns at so many levels?

It's fun to think about.

</comment><comment><USER12> > While some of us can name the inventor of the light bulb

Okay, quiz time, no using the internet: who invented the light bulb?

<comment><USER13> I'll bite. It has a storied history...good stuff.

[http://www.unmuseum.org/lightbulb.htm](http://www.unmuseum.org/lightbulb.htm)

<comment><USER14> > no using the internet

</comment></comment></comment><comment><USER15> Just a counter point, whenever anybody wants to solve _really_ hard problems, the think tank, lab, whatever that's setup to do the solving is very rarely near a city.

<comment><USER16> For what it's worth [ _]_ , I'm pretty sure CERN(/LHC), ESRF, SLAC Nat. Accelerator Labs, Lawrence Livermore, Lawrence Bekeley, Sandia, Princeton Plasma Phys. Lab, Brookhaven, Fermilab, Argonne Nat. Lab, TRIUMF, Sudbury Neutrino Observatory, Oak Ridge, Research Triangle Park, various institutions within the Max Planck Society, Institute for Advanced Study, Bell Labs, Xerox PARC, RAND Corporation, (and likely many others...), all have/had facilities within (at most) 100 km of a city with a population of at least 150,000 people. Several of these institutions are within commuting distance of major world cities.

In the set of nation state funded research facilities (e.g. funded by US Dept. of Energy, European Research Council, etc.) I'm willing to bet that most of the facilities are _more often_ relatively near cities (greater than 150K pop). Many (high energy / high capital cost) facilities, employ on the orders of 1-10K staff alone.

[ _]_ (... and that Basic Science, High Energy Physics, Computational Science, Basic Tech, Corporate Research, and/or Public Policy/Economic/GeoPol research are the hard problem spaces you are alluding to...)

<comment><USER15> > have/had facilities within (at most) 100 km of a city with a population of > at least 150,000 people.

I mean, yeah, _everywhere_ is basically within walking distance if you change the definition around enough.

I've actually spent a bit of time at towns near big labs, and for the most part these are not the kinds of big cities solving problems the article is trying to assert. They're sleepy little towns, many of which barely have a building over 2 stories. In fact, I live 20 minutes from a major biomed lab. I definitely do not live in a major city. 20 year ago people would have called this entire area "farms".

The article is positing that _Cities_ , BIG Cities, are the places hard problems get solved. Not places withing an hour drive of a big city. CERN, ESRF, SLAC, etc. aren't _in_ major cities of their respective countries. 150,000 people is not a big city. If the supposition was correct, it would make sense to set these kinds of hard problem solvers in your densest, most populated cities. NYC would be thick with labs, Paris, Tokyo and London would all be centers of major Research facilities.

I think the Soviets got it right with not being coy about what these places are: наукогра́д "science cities". For example, Tri-cities, WA is a "farm" town that only exists because one of the biggest U.S. nuke labs was built there. It was built there because it's out in the middle of nowhere. ORNL is a 2.5 hour drive from Nashville. Brookhaven, 1.5 hours.

My point is that cities don't solve hard problems. Cities merely bring efficiency that can be a useful tool in aiding this process. It'd be hard and more expensive to build something like the LHC without good infrastructure in place to move all the equipment in, so of course you're going to have good transit links and communications infrastructure etc. But the LHC wasn't built in downtown Paris was it?

[] (... and that Basic Science, High Energy Physics, Computational Science, Basic Tech, Corporate Research, and/or Public Policy/Economic/GeoPol research are the hard problem spaces you are alluding to...)

yeah basically. Let's be real honest here, the kinds of "problems" the Bay Area seems to spend most of it's time solving appear to be mostly figuring out how to extract the maximum amount of money out of fairly uninteresting technical implementations designed to let people chat and share cat pictures. In other words, money for engineering, not problem solving.

</comment></comment></comment><comment><USER17> This is the first time I've seen this metaphor applied to San Francisco, and not New York.

<comment><USER18> The article is about intelligence, not narcissism. :)</comment></comment>


When you're a Rails developer but your client wants WordPress - <AUTHOR> https://buttercms.com/blog/when-youre-a-rails-developer-but-your-client-wants-wordpress <comment><USER2> "You explore the workaround of setting up a reverse proxy, but it looks extremely unpleasant."

not really... sooner or later having the flexibility of a reverse proxy will be extremely beneficial.</comment>


Video game training enhances cognitive control in older adults - <AUTHOR> http://www.nature.com/nature/journal/v501/n7465/full/nature12486.html or, rephrased,<p>"training on a task designed to tap into cognitive control, enhances cognitive control, in older adults ... and we named the task 'a video game'" <comment><AUTHOR> or, rephrased,

"training on a task specifically designed to tap into cognitive control, enhances cognitive control, in older adults ... and we have named this task 'a video game' "</comment>


Show HN: Python 3.5 Async Web Crawler Example - <AUTHOR> https://github.com/mehmetkose/python3.5-async-crawler <comment><USER2> Guido van Rossum, the creator of Python, wrote a web crawler as a motivating example for asyncio. You can find the code for it here:

[https://github.com/aosabook/500lines/tree/master/crawler](https://github.com/aosabook/500lines/tree/master/crawler)

And a detailed post about its design, co-written with A. Jesse Jiryu Davis, here:

[http://aosabook.org/en/500L/a-web-crawler-with-asyncio- corou...](http://aosabook.org/en/500L/a-web-crawler-with-asyncio- coroutines.html)

</comment><comment><USER3> I was investigating how to add asyncio / async def support to Scrapy (see [https://github.com/scrapy/scrapy/issues/1144#issuecomment-14...](https://github.com/scrapy/scrapy/issues/1144#issuecomment-141843616)). Small examples like the one at the link look neat, but it is not all roses as you go further. The problems are not specific to Scrapy; I think any advanced `async def` based crawler will face them.

There are 2 challenges with async def I don't know how to solve elegantly:

1\. how to integrate coroutine-based scraping code with on-disk persistent request queues;

2\. how to deallocate resources without boilerplate in coroutine-based scraping code.

(1) is easier with callbacks-as-methods because this way state is passed explicitly (it is not in local variables), so Scrapy can choose to save it to disk.

Example of (2) is this code:

        async def parse(self, response):
            resp = await self.fetch(url)
            # ... find another URL to follow

            # Here we have the problem:
            # response object is kept in memory
            # until second response is fully received.
            # This is a problem if 10s and 100s
            # of requests are processed in parallel
            # and responses are large.
            # Because of refcounting, with callbacks
            # response would have been kept in
            # memory only until second request
            # starts - callbacks+refcounting provide
            # an elegant way for resource deallocation.
            resp = await self.fetch(url2)

If anyone has suggestions please comment on [https://github.com/scrapy/scrapy/issues/1144#issuecomment-14...](https://github.com/scrapy/scrapy/issues/1144#issuecomment-141843616).

<comment><USER4> I don't know scrapy enough to understand your first problem.

As for the second one, wouldn't using streams to process the data solve the memory usage issue?

<comment><USER3> The first problem is not specific to Scrapy; it is the same if you're using inlineCallbacks from twisted, tornado.gen or async def: there is no way to serialize tasks inside coroutine and save them to disk, e.g. to be able to stop the process and then restart it from the same point, or to avoid keeping the whole request queue in memory.

Stream processing could fix the second problem, but it is not a practical solution: in most cases one needs to build HTML tree in memory to do further processing (e.g. extract links). I'm also not aware of streaming regex libraries.

</comment></comment></comment><comment><USER5> This example isn't really making use of asyncio. asyncio.run_until_complete() is a blocking method (note that you don't use await when calling it, as it's not a coroutine.) You'd want to use something like asyncio.wait() with multiple futures to achieve some concurrency.

<comment><AUTHOR> Thanks for the notice. I just trying python 3.5 goodies. I updated the code now and I added a queue.

</comment></comment><comment><USER4> While you're using AsyncIO, your requests are still done serially due to using loop.run_until_complete().

</comment><comment><USER6> What is the advantage of this over say using threads? Web scraping is pretty much all IO so you get big wins using threads in Python and Ruby.

<comment><USER7> Threads in Python creates a full posix thread which is very heavyweight compared to using AsyncIO. You should get the same throughput for a far lower resource usage.

<comment><USER8> does asyncio in python = green threads in python, therefore defeats the GIL?

<comment><USER4> It does not defeat GIL, but the way asyncio is working makes GIL irrelevant.

In simplest use case you have a single thread (an actual OS thread) and as soon as certain operation blocks (e.g. waiting a response from server) another coroutine is scheduled.

The reason why GIL is irrelevant is because only a single coroutine executes at given time[1]. The way it works is similar to cooperative multitasking. The new task that was scheduled will execute until it encounters an operation that will make it block (typically an I/O operation).

[1] You actually can spin multiple threads and each could have its own event loop and then schedule coroutines on different threads.

</comment></comment></comment></comment><comment><USER9> Here's a proper example written for 3.4+: [https://gist.github.com/madjar/9312452](https://gist.github.com/madjar/9312452)</comment>