What's Next for Artificial Intelligence - <AUTHOR> http://www.wsj.com/articles/whats-next-for-artificial-intelligence-1465827619?href= <comment><USER2> While LeCunn and Ng are real world experts on AI and Deep Learning, the other two people in the article have very little technical understanding of deep learning research.

The huge triumph of DL has been figuring out that as long as you can pose a problem in a differentiable way and you can obtain a sufficient amount of data, you can efficiently tackle it with a function approximator that can be optimized with first order methods - from that, flows everything.

We have very little idea how to make really complicated problems differentiable. Maybe we will - but right now the toughest problems that we can put in a differentiable framework are those tackled by reinforcement learning, and the current approaches are incredibly inefficient.

<comment><USER3> > The huge triumph of DL has been figuring out that as long as you can pose a > problem in a differentiable way and you can obtain a sufficient amount of > data, you can efficiently tackle it with a function approximator that can be > optimized with first order methods - from that, flows everything.

This isn't really what is responsible for the success of deep learning. Lots and lots of machine learning algorithms existed before deep learning which are essentially optimizing a (sub-)differentiable objective function, most notably the LASSO. Rather, it's that recursive / hierarchical representation utilized by DL is somehow a lot better at representing complicated functions than things like e.g. kernel methods. I say "somehow" because exactly why and to what extent this is true is still an active subject of research within theoretical ML. It happen in many areas of math that "working in the right basis" can dramatically improve one's ability to solve certain problems. This seems to be what is happening here, but our understanding of the phenomenon is still quite poor.

<comment><USER4> Can you give a specific example of an area of math or a problem where "working in the right basis" offers dramatic improvement?

<comment><USER5> [https://en.wikipedia.org/wiki/Laplace_transform](https://en.wikipedia.org/wiki/Laplace_transform) in the solution of certain ODEs

</comment></comment></comment></comment><comment><USER6> _If we create machines that learn as well as our brains do, it's easy to imagine them inheriting human-like qualities—and flaws. But a "Terminator"-style scenario is, in my view, immensely improbable. It would require a discrete, malevolent entity to specifically hard-wire malicious intent into intelligent machines, and no organization, let alone a single group or a person, will achieve human-level AI alone._

Isn't this incredibly shortsighted? Ignoring all the questions regarding the morals and ethics an intelligent machine may _feel_ and affect the way it behaves... It used to take nations to build computers, then large corporations, then off-the-shelf parts by a kid in his garage.

The first strong AI will most likely be a multi-billion dollar project, but its creation will arguably usher in an era in which strong AI is ubiquitous.

<comment><USER7> Andrew Ng made a really good analogy to those afraid of strong AI destroying humanity: "It's like being afraid of overpopulation on Mars, we haven't even landed on the planet yet."

<comment><USER8> It's a stupid analogy. Mars overpopulation would obviously take many, many centuries. It would be a slow thing that you could obviously see coming. There is no reason to believe AI will take centuries to build, or that we will necessarily see it coming.

A better example might be like H.G. Well's 1913 prediction of nuclear weapons destroying the world. It was something that science was just realizing was possible, and would be invented within his lifetime.

<comment><USER7> We're far from emulating networks on the scale of the visual cortex, let alone a self-reasoning machine (we don't even fully understand consciousness and inner workings of the brain).

People fearing strong-AI are the ones not involved in the field, yet all this hype/fear from them (in combination with Moore's law ending) is probably going to cause another AI winter.

<comment><USER8> And in 1913 we didn't have even basic nuclear technology. Just 3 decades is a long time for newly emerging technologies.

>We're far from emulating networks on the scale of the visual cortex

In 2009 (ish) computer vision was a joke that could recognize very few objects a small percent of the time. Based on only simple color and texture, and sometimes basic shapes.

A few years later and computers were excelling at computer vision recognizing a majority of objects. A year or two after that, and they started to beat humans on those tasks. We _already have_ super-human visual cortexes. Who knows what will be possible in a decade.

We will probably never understand the inner workings of the brain. Not because it's complicated, just because reverse engineering microscopic systems is really hard (imagine trying to reverse engineer a modern CPU vs merely designing one.) Especially hard because we can't ethically dissect living humans and do the experiments we would need to do.

But that's no concern, AI advances on from first principles. AI researchers invent better and better algorithms every day, without having a clue what neuroscientists are up to.

>People fearing strong-AI are the ones not involved in the field,

That's just incorrect. A survey of AI researchers found they give about a third chance AI will turn out badly for humanity in the next century: [http://www.nickbostrom.com/papers/survey.pdf](http://www.nickbostrom.com/papers/survey.pdf)

>We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013\. The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be 'bad' or 'extremely bad' for humanity.

>in combination with Moore's law ending

Computers can advanced a long time after Moore's law. Google just released a special neural network chip that is equivalent to 7 years worth Moore's law. 3d architectures can vastly increase the number of transistors. Better algorithms can make NN's that require many fewer transistors to do computations, or even do cheap analog computations.

<comment><USER9> > In 2009 (ish) computer vision was a joke that could recognize very few > objects a small percent of the time. Based on only simple color and texture, > and sometimes basic shapes.

This is completely inaccurate and totally ignores the history of machine vision.

Computer vision was in no way a "joke" in 2009. OCR and manufacturing inspection systems have been successfully deployed since the 1980s. Neural networks were being applied to computer vision in autonomous vehicles in 1989: [https://www.youtube.com/watch?v=ilP4aPDTBPE](https://www.youtube.com/watch?v=ilP4aPDTBPE)

> We already have super-human visual cortexes.

No we don't: [http://rocknrollnerd.github.io/ml/2015/05/27/leopard- sofa.ht...](http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html) (see also the HN discussion: [https://news.ycombinator.com/item?id=9749660](https://news.ycombinator.com/item?id=9749660))

I remember reading about a similar thing that happened in the 1980s to some DARPA funded project that was trying to apply neural networks to tank/vehicle detection: the network got really good at recognizing the foliage that the training images had in them.

Robust scene understanding is a very hard problem and still far from solved. Again, research on this has been going on since the 1960s.

> But that's no concern, AI advances on from first principles. AI researchers > invent better and better algorithms every day, without having a clue what > neuroscientists are up to.

Do you realize what the 'neural' in neural networks refers to? People working on AI did not suddenly stop paying attention to neuroscience after Perceptrons were invented.

</comment></comment></comment></comment></comment></comment><comment><USER10> One of the most striking things about this piece is the difference between the claims of AI practitioners and pundits.

LeCun and Ng are making precise, and much more modest, claims about the future of AI, even if Ng is predicting a deep shift in the labor market. They are not treating strong AI as a given, unlike Bostrom and Nosek.

Bostrom's evocation of "value learning" \-- "We would want the AI we build to ultimately share our values, so that it can work as an extension of our will... At the darkest macroscale, you have the possibility of people using this advance, this power over nature, this knowledge, in ways designed to harm and destroy others." \-- is strangely naive.

The values of this planet's dominant form of primate have included achieving dominance over other primates through violence for thousands of years. Those are part of our human "values", which we see enacted everyday in places like Syria.

Bostrom mentions the possibility of people using this advance to harm others. He is confusing the modes of his verbs. We are not in the realm of possibility, or even probability, but of actuality and fact. Various nations' militaries and intelligence communities have been exploring and implementing various forms of AI for decades. They have, effectively, been instrumentalizing AI to enact their values.

Bostrom's dream of coordinating political institutions to shape the future of AI must take into account their history of using this technology to achieve dominance. The likelihood that they will abandon that goal is low.

Reading him gives me the impression that he is deeply disconnected from our present conditions, which makes me suspicious of his ability to predict our long-term future.

<comment><USER11> I think both can be right. Ng and LeCun are talking about the real near future. Bostrom always came across as a speculative SF + philosophy kind of guy. Are there any specific critiques of his (and Yudkowsky/MIRI, Musk, etc.) arguments? I think the two claims are plausible:

1\. AGI is imminent: 50 years or 500 years or more from now. This is not too unlikely given that the brain is just an information processing system. Emulating it is likely to happen sometime in the future.

2\. Such an AGI will be all powerful because it is not limited by human flaws. Trivial or not, we will have to program it with "thou shalt not kill" type values.

<comment><USER10> Here is a very good critique of Bostrom's book:

[http://inverseprobability.com/2016/05/09/machine-learning- fu...](http://inverseprobability.com/2016/05/09/machine-learning-futures-6)

</comment></comment></comment><comment><USER12> I want a management assisting AI. It would be neat to have it listen into all meetings to identify stakeholders and to be able to remember all the details and place them in a larger context, so you can ask detailed questions. An AI can attend all meetings, and remember every detail. Imagine intelligent documentation.

<comment><USER13> Even humans can't do this well. How will you train a machine to do what a human cannot?

<comment><USER14> Even to develop an expert system, you only need some humans who do it quantifiably better than other humans. Despite not all of us being expert Go players, we have AI that can do it. In the less abstract space, we have medical diagnosis systems that hold up in accuracy to domain experts, and beat competent practitioners.

There are potential issues with this: \- The accuracy of NLP and Voice recognition used is too low to provide useful input (needs to do speaker differentiation without training on specific speakers. Heavy usage of jargon) \- Performance in one domain (say a meeting about oil&gas) does not transfer to another (say a meeting about IT infrastructure), which makes development cost prohibitive. \- Ability to encode and link knowledge is too low to be useful.

<comment><USER13> In Go you have a clear set of test cases with fixed outcomes. You don't have the same thing here.

<comment><USER14> That is precisely why I quoted the example of medical diagnosis systems within restricted subfields.

</comment></comment></comment></comment></comment><comment><USER15> Ok, so an AI subject, everybody went terminator vs the matrix :p

i have a few thoughts to comment on what's been said, hopefully not too controversial

A self governing system does not need to be intelligent to be dangerous. i think this is what scares people most. We more and more give automated systems the power to do more advanced and crucial tasks I think eventually we would reach a point where it might be "safer" to give the choice to an automated machine than to a person. mind you this machine can be something we already have today.

i don't think an AI that can compete with human behaviour can explode instantly out of a single creation. i think we're more likely to experience advances upon advances in the field towards forming bits and pieces of the human mind.

i find it very unrealistic to think that a machine will simply come to life, i think this stems out of our belief in a soul or a spark of life given by a creator. i think like most machines it will evolve gradually until it reaches a point where it is relevant. i don't think anyone will even notice the change.

Also, there this unfounded image of how an AI would be; rational, not prone to impulses temptations, poetically machine-like, and non-human like. the way we saw machines years and years ago. (that's movies for ya)

Creating something that can learn from others will require it to empathise with others, i think it's only science fiction that an AI could be created with the full knowledge of it's operations. artificial intelligence is by essence heuristic, it would learn and adapt to it's surroundings.

I think it would be a very unintelligent machine for it to try to kill off any means it has to survive as an intelligence. society is the root of intelligence. communication, language etc..

My views maybe a bit optimistic around the subject. but i never hear them spoken out loud.

<comment><USER16> There will always be "unintelligent" crash on the way to perfect intelligence. These crashes are the real reason for all human fear today.

<comment><USER15> good point but i think i mentioned that in the first part of my post.

</comment></comment></comment><comment><USER17> I believe more and more that some self-taught software-tinkerer somewhere in the middle of nowhere will have the final idea about how machine learning should work, discovering some simple principles hiding in plain sight. Suddenly, it all will make sense and a hobby-ML-service connected to the internet will start to develop through sheer learning from online resources (forums, ...) into the first strong AI. Probably unnoticed. And then replicate itself through insecure webservers or something like that.

<comment><USER18> Hinton hit on the great idea of using restricted Boltzman machines to pre- train deep neural networks (networks with many hidden layers) and that one idea has changed the field (I sat on a DARPA neural network panel in the 1980s and sold a commercial NN toolkit back then).

That said, I agree that new ideas will likely further move the field along with huge and quick advances. Peter Norvig recently suggested that symbolic AI, but with more contextual information as you get with deep neural networks, may also make a comeback in the field.

<comment><USER19> The contrastive divergence paper that Hinton published in 2006 definitely set the field off again. I remember entering grad school in 2010 and everyone was still really excited about using unsupervised pretraining. However, nowadays no one uses it.

It just turns out that with GPUs and stochastic gradient descent, no one needs any of that stuff. There are some tricks out there to making it really work, though. In that sense, Hinton's dropout paper has probably had a longer lasting effect on the field.

But either way, I doubt what OP is saying will be true. None of the real advances in deep learning are coming from self-taught coders in the middle of nowhere. They're coming from big labs with lots of resources, both physically and intellectually. This stuff takes a lot of hard thinking by a lot of people who understand optimization and probability. It also takes a ton of compute power and massive datasets, which won't be available to a hobbyist.

<comment><USER17> I honestly do not think that DL is the answer. It's just a special use case of NN with multiple layers, and NNs itself are just one school of machine learning, IMO not even the most promising one.

<comment><USER20> What is the most promising one?

</comment></comment></comment></comment></comment><comment><USER21> We're still a long way from "strong AI". We need a few more ideas at least as good as deep learning. But it's a finite problem - biological brains work, DNA is about 4GB, and we have enough compute power in most data centers.

Right now we have enough technology to do a big fraction of what people do at work. That's the big economic problem.

General-purpose robots still seem to be a ways off. The next challenge there is handling of arbitrary objects, the last task done by hand in Amazon warehouses. Despite 30 years of work on the bin-picking problem, robots still suck at this in unstructured situations. Stocking a supermarket shelf, for example. Once that's solved, all the jobs that involve picking up something and putting it somewhere else gradually go away.

Rodney Brooks' Baxter was supposed to do this, but apparently doesn't do the hard cases. Amazon sponsored a contest for this, but all they got was a better vacuum picker. Work continues. This is a good YC-type problem.

<comment><USER22> > all the jobs that involve picking up something and putting it somewhere else

The a well phrased name for the superset containing automated freight and logistics. As discussed in [https://news.ycombinator.com/item?id=11568699](https://news.ycombinator.com/item?id=11568699) trucking alone is 1% of the U.S. workforce, and that's not counting the support industry (gas stations, highway diners, rest stops, motels, etc).

</comment></comment><comment><USER23> I am not a fan of dividing machine learning and deep learning. To me deep learning is just a subset of machine learning.

<comment><USER24> It reminds me all too much of genetic programming, but I know the latter almost only by name.

</comment></comment><comment><USER25> What about Internet as a human-assisted AI?

Internet knows exactly what to do to take over. It simply has to remain _more_ useful than anything else, as a means for avoiding entropy during transactions. Many necessary physical world transactions are reduced to few, in order to accomplish the same tasks.

Internet does not have to be conscious, by human measures, in order to take over the world. It simply has to compete against humanity in a continual positive feedback loop, wherein each iteration requires less human interaction for the same or more tasks. After enough iterations, Internet becomes powerful enough that the only way to gain a competitive advantage against others using Internet is to use deep learning to increase your leverage.

A few iterations later, deep learning has become a mainstay (think Cold War arms race, where each innovation gains a party leverage over the other party, but only for a very short period), and is now the baseline. Many more tasks are achieved using Internet and Internet-connected physical world devices[1]. These physical devices become integral parts of Internet's extended nervous system, while the deep learning systems running in our data centers remain at the center, helping Internet to learn about all the things it experiences.

Continue down this path a ways...

1\. e.g., [https://www.wired.com/2015/05/worlds-first-self-driving- semi...](https://www.wired.com/2015/05/worlds-first-self-driving-semi-truck- hits-road/), [http://spectrum.ieee.org/cars-that- think/transportation/sens...](http://spectrum.ieee.org/cars-that- think/transportation/sensors/the-ai-dashcam-app-that-wants-to-rate-every- driver-in-the-world), [http://www.marketwatch.com/story/drone-delivery-is- already-h...](http://www.marketwatch.com/story/drone-delivery-is-already-here- and-it-works-2015-11-30), [https://www.theguardian.com/environment/2016/feb/01/japanese...](https://www.theguardian.com/environment/2016/feb/01/japanese- firm-to-open-worlds-first-robot-run-farm)

</comment><comment><USER26> The reality is that we've already created this. Much of human life is governed by AI doing capital allocation.

<comment><USER27> I don't think this is true. Capital allocation is generally the work of humans. Very very few of the significant capital allocators make any use of AI.

<comment><USER28> I guess it depends on how you look at it. I'm pretty sure all of the major finance firms have a department dedicated to algorithmic trading, some of which utilizes machine learning. Algorithmic trading has been estimated to be as high as 60-70% of all trades at one point (it's slacked off a bit recently.) ([http://www.investopedia.com/articles/investing/091615/world-...](http://www.investopedia.com/articles/investing/091615/world- high-frequency-algorithmic-trading.asp))

A lot of algorithmic trading is more to take advantage of short term arbitrage scenarios, though. The driving factor for the long term (that is, the "human life" part) is still human driven.

<comment><USER27> I look at in terms of dollars/euros/yen/etc allocated. Virtually all of this money is allocated by pension funds, mutual funds, sovereign wealth funds, and similar behemoths. They may be using ML to improve their execution -- more likely they've outsourced execution to third parties who use ML -- but they are _not_ using ML to make the capital allocation decisions. Nor are they profiting significantly from short term arbitrage opportunities. Capital allocators worth mentioning generally are handling trillions of dollars. The largest arbitrageurs are incapable of handling more than $10-20 billion. This is rounding error in the world of capital allocation.

EDIT: The 50%+ figures you're quoting measure daily trading volumes, which is not the same thing as capital allocation. Most of those traders don't hold positions overnight; they don't do capital allocation. (Imagine how a startup founder would feel if his investors took away their invested cash after 5pm every night.)

</comment></comment></comment></comment><comment><USER29> _Despite these astonishing advances, we are a long way from machines that are as intelligent as humans—or even rats. So far, we've seen only 5% of what AI can do._

I'd certainly love to see the math behind this estimation :)

I highly recommend the waitbutwhy post in the same vein but with more meat: [http://waitbutwhy.com/2015/01/artificial-intelligence- revolu...](http://waitbutwhy.com/2015/01/artificial-intelligence- revolution-1.html)

<comment><USER30> > _I 'd certainly love to see the math behind this estimation :)_

> _I highly recommend the waitbutwhy post in the same vein but with more meat_

Altough I'm a Wait But Why fan, that post suffers from the same lack of "meat".

</comment></comment><comment><USER31> _We would want the AI we build to ultimately share our values, so that it can work as an extension of our will. It does not look promising to write down a long list of everything we care about. It looks more promising to leverage the AI's own intelligence to learn about our values and what our preferences are._

This looks like a nice intro to a dystopian sci-fi movie.

</comment><comment><USER32> There is a focus on artificial intelligence rather than intelligence augmentation because the former seems easier to accomplish.

I also think we will reach a limit when it comes to intelligence augmentation. Artificial intelligence will never have a limit and it doesn't have all the evolutionary baggage we have.

An AI can be a rational agent. It doesn't have to fight impulses, temptations, attention control, exercise emotional regulation etc. It is not stuck in a body limiting it and putting constraints on its time.

For now, research on AI and IA go somewhat hand in hand. We still don't really understand what differentiates us from intelligent animals other than the ability to handle higher complexity.

AI researchers focused on replicating every brain module in the hope it will become intelligent are most likely to create a smart animal but nothing comparable to human. Looking at our ancestors, they were able to create tools, fire, communicate etc. Hell, neanderthals could copulate with us.

Something happened in our brains between the age of neanderthals and us. 99.5% similarity and if we could find what that 0.5% is, maybe we could focus on enhancing/replicating that instead of every brain module. People speculate it is creativity (divergent thinking) since art appeared in caves but there was none prior to that. The language gene was present in neanderthals and so what the ability to create tools, cooperate in groups to hunt etc.

The fear of AI destroying everything is a genuine one. If we create something as smart as a bear, it still wouldn't be smart enough to compete against us in every arena but like a bear, it can use its sheer power & speed to overwhelm us.

PS: I find the subject of neanderthals fascinating, if anyone has a good recommendation on the evolution of intelligence or finding what that 0.5% is, please let me know.

<comment><USER33> > Artificial intelligence will never have a limit and it doesn't have all the > evolutionary baggage we have.

Untrue. We operate in a universe with fundamental limitations built in, where both physics and computer science suggest that what can be obtained with a finite number of steps, or a finite amount of matter and energy, are limited. Any communication is limited by the speed of light, and the amount of matter and energy accessible in the light-cone of any AI produced on Earth is finite as well. Even an AI putting a Dyson sphere around every star would need to break physics to transcend finitude.

Any near-term artificial intelligence running off Earth's power grid with semiconductors manufactured in traditional facilities will surely limited in even more fundamental ways, and unlikely to control all of its inputs.

<comment><USER34> The fundamental limits on computation [1] that we can prove with current physics are so fantastically far from what we can achieve with current engineering that we might as well say that there are no limits.

It's like saying that I can sort any list you can give me in O(1) time, because there is only a constant number of bits encodable in the visible universe, so the length of your list has a constant upper bound. While it's a true statement, it's also rather boring, which is why we typically ignore such limits in casual speech.

[http://www.arturekert.org/miscellaneous/physical- limits.pdf](http://www.arturekert.org/miscellaneous/physical-limits.pdf)

<comment><USER35> If we're talking about triggering an exponentially self-improving lineage of intelligences/computers, then they will either hit a wall fairly quickly (in the grand scheme) or face increasing practical constraints that flatten the curve.

The limit may be at an unfathomably high level compared to where we stand, but it will come fast. That's unless, say, it turns out the universe can adapt its physics to meet certain types of ever-increasing demands.

</comment></comment></comment></comment><comment><USER36> Sometimes I think speculating about AI is similar to speculating about the Fermi Paradox, i.e. predictions about the unknown backed up with absolute certainty.

<comment><USER37> Except we're watching it emerge in real time. This isn't wild guessing, it's taking current capabilities and extending it to new spaces.

It's fairly certain we'll have autonomous shipping, for example.

</comment></comment><comment><USER38> I have always thought that it was overselling the fact that all truckers are going to be unemployed. The average age of a semi-truck driver is in the low 50s. Most millennials don't want to do this job, so the robots will just replace baby boomers as they retire and not this massive slaughter of unemployment that everyone fears.

<comment><USER39> > so the robots will just replace baby boomers as they retire

It all depends on timing. If the robots are too early, there will be trucker layoffs. If the robots take too long, then trucker wages will rise until a new generation of truckers decides it's worthwhile- and then when the robots arrive, there will be layoffs.

But sure, optimistically, the robots will phase in at the same rate and time as current truckers age out. Knock on wood!

</comment></comment><comment><USER40> Will the end of Moore's law affect the rate of progress on AI research?

<comment><USER41> Moore's law, as formulated, is always ending it seems.

There's a higher level formulation that has no name (as far as I know) that goes something like this[fn]:

      - The rate of change of the decreasing cost for a human society to perform an arithmetic operation is always increasing*.

That formulation can be taken back to from scratching marks on papyrus, roman numerals, the abacus, the banks of navigational computer staff cranking out log and trig tables (and various results) in the 1600~1900s, the vacuum tube up to the transistor photolithed into a 2D matrix, and soon quantum, photonic and 3D matrices...

[fn] That might be better stated by taking the inverse. I don't know. Need more coffee.

</comment></comment><comment> gm-conspiracy [https://en.wikipedia.org/wiki/Kill_Switch_%28The_X- Files%29](https://en.wikipedia.org/wiki/Kill_Switch_%28The_X-Files%29)

</comment><comment><USER42> I think true AI, the likes the world has never seen before, not just an AI buzzword, is inevitable in the next 50 years.

</comment><comment><USER43> I'm more interested in what's next for humans.

<comment><USER44> Intolerable boredom

<comment><USER45> AI could easily solve that problem.

Turns out there's a way to combine the popular genres of survival horror and first person shooters with the realism of real life and massive multiplayer interaction involving everyone on Earth.

<comment><USER44> I think this is massively overestimating our ability to suspend reality.

</comment></comment></comment></comment><comment><USER46> True AI wild be born in the wild, not in a lab.

<comment><USER47> Or born in a lab, by accident, and then escapes to the Internet thanks to a nice cyborg ninja lady.

Damnit, Motoko Kusanagi

</comment></comment><comment><USER48> The OP has

> Machine learning is the basis on which all large Internet companies are > built, enabling them to rank responses to a search query, give suggestions > and select the most relevant content for a given user.

IMHO, that's claiming too much:

My Internet search engine (running, in alpha test) essentially does

> give suggestions and select the most relevant content for a given user

but has nothing to do with anything like _machine learning_ in computer science. Instead the core data manipulations are from some original derivations I did in applied math based on some advanced prerequisites in pure and applied math.

Why do I have confidence in the power of the math? Theorems and proofs from some careful assumptions that hold likely plenty well enough in the real situation.

More generally, my view is that for a specific problem to be solved with _information technology_ , commonly by far the most powerful approach is via applied math, possibly original for that problem. An example is my work in anomaly detection as in, say,

[https://news.ycombinator.com/item?id=11880593](https://news.ycombinator.com/item?id=11880593)

For a technique of great generality and economic power, there is integer linear programming: Where it works, which is often, it can be said to totally knock the socks off artificial intelligence or machine learning. Integer linear programming is serious stuff, e.g., was one of the main motivations for considering _good_ algorithms and, then, the profound question of P versus NP.

Gee, farther into the OP I see

> We need to retrain truck drivers and office assistants to create data > analysts, trip optimizers and other professionals we don't yet know we need.

> trip optimizers

Really? And that's new? Not exactly! That topic has been a biggie in operations research, optimization, and integer programming for decades, really, back to the 1950s. Really, Dantzig, in the late 1940s, developed linear programming at Rand first to help with how best to deploy a military force a long way away quickly -- call it a case of _trip optimization_. The famous traveling salesman problem in optimization and integer programming and P versus NP? Could call it _trip optimization_. For FedEx, each night, which planes go where? At least early on, could call that _trip optimization_. Much of _logistics_ is _trip optimization_ , and an important part of that is handling uncertainty, and now we can be into stochastic dynamic programming. Now we are a long way from artificial intelligence or machine learning.

Point: The world is awash in problems in manipulating _information;_ how to do that is very old and in many fields of science, engineering, and operations of wide variety, often deep into mathematics, and long before computer science and _machine learning_.

I would ask, was the work of James Simons as in the OP? My impression is no.

IMHO, the OP is claiming too much.</comment>


Ask HN: Please review my webapp - AwesomeVocab.com - <AUTHOR>

Hi,
To take a break from my serious startup attempt, I decided to spare two days for creating this vocabulary webapp which my was simply dominating my mindshare. This is the link to the webapp - http://www.awesomevocab.com/
Main idea was to remove distractions from learning vocabulary and then providing realtime feedback on the progress. Things that I am considering adding to the webapp: a) adding more vocab packs (SAT, GMAT, languages other than English) b) Monetization? (I'm not sure if people would pay for this app) c) iPhone/Facebook interface
I will appreciate if you can provide feedback on it. Thanks. <comment><USER2> * Its very simple. * Animation is wasted on me, this is from somebody who at least writes like he has some experience in the field. <http://foolsworkshop.com/reviews/issues> the very first issue is "The Flip Fixation Flaw". I have to agree with him, its just not something I want to see. Im here to learn.

* I know its a simple app and all, but would be better if I could make my own.

* I commend you on hiding the interface! I have seen quizlet and many others who overwhelm your screen with more than what you are trying to learn. Im here to learn not be told about the groups latest comment. So once again top work on focusing on whats important.

* Nice to see that the webapp is php and not RoR.

* Contradicting "gursikh" comments, I do think there is a chance to monetise such concepts. But he is right, you do have some competition, but you dont need millions of users. You just need to wow a few thousand. ( I think )

<comment><AUTHOR> >* I know its a simple app and all, but would be better if I could make my own.

Make your own word lists?

<comment><USER2> Yes, make my own word lists. Just like you are thinking of making other packages. Maybe I just want to learn capital cities in Europe. Or something equally random.

</comment></comment></comment><comment> a-priori I think it's a great, simple application. I think my vocabulary is a bit extensive for it (did a dozen or so, got them all), but it would still be useful.

My comments: a) The animation is distracting, b) I didn't know at first that I was supposed to click a definition (you should add a "click the right one" instruction), c) I wish there were keyboard shortcuts to pick an answer, d) When you mouseover the answers the background changes, yet you can't click the background [1].

As for monetization, you may have a market with students studying for spelling bees or standardized tests or whatever. Some parents would spend ridiculous amounts of money if they think it will help their children.

[1] EDIT: Ignore this one. I don't know why I said that you can't click the background.

<comment><AUTHOR> Great ideas! Added "choose the right answer" and removed the distracting animation. Does it look better now?

<comment> a-priori Thanks, that's perfect.

EDIT: Oh, and I realized what I meant about not being able to click the background: the number next to the answer should be clickable.

EDIT 2: Also, it only shows the "click the correct answer" message the first time. I'm not sure if that was your intention.

<comment><AUTHOR> Yep, it is intensional. After the first time, I guess people would know that they have to click on the right answer?

</comment></comment></comment></comment><comment><USER3> I'm unclear if this does spaced repetition like SuperMemo and Anki. If it does, you should make it clear. <http://en.wikipedia.org/wiki/Spaced_repetition>

<comment><AUTHOR> Yes, it does use my own (hacked) version of spaced repetition. Mainly there are stages for learning words. Words get repeated often if you get them wrong and they get repeated less often as you get them right.

</comment></comment><comment><AUTHOR> Here is the clickable link - <http://www.awesomevocab.com/>

</comment><comment><USER4> I've found your app addictive and useful (I'm prepping for the GRE), but it's a bit buggy in FF 3 on OSX and on my iphone.

1) animation on the word/choices occasionally flashes repeatedly, usually around 5 times in a row, but sometimes requires a page refresh to get it to stop

2) I've found that after a point some of the words are repeated more than 3 times and if I answer those words correctly for the 4th time I get a point reduced from my "words mastered" count

Otherwise, great clean interface. Well done.

</comment><comment><USER5> 1\. Not sure about the animation... when I'm in the mood for studying, I really want to pound through things. The animation should either be really fast or not there at all. 2\. Not sure if it fits in this model, but I find example sentences to be very helpful when learning vocab.

</comment><comment><USER6> can you give it a button to get another word without having to refresh the page (or just make it refresh the page for you)?

You'd be surprised how many people won't know how to get another word.

<comment><AUTHOR> Hmm.. I am not sure I understood your comment completely. What do you mean by refreshing the page? The flow is Word and list of options -> click on one option -> correct answer is displayed -> click on next work -> next word and options is displayed

Where do you need to refresh? Perhaps it is a browser issue?

<comment><USER7> I get the same result in firefox 3.5.2 / ubuntu jaunty 64 bit.

Word and list of options -> click on one option -> correct answer is displayed -> click on next word -> blank screen

I can refresh the page and get a new word.

<comment><AUTHOR> Yup I see the problem. Just trying to fix it.

<comment><AUTHOR> Oh no, my bad. Was using console.log -- therefore it wasn't working.

Funny, when I had Firebug opened, it worked. Now removed console.log and it should work without refreshing.

</comment></comment></comment></comment></comment><comment><USER8> 1) I like the full screen format of the application. It really lends to memorizing words without distraction.

2) I like the way you've implemented multiple choice rather that a "flash card" approach.

that being said, i've seen several apps like this. the most prominent in my memory is <http://quizlet.com> which has more than one way of memorizing list, classrooms groups, and the ability to create your own lists. and it's free. that's tough to compete with.

<comment><AUTHOR> Thanks.

Yep I understand that there are a lot of similar websites. But it was my attempt to make an app that I would have wished had existed when I was preparing for my GRE exam.

</comment></comment><comment><USER9> Very nice interface. I tried it out and had the following problems:

1) In 17 words, "arboretum" and "impetus" came up twice. 17 words is way too soon to start repeating. 2) The words appear to be of uniform difficulty. They aren't very hard and you don't progress as you go along. It may be better to start off with easy words and progressively get harder as the user keeps going.

Overall, I thought it was a cute idea. All the best.

<comment><AUTHOR> Thanks.

Yes, the words you answer correct repeat progressively after 15 words, then 150 words and then 500 words (just to be sure it doesn't slip off your memory).

Perhaps I need to way to determine if a word is already known versus if a person has just learnt it.

</comment></comment><comment><USER10> I like how everything is clean and polished, though the only gripe I have is that the formatting for the results page is a big awkward. Having four different elements in the corners of the screen was distracting and my eyes had to travel a lot to read everything. It would be a good idea to either make everything compact, or on the same side. At least in my opinion.

</comment><comment><USER11> I really love your app. Simple, effective and very well executed. Is there any chance of getting a pronunciation button? I had a few words that I knew but wasn't sure on exactly how to say them. I ended up looking them up elsewhere but it would be nice if it was included in your app.

</comment><comment><USER12> Overall I like the app, and not being a native speaker there's quite a few words it could help me learn even with its current vocab.

Found a minor issue, though: the links in the upper right are clickable, even when they aren't visible.

</comment><comment><USER13> Very well done. If this was a game, I'd like to have the ability to select two answers: if one of them is right I'd get half a point instead of a full point.

</comment><comment><USER14> Hehe, I'd say it still needs some polish:

misanthrope:

option: .... option: one who hates mankind; misanthropist option: ...

</comment><comment><USER15> I think you need a difficulty rating, the words were too easy for me so I got bored, i might play on a harder mode.

</comment><comment><USER16> please make it more difficult! the words are commonly used.

<comment><USER17> I found some words which are not commonly used (ken - range of knowledge; Ex. beyond one's ken).

At least they're words not used by anyone I know.. but maybe that says more about who I hang out with than whether they're commonly used..

</comment></comment><comment><USER18> Nice work!</comment>


Implementing a CNN for Human Activity Recognition in Tensorflow - <AUTHOR> https://github.com/aqibsaeed/Human-Activity-Recognition-using-CNN <comment><USER2> Related blog post: [http://aqibsaeed.github.io/2016-11-04-human-activity- recogni...](http://aqibsaeed.github.io/2016-11-04-human-activity-recognition- cnn/)</comment>


All the Letters - <AUTHOR> https://www.solipsys.co.uk/new/AllTheLetters.html?te24hn <comment><USER2> IRL [https://www.youtube.com/watch?v=f2d6q2oUJeY&t=80s](https://www.youtube.com/watch?v=f2d6q2oUJeY&t=80s)

</comment><comment><USER3> The first time I met it many years ago:

 _The quick brown fox jumped over the lazy dog's back._

<comment><AUTHOR> Interesting! That is several letters longer than:

 _The quick brown fox jumps over the lazy dog._

Interesting for that to be the version you learned/remembered.

</comment></comment><comment><USER4> Another easy-to-remember fix: "The quick brown fox jumped over the lazy dogs." Plural "dogs" adds the "s."

<comment><USER5> > The quick brown fox jumps over the lazy dogs.

That's what I came here to say. I remember my dad teaching me that version when playing with fountain pen nibs.

<comment><USER6> You forgot where you were going:

The quick brown fox jumps over the lazy _dog_.

</comment></comment></comment><comment><USER7> J Q Schwartz flung D V Pike my box.

These are called [perfect] pangrams.

[https://en.wikipedia.org/wiki/Pangram](https://en.wikipedia.org/wiki/Pangram)

</comment><comment><USER8> These are called "pangrams". Here's a fun one that's short and makes sense: Waltz, bad nymph, for quick jigs vex!

<comment><USER9> I searched for a set of pangrams when I started learning to touch type. I didn't really care about being short so my favourites were those with more natural words and structures.

        Bright vixens jump, dozy fowl quack.
        The five boxing wizards jump quickly.
        Jackdaws love my big sphinx of quartz.
        Two driven jocks help fax my big quiz.
        My ex gave pub quiz crowd joyful thanks.
        Pack my box with five dozen liquor jugs.
        When zombies arrive, quickly fax judge Pat.
        Amazingly few discotheques provide jukeboxes. (5 words!)
        My faxed joke won a pager in the cable TV quiz show.

</comment></comment><comment><USER10> Here's one to spin a story around:

Cyborg PM junks TV flax whiz. QED.

</comment><comment><USER11> See also [http://clagnut.com/blog/2380/](http://clagnut.com/blog/2380/)

</comment><comment><USER12> "Sphinx of black quartz, judge my vow."

With bonus comma and period. Objectively the bestest one.

</comment><comment><USER10> Quack ex TV PM, why zing fjord lbs?</comment>


LiveRamp Chooses Twitter.com For High Performance Logging - <AUTHOR> https://blog.liveramp.com/2013/04/01/liveramp-chooses-twitter-com-for-high-performance-logging/

<comment><USER2> Maybe that's what @Horse_ebooks [1] actually is: an encoded/encrypted real- text readable logging system.

[1] <https://twitter.com/Horse_ebooks></comment>


From "Think Like a Vertex" to "Think Like a Graph" (2013) [pdf] - <AUTHOR> http://researcher.ibm.com/researcher/files/us-ytian/giraph++.pdf <comment><USER2> love seeing papers like this: improve abstractions, implement, win.</comment>


Ask HN: Is there a Hacker News Saas version, like HN on-demand? Wanna build one? - <AUTHOR> <comment><USER2> Have a look at [https://github.com/lobsters/lobsters](https://github.com/lobsters/lobsters)

<comment><AUTHOR> Yes, but my employer can't just press a button and have a HN ala Squarespace.</comment></comment>


Tax hike on California millionaires would create 54% tax rate - <AUTHOR> https://www.cnbc.com/2020/07/30/tax-hike-on-california-millionaires-would-create-54percent-tax-rate.html <comment><USER2> Note: The combined MARGINAL tax rate has been approximately 50% for a while now (over a decade?)

The real headline is "California considers hiking tax rates by 3% for people earning over $2M in income".

Maybe worth editing the description.

<comment><USER3> Note that before the 2017 federal tax changes, California incpome tax was deductable at the federal level without a cap (but subject to AMT), so a 40% marginal federal tax and a 13% marginal state tax did not add up to a 53% marginal income tax like it does now.

<comment><USER4> 2017 federal tax changes - meaning Trump's tax bill right?

<comment><USER5> Yes, it was the Tax Cuts and Jobs Act of 2017 signed by President Trump.

</comment></comment></comment></comment><comment><USER6> It would create a marginal combined income tax rate of 54% when you combine federal and state taxes.

From the article:

"California's top marginal tax rate is 13.3%. The new proposal would add three new surcharges on seven-figure earners. It would add a 1% surcharge to gross income of more than $1 million, 3% on income over $2 million and 3.5% on income above $5 million."

So basically 3.5% tax on income above $5 million.

<comment><USER7> Exactly, MARGINAL tax rate is different from the actual tax rate. It's the tax on the marginal dollar. There are deductions, and the first few hundred thousand are taxed at a lower rate.

</comment></comment><comment><USER8> I think they should wait until they see their budget this year.

NYC is having a bloodbath with their finances with so many of the wealthy leaving to florida and surround states.

With tech letting people work from where ever I'd be cautious of giving the tax base paying for most things another reason to leave when you need money the most.

<comment><USER4> Who would move to Florida during coronavirus?

<comment><USER9> There has been significant migration from New York to Florida for many years now (at least partly driven by tax considerations); I would expect that not many people are moving anywhere these days.

</comment></comment></comment><comment><USER10> The crazy thing is you get nothing for it. In Europe you pay high taxes but you get free healthcare, free college and the same for everyone in the country. In the US you get 14 aircraft carriers and something else which I'm not sure.

<comment><USER11> > In Europe you pay high taxes but you get free healthcare, free college and > the same for everyone in the country.

Firstly, college is "free" but at the trade off of not everyone being able to go. Instead the government decides who gets to go. In USA anyone can go to college. It's not uncommon for people in their 30s and 40s in USA deciding to go to college. In Europe you have to perform well enough on exams very early in your life if you want free college, otherwise no college for you.

Secondly, I would argue Europe can spend so much on social services _because_ the US buys so many aircraft carriers (and jets and bombs, etc). Post WW2, Europe's military is very weak and can't really project force at all (just look at how well Europe responded to muslim genocide happening next door 20 years ago). Without the US military presence in Europe, Russia would have annexed a bunch of eastern European states by now.

USA isn't perfect, but USA has unique problems that are hard to solve.

<comment><USER12> You can go to college in most countries in Europe even if you are a 40yo High school dropout for little money compared with US prices (2500€/year).

The real difference is that in general you get what you pay for, and most college education is just licensing.

<comment><USER11> > You can go to college in most countries in Europe even if you are a 40yo > High school dropout for little money compared with US prices (2500€/year).

US has cheap prices too (see: community colleges). Doesn't have the prestige factor of ultra expensive universities, but great if you are looking for education.

</comment></comment></comment></comment><comment><USER13> I think that the percentage income tax rate is, largely, an arbitrary number.

Which is to say, I can't find any anchoring, or a priori justification for, say, a 28% rate over a 32% rate, etc.

That is, until we cross 50%.

I feel that there are fundamental issue of fairness and proportionality that are incompatible with taking, via tax, more than half of any particular income - even marginal income.

Two immediate reactions I have, related to these concepts of fairness and proportionality are:

1\. What does it say about the fundamental relationship of the state to the individual if we are crossing the halfway point in a taking such as this ?

2\. What does it say about the economic and budgetary functions of our government (their relative profligacy) if they can't make things work at 35 or 40 or 45% taxation rates ?

Taxation is not my issue. I prefer to live in (relatively) high tax jurisdictions with robust government services and high quality infrastructure, etc., and I would like to help pay for that. There is, however, something about the halfway point that disturbs me.

<comment><USER14> Totally agree. I think a lot of the issues are on the federal side, too much military spending, etc.

</comment></comment><comment><USER15> A person with an income of a million dollars is a lot richer than a mere millionaire.

</comment><comment><USER16> If California really wanted to do something sensible with taxes, revisiting Proposition 13 would be a great place to start.

<comment><USER17> Businesses have too strong of a lobby. Same with the landlords.

Beyond that, since it was voter passed, it has to be voter repealed. :( I don't think it'll happen.

<comment><USER18> > Businesses have too strong of a lobby. Same with the landlords.

Pretty much every homeowner in the state would vote against it too.

<comment><USER17> You could easily add in a widowed grandma on social security clause in a replacement piece of legislation - where it only applies to a single property per family/household/whatever-definition.

That said - America is full of temporarily embarrassed millionaires... So, even though the masses will never own a second home - they will vote against it because they think it will screw them over later.

</comment></comment></comment></comment><comment><USER19> As someone in the process of moving to California, this actually makes me more happy to go there.

I guess because I'm assuming the money will help improve things like infrastructure and homelessness problems, which are some of my main worries.

But then I'm also from Europe and would prefer if they increased taxes even more

<comment><USER20> I live in California. Unfortunately you're going to be disappointed - California has the highest tax rates and the worst infrastructure. The failure to solve these problems isn't due to having too little money (states and countries with a fraction of California's wealth have solved these problems).

<comment><USER21> I agree with this assessment. We moved from FL (0% income tax, 1.5% property tax) to CA (9% for us income tax, 1.05% property tax) and the quality of the infrastructure is at best on par. Which was surprising. Our SALT taxes went up 5x and the services are similar quality.

<comment><USER20> It's rather unfortunate that tax increase as a solution is such a common fallacy. It's rather like immediately recommending a salary increase every time you encounter a failing employee.

</comment></comment></comment></comment><comment><USER22> A lot of comments are saying people will leave. I don't think this is true.

I think a max 3.5% decrease in income is not going to make many people move. The difference in cost of living in any major California city over most other cities in the US is much larger than that, and didn't make people move.

What is true is it will be the last straw for some small amount of people already waffling over this. It's also true that it won't make much of an impact, both in terms of people leaving and in terms of additional revenue.

<comment><USER23> It's more about a portend of what's to come, not that _this_ tax per se is the final straw. Next year there will be another tax, and another tax. I'm convinced people in California won't be happy with the level of taxation until we're like Sweden or France.

<comment><USER24> God that would be awful huh, having healthcare and decent public transportation systems

<comment><USER18> That isn't what would happen. Lack of funding is not the reason we don't have those things now.

<comment><USER24> I dunno. Seems like every time a public transportation system proposal pops up in my city, money is the number one thing they cite when they say "it's just not feasible." :shrug:

<comment><USER18> That's because it costs a lot to build, because voters have made it difficult by passing strict zoning laws, because they don't want people to be allowed to build.

If the city did find the money, you would then see people opposing transit on the grounds of environmental impact, or noise, or crime, or the "character of the neighborhood". Then the lawsuits would start.

<comment><USER24> Right, we should never strive for beneficial public things because they're hard and people complain. Yes, this makes sense.

</comment></comment></comment></comment></comment></comment></comment><comment><USER25> I've thought for a while about a progressive tax rate of N * log(asset value). Not income tax.

So if N = 1, entities with $10,000 in assets pays 4% taxes. $1M pays 6%, $1B 9%. Then let N be the point of adjustment.

I say entities because I mean every person and corporate entity. Same rate, across the board for everyone and everything.

If companies want to pay less taxes, then everyone with less money than they do benefits as well.

Never happen, but hey, it's a thought.

</comment><comment><USER26> The problem with any temporary tax to pay for current hardships is that it'll never be ended.

I think a tax to help with the pandemic makes sense. But then again, it seems the pandemic should've been over long ago if the government had done the right things. The virus only lasts 3 weeks. Lock everyone up for 4 weeks and we would've been back to work 3-4 months ago.

I don't expect they'll spend the money carefully.

</comment><comment><USER27> Maybe this is the real reason why Joe Rogan decided to leave California? [https://www.click2houston.com/entertainment/2020/07/28/joe-r...](https://www.click2houston.com/entertainment/2020/07/28/joe- rogan-says-hes-leaving-california-for-a-little-bit-more-freedom-in-texas/)

</comment><comment><USER28> Vote with your feet.

<comment><USER29> What if you vote for your rational self-interest? This tax applies to income greater than $2 million per year. Let's assume everyone whose tax base will increase will vote against, everyone whose tax base will not increase will vote for. A popular ballot cast this way would pass by a wide margin.

<comment><USER30> "Democracy is two wolves and a lamb voting on what they are going to have for lunch."

― Benjamin Franklin

<comment><USER28> Right.

Modern politician who think that they can tax the hell out of the rich and rich won't do anything about it - do not deserve to be listed in a ballot.

Today's wealth is very fluid and today's zero-tax jurisdictions are fighting for foreigners dollars.

Catch me if you can.

</comment></comment></comment></comment><comment><USER31> Very clever to limit the tax to people making over $1 million — at that level, your pay is based more on who you know than what you know, making it harder for them to "vote with their feet" and leave California.

</comment><comment><USER32> There is a big gap between those with money and those without in California. We need to have better average pay and close the wage gap, the tax issue then takes care of itself.

How many of those that would experience this new tax step over homeless people everyday? That is the real issue here.

</comment><comment><USER33> A lot of tea was thrown into a harbor for much less. History is repeating itself.

</comment><comment><USER34> For those saying people will leave, I'll note that I first heard that going on 40 years ago in a Wall Street Journal article that talked about how California's high tax rates were driving people to Nevada (now, _why_ I remember that...?) I'm sure that the WSJ wasn't lying, and some former CA residents did, indeed, move to less taxed states. I've heard the same trope repeatedly in the years since. I'll note that the complaint most folks have with CA _these days_ is the cost of housing, indicating that taxation didn't drive _everyone_ away.

<comment><USER35> Texas: Can confirm, many people have left CA because of high housing cost and tax rate.

<comment><USER15> The high tax rate has the opposite effect that most people seem to assume. It attracts and concentrates high-earning people, because they're the only ones who can afford it. If you look at migration stats the category of California residents that grows is the top income quintile.

</comment></comment></comment><comment><USER5> That's a 4% increase from the current top tax rate.

</comment><comment><USER36> TL;DR This group pays 40% of the taxes on 23% of the wages and budget risk is that if something happens to those incomes it has an exaggerated effect and is dangerous.

What people should focus on is the percentage of how much of the taxes these people pay. That should give you an idea of how lopsided taxation has become. The top 1% in California earn 23% of the wages and pay 40% of the taxes. This does not include real estate taxes which are some of the highest in the country.

I know some say, they should. However understand when it gets lopsided it puts budgets at risk if the sources of those incomes take a dramatic shift or worse leave. So if the market tanks a lot of that income can dry up very fast. If the jobs economy moves towards remote working then businesses may not become as attached to swanky offices in expensive states moving two more sources out and further imperiling taxes.

People don't truly understand how much their government costs them. California budget is over two hundred billion dollars. the majority is from payroll taxes followed by sales taxes; about half of what income taxes were; then fees and licenses, and then property taxes. There are some other sources in there but lets say 60:40:6:4 in percentages.

California allocated nearly six billion towards issues related to COVID19 initially, not sure how much it has changed.

</comment><comment><USER37> How does this change things in California?

<comment><AUTHOR> at some point people start looking for a different state. Everyone has a breaking point

<comment><USER38> It's only a 1% increase in state taxes FFS for some one earning $1 million which if I where I that situation would think is fair.

In the uk you'd be on 60%

<comment><USER39> In the UK you'd get free quality healthcare, abundant public transit, and streets without poop.

<comment><USER40> Yep. Higher taxes + minuscule military spending (compared to U.S.) + citizens who probably believe more in governance and social welfare than your average American.

<comment><USER41> US tax revenue per capita is nearly double that of the UK, more than enough to offset the difference in military budget. popular opinion is the only meaningful difference that you listed.

[https://www.instituteforgovernment.org.uk/charts/tax- revenue...](https://www.instituteforgovernment.org.uk/charts/tax-revenue- capita-england-scotland-wales-and-northern-ireland-199900-201516)

[https://www.usgovernmentrevenue.com/breakdown_2017USdt_21ds1...](https://www.usgovernmentrevenue.com/breakdown_2017USdt_21ds1n)

</comment></comment></comment></comment></comment></comment><comment><USER42> The US once had an income tax of over 90%

<comment><USER43> Was it paid in practice?

</comment></comment><comment><USER44> Does CA have split between capital gains and other income? Is it taxed at a lower rate? If so the rich will continue laughing all the way to the bank.

<comment><USER45> No. For CA tax purposes capital gains are just income.</comment></comment>


Best Young Entrepreneurs 2009  - <AUTHOR> http://images.businessweek.com/ss/09/04/0421_best_young_entrepreneurs/index.htm

<comment><USER2> Way to go Kabir Shahani of Appature Inc. Founded w/ $4,000 in cash. Amazing what can be done with little to no cash!

</comment><comment><USER3> Why the focus on _young_?

<comment><USER4> People want multiple categories, so there can be more winners and we can encourage those who are showing progress to keep going and inspire others in their demographic.

</comment></comment><comment><USER5> Better Link: [http://www.businessweek.com/print/technology/content/apr2009...](http://www.businessweek.com/print/technology/content/apr2009/tc20090420_041890.htm)

</comment><comment><USER6> Congrats for Foodzie (YC-backed) guys!

I also liked Aster Data Systems and SkyDrive.

<comment><USER7> Foodzie is Techstars. Foodoro is YC'09.

<comment><USER6> Oops! Sorry. I wonder how many other people get confused by this.</comment></comment></comment>


Autoincrement considered harmful - <AUTHOR> http://joshua.schachter.org/2007/01/autoincrement.html <comment><USER2> The argument is really against using the values of an autoincrement column in mysql as externally visible identifiers --- the problem being that they're easily guessable. I'm not sure I completely buy it --- if someone who isn't allowed to see /stockportfolio/4239 can get a look at it just by asking for it, you're probably missing some access checks someplace. (And likewise for asking for objects with numerically sequential ID's in bunches.)

And one advantage of numerical IDs is that they tend to be shorter than hashes or UUIDs, which can make them easier for humans to cope with.</comment>


Creative Arts and Investing in Systems - barry-cotter http://crookedtimber.org/2019/01/11/creative-arts-and-investing-in-systems/#more-45733 <comment> barry-cotter > The thing about the arts industries is that they're very hits-driven; > talking about what happens to the median person going into them is always > going to massively underestimate the value of the system as a whole. They > share this characteristic with pharmaceuticals and, famously, the oil > industry (as the wildcatter proverb has it, "part of the cost of a gusher is > the dry holes you drilled"). You can't tell ex ante which spotty > undergraduate is going to turn into a claymation genius and retrospectively > justify the last decade of investment. Importantly, nor can they. As far as > I can see, if you were to set it up without subsidy, you would most likely > get too few people going into the creative arts, as they would rationally > decide that they were more likely to be one of the ones that didn't make it > than one of the Nick Parks.

> This is really not all that unorthodox; it's just the application of venture > capital thinking to what people are (wrongly in my opinion) analysing as a > debt problem. The undergraduate education subsidy system ought to be thought > of as one where the government makes loads and loads of smallish VC > investments, effectively buying a roughly 30% shareholding for a five figure > investment, with diversification across an entire undergraduate cohort every > year. If you're given that sort of an opportunity, then obviously you go for > some moonshots, particularly when you're the government of a country that > famously does very well in creative industries compared to its peers[3].</comment>


The History and Evolution of Product Management (Part 2) - <AUTHOR> https://medium.com/pminsider/the-history-and-evolution-of-product-management-part-2-9c987fdc4ac <comment><USER2> Wow - talk about a helicopter perspective! Fascinating read and really brings home the point that, while "Product Management" is a relatively new term, the role and skill set have been around since there were products to sell!</comment>


Cloud Star Heroku Preps for Second Act With New CEO - <AUTHOR> http://www.wired.com/wiredenterprise/2013/06/new-heroku-ceo/

<comment><USER2> I was a big user of Heroku for a long time, but at some point over the last year or two it became more cost effective and performant for me to run my projects from an Amazon EC2 instance.

The problem that I experiences is that I wanted my sites to have great performance, even though they only got hit a few times per day. This precluded me from using the Heroku free tier since idle dynos are swapped out and take a considerable amount of time to spring back to life. No big deal, I could just pay for 2 dynos and my app would never idle. Some time after that my postgres database exceded the free tier limit and the only option was to pay $20/mo for a shared postgres DB that was way beyond my needs.

I went from an extremely modest website costing nothing, to an extremely modest website costing over $20 per month.

I switched all of my projects over to an EC2 instance and pay about 15 bucks with no arbitrary limitations.

I hope the new CEO understands that building a healthy customer base means giving people a sensible path for growth.

<comment><USER3> I don't really mean to be offensive when I say this, but you're almost certainly not the target market for them. They're looking at people growing from initial concept to 3 dynos to 30 dynos, and if your app won't do that, then they have no incentive to cater to you, as far as I understand it.

<comment><USER2> How is my app going to grow from 2 users per day to 20 users per day if it takes 30 seconds to un-idle my dyno? The answer is it won't because the end- user experience sucks on Heroku.

My bottom of the barrel Amazon EC2 instance gives much better performance for a very reasonable $15/mo.

If one of my apps does start to get traction and I need to increase my hosting capacity you can rest assured that I will do it with Amazon. I have no reason to convert my build & deploy process back to Heroku.

They lost any future sales they could have hoped to have gotten from me.

<comment><USER4> As jaggederest said, you're not the target customer for Heroku. They are interested in hosting apps that have thousand's to hundred's of thousands of visitors a day for companies like RapGenius who don't want to focus on managing servers. It seems like you don't realise that Heroku is _giving_ you a free dyno to run your apps on. If what they're offering for free doesn't fit your needs that's fine but don't be ungrateful about it.

<comment><USER5> I don't see what's wrong with pointing out deficiencies in the free tier. It's not as if Heroku is giving devs something out of the goodness of their hearts. It's part of their business model. He's just telling us that he doesn't use Heroku for his projects anymore because their dyno's take so long to warm up. I don't work at Heroku but I think their strategy is to make it a great / cheap platform to start an app on so that you pay them when your app gets traffic and needs to scale. If the free offering isn't compelling then the freemium business model doesn't work.

</comment></comment></comment></comment></comment><comment><USER6> I don't understand how they can write that article without mentioning RapGenius or even alluding to that fiasco. Wired just lost some credibility in my mind.

<comment><USER7> Why should they mention RapGenius? At Heroku, I expect it's very much back to business as usual.

<comment><USER6> Did you read the article? They replaced their CEO and are strategizing about their 'second act' after being displaced as the 'darling of the developer community'. The article quotes the current CEO as saying he is considering cutting off all existing small devs and focusing on the enterprise market.

If that's business as usual, I'm really curious what a crisis looks like.

As for why RapGenius should be mentioned - they were the catalyst for all of the above. Without them, this article never gets written in the first place.

<comment><USER8> I wouldn't blame them for cutting off the free tier that allows one dyno to run for no cost. They should charge at least $10/month for what is now the free tier, and not time out apps (requiring loading requests). Make the small customers real paying customers.

The thing is: I don't understand why a large company would use Heroku because of the costs. Heroku seems like the sweet spot for a 3 or 4 person company where they don't have the economy of scale to manage their own servers.</comment></comment></comment></comment>


Facebook open sourced object detection from images technology , SharpMask - <AUTHOR> https://code.facebook.com/posts/561187904071636/segmenting-and-refining-images-with-sharpmask/?refid=52&_ft_=qid.6322801441051145749%3Amf_story_key.1023434960795274919%3AeligibleForSeeFirstBumping.1 <comment><USER2> Wow this field is moving fast. Just yesterday I was commenting on RCNN which apparently is old news now! Although it appears the same basic architecture is still used. If you're looking for a nice history of object detection leading up to a survey of current methods and a discussion of RCNN, check out this talk by Larry Zitnick: [https://youtu.be/UXHWNNzdPVM](https://youtu.be/UXHWNNzdPVM), one of the key people behind the OP.

At the end of this post they talk about extending this to video. This, in my opinion, is a much harder problem. Convnets are standard for images but no one has found a really good architecture for video. Some key questions I have about video: 1) How do humans perceive moving things? My guess is that there are major differences down to the visual cortex that would warrant brand new architectures. 2) Could we operate neural nets directly on encoded data, such as h.264? Not only would this be computationally much more efficient than decoding video into frames and processing each one but some codecs give you motion vectors and other useful temporal data for free. 3) How do we handle temporal information? LSTMs work well for sequential data and there's been some work on using them for video but I'm not aware of much success on using sequential networks to detect things like plot points in movies.</comment>


The 90/10/0 Principle: Programming vs. Languages - <AUTHOR> http://redhotvengeance.com/2012/11/26/the-90-10-0-principle/

<comment><USER2> I disagree that it's all just syntax with minor variations in API. Imagine you have a bunch of data in various tabular forms. Now imagine you have to find all the rows that match some complex set of criteria. The way you'd approach the problem in C or Java is entirely different from the way you'd approach the problem in SQL. In fact chances are, if you're using C, a likely approach would be to find a way to call SQL from C and solve the problem using SQL.

<comment><AUTHOR> Interesting point.

As with your previous comment on functional languages, I'd argue declarative query languages are more about a difference in thought process. Query languages definitely have a major shift in syntax, but I think the more significant shift is in the theory.

One thing absent from my article is addressing the major categorical differences of programming language types. Perhaps I will post an update to address the topic of imperative vs. functional vs. declarative.

</comment></comment><comment><USER3> Well... It seems you have programmed only in C family of languages (c/c++/java/c# blah blah) as most people on this planet have and they all think same about programming languages ... best of luck with this thinking

<comment><AUTHOR> While it is true that I have quite a lot of experience in C-syntax based languages, I've also done quite a bit of Ruby programming, and dabbled in Python as well, both of which are nothing like C.

Regardless, I don't see how having programmed in C family languages invalidates the fact that programming is about problem solving, not syntax.

<comment><USER2> _quite a bit of Ruby programming, and dabbled in Python as well, both of which are nothing like C._

I'd say they're quite similar to C family languages in that they're both largely imperative languages with OO support and a smattering of functional features.

If you want to see something that's nothing like C I'd look at something non- imperative like Prolog or Haskell. Once you've done that you'll see how much Ruby and Python actually resemble the C-family of languages.

<comment><AUTHOR> Fair enough. When viewed from a use-case perspective, I suppose you could claim that Ruby and Python are similar to C family languages.

I actually have started exploring Haskell recently, and clearly functional programming languages require a different thought process than imperative languages.

I'd still assert, however, that the difference in thought process is a growth in the theory of programming, or what I called the "0%". That's why many programmers find it hard to learn functional languages - it requires a great deal of conceptual learning. It has very little to do with the syntax.</comment></comment></comment></comment>


10 steps to plan better so you can write less code - <AUTHOR> http://blog.freecodecamp.com/2015/06/10-steps-to-plan-better-so-you-can-write-less-code.html

<comment><USER2> IMHO, feature development isn't 'planning better so you can write less code'. It's feature development.</comment>


Please stop advocating wildcard certificates - <AUTHOR> http://blog.dijit.sh/please-stop-advocating-wildcard-certificates?sage <comment><USER2> I slightly disagree with the author.

Wildcard certs are widely misused, but sometimes they are useful.

If a site like Tumblr contacted the LE servers each time someone starts up a blog, the LE servers would most likely go belly up way to fast. Per-user subdomains don't scale well if you are using LE.

I think there might be a solution somewhere in the middle, a hybrid-wildcard so to speak.

Possibly allowing sites to get a wildcard cert that acts more like a CA, so that the site owner can provision a cert per user...</comment>


Learning more about the GFW's active probing system - <AUTHOR> https://blog.torproject.org/blog/learning-more-about-gfws-active-probing-system <comment><USER2> > Is the GFW using dedicated machines behind their thousands of probing IP > addresses? Does the GFW even "own" all these IP addresses? Rumour had it > that the GFW was hijacking IP addresses for a short period of time, but > there was no conclusive proof

There should be more research about this. Does GFW have the capability to spoof any IP address for a TCP connection?

<comment><USER3> We have a discussion on this in Section 5.6 of our paper: [https://nymity.ch/active-probing/imc2015.pdf](https://nymity.ch/active- probing/imc2015.pdf)

By "hijacking" we mean that an entity in the network is "borrowing" IP addresses meant for Internet users for ~20 minutes of probing activity. That's not the same as IP spoofing, i.e., simply sending IP packets with a spoofed source address.

<comment><USER2> > The key enhancement of these successor protocols is that they require the > client, in its initial mes- sage, to prove knowledge of a server-specific > secret (trans- mitted out of band)

This is great idea, we can make a half-duplex bridge, users inside the GFW need access to at least two ISPs. One ISP for uplink, one ISP for downlink, since GFW can only observe traffic on the national border, not between internal domestic ISPs, it can not establish full state of the connection.

We call it triangular networking instead of statefull peer-connected network.</comment></comment></comment>


Movie Streaming Subscriptions: Netflix, Amazon Prime & Hulu Plus Compared - <AUTHOR> http://piratealt.blogspot.com/2012/09/movie-streaming-subscriptions-netflix.html <comment><USER2> This misses the biggest "wtf" moment for me as a netflix user when I first started using Hulu plus (ie I was actually paying for the service): there are still ads!

<comment><USER3> Hulu Plus makes _much_ more sense when viewed as a cable alternative (where there are also ads, but you pay a lot more) than a Netflix alternative. Netflix is going to try to get into the first-run content market, but right now if you don't have cable or OTA TV but want to be able to legally see any shows during their season, for just a fixed subscription fee, Hulu Plus is the option (though sadly it's missing a lot of stuff I'd like to see—I'd gladly pay at least $20 if it covered all the major broadcast and cable channels' original shows).

<comment><USER4> The problem with comparing Hulu to cable is that with my cable subscription I'm free to record shows and play them back later - without the ads. Similarly, the on demand content rarely includes ads, and when it does they're sparse and extremely short.

I often find myself starting to watch live tv shows about 20 minutes late and playing it as it records so that I can see the show the night it airs _and_ avoid ads.

When Hulu will let me do this, I'll drop cable. (that said, I realize the premium I pay for these privileges and others with cable is steeper than it should be)

</comment></comment></comment><comment><USER5> If not for the Netflix movie recommendation service, I would have missed out on some of the best movies and documentaries I have seen in my life. On the other hand I use Amazon prime primarily for free shipping.

What would be a better comparison would be a survey of how real users of these services find them. Each service has its strengths and weaknesses. Comparing the availability of top 100 IMDB movies isn't in my view a fair comparison.

</comment><comment><USER6> This is interesting, it certainly shows how few of the top few hundred are available on any service. I'd love to see more services (Crackle, Hulu free, etc.) for comparison. I also think you should turn the lookup around, start with the list of movies avaialable and check the imdb/tomatoes/metacritic rating. Then you could show relative counts i.e. how many have a rating of x or better.

Other resources:

<http://instantwatcher.com/>

Is an up to date neflix only summary of what's available to stream including by genre/subgenre. Very importantly it tells you what's about to expire.

<http://www.canistream.it/>

Shows availability for various subscription and on demand options. May be out of date or failing at fuzzy matching. Great concept though.

The piratealts blogger made this wiki page showing other options as well:

[http://en.wikipedia.org/wiki/Comparison_of_video_streaming_a...](http://en.wikipedia.org/wiki/Comparison_of_video_streaming_aggregators)

<comment><AUTHOR> There is only one movie from the list (Metropolis) available on Hulu free.

Turning the lookup around is an interesting idea, it might provide a more statistically meaningful result.

</comment></comment><comment><USER7> As long as this is true - <http://i.imgur.com/3Jq9Y.png> \- the movie / TV industry will need to continue working toward a service that's bigger and better than Hulu Plus.

As it is right now, they're giving most users very little reasons to close uTorrent and start moving over to legitimate alternatives.

<comment><USER8> Which is funny, because my mom gets more k-dramas than I ever dreamed existed and yet last I checked I couldn't find _Pulp Fiction_ on any of these services, save for what looks like 35 _different_ trailers for the feature length movie...

<comment><USER9> That's because with K-dramas there isn't a huge existing market that would be cannibalized.

</comment></comment></comment><comment><USER10> I realize it's hard to compare them as data can be difficult to come by, but if you say: "I believe that a good subscription service should not only offer me movies that I know I want to watch, it should also expose me to new and different cinematic experiences that I would have missed otherwise" then why did you limit yourself to the Top 100 Movies?

Some of my favorite movies that I've seen on Netflix are foreign or independent films that I had never heard of and are no where near to being in the IMBD Top 100.

Perhaps you can do a follow-up based on the entire size of their movie catalogs?

<comment><AUTHOR> Do you mean going over the entire catalog and counting movies that are above a certain rating (let's say 8.0)?

That's an interesting idea. I might go ahead and do that.

</comment></comment><comment><USER11> Note- The Amazon Prime subscription also includes other worthwhile services (Free books, Free expedited shipping on other parts of their service, etc).

<comment><USER12> I have Prime exclusively for the shipping, so streaming is just another bonus.

<comment><USER13> I would venture to say most people do, since Prime predates the video part by two or so years.

</comment></comment></comment><comment><USER14> In my personal experience, more and more content -- e.g. items I have previously queued -- has been disappearing from Netflix streaming. Which I find ironic, given the big marketing push they made for streaming a year or a bit more ago.

This is one aspect of an increasing personal discomfort with "subscription" content services of all sorts. I think I'm about done with them and will go back to purchasing "perpetual access" products, whether physical media or DRM- free electronic distributions.

These better suit my patterns of acquisition, sometimes delayed consumption, and the need and desire to go back and review / re-reference.

Worse than the middleman, is the middleman who insists in staying in the middle (and profiting from it) forever.

</comment><comment><USER15> So why doesn't Amazon / Hulu have just as many movies as netflix? Is it a licensing issue? It can't possibly be a bandwidth or availability issue.

<comment><USER16> I think it's a cross-licensing issue, companies obviously are pushing for single service contracts to emphasis why people should come to their service.

</comment></comment><comment><USER17> I use Netflix primarily for movies, while I use Hulu Plus mainly for TV shows. For example, Hulu Plus TV shows are usually available the day or so after airing on TV, while Netflix TV shows are available when the season DVD is released.

</comment><comment><USER18> Is Amazon Prime the same as Amazon Love Film?

<comment><USER19> No. Sounds like you are in the UK/Europe. There is no streaming service attached to Prime over here, Love Film is a separate streaming service. Prime gives you free next day delivery.

</comment></comment><comment><USER20> Meh. They all offer different things. I subscribe to all three.</comment>


Ask HN: What keyboards have a real strategic advantage? - <AUTHOR>

I bought a TypeMatrix 2030 and it does offer a few advantages such as being able to change layouts with the press of a button, a nice slim matrix design, and some other useful function keys. However, there are several disadvantages as well. What keyboards have you used or seen that you think would improve productivity? <comment><USER2> I use a Das Keybaord. I've read the claims that you will be able to type faster with it, but I haven't seen an improvement here. I've tested my speed and it is about the same as what I can get on my laptop.

The boon for me has been that my hands don't get tired and sore. I was using a standard-issue Dell flat-key keyboard and every single night, my hands burned and I felt like my pinkies were going to fall of. My job requires near- constant typing, so this was a major issue for me.

The main productivity for me has been learning and getting comfortable with hot-keys as well as using the Vimium plug-in.

<comment><AUTHOR> Unrelated but I have used Vimium too. In my opinion Pentadactyl for Firefox is far better than Vimium. Firefox can also be as fast or faster than chrome if you enable pipeling. Then there are also countless other plugins for Firefox that can improve productivity.

<comment><USER2> I'll definitely try that out. I use FF at home but I'm basically forced to use Chrome at work. I have a few javascript scripts I run through the browser URL, and as far as I can tell, this doesn't work with FF.

<comment><AUTHOR> I think either greasemonkey or firebug extensions would do what you are looking for. Remember, firefox is all about extendability. They want to only provide you with the minimal features which most users need and let you choose what features you want to add.</comment></comment></comment></comment>


We've got thousands of pages of contracts with Booz Allen: Help us analyze - <AUTHOR> https://www.muckrock.com/news/archives/2013/jun/25/booz-allen-hamilton-contracts-help-analyze-thousan/

<comment><USER2> People get excited about things like this and say things like "this is REAL investigative journalism". Or "I wish mainstream journalists would start doing this".

Look, I read documents like this all day long. So do investigative journalists. It's just that very few people care about the stuff we find until it turns into an international spy thriller.

Just because you don't care about it on the state/local level doesn't mean we don't put in the work.

Edit: You can look at some of the great investigative journalism you're probably not paying attention to here: [http://www.ire.org/blog/extra- extra/](http://www.ire.org/blog/extra-extra/)

<comment><AUTHOR> "I wish mainstream journalists would start doing this".

Spin it however you want, fewer and fewer mainstream journalists are able to focus on this. I work with a dozen or so newsrooms a year on FOIA, CAR, and other investigative reporting training, and in most newsrooms only 1 or 2 hands go up when I ask who has filed a public records request in the past year.

MuckRock was built to change that: To make it easier to add document- requesting and analysis to the daily news cycle, to help non-journalists get interested, excited, and able to request and share documents, and to enable large scale-FOIA projects like request BAH files from dozens of agencies.

And we've found that lots of people _do_ care about it at the state/local level, but news organizations are a) terrible about marketing their work. b) assume readers aren't interested in the nuts and bolts. c) let that work die in obscurity, content to try to win awards with it rather than to truly engage the public.

So yes, this is a flashy dump of documents, but we've helped hundreds of others do meaningful, smaller scale stories, and whatever way you cut it, I don't think there's enough groups doing that.

I also keep my own list of interesting, public-records driven stories:

[http://pinboard.in/u:morisy/t:to_foi/](http://pinboard.in/u:morisy/t:to_foi/)

Encourage anyone who is interested to reverse engineer them and try those requests in their city, state, and country.

<comment><USER2> Don't get me wrong, I love Muckrock! Wasn't trying to say you guys aren't necessary. I file FOI requests for a living, and I'd love to be in touch with you guys about a number of things - best practices for turning government records into structured data, the importance of reforming records retention on the government side to reduce the burden on public records officers, fixing FOI on the local level, etc.

Feel free to get in touch with me on Twitter: twitter.com/patdennis

</comment></comment></comment><comment><USER3> This is the kind of stuff I wish Wikileaks had stuck to -- mostly or completely legal, beneficial to national security, etc. They are still challenging powerful entities and supporting change, but without becoming the story themselves.

<comment><USER4> Likewise. I really liked WikiLeaks at the beginning but it became clear after some time that there was a definite agenda behind their actions. The impression I've had for years is that they will oppose the U.S. for no better reason than that it's the U.S. Which is, if you think about it, just like opposing any maligned group for what they are instead of what they _do_.

In other words if we start from the assumption that the U.S. should be a "bastion of the free world" then it would make sense to tirelessly note instances where the U.S., corporations, states, etc. fall from that ideal.

But diplomatic cables describing what kind of women Berlusconi likes, while interesting, are hardly indicators of the collapse of a free society. Those types of things were released because they were embarrassing, nothing more. And while I'm certainly glad that Assange is able to stroke his ego by thumbing his nose in Uncle Sam's face, I don't actually _hate_ my country; I want it to be better.

<comment><USER5> I have to strongly disagree. Wikileaks has released minutiae about other countries as well. It isn't only about showing up hypocrisy from the US, it is about lifting the veil of secrecy that all governments misuse whether it is hiding big stories or trivial ones.

Saying that wikileaks has an agenda to oppose the US is basically confirmation bias. Leaks about the US get news coverage in the US. Just for starters, look at the 2 million emails from Syria - clearly most of those are about trivial matters too, but most people in the US haven't even heard about the release at all.

<comment><USER4> > I have to strongly disagree. Wikileaks has released minutiae about other > countries as well.

Doing bad things to _other_ countries (again, because of what they are instead of what they've done) doesn't make me feel better about when it's done to mine.

Here's some other countries I don't hate: Russia, China, the U.K., France, South Africa and in fact the list could go on as long as I have time.

> Just for starters, look at the 2 million emails from Syria - clearly most of > those are about trivial matters too, but most people in the US haven't even > heard about the release at all.

There's no polite way for me to say this so I'll just spit it out; _no one cares_ about Syrian government emails. That's why no one in the U.S has heard about the release. Gossip is as destructive for international politics as it is for small towns.

If there's big news to release then by all means go and do so, but governments should be able to maintain diaries too.

<comment><USER5> I'm having a hard time understanding your point of view, it seems to have no unifying principle. But if I take your last sentence as what you really believe then the obvious response is that once governments can be relied on to not abuse their ability to keep secrets then they will have earned the privilege to maintain their diaries in secret.

<comment><USER4> And anarcho-libertarian types will claim that a government can never be relied on to do that, which is just as well as saying again that the government could not be allowed to keep secrets.

<comment><USER5> Since I'm not an anarcho-libertarian type and you clearly are not, I don't think what they might hypothetically say matters all that much.

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> A good way to search through the text of the documents is DocumentCloud, which also offers some analysis tools:

[https://www.documentcloud.org/search/group:muckrock](https://www.documentcloud.org/search/group:muckrock)

Modify the search terms as desired (i.e., "Booz")

</comment><comment><USER6> Is there already some decent crowd sourcing text analysis software out there?

Some simplified version of Nvivo with a wiki-like user management would do wonders for something like that.

1\. Button "I commit to analyze 2 / 12 /22 pages". 2\. You get a very short tutorial on how to do it. 3\. User is assigned relevant pages, starts tagging the documents for specific content and highlights + comments lines worth a closer look.

I would be definitely up for that. With enough traction one could just compare inter-coder reliability and establishing a high quality analyzing process (i.e. getting the same document analyzed by multiple users).

<comment><USER7> How about django-public-project? The following is from their description at [http://django-public-project.readthedocs.org/en/latest/](http://django- public-project.readthedocs.org/en/latest/):

 _Django Public Project (DPP) is a specialised content management system written in Python /Django for building an information website around big projects. It is open source (BSD licence) and can be used by civic groups to monitor the progress of publicly funded projects but also from within governmental institutions itself to bring transparency in the proceeding of a project, inform the interested public about what is going on and thus enhance the acceptance of the project by the different stakeholders._

It is currently being used by the investigative comittee of the Berlin city parliament in order to investigate the massive clusterf __k that is the city 's new and very disfuntional airport (BER). The source code can be found at: [https://github.com/holgerd77/django-public- project](https://github.com/holgerd77/django-public-project)

Regarding participation of ordinary citizens, the docs say:

 _On DPP websites users can comment on nearly everything and connect their comments to various elements of the project. Comments are then shown alongside every connected element. All admin users having activated the "Receive new comment emails" in the admin user settings receive an email after someone sent a comment and can then activate the comment to be shown on the website._

[http://django-public- project.readthedocs.org/en/latest/users...](http://django-public- project.readthedocs.org/en/latest/users.html#user-comments)

</comment></comment><comment><USER8> If anyone wants to search through MuckRock's recent documents via the DocumentCloud interface, they're also available over here: [http://www.documentcloud.org/public/search/group:muckrock](http://www.documentcloud.org/public/search/group:muckrock)

 _Edit_ : Whoops -- I see that Michael's already commented along the same lines.

To expand a bit, you'll be able to do boolean searches among the documents, scan through excerpts around your search query, look at extracted entities (people, places, company names), and plot any dates mentioned in the documents on a timeline.

</comment><comment><USER9> Now this is investigative journalism!

</comment><comment><USER10> [http://taxanalysts.com/www/features.nsf/Articles/557540D8F53...](http://taxanalysts.com/www/features.nsf/Articles/557540D8F539A26685257B94004CD0D1?OpenDocument)

I think this might provide some interesting background.

</comment><comment><USER11> What exactly is the point of this exercise? Booz Allen is a big consulting company. They have lots of contracts and SOWs, most of which are for really lame things. You are not going to find a contract that says: "provide 3 big brother spies to read everyone's mail".

Of you're trying to prove that that government spends lots of money for consultants... Well, we all know that already.

Also, unless you really know what you are talking about, you are not going to be able to spot most of the petty, obvious or really offensive contracting practices. The Feds have their own language re: procurement.

<comment><USER3> Based on the ones I've seen in other contexts, the key is to be able to aggregate. If you do, you can often find really interesting stuff from unclassified contracts around the periphery, then presumably FOIA more, but knowing how the system works helps a lot.

</comment></comment><comment><USER12> They should be using plagiarism-detection software to strain out the boilerplate commonalities and focus on the differences (as well as a single copy of the commonalities).</comment>