Nassim Taleb: End Bonuses For Bankers - <AUTHOR> http://www.nytimes.com/2011/11/08/opinion/end-bonuses-for-bankers.html?_r=1&pagewanted=all <comment><USER3> As a former banker with 18 years experience, this article is 100% on the ball. Having worked as a senior executive during the times of Global Crossing and Enron I saw how the system was gamed!.

Working at this large institution I saw how the bonus system, made the supposedly senior bankers act like a group of Mary Kay cosmetic sales girls, seeing how they could optimize their bonuses by playing the game, and how they got the lower levels of the pyramid to play along because of the partial subjectivity and discretionary aspect of the bonus system. Because of this discretionary aspect, lower levels of the pyramid, we're unlikely to question the creation of complex and funky new products specifically designed to overcome impediments to maximize that short term bonus.

When this giant "ponzi" scheme began to collapse, I saw how those same greedy senior executives proceeded to panic and destroy significant strategic parts of the business solely to stop the leakage of their bonus pool and try and cosmetically dress up the banks short term results to justify and maintain those 6-8 figure bonuses they had thought they were going to receive.

Many of these executives later "resigned" or were "retired" by their boards who should have been accountable for the damage reaped by these masters of gaming. Most of them(I think all of them!) retained huge bonuses all at the expense of the shareholders and employees. Writing off 100's of millions of $ of shareholder and depositor value. With middle class retail shareholders, depositors, and employees paying the price of this borderline criminal behavior.

Most galling to me is that one of these executive used some of his "hard owned bonus" to have a faculty/ building at my alma mater named after him. I believe this was probably more driven by ego than guilt!

Nassim is 100% on the ball. Nothing has really changed and history repeats itself, and unless government starts to listen then I fear the outcome will either be financial collapse or revolution (#occupywallstreet?).

<comment><USER4> As publicly traded companies (eg. Goldman, Morgan Stanley), don't the executives have legal obligations towards the shareholders? Isn't it illegal for them to game the rules to maximize their bonuses at the expense of the company's long-term viability? Why don't the shareholders fire these guys out of simple self-interest to protect the value of the stock they're holding?

If I were the owner of a company and the CEO I hired to run it for me were to put my company at risk for his bonuses I'd be hitting my head against the wall and firing him. No gov't regulation required.

<comment><USER5> Nobody wants to ruin the party. You're asking someone to stand up and say "stop making so much money, this isn't sustainable" when EVERYONE is making money hand over fist. Sure, it may be the right thing to do depending on your point of view, but nobody will do it because they will be laughed out of the room and cast aside.

Retail share holders are ignorant, so they don't matter. Big share holders know that there is a time frame that they care about that is as short as they want it to be. They don't care about the long term. The board of directors (in theory) should, but even they are short-timers. Asking them to stop making money in the short term to help the long term is counter to their personal interests. It's just a flaw in the system, and a major argument against going public.

Interestingly, the handful of folks I know who have served as directors for public companies all share certain personality trais: A strong extroversion (in that they seem to care a lot about what others think of them), a tendency towards overconfidence, and an aloof demeanor. Not sure if that's universal, but it's something I've noticed. It probably doesn't help.

<comment><USER4> > Big share holders know that there is a time frame that they care about that > is as short as they want it to be.

It's always said that the it's the "public" [through centralized funds like retirement funds] that get the short end of the stick at the end of a bubble. It seems to me these _are_ big share holders and they _should_ care.

<comment><USER5> Everyone should care. But the reality is that almost nobody does in the public market system.

</comment></comment></comment></comment></comment><comment><USER6> I love hacker news for discussions about business and about learning about how the smartest programmers in the world think about programming.

When articles that deal with the world outside of startup finance appear on Hacker News, the articles and comments usually have so little knowledge behind them they are practically unreadable.

It would be nice if this community could keep the articles they post based on VC funding, angel funding, debt funding for start ups, option pools, etc.

The comments here are more representative of political ideals and not based on facts.

(Also, I understand that is not entirely true as some comments are actually quit interesting, but I have to wade through so much garbage to find them that it isn't worth it.)

<comment><USER3> I would tend to disagree that this article is not relevant, startup finance and access to funding for startups is highly dependent on the health of the financial system, and this article deals with the economics and the gaming of that system.

<comment><USER6> You mentioned how bankers you worked with gamed the system in your other post. If gaming the system is so easy as a banker, please tell us how much money you gamed?

If you were there for 18 years as you said you were then you should have gamed a ridiculous amount of money out of the system and you would have been through at least 1 if not 2 recessions, 1 of which as a fairly senior banker.

<comment><USER3> I never said the game was easy and yes I saw several economic cycles, at several different institutions, on both the credit side (when Glass-Steagal actually meant something) and the investment banking side. I was well rewarded for the efforts I put in becoming a junior partner/MD and receiving adequate bonuses a year before the first collapse. Not everyone was consciously gaming the system, but you started to realize it when on successful deals like Global Crossing, which went from (a market cap of US$ 5 bn to chapter 10 bankruptcy the following year), senior exec were taking 20 million+ bonuses, and then being fired a year later. A clear indicator that there was something significantly wrong with the corporate governance bonus systems then.

Now 10+ years later when you see things like the 2008 recession, the current bankruptcy of MF Global and some of the Internet IPOs that are being pitched and sold to smaller institutions and retail investors, and the fees that are being taken, you have to wonder whether anything has really changed and really see how such compensation systems are not in the interests of a healthy financial system. Especially when taxpayers and shareholders have to bailout or bear the economic costs of the distortions created by these bonus systems.

Don't get me wrong there is nothing wrong with people being paid good bonuses and good compensation but as Nassim indicates the amounts being paid are excessive and don't truly reflect the risks being taken.

I don't know how relevant any compensation number would be to the point I am making. But if you did your research amongst the sec filings I am sure you will raise your eyebrows about how much money has been gamed and how disproportionate this is compared to the costs borne by taxpayers and others.

<comment><USER6> As a banker please recommend SEC filings that would be helpful in determining "gaming". Typically the media likes to use ridiculous numbers like average salary or average bonus per employee etc. (I've never seen a gaming loss section on a 10-K)

You do not need to take risk to make a lot of money in the United States. That is what makes us the greatest economic power house in the world. (yes, far greater than China who's average income is less than $5,000 per day) and especially Europe who is transferring all of their wealth to US treasuries to prevent loosing their money.

The rest of the world isn't even worth mentioning.

</comment></comment></comment></comment></comment><comment><USER7> This sounds like we're tackling a symptom not the disease. Instead of figuring out how to stop bankers from being paid so much in bonuses, we should answer ask why a few times.

"Why can banks pay their employees so much?" - because they make so much money, and have so few employees.

"Why can banks make so much money?" - I'm not sure but it seems like banks can take risks, but pass off the real risk to others.

"Why can banks take risks, but not have to worry about the downside of those risks" - ...

I think if you follow that train of thinking you'll get to some structural problem in our current system. It doesn't seem like there is an easy fix here.

<comment><USER8> Well, it used to be that banks were prohibited from taking risks. 100 years ago banking was a boring job.

<comment><USER9> In fact 100 years ago it was quite the opposite.

<http://en.wikipedia.org/wiki/Panic_of_1907>

</comment></comment></comment><comment><USER10> Or the US could simply re-regulate financial institutions the way it did during the period from the 1930s through the 1990s.

<comment><USER11> I do like this idea, but I'm a bit skeptical that a relic of the 1930s would work in our modern economy. Thus, I suspect the answer is a bit more complex than "Bring back Glass-Steagall".

<comment><USER12> I think the economics is pretty timeless, actually. A lot of the regulation was aimed at preventing catastrophic bank failures, before the idea of Too Big To Fail was invented. Glass-Steagall contains the idea that a commercial bank, which gets major regulatory freebies by virtue of being a commercial bank (FDIC, Fed access, &c), shouldn't have access to unfettered financial markets. And so on.

</comment></comment></comment><comment><USER13> Another option is for banking (of the high risk investment banking kind) to be conducted by partnerships, with only natural persons as members, rather than corporations. One's ownership would be long-term, and actions could claw back previous upside.

Goldman Sachs was a partnership until its IPO in 1999. A lot of finance houses were closely held partnerships until the late 20th century.

<comment><USER14> That would work if we wanted to stop banks from transferring losses to shareholders - the real problem is that they transfer losses to taxpayers. Pre-IPO Goldman Sachs was probably considered "systemically important" enough for their losses to have been covered in the event of a large trading loss.

<comment><USER13> Yeah, there needs to be a clear distinction in advance between entities which can be bailed out (and thus must be highly regulated) and entities which can experiment/innovate freely (but which can't socialize their losses).

LTCM would have clearly been "not bailed out", except it was. Admittedly not directly by the USG/FR, but at the direction of the Federal Reserve.

<comment><USER14> And even then it's tricky, because a non-regulated entity can make a bet with a regulated one and end up getting bail-out money by virtue of that bet. That seems undesirable, but you also can't really stiff the non-regulated entity, because then nobody will ever want to bet with the regulated one again, thereby destroying its ability to hedge.

<comment><USER13> I wonder what the costs to the economy would be if we had Canadian levels of regulation of the financial industry, or even greater (something like how utilities themselves are regulated).

<comment><USER15> Canada believes in allowing only a few, highly regulated banks. Since there are only a handful of banks, it makes it very easy to regulate them.

But this model does hurt the little guys. Since Canadian banks form an oligopoly, they charge _you_ for lending them money; account maintenance fees are standard. This hurts the smallest savers the most. Similarly, there is minimal competition in setting interest rates. Finally, financial innovation, done right, does help people---think Vanguard, index funds, ETFs, etc. All much more limited and much more expensive in Canada (I think).

In contrast to Canada, I think we would be better off with many small institutions, lightly regulated. Then they can compete for your business and do not need taxpayer support when they fail. When they become too big, break them up.

<comment><USER16> The banking system in Canada is not significantly different than the United States, except that it is just a banking industry. The U.S. has a banking/gambling industry. Actually it's more of a gambling industry with a small banking sideline.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> I think a better solution for "too big to fail" banks would be to require bonuses are paid in restricted stock. That way if the bank needs a bailout, then bonuses would be automatically clawed back. The bigger the bank, the larger the portion that is required to be RSUs. This would give a recruitment advantage to smaller banks, thus serve as an automatic limiter to bank size.

<comment><USER18> An even better solution would be to just allow the market to take its course. Thinking up ways to allow to big too fail institutions to keep existing is counterproductive. Just allow it to fail and people will stop doing those things. If your bank doesn't exist you can't very well collect a bonus from it.

I bet you if the big money families started losing big chunks of their fortunes that there would be serious reform. Same with retirees and their pensions.

How do you think the prosecution of bankers would go if bankers put the DOJ pension at risk?

<comment><USER19> We already tried this in the 1930s and that didn't go so well. Banks are different than other business because they act like a heart to pump cash through our economy. Allowing banks to fail would only make it harder for new businesses to find cash and for people to trust saving their money.

<comment><USER18> Actually in the 30s we did exactly what we're doing now, deflating our currency, passing a bunch of legislation to prevent competition, disallowing people from holding solid investments, paying people to be unproductive, a massive expansion of government predicated upon a weird interpretation of the commerce clause, and surprise surprise, it's having the same effect it did in the 30s.

Keep in mind that the economy 'failed' in 28, but it wasn't until FDR that things really got bad.

<comment><USER20> Are you sure?

I just pulled up the a graph of US GDP from Google: <http://www.housingbubblebust.com/GDP/Depression.html>

FDR took office in March of 1933, which looks like when things got better.

</comment></comment></comment></comment></comment><comment><USER21> This is absurd. Regulating private companies' compensation? Instead, the government should either make sure companies don't get "too big to fail", or make it very clear that banks don't have the Geithner Put option.

Simple and unobtrusive.

<comment><USER22> >> the government should either make sure companies don't get "too big to fail"

Exactly. And the best way to do that? Never bail them out.

<comment><USER23> "Never bail them out."

Even if it destroys the economy for everyone?

<comment><USER24> Bailing them out is the same as adding RAM to a server to fix a memory leak.

The problem isn't a temporary, transitory state - it's a systemic design flaw in a system filled with rogue actors.

<comment><USER23> Right, you need to combine saving the financial system with systemic change. Letting them fail is equivalent to letting the system crash and the doing nothing.

<comment><USER25> I just dont understand why people talk about the "financial system". Its a bunch of banks and some of them fail. The only banks that failed are those who are bad.

<comment><USER26> A financial panic is not an avenging angel that slays the wicked and leaves the righteous untouched.

Without the possibility of a bailout, if, say, Citbank went under, this would not only screw Citibank's depositors—most of whom were hardly in a position to audit Citibank's books before opening their accounts—but also every bank that had loaned money to Citibank. If a bunch of people who deposited money with Citbank owe money to Wells Fargo, then Citibank's failure hurts Wells Fargo. And if Citibank's failure led the depositors at Bank of America to get nervous and withdraw their money, then BoA would be at risk _even if_ it had been prudently managed up until the crisis. And then BoA's and Wells Fargo's creditors... etc., etc., etc.

<comment><USER25> Well yes some banks can be hurt in the process that are not acctully bad but history shows that in most cases bank runs happen on banks that are really bad. If you are a conservative bank you will acctully get a lot of cash (this happend in switzerland) witch you can then lend at high rates to banks that need it (if there wouldn't be the fed pushing the intrest rate down). Witch banks acctully are in a bad place and witch are not will be the task of bankers and private investors. The banks that are acctully good will probebly get survive. So yes there will be some collateral damage but that just cant be avoided in a cycle. It would be much better then the mess we are in know.

That all beeing said in a system where you have fractional reserve banking (witch is a bad idea anyway) there is a case (not one that I totally agree with) to be made that we need a "lender of last resort". But beeing a "lender of last resort" is quite diffrent then what the fed did. They literly flooded the hole bankingsystem with cash, knowbody knew what was going on, how will be saved who want, witch banks acctully are still liquid and witch will bust when the flow of cash stops. Why should banks lend to each other if the get free money. If I would get free money, I too would just sit around and wait until the worst is over. If every body does that we acctully have a bigger problem a long rescession instead of a short crash.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> >> I HAVE a solution for the problem of bankers who take risks that threaten the general public: Eliminate bonuses.

I'm sorry but that's not the solution.

The solution is for the government to stop bailing out private institutions, regardless of whether they are deemed "too big to fail."

There are some incredibly elegant natural laws built into the fabric of the universe, one of which is expressed through economic systems in which corrupt, reckless institutions are eliminated because they go broke.

The only way corrupt, reckless institutions are allowed to persist are when they are propped up by taxpayer money.

<comment><USER27> And if tens of thousands of people lose their jobs because the corporate credit markets freeze and otherwise-profitable companies can't fund their day- to-day operations, well, "externality" is just a fairy-tale idea thought up by ivory tower economists?

I'm sorry to be so glib, but if the answer isn't as simple as "end bonuses", it also isn't as simple as "just let the banks fail".

<comment><USER22> >> I'm sorry to be so glib, but if the answer isn't as simple as "end bonuses", it also isn't as simple as "just let the banks fail".

But there is a "simple" difference between philosophies of how to address corruption in financial markets.

>> And if tens of thousands of people lose their jobs because the corporate credit markets freeze and otherwise-profitable companies can't fund their day- to-day operations

There is no way to avoid the day of reckoning.

Don't pretend that any crises that were avoided in 2008 won't be that much larger when they eventually materialize.

<comment><USER28> >> There is no way to avoid the day of reckoning.

That would be true if it weren't for the fact that a good bit of our economy relies on confidence and trust. Confidence that there isn't a huge recession or depression in the near future, and trust that others will remain in business if you deposit/lend them money/invest.

So in a very real sense, a panic, even if it isn't based on anything fundamental, can set you back very significantly.

</comment></comment></comment></comment><comment><USER29> There are plenty of fine ways of reducing risk for companies which pose a systemic threat the US economy, eliminating bonuses is probably not one of them.

The most obvious is, if a company ever becomes Too Big To Fail, you simply force them to break up. We do this with monopolies because they could harm competition. We have plenty of experience with it. Surely we could do it with companies that represent a massive threat to our economy.

The second one I see is to force any Too Big To Fail company to hold a very large percentage of their value in a bond they hold with the government. Now, they could be a standard federal bond or a special insurance bond, but it would basically mean that if the sh*t hit the fan, there would be enough company assets in safe holding to fail in a more controlled manner.

<comment><USER30> > The most obvious is, if a company ever becomes Too Big To Fail, you simply > force them to break up.

Surely that also applies to GSEs (such as Fannie Mae and Freddie Mac), govt programs, and even govts themselves.

If not, why not?

</comment></comment><comment><USER31> This fails because it ignores the root cause. All banking crises are informational in nature. Circulating more of the right kind of information is nearly all the regulation you need. What information? Positions, live or daily, in detail. No more too big to fail, and many other problems simply disappear.

In contrast, most regulatory proposals betray a belief in installing a tough cop of some kind to combat 'evildoers'. Guess what, the evildoers are just people doing their jobs. We need to tweak the system to stabilize it, and for a regulated industry like banking the government has all the power it needs to do so.

<comment><USER32> What about accounting tricks which seem to move risks off an institution's books, but don't actually do so, and nobody notices until it's too late?

<comment><USER31> What I am suggesting is to allow the public and analysts to see the detailed positions of the institution and make accounting superfluous. We can view accounting as an extremely flawed model that lacks timely updates. By operating in the sunshine an array of activities permitted by information hoarding will no longer be sustainable.

</comment></comment></comment><comment><USER33> Mixed feelings about this article.

> The potency of my solution lies in the idea that people do not consciously > wish to harm themselves; I feel much safer on a plane because the pilot, and > not a drone, is at the controls.

What about myopia, self-delusion, panic, sheer intellectual dishonesty or even disability, and all the other assorted biases that afflict human judgment? Humans hurt themselves all the time. In fact, there are instances when putting more pressure or increasing the (financial) incentives _hurts_ performance and _increases_ risk.

> I believe that "less is more" — simple heuristics are necessary for complex > problems. So instead of thousands of pages of regulation, we should enforce > a basic principle: Bonuses and bailouts should never mix.

Having said that, I'm still all for the use and (re?)discovery of heuristics in regulation, combined with judgment on the part of the enforcer. It's high time we moved past the game of who can outlawyer who. There is a reason that posts like yesterday's knife maker capture the attention of many people these days and why firms like Apple or Leica are so successful these days. Life is becoming so complex, we can't write every contingency into a law; so much is becoming possible today that we need more and more conscious, i.e. editorial constraint.

</comment><comment><USER34> The problem that Taleb unfortunately doesn't address is that the bankers have captured the political system. For this reason his solution unlikely to transpire.

</comment><comment><USER35> I like the general idea of linking the consequences of some actions to the actors involved to prevent cheating and bad behavior. The examples from Babylon and Rome were probably much more effective in this than cutting the bonus of a banker (ie they paid with their lives).

I had a similar thought when I started reading about fracking (<http://www.propublica.org/series/fracking>). A simple solution there for the pollution caused by the wastewater is to force the executives and their families from the companies doing fracking to live in the communities they affect and use/drink the water they claim is safe.

</comment><comment> brown9-2 _Instead, it's time for a fundamental reform: Any person who works for a company that, regardless of its current financial health, would require a taxpayer-financed bailout if it failed should not get a bonus, ever. In fact, all pay at systemically important financial institutions — big banks, but also some insurance companies and even huge hedge funds — should be strictly regulated._

I'd really like to see someone attempt to express such a condition in the type of legalese that appears in legislation.

How do you ban something based on a hypothetical possibility? How do you write this condition down?

</comment><comment><USER36> >The asymmetric nature of the bonus (an incentive for success without a corresponding disincentive for failure) causes hidden risks to accumulate in the financial system and become a catalyst for disaster.

I think this claim contains a pretty elementary mistake. No bonus is a disincentive, because base salaries can be relatively low; not receiving a bonus is a large opportunity cost. If I could get a 400k salary, but I instead opt for a 200k salary with a 400k expected bonus, then if I don't get my bonus I'm 200k behind where I could have been if I just took the salary.

<comment><USER37> This is true, but the downside is capped: a bonus can't go below zero. Thus traders may become insensitive to the amount of money they lose and thus increase risk.

<comment><USER36> A bonus can go "below zero" if you need to get a >0 bonus just to reach your market salary.

<comment><USER14> But it can only go as far below 0 as the amount in salary you sacrificed in exchange for a potential bonus.

<comment><USER36> Yyyyes...? What is your point?

Edit: To clarify (since someone downvoted me), you are making a true statement, but I don't see how it is in conflict with anything I'm saying, or what other point it supports.

<comment><USER14> What Taleb is saying is still true. If I have a 400k market salary and I give up 200k of it for the right to 1% of the profits I generate, then I still maximize the expected value of my compensation that year by maximizing the size of my bets. I could bet a billion dollars on a coin flip, get 9.8 million (after recouping foregone salary) on heads and lose 200k on tails. Moreover, there is a well established history of traders that lost large amounts of money finding gainful employment regardless. See Boaz Weinstein, for example.

</comment></comment></comment></comment></comment></comment><comment><USER38> If you don't pay bonuses, do you give a managing director at a bank a 1 million dollar salary independent of performance? How is that a good thing? And if you tried to pay her very little, the bank would just turn itself into a shadow bank and proceed as before. Remember, GM was not a bank, at least nominally, and it was bailed out

</comment><comment><USER39> In order to understand why bankers get such massive bonuses it would probably be good to read this first:

Michael Lewis - Liar's Poker

[http://www.amazon.com/Liars-Poker-Michael- Lewis/dp/039333869...](http://www.amazon.com/Liars-Poker-Michael- Lewis/dp/039333869X/ref=sr_1_3?ie=UTF8&qid=1320795652&sr=8-3)

</comment><comment><USER40> I think strict regulation is the way to go. Banking in a modern economy is really a utility like gas, electric, and water.

Another solution that is more long term oriented and market based: create rival capital-formation pools outside of Wall and Broad, say in the Midwest, South, and West Coast. That way if one pool blows up, we can let them fail and it won't take out the whole economy. It also removes single points of failure from the system. I think the crowdsourcing bill floating in Congress is a great start as it decentralizes capital-raising.

</comment><comment><USER41> The real solution is to either disintermediate the banks or eat away at their profits. LendingClub, Covestor, SecondMarket, Simple, Square, WePay, and Mint come to mind.

<comment><USER42> agreed. it's simple and it only requires direct action on the part of the "little guys", and doesn't require Congress. also, shift more money into credit unions rather than traditional banks. generally, if we as a society make less money available for the Wall Street types to play with, they'll have less to play with, and the chance/magnitude of them fucking with the normal/base/productive part of the economy will be lessened in the future.

</comment></comment><comment><USER43> The fundamental problem is that when a corporation has so much money they can throw tons of it at the upper management and executive, and not go out of business because they are undercut by a competitor that doesn't waste money like that, then something is wrong with capitalism at that point. That is the problem you try to fix, you don't just throw more regulation on the banks that will backfire like software patents.

</comment><comment><USER44> Not everyone who works at a bank is a "banker" -- going after bonus structure holistically without regard to station/role seems impractical bordering on impossible.

Beyond even that, one might imagine that if US banking regulations get extremely prohibitive, banks will just move offshore. (And still be too big to fail with regard to the US economy)

I think that a little divide-and-conquer is needed to fix some of the smaller sub problems...

<comment><USER45> Part of the issue you're highlighting has to do with banks being able to engage in risky behavior - the traders that argued, after the collapse of, say, AIG, argued that they still deserved their bonuses. End the mixing of risky and non-risky behavior and you put most of that problem to bed.

Banks moving offshore isn't really realistic - you can't avoid, at the moment, having America as part of your bank - too much of a market. So even if, like UBS, the bank isn't in the US, it doesn't mean that they wouldn't be subject to regulation.

</comment></comment><comment><USER46> How about structuring bonuses more like an earn out. When companies are acquired an earn out is typically included so that the owner of the acquired company has an incentive to run the company in the most profitable way. If bankers bonuses were based on earn outs that are paid over 3-4-5 years then they will reconsider risk since compensation is deferred.

</comment><comment><USER47> Bonuses are incentives. What they are giving bonuses for is wrong. Not the bonus concept itself. Eliminate the bailouts, require a certain percentage liquidity, force assets to be marked to market on a certain interval. Suddenly the bank's finance sheet to it's investors won't be so rosy, investors will demand change, change will happen.

</comment><comment><USER48> You cannot control payment to employees like that, companies just find a way around those kinds of laws.

Better to regulate the institutions themselves so that they dont get 'to big to fail', don't get to combine access to cheap money with access to the financial markets, don't get to insure themselves and so forth.

</comment><comment><USER49> Nassim Taleb is making an argument about the structure of the banking system's risk management schemes. This is an attractive article for us to comment on because, doh, we'd like to punish somebody for the 2005-2008 Collateralized- debt-obligation and Credit-default-swap Cluster-f __* (I'll call that the C3F from here on). And, surely it would be good to hold somebody accountable for what happened. Not that they'll be able to fix it. But still, it will feel like justice.

But that's not Mr. Taleb's point.

The real C3F problem is of TREMENDOUS professional interest to creative software folks. Most of us work on systems that can aggregate lots of measurements to try to get a big picture of the system. Some of us look at video and audio signals. Others of us look at web server logs. Today, I'm trying to troubleshoot slow DBMS performance. Our brothers and sisters in banking and trading look at measures of risk.

And we all know what we do to make sense of these measures. We average them. We sometimes throw out the outliers. We measure their standard deviations, or maybe their quintiles if we're sophisticated. And then we track the averages and other aggregates, assuming that it's sound to do so.

My boss asked me today, "is the average query time going up?" I responded, "wrong question! we need to look at the outliers."

This approach to averaging measurements feels like something we got from our mothers' milk as infants. But it's based on Gauss's Central Value Theorem, which shows that independent (repeat INDEPENDENT) measurements tend to have a normal bell curve distribution. (We call that a Gaussian distribution in honor of the Central Value Theorem).

So, what the heck, let's sell mortgages to poor folks, and huge mortgages to rich folks. They can't ALL fail to pay, can they? The ones who fail to pay will be the outliers, won't they? The Central Value Theorem teaches us that the average person will pay up. So we can manage the two-sigma risk by buying a credit default swap (you have AIG's phone number, call them!), and all is well.

Except for one thing. Mortgage defaults aren't independent of each other. When one property on the block goes into default, it becomes harder to sell the others or refinance them. So the Central Value Theorem's premise of INDEPENDENT measurements fails. Big time. Lo and behold, C3F.

Mr. Taleb's point is that in the real world of risk management, things aren't Gaussian. The events he calls black swans are long-tail events (that is, their probability curve falls off far slower than the Central Value Theorem predicts).

Why is this relevant to HN? Because we can easily deceive ourselves by ignoring outliers (black swans) in our fields of work. Hopefully it won't be as catastrophic as C3F, but we should beware.

Seriously, if you haven't read Mr. Taleb's book The Black Swan, it's worth your trouble.

<comment><USER50> You are correct in most of your points, but it's worth pointing out that for the 25 or 30 years prior to 2007 mortgage defaults did tend to be uncorrelated. During that entire time frame, if you were a trader who chose to insure yourself against black swan events, you were not likely to have kept your job for long since the cost of that insurance would have been prohibitively high cutting deeply into your profits.

There's a great line (one of many) in Andrew Ross Sorkin's book "Too Big To Fail" in which Jamie Dimon, the CEO of JPMorganChase told Hank Paulson, at the time the Treasury Secretary, "You've got to make us do it". It's essentially the prisoner's dilemma. It would be advantageous for all the banks to institute a given change, but none of them can do it by themselves without being chewed into little pieces by the others.

</comment></comment><comment><USER51> Taleb is an excellent writer and not (as someone here has stated) someone completely delusional. He, in fact, worked in the financial industry for many years and I thoroughly enjoyed reading his book "The Black Swan"[1] about how we've basically all been duped by large banks that expose us to risk and keep the profits for themselves.

[1]<http://www.fooledbyrandomness.com/imbeciles.htm>

</comment><comment><USER52> Taleb rocks. The funniest thing is that he hates the NYT but is able to recognize the chance for a big audience.

<comment><USER52> Why did this get downvoted?

</comment></comment><comment><USER53> A debate with some merit, but not "hacker news". I say flag it.

</comment><comment><USER54> What people seem to often forget is... during the financial crisis following the housing crash governments (especially in the USA) passed out significant amounts of bailout money for all those banks but completely failed to get them under control - so banks just used that bailout money to temporarily guarantee their vital liquidity and once things were looking up again a few months later, they just kicked them out and went right back to their old business, including bonuses and everything.

There were even farce attempts to "get them under control" and the banks themselves would have votes in these decisions HOW they were going to be controlled and regulated. Excuse me??? That is like putting the mafia on trial and making their family the grand jury.

"Too big to fail" has proven time and again to be the absolutely best way of gaming the system almost any way you want. We even bent the very rules of capitalism and free market backwards multiple times to accommodate for these behemoth institutions and to this day, justice hasn't happened and nobody has ever been responsible for this mess.

</comment><comment><USER55> The problem is what we are attempting to regulate culture wise does not view at as a problem..to change that culture requires some heavy duty court convictions and jail time...if Obama get his 2nd term some of those court cases in fact come to trial via the DOJ and SEC, etc.

<comment><USER56> What evidence do you have that Obama administration will prosecute their campaign financiers in the 2nd term?

They had more (electoral) incentive to prosecute in the first term... They have no reason in their 2nd term.

</comment></comment><comment><USER57> End Bonuses For Software Engineers!

</comment><comment><USER58> In a Ponzi scheme, you give money to the Ponzi operator, the operator does "complicated" things with it, those who withdraw early do well but at some point there is no money.

How is this different from the bailed-out banks, or an over leveraged banking system in general?

<comment><USER59> In a Ponzi scheme you have to actually participate to get burned.

<comment><USER18> If you don't participate in the global economy you can't get burned by it. People who subsistence farm are largely immune to problems in the global economy.

</comment></comment></comment><comment><USER60> I like the decimation idea. Before they can be bailed out, capital punishment should be randomly applied to one-tenth of the bankers of the failed institution. The lottery can be weighted based on bonus size, to ensure that the big wigs are fairly represented in the sample.

</comment><comment><USER61> The author of the article and the people who advocate his opinion are delusional! Giving bonuses is how banks work these days.

<comment><USER62> Because it currently is thus, therefore it must be efficient and optimal, and any attempt to change it is delusional... you sound like an economist! But I think a lot of HN is about challenging incumbents and the status quo...

<comment><USER62> It exists, therefore it must be optimal (unless government is involved) - <USER2> economists' credo</comment></comment></comment>


Lots of Animals Learn, but Smarter Isn't Better - <AUTHOR> http://www.nytimes.com/2008/05/06/science/06dumb.html?partner=rssuserland&emc=rss&pagewanted=all

<comment><USER2> in the immortal words of ogden-nash:

here's a good rule of thumb:

too clever is dumb

what's more interesting is that it applies to programming. quite nicely too!</comment>


The Dangerous Academic Is an Extinct Species - <AUTHOR> https://www.currentaffairs.org/2017/04/the-dangerous-academic-is-an-extinct-species <comment><USER3> This perfectly describes my experience in college. I waited for my assumptions to be challenged, but all I found was more boring conformity.

<comment><USER4> Me too. College turned me into a men's rights activist

</comment></comment><comment><USER5> Jordan Peterson says that The University has moved online.

The shells that remain on campuses are expensive finishing schools and a way around the IQ testing ban.

</comment><comment><USER6> The first two paragraphs fits in the most stereotypes of any article I've read. A better one is Matt Walsh's, [http://matt- welsh.blogspot.com/2010/11/why-im-leaving-harvar...](http://matt- welsh.blogspot.com/2010/11/why-im-leaving-harvard.html)

</comment><comment><USER7> This article is a little odd. It describes conservatives as being worried that professors are "subversive", but it's exactly the opposite - <USER2> worry that colleges are enforcing and indoctrinating people in the orthodoxy.

It's the rare subversive academics, e.g. Charles Murray, who are being assaulted and chased of campus for holding unorthodox opinions.

And it's happening off campus as well - for instance, we recently had a peaceful (but unorthodox) anti-war activist assaulted: [https://antifascistnews.net/2017/04/09/richard-spencer- just-...](https://antifascistnews.net/2017/04/09/richard-spencer-just-got-his- ass-kicked-and-glitter-bombed-videos/)

<comment><USER8> Orthodoxy is not a few people (correctly) arguing that we should STOP stating that black people are inferior. Furthermore, a Nazi who happens to be against a particular war is not "peaceful but unorthodox".

<comment><USER7> Charles Murray wasn't discussing human biodiversity at all. His talk was on an entirely different topic. No one argued against him at all. They merely disrupted his talk and then assaulted him and others.

The Nazi in question did not engage in any violent acts at that gathering. That makes him a peaceful protester.

<comment><USER8> That Murray was attacked during an unrelated talk is uncontested. My objection is that you referred to the defenders of centuries-old superstition as unorthodox simply because the cultural tide began to turn against them in the 20th century.

<comment><USER7> I agree that in some historical time, some of Murray's views might be considered orthodox. That doesn't make them orthodox today - nowadays he's subversive.

<comment><USER8> Only if we accept that racism is dead. To find proof to the contrary, we need only look to popular responses to videos of unarmed black men being killed by men with guns. And that's just scratching the surface.

<comment><USER7> Similarly, flat earthism is an orthodox theory since it was orthodox at one point and is not completely dead.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Eric Garner wasn't choked to death. Even then, what does he have to do with conformity?

</comment><comment><USER10> The take on adjuncts is truth. Just about everything else is pure rubbish and I'm disappointed I wasted my time reading even half of it.</comment>


[pdf] "Galileo", Intel's new arduino-compatible dev board's datasheet - <AUTHOR> http://download.intel.com/support/galileo/Galileo_Datasheet_329681.pdf

<comment><AUTHOR> DISCLAIMER - I work at Intel, but am by no means speaking on their behalf. I speak for myself as an individual.

As a guy that works at Intel, it excites me to see Intel taking steps in the right direction. There have been people clamoring for this at meetings since I joined. It's been posted up here before, but I just had to post it again.

No more cross-compiling for me.</comment>


Rabbit Holes - <AUTHOR> https://github.com/mrdrozdov/rabbit-holes

<comment><USER2> What the fuck is this? This is soooo fucking shit. It is literally a random collections of websites. I am so angry and disappointed. I going to slap you with an oily dorrito flavoured fish</comment>


Face Surveillance Is Not the Solution to the Covid-19 Crisis - <AUTHOR> https://www.eff.org/deeplinks/2020/03/face-surveillance-not-solution-covid-19-crisis <comment><USER2> Uhhh, right... nobody thought that it was.</comment>


Super-Microphone picks out Single Voice in a Crowded Stadium - <AUTHOR> http://www.wired.com/gadgetlab/2010/10/super-microphone-picks-out-single-voice-in-a-crowded-stadium/

<comment><USER2> Neat. The sound-wave equivalent of a Mills Cross style radiotelescope, which can be electronically "aimed" at any point in the sky by adjusting the signal delay from each individual reciever.

<http://en.wikipedia.org/wiki/Mills_Cross_Telescope>

<comment><USER3> Yes, it looks like an interferometer. The microphone array looks very symmetric, and usually with radio interferometer's you set up the antennas to not be symmetric, in order to prevent ringing (ie areas on the sky where you get very good signal, and other area's where you get very poor signal. With a circular aperture, you'd get a response function resembling a Bessel function). So I wonder if they could get better results with a less symmetric microphone array.

Hmm... Or maybe the array is large enough that the entire court is inside the central spot, meaning all the ringing is outside the court. I'll have to think about that.

</comment></comment><comment><USER4> Awesome possibilities for entertainment, but terrifying possibilities for surveillance in public spaces.

</comment><comment><USER5> I've always wondered how much of this would be possible with just a standard stereo recording. Does anyone have an idea?</comment>


How I Scrapped MySQL for MongoDB in Three Hours - william-shulman http://blog.programmableweb.com/2011/08/15/how-i-scrapped-mysql-for-mongodb-in-three-hours/

<comment><AUTHOR> In this light, <http://mongodb-is-web-scale.com> is maybe relevant.</comment>


HP TouchPad will be the first real competitor for the iPad - <AUTHOR> http://ipadwatcher.com/2011/02/09/hp-touchpad-will-be-the-first-real-competitor-for-the-ipad/ <comment><USER2> I don't think the few gee-whiz features will set it apart from the upcoming iPad 2, Blackberry Playbook or the legions of Android tablets unless it is competitive on price, availability and battery life. and sadly nothing was said about price or battery life and availability was "summer" for wifi and "later" for 3G. Not promising.</comment>


Show HN: Screensharing plugin for FaceTime (macOS) - <AUTHOR> https://tryscreentime.com <comment><AUTHOR> I have 100 coupons to give out. Please email me to claim it (my email's on my profile).</comment>


Billionaire's Supersonic Private-Jet Plan Hits Engine Snag - <AUTHOR> https://www.bloomberg.com/technology <comment><USER2> The link is not to the story, but to the general technology page:

[https://www.bloomberg.com/technology](https://www.bloomberg.com/technology)</comment>


Generalized Covert Data Exfiltration with Funtenna - <AUTHOR> https://www.blackhat.com/us-15/briefings.html#emanate-like-a-boss-generalized-covert-data-exfiltration-with-funtenna

<comment><USER2> [http://ossmann.blogspot.ch/2013/01/funtenna.html](http://ossmann.blogspot.ch/2013/01/funtenna.html)

</comment><comment><USER3> Here's the repo - [https://github.com/funtenna/REF_DESIGN](https://github.com/funtenna/REF_DESIGN)

</comment><comment><USER4> I am immediately reminded of [http://zapatopi.net/mindguard/](http://zapatopi.net/mindguard/)</comment>


Duolingo announces Tinycards - <AUTHOR> https://www.duolingo.com/comment/16875822 <comment><USER2> Anybody know what is going on with Duolingo's monetization model? They where supposed to sell crowd sourced translations, but my impression is hardly anybody is buying and the few users that do translations are mostly translating stuff that doesn't appear to be for commercial purposes.

<comment><USER3> They dropped translation as a profit model (newer language groups lack "immersion" and they are trying to remove it from older languages with some backlash from existing users.)

They released a remote testing/certification initially for English. They also added and continue to build on class management, but so far that seems to be free.

</comment></comment><comment><USER4> I'll save a lot of you some time. This is iOS only for now.

</comment><comment><USER5> Well, this has totally ruined my day. I've ben working on a side project that is essentially this: making and sharing of cards and decks.

<comment><USER3> > making and sharing of cards and decks.

Anki already ruined my whole decade AFA writing my own version of this.</comment></comment>


Improved write barriers in Factor's garbage collector - <AUTHOR> http://factor-language.blogspot.com/2009/10/improved-write-barriers-in-factors.html <comment><USER2> It just occurred to me that Slava is doing something a little innovative with this blog. Most articles with code in them use sample code to illustrate some abstract idea. Conversely, most articles about working systems are written at a feature level. Slava is producing posts about a real system, organized around real code. It's a kind of documentation, really, but a publicly accessible, general-purpose variety. Now that I think of it, I'm surprised that this is not more common. Can you guys think of other instances?

<comment><AUTHOR> Ola Bini has done something similar with his blog posts about Ioke.</comment></comment>


Taking an app from good to great: 6 app pros critique Hawaiian Words - <AUTHOR> http://blog.tapstream.com/post/47028307694/taking-an-app-from-good-to-great-6-app-pros-critique

<comment><USER2> For those that don't know: Hawaiian Words is a 1600 word translation dictionary. It has ads that can be removed via an in-app purchase, music and customizable backgrounds, and purchasable text-to-speech for the Hawaiian words. It also has word of the day.

It might not be for me or for you, but if people are buying it, I assume there's a niche market for this kind of thing.

</comment><comment><USER3> I'm the developer of this app. The critique was really useful; I'm already starting to implement some of the suggestions.

Feel free to ask me any questions. I have a lot of data on which marketing techniques have paid off and which haven't. I just started running an online and print ad campaign in Hawaii Magazine.

<comment><USER4> I ran an ad in an email blast to 11,000 people for $300. 12-20 visitors was all I saw through Google Analytics, and the App Store downloads did not increase.

Since that experience, I haven't spent any more on marketing, but I'm going to give adwords a shot for my Photo Investigator landing page. Are there any other marketing avenues that you'd recommend?</comment></comment>


Avoid Gulf stream disruption at all costs, scientists warn - <AUTHOR> https://www.theguardian.com/environment/2018/apr/13/avoid-at-all-costs-gulf-streams-record-weakening-prompts-warnings-global-warming <comment><USER2> There's a great little book called "Ice Age"[a], written by a Cambridge University astrophysicist and his wife, in which they discuss in lay language what would happen if this "underwater conveyor belt" in the Atlantic, which helps regulate global temperatures, were ever to stop or (gasp!) reverse: global weather patterns likely would drastically change and the planet would likely accelerate towards a _glacial period_.[b]

For example, if you look at a map of the world, you will notice that the UK is at roughly the same latitude as some of the northernmost parts of Canada but is much warmer. This is partly due to the warm waters brought by the Atlantic's "conveyor belt" to the waters surrounding the UK's coasts. If these warm waters stop flowing toward the UK, its weather would likely evolve to resemble that of northern Canada.

There are other complicated factors and feedback loops involved, but this example should give you an idea of how important this Atlantic current is to the stability of global climate.

[a] [https://www.amazon.com/Ice-Age-John- Gribbin/dp/1511926422](https://www.amazon.com/Ice-Age-John- Gribbin/dp/1511926422)

[b] [https://en.wikipedia.org/wiki/Glacial_period](https://en.wikipedia.org/wiki/Glacial_period)

</comment><comment><USER3> [https://news.ycombinator.com/item?id=16838058](https://news.ycombinator.com/item?id=16838058) (My post before I found this one.)</comment>


There Is No Theory of Everything - <AUTHOR> http://opinionator.blogs.nytimes.com/2015/09/12/there-is-no-theory-of-everything/ <comment><USER2> > One huge problem with scientism is that it invites, as an almost allergic > reaction, the total rejection of science. As we know to our cost, we witness > this every day with climate change deniers...

Just a few paragraphs later he celebrates how irritating philosophy is while sidestepping how much its pompous pedanticism encourages anti-intellectualism. If it's a "huge problem with scientism" then it should also be a huge problem with philosophism.

It's also unclear how exactly "scientism" is responsible for climate change deniers. Somebody says "I believe someday science will explain everything!" and so somebody else says "Oh yeah? Climate change is a hoax by the government to get money!" Perhaps the author uses the word to describe being arrogant about science but you can be plenty arrogant about it without adopting a view of "scientism".

</comment><comment><USER3> "As we know to our cost, we witness this every day with climate change deniers".

Denial? No one will deny there's an asteroid bearing down on us, if there is. No one will deny that an appalling virus released from a research lab can kill us all, if in fact it was released and can do that. Denial is reserved for deviation from a faith. Such deviants are called heretics. The religious terminology says it all. Why talk about denial if the evidence is screaming at us. Please point to it; that's all you have to do. The heretics will recant immediately. Is CAGW a 'faith'? I guess so.

<comment><USER2> A quick Google search gives these top links for starters:

NASA: Climate Change Evidence - [http://climate.nasa.gov/evidence/](http://climate.nasa.gov/evidence/)

National Oceanic and Atmospheric Administration: [https://www.ncdc.noaa.gov/indicators/](https://www.ncdc.noaa.gov/indicators/)

Union of Concerned Scientists: [http://www.ucsusa.org/our-work/global- warming/science-and-im...](http://www.ucsusa.org/our-work/global- warming/science-and-impacts/global-warming-science)

Weather Underground: Evidence of Climate Change: [http://www.wunderground.com/climate/evidence.asp](http://www.wunderground.com/climate/evidence.asp)

<comment><USER4> None of that evidence counts. It's all falsified. Or the people producing it are stooges. Or they're so stupid they get it wrong. Some of it is a conspiracy. Some of it is misinterpreted. And it's a good thing anyway. And even if it's real (which it isn't) it's not caused by anything humans do.

I can come up with a million of these. When an axiomatic starting point is fixed (hello religion and other cultural beliefs), you can come up with post- hoc rationalisations that not only mean you get to be right, but you get to feel smart and special (and with this one, you get to make people who have no idea what they're talking about feel that their street-smarts or some bullshit "common sense" is superior to actually knowing things, like those scientists who think they're so much better than everyone else). Trying to reason anyone out of a position they did not reach through reason is a wasted effort. You'll just go insane if you try :)

<comment><USER5> You forgot, "whatever humans have contributed is minuscule compared to how much the sun has affected things."

<comment><USER4> Now that's a clever one. It's true in that the energy being dumped on the planet every day by the sun is huge compared to what we burn, so at first glance people will think it's a decisive argument. It's easily understood, so people with little understanding of the situation can think they've seen right through all the conspiracy etc. I'll add that one; if you can make your audience feel clever doing it, they'll believe anything you want.

<comment><USER5> Hence the common refrain in a lot of the denialist screeds of, "This is just common sense", "we already know this", "it's simple science", "THINK, PEOPLE!!!!', etc.</comment></comment></comment></comment></comment></comment>


Sculptor Dangles Clues to Stubborn Secret in CIA's Backyard - <AUTHOR> http://www.nytimes.com/2010/11/21/us/21code.html

<comment><USER2> What do you want to bet that it's not the word "Berlin" but pieces of two adjoining words like "number line?"

[http://www.wordbyletter.com/suffixes/words_finishing_with_be...](http://www.wordbyletter.com/suffixes/words_finishing_with_ber.php) [http://www.wordbyletter.com/prefixes/words_starting_with_lin...](http://www.wordbyletter.com/prefixes/words_starting_with_lin.php)

</comment><comment><USER3> Previously:

<http://news.ycombinator.com/item?id=573192></comment>


Programming languages worth checking out  - <AUTHOR> http://www.h3rald.com/articles/10-programming-languages

<comment><USER2> 11\. F#

F# is a mixture of C#, OCaml and Haskell. F# will be included in the next version of Visual Studio so it's no longer just a research language. Performance is similar to C# but coding in it is faster in my experience and above all a lot nicer if you like functional programming. Because it runs on the .NET VM or Mono VM it has good multicore support unlike OCaml. Haskell is more powerful as a language, but also has it's weak aspects. The main thing F# is missing imo in comparison to Haskell is type classes. And of course the Haskell community.

<comment><USER3> If Microsoft can work out a sane licensing arrangement for compute farms/clusters/clouds (whatever we're calling them this week) of Win 2008 Server, then F# is a game-changer of a similar scale to Google's MapReduce. That's a big _if_ tho'.

Having said that, Windows is cheaper than RHEL already...

<comment><USER4> I think I can see how MapReduce could be argued to be a "game-changer" (although I'm not sure it's true), but why F#? If I understand correctly, it's basically OCaml on the CLR. So you get a Python-like brevity of code (and ease of programming? maybe it has better error messages than OCaml?} with C# performance and access to the CLR libraries. That sure sounds useful, but why is it a "game-changer"?

<comment><USER3> Because functional languages are a good choice for extracting as much parallelism as you can get from your algorithms.

<comment><USER4> No.

CUDA yes. Cg yes. HLSL yes. Verilog yes. VHDL yes. C++ or Java with MapReduce yes. PHP and MySQL with memcached yes. Erlang yes — and it really _is_ functional, inside each process, anyway — that is, the level where you _aren't getting any parallelism_. Octave or R, potentially, but not today, as far as I know. Mathematica yes, and it, too, is mostly functional.

In theory, side effects are what make parallelism hard, and so languages whose semantics are side-effect-free (unlike F# or Mathematica or Erlang) should make it easy. So we all thought in 1980. Since then we spent 20 years or so trying to make that happen, and it basically didn't work.

There are basically four kinds of parallelism within easy reach today. There's SIMD, like MMX, SSE, 3DNow, AltiVec, and the like; you'd think that data- parallel languages and libraries like Numpy and Octave would be all over this, but except for Mathematica, that doesn't seem to be happening. There's running imperative code on a bunch of tiny independent processors that share no data; AFAIK that's what the shader languages are doing. There's instruction-level parallelism on a superscalar processor, which largely benefits from things like contiguous arrays in memory, or maybe what Sun is doing with Niagara, where the processor pretends to be a bunch of tiny independent slow processors. And then there's splitting up your data across a shared-nothing cluster, which is how every high-traffic web site works, and that's what MapReduce makes simpler.

Uh, and then there's designing your own hardware or programming FPGAs, which is what Verilog and VHDL are for.

Languages like OCaml (I don't know anything about F# except that it's like OCaml, but for the CLR) have no special advantage for any of these scenarios. They don't even have the _theoretical_ advantage that they have no side effects and therefore you can speculatively multithread them without breaking the language semantics. They do have the massive practical disadvantage, in most of the scenarios I described, of needing unpredictable amounts of memory, having massive libraries, and using pointers all over the place. Using pointers all over the place kills your locality of reference and your ILP. Having massive libraries and using unpredictable amounts of memory makes it impossible to run them inside your GPU and means they can't run on an FPGA (except by using external memory, like the awesome Reduceron). And nothing about the language semantics helps with SIMD either.

So, sheesh, go read Alan Bawden's dissertation or whatever, but don't go around claiming that ML (or even Haskell) is going to magically make your algorithms parallel. We tried that. It didn't work. We're trying something else now.

<comment><USER5> Just for my own edification, since you seem quite familiar with the subject: Is Obsidian* a big fat waste of time that'll never be as good as just compiling Haskell for a CPU like a normal person? 'cause I was considering investing some time in learning it, when I wouldn't have the free brain cycles for CUDA.

* <http://www.google.com/search?q=obsidian+haskell>

<comment><USER4> I didn't know about it! From a cursory look (all I can find is presentation slides?), it doesn't sound like you'll be able to take any off-the-shelf Haskell subroutines and run them in parallel on the GPU; rather, it's an embedded DSL for constructing shader programs. So I imagine you'd have to learn both CUDA and Obsidian to use it.

But it would be awesome if someone came along and proved me wrong.

<comment><USER5> Thanks for the read--now, how 'bout another? <http://www.cse.unsw.edu.au/~chak/papers/gpugen.pdf>, another Haskell-embedded GPU DSL, claims to work at a higher abstraction level than Obsidian, but still provide high performance general-purpose GPU programming capabilities.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> I find Self to be worth checking out, as well (<http://research.sun.com/self/>) if for no other reason than seeing how fast dynamic languages can be. The research that went into it--techniques for blazing fast implementations of highly dynamic, object-oriented languages--has become especially relevant and provides the basis for many of the numerous new Javascript vms (especially V8).

With regards to Smalltalk and its incredible reflective abilities, this video from OOPSLA'08 ("Smalltalk Superpowers") is particularly amusing: <http://www.veoh.com/videos/v163138695pJEMGmk>. Unger even does a pretty neat demo of Self.

<comment><USER3> Self seems to be a dead project?

<comment><USER6> Actually, checking out the cvs repository on sourceforge (<http://self.sourceforge.net/>), there seem to have been some recent commits; though, in general, I think you're right that there isn't much active development.

There's also work on a linux port at <http://www.gliebe.de/self/download.html>, but I think it might be relatively defunct as well.

</comment></comment></comment><comment><USER7> I'm curious if anyone has any experience with Io (<http://iolanguage.com/>). The description sounds like an appealing combination:

 __ _Its unusual, minimalist and yet elegant and powerful syntax reminds of Smalltalk, but the language goes far beyond that. Io is an object-oriented, prototype-based, message-based and fully-reflective programming language. This means that you use messages like in Smalltalk, you create objects like in Javascript and every bit of your code can be inspected and passed around as you see fit._ __

<comment><USER8> I co-wrote (with Steve) SoundConverter for Windows:

<http://dekorte.com/projects/shareware/SoundConverter/>

The interface talks to an Io "server" which handles business logic.

We're currently working on a shopping aggregation startup. The crawler is written in Io.

Pixar's RenderMan image tool uses Io as its scripting language:

<https://renderman.pixar.com/products/tools/it.html>

Io is still under development so you might run into the occasional bug. Bugs are encountered more often in the addons than in the core interpreter (which is stable).

<comment><USER9> Ah, I see you're the creator of Io. Do you know of any Io tools for Vim? Syntax and indent would be great if nothing else...Google didn't turn up anything.

<comment><USER8> No, Steve Dekorte is the creator of Io. I work with him on some projects (including Io).

I'm not sure about vim. I use TextMate (there is an Io bundle).

<comment><USER9> Oh, I saw the link and didn't read your username clearly. I have the TextMate bundle here...I suppose _some_ interested person will have to get to writing a few io.vims :^)

</comment></comment></comment></comment></comment><comment><USER10> Is anyone using D? Is it a lot better than C++ or just a little?

<comment> old-gregg There is also an issue with an implementation: the official implementation from Walter is awkwardly packaged and lacks the source (you can't build it yourself). This pretty much turns it into a toy for Linux/OSX programmers: you can't distribute your D code, nobody will figure out how to build it.

And GCC-based implementation is lagging behind in features, and also scores consistently 10-30% slower than C/C++ in benchmarks, which kills D's appeal as a performant replacement for C.

What Walter needs to do, IMO, is to abandon his own implementation completely and closely work with GCC/Linux community to include high-quality D into standard GCC package. _Then_ we may start seeing decent software written in it.

</comment></comment><comment><USER11> Noticeably missing from this is Maroon, the Ultra High Level Programming Language

<http://mattmaroon.com/?p=337>

I guess I should have paid that guy on elance his $1,000 to code it up in C for me.

</comment><comment><USER12> _"Unlike other Lisps (and Schemes) you may have encountered before, Clojure comes with some interesting additions: [...] Many pre-built data structures, like Vectors, Maps, Sets, Collections, …"_

Here I stopped reading.

<comment><USER13> Why? It's a nice adition to the language and makes using it a much better experience.

My first big project in LISP was making a Sudoku solver for a school project. Could only use basic instructions and lambdas. It was not pleasant trip :P

<comment><USER14> But common lisp does come with vectors (make-array), hashtables (make-hash- table), and set-operators (intersection, union, pushnew, etc) that work on lists.

Also, Common Lisp (CLOS, in particular) supports multimethods (defmethod), contrary to what the author claims. And all of the special forms I saw in clojure have a counterpart in common lisp.

In fact, a lot of common lisp implementations (I like SBCL) have support for threading/asynchronous action built in. The one problem is that none of that is part of the ansi spec, so most multithreaded code won't be portable between implementations.

It isn't the language's fault that your teacher wouldn't let you use all these features.

<comment><USER15> What's distinctive about Clojure is not that it has them, but rather that it makes them first-class citizens in the way that lists are - i.e. provides a universal set of operators for manipulating them (or so I hear anyway). This is certainly a weakness of CL. I think this is the original point underlying the author's garbled statement, and it's an improvement specifically to CL that has nothing to do with the presence of arrays and hashtables in the language.

Edit: I realize this is probably obvious, but let's not confuse what this guy says about Clojure with what Hickey has to say. He's well aware of CL and doesn't make silly claims about it.

<comment><USER14> That's very true. I personally like clojure - and it does provide a lot of potential advantages to common lisp. I just don't think the author of this article understands them - he's just spewing bs.

Hickey does make a lot of valid points about the advantages/differences of clojure (<http://clojure.org/lisps>) though.

</comment></comment></comment></comment></comment><comment><USER16> What about Prolog? Why it's not in the list?

<comment><USER17> Is it still being used at all? I would be interested in it, too (and I don't think Erlang is very similar to it at all???).

<comment><USER4> Erlang gets a lot of its syntax and implementation technology from Prolog, but it's true that the distinctive things about Erlang (massive shared-nothing concurrency, supervision trees, tuples, pattern-matching on binaries) have nothing in common with the distinctive things about Prolog (backtracking). They both have pattern-matching, but so do lots of other languages (even Python to a small extent).

I'm pretty sure Prolog is still being used, but it doesn't have the buzz of, say, Fortran or COBOL or Pascal.

<comment><USER18> Wasn't Erlang originally implemented in Prolog?

<comment><USER4> Yes.

</comment></comment></comment></comment></comment><comment><USER19> experimenting with lua this evening. be curious to hear if others have had good/bad experiences with it.

<comment><USER20> It's one of my favorite languages, and is rapidly crowding out Python. It's a _tremendously_ clean and powerful language. Since Lua was designed for embedding, its design favored adding a few meta-features that could be used to build task-appropriate features into the language, rather than adding to the language core. It's also a relatively quick language to learn, and is extraordinarily portable - the whole language is written in pure ANSI C.

It also has tail-call optimization, closures, coroutines, and other aspects that are interesting from a pure CS standpoint, such as the way the table datatype is implemented and its register-based VM.

For a good taste of Lua, skim "The Implementation of Lua 5.0" (<http://www.tecgraf.puc-rio.br/~lhf/ftp/doc/jucs05.pdf>), the quick intro to Lua in the LPEG paper (<http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html> , around page 6 of the PDF), or _Programming in Lua_ (<http://www.inf.puc- rio.br/~roberto/pil2/>). The first edition of the latter is online, though the second edition has a bunch of added material, and IMHO is well-worth buying if you get into Lua. Also, check out LuaJIT (<http://luajit.org/>), if you're using i386 hardware -- it's a _FAST_ JIT.

</comment></comment><comment><USER21> Try minikanren for all your logic programming needs: <http://kanren.sourceforge.net/> . :)

</comment><comment><USER22> Hmm,

Nice to have all the up-and-coming languages lined-up in an internal comparison. I like Lua, Haskell and Io. I think that future languages will have to have a cleaner rather than an uglier look and syntax. For that reason, I have a visceral reaction to scala and factor. Even the one-line examples I've seen of these language have look like gibberish. I'm sure they are great in many ways but don't want them to succeed since they will make my brain hurt more. Also, I think that they won't succeed for similar reasons.

<comment><USER23> Don't forget Eiffel and its progeny. Design By Contract and heavy use of assertions feels to me like it's closely related to Test First development. You also get fast runtime code out of it (as in C++ fast) in a very Pure-OO environment. (By many accounts, even more so than Java and C++.)

<comment><USER24> I played with Eiffel for a time, but found that Design By Contract was a bit overly restrictive for amateur programming. Now that I'm doing more professional stuff, I think it's worth looking at again. There's also a lot of Eiffel influence in Ruby, so moving over from Ruby is not so hard.

<comment><USER23> What I've found is that you actually want restrictive when you're doing maintenance. You want to know as much as possible about the code you're changing. You want to know exactly what's in that instance variable.

When you're doing prototyping, you want to delay that decision as long as you can, so you are making the most informed decision possible. And nothing informs your decisions like the act of developing and getting feedback from users.

Optional static typing gives you the best of both worlds. But I don't know how you'd get the equivalent with Design By Contract/Unit Testing. Both of those feel a little unnatural to me. I'd rather just tinker, and not have to set up extra stuff before or after I code.</comment></comment></comment></comment>


CodeGear (Borland) releases 3rdRail - "High Power Ruby on Rails IDE" - <AUTHOR> http://www.codegear.com/products/3rdrail <comment><USER2> Am I missing something here? Why would I pay $399 for this "new" IDE, when the existing competing products are either free (Radrails & Netbeans) or cost about 1/5th as much (Textmate) ?

<comment><USER3> Like they say, no one ever got fired for buying IBM.

It's expensive because it's aimed at accounting drones who make the decisions about what to buy and who want software from a "big name" company like Borland.

On a related note, there is at least one other similarly expensive rails IDE, but the name escapes me right now.</comment></comment>


MG4J – A free full-text search engine for large document collections - <AUTHOR> http://mg4j.di.unimi.it/ <comment><USER2> That name sound very familiar, as does the feature set. Managing Gigabytes[1], or "mg" was the output of a University of Melbourne and RMIT research in the 1990s. It went on to be commercialized as SIM and later TeraText[2] and has largely disappeared into the government intelligence indexing and consulting- heavy systems space (where it is now presumably being trounced by Palantir).

[1] [https://www.amazon.com/Managing-Gigabytes-Compressing- Indexi...](https://www.amazon.com/Managing-Gigabytes-Compressing-Indexing- Documents/dp/1558605703) \- Note review from Peter Norvig!

[2] [http://www.teratext.com/](http://www.teratext.com/)

<comment><USER3> That's exactly what I thought - I worked on index construction for MG back in 1994. (Note, although my name is Tim Bell, I'm not Timothy C. Bell, the coauthor of "Managing Gigabytes".)

</comment></comment><comment><USER4> I don't how this project ended up here in this moment, but as one of the authors let me answer the main questions.

1) The name is just a coincidence. I learned originally about indexing from the "Managing Gigabytes" book, and that's the reason for the name, but the book is now completely obsolete, and, even at that time, it contained a significant number of red herrings. There's no connection or code or idea sharing of any kind.

2) MG4J is our playground for doing research in information retrieval. This means, for example, that we designed new data structures, such as Elias-Fano indexing, which make MG4J have ridiculously faster times in benchmarks (see [https://github.com/lintool/IR-Reproducibility](https://github.com/lintool/IR- Reproducibility)). Elias-Fano is now the main Facebook indexing algorithm and it is slowly percolating to Lucene (look in the sources).

3) You can define your queries using a very rich interval language with a very fast implementation based on new algorithms. You can easily create parallel indices with text and tagging and ask whether a phrase falls into an area tagged as "location", for example.

2) MG4J is a project of two people and at this time I'm the only maintainer. You cannot expect that it is refined as Lucene or Solr. But you can very easily hack into it (even without modifying the sources), which is why it has been popular with people experimenting with indexing. For example, there are many tools to manipulate index, splitting them with a specified strategy, combining them, etc.

3) So if you want an out-of-the-box solution for indexing, forget about it. If you want a fun playground for doing research or a very efficient backbone on which to build your infrastructure, MG4J might be useful to you. We used it recently for [http://wikirank.di.unimi.it/](http://wikirank.di.unimi.it/) .

</comment><comment><USER5> Blast from the past! Distributed is a bit of a stretch, I think you need to coordinate all of that yourself. It is no more distributed than Lucene (I think).

Their fastutil stuff is pretty interesting though for creating highly optimized algorithms. Lot's of primitive based data structures that are fast and memory efficient.

</comment><comment><USER6> How does this compare to Elasticsearch or Solr?

<comment><USER7> I think it makes more sense to compare it with Lucene (which both ElasticSearch and Solr are based on) or, say, Xapian.

Based on a PDF [http://grupoweb.upf.es/WRG/mir2ed/pdf/slides_appendixA.pdf](http://grupoweb.upf.es/WRG/mir2ed/pdf/slides_appendixA.pdf) page 14 (linked from [http://stackoverflow.com/q/5028314/116546](http://stackoverflow.com/q/5028314/116546)), I think the differences are MG4J has constant RAM usage (as opposed to Lucene's linear one), but is somewhat more CPU intensive.

Haven't used either directly.

<comment><USER8> The comparison is with an 8-year-old version of lucene. Lucene is (optionally) constant RAM now.

</comment></comment></comment><comment><USER9> Some links are broken inside the unimi.it

</comment><comment><USER10> how is this on the first page of hackernews?

<comment><USER11> It's hard to answer that without any idea of why you find its presence surprising.</comment></comment>


Show HN: A raytracer to shade topographic maps in R - <AUTHOR> http://tylermw.com/throwing-shade/ <comment><USER2> That's a cool project and a nice write up. I remember as a 13 yr old being fascinated by 3D software. I think I just wanted to create something photo- realistic with a computer. I couldn't afford 3DSMax, so started on the steep learning curve with Blender. About the same time, they integrated Yafray into Blender, resulting in some outstanding renders, similar in quality to commercial packages. I was amazed and started delving into the maths - what a fun and confusing journey it was! Five years later when the maths was finally explained to me at university, I truly appreciated the work that went into things like Blender and Yafray. So kudos to you for your raytracer!

<comment><USER3> Aside: At various points I'd gotten deep/prolific in 3DS Max, Maya, Softimage, Cinema 4D, and Lightwave — but to this day, I cannot grok blender. I never get past using it for a day and feeling like the controls are completely against my brain.

<comment><USER4> When did you last try? I'm a small-time blender dev and lots of work is being done to improve the usability. Any comments you have would be appreciated

<comment><USER2> I think the way split screening works is quite confusing. As a frequently returning user, I often forget how to merge a split back and end up having to Google it. Or without realising it, I end up splitting the screen too many times. I wished that part was more intuitive. The fact that it isn't means I am reluctant to use this rather powerful feature.

<comment><USER5> Funny - I was about to hop in and defend Blender, before I read your comment. Splitting the screen is annoyingly fiddly.

In general, I actually like Blender's interface. I'd like it more if it was a little more vimlike (all the screen splitting stuff could be under 'Ctrl-W', for instance).

My problem with Blender is more about how easy it is to lose data. A reliable autosave would be amazing.

</comment></comment></comment></comment></comment><comment><USER6> I think the O(row * col * light_count * dist) algorithm could be optimized to run in O(row * col * light_count).

The idea is, imagine the light is coming from the left, and our height map consists of a single row that looks like (0,5,2,7,1).

By using the angle of the light, we can compute from left to right whether the current point should be visible. We only need to store the last point that was also visible since that is the only point that can block us from this light source. For the given example, at point of height 2 we only need to care about the previously visible point of height 5. If 2 isn't visible, 5 remains the "highest" point, if 2 is visible, 2 becomes the "highest" point, there is no danger that 5 will block something beyond two since 2 wasn't blocked. So at each point we only have to check with one previous point, not all.

If the light isn't coming from exactly the left side, then we can tranform the image so in such a way that the light is now from the left. This transformation will take O(r * c) time per light.

Total complexity: O(transforms + visibilityChecks + mergingResults) = O(l * r * c)

<comment><AUTHOR> One thing I didn't mention in the article are a few optimizations I made that make it significantly less than O(row * col * light_count * dist). First, if the ray exits the map region (either by going higher than the highest point on the map, or outside the boundaries), it stops propagating that ray and calls it a non-intercept (cutting down on the "dist" complexity). Second, if you're drawing multiple rays for a finite sized light source, I first order them by angle and send the rays out from the lowest to the highest angle. If a ray at a lower angle doesn't intercept a surface, higher angles won't either. So you save a lot on the "light_count" complexity.

</comment></comment><comment><USER7> Your raytraced shadow maps have a bit of a "glossy" feel, it suggests that you don't use linear colorspace for rendering these. I think colormagick also needs additional options to handle multiplication in linear colorspace. I didn't look into any of your code, so I'm sorry if I'm wrong about this.

<comment><AUTHOR> I'll look into that, thanks for the note. My Imagemagick knowledge is by far my weakest link.

<comment><USER8> Something like "-set colorspace RGB -colorspace sRGB" converts from linear RGB to sRGB. Switch those two to go the other way as needed.

</comment></comment></comment><comment><AUTHOR> If you missed it, the github repo for the R package is located here:

[https://github.com/tylermorganwall/rayshader](https://github.com/tylermorganwall/rayshader)

</comment><comment><USER9> I have a bit of a complaint. I've used something similar to the "derivative based model" (DBM) for 3D shading before, and tested them on moonscapes.

The angle-to-brightness curve can be adjusted as needed to avoid under or over-emphasizing mountain slopes. The "Andes" example is probably a case of exaggerating topology to illustrate relative differences, and NOT in inherent fault of slope-to-brightness maps (DBM-like). I can't say which one is more practical without going to the actual slopes to check the look and feel of the terrain and "feel" of the slopes. For technical accuracy, color coding is probably better (color-per-height), but slope-shaded maps are easier for regular folks to relate to.

Ray tracing is probably better when dealing with direct shadows and reflections, such as found with buildings. But for high level maps with mountains and rivers, I believe DBM is just fine if tuned right, and won't differ noticeably from comparable ray tracing.

To avoid misinterpretations, the large-scale shading should more or less mirror a semi-cloudy day such that direct shadows and reflections would play an extremely minor role anyhow.

I should point out there is an element of subjectivity here. Illustrations often exaggerate to emphasize various attributes of the illustration's target. Often there is a trade-off in terms of aiding the quick mental digestion of information versus "native" accuracy. Abstractions lie on purpose, or we wouldn't call them "abstractions".

<comment><USER10> It looks to me like the original gradient based shading code is just wrong. Maybe there are numerical problems due to its frivolous use of trig functions, or more likely the way it calculates gradients is just broken. But shading based on slope should obviously make all zero slope areas (like water) the same color, and the fact that it doesn't means it isn't working.

The raycasting code in this article, on the other hand, seems to be _missing_ the Lambert term! Just because a surface is not in shadow doesn't mean that it will appear at uniform brightness. If it is diffuse/matte, which seems like a reasonable starting point for most terrain, each light ray L should have a contribution to the brightness of the point proportional to L dot N, where N is the normal vector of the surface (and to find that you need, sorry, the gradient). In short, light almost tangent to the surface doesn't illuminate it much. Other kinds of surfaces will have different BRDFs (e.g. they can be shiny) but none of them will have the completely unshaded behavior shown. Of course, it's fine to do totally nonphysical things if it helps emphasize some aspect of the data, but it doesn't seem to be done knowingly and I think it produces a misleadingly flat look.

<comment><USER11> His dataset includes underwater topography

</comment></comment></comment><comment><USER12> This would be fantastic as a QGIS plugin, have you thought about applying to their grant program?

<comment><AUTHOR> First time I've heard of QGIS, so nope! But it looks like they support C++ plug-ins, so it wouldn't be hard to port it over. Don't know anything about their approval process, though.

<comment><USER12> Here's a brief outline of the requirements: [https://plugins.qgis.org/](https://plugins.qgis.org/) they advertise a 2 weeks review time maximum.

</comment></comment></comment><comment><USER13> What's really baffling about all things rendering is when you realize that almost anything you can think of and optimize to render in 10ms with a ton of sweat, was already done in 1ms in a game engine. Ten years ago.

<comment><USER14> Except that's not true is it?

3d rendering in games even today don't (in general) raytrace. While some things are raytraced in games /today/, getting performance is still achieved largely by leveraging the GPU, which in nowhere near as good at raytracing as general rastering. Essentially GPUs make raytracing faster simply because it is ridiculously parallel, and by limiting the kind of features that are supported (vs. general cpu raytracing)

And also "cheating" \-- raytracing is ideally "correct", but for a game you don't need to be correct, you just need to be good enough (or rather, you need to be pretty, which isn't necessarily physically correct).

That said based on the content of the article this could be trivially improved with better data structures -- it's just walking a height map, but the article makes no mention of quad or kd trees which are iirc the standard go to for height map traversal.

<comment><USER13> My comment came from the observation that making sampled tracing of a depth buffer is similar to what (current, rasterizing) game engines do for ambient occlusion.

[https://en.m.wikipedia.org/wiki/Screen_space_ambient_occlusi...](https://en.m.wikipedia.org/wiki/Screen_space_ambient_occlusion)

It's correct that it cheats (samples a few samples and blurs the results) instead of sampling adequately to get "correct" results

<comment><USER14> SSAO is not remotely close to raytracing (as done here). This post is not discussing ambient occlusion, but actual direction shadows.

SSAO doesn't sample a path across the depth buffer (because that wouldn't make any sense at all for shading), or consider light direction. This is very much my recollection from many many years ago, but basically my understanding is, but I recall that it's just a random sample of depth information surrounding a given fragment, and uses the depth of those samples relative to the current fragment as a scaling factor for the ambient contribution to to the final fragment light. The end result isn't even remotely close to correct, but again, correct isn't the same as being looking good enough (or even just looking good - in games you don't have totally control of how a scene may be seen by a player. Incorrect lighting can often be far superior to correct lighting from a gameplay pov.

</comment></comment></comment></comment><comment><USER15> Neat project!

I'm not sure if the author is familiar with ambient occlusion, but the method described here can be extended to generate ambient occlusion data which helps with a couple flaws. In particular:

\- Although the shading is very nice (see Figure 8), the long shadow rays are distracting and are not very useful for cartography.

Ambient occlusion extends the idea presented here, except that instead of sending rays towards a relatively small light source, consider the light source to be the entire hemisphere of the sky, like on an overcast day. The result is wonderfully soft light that conveys form much like Figure 8, but without shadows.

Simply send out more rays to the entire sky hemisphere to create this effect.

</comment><comment><USER16> I think it is likely possible to do this in screenspace ultra fast on demand using a shadow map / SSAO technique.

<comment><USER17> In this application you're probably interested in something more accurate, more than the speed. But it could be worth trying to see what interactive re- mapping bought you.

</comment></comment><comment><USER18> I made something similar that runs in real time in 30 lines of code.

</comment><comment><USER19> awesome stuff, how'd you make the illustrations (like Figure 6. )?

<comment><AUTHOR> Thanks! I drew the frames in Inkscape and animated them together using the Scrollmagic JavaScript library.

<comment><USER19> looks good. would love to see more !

<comment><AUTHOR> Check out the last few posts on my blog! I use this method a few times to help demonstrate points:

[http://www.tylermw.com/visualizing-a-reddit-hug-of-death- and...](http://www.tylermw.com/visualizing-a-reddit-hug-of-death-and-how-to- reddit-proof-your-website-for-pocket-change/)

[http://www.tylermw.com/soma-water-filters-are-worthless- how-...](http://www.tylermw.com/soma-water-filters-are-worthless-how-i-used-r- to-win-an-argument-with-my-wife/)</comment></comment></comment></comment>


Blockchains and Reality – Introducing Rlay - <AUTHOR> https://medium.com/rlay-official/introducing-rlay-a-decentralized-protocol-for-blockchains-external-data-problem-c0e6947a193d?ref=hn <comment><USER2> Ok, this is worth reading!</comment>