Optimizing Solr (Or How To 7x Your Search Speed) - <AUTHOR> http://carsabi.com/car-news/2012/03/23/optimizing-solr-7x-your-search-speed/ <comment><USER2> My experience working with Solr is that a lot of the time people don't have a good working knowledge of how to optimize an index because it's so easy not to. At my last job, the initial implementation they we involved storing the full text of millions of documents, even though they never needed to be retrieved (just searched). If you're running Solr as a front-end search for another database, the best way I've seen to optimize performance is just to make sure you're not storing data unnecessarily.

Maybe everyone already should already know this, but I was working on a very smart team, and we totally missed this initially. Setting "stored" to false for most fields resulted in a 90% reduction of the index size, which means less to fit into RAM.

<comment><AUTHOR> Yep, totally agree with this. Last month we spent an hour or two going through the schema and removing any fields that didn't need to be stored, and making sure that only fields which we actually do queries on had index=true. I didn't test before and after results, but qualitatively it seemed to be faster afterwards.

</comment></comment><comment><USER3> Hey, Websolr founder here.

Websolr's indexes return in under 50ms for queries of average complexity.

The more expensive queries usually involve "faceting" or sorting a large number of results. For an example, say you search Github for "while." Github used to do language facets, where it would tell you that out of a million results, 200103 files were in javascript, 500358 files were in C, etc.

The problem with this is that you have to count over a million records, on every search! Unlike most search operations which are IO bound, the counting can be CPU-bound, so sharding on one box will let you take advantage of multiple cores.

Racoonone is "sorting on two dimensions, a geo bounding box, four numeric range filters, one datetime range filter, and a categorical range filter." This should put him in a cpu-bound range (in particular because of the sort).

Websolr has customers on sharded plans, but they are usually used in custom sales cases where we're serving many, many millions of documents. We'll look at adding sharding as an option to our default plans, so that they'll be more accessible for people like raccoonone. In the meantime, if you send an email to firstname.lastname@example.com, we'll try to accomodate use cases like this.

Edit: Also, other possible optimizations include (1) indexing in the same order you will sort on, if you know ahead of time, and (2) using the TimeLimitedCollector.

<comment><AUTHOR> Awesome, lemme know when you guys have sharding available! Would love to not have to worry about running our own index, again.

<comment><USER3> It's available now, just not as part of a standard plan. I'll send you an email, just to make sure we're getting you the best setup for your needs.

</comment></comment></comment><comment><USER4> There are quite a few other performance related points to think about for Solr speed for queries and indexing.

Here are some that come to mind right now that are very useful:

\- Be smart about your commit strategy if you're indexing a lot of documents (commitWithin is great). Use batches too.

\- Many times, i've seen Solr index documents faster than the database could create them (considering joins, denormalizing, etc). Cache these somewhere so you don't have to recreate the ones that haven't changed.

\- Set up and use the Solr caches properly. Think about what you want to warm and when. Take advantage of the Filter Queries and their cache! It will improve performance quite a bit.

\- Don't store what you don't need for search. I personally only use Solr to return IDs of the data. I can usually pull that up easily in batch from the DB / KV store. Beats having to reindex data that was just for show anyway...

\- Solr (Lucene really) is memory greedy and picky about the GC type. Make sure that you're sorted out in that respect and you'll enjoy good stability and consistent speed.

\- Shards are useful for large datasets, but test first. Some query features aren't available in a sharded environment (YMMV).

\- Solr is improving quickly and v4 should include some nice cloud functionality (zookeeper ftw).

<comment><USER5> These are good points. Solr/Lucene tuning is an art, so much so that some search consulting companies charge tens (or hundreds) of thousands of dollars for these services. That's the value proposition of Searchify's hosted search - if you just want search, you shouldn't have to worry about shards, commit strategies, batching, GC, etc. You just want to add your documents, search them, and get great, fast results, without having to become a Lucene expert in the process.

If this sounds interesting, check us out at <http://www.searchify.com> \- We offer true real-time, fast hosted search, without requiring you to learn the innards of Solr or Lucene.

<comment><USER4> Good stuff. I see you're on Heroku as well which is always a win.

Now if someone could put SenseiDB on the cloud, i'd pay for it...

</comment></comment></comment><comment><USER6> I'm curious what your queries look like, because these performance numbers are awful.

I'm currently running an index that is 96 million documents(393GB) using a single shard with a response time of 18ms.

If you're comfortable with it, I'd suggest profiling Solr. We found that we were spending more time garbage collecting than expected, and spent some time to speed up an minimize the impact of it. Most of this was related to our IO though.

Second, don't use the default settings. Adjust the cache sizes, rambuffer, and other settings so they are appropriate for you application.

I'd also start instrumenting you web application such that you can start testing removal of query options that may be creating your CPU usage issue. You get a lot of bang for your buck this way, and you may find the options you were using provide no meaningful improvement in search. A metric like mean reciprocal rank can go a long way to improve your performance.

<comment><USER7> Re GC - were you using standard or parallel GC?

</comment></comment><comment><USER8> Our company had to set up a Solr implementation with some pretty crazy requirements (hundreds of shards, tens of thousands of requests per second, etc), and we ended up with 4 machines - one for indexing, one as a backup indexer/searcher, and 2 just doing load balanced searches. Replication was interesting but easy to set up (since it's basically an rsync of the indexes between servers).

The end result works very well, though it's a real memory hog when you get into the "hundreds" of shards on an individual server.

<comment><AUTHOR> What was the reason for having hundreds of shards on each server? Were you still seeing performance benefits to sharding it that aggresively?

<comment><USER8> Multi-tennent data separation required by contract.

</comment></comment></comment><comment><USER9> One thing I think would be valuable to know here is how many threads each shard is using, and what effect changing that number would have.

(rather: why is it useful to explicitly shard vs running one big instance with all of the memory and the same total number of threads? queuing theory would lead me to believe the latter would be better)

</comment><comment><USER10> take a look at this presentation if you are interested in NRT Solr (although it was done before Solr added the latest NRT features):

Tuning Solr in Near Real Time Search environments: <https://vimeo.com/17402451>

</comment><comment><USER11> Thoughtful sharding is not an optimization, it's a _requirement_ at scale.

</comment><comment><USER12> does any one have experience with adding shards on the fly??we have a requirement weher we get millions of docs every day and we need to have an environment that can handle real-time as well previous days' data...any thoughts on this will be appreciated...

<comment><USER13> The shards that are used in a Solr query are specified at runtime (you pass a list of shard URLs as part of the search query string) so adding new shards on the fly should Just Work.

</comment></comment><comment><USER14> anybody can recommend a good way to get startet with Solr ?

<comment><USER13> If you're using Django, Haystack makes it trivially easy to set up a Solr index against your existing Django models.

<http://haystacksearch.org/>

<comment><USER15> in a similar vein, if using ruby/rails Sunspot(sunspot_rails)[1] is awesome. <http://sunspot.github.com/>

</comment></comment></comment><comment><USER5> I'd love to have them try out Searchify's hosted search and see how fast it is. The key to fast search is RAM, which is why we run our search indexes from RAM (not cheap), and most queries are served within 100ms. If you're the author of the blog post, please contact me, chris at searchify, if you'd like to do this comparison, and I'll set you up with a test acct.

<comment><USER16> Hi Chris, what is Searchify's relationship with the old IndexTank (<http://indextank.com>) team?

<comment><USER5> There's no formal relationship, although I have met several of the IndexTank guys and they're a cool group. And Searchify is based on the IndexTank open- source project.

</comment></comment></comment><comment><USER17> I improved solr performance by switching to elasticsearch. =)

<comment><USER18> Same here. Just make sure to set your inital shards correctly so that you can grow for a while. Once you have to increase shards you have to re-index :(. But that's fine. Just go with 4x or more the shards you need now and you'll be able to scale out to that many boxes.

</comment></comment><comment><USER19> This is the founder of Bimaple. We provide hosted search and license our engine software with significantly better performance and capabilities than Lucene: (1) Supporting "Google-Instant" search experiences on your data; (2) Powerful error correction by doing fuzzy search; (3) Optimized for mobile users by doing instant fuzzy search with a speed 10x-100x higher than Lucene; (4) Optimized for geo-location apps; (5) Designed and developed ground up using C++. We have demonstrations on our homepage. If interested in using our service or software, please email contact AT bimaple.com.

<comment><USER20> Hi Chen,

It seems like an interesting product. Advertising here on HN is always tricky --- it's a balancing act between self-promotion and restraint. I voted your comment up in the hope that you'll stick around and tell us more about it. But perhaps more on the tech side, and less on the marketing.

Have you published any papers describing your approach? Or white papers with more meaty details? I work on the Apache Lucy project, and am very interested in things that work better than Lucene.

<comment><USER19> Hi nkurz,

Thanks for voting my comment up. Based on your request, I provide the following information. Let me know if you want to move this discussion to a private email discussion.

We have two published white papers to compare our engine with Lucene/Sor:

(1) A comparison white paper in the case of traditional keyword search: [http://www.bimaple.com/files/Bimaple-Keyword-Search- Comparis...](http://www.bimaple.com/files/Bimaple-Keyword-Search-Comparison- With-Solr.pdf) . We have a demonstration site to support instant, fuzzy search on Stack Overflow data (600K question titles as of January 2011): <http://demo.bimaple.com/stackoverflow>

(2) A comparison white paper in the case of geo-location keyword search: [http://www.bimaple.com/files/Bimaple-Map-Search-White- Paper-...](http://www.bimaple.com/files/Bimaple-Map-Search-White- Paper-201108.pdf) . We have a demonstration site to support instant, fuzzy search on 17 million US business listings: <http://www.omniplaces.com> . It has an iphone app at: <http://itunes.apple.com/us/app/omniplaces/id466162583?mt=8>

If you have questions, please feel free to contact me at chenli AT bimaple DOT com. Thank you.</comment></comment></comment>


Snowden Leaves Hong Kong on Commercial Flight to Moscow - <AUTHOR> http://www.scmp.com/news/hong-kong/article/1267261/snowden-leaves-hong-kong-commercial-flight-moscow <comment><USER3> Man, how times have changed. I was born in the 70s and the US was the country you would run _to_ to escape retribution for whistleblowing you did back home. (†)

For Snowden to be running to Russia to escape extradition from Hong Kong (!) is just fucking bizarro world for me. It is actually quite jarring. I don't trust Russia at all. But even thinking about it forces one to ask whether one trusts the USA. The answer to that is pretty jarring, too.

(†) I'm not making a right/wrong judgement about his actions. He blew a whistle, a whistle he thought needed to be blown. In my opinion that is a necessary check on state power and is a defence in and of itself.

<comment><USER4> Really, they haven't. Daniel Ellsberg, the leaker of the Pentagon Papers (1971), was also also charged with espionage. In his case all the charges against him were dismissed. Not, to be clear, because of any contriteness on Nixon's part. Instead, it was because the government was found to have illegally bugged his phone and broken into the office of his psychiatrist in order to find material with which to discredit him. Several of those involved were later convicted as part of the watergate trials [0].

If you think things were better in the 70s, you really weren't paying attention.

[0] [http://en.wikipedia.org/wiki/Daniel_Ellsberg#Trial_and_mistr...](http://en.wikipedia.org/wiki/Daniel_Ellsberg#Trial_and_mistrial)

<comment><USER3> > If you think things were better in the 70s, you really weren't paying > attention.

Yeah, well I was 3.

Even so I don't seem to recall Ellsberg fleeing the country to escape absolute certain injustice. I don't recall his video interview with a foreign paper while he hid in China. Do you?

Something has definitely changed.

<comment><USER5> Imagine how things might have turned out differently for Bradley Manning if he had a day or two to get out before his arrest.

<comment><USER6> Manning was a PFC in the Army. He wouldn't have been able to afford fleeing to Hong Kong or Moscow making less than $30,000 per year. He would have had a much more difficult time evading arrest.

<comment><USER7> How much money could such a fleeing really take? I mean, 30k isn't much at all, but would he even have been paying for rent or food while enlisted? If he managed to save up a few thousand for a few plane tickets and maybe a few days in a hotel on his own wallet, then that would be enough. I don't think a few thousand is a particularly unreasonable ask.

</comment></comment></comment></comment></comment></comment><comment><USER8> This is strange, because there are no scheduled nonstop flights between Moscow (any airport) and Iceland. There are flights which make a connection in Europe, e.g. Helsinki, but then there are already nonstop flights to Helsinki from Hong Kong, so what does he gain from an additional landing in Moscow?

(Assuming Iceland is where he is trying to go).

[http://matrix.itasoftware.com/](http://matrix.itasoftware.com/)

[http://flightaware.com](http://flightaware.com)

(Note that the hub in Iceland is Keflavík (KEF), not Reykjavík (RKV) which mostly does domestic).

<comment><USER9> Furthermore, in Russia, he will have to go to the Danish embassy or consulate to get asylum in Iceland. I am not sure how this helps or disadvantages him, but I thought I'd mention that Iceland themselves don't have an embassy/consulate in Russia, but is represented by Denmark.

<comment><USER10> Dane here. Our politicians will hand him over to the US if they can, but by law they can't if he faces the death penalty or if he is likely to be treated inhumanely.

<comment><USER8> The US can guarantee to waive the death penalty in exchange for an extradition. I've read some lawyers think the US is likely to try this.

<comment><USER11> For some reason the fact that he can be facing _death_ just struck me after seeing you mention it.

<comment><USER12> Well, he's being charged with treason isn't he? I know treason doesn't really sound _that_ serious to many people any more, but historically it is one of the "big three" that could earn you the death penalty, even in progressive countries that don't like the death penalty. The others being espionage, and of course, murder.

<comment><USER13> No, so far he's not been charged with treason. The US allows the death penalty for divulging certain types of classified information - for example information directly concerning nuclear weapons, and communications intelligence.

I don't think any progressive countries impose the death penalty at all, but I suppose this depends on what countries you would include in that category.

<comment><USER12> _I don 't think any progressive countries impose the death penalty at all_

Would you consider Canada progressive?

[http://en.wikipedia.org/wiki/Kanao_Inouye](http://en.wikipedia.org/wiki/Kanao_Inouye)

I didn't mean it is necessarily like this _today_ , but rather that even progressive countries continued to execute for treason at least partway through their history of progressiveness. It took longer for that piece (death for treason) to die out than most other pieces (like, I dunno, denying women suffrage?). It's sort of an awkward idea to express.

I could start listing, but you should really just peruse this list:

[http://en.wikipedia.org/wiki/List_of_people_convicted_of_tre...](http://en.wikipedia.org/wiki/List_of_people_convicted_of_treason)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> _> Moscow will not be his final destination. Possible final destinations are either Iceland and Ecuador, according to previous media reports._

I first thought that Ecuador would be more likely, since had Iceland been his final destination, he probably would have taken the Icelandic businessman's offer to hire him a chartered flight. But then I realized that perhaps Snowden places more trust in a commercial flight and not being detained in Moscow than a stranger's chartered flight which could potentially backstab him and hand him over to US authorities.

<comment><USER15> Iceland hasn't exactly been very vocal about supporting him, so Ecuador might be the best choice. Russia I think was pretty vocal, but maybe they just wanted to make fun of US, and aren't really serious about protecting Snowden. Plus, they support US's surveillance state, so there's little reason why they'd want to protect him, other than being vengeful against US for whatever reason.

Ecuador so far has been pretty good for Assange, and they might continue to want to be seen as "the country that stood up to US", but nobody really knows how far they are willing to go to do that. At least Assange hasn't even been charged with anything by US, yet.

<comment><USER16> Putin didn't really say they support the US's surveillance state, he said surveillance is necessary but you need a warrant.

<comment><USER17> Yeah, careful with that guy. A lot of people who didn't get on with him seem to die in strange ways. But don't talk about it - especially not Pussy Riot style as dissent isn't ok.

<comment><USER18> You know, I find it puzzling that, even though USA has proven itself to be literally on the same level as the "communist" countries(read: China, Russia), people still apply selective blindness.

People die in strange ways in the states too. Michael Hastings much?

<comment><USER19> > though USA has proven itself to be literally on the same level as the > "communist" countries

And people wonder why I complain about seeing political discussions on this site. How can you have a serious discussion with bizarre comments like the above? And just so it's clear: one can simultaneously hold in one's mind the view that there are some serious imperfections and injustices in the US, and that there are a lot of worse places - and some that are probably better.

<comment><USER18> If by serious imperfections and injustices you mean the violation of fundamental human rights, including freedom of speech, that America flaunts right and left.

"What makes America the greatest country in the world? re: Freedom and freedom, and let's keep it that way."

<comment><USER20> And you've got your blinkers on if you think that's really the worst it can get, or if you think Putin "really isn't that bad".

<comment><USER18> That's not at all what I said. The difference is that Putin doesn't spread propaganda through every news channel about how Russia is a truly democratic country and everyone has their freedoms according to the law. Everyone KNOWS he is a tyran, and he doesn't do much to change that.

Now let us observe the american media - 1)America birthplace of freedom? check. 2)America greatest country in the world? check. Despite not leading the world in a single category(besides defense spending). 3)Ideologically and officially different but pragmatically similar political parties? check. America - the birthplace of hypocrisy and the God complex.

<comment><USER20> So this is you being one of those Americans who thinks America is the center of everyone's world. Putin, doesn't broadcast on American networks, in America, because they're American.

What do you think happens in Russia?

<comment><USER18> I'm eastern european, meaning I do have some inside knowledge on how Russia operates, especially since it's still trying to regain political power in the post-soviet countries 23 years after its fall. Everybody knows Russia wants its old empire back.

I do believe my perception of the american media is accurate as well. You're free to dispute that.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> It seems that he got help from Wikileaks' legal team:

>FLASH: WikiLeaks has assisted Mr. Snowden's political asylum in a democratic country, travel papers ans safe exit from Hong Kong. More soon.

[https://twitter.com/wikileaks/status/348724514135347200](https://twitter.com/wikileaks/status/348724514135347200)

>FLASH: Mr. Snowden is currently over Russian airspace accompanied by WikiLeaks legal advisors.

[https://twitter.com/wikileaks/status/348732325909102593](https://twitter.com/wikileaks/status/348732325909102593)

<comment><USER22> Wikileaks legal advisers - he's flying to a country that is putting a dead whistleblower on trial in order to demonise him and cover up government corruption.

Relying on the legal process in Russia is a really good way to find out how different it is from America, however much certain people would like to think differently.

Because that dead whistleblower - he was a lawyer.

</comment></comment><comment><USER23> This story is part of the ongoing propaganda war between the USA and China. The only thing we know for certain is that the Chinese say he has boarded a plane to Moscow, he may or may not have done. He may or may not still be alive. The Chinese may even have secretly handed him back to the Americans. We don't know and it is probably best to be sceptical about everything you read until a journalist and film crew meets him face to face.

<comment><USER24> It seems he's definitely left HK - [https://news.ycombinator.com/item?id=5926668](https://news.ycombinator.com/item?id=5926668)

</comment></comment><comment><USER25> WikiLeaks (@wikileaks) tweeted at 11:20am - 23 Jun 13:

FLASH: Mr. Snowden is currently over Russian airspace accompanied by WikiLeaks legal advisors. ([https://twitter.com/wikileaks/status/348732325909102593](https://twitter.com/wikileaks/status/348732325909102593))

Edit: Oops sorry, somebody posted this already. Anyway, interesting news!

</comment><comment><USER26> For those who want to hypothesize: Here you can check the departures from Moskow after 17:15h

[http://svo.aero/en/timetable/today/](http://svo.aero/en/timetable/today/)

I think destinations in Russia, EU and Asia can be safely filtered out.

<comment><USER27> Thanks for that link. Seem to be a few flights out to Havana. That would be an interesting destination.

<comment><USER28> Seems he is heading to Venezuela through Cuba.

</comment></comment></comment><comment><USER29> "I don't want public attention because I don't want the story to be about me. I want it to be about what the US government is doing."

"he fully expects the US government to attempt to use all its weight to punish him. 'I am not afraid,' he said calmly, 'because this is the choice I've made.'"

[http://m.guardiannews.com/world/2013/jun/09/edward- snowden-n...](http://m.guardiannews.com/world/2013/jun/09/edward-snowden-nsa- whistleblower-surveillance)

</comment><comment><USER30> People interested in this might also want to read the Guardians latest GCHQ revelations. Worryingly they seem to have been gathering more data and there is an interesting section on NSA involvement.

[https://news.ycombinator.com/item?id=5926608](https://news.ycombinator.com/item?id=5926608)

</comment><comment><USER31> I'd be happier if he weren't putting himself under the control of governments that can reasonably be regarded as US adversaries. Thus, I'm with Assange that Ecuador or something was the way for Snowden to go.

Yes, there would be a risk to him of regime change in any such place, but so be it ...

<comment><USER32> I'd like to see what your actions would have been if you were facing life in prison under solitary confinement like Bradley Manning does (or even worse - torture & execution)

Any government that isn't a US adversary would surely extradite him, so it seems reasonable to me that he was in China and now travels through Russia.

Chinese dissidents do the same thing, they flee to Chinese adversaries (like the US)

<comment><USER33> > or even worse - <USER2>

All studies that I know of show that solitary confinement _is_ torture after a few weeks.

<comment><USER32> I absolutely agree with you.

</comment></comment></comment></comment><comment><USER34> Snowden is going to land in Oslo airport, Norway and meet with the leader of our Pirate Party, before going on to his final destination which is supposed to be Iceland. Source, in Norwegian: [https://www.aftenposten.no/nyheter/uriks/Piratpartiet- Snowde...](https://www.aftenposten.no/nyheter/uriks/Piratpartiet-Snowden- mellomlander-pa-Gardermoen-7238358.html)

<comment><USER35> Amazing publicity, there are still a few hours before he lands in Norway then. Sunday is the best day to travel for him propably...

Here is Google Translate of the above article: [http://bit.ly/14RbV5Y](http://bit.ly/14RbV5Y)

<comment><USER36> I wonder if this was an intentional leak. This was reported by the leader of Norway's Pirate Party on Twitter. With the misdirection regarding Cuba and Venzuela, I'm wondering whether he is actually headed for Iceland or somewhere else. Seems weird that they would publish something like this.

Oslo seems like a risky place to land since Norway has historically been very open to requests from US officials. Is anyone able to get in touch with someone involved and see if the leak was intentional, and if not get in touch with Snowden's Wikileaks entourage? (I'm from Norway, btw).

The latest flight leaving Oslo for Reykjavik today is Icelandair's FI325 at 21:55 local time.

</comment></comment></comment><comment><USER37> The flight can be followed in real time on FlightRadar24: [http://fr24.com/AFL213](http://fr24.com/AFL213)

</comment><comment><USER38> Not only his leaks, even his itinerary makes the US look bad.

<comment><USER15> Republican media: "Now he's aiding _all_ the enemies, not just China!"

Because in their minds China and others are actual "enemies".

<comment><USER24> You could have just left this at 'american media'

<comment><USER39> Nope, it's just GOP media, GOP executive branch, GOP congress that are behind the largest crackdown on whistle-blowers in U.S. history~

<comment><USER40> Isn't Obama to blame for this too?

<comment><USER39> Sarcasm. I even used the ~ sarcasm terminator from usenet.

<comment><USER40> I didn't know ~ meant sarcasm.

</comment></comment></comment></comment></comment></comment></comment><comment><USER41> Seems like this is where his carefully considered plan gets much more improvisational. Bad optics, just the same - even if he is on his way to Iceland.

<comment><USER42> Why is Iceland such a bad idea?

<comment><USER43> My guess is that Iceland would probably bend to the US's will pretty easily, especially over such a high profile individual.

<comment><USER18> It always puzzles me how people, especially citizens of Murrica think everyone will bend over and let U.S. have their way.

Small countries? Maybe. See case of latvian hacker Deniss Calovskis [http://bits.blogs.nytimes.com/2013/01/23/u-s-charges- three-w...](http://bits.blogs.nytimes.com/2013/01/23/u-s-charges-three-with- hacking/?_r=0)

Now Latvia is widely regarded as a country that will definitely follow US's whistle. And this extradition case has been going on for, eh, 9 months now?

Think about what independent and "ballsy" countries will do.

</comment></comment></comment></comment><comment><USER44> Imho, given the publicity of the issue, there's no way the current US judicial system could find Snowden guilty of treason without creating a huge domestic and international scandal. Politically, this would turn into an enormous backlash. To maintain the image of a state of law, the US has no other choice than prosecute, but the outcome will turn out as either:

\- a mistrial, followed by a strong attempt at burying the case somehow (probably with other events)

\- a non guilty verdict, with spin doctoring to display the whole thing as a demonstration of the fact that the US constitution and overall legal system is sound,

\- a "guilty" verdict, but with so much attenuating circumstances than there will be no significant sentence

However, for Snowden, I doubt that he will be able to find a job again in his line of work, after breaking the "ethic rules". What security company would hire a (skilled even) security analyst (or whatever Snowden was actually doing) if he cannot keep it shut on the internal practices? The scope could widen to any entreprise actually.

</comment><comment><USER45> "As the HKSAR government has yet to have sufficient information to process the request for provisional warrant of arrest, there is no legal basis to restrict Mr Snowden from leaving Hong Kong."

I think Washington will wake up with a bitter taste in their mouths…

<comment><USER46> After he spent the last two weeks releasing info that helped both Hong Kong and China - I don't think they'll be all that surprise he was granted the ability to leave the country.

<comment><USER47> He also released info that helped US citizens, as we are being kept in the dark in regards to the country's unconstitutional surveillance programs.

</comment></comment></comment><comment> mh- his fellow passengers are going to be rather annoyed if he causes this flight to be detained on the tarmac for hours.

<comment><USER48> Actually it would be awesome to sit next to Snowden on this flight.

<comment><USER40> I wish Snowden good luck, but I wouldn't like to be in the same flight with him. I would be that such flight would have a higher risk of "an accident".

</comment></comment></comment><comment> brown9-2 Didn't he say that he intended to fight any extradition in the HK courts?

</comment><comment><USER49> Cool website. This is what I see: [http://i.imm.io/1a7pX.jpeg](http://i.imm.io/1a7pX.jpeg)

</comment><comment><USER50> Running away from lucifer to fall in the arms of beelzebub?

Weird world.

</comment><comment><USER51> Not sure why he just didn't go to China initially and then use the Phoenix to fly wherever he wanted?

</comment><comment><USER52> In the modern era you can't really hide so he might as well be as public about it as you can be.

Russia is definitely an interesting move, and Putin has been playing the whole thing very smart. I wonder what he knows that the Russians don't, or what they can now reveal having found out through Snowden rather than the usual sources.

It seems to me the value in Snowden is mostly making public what everyone already knew.

<comment><USER3> > making public what everyone already knew

I can't find a link, but there has been a "meme" graphic going around with, I think, a very insightful caption.

For years, you go around trying to raise the alarm, and you're a crazy conspiracy theorist.

Then when you finally get conclusive proof - it's old news, everyone knows, you've been saying it for years!

<comment><USER53> Welcome to rhetorics 101

</comment></comment></comment><comment><USER54> Never heard that we've implemented visa-on-arrival scheme.)

<comment><USER55> No visa necessary if he doesn't leave the airport and gets on another flight within 24 hours.

</comment></comment><comment><USER56> Let's not get cute here.

If the United States wanted him extracted (on the way to Hong Kong) or worse, killed during his sojourn in HK, Snowden would have already faced a fate several orders worse than Alexander V. Litvinenko.

It isn't a huge tab for the U.S. to foot in the way of deploying deep cover teams to "take care" of Snowden - for what had been alleged to be the biggest U.S. intelligence leak in a generation - in the least suspicion-arousing manner long before Greenwald ever published his story.

For a nation that outspends the next ten defense budgets combined (even with the off late sequestration cuts accounted for) this is not much of a task.

The top brass could have taken care of this in a manner several orders more believable than the 2006 killing of Alexander V. Litvinenko in which the uber rare, and thus strictly available to only government bodies, Polonium-210 was used.

They could have made it seem like a forlorn affair of unrequited love with that dancer-acrobat girlfriend of his.

It is acutely nonsensical to assume that movements of sensitive personnel, like Snowden, are not tracked, especially to far-flung locations like HK given that he is not a field operative of any kind.

The NSA and other assorted intelligence/military organizations probably have a protocol for dealing with these things,in conjunction with the Justice department.

If they didn't before, Manning's case would have elicited the need for one.

Source:

[http://www.nytimes.com/2006/12/03/world/europe/03russian.htm...](http://www.nytimes.com/2006/12/03/world/europe/03russian.html?pagewanted=all)

[http://en.wikipedia.org/wiki/Poisoning_of_Alexander_Litvinen...](http://en.wikipedia.org/wiki/Poisoning_of_Alexander_Litvinenko)

[http://nation.time.com/2012/09/25/comparing-defense- budgets-...](http://nation.time.com/2012/09/25/comparing-defense-budgets- apples-to-apples/)

[http://www.forensicmag.com/articles/2009/06/polonium-210-and...](http://www.forensicmag.com/articles/2009/06/polonium-210-and- assassination-alexander-litvinenko#.Uca0-zvs3N4)

[http://www.buzzfeed.com/mbvd/this-might-be-the-girlfriend- ed...](http://www.buzzfeed.com/mbvd/this-might-be-the-girlfriend-edward- snowden-left-behind)

<comment><USER5> Your suggestions are far too cute, actually. By going public, and periodically reminding everyone that he could be targeted for assassination, Snowden has created a strong incentive for FedGov to keep him very much alive.

<comment><USER56> I was referring to the fact that had they wanted they could have scuttled his efforts even before he arranged to speak to Greenwald and Poitras.

If the NSA or BAH were to submit that they were unaware of Snowden's whereabouts or movements when he arranged these interviews, they would in all likeliness be resorting to feigning ignorance.

It's the height of silliness to suggest they were clueless about Snowden's motives before his junket began.

Source:

[http://www.salon.com/2013/06/10/qa_with_laura_poitras_the_wo...](http://www.salon.com/2013/06/10/qa_with_laura_poitras_the_woman_behind_the_nsa_scoops/)

</comment></comment></comment><comment><USER57> Snowden running is not something to be confused with an admission of guilt, but at the same time, it does prove that he is not a patriot. With his worldwide recognition: the safest course would be for him to publicly turn himself in to a US police station or deliver himself to a courtroom. He would have a very public trial and would have an even farther reaching message in this country. By running he shows that he has no faith in this country and no longer considers himself a citizen. Not only that but he opens himself up to a ridiculous amount of risk by being wide open in foreign countries where things are semi-expected to happen to travelers by this country. If something happens to him while he's in the legal system of this country, the people responsible for him would be crucified by the media and everyone else. Granted he could be silenced and have his reputation destroyed by the media while incarcerated in this country but I believe a patriot works with the system or fights within it, only a coward runs.

<comment><USER58> Right, because the United States justice system works so well when its (the US) ass is on the line. Just ask this guy:

[http://www.saveamericafoundation.com/2013/06/21/top-us- journ...](http://www.saveamericafoundation.com/2013/06/21/top-us- journalistmichael-hastings-assassinated-while-attempting-to-reach-israeli- consulate-assassinated-by-sorcha-faal/)

<comment><USER57> This was a man who was running before facing a trial or even charges. Not a man in the justice system. Who knows what really happened to him? Nobody. Because he was being secretive about his actions. He has allowed nothing but conjecture to his fate. I have been under investigation by the FBI; it doesn't mean they're going to kill you.

<comment><USER58> Just the fact that the US President can order an extra-judicial killing of any US citizen, if he deems it fit, should be enough to raise your ire, citizen. With such tyranny allowed, justice is no longer possible in US society.</comment></comment></comment></comment>


Why I'm a Creationist (2013) - <AUTHOR> https://www.yahoo.com/news/why-im-a-creationist-141907217.html <comment><USER2> The author makes several mistakes:

1\. "I still read and read and listen and listen. And I have never found a more compelling story of our origins than the ones that involve God." This is an esthetic judgement, and is therefore beyond criticism. But it doesn't hold for everyone. You find creationism beautiful? Great. Teach it in an art elective.

2\. " I guess I don't "believe" that the world was created in a few days, but what do I know? Seems as plausible (to me) as theoretical astrophysics..." The author doesn't particularly understand a theory, therefore the author throws out the theory. I believe this is the famed "Argument from Ignorance". Please, please, please don't push your ignorance on me.

3\. The author spends two paragraphs describing how science, after collecting more evidence, changes conclusions. Shocking, I know, that when you gain knowledge, your conclusions change. Creationism doesn't change, and can't change, and so eliminates the author's confusion. A fancier, longer way of saying "God did it, so I don't have to think". That's fine. I would be embarassed to admit in public that further evidence can't change my mind.

These kinds of articles about rejecting up-to-date conclusions in favor of some kind of homespun ignorance annoy me to no end. The subtext is that everyone should go on believing what they were taught as children by authorities. I suppose that's whay corporations like Yahoo run them: Yahoo is in the business of being an authority that tells you what to believe. If your belief changes with further evidence, Yahoo just might be out of a job.</comment>


String tokenization in C - <AUTHOR> https://onebyezero.blogspot.com/2018/12/string-tokenization-in-c.html <comment><USER2> The actions of _strtok_ can easily be coded using _strspn_ and _strcspn_.

[https://groups.google.com/forum/message/raw?msg=comp.lang.c/...](https://groups.google.com/forum/message/raw?msg=comp.lang.c/ZhXAlw6VZsA/_Y5evTIkf6kJ) [2001]

[https://groups.google.com/forum/message/raw?msg=comp.lang.c/...](https://groups.google.com/forum/message/raw?msg=comp.lang.c/ff0xFqRPH_Y/Cen0mgciXn8J) [2011 repost]

 _strspn(s, bag)_ calculates the length of the prefix of string _s_ which consists only of the characters in string _bag_. _strcspn(s, bag)_ calculates the length of the prefix of _s_ consisting of characters _not_ in _bag_.

The _bag_ is like a one-character regex class; so that is to say _strspn(s, "abcd")_ is like calculating the length of the token at the front of input _s_ matching the regex [abcd]* , and in the case of _strcspn_ , that becomes [^abcd]* .

<comment><USER3> And it's nicer, since you can pass in a const char * and use it in concurrent code.

</comment></comment><comment><USER4> strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens.

If you program in C please just write those four obvious lines yourself.

<comment><USER5> _If you program in C please just write those four obvious lines yourself._

Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages.

In contrast, an example of truly sickening part of stdlib is converting strings to number. The atoi()/atol() family doesn't check for errors at all so you want to use strtol(). But the way error checking works in strtol() is so complex that the man page has a specific example of how to do it correctly. All sane programmers quickly write a clean wrapper around strtol() to encode the complexity once. Now, strtok() is nothing like that.

In its simplicity, strtok() is quite versatile. A few strtok() calls can easily parse lines like:

        keyword=value1, value2, value3

that you might find in configuration files. And I mean truly in just a few lines which you might expect in Python but with C string handling? No.

<comment><USER4> Here is the musl implementation.

> > [https://github.com/esmil/musl/blob/master/src/string/strtok....](https://github.com/esmil/musl/blob/master/src/string/strtok.c)

It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as soon as you want to do something as basic as parsing from a stream instead of a string that is completely in memory.

So to clarify, of course you are never done with parsing in 4 lines. But even if it wasn't as braindead to overwrite the input string, the functionality strtok provides would not be worth more than 4 lines.

<comment><USER5> So, here's that implementation:

        static char *p;
        if (!s && !(s = p)) return NULL;
        s += strspn(s, sep);
        if (!*s) return p = 0;
        p = s + strcspn(s, sep);
        if (*p) *p++ = 0;
        else p = 0;
        return s;

Instead of carrying that code, or something similar, with my source code or my own utility library I'd much rather have the already debugged version from the standard library.

Overwriting the input in C is more efficient than maintaining more internal state and returning a pointer and the length of each token which you would need to strncpy() to get the token into a C string. strtok() does not want to do the initial strdup() for you because only you will know whether your input can already be mutated or whether you need to use a copy.

As I pointed in the other reply, strtok() _does not break_ on strings like "keyword=value1,, , value3" unless you skipped RTFM and expect it to do something completely different. And more often than not that's exactly what you want when parsing non-computer readable input which you can expect to take a specific form.

If you want to handle escape sequences, parse from a stream (without having the option to fgets() the next line into memory), or parse CSV tables without collapsin colunms then you will want to use something more specific to that. Luckily, strtok() was not advertised as a Swiss army knife so it's off the hook for specific parsing purposes like those.

<comment><USER4> As someone else pointed out what you really want is the implementation of strspn/strcspn which is where the loop is. You don't "carry" that code along. You just write

        while (i < len && !is_token_begin(buf[i]))
            i++;
        if (i == len)
            error("End of input\n");
        start_token(tok);
        while (i < len && is_token_char(buf[i])) {
            i++;
            add_to_token(tok, buf[i]);
        }
        end_token(tok);

or something along those lines. Whatever you need. It's not rocket science. Putting highly fluctuating and project-specific code like this in a library would only have disadvantages. Not everything should be in a library. In fact, most things should not be.

</comment></comment></comment></comment></comment><comment><USER6> I recommend ksplit/ksplit_core from Heng Li's excellent klib kstring.{h,c}[0]. It modifies the string in-place, adding null terminators, and provides a list of offsets into the string. This gives you the flexibility of accessing tokens by index without paying costs of copying or memory allocation.

[0] [https://github.com/attractivechaos/klib](https://github.com/attractivechaos/klib)

</comment><comment><USER7> I have an obsession with unsafe example code:

      strcpy(str,"abc,def,ghi");
      token = strtok(str,",");
      printf("%s \n",token);

Even if the author knows how many tokens are returned I would prefer a check for NULL here since a good fraction might not read further than this bad example.

<comment><USER8> > I have an obsession with unsafe example code:

It is perfectly OK for example code to be unsafe. You do not wear a parachute when you learn to fly using a simulator. You realize that things will become more serious and complicated in the future, but you have to start with something simple and unsafe, no big deal. Otherwise you will never see the consequences of unsafe code in simple cases.

<comment><USER9> I think you underestimate how many people blindly copy examples without understanding them. Safe example code results in more correct programs.

<comment><USER8> > I think you underestimate how many people blindly copy examples without > understanding them. Safe example code results in more correct programs.

Even if this is true, the reasoning here is disturbingly short-sighted. Copying code that you do not understand is unacceptable behavior, and I'd say the sooner it blows up in your face, the better. The goal of code examples is to illustrate how things work in a simplified way, and code without error checks is often easier to understand at first. Imagine a hello world with all the possible error checks. That would be incomprehensible.

</comment></comment></comment></comment><comment><USER10> Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

<comment><USER4> You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.

<comment><USER11> Most lexers are state machines, either explicit with tables (like you get from lex) or implicit with program counter (with loops and switches). Those state machines implement matchers for regular languages; they're effectively hand- coded implementations of regular expression matching.

Regular expressions don't show up outside the spec, sure; but if you're writing the code (for implicit state machine), you need to know exactly where you are in the regular language that defines the tokens to write good code. Writing a regex matcher in code like this is like writing code in assembly - mentally, you're mapping to a different set of concepts all the time.

<comment><USER4> Yes. I don't think anyone is disagreeing here.

If you're implying that we should then use a regex implementation instead: Coding up a lexer (for a mainstream programming language) using simple counter increments and such is not a lot of work. It has the advantage that it results in faster code (unless you're going for real heavy machinery) and that you can easily code up additional transformations. For example, how would you parse string literals (including escape sequences) with a regex?

<comment><USER12> "(.|\\.)*"

<comment><USER4> "Evaluation:\tWrong!\x07\x07\x07\r\n"

You want to convert the string literal to an internal buffer, interpreting the escape sequences. In the same way, you want to parse integers. You cannot really do that with a regular expression. RE is for _matching_ , not for transforming.

</comment></comment></comment></comment></comment></comment><comment><USER13> A lot of experience shows that the string tokenization in Open Object Rexx is darned useful. E.g., for many years, IBM's internal computing was from about 3600 _mainframe_ computers around the world running VM/CMS with a lot of _service machines_ written in Rexx. Rexx is no toy but a powerful, polished, scripting language and really good at handling strings.

A little example of some Rexx code with some string parsing is in

[https://news.ycombinator.com/item?id=18648999](https://news.ycombinator.com/item?id=18648999)

</comment><comment><USER14> It used to be that gcc will warn against strtok and recommend strsep instead. Do not know what the status is today

<comment><USER15> Strtok is not thread safe and can't be made thread safe without changing the API. You should not use it.

<comment><USER16> Well, strtok could use thread local variables to store intermediate state, to make it threadsafe while maintaining the same API. Not saying this is a good idea, but technically it would work, no?

<comment><USER17> Yes, as long as you can guarantee that there's only one tokenization going on per thread at a time.

</comment></comment></comment></comment><comment><USER18> Note though that strsep() is not as portable, because it is an extension to standard C.

<comment><USER19> It's a tiny function, written in ANSI C, so if you're really concerned about this, just include it with your program. It's an extension to the standard C _library_ , not to C itself.

<comment><USER20> Except then you have the issue about compilers complaining about double- declarations of the function, meaning you'll either have a lot of warning spam on every #include or now hard require some kind of header defines for HAVE_STRSEP. Once you go that way, there's no going back and it's only gonna become more and more.

<comment><USER19> People have been including strsep in packages since the 1990s (people used to include their own snprintfs, ffs). If you're really this freaked out about it, call your local copy "mystrsep" or something like that.

You know what else isn't _in POSIX_? All the rest of your C code.

</comment></comment></comment></comment><comment><USER21> > Next, strtok is not thread-safe. That's because it uses a static buffer > internally. So, you should take care that only one thread in your program > calls strtok at a time.

I wonder why strtok() does not use an output parameter similar to scanf() — and return the number of tokens. Something like:

      int strtok(char *str, char *delim, char **tokens);

Granted, it would involve dynamic memory allocation and the implementation that immediately comes to mind would be less efficient than the current implementation, but surely it's worth eliminating the kind of bugs the current strtok() can introduce?

Does anyone here have the historical prospective?

</comment><comment><USER22> Other approach from library calls and flex is re2c. It preprocesses the source code and inlines regular expression parsing where you needed. It's very powerful in combination with goto.

</comment><comment><USER3>
      str = (char *) malloc(sizeof(char) * (strlen(TESTSTRING)+1));

      strcpy(str,TESTSTRING);

str = strdup(TESTSTRING)?

</comment><comment><USER23> AFAIK strtok has restrict on both args since C99. And the safe variants strtok_s and esp. wcstok_s are missing. Strings are unicode nowadays, not ASCII.

[https://en.cppreference.com/w/c/string/byte/strtok](https://en.cppreference.com/w/c/string/byte/strtok)

</comment><comment><USER24> ...And then the application is required to implement variable length characters, a la Unicode, and you start your strings logic all over...

<comment><USER25> As long as you're fine with ascii delimiters, strtok et al. work fine for utf-8 strings.

<comment><USER24> Would you happen to be aware of good Unicode normalization function/lib in C/C++?

</comment></comment></comment><comment><USER26> Problem is that your token string is going to be quite large. Is there a built-in solution for when tokens are just single chars?

</comment><comment><USER27> I just use flex. You don't have to ship flex as a dependency either.

</comment><comment><USER28> How about just using a properly suited language por string manipulation?</comment>


Hooking up: zapping your brain - <AUTHOR> http://thelongandshort.org/issues/season-four/hooking-up-zapping-your-brain.html

<comment><USER2> I thought tDCS was debunked a while ago according to an article that was later posted on reddit's science subreddit.

<comment><USER3> Yes. This one: [http://www.sciencedaily.com/releases/2015/05/150505152140.ht...](http://www.sciencedaily.com/releases/2015/05/150505152140.htm) "Popular electric brain stimulation method used to boost brainpower is detrimental to IQ scores"

</comment></comment><comment><USER3> > Would I get a buzz off electricity? I secretly hoped so.

Great way to have unbiased results.

</comment><comment><USER4> nothing constructive except to applaud "... fitter, happier, and more productive"</comment>


Popcorn Time Just Went Mobile - <AUTHOR> http://techcrunch.com/2014/05/13/popcorn-time-is-now-on-android/?utm_campaign=fb&ncid=fb

<comment><USER2> i think this should be mentioned

[http://www.reddit.com/r/time4popcorn/comments/24a7qd/why_tim...](http://www.reddit.com/r/time4popcorn/comments/24a7qd/why_time4popcorn_is_incredibly_dangerous/)

i would recommend show box as a mobile alternative and the original popcorn time for your PC</comment>


CamFind - Better Image Recognition Than Google Goggles - <AUTHOR> http://camfindapp.com/

<comment><AUTHOR> Yesterday I came across this app. You take the photo and it tells you what it is. We tried lots of shoots. If it cannot find -say the brand name, etc.-, it interprets the taken photo and defines the object.

The results that i received was like below. Black Necklace, Child's Toy Fan, "X" Remote Controller , "X" Water Bottle, Black hanging light with metal surrounding frame. ("X" is the brand name)

How can it do it? Ok, Goggles searches in the images databese to match the image... But this one is interpreting the image. How???

</comment><comment><USER2> No Android - no fun!</comment>


Intelligence Without Reason (1991) [pdf] - <AUTHOR> https://www.ijcai.org/Proceedings/91-1/Papers/089.pdf <comment><USER2> FYI, this is one of the papers discussed in a critique of peer review by author in post discussed recently on HN: [https://news.ycombinator.com/item?id=23280372](https://news.ycombinator.com/item?id=23280372) . It's been on my list to read, so it's great to have a link!

What's surprising to me is a paper that's been cited over 1,000 times had such a difficult time getting published in the first place. It reminds me of a piece of advice for startup ideas that if everyone agrees with you it's probably a bad startup idea, but on the other hand if only a small number agree with you it might be a valuable insight and startup idea. That's a terrible paraphrase, I think, from a PG essay? A similar thing occurs in science it seems, especially in fields where it's harder to test/verify. Truly revolutionary ideas will often be greeted with more skepticism by experts in a field.

<comment><USER3> > That's a terrible paraphrase, I think, from a PG essay?

PG likes to use the phrase _counter intuitive_. The idea, in fewest words, strive for originality/disruption not acceptance. Original ideas are not instantly popular as most people fear originality.

<comment> bra-ket but dumb ideas are often counterintuitive and not instantly popular, so how do you tell them from good ideas

<comment><USER3> Good: It achieves product/market fit and solves its stated problem.

</comment></comment></comment></comment><comment><USER4> Another relevant paper is Intelligence Without Representation

[https://people.csail.mit.edu/brooks/papers/representation.pd...](https://people.csail.mit.edu/brooks/papers/representation.pdf)</comment>


Woman Who Couldn't Be Intimidated by Citigroup Wins $31 Million - <AUTHOR> http://www.bloomberg.com/news/2012-05-31/woman-who-couldn-t-be-intimidated-by-citigroup-wins-31-million.html

<comment><USER2> _Hunt still remembers her first impressions of CitiMortgage's O'Fallon headquarters, a complex of three concrete-and-glass buildings surrounded by manicured lawns and vast parking lots. Inside are endless rows of cubicles where 3,800 employees trade e-mails and conduct conference calls. Hunt says at first she felt like a mouse in a maze._

Accountability is a tough thing to manage when companies get to this stage. The larger the institution, the less accountable every individual within that institution tends to act. Organizational architecture pushes the risk downstream while pulling the reward upstream.

 _Workers had a powerful incentive to push mortgages through the process even if flaws were found: compensation. The pay of CitiMortgage employees all the way up to the division's chief executive officer depended on a high percentage of approved loans, the government's complaint says._

How does this organizational structure differ from the pyramid scheme, or MLM? With scrutiny, it really doesn't. This organizational structure is what has kept the Realtor-Broker model perpetuating for as long as it has: risk essentially becomes decoupled from the reward for Brokers; they can push their Realtors to pursue risky prospects because they get a cut every time. Furthermore, when Realtors collude with their Brokers, they have collective incentive to both negotiate prices _upward_ and take as large of a cut as possible of that inflated amount. Missing from the equation is a system of checks and balances.

CitiMortgage mortgage brokers' compensation being tied to volume means that _turnover_ in the housing market is something they aim(ed) for. There is something seriously wrong with this model.

The fact that nobody in the NAR or mortgage industry is being held accountable is puzzling.

<comment><USER3> > The larger the institution, the less accountable every individual within > that institution tends to act.

I'm reading _Liars and Outliers_ , by Bruce Schneier, and this is one of the big points he makes in his discussion of how trust and security balance to allow society to function.

There is a huge difference between personal morality and organizational morality, for instance. Even if every person working for Citigroup had been basically moral (in the sense that they wouldn't have committed fraud alone) this still could have happened, and probably would have.

The argument that gets repeated ad nauseum, that things like this are the fault of "a few bad apples", is just plain false. The bad apples certainly exist (especially given that our society and economy reward anti-social behaviors), but the incentive structure is the real problem. It's systemic, there's no getting around that fact.

<comment><USER4> TLDR: The more a company grows, the more inefficient and dysfunctional it becomes.

<comment><USER5> Try this: The more a company grows, the more efficiently dysfunctional it becomes...

</comment></comment></comment></comment><comment><USER6> The total fine of $158 million, after Citi got billions in support, is nothing, a mere rounding error on one of their quarterly reports. A simple cost of doing business...

The key take-away is: no one at Citi went to jail for fraud.

<comment><USER7> Bingo! Give this man a cookie. It's bullshit that these guys perpetrated outright fraud like this, and - at the end - nobody was really punished, even though they basically _admitted_ they were guilty.

This shit is what gives capitalism a bad name... I'm as much a proud anarcho- capitalist, Ayn Rand reading, Ludwig von Mises quoting, Gary Johnson supporting libertarian as the next guy, and it pains me to get beat up for believing in capitalism because of these crooks at Citi and BoA, when nobody is being punished for their fraud. Capitalism does _not_ inherently entail accepting fraud like this, but every time these asshats get away with this kind of behavior, it reinforces all the negative images that people have about capitalism.

If we're going to accept the idea that government (or something like government) is to protect property rights and enforce a "rule of law" that protects against this kind of fraud, then our government needs to step the %$!# up and prosecute some of these assholes.

<comment><USER8> In a corporate case it's hard--both legally and practically--to pinpoint someone of guilty mind performing guilty acts beyond reasonable doubt. That's why the corporation is a "legal person" that can itself be legally responsible for stuff and get sued: corporate personhood arose partially as a solution to this problem. This has borne out in legal practice, where even when there's obvious blood in the water, prosecutors have trouble convicting all but a few top guys.

Anyway, I also find your profanity-laden vitrol unbecoming of discussion on this site.

EDIT: I guess it's now considered a sin to state an easily-verifiable truth and ask for polite discourse.

<comment><USER9> > In a corporate case it's hard--both legally and practically--to pinpoint > someone of guilty mind performing guilty acts beyond reasonable doubt.

What? We solved this problem a long time ago. Corporations are beneficial because they provide liability protection to the employees. That's it.

In the case of fraud, we decided the corporate veil should be lifted.

So all you have to do is go through each individual in the corporation, see what laws they broke, and go after them. It's very straightforward.

The problems occur when the corporations received ungodly amounts of money from the government, when a government "appoints" individuals to positions within the corporation, and when there's quite a bit of collusion between the corporate entity and the government.

At that point, the government has very little incentive to prosecute and the corporate entity has even more incentive to commit fraud.

<comment><USER8> "Limited liability" and "corporate veil" refer to shareholder protection, not employee protection. Traditionally shareholders, not employees, were on the hook for lawbreaking.

Anyway, you say something like " _go through each individual in the corporation, see what laws they broke, and go after them_ " as though it's easy. In practice prosecutors have a hard time doing this even when they want to for the reasons I just said. Some of the Enron guys were acquitted by a jury, for example.

</comment></comment></comment></comment></comment><comment><USER10> I've been hearing about this story a lot. Is there an angle?

Is it the idea that folks always have about how "if you go up against company x's legal department you'll get destroyed no matter what"?

One lady winning doesn't really change the odds in general.

<comment><USER11> But it is worth learning about. Also, it shows that it _is_ possible though very unlikely.

</comment></comment><comment><USER12> It sickens me that Citigroup acknowledged they fraudently bought loans without the required documentation and even in some cases said documentation was forged or incomplete and all they got was a $185 million slap on the wrist. What makes matters worse is that the people who have these mortgages are completely oblivious as to what is going on. What makes the whole situation even more laughable is the fact they got billions in bailout cash and yet were only fined $185 million.

They didn't deny the allegations, why didn't someone go to jail over this? They breached mortgage regulations and by the sounds of it most of their purchased loans were fraudelent. Disgusting.

Even after the housing bubble popped it surprises me that banks are still doing these kinds of things. The saying history repeats itself couldn't be more true even if it tried.

</comment><comment><USER13> It's a great story however skeptical of these retrospective views. I wonder how much of her motives were driven by protecting the mortgage holders and the economy; or protecting her career prospects.

Saying this, Hunt seems much more deserving than the former banker in this article:

<http://gawker.com/5553737/>

</comment><comment><USER14> Can we get a version of this story with a less editorialistic headline?

This is clearly a "fits the narrative" type headline, hardy individual versus big bad bank; and I don't _trust_ stories when they fit popular narratives.

Tell me the facts of the case, Bloomberg, I'll make up my own mind how I want to _feel_ about it.

<comment><USER15> Run that in reverse: if the headline were less sensational, you'd automatically trust the article more?

Just read the article. The headline is one sentence. You have to evaluate the whole thing anyway.

</comment></comment><comment><USER16> Wow. that's not an editorialized headline by bloomberg.

<comment><USER17> It's a post of an article from the Bloomberg Markets Magazine.

<comment><USER16> In any case. It's not editorialized at all.

i'll hold up my sarcasm sign in case you missed it.

edit: the 'by bloomberg' remark was meant to convey that it wasn't the OP who editorialized things.</comment></comment></comment>


Inside Elon Musk's Reversal on Taking Tesla Private - <AUTHOR> https://www.nytimes.com/2018/08/25/business/elon-musk-tesla-private.html <comment><USER2> What reversal? He never had the funding to begin with, this is just a damage control exercise.</comment>