Why a Post-Nuclear World Would Look Nothing Like "Mad Max" - <AUTHOR> http://nautil.us/blog/why-a-post_nuclear-world-would-look-nothing-like-mad-max <comment><USER2> A somewhat less popular cultural reference could be The Road from McCarthy.

Human scavengers roaming what's left, true, but mostly famished and trying to find food before anything else.

Maybe this time is the time we will renew the fear of nuclear annihilation. With Trump elected, Europe is turning to itself for its defence, and is already considering a common nuclear deterrence. Depending on its implementation, this could mean the end of the non-proliferation treaties.

Time to watch Threads again.

<comment><USER3> I must watch Threads given all the talk about it in this thread, but I guess I'll be the lone dissenter that will come out and say that _The Road_ was an absolutely horrible movie. I found it so drawn-out, boring, interminably long. I realize it was presenting life in a post-apocalyptic future, but that's no excuse to have no plot to truly speak of in its 111 minutes of runtime.

Then again, I hated _Mad Max_ and only sat down to watch it at the theater because of the ridiculously-good critical reviews it had amassed. I found it a pointless exercise in hedonism and a testosterone-fueled car chase filled with gore and nudity for their own sake.

<comment><USER4> I'm glad I'm not alone in hating Mad Max. When the credits rolled I was left wondering what the plot even was. It felt like nothing more than an excuse to blow stuff up and chase around cars. Absolutely no idea how it got such high marks.

<comment><USER5> > excuse to blow stuff up and chase around cars.

Is that such a bad thing?

It had a guy playing electric guitar on top of a giant flamethrower, because that's the kind of movie it was.

Maybe not for everyone, but I thought it managed to do what it was trying to.

<comment><USER6> _" It had a guy playing electric guitar on top of a giant flamethrower"_

I'm reminded of the movie _Idiocracy_.

</comment></comment></comment></comment></comment><comment><USER7> Water shortages is a weird topic. The article focuses on finite underground aquifers, but how much world water consumption depends on that (I actually would like to know)?

From what I gather many (most?) cities have water "systems". Dirty water is flushed, evaporation and human processes clean it, then we drink it on the other end. It's a continuous renewable cycle. We are reusing the same molecules, not consuming a limited resources. In many cities, water is nearly unlimited. The system just has to be able to cycle enough for the pipes to be full at any given moment.

Some places like California have too much demand, for the rate at which their water cycles, so they have to be careful. But places like NYC or Chicago have nearly unending supplies of sewage to process.

The amount of water on earth today is the same as 4 billion years ago. It doesn't disappear, it just gets dirty.

The water problem is actually an energy problem. There is more than enough water on earth for probably a trillion people. The problem is it's salty, and desalination is extremely energy intensive. If we figure out cheap renewables, or fusion, or just have to pay a greater percentage of our salaries for water, we're really not going to run out.

I know tfa talks about nuclear holocaust but I wanted to discuss regular water issues/fears.

<comment><USER8> Use of sewage for potable water is still pretty rare.

New York City famously has tunnels up into the mountains for most of their water supply. One of them was featured in Die Hard III, where it was used (while under construction and dry) to smuggle dump trucks full of gold.

Chicago draws water from Lake Michigan and discharges sewage into (artificial) headwaters of the Mississippi.

<comment><USER9> Folks in upstate New York _hate_ NYC for the water supply. Much of the economy of the "southern tier" region is crippled by the requirement to keep water quality pristine (NYC being one of few cities that doesn't significantly process incoming water), and to manage water flows (including that of the Delaware River itself) to optimize the reservoirs, at the expense of fisheries and stuff.

<comment><USER10> Mind, having the Hudson horribly polluted from previous industrial activity in regions like Schenectady/Albany/Troy isn't all that great, either.

</comment></comment></comment></comment><comment><USER11> tl;dr: Sensationalist but unconvincing.

This is because of statements like made here:

> _"But global warming would be over." Why? "Because CO2 production would stop > with the destruction of civilization." Man-made climate change, and the > drought conditions that come with it, would cease._

I find this is just ignoring reality. CO2 content in atmosphere is the result of a balance in the world. It's a result of processes that bring more CO2 there, and process that remove it.

There's plenty of CO2 production that happens without civilization. And almost all of CO2 removal is not the result of civilization. The balance is not quite solid, but it is there.

The man-made climate change is about changing that balance (and the balance with some other gasses like methane). Annual man-made CO2 emissions are however only a small portion of the total CO2 currently in atmosphere, and of total CO2 removed. Even if civilization wouldn't make any CO2 emissions, the CO2 content of atmosphere would not go to zero.

For CO2 levels in atmosphere, there is an annual fluctuation of about 3–9 ppm which is negatively correlated with the Northern Hemisphere's growing season. [1]

Thus, a far more important factor in the CO2 balance of Earth after a nuclear winter would be how vegetation and seawater plankton survive. If large areas of forests were destroyed, and if seas would be poisoned, then the conversion of CO2 to oxygen and absorption of carbon to plankton would change drastically. The impact of that would be vastly larger than the CO2 output of any human industries.

Climate change would not be over because of lack of civilizsation; climate change would be unpredictable because we wouldn't know what are all the things that blew up in the air and what impact they have.

[1] [https://en.wikipedia.org/wiki/Carbon_dioxide_in_Earth's_atmo...](https://en.wikipedia.org/wiki/Carbon_dioxide_in_Earth's_atmosphere)

<comment><USER12> I think the bigger point is that in the face of nuclear annihilation CO2 doesn't matter.

Just maybe the particles launched into th atmosphere, the radioactive fallout and the death of the human race are more important than CO2

<comment><USER11> The death of human race would mainly come via climate change. No nuclear war will directly kill all people, nor will radioactive fallout of even an extensive one. But if the climate would get _really_ much colder, then the human race might be in danger.

For the majority of existence of anatomically modern human being (50 000 years to 200 000 years, depending on definition) the world population has been less than 5 million people (the population of world 10 000 years ago). I think it is quite likely that a similar population could be maintained today, with today's technology, by building artificial environments, running on nuclear power and other modern technology. Think of Caves of Steel. Major powers surely have done some preparations towards that.

It would be what was attributed to Stalin having called "statistics", of course [1], but it wouldn't be the extinction of the human race.

[1] [http://quoteinvestigator.com/2010/05/21/death- statistic/](http://quoteinvestigator.com/2010/05/21/death-statistic/)

<comment><USER12> I am curious as to why you think "No nuclear war will directly kill all people" and then after that climate change would be a problem.

I agree a limited exchange might leave people in hardened bunkers alive. But a full exchange, we have sop many weapons[1] it seems entirely that only the luckiest in hardened bunkers would survive and largely they would emerge in a polluted radioactive wasteland.

There are only about 3,000 cities[2] with more than 10,000 people, and only about 19,000 cities in total. We very nearly have a warhead per city, but a city does not need a warhead to hit it to die, irradiating the farmland will kill off everyone just fine. They will have trouble growing food because of pollution levels, radioactivity.

I guess you could call a spontaneous transformation in a barren wastes "climate change" but I think that would be underselling the problem. I also think doing that does a disservice to the modern political about climate change.

[1] [http://www.icanw.org/the-facts/nuclear- arsenals/](http://www.icanw.org/the-facts/nuclear-arsenals/) [2] [https://www.quora.com/How-many-towns-counties-and-cities- are...](https://www.quora.com/How-many-towns-counties-and-cities-are-in-the- USA)

<comment><USER11> There are lots of smaller cities and towns, and not all people live in cities or towns. Nuclear attacks would not cover all land areas where people live. Thus the detonations would not kill all people directly.

Would the radioactive waste kill people? That is something which is harder to calculate. But the dangers of radiation are much exaggerated, and humans are generally more resilient than most people think.

But nuclear winter, drastically reduced temperatures, particles in the air that prevent vegetation from growing and which thus changes the composition of the gasses in atmosphere? That sounds a real threat.

<comment><USER12> I agree that radiation dangers are exaggerated for most things. The amount of radioactive material left over in most nuclear detonations is staggering. Most fission bombs consume only a tiny percent of their fuel and scatter the rest as particles. Unlike a simple meltdown like Chernobyl there is no container to hold this. Put another way all nuclear weapons are also immensely powerful dirty bombs.

To contrast further, I am well aware of stories of survivors in Japan, even the one survivor who is still alive (last I heard) who was thrown tens of meters twice, once per blast. He and most others only survived because of immediate medical attention and then sustained infrastructure that could not survive a doomsday scenario.

Uranium and Plutonium are some of the most dangerous materials to ingest tiny particles of and these are exactly what will be launched and scattered into all the food supplies. The radiation of being near tiny specks of dust can consume the safe dose of a person for the year and likely do nothing more than increase their chance for cancer in 30 years. Eat that same dust and die of horrible cancer and burns in a few months. It will be in all the food because we have enough warheads, as stated in my previous post, to erase most cities, the resulting damage and fallout would leave no on the planet untouched because the dust would scattered high into the upper atmosphere [1] and it will settle every where

The preposterous devastation of a total nuclear war cannot be understated and again I say that comparing it to climate change does climate change a real disservice. CO2 based climate will disrupt industry and cause wars, but it cannot exterminate humanity. The first volley in a nuclear exchange can exterminate all of humanity before the the next planting of crops. Any comparison to nuclear extinction makes it easy for climate change deniers to claim climate change is hyberbole.

[1]: [https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions](https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions)

</comment></comment></comment></comment></comment></comment><comment><USER13> Please explain to me -- how could 50 measly "Hiroshima-size nuclear weapons" (in TFA) have an effect multiple factors of ten worse than Krakatoa, with ca 200 MT...?

[https://en.wikipedia.org/wiki/Krakatoa#1883_eruption](https://en.wikipedia.org/wiki/Krakatoa#1883_eruption)

Edit: Also, all the fire bombed cities in WW II should at least be in the neighborhood of a large fraction of 100 small nuclear devices. Then we have this: [https://en.wikipedia.org/wiki/Kuwaiti_oil_fires#Oil_fire_smo...](https://en.wikipedia.org/wiki/Kuwaiti_oil_fires#Oil_fire_smoke)

<comment><USER14> So it seems nuclear winter occurs not from the bombs but from the smoke from all the fires they would cause.

[https://en.wikipedia.org/wiki/Nuclear_winter](https://en.wikipedia.org/wiki/Nuclear_winter)

<comment><USER13> So if it was burning cities -- how much worse were Hiroshima/Nagasaki than the firebombing of Tokyo, Dresden and so on?

The years of WW II ought to play in the same division?

Edit: Also see [https://en.wikipedia.org/wiki/Kuwaiti_oil_fires#Oil_fire_smo...](https://en.wikipedia.org/wiki/Kuwaiti_oil_fires#Oil_fire_smoke)

</comment></comment></comment><comment><USER15> It's worth noting that nuclear winter is just an educated guess about what would happen; we obviously don't have any data on how a large-scale nuclear exchange would affect the climate, and there may be other climatic effects that are harder to predict.

I'm skeptical that we would see much effect from an exchange of 50 weapons; the year 1962 saw 178 detonations.[1]

[1] [https://www.armscontrol.org/factsheets/nucleartesttally](https://www.armscontrol.org/factsheets/nucleartesttally)

<comment><USER16> Big difference between dropping a nuke on a remote atoll in the middle of the pacific to a dropping one on a densely populated city.

<comment><USER17> We turned a ton of cities into ashes during WWII without the help of nuclear weapons (except at the very end).

<comment><USER18> The current nuclear arsenal is over 2000 times greater than all of the munitions used in WWII (including the nuclear weapons dropped on Japan)

[https://en.wikipedia.org/wiki/TNT_equivalent](https://en.wikipedia.org/wiki/TNT_equivalent)

* The total energy of all explosives used in World War Two (including the Hiroshima and Nagasaki bombs) is estimated to have been three megatons of TNT

* The total global nuclear arsenal is about 30,000 nuclear warheads with a destructive capacity of 7,000 megatons or 7 gigatons (7,000 million tons) of TNT

<comment><USER13> Firebombings cause a lot of ash with little boom.

And TFA talked about a nuclear winter from burning cities when using a hundred Hiroshima bombs, not the US/Russian arsenal. We ought to have learned about those effects, just from WW II.

</comment></comment></comment></comment></comment><comment><USER19> If any of you want practical advice on how to rebuild after a nuclear (or other) apocalypse let me recommend _The Knoweldge_ by Lewis Dartnell. It's interesting if you're just interested in the history and theory of technology, even.

[http://the-knowledge.org/en-gb/the-book/](http://the-knowledge.org/en-gb/the- book/)

</comment><comment><USER20> Does the whole world really look like that in Fury Road? I haven't seen it, but in the original I don't recall any suggestion that it's global, or even that there was a nuclear war. It was just collapse of society in Australia, which is mostly desert or semi-arid climate.

<comment><USER21> That was always one of the weird things in both the original and Fury Road to me: there are sort-of hints that there are lush (or at least non-desert) lands within a few days, or weeks at most, driving. Why do they spend all their time wasting fuel by driving around in small local circles?

<comment><USER22> Mainly, because that's where you go, if you're a bandit.

The first movie showed a relatively normal city, bordered by a no-mans land. Max + family vacationed on the coast. The last scene, before Max drives out into the desert is basically in a forest. Max's job was to keep the bandits out of the relatively normal area of civilization.

The third movie showed a paradise hidden from most people, cut off by a lifeless desert. It was only by using a plane that they were able to get to a former city with enough resources to build families, again.

It sounds like there's islands of habitable land, and difficult if almost impossible distances between them. No one knows what's happening outside their sphere, except rumors.

<comment> crooked-v Trying to make all of the Mad Max timelines fit into a single-generation timeline is hard at best.

Think about the kind of timescale needed for those War Boys to exist as an entire subculture, or for the Green Place to be populated and well-known before withering up and disappearing. The only way it makes sense to me is to consider it happening over generations - and yet in the first movie, the collapse of society is still only in progress.

</comment></comment></comment></comment><comment><USER23> As far as I know, the post-nuclear world is most accurately described in the 80s UK movie "Threads". It's the bleakest and most depressing movie I've ever seen, like a never ending nightmare. Don't watch it, unless you like being in a bad mood for days.

<comment><USER24> Sheffield was actually just like that in the early 80s (but with more hippies).

</comment></comment><comment><USER25> Similar predictions of catastrophic global winter were made by Carl Sagan regarding the Gulf War oil fires.

In what way is this model any better?

<comment><USER26> In the sense that this talks about a devastation 10000s of times worse than some oil fires.

<comment><USER27> Not really. This talks about a devastation of:

> A limited, regional nuclear war between India and Pakistan in which each > side detonates 50 15 kt weapons could produce about 5 Tg of black carbon > (BC). [1]

That's 5 million metric tons. The footnote on the Wikipedia article [2] says the oil fires produced:

> 16,000 metric tons of actual soot is produced from 220,000 metric tons of > oil burned every day.

They started doing this in January and they didn't finish putting out the fires until November, or almost 300 days. But let's use 200 days as a reasonable approximation, given that some of the fires were out earlier than that. Multiplying 16,000 by 200 gives 3.2 million metric tons of soot, so the nuclear war soot is about 1.5 times worse than the oil fire soot.

Also, consider that humanity has actually performed some 520 atmospheric nuclear explosions (including 8 underwater) with a total yield of 545 Megaton [3]. And yet this 0.75 Megaton conflict is going to plunge the world into winter? I'm not buying it.

[1] [http://onlinelibrary.wiley.com/doi/10.1002/2013EF000205/full](http://onlinelibrary.wiley.com/doi/10.1002/2013EF000205/full) [2] [https://en.wikipedia.org/wiki/Kuwaiti_oil_fires#cite_note-37](https://en.wikipedia.org/wiki/Kuwaiti_oil_fires#cite_note-37) [3] [https://en.wikipedia.org/wiki/List_of_nuclear_weapons_tests](https://en.wikipedia.org/wiki/List_of_nuclear_weapons_tests)

</comment></comment></comment><comment><USER28> No one seems to have mentioned When The Wind Blows.

[http://m.imdb.com/title/tt0090315/](http://m.imdb.com/title/tt0090315/)

It's a gentle little film I saw as a kid. It left an impression. :(

<comment><USER29> I remember it well and share your gloomy response to it.

</comment></comment><comment><USER22> _So why the clear, blue skies, George Miller? Perhaps the once smoke-filled air in Max's world has already cleared._

I'm not sure which Fury Road the author saw, but there are many scenes where there is little daylight coming through, for a variety of environmental reasons.

</comment><comment><USER30> Well thank goodness for this revelation. My addiction to water would have made me quite resentful of those who control it.

</comment><comment><USER31> What is wrong with dressing up in old rubber tires, driving around in sand buggies with flame throwers?

<comment><USER32> The EPA might want a word, which is presumably why they've got to go.

</comment></comment><comment> guard-of-terra Can we try to reproduce Nuclear Winter in case of runaway global warming? Seems that we do have the emergency brake here. Won't even take nuclear weapons - introducing soot into higher layers of atmosphere doesn't seem to hard.

However, this will decrease carbon dioxide consumption by plants and might allow unsafe buildup in the atmosphere.

<comment><USER33> How do you make the soot stay in the higher layers?

Soot partly is a problem now. It lands on the ice/snow and makes the ice melt much faster (due to being black/brown)

</comment></comment><comment><USER34> True, volcanic eruptions in the past caused temporary much colder temperatures: [https://en.wikipedia.org/wiki/Year_Without_a_Summer](https://en.wikipedia.org/wiki/Year_Without_a_Summer)

<comment><USER35> And mass starvations, just from blocked sunlight and decreased crop yield due to sunlight decrease.

</comment></comment><comment><USER36> >> "Because CO2 production would stop with the destruction of civilization."

except CO2 produced by "civilization" is at around 3-4% of the total atmospheric CO2.

</comment><comment><USER37> Isn't Mad Max the film where paper is the most highly prized commodity, but everyone smokes cigarettes?

It's easy to forget that the vast majority of popular speculative fiction is utter bollocks and could never happen. I wish we had more "plausible" sci-fi.

</comment><comment><USER38> Maybe the only larger swaths of lands not carpet bombed with nuclear payloads are arid areas, devoid of any current human population.

After the nuclear fallout these would be the only areas where sustained human life would be possible.

Even a recovering climate wouldn't change much about that.

<comment><USER39> I've been to Ground Zero at both Hiroshima and Nagasaki, and I've spent a lot of time in Nevada - and I'm just fine.

People grossly, grossly overestimate the long-term effects of fallout. The main negative effects of nuclear warfare would be infrastructure destruction and the collapse of food/medicine distribution - fallout wouldn't even register.

At the height of the Cold War, a full-on strategic exchange between the US and the USSR wouldn't have ended human civilization. Things would've been unpleasant in parts of North America and Eurasia for a time, but all these extreme scenarios that are sensationalized wouldn'tve come to pass.

Look at Chernobyl - it's essentially a thriving wildlife reservation, now.

<comment><USER40> You grossly overestimate the relevance of your own experience to the discussion of modern nuclear weapons. Those explosions were all either airbursts, purposefully designed to avoid fallout, or very low yield weapons when compared to modern weaponry.

All of which is of very little relevance to discussing modern nukes intended to cause fallout.

<comment><USER41> An airburst not only minimizes fallout, it also maximizes the damage done by a nuclear weapon. I would expect a nuclear exchange to be primarily airbust weapons, considering the intent is to destroy the enemy's infrastructure and fighting capability.

Mutually Assured Destruction refers to the combatant countries, after all, not to the human species.

<comment><USER40> One would hope so, but tactical ground bursts could irradiate entire agricultural regions in ways that airbursts couldn't.

I'd expect any MAD plan to include airbursts for big cities and ground bursts for breadbaskets.

</comment></comment></comment></comment></comment><comment><USER42> Mad Max is not a post nuclear world.

<comment><USER26> Following a nuclear holocaust, the world has become a desert wasteland and civilization has collapsed.

[https://en.wikipedia.org/wiki/Mad_Max:_Fury_Road#Plot](https://en.wikipedia.org/wiki/Mad_Max:_Fury_Road#Plot)

<comment><USER43> Yes, Fury Road was the first one to mention that. It's basically a retcon of what the proper Mad Max films have established.

<comment><USER44> They're all by the same writer/director, so "proper" doesn't really apply.

</comment></comment></comment></comment><comment><USER45> Reading that I'm left thinking that perhaps nuclear war wouldn't be so bad, well except the billions dead and the radioactivity. But we don't have to worry about global warming and water scarcity so we have that going for us.

</comment><comment><USER46> This hasn't been thought through enough, with nuclear reactors not maintained by people they will melt-down and pour out lots of radiation on top of that produced by the bombs. A lot of life will be destroyed.

</comment><comment><USER47> wait, Mad Max is not a documentary ;)

With a full scale nuclear exchange we would have a nuclear winter for a the first few years (when the radiation would be the highest). This is an interesting read that looks at actual US government procedures following and exchange. It's a bit dated, but was written during the height of the cold war. [https://www.amazon.com/Day-After-World-War- III/dp/0670258806](https://www.amazon.com/Day-After-World-War- III/dp/0670258806)

</comment><comment><USER48> I assume a worldwide nuclear war would be an extinction level event. Perhaps roaches will inherit the earth, but I doubt humans will.

<comment><USER49> I don't think that was ever really the case - there was a vast amount of over targeting where the same location would be hit tens or even hundreds of times (notably Moscow - apparently due for ~400 warheads in the worst days of the Cold War).

</comment></comment><comment><USER50> Did they forget Mad Mx takes place in the Australian Outback? It currently isn't so different than the Mad Max version...

</comment><comment><USER3> > So why the clear, blue skies, George Miller? Perhaps the once smoke-filled > air in Max's world has already cleared. But Robock predicts that process > would take around 30 years. Why, then, does Max still look so young?

I think they accidentally a word. 30 years == young.

<comment><USER51> Max was already an adult at the time of the apocalypse.

<comment><USER3> Thanks, I thought they meant the world of _Mad Max_.

</comment></comment></comment><comment><USER52> Next he'll be saying the Zombies / Infected arent right in Walking Dead / 28 days.

This is a work of fiction, no ?

</comment><comment><USER53> The whole article and not one mention of radioactive fallout? The radioactivity from a thermonuclear bomb spreads through the regional atmosphere and remains in contaminated soil for centuries...

</comment><comment><USER54> What worries me is that even today, there're nut jobs on our side (and I'm sure on the other side also) who believe that a nuclear war is winnable.

During the very brief Russian-Georgian war, some folks under Bush (Bush disagreed with them) actually toyed with the idea of bombing Russia.

And we're still baiting them in Syria.

<comment><USER6> You might enjoy a movie called _" Seven Days in May"_:

[https://en.wikipedia.org/wiki/Seven_days_in_may](https://en.wikipedia.org/wiki/Seven_days_in_may)</comment></comment>


Early access open for rumii, VR collaboration for remote teams - <AUTHOR> http://www.dogheadsimulations.com/earlyaccessprogram <comment><AUTHOR> This is very early access for rumii, VR teleconferencing and collaboration for remote teams. Currently supports HTC Vive and desktop. Atlassian Jira integration, shared web browser, shared desktop, positional audio, multiple rooms, web cam support. Upcoming features: FBX import, record your meetings and re-visit in VR, avatar customization, many 3rd party widgets.

I've been using rumii on average for 7 hours a week with people in 3 continents to do all my sprint planning meets with Jira. Zero lag issues.</comment>


Pybind11 — Seamless operability between C++11 and Python - <AUTHOR> https://github.com/pybind/pybind11 <comment><USER2> Some may find it useful: I've written a Jupyter notebook extension for pybind11 a while ago, so you can write C++ in the notebook and it gets automagically compiled and imported into the kernel (plus you get C++ syntax highlighting and a few other goodies):

[https://github.com/aldanor/ipybind](https://github.com/aldanor/ipybind)

(still not on pypi but the plan is to release it soon)

</comment><comment><USER3> I've been using boost::python, and the experience is what I'd call cargo-cult programming. You copy and paste stuff off the tutorial, and when you get a compiler error, you just try other stuff at random, because the inner workings of the library are inscrutable. I vastly prefer cython to this. Is Pybind11 any better?

<comment><USER4> Pybind is derived from boost::python, so it's not much different. I wrote a large Python binding in the past for a moving C++ library target, using boost::python, and keeping the C++ binding code up to date was a nightmare akin to maintaining a fork of any fast-moving project.

Try cppyy [1]. It's very nice, though quite fresh. It's used extensively (as far as I can tell) at CERN, and derives from their cling C++ JIT interpreter. Plus it does lots of nice automatic transformations to make a C++ API more pythonic.

[1] [http://cppyy.readthedocs.io/en/latest/index.html](http://cppyy.readthedocs.io/en/latest/index.html)

<comment><USER5> Not quite -- while pybind11 was originally inspired by boost::python, it is an entirely different project that is intended to address many of its problems.

<comment><USER4> Indeed; I was using "derived" somewhat loosely. But I think there is certainly a visible lineage there. Indeed, pybind11 is much nicer than boost::python (and I did evaluate it for the future of the project I mentioned above).

However, the very nice feature of cppyy is that it does much (most?) of what pybind11 does, but it can also be completely on-the-fly, in the sense that it relies on the cling JIT interpreter. This means that there is absolutely no need to maintain a compiled C++ part for your bindings, and so the problem of keeping the interface up to date is greatly mitigated: the equivalent changes to match the interface when using cppyy are _much_ smaller.

Often, one maintains a "C++ interface" layer in one's Python bindings (which could be created by pybind11, boost::python, or SWIG, for example), with a pure Python shim layer on top of that. cppyy allows you to do away with this two-layer structure entirely; all you need is the shim, if you need anything at all.

<comment><USER2> The catch with cling and the derived libraries is in that you have to download a whole bunch [1] of CERN stuff and then build a customized LLVM as part of the build process. That's a bit too heavy for the nice reflection- and REPL- like features that you gain.

        [1] https://github.com/antocuni/cppyy-backend/blob/master/create_src_directory.py

Boost.Python is better in that regard since you "just" have to build boost; on some platforms, you can just snatch that via a package manager; that being said, you still need to build it. SWIG, aside from being ugly, requires an extra build step.

> This means that there is absolutely no need to maintain a compiled C++ part > for your bindings, and so the problem of keeping the interface up to date is > greatly mitigated

I tend to disagree. I would never consider the raw (swig or cling) 1-to-1 bindings of C++ code satisfactory for end-user use in Python. Ideally (in my subjective opinion and previous experience) Python-side bindings would closely mirror the C++ API, to the point where downstream code in either language looks very similar, but they don't reference any C++ stuff, be it vectors, or maps, or template arguments, or anything else. This implies you would have to maintain a set of higher level bindings on top of swig/cling ones anyway -- and _these_ are the ones that'll break as the code evolves and that you'll have to maintain manually. As such, I'd rather maintain one set of bindings than two.

<comment><USER4> You're right about the CERN stuff, though recent efforts seem to have been made to split at least some of that out. I hope that it continues. I think, aside from anything else, that cling is a really cool project, and if it could be easily available more widely, that would be great.

I was being quite literal when I wrote "no need to maintain a compiled C++ part": of course, you probably do want to maintain /some/ extra layer! And, in that sense, I do think that cppyy lets you maintain just one set of bindings (not two); and, as in pybind11, the ultimate aim is to transparently translate any "vectors, or maps, or template arguments" into idiomatic Python: this is why cppyy has a 'Pythonization' API.

Perhaps there are just two slightly different niches: cppyy is good when you need more a interactive interface to C++, for prototyping or exploration (for example), because of its JIT nature; and pybind11 is good for building something more static in the longer term, and where you don't mind the cost of keeping the compiled part up to date with the relevant C++ API.

It's certainly an interesting space at the moment, and I do hope both projects keep the momentum up and keep innovating!

<comment><USER6> I'm the author of cppyy and was just made aware of this thread.

The big dependency is LLVM, not any more CERN code (there's some left, but it nowhere near takes up the disk space or compilation time than the patched version of LLVM does). The CERN code exist b/c LLVM APIs are all lookup based. The bit of leftover code merely turns that into enumerable data structures. Once pre-compiled modules can be deployed, everything can be lookup. That will greatly reduce the memory footprint, too, by making everyhing lazy.

It is hard to trim down the source of LLVM, but trimming the binary is easier to achieve and that's what I'm working on right now. The end result should be a binary wheel less than 50MB that is usable across all python interpreters on your system, and would be updated something like twice a year. Since that gets it down to a level where even an average phone won't blink, pushing it beyond that leads to vastly diminishing levels of return, and I'll leave it at that unless a compelling use case comes along.

That said, there is an alternative: on the developer side, you can use cppyy to generate code for CFFI ([http://cffi.readthedocs.io/en/latest/](http://cffi.readthedocs.io/en/latest/)). The upshot is that LLVM only has to live on the developer machine, but would not be part of any deployed package. Of course, w/o LLVM, you have to do without such goodies as automatic template instantion.

Finally, note that cppyy was never designed with the same use case as e.g. pybind11 in mind. Tools like that (and SWIG, etc.) are for developers who want to provide python bindings to their C++ package. The original idea behind cppyy (going back to 2001), was to allow python programmers that live in a C++ world access to those C++ packages, without having to touch C++ directly (or wait for the C++ developers to come around and provide bindings). Hence the emphasis on 100% automation (with the limitations that come with that). The reflection technology already existed for I/O, and by piggy-backing on top of it, at the very least a python programmer could access all experimental data and the most important framework and analysis classes out-of-the-box, with zero effort.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> other projects in that space are: cython (need to manually do wrapping), xdress, SWIG, SIP, clif.

Does anyone have experience with using these and able to compare/contrast?

I'm afraid of C++, but if I can wrap it and learn how to use from a python REPL, I think I can handle... any recommendations/tuotirals/howtos would be much appreciated. (The library I want to wrap is [https://github.com/openzim/libzim](https://github.com/openzim/libzim) )

<comment><USER8> I built bindings for various C++ libraries in SWIG (not only for Python but also Lua and Ruby) and I was always very impressed with the result. What's amazing about SWIG is its ability to easily port very complex and rich code. I successfully exported very complex class hierarchies and code relying heavily on pointers and STL to Python and/or Lua. The process consists of writing several header-like configuration files that SWIG ingests and uses to generate bindings, which is often pretty straightforward. Often you can simply import the normal C/C++ header files in those configs and add some glue code / extra hints to help SWIG in cases where it can't figure out what to do with a particular type.

In my experience, the most difficult aspect of binding generation is not writing the glue code, but doing so in a scalable way. Manually writing bindings for any real-world C++ codebase would therefore be extremely tedious (IMHO), so having an automated system that does this work for you is a huge time saver.

Some larger libraries / frameworks have their own wrapping generators Btw, PyQt (which provides extremely good bindings to QT) for example has SIP which is worth looking at (it's open source).

<comment><USER9> Have you used pybind? I've used SWIG extensively so I'm familiar with that - you talked about SWIG but not how it compares to this new one.

Reading through some docs, it looks like pybind suits better for a python codebase that occasionally needs C++ pieces added in as you have to setup the C++ code to handle the py needs.

SWIG I was always impressed with exactly your point. You could sometimes just make 1 big header file in a facade pattern to expose some "start" buttons needed to run a huge c++ application and nothing else needed to be setup. No going back and forth.

Example usage here was a single heavy duty image processing application but using python has the distributor and work manager. C++ code was a "worker" and python and zeroMQ did everything else. So the only "connection" we needed was a start button and some metadata to the c++. Maybe 1 header file with 100-200 lines of code to make the bridge for SWIG.

(to anyone else reading) does that sound trivially simple to the do the same with pybind?

<comment><USER8> I haven't used pybind yet, so I can't compare the two unfortunately. From how I understand the docs it seems you would have to write the interface definitions in C++ and the compiler would generate the bindings for you, which sounds like a nice approach that could be a bit more cumbersome though as (again, in my understanding) forces you to write and adapt all bindings by hand.

<comment><USER9> >could be a bit more cumbersome though as (again, in my understanding) forces you to write and adapt all bindings by hand.

That was my interpretation as well.

</comment></comment></comment></comment></comment><comment><USER10> At what point should I start thinking about rewriting computationally expensive parts of my Python application in C++ and making bindings?

Does anyone have experience taking one expensive method and replacing it with C/C++? Were the trade offs worth it?

<comment><USER11> First, try to optimize your Python. It's unexpected what you can do with it. E.G: slicing assignment is crazy fast, removing calls and lookups goes a long way, using built-ins + generators + @lru_cache + slicing wins a lot, etc. Also Python 3.6 is faster, so upgrading is nice.

Then, you try pypy. It may very well run your code 10 times faster with no work on your part.

If you can't or the result is not as good as expected, you can start using EXISTING compiled extensions. numpy, uvloop, ujson, etc.

After that, and only now, should you think about a rewrite. Numba for number related code, cython for classic hot paths or nuikta for the entire app: they can turn Python code into compiled code, and will prevent a rewrite in a low level tech.

If all of that failed, congratulation, you are part of the 0.00001%.

So rewriting bottlenecks in a compiled language is a good option. C or C++ will do. But remember you can do it in Rust too!

<comment><USER12> I love Python, but this is one of the pain points. Working through a sequence of domain specific languages when you could have just written it in a fast one to begin with (e.g. Julia or C++).

<comment><USER11> You usually don't, that's the point.

You may, late in the project, write partially some of your code in a DSL.

While with C++, you start from the beginning with a handicap for the whole project.

<comment><USER13> The reason so few projects are rewritten in C/C++ is that many people know up front that their project will require that performance and just start there.

If you are building a high end 3d video game with anything like current fancy graphics no amount of python or ruby is going to make it work. You must start with C or C++ to make effective use of modern hardware (even using the C# unity provides leaves a lot of performance on the table).

If you are building a system designed to be faster that some other well defined system then starting with C or C++ is a good idea. If your Java or C# system could handle 1 million transactions a second you might be able to complete 1.5 million/s with C++.

Some projects never need that level of performance, building those projects on C++ can cost you some time. Most webpages are in that vein, how many hits a day does a typical website get? only a few of the biggest retailers and search engine need that level of performance.

That time cost is also shrinking, but not shrinking as fast as I would like. C++11, 14 and 17 it took chunks off development time by polishing some of the sharp corners of the language. Memory leaks are harder to make. Threads and time are easier to work with. Error message are better than ever.

There is still progress to make. Every C++ project still needs some time dedicated to configuring the build system. There needs to be some plan for checking for memory issues, there needs to be... I think C++ will continue to get more Rust-like and Rust will continue to grow in popularity and performance. Eventually, I think Rust or something like it will be the preferred high performance language.

</comment></comment></comment></comment></comment><comment><USER14> Has anyone looked at [https://pypi.python.org/pypi/cppyy](https://pypi.python.org/pypi/cppyy) ? This seems to be even easier than pybind11

</comment><comment><USER15> I like how the first basic example invokes undefined behaviour without mention:

      int add(int i, int j) {
          return i + j;
      }

<comment><USER16> Looks fine to me. Are you supposed to check for integer overflow before doing the addition, or something?

<comment><USER17> Yep,

        add(INT64_MAX, 1)

Is undefined.

If using Ada, similar kind of code would produce "raised CONSTRAINT_ERROR : program.adb:7 overflow check failed"

To be fair, Ada's behavior is easy to reproduce with a checked integer class like [https://accu.org/index.php/journals/324](https://accu.org/index.php/journals/324)

<comment><USER18> So?

There millions of lines of code, from the best C/C++ programmers, even the kernel, that add two ints in all kinds of programs.

Why is this suddenly a valid concern, especially for a code example, not NASA's missile code or Tesla's self-driving libs?

<comment><USER17> It is not a sudden concern, it has been ignored since C and C++ exist, and only became worse with code exposed to the world via the Internet.

"Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to--they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."

\-- Tony Hoare, "The 1980 ACM Turing Award Lecture"

Millions of people also drive without security belt or helmet, apparently they are an useless extra.</comment></comment></comment></comment></comment>


Command-line tools can be faster than a Hadoop cluster (2014) - <AUTHOR> https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html <comment><USER2> With all the Hipster tech being released recently, the headline statement holds true for a lot of things, unfortunately.

We recently discussed new logging tools at work. It was either a redundant Amazon EC2 cluster with ElasticSearch for $50K monthly, or two large bare metal servers with rsyslog and grep for $400 monthly. The log ingestion and search performance was roughly the same...

EDIT: To give everyone a sense of scale, those $200 each bare metal servers are 2x Intel Xeon 6-core + 256GB RAM + 15x 10TB 7200 rpm. We retain logs for 30 days and handle 4-5TB per day.

<comment><USER3> What cloud advocates always say is that the $50k monthly will save you money from not needing to hire a team to manage it for you, and that over the course of 10+ years you will be ahead.

Is that true in anyone's experience? Every once in a while somebody posts about their competing bare-metal system and it looks like a lot of people have managed to cut their server costs by 99% (based on the numbers they post) by avoiding the cloud as a service

Honestly curious

<comment><USER4> We maintain 30 bare metal servers at a colo center, and between me (primarily a developer) and the CTO we spend maybe 1 day per month "managing" them. The last time we had a hardware failure was months ago. The last hardware emergency was years ago.

Servers run on electricity, not sysadmin powered hamster wheels.

<comment><USER5> Yes, the maintenance is cheap. The changes are more costly.

We run a dozen bare metal servers and I see the difference what it takes to spin up a new VM vs. set up a new physical server. There's planning, OS installation (we use preseed images but we weren't able to automate everything), sometimes the redundant network setup doesn't play well with what the switches expect (so you need to call the datacenter).

Still, it works out in favor of the bare metal servers. But I'm looking forward to a bit bigger scale to justify a MaaS tool to avoid this gruntwork.

<comment><USER6> I completely agree that some types of changes are much more expensive with a bare metal architecture than with cloud.

6 years ago, I worked for a company in the mobile space. This was around the time of the Candy Crush boom, and our traffic and processing/storage needs doubled roughly every six months. Our primary data center was rented space co- located near our urban office. For a while, our sysadmins could simply drive over and rack more servers. We reached a point where our cages were full, and the data center was not willing rent us adjacent space. We were now looking at a very large project to stand up additional capacity elsewhere to augment what we had (with pretty serious implications on the architecture of the whole system beyond the hardware) or move the whole operation to a larger space.

This problem ended up hamstringing the business for many months, as many of our decisions were affected by concern about hitting the scale ceiling. We also devoted significant engineering/sysadmin resources to dealing with this problem instead of building new features to grow the business. If the company had chosen a cloud provider or even VPS, it would have been less critical to try to guess how much capacity we'd need a few years down the road to avoid the physical ceiling we dealt with.

<comment><USER5> Yes, the cloud premium is also a kind of insurance - you know you'll probably be able to double your capacity anytime you need it.

</comment></comment></comment></comment></comment></comment><comment><USER7> I did some testing on the same (kind of) dataset and task:

First test: A single 2.9GB file

time rg Result all.pgn | sort --radixsort | uniq -c 13 [Result " _" ] 1106547 [Result "0-1"] 1377248 [Result "1-0"] 1077663 [Result "1/2-1/2"] rg Result all.pgn 1.12s user 0.55s system 99% cpu 1.680 total sort --radixsort 3.87s user 0.37s system 71% cpu 5.911 total uniq -c 2.69s user 0.02s system 45% cpu 5.909 total

Using Apache Flink and a naive implementation It took 13.969 seconds.

Second test: same dataset, split between 4 files

time rg Result chessdata/_ | awk -F ':' '{print $2}' \- | sort --radixsort | uniq -c 13 [Result " _" ] 1106547 [Result "0-1"] 1377248 [Result "1-0"] 1077663 [Result "1/2-1/2"] rg Result chessdata/_ 1.70s user 0.97s system 42% cpu 6.292 total awk -F ':' '{print $2}' \- 5.47s user 0.07s system 88% cpu 6.289 total sort --radixsort 4.13s user 0.42s system 43% cpu 10.559 total uniq -c 2.73s user 0.03s system 26% cpu 10.559 total

Flink: 12.724s

Conclusion: For this kind of workload, both approaches have comparable runtimes, even tough taco bell programming has the upper hand (as is should for simply filtering a text file). It took me about equally long to implement both. I think both approaches have their use case.

I ran this locally on my Laptop with 4 logical cores.

<comment><USER8> Hadoop is very slow, because it persist the data to disk before every stage. You really wouldn't want to use Hadoop if you don't have a good reason too. More modern tools like Spark and Flink fare better there.

</comment></comment><comment><USER9> A thread from 2018: [https://news.ycombinator.com/item?id=17135841](https://news.ycombinator.com/item?id=17135841)

2016: [https://news.ycombinator.com/item?id=12472905](https://news.ycombinator.com/item?id=12472905)

2015: [https://news.ycombinator.com/item?id=8908462](https://news.ycombinator.com/item?id=8908462)

<comment><USER10> It's just the annual expedition for HN where everyone gets their turn to be smarter than everyone else.

Even more irrelevant now because Hadoop is largely a dead-end technology.

</comment></comment><comment><USER11> A classic from 2015 along the same lines: Scalability, but at what COST?

[http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...](http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html)

</comment><comment><USER12> The author is experimenting with 1.75Gigs of data. At that scale sure, a local machine will be faster. Hadoop's real use-case though is when your data doesn't fit in memory and even this is kind of debatable. It makes sense to measure the performance with some prototypes and then make a final design rather than just use whatever AWS offers. Besides packaged services in AWS are also a bit more costly than basic services like EC2 instances and network goodies.

<comment><USER13> These days you can get servers with Terabytes of RAM, so a lot of people (most?) could fit their data in memory.

I just took a gander to HPE's website and you can get proliant servers with up to 12 TB of RAM (you might be able to get them with more RAM, did not check in detail)

<comment><USER12> HPE ?

<comment><USER14> Hewlett-Packard Enterprise

</comment></comment></comment></comment><comment><USER5> This reminds me my experience from a company internal hackathon. My colleague started writing a Spark program that would process the data we needed (a few hundreds GB uncompressed). Before he finished writing it, I was able to process all the data on a single machine with a unix pipeline. The computationally intensive steps were basically just grep, sort and uniq. When he finished the program, it couldn't run because of some operational issues on the cluster at the moment, so we didn't even find out the speed to compare.

For me, the morale is that the cheap hardware saves money/time twice:

1\. It's faster if a program can run on a single machine.

2\. It's easier to write a program that runs on a single machine.

With this in mind, cloud works great for analytical data processing. Just start a big enough machine, download data, do the computation, upload the result and turn the machine off. If you develop the program on a sample of the data so you can do it locally, it will be even cheap because you use only short time of the powerful server.

<comment><USER15> Given you can soon get beefy 64 core Threadripper based workstation under 10K just running the analysis locally looks like a very decent option.

</comment></comment><comment><USER16> The two approaches aren't necessarily mutually exclusive. Spark can easily shell out using pipe(). Plus you can use that to compose and schedule arbitrarily large data sets through your bash pipeline through a multi-node cluster.

Beyond that, while the Unix tools are amazing for per-line FIFO-based processing, they really don't do a great job at anything requiring any sort of relational algebra.

<comment><USER17> join and comm would like a word with you.

You can't match SQL expressiveness, but you can definitely handle set-based stuff.

</comment></comment><comment><USER18> Wait... are you telling me people over-engineer solutions to ultimately simple problems? You're _kidding_.

</comment><comment><USER19> very simple processing, not memory bound, tiny data-set - of course its going to be faster locally when the command itself takes less time than the networking, distribution, coordination and collation overhead of using any distributed tool...

<comment><USER20> You know that, I know that, and we can be happy that we have the experience to know what the right tool for this job would be by sizing up and describing the characteristics of the problem like you just did. But those with less experience may not be able to do that unless shown stuff like this in practice.

Some may think this problem requires MapReduce. The quote from the original implementation blog post certainly seems to indicate so.

<comment><USER10> MapReduce as a paradigm and technology was popular about a decade ago and then died shortly after in favour of Hive, Spark etc.

Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.

<comment><USER21> Some places still seem to go for large Kafka clusters just to calc some stats and forward some messages. I am sure some of their solutions are MapReduce below.

<comment><USER10> Very curious to understand more about these mythical developers who are recommending a technology/paradigm that stopped being used a decade ago.

You've seen people writing pages and pages of Java code to do ETL ?

<comment><USER22> I worked for a large multinational last year that had multiple teams rolling out terrible Hadoop solutions.

The rest of the world does not particularly care about SV fashion trends.

</comment></comment></comment></comment></comment></comment><comment><USER23> once you get to the stage where your laptop is just not enough anymore (or your laptop has some cores you want to add to the processing as well), gnu parallel might be of use.

[https://www.gnu.org/software/parallel/](https://www.gnu.org/software/parallel/)

</comment><comment><USER24> Is there any benefit of

      cat files* | grep pattern

over this

      grep -h pattern files*

aside from result color highlighting?

<comment><USER25> With the first method, you can use xargs to run multiple copies of grep running in parallel like he did later in the article.

<comment><USER26> The author switched to using _find_ before doing the parallelization.

</comment></comment></comment><comment><USER27> When all you have is a hammer, every problem starts looking like a nail.

The basic premise is fine: If you have a simple problem, using simple tools will give you a good result. Here you have text files, you just want to iterate through them and find a result from ONE line that's the same in every file, collate the results. No further analysis required.

Every problem in the world can be solved by a bash one-liner, right!?

There's an interesting dichotomy with bash scripts: One school says any bash script over 100 lines should be rewritten in Python, because it's overcomplex already. Another school says any Python script used daily over 100 lines should be rewritten in bash so there are no delusions about it being easy to maintain.

The original article is from 2013, and doesn't try to do any optimization (I guess, the original article is unavailable at the time of writing of this comment), so it would be an interesting question to see what you could do at the Hadoop end to make the query faster. I would imagine quite a lot.

</comment><comment><USER28> If you can fit your data on a single disk drive, you don't need Hadoop.

</comment><comment><USER29> The bottom 90% of data users are in the gigabytes range. Anything works.

<comment><USER30> I've been in a meeting where the amazing scalable cloud solution for the huge data warehouse was laid out. Turned out to be 500GB. Judging by the death stares I got I don't think I was supposed to say "Wow, the whole thing would fit on the 512GB SD card I bought last week".

</comment></comment><comment><USER31> We had a poorly performing service which reads from a number of rest endpoints and writes to s3 in date prefixed format. Offshore wrote 3,600 lines of codes targeting kinesis firehose. By just piping the url endpoints to a named pipe and cycling the s3 file in python, my code was 55 lines of code and did the same thing without kinesis. Wrapping things in GNU parallel and using bash flags, it handles any failure cases super gracefully, which is something the offshore code did not do. The India offshore code had a global exception catch-all, and would print the error and return exit success return code... but I guess someone got to put Kinesis on their resume.

</comment><comment><USER32> I maintain here a very small command line cheatsheet that I get back for reference for mostly data analysis tasks [https://tinyurl.com/tomercli](https://tinyurl.com/tomercli)

<comment><USER33> It's not accessible. Can you publish it?

<comment><USER32> yeah sorry here: [https://sites.google.com/view/tomerbendavid/commandline](https://sites.google.com/view/tomerbendavid/commandline)

</comment></comment></comment><comment><USER34> Been saying that for years. Also, get this, 99.999% of companies do not need "big data" or distributed systems of any kind. I feel like the old "cheap commodity hardware" pendulum swung way too far. More expensive, less "commodity" hardware can often be cheaper, if correctly deployed. I.e. you don't need a distributed database if your database is below 1TB and QPS is reasonable (and what's "reasonable" can surprise you today with large NVME SSDs, hundreds of gigabytes of RAM, and 64-core machines being affordable).

</comment><comment><USER35> This was a straw man article in 2014, it was a straw man article the other times it's been posted to HN in the intervening years, and it's still a straw man article in 2020. As noted in another comment here, the contemporary technology of Apache Flink really isn't far off command-line tools running on a single machine. Meanwhile, HDFS has made a lot of progress on its overhead, particularly unnecessary buffer copies. There are datasets where a Hadoop approach makes sense. But not for ones where the data fits in RAM on a single system. No one has ever argued that.

</comment><comment><USER36> While I personally would use a similar pipeline as OP for such a small data set, saying Hadoop would take 50min for this is just flat wrong. It shows a clear lack of understanding of how to use Hadoop.

</comment><comment><USER37> Amen. You can do a lot with pipes, various utils (sed, awk, grep, gnu parallel, etc.), sockets, so on and so forth. I see folks abuse Hadoop way too often for simple jobs.

<comment><USER38> I am always tempted to say too that "vim can be faster than IDE x"... But I guess that is a bit more subjective.

</comment></comment><comment><USER39> That's because Hadoop is a big favorite of wining and dining suits, who scram at the sight of the command line.

</comment><comment><USER40> If you're disappointed with the speed and complexity of your Hadoop cluster, and especially if you're trying to crack a bit, you should give ClickHouse a spin.

<comment><USER41> >crack a bit

What does that mean? I don't understand if you're trying to endorse ClickHouse or make fun of it.

<comment><USER40> Phone typo. Should have been 'nut'.

And yes, I'm endorsing ClickHouse; it scales down much better than Hadoop.

</comment></comment></comment><comment><USER42> if you're doing spark or hadoop today and are a python shop...you should definitely look at Dask [https://dask.org/](https://dask.org/)

works as good as spark. very lightweight. works through docker.

Ground up integrated with kubernetes (runs on EKS/GKE,etc).

and no serialization betweek java/python, fatjar stuff, etc

</comment><comment><USER43> command line tools like grep,awk,sed,etc are great for structured and line- based files like logs. For json documents I can add a recommendation for jq:

[https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)

</comment><comment><USER44> Cloud computing is kind of a joke. Yeah keep paying someone for shared "virtual computers", that sounds suspiciously similar to shared hosting from a decade or 2 ago... Oh but this is different, you get isolation from containers/VMs! Yeah ok, meanwhile new exploits emerge every couple weeks. It's like tech debt ideology on steroids... just keep pumping out instances until the company either goes hyperbolic, or goes bankrupt. Realistically, just buy a few physical servers and actually work to build efficiency into the system instead of just throwing compute at your public-facing web app.

I recently bought a DELL r710 just for fun and was pleasantly surprised how even days after spinning up a bunch of VMs, I don't have a 30gb logfile for all the failed attempts at getting into my instance (this was my experience recently with 2 cloud providers!)

It'll be interesting to see how the mkt reacts when you have a "first-of-its- kind" massive, massive security breach that affect popular "pure play" internet companies hosted on top of the mythical "cloud."

Seriously, $READER, look at your cloud computing-dependent startup, and calculate egress costs for your storage, as if you HAD to stop using cloud tomorrow. How much does it cost you? How could you adapt? It's designed to keep you dependent on 3rd parties... Idk, IMO it is really not great.

<comment><USER2> I see one really positive point in cloud computing and that is that I can soundly sleep at night :)

Of course, cloud is overpriced, slow, and suffers from noisy neighbors. And keeping things running in the cloud is about the same amount of work as keeping it running on bare-metal. But for customer-visible things, I want to use cloud so that someone else has to get up in the middle of the night when apache crashes.

Sleeping peacefully makes it worth for me to pay $5,000 monthly to Heroku when 2-3x $100 bare metal servers would do. Plus I can cheaply insure against supplier negligence, whereas insuring against employee negligence would be much more expensive.

<comment><USER45> Noisy neighbors tracking can be automated and partially solved then. With cloud you gain access to the provisioning API and with Terraform/Ansible stack (as for example) you are able to build up and manage infra quickly, efficiently and declaratively. Bare metal provisioning can also be automated (as via private cloud solution, for example) but you need dedicated team for that (and nice OpenStackers aren't cheap). I was solo managing 500+ hosts once on public cloud, there is no chance you can do this without what you call "hipster tech" and modern devops toolchain.</comment></comment></comment>