The future of databases is RAM clouds and SSD - <AUTHOR> http://www.igvita.com/2009/12/07/future-of-rdbms-is-ram-clouds-ssd

<comment><USER3> The problem for RethinkDB is that the firmware on Intel SSDs is advanced enough that it can speed up the normal, free MySQL engine (or postgresql, or oracle, ...) to almost the same sort of speeds.

I am yet to see any evidence that RethinkDB's approach is anything but a niche for those who need to squeeze an extra 5-10% that advances in filesystems and flash firmware cannot.

<comment><USER4> It's not that simple. The FTL addresses two fundamental problems with flash - slow random write performance and wear leveling. It works beautifully on consumer hardware, but performs rather poorly in an enterprise environment. For some workloads you can squeeze a lot more than 5-10% improvement out of the drives. But that's not the interesting part (most of today's workloads are read-intensive anyway).

The interesting part comes from being able to execute existing operations in fundamentally different ways. When you're not restricted to sequential reads you can implement a huge chunk of database internals differently - <USER2> concurrency models, different implementations of certain joins, etc. This is where the bulk of the performance improvement comes from.

Once you do this, you realize that a traditional bottleneck (I/O) is completely blown away. Suddenly, all kinds of things nobody ever bothered to optimize become performance bottlenecks (for example, the query plan is traditionally interpreted, not compiled to native code - not good enough once you get an I/O subsystem that's 250 times faster than what you used to have).

Once you get rid of the software bottlenecks, you realize that you're limited by the bus. Since SSDs are highly parallelized devices, you have to change the way you interleave operations for optimal performance. That alone is another order of magnitude.

In three years or so, the flash manufacturers will put chips on the SSDs that will allow you to send down programs to them (not unlike the shaders in video cards). That will likely be a bigger revolution than SSDs themselves. Where we are now is just the beginning.

You're right that we haven't shown any evidence that our approach is anything but a niche. We will. This stuff just takes a long time to build.

<comment><USER5> _In three years or so, the flash manufacturers will put chips on the SSDs that will allow you to send down programs to them (not unlike the shaders in video cards). That will likely be a bigger revolution than SSDs themselves. Where we are now is just the beginning._

This sounds crazy. SSDs have all this firmware and processing power to make them behave like hard drives. If we want to drive the underlying storage directly we might as well rip all that out and let the OS drive the flash directly from the main CPU. Why would we want to be sending programs into the black box? What can the firmware do that the main CPU couldn't if we stopped attaching flash as SATA and instead just created a direct connection that can DMA back and forth to main memory?

Some Linux developers have made that point, that they would rather be implementing their own filesystems directly on the flash than dealing with a firmware that is trying to make it appear that the SSD is a hard drive but that then needs TRIM to work around the fact that that is not true. These firmwares are nice for Intel by letting them turn a commodity (flash) into a differentiated (and higher margin) product based on the code in the firmware. Considering Linux already has filesystems that can operate directly over flash and that better ones can be written I'd say we would be better off just getting direct access to the flash.

<comment><USER6> I agree with your point in general, but if coffeebus' assertion that SSD access will become limited by the bus, it still makes sense to allow programs to be sent to run directly on the SSD. Remember, video cards originally were just big ram buffers that were accessed directly by the CPU, but that wasn't fast enough so now we have shaders on the cards. Having something similar in an SSD will allow a lot of data searching and crunching to happen on the SSD, and it'll run in parallel with whatever else the CPU might be busy doing.

<comment><USER7> I see your point, but I'm not convinced that the amount of data (raw volume size) and the read speeds of flash lead to the same behavior as shaders + GPU. GPU's are processing an order (or several) of magnitude more data, which makes shaders a useful feature: data and computation locality.

Flash is 10x slower than RAM (in an ideal case), and even then, we haven't had the need to introduce 'shader like' features into RAM - yes, because historically we haven't stored all that much in it, but even with modern 64Gb configurations, same thing...

<comment><USER6> I don't think the comparison is Flash vs RAM, I think it's "directly-accessed Flash" vs "Flash on the other side of a PCI-E bus". We don't need anything like a shader for RAM because the bus RAM is attached to is fast. The question at hand is whether the Flash on an SDD is best accessed as a SATA device, DMA Flash over PCI-E (or some new slot technology), or through shader-like cpus on the same card as the Flash.

</comment></comment></comment></comment></comment></comment><comment><USER8> That's interesting and very promising. But what I miss in most discussions of relational vs NoSQL is some kind of separation of data model related, performance related and scalability related arguments. These are three rather different issues.

There is no way to have a benchmark alone show me what it means to denormalize my data model. Denormalization can help scalability and sometimes performance, but it has huge implications for maintainability, data quality, code complexity and even organisational matters.

I do understand the advantages of being able to run any algorithm on data and not be limited by SQL. But abolishing normalized data models altogether and replacing them with application code and custom data structures everywhere makes me very skeptical as someone who had to deal with data quality issues for many years.

So that's why I'm glad to see some performance issues of RDBMS improved by new storage architectures and also the innovation going on in terms of recursive queries to deal with hierarchical and graph data structures.

<comment><USER9> The missing discussion point for me is more fundamental - how and when should I break with my RDBMS data modeling habits?

I can't say for sure, but suspect some of the headlong rush into denormalized NoSQL stores is motivated by the ease of the data model. I wonder if once you go down that road, if your "program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of" what you get in a decent RDBMS. (Forgive my hijacking of Greenspun)

That said, I'm usually working in denormalized models anyway (star schemas in data warehouses for handy-dandy slice/dice/drill), so I'm completely open to different ways of approaching data processing and analysis problems.

I actually feel more free with SQL than limited though. What I would really enjoy would be to have the fully expressive power of SQL embedded in a natural feeling way in a higher level language. I'm interested in Q (of KDB) for this reason, have also been baking my brain in prolog to tear down old data model habits, and hacking some in Ruby and Common Lisp for the same reasons.

Of course, I'm probably breaking my career in the process, since I've made a habit of getting paid to work on RDBMS walking the line between hardware/software performance tuning. But because of that, I'm also glad to see new approaches to RDBMS storage and innovation in the whole DB space.

<comment><USER8> I do work with denormalized data structures and schemas as well, but that data is usually derived from a normalized source and it's read-only. All the issues related to redundancy don't exist in a DW/BI scenario.

One area where SQL is clearly limiting is text processing and I'm doing a lot of that. There's just no way you could do text analytics in a regular SQL database. I'm sure there are other areas like that, so I agree with you that innovation in the database space is welcome, even if it has acquired that slightly unfortunate name "NoSQL".

</comment></comment></comment><comment><USER10> One of the most actively-used databases at my company is about 1GB in size and grows at about 100MB/month. It is a Windows-based DB and is very heavy on the reads. I installed SuperVolume <http://www.superspeed.com/desktop/supervolume.php> on the DB server and created a 4GB disk partition to store the DB files. SuperVolume mirrors the entire 4GB disk partition in RAM and works flawlessly and transparently.

I was going to get an SSD for performance enhancement but I don't much see a reason to do that anymore. All the reads happen at the speed of RAM and writes are delayed and non-blocking. Nested queries that used to take 15 minutes now take 45 seconds.

Right now the speed issues we have are solely due to unoptimized algorithms and not due to hardware limitations. This is a wonderful spot for a coder to be in because there is so much room for improvement now. If you have Windows systems serving files and data on slow systems, I would highly recommend giving SuperVolume and its related apps a shot before investing in expensive SSDs.

</comment><comment><USER11> I put SSDs in all my development machines, and it indeed makes compiling faster (about three times). When prices drop a little, I'll also equip our database server with SSDs. Luckily the database is only 60G.

<comment><USER12> Interesting size; I was talking to guys at a gay dating site, and their worldwide db was around 60 GB also; it could fit in RAM, on the 4XL amazon ec2 instances.

</comment></comment><comment><USER13> I'd like to see a proper quantitative analysis done on this subject.

When you take money into account, are SSDs really worth it, or should I just buy more RAM?

<comment><USER14> Properly, Flash is another tier in the storage hierarchy. Faster than disk in important measures, but more expensive. Slower than RAM, but cheaper, lower power, and durable. Unfortunately, the infrastrucure for taking easy advantage of Flash isn't very mature yet. SSDs make flash act like a hard disk, but OSs and applications generally don't allow transparent fined-grained data management to take advantage of it. I guess ZFS is best in this regard, since it can migrate hot data to fast devices, but I haven't seen good options on Linux, and I haven't seen hardware RAID controllers that could fill the gap. In your DB, you can place tables on different devices, or use table partitioning to separate recent data from cold, but that isn't as easy as adding more RAM.

Ultimately though, cost effectiveness really depends on your workload. MySpace is apparently finding it cost effective to use SSDs for their cache tier. A few months back James Hamilton linked to a study that suggested that for some common workloads, the case for SSDs had to include both price/performance and performance/watt, since it didn't pencil out for each individually. That'll change though, as prices drop and performance improves.

</comment></comment><comment><USER15> "RDBMS systems are headed into main memory, which changes the playing field all together"

It doesn't change the fact that RDBMSs don't do a good job of returning representations of graphs.

<comment><USER8> All the major RDBMS do recursive queries nowadays.</comment></comment>


Show HN: Road to Scale – 300+ content items for every stage of your startup - <AUTHOR> https://roadtoscale.com/showhn <comment><AUTHOR> What's up HN! Today I'm launching a fun project that I've built without code (sorry!). Using Carrd, Zapier, Airtable, Google Sheets & Gmail.

Nowadays there's a lot of startup related content around the web. But it's difficult to navigate what to read, watch or listen to. Because in every stage of your startup journey there is different kinds of relevant content you can consume and draw inspiration from.

That's a problem I've been wanting to solve :)

So today I'm launching Road to Scale

It's a curated knowledge library with 300+ content items for every stage of your startup journey. So whether you're just starting out with your idea or scaling like crazy, Road to scale has got your back with insightful, practical and useful knowledge you can apply to your startup.

I hope you like it, thanks for checking it out</comment>


First XBox commercial? - <AUTHOR> http://www.winsyde.com/retro-rewind-watch-one-of-the-first-xbox-live-commercials-from-2002/?utm_source=dlvr.it&utm_medium=twitter

<comment><USER2> Were our tv's really that bad back then?</comment>


This $153k rattlesnake bite is everything wrong with American health care (2015) - <AUTHOR> https://www.washingtonpost.com/news/wonk/wp/2015/07/20/this-153000-rattlesnake-bite-is-everything-wrong-with-american-health-care/ <comment><USER2> I know if I had no health insurance I wouldn't hesitate to go to the emergency room for a snake bite, because I'd rather go bankrupt than die. I wonder if there is a threshold where since the cost of bankruptcy is basically constant (it doesn't matter if you go bankrupt for a $100K bill or a $100M bill unless your assets are >$100K), that the face value of the bill from the hospital can grow without bound beyond that upper boundary, since no one is actually paying it?

<comment><AUTHOR> He [http://www.dailymail.co.uk/news/article-3099453/Woman- wakes-...](http://www.dailymail.co.uk/news/article-3099453/Woman-wakes- boyfriend-37-dead-bed-one-day-bitten-poisonous-water-snake-refused- treatment.html) probably thought that he was going to make it, financial troubles are real and creditors can make your life virtually impossible.

</comment></comment><comment><USER3> I thank Lady Luck that I live in a country with a single-payer system, and health insurance for 'extras'.

Some years back I found I required an urgent triple-bypass following a routine cardiac angiogram. 14 days later I had the triple bypass followed by 2 days in intensive-care. Cost: NIL.

I then had 5 days post-op in a one-bed private room. That was followed by six weekly Cardiac Rehab sessions. Both of these were covered by my health insurance. Out-of-pocket: NIL

<comment><AUTHOR> Your life would have been saved in USA too---followed by a massive and fatal heart attack the second you the preliminary $875,214 bill :). Doctors at the hospital bill you extra a lot of times so the bill just keep growing.

You pay for the medicine, for the nurse administering it, for the syringe, for her breaking the ampule, for mixing it, for disposing the syringe and for the gloves the nurse used. I'm exaggerating somewhat, but they do try to find a lot of little things to pad the bill. So what in another country would have cost $2, in a USA hospital might cost well over $1000.</comment></comment>


Ask HN: What are some good sources of climate data to analyze? - <AUTHOR> I've noticed that articles mentioning climate and climate change that make it to the front page end up being flagged.
I think this is probably fair, because discussions about such articles invariably end up being political and don't end up contributing very positively to HN.
I certainly have thoughts and opinions on this topic, but upon reflection I realize they're not especially well informed. They're mainly based on what I've read in popular media, and I'm hoping I can do better than that.
So I'd like to approach it from another direction. If I want to get some raw climate data for myself, and analyze it using R or Python, what would be a good starting point?
Are there specific data sets that I should focus on? Are there any well known papers that I should read? <comment><USER2> "No One Gives a Fuck About Climate Change" made the front page of Hacker News in June. This inspired a team to created the first Open API for Climate Data. It's a repository of carbon dioxide data as measured on Mauna Loa by NOAA.

The prototype is up and running with data from 1958 - 2017! [http://api.carbondoomsday.com](http://api.carbondoomsday.com) prototype frontend: [http://carbondoomsday.com](http://carbondoomsday.com).

The project is open source on GitHub: [https://github.com/giving-a-fuck-about- climate-change](https://github.com/giving-a-fuck-about-climate-change)

The original blog post that inspired the project: No One Gives a Fuck About Climate Change ([http://titojankowski.com/no-one-gives-a-fck-about-climate- ch...](http://titojankowski.com/no-one-gives-a-fck-about-climate-change/))

Would love to get your perspective and feedback! Maybe you want to follow the project on Github?

<comment><USER2> Also, continue this discussion with other like-minded hackers!

Join the "Giving Fucks about Climate Change" Google Group: [https://groups.google.com/d/forum/giving-fucks-about- climate...](https://groups.google.com/d/forum/giving-fucks-about-climate- change)

</comment></comment><comment><USER3> I have a Master's degree in physics from the University of Copenhagen and I have taught together with one of the leading climate scientists from that university Bo Møllesøe Vinther. He claimed that the best dataset on global average temperatures is satellite based and is the University of Alabama in Huntington dataset (UAH) which goes back to 1979.

You can find it here: [https://www.nsstc.uah.edu/climate/](https://www.nsstc.uah.edu/climate/)

Direct link for monthly temperature averages: [http://www.nsstc.uah.edu/data/msu/v6.0beta/tlt/uahncdc_lt_6....](http://www.nsstc.uah.edu/data/msu/v6.0beta/tlt/uahncdc_lt_6.0beta5.txt)

<comment><USER4> Wouldn't you need data going back for thousands of years? 40 years of data won't tell you anything about the cause of the temperatures in this data set. You'll need to eliminate the possibility that we are in a Little Heat Wave cycle; if you got forty years of temperature data during the beginning of the Little Ice Age, you won't have a representative sample, for instance.

<comment><USER5> Its a unique dataset. it does not portend to be a multi-source long history dataset. you raise a question re: long term trends which is another topic.

</comment></comment></comment><comment><USER6> [http://berkeleyearth.org](http://berkeleyearth.org)

Not just data, but analysis tools as well. Run by a Berkeley physicist who was originally a climate change skeptic and changed his mind after doing this analysis.

<comment><USER7> I looked into this a while back and came to the conclusion that this is the best source.

To combat the "oh no they are 'corrected'" claims, they host the uncorrected, completely raw files too[1]. Then you can have the enjoyable task of trying to work out how to correct for sensors moving, cities growing, etc. Look for the "raw" datasets, as opposed to the Breakpoint Adjusted files.

[1] [http://berkeleyearth.org/data/](http://berkeleyearth.org/data/)

</comment></comment><comment><USER8> It's slightly tangential, but you might find this project, and the surrounding community, to be of interest.

[https://climate.apache.org/](https://climate.apache.org/)

Also, in addition to historical data that's already collected and available online, an interesting side note is that it's possible to directly receive some satellite data yourself using a relatively inexpensive SDR (Software Defined Radio) device. This article deals specifically with receiving images from NOAA satellites, but there are probably other signals floating around out there that would be useful to you.

[https://www.rtl-sdr.com/rtl-sdr-tutorial-receiving-noaa- weat...](https://www.rtl-sdr.com/rtl-sdr-tutorial-receiving-noaa-weather- satellite-images/)

<comment><USER9> The Apache project you link is not tangential at all -- it's spot on! See the examples at:

[https://github.com/apache/climate/tree/master/examples](https://github.com/apache/climate/tree/master/examples)

The people advising the project have significant publications and research experience. I believe some of their examples are replicating peer-reviewed work with that software.

<comment><USER8> Fair enough. I just meant tangential in the sense that the OP was asking about data, not software specifically. But yeah, it is very much about working with climate models. Unfortunately I can't claim any experience working with the project myself... I'm only familiar with it in passing.

<comment><USER9> The authors inhabit the same floor of the lab where I work, so I may have a touch of bias. ;-) I think some data sources are linked with the examples, but I have not checked myself.

</comment></comment></comment></comment><comment><USER10> I follow some websites casually, usually daily, to get an idea of what the numbers are showing me. It's just a ritual I go through in the mornings before going to work.

When I want to see where we are temperature-wise, I go to [http://www.drroyspencer.com/latest-global- temperatures/](http://www.drroyspencer.com/latest-global-temperatures/) for the latest satellite monthly temperatures.

The What's Up with That? website (which is strongly biased towards the "denier" side of the debate) has a great collection of graphs related to arctic and antarctic sea ice: [http://wattsupwiththat.com/reference-pages/sea- ice-page/](http://wattsupwiththat.com/reference-pages/sea-ice-page/)

I go to [http://spaceweather.com/](http://spaceweather.com/) to watch the sunspot numbers (just for curiosity sake), [http://sealevel.colorado.edu/](http://sealevel.colorado.edu/) for sea level information, and [http://www.cpc.noaa.gov/products/analysis_monitoring/ensostu...](http://www.cpc.noaa.gov/products/analysis_monitoring/ensostuff/ensoyears.shtml) to see if we're in an El Nino or a La Nina (or somewhere in the middle).

</comment><comment><USER11> Google's Bigquery has the following huge set of weather measurements, and you can use both the initial trial credit, as well as a substantial free quota every month to run analysis on it right within Bigquery: [https://bigquery.cloud.google.com/dataset/bigquery-public- da...](https://bigquery.cloud.google.com/dataset/bigquery-public- data:noaa_gsod)

Quote: "This public dataset was created by the National Oceanic and Atmospheric Administration (NOAA) and includes global data obtained from the USAF Climatology Center. This dataset covers GSOD data between 1929 and present, collected from over 9000 stations.

Dataset Source: NOAA

Category: Weather

Use: This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source — [http://www.data.gov/privacy- policy#data_policy](http://www.data.gov/privacy-policy#data_policy) — and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.

Update Frequency: daily"

</comment><comment><USER12> Here's a compilation of links to get you started: [https://tamino.wordpress.com/climate-data- links/](https://tamino.wordpress.com/climate-data-links/)

Climate science is a big subject and you should start by understanding the basics of the subject. Spencer Weart's history of climate science is a good place to start: [https://history.aip.org/climate/index.htm](https://history.aip.org/climate/index.htm)

Also, I see many people with strong statistics/engineering backgrounds analyzing climate data and coming to wrong conclusions because they never bothered to learn the physics. Don't be those people.

<comment><AUTHOR> Thanks for the suggestions.

And thanks for the heads up regarding coming to wrong incorrect conclusions. That's important and useful advice.

</comment></comment><comment><USER13> The IPCC full reports are probably the place to start.

However, before embarking on your own analysis, you should consider what questions you're trying to answer and how to avoid misleading yourself. There are two very important and extremely complex areas: proxies and forcing.

Proxies are the use of data other than temperature measurements to infer temperature, such as tree rings and ice cores.

Forcing is the second-order effects of more CO2 such as causing more water evaporation - whic is itself a greenhouse gas.

<comment> pale-hands I think you've identified the two most important topics. A nitpick, though: I disagree with your definition of forcing, it's the same as my definition of feedback.

A forcing is anything that moves the temperature away from the equilibrium between radiative losses and insolation, whether higher or lower. Additional CO2 is itself a forcing.

</comment></comment><comment><USER14> I believe that the dataset published by ICARUS in Maynooth University might be more complete and up to date than some of the others mentioned here. It is a combination of a large number of datasets, is global in scope and runs from 1900 to 2012.

[http://maynoothdigitalcollections.ie/catalog/bg257f05g](http://maynoothdigitalcollections.ie/catalog/bg257f05g)

It is the basis for the two following articles:

[http://onlinelibrary.wiley.com/doi/10.1002/2015JD024583/abst...](http://onlinelibrary.wiley.com/doi/10.1002/2015JD024583/abstract)

[http://onlinelibrary.wiley.com/doi/10.1002/2015JD024584/abst...](http://onlinelibrary.wiley.com/doi/10.1002/2015JD024584/abstract)

IIRC the largest files might not download correctly from this repository linked to above. If that it the case you should contact the researcher Peter Thorne:

[https://www.maynoothuniversity.ie/people/peter- thorne](https://www.maynoothuniversity.ie/people/peter-thorne)

</comment><comment><USER15> I like this data:

[https://gcmd.nasa.gov/KeywordSearch/Metadata.do?Portal=amd&K...](https://gcmd.nasa.gov/KeywordSearch/Metadata.do?Portal=amd&KeywordPath=%5BKeyword%3D%26%23039%3BCH4%26%23039%3B%5D&OrigMetadataNode=GCMD&EntryId=EARTH_LAND_NGDC_PALEO_VOSTOK1&MetadataView=Full&MetadataType=0&lbnode=mdlb5)

And from there:

[https://www.ncdc.noaa.gov/data-access/paleoclimatology- data/...](https://www.ncdc.noaa.gov/data-access/paleoclimatology- data/datasets/ice-core)

And a Wiki article discussing it:

[https://en.wikipedia.org/wiki/File_talk:Vostok_Petit_data.sv...](https://en.wikipedia.org/wiki/File_talk:Vostok_Petit_data.svg)

Where this chart is interesting:

[https://en.wikipedia.org/wiki/File_talk:Vostok_Petit_data.sv...](https://en.wikipedia.org/wiki/File_talk:Vostok_Petit_data.svg#/media/File:Vostok_Petit_data.svg)

</comment><comment><USER16> Right now I am using this data: [https://data.giss.nasa.gov/gistemp/station_data/](https://data.giss.nasa.gov/gistemp/station_data/)

While my motivation is to demonstrate matrix decomposition (see: [https://twitter.com/pmigdal/status/902563470125789188](https://twitter.com/pmigdal/status/902563470125789188)), I wanted to use a credible source. See also a few threads on Open Data Stack Exchange (e.g. [https://opendata.stackexchange.com/questions/1546/how- might-...](https://opendata.stackexchange.com/questions/1546/how-might-i-go- about-visualising-historical-temperature-csv-data)).

<comment><USER8> Since you mentioned the Open Data Stack Exchange, it might also be worth pointing out a couple of sub-reddits that might also be of use to the OP (or anybody):

[https://www.reddit.com/r/datasets/](https://www.reddit.com/r/datasets/)

[https://www.reddit.com/r/opendata/](https://www.reddit.com/r/opendata/)

</comment></comment><comment><USER17> If you're looking for raw data, you can find measurements from NASA's earth observation missions here: [https://earthdata.nasa.gov/](https://earthdata.nasa.gov/)

NASA's Worldview application presents the data as overlays on maps of the Earth and it lets you scroll backward in time from the start of the missions (around 2000 or so) up to near real time (a few hours old). If you want to make sense of the scientific data, you'll still need to put some work into understanding what the instruments are measuring: [https://worldview.earthdata.nasa.gov/](https://worldview.earthdata.nasa.gov/)

</comment><comment><USER18> The National Centers for Environmental Information are probably a good place to start. Collection of links to data can be found here: [https://www.ncdc.noaa.gov/data-access/quick- links](https://www.ncdc.noaa.gov/data-access/quick-links). Alternatively there are some collected sets available here (which may be a little cleaner/ have some metatdata annotation): [https://public.enigma.com/search/climate](https://public.enigma.com/search/climate).

</comment><comment><USER19> Data from European Union's satellites (Copernicus Program) is available for free.

Copernicus Climate Change Service is under development, some data is already available:

[http://climate.copernicus.eu/](http://climate.copernicus.eu/)

Some other Copernicus services may also contain relevant information related to climate change: atmosphere, land, marine..

[http://www.copernicus.eu/main/services](http://www.copernicus.eu/main/services)

</comment><comment><USER20> Oregon University has precip/temperature data for CONUS freely available!

[http://prism.oregonstate.edu/](http://prism.oregonstate.edu/)

</comment><comment><USER21> [http://www.woodfortrees.org/](http://www.woodfortrees.org/) \- excellent stuff here

There's an interesting wrinkle going on in Australia at the moment. The Bureau of Meteorology routinely adjusts the "raw" data (for entirely sensible statistical reasons). However, the net effect of these adjustments is apparently to amplify the warming trend. There's controversy even in the raw data - any analysis produced from it can be criticised. I don't think going to the source is going to resolve the politics.

</comment><comment><USER22> [https://www.climatedataguide.ucar.edu/](https://www.climatedataguide.ucar.edu/)

Quoting from the website "Search and access 194 data sets covering the Atmosphere, Ocean, Land and more. Explore climate indices, reanalyses and satellite data and understand their application to climate model metrics. This is the only data portal that combines data discovery, metadata, figures and world-class expertise on the strengths, limitations and applications of climate data."

</comment><comment><AUTHOR> There's a reply that's dead, but I'd like to address it. It asked:

"We have 25 years or so invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it?"

That's a fair question, but my aim definitely isn't to try to find something wrong with the data. My aim is only to explore the data and find out what's right and what's wrong about my current understanding of the situation. I don't plan to blog about or even discuss the work I do.

<comment><USER10> Also, isn't publishing something in order for others to "find something wrong with it" a core component of the scientific method?

<comment><USER23> sure, but statistically someone who knows how to correctly interpret this data probably already has the academic communication channels to solicit it directly, so someone randomly asking for it online probably has an axe to grind

<comment><AUTHOR> Right, which is why I thought it was a fair point for the original poster of the message to bring up. And it seems like there are lots of people with an axe to grind on this topic.

I'm just looking to learn more for myself, and I decided to ask HN because I was having trouble deciding where to start. I realize it's tough to look at data on a complex topic without a thorough academic background in it. But I figure I'll never learn more if I don't at least get started.

<comment><USER24> Thanks for explaining this, I would have thought the same: I'm a novice, looking for good climate data sets to analyze... asking HN could be a good place to start. Based on the feedback/links provided so far, I think it is!

</comment></comment></comment></comment></comment><comment><USER25> [https://data.giss.nasa.gov/](https://data.giss.nasa.gov/)

</comment><comment><USER26> I've never used it myself, but Richard Muller's Berkely Earth project publish their datasets online.

</comment><comment><USER27> [http://www.ncdc.noaa.gov](http://www.ncdc.noaa.gov)</comment>


Python Decorators: Wonderful and beautiful syntactic sugar  - <AUTHOR> http://avinashv.net/2008/04/python-decorators-syntactic-sugar/

<comment><USER2> Decorators are very wonderful things, but if used as this article shows, they destroy the method signature of the function; fibonacci() goes from being a function that takes one argument to a function that appears as if it will take as many arguments as you give it. (That's the *args syntax; it collects all the arguments into a list.)

Usually this is not a problem, but some people want to see the correct method signature when they look up the documentation (often auto-generated) for a function. And there are some programs that inspect the method signature and expect it to be correct.

[http://www.phyast.pitt.edu/~micheles/python/documentation.ht...](http://www.phyast.pitt.edu/~micheles/python/documentation.html) provides more details on this subject, as well as a library to mitigate this problem. I believe functools.wraps in python2.5 also solves this problem: <http://docs.python.org/lib/module-functools.html>

<comment><AUTHOR> I agree, but as I tried to make clear in the article, I've only tried to make the code as generalized as possible. For the memoize class for example, you wouldn't care. For the safe class, you'd definitely write in more exceptions to handle more issues your source might have. In this process, you'd be writing the proper method signature in.

You're right though, I should have made that clearer.

</comment></comment><comment><USER3> I first came across decorators in TurboGears where they are used extensively for exposing methods, pagination etc (similar to pylons and Django).

They are very useful and insanely easy to use in python.

<comment><AUTHOR> Not really familiar with TurboGears, but it's always awesome to see an app leveraging a language's natural power and flexibility.

I concur about decorators being "insanely easy to use"--taking the memoization example I presented in the article; it's far, far simpler to tack on "@memoize" than to incorporate it into the flow of a function and risk losing clarity. I am sure TurboGears (and Django/Pylons) authors love that you can decorate something with @login_required (or equivalent) and have that method exposed to authentication.

</comment></comment><comment><USER4> This looks similar to aspect-oriented programming. For example, post-sharp (aspects for c-sharp).</comment>


Freelance Camp PRO, San Francisco 2010 - Anybody going? - <AUTHOR> http://freelancecamppro2010sf.eventbrite.com/

<comment><AUTHOR> I'm planning to be in SF during WWDC (even if I won't attend, one of my employees will), so I'm looking for events to attend and meet local people.</comment>


A Git Style Guide - <AUTHOR> https://github.com/agis-/git-style-guide

<comment><USER2> I'm very happy to see this guide encouraging the use of squashing and rebasing. It's really frustrating to see patch sets being merged that look like:

* Add widget

<code review happens>

* Fix typo

* Add test

* Fix test

* etc.

People fear rebasing because they already pushed to a remote server. Just delete the remote branch and re-create it after you've rebased. Assuming, of course, that this is a feature branch of some sort that no one else depends on.

Clean, logical commits and a mostly linear history make me a happy developer.

</comment><comment><USER3> A lot of these items seem to go against current standards for no particular reason (or with no clear explanation). I've mentioned two here [1], but there's several others.

[1]: [https://github.com/agis-/git-style- guide/issues/4](https://github.com/agis-/git-style-guide/issues/4)

<comment><USER4> Old, but still valid: [http://tbaggery.com/2008/04/19/a-note-about-git-commit- messa...](http://tbaggery.com/2008/04/19/a-note-about-git-commit- messages.html)

</comment></comment><comment><USER5> >Prefer dashes over underscores.

Why? Is this just an arbitrary style preference, or is there some practical basis?

<comment><USER6> Alot of (Linux) package managers use dashes within package names and within package versions and underscores to split name from version. This is the convention I try to follow.

Example: package-name_1.0-rc1.pkg</comment></comment>


Tadepalli V. Uber Technologies, Inc - <AUTHOR> https://eclaim.kccllc.net/caclaimforms/utd/home.aspx <comment><USER2> I just received an email about this. It looks like if you used Uber since November 1, 2014, your reimbursement (if it happens) will be in the form of credit to your Uber account. If you haven't used it since then, Uber will issue a credit to the payment account associated with your Uber account. Kind of interesting.

As for how much: "The total amount of Unremitted California Airport Fee Tolls that were charged to Class Members and that Uber will reimburse is approximately $1,785,912.85. The amount that Class Members will receive will vary depending on the Unremitted California Airport Fee Tolls they were charged. The settlement payments to Class Members will not be reduced or used to pay the settlement administration costs, a Plaintiff's service award as approved by the Court, or attorneys' fees as approved by the Court. Uber will pay these expenses separately and in addition to reimbursement of the Unremitted California Airport Fee Tolls it will pay to Class Members."

</comment><comment><AUTHOR> Via an email from Uber California Airport Fee Settlement Administrator <email@example.com> opening with:

> Why did I get this notice? This Notice relates to a proposed settlement > ("Settlement") of a class action lawsuit involving claims that Uber > Technologies, Inc. ("Uber") charged users of its mobile application (the > "Uber App") fees that were described as "Airport Fee Tolls" when users > arranged for rides to or from certain California airports, and that Uber did > not remit those fees to the airport ("Unremitted California Airport Fee > Toll"). You are receiving this notice because you may have paid an > Unremitted California Airport Fee Toll between June 1, 2010 and November 20, > 2015 and may be a "Class Member" in the lawsuit.</comment>


Genetic Algorithms - <AUTHOR> http://xkcd.com/534/

<comment><USER2> Ok, very funny. I just wish that I could actually share xkcd with someone. Even most of my programmer colleagues aren't able to understand the jokes... And this one was particularly bad - I mean, geez, all you need to know is what a Python program looks like, how genetic algorithms work, and what SkyNet is.

I guess it's all just all part of a hacker's curse - we find this stuff funny, and the rest of the world just thinks it's gibberish.

<comment><USER3> Weird. I was just thinking that XKCD has become much too conservative in its geek humor; often, I sigh when reading XKCD because it's become so close to speaking to "laypeople" (abstruse goose does a slightly better job). Then again, the only people I associate with at the moment are academics, as I'm in grad school.</comment></comment>


The "C Is Efficient" Language Fallacy (2006) - <AUTHOR> http://scienceblogs.com/goodmath/2006/11/02/the-c-is-efficient-language-fa/ <comment><USER3> Mods- put a (2006) in there please. The number-crunching ecosystem has changed drastically in 10 years. He also didn't mention what compiler he's using. Intel's ICC had a _huge_ advantage over gcc 2.95.3 (or whatever was used in '06) _especially_ in numerical methods. (Though, I'm not sure when Intel's MKL became semi-freeware; at which point there was a significant performance jump if one opted to use the lib appropriately in tandem with gcc.)

This guy also could have at least linked to a repo of his code... though none of that fancy github stuff was around, Sourceforge and CVSweb was around back then. Without the source we can't really accurately judge, well, much of anything. Compiler flags, JVM tunes (was it warm?), source code, data sets, or hell, even mention of compilers.

Edit: Though his point held hm, not quite 'true' categorically, but you could argue the point. Around '06, Fortran was still largely used, at least bioinformatics (sequencing, attempts at heuristic folding .. stuff), weather prediction, and was a perfect fit for linear algebra.[1] It was accessible and fast enough to be used by post-docs who didn't write software for a living, so presumably the ecosystem was founded then continued to develop within Fortran. Even if it was possible to outperform your simulations in C++ (I'm sure you could get at least fairly close), all of the work you were building off of (e.g. with your colleagues, advisors, etc) were Fortran-based.

[1] This is all hearsay from my father's / his doctoral students work from the late '90s/early '00s when I was in high-school, so I can't really offer sources on hand, but if someone _really_ wants me to, I can fire off an email to him to gather specifics.

<comment><USER4> Fortran is still used in Physics. They have a lot of libraries, and they usually have a some program of a previous paper that almost do what they want now, and they know the technology stack, so many physicist are still not planning to change.

(I work with a Physics group. Sometimes I found programs with an old part that still use numbered lines for the "do". We are still trying to get rid of numbered lines ...)

<comment><USER5> Not to mention that Fortran has come a long ways from your father's (or my) Fortran IV.

<comment><USER6> RatFor for the win... :)... :(

</comment></comment></comment></comment><comment><USER7> An underlying point to this article that is still true is that C isn't inherently fast -- you have to work together with the compiler to make sure it generates what you want.

Why is this even worth pointing out? Well, in many languages communities (Common Lisp is a good example), the speed argument comes up, and it's pointed out that carefully working with the compiler (declares in all the right places and so on) makes the implementation competitive, and this is often dismissed as "too much work", ignoring the fact that even in C, there's no free lunch.

When one recognizes that good performance often requires a dialogue with the compiler, one starts to recognize that a criteria for performant language implementations is not "it's just like C" but rather "it's possible to tell the compiler to make this fast".

This makes many alternate languages viable choices for high-performance work (and eliminates some!), at least based on the current state of their implementations.

Imagine if as much work had gone into their compilers as has gone into the production C and Fortran compilers of today.

<comment><USER8> Except that the default, basic, recommended style for writing in C and writing in some other language X often produces very reasonably fast code in C, and horribly slow code in X. Which was the whole reason people advanced the argument that you need to use contortions to get X to generate code that even approaches the speed of normal, uncontorted C.

<comment><USER9> Except for the little fact that in the 80's and 90's most hobby coders still managed to write better Assembly than C compilers for home computers were able to generate.

C compilers are fast in modern times, because of the amount of industry money spent in writing C optimizers since the industry adopted it.

<comment><USER10> You can still write better assembly today. You just probably wouldn't be able to keep up the effort for the entirety of your code base.

</comment></comment></comment></comment><comment><USER11> The author is wrong about the details but right regarding the general point. Wrongness:

1\. C99 has the restrict keyword to declare no aliasing.

2\. there are still many places dropping down to assembly is better than what your compiler can do, mostly since C is too portable to provide access to processor flags directly (see strlen.s/asm for your platform for an example, e.g. [http://lxr.free- electrons.com/source/arch/arm64/lib/strlen.S](http://lxr.free- electrons.com/source/arch/arm64/lib/strlen.S) ).

3\. (nitpick) Arrays are actually types distinct from pointers in C/++ (e.g. char a[5]; char * b="asdf"; a=b; will not compile, and arrays aren't interchangeable with const-value pointers like char * const either - <USER2> passing a pointer to a function accepting an array).

edit: spacing, fixed italics instead of pointers

<comment><USER12> 3\. nitpick to the nitpick: your example is a bad way to prove that arrays and pointers are distinct: a = a; // also doesn't compile const int x, y; x=y; // also doesn't compile :) The following is more helpful I think: a=b; //error b=a; //fine! sizeof a != sizeof b; // true

</comment></comment><comment><USER13> 0.8 seconds for C and 2.3 seconds for C++? That discrepancy is a huge red flag in this analysis, given that the code is so naive and simple that the program should effectively be the same between C and C++, so I would expect them to have nearly identical timings.

<comment><USER14> We can't know how naive and simple the code is. As far as I can see, the timings have nothing to do with the code snippet in the blog post. Instead they are from some totally separate program, with the source code not available in any language.

<comment><USER13> That's a good point, and an even bigger red flag.

</comment></comment></comment><comment><USER15> The exact meaning of the statement "C is efficient" is that there is no built- in overhead: no virtual machine, no garbage collector, etc. Incidentally, the same is true for C++, with its mantra "you don't pay for what you don't use", except that there is a caveat: the mindless using of the C++ standard library classes could easily result in a significant overhead - mostly as a consequence of a liberal use of the heap for memory allocation. (But therein also lies an advantage: you could write C++ as if it was Java and still get a better efficiency.)

</comment><comment><USER16> Reeks of an anti-C bias. This comment by the author is flat out wrong, and I quote:

""" `int x[1000]` in C or C++ doesn't have much meaning. It's semantically equivalent to declaring "x" as `int *x`. You can reassign X any time you want, to make it point to a different location. """

C is also a pretty different language style-wise than C++. Aside from that, they're both good languages for building runtimes since their memory model resembles physical memory and they can run in an unhosted (runtime-less) environment.

Just a childish rant really.

<comment><USER17> _Obviously_ , the OP was talking about function parameters, where "int x[1000]" _is_ semantically equivalent to "int *x", see C11, 6.9.1, item 10:

[http://www.iso-9899.info/n1570.html#6.9.1](http://www.iso-9899.info/n1570.html#6.9.1)

<comment><USER16> It's not obvious what he was talking about. In context he was responding to this comment:

"While I know that C compilers are really conservative about this, doesn't it matter as to how x and y are declared? After all if they are dclared as: int x[1000], y[1000];"

Those array declarations aren't function argument declarations (except in pre-C89), so not sure why he would talk about semantic equivalence in response to that if it only applies to function arguments.

</comment></comment></comment><comment><USER18> Ehhhhh.... I can't help but think that it's _never_ completely necessary to use totally unrestricted pointers in 'C'. Might be convenient, but very nearly certainly not necessary. Even in systems calls and library functions, it's possible to never dereference a pointer in an unconstrained manner. But you have to be prepared to reason rigorously about those constraints.

Noting that the semantics of array access semi-equate to pointer dereference is well beside the point.

This fellow's "for" loop: for (int i=0; i < 20000) { for (int j=0; j < 20000) { x[i][j] = y[i-2][j+1] * y[i+1][j-2]; } }

shows that you don't even have to have unrestricted pointer access to have bounds violations - what's "y[i-2]" when i==0?

'C' is not for everybody. It's a systems programming language. As the space of activities related to computers grows, there's no good reason to continue to erect a strawman about the language and set fire to it.

<comment><USER19> You can't pass an array to a procedure without it becoming a pointer. But that is not even the issue ... the issue is that even if you know something is an array, any pointer of any type could be pointing anywhere into the interior of that array.

<comment><USER18> If you

1) Know the base address of the array.

2) Know the offset of the element(s) you're passing

2a) Know the declared length ( or other upper limit for the array ).

then you need never refer to any element of an array that doesn't exist.

<comment><USER19> That's not the problem. The problem is any other live pointer could point into that array, for all the compiler knows. Unless the array was generated locally and never aliased, then if you ever call through some set of procedures that the compiler doesn't have complete visibility into, then it is possible that that code generates a pointer into that array and then uses it. The compiler has to be VERY conservative.

</comment></comment></comment></comment><comment><USER20> True, the efficiency argument is wrong. Thinking about your algorithm and knowing where you actually need to be efficient are far more important.

What I appreciate about C in particular is that it seems harder to "program by accident" than it is in most languages. C was my first language. After I learned C, I programmed in a higher level language for a while later before discovering that its runtime threw exceptions on 'array subscript out of bounds.' I discovered it by looking over someone else's shoulder in a lab. I realized then that I had just learned not to do that because C was so unforgiving.

We can argue whether the kind of discipline of thought that you gain from that experience is good or even useful today, but I suspect it is. It's different from learning how to do mental math in a world full of calculators - that's just raw mental exercise.

<comment><USER21> I think that C is a very good first language to learn, not because it hardens you against errors, but because it has a very simple mental model that you can use to understand how other might languages work in terms of C semantics, which provides useful clues for what the costs are for features in those languages.

<comment><USER22> I'm pretty sure that if C was the first language presented to me back in college, I would have found a new major.

C was more convenient than doing assembler, but not much more, compared to say Pascal. (Having learned BASIC, FORTRAN 77, Pascal, PDP-11 ASM, Lisp at school, and some dBASE at work, before we switched over to C at school, which seemed like a big step backward)

        var s : string; multi : integer;
        ...
        s := 'Hello' + ' ' + 'world';
        multi := 616;
        writeln( s, ' ', multi : 4);

vs

        char s[ 256 ]; int multi;
        ...
        s = strcat( strcat( "Hello", " "), "world");
        multi = 616;
        printf( "%s %d4\n" s, multi);

It's a shame no work was put into good Modula 2 compilers (for various platforms) around 85-90, rather than running with C as the standard. Imagine all the security disasters that probably would have been avoided.

<comment><USER21> C was my first language, and I _did_ find a new major (Physics/Mathematics.) I only considered going back for a CS degree after learning Haskell.

</comment></comment></comment></comment><comment><USER23> > In C and C++, there's no such thing as an array – there's just pointers, > which you can subscript and a shorthand for pointer arithmetic and > indirection

This is fallacy #1 about C. C does have arrays which constitute a contiguously allocated piece of memory, it's just that the identifier decays to a pointer in most contexts. When dealing with arrays, there is no indirection involved.

<comment><USER24> He's obviously making comparisons to Fortran's arrays, which would be equivalent to using `restrict` for pointers (in addition to passing the size). This enables some compiler optimizations that can't be done if there is aliasing (ie. two pointers pointing to overlapping areas of memory).

In other words, this claim in the original article is incorrect: "But there's no way to write code in C or C++ that guarantees that. "

</comment></comment><comment><USER25> From the empirical mini-"benchmark" in the article:

      * C: 0.8 seconds.z
      * C++: 2.3 seconds.
      * OCaml: 0.6 seconds *interpreted*, 0.3 seconds fully compiled.
      * Java: 1 minute 20 seconds.
      * Python: over 5 minutes

I have a hunch the Python version did not use numpy because numpy calls to a BLAS library in either C or Fortran to vectorise stuff and should have been close to C and C++ in performance.

Edit: numpy was available in 2006, and even earlier than that.

</comment><comment><USER26> Really good points! For me, the best approach to efficiency in terms of development speed and performance is to use a fast language for critical code paths and then make it accessible using a scripting language. Python and C++ work really great in this respect, as it's very easy to generate Python bindings for a C++ library using e.g. SWIG, SIP or Boost. Lua is also a good choice as an embedable scripting language and widely used e.g. in the game industry (it's faster than Python and has a tiny code footprint).

Recently I also investigated Go as my new "fast language", generating bindings for other languages such as Python is still very difficult though and many of the emerging tools (e.g. gopy) are not very well maintained our outright broken. If integration with other languages was easier I think Go would be a serious contender for this kind of scenario though.

<comment><USER27> From your username & requirements, I'd definitely recommend looking at Julia: [http://docs.julialang.org/en/release-0.4/manual/introduction...](http://docs.julialang.org/en/release-0.4/manual/introduction/)

It can be very fast by itself, and can also be written extremely high level & is very easy to form bindings to other languages, thanks to the multiple dispatch. Even small things such as having full unicode support is very nice to write maths code in a very 'pen and paper' kind of way.

<comment><USER26> Yes Julia is interesting, but for most use cases the languages is (to my knowledge) still significantly slower than C, while not having the great library ecosystem that e.g. Python has. This might change in the future though, and I know that really bright people like Stefan Karpinski are behind the language, so I'm sure it might become a serious contender to Python in the future.

</comment></comment></comment><comment><USER28> I'm not expert but looking at the benchmarks game data:

[http://benchmarksgame.alioth.debian.org/u64q/fortran.html](http://benchmarksgame.alioth.debian.org/u64q/fortran.html)

C seems to match Fortran on about half of them and be 2 or 3x quicker on the others.

<comment><USER29> Beware of putting too much into these benchmarks. They are far too simple to say anything about larger systems. In particular, large systems are to a great extent measured in how well they optimize interactions between different modules in the system. Stuff such as link-time-optimization is not going to be shown via the rather simple benchmarks in the shootout.

<comment><USER30> Yes, but they serve as a reasonable refutation of the specific points made in the article, which focus on small, simple programs.

</comment></comment></comment><comment><USER31> Loved the blog post. Too many people just love jumping the gun and saying that C/C++ are fast languages. I believe it was Chandler Carruth who said that C/C++ aren't performance languages, but rather languages that give you the largest control of your performance.

</comment><comment><USER32> If there is a universal, relatively simple formula for general efficiency, then compilers may be constructed with such algorithm to render certain language more efficient than other. Alas, there is no such thing.

Efficiency, as current situation (before true AI), are always specific. The compiler may implement certain specific optimizations that enable efficiency for certain specific applications -- inline here and there, vectorization for trivial cases, loop unwrapping for some patterns -- but hoping the compiler can work out the true efficiency is naive.

So true efficiency often requires true intelligence. I'll list one example, the famous platform specific linear algebra library blas. It is really efficient but FORTRAN is not the real reason; it is due to the human expert tweaking.

Under the context that real efficiency require human tweaking, there is some advantage of choosing C over most other language, in that C allows more room and more straight forward path for tweaking.

Comparing C against other language (not FORTRAN, which is very much like C), the efficiency of the language is achieved by its compiler doing less rather than doing more. Because the compiler is doing less, it is more straight forward on how to reach efficiency (if the coder sees the efficiency). _With C, getting efficiency is programming_. In comparison, for some higher level programming languages, efficiency is having _faith_ in the language designers and compiler implementers; and tweaking often requires specific knowledge of the language and compiler. For example, knowing that using higher-order functions can be more efficient than a straight loop -- that is beyond general knowledge of programming.

On this, I would like to differentiate C from C++. C++ language (and compiler) is much more complicated than C and much of its efficiency is tweaked by the compiler. For example the '-O' flag is often not so significant for a C program but it is often day and night kind of different for a C++ program (often -O2 or even -O3 is necessary). For C++, significant portion of efficiency is dependent on the compiler, and as such, requires additional knowledge on the coder, therefore, more difficult to reach the efficiency. Of course theoretically, one can write C in C++; and many use that theory to derive that C++ is as if not more efficient than C. That is not the case in reality. Few can contain themselves in a C subset when writing C++ (if they can, why C++ instead of C?) So in reality, if you are using C++, you almost certainly are not writing C and often the efficiency path is more complicated comparing to C.

<comment><USER33> The most efficient thing to do is nothing at all.

What we want is effectiveness.

<comment><USER32> Efficiency is measured by cost and result. Doing nothing end up with no result, how is that "most efficient"? And what is the definition of effectiveness? ... maybe the word you are looking for is convenience?

<comment><USER33> no, efficiency is expenses / revenue

</comment></comment></comment></comment><comment><USER34> So is this mostly just a case of the author never having heard of the restrict keyword?

</comment><comment><USER35> Seeing O(N^3) as runtime complexity for computing LCS was enough of a red flag for me to completely ignore anything that came after the incorrect claim. (For more info, see [https://en.wikipedia.org/wiki/Longest_common_subsequence_pro...](https://en.wikipedia.org/wiki/Longest_common_subsequence_problem))

</comment><comment><USER36> The post was published in 2006.

The claim that Fortran arrays are easier for the compiler to optimise than raw pointers in C/C++ may well be true. But I imagine that there has been progress on compiler technologies to address auto-vectorisation in C/C++ since the post was written (and since C99, C has the restrict keyword too). Can anyone comment?

<comment><USER37> FORTRAN disallows pointer aliasing, which allows the compiler to perform certain instruction re-ordering optimizations. As far as I know this helps more with instruction level parallelism, as opposed to auto-vectorization. C99 introduced the restrict keyword in order to solve the pointer aliasing / instruction reordering issue.

Auto-vectorization is a bit different, it uses mathematics of integer polyhedra to perform loop transformations allowing the compiler to replace scalar operations with SIMD vector operations. The Intel C compiler and IBM XL C compiler have been quite good at this for a long time (at least on simple loops that don't really require transformations). More recently GCC implemented the "Graphite" optimizer to handle this type of auto- vectorization, and LLVM has the the "Polly" optimizer. FORTRAN automatically benefits from these optimization improvements because all these compilers have FORTRAN front-ends that translate these languages to an intermediate representation, where the optimizer don't "know" what the source language was.

I think the pointer aliasing issue still haunts C because the "restrict" keyword may not be commonly used. So FORTRAN seems even today to have a reputation of being "faster" than C for numerical codes.

</comment></comment><comment><USER38> That's not "C is inefficient", it's "Sequental computation is inefficient". Want a lot of "number-crunching"? Learn CUDA (or similar non-hardware-locked thing). It's quite simple to get into and you can get orders of magnitude speed boost. Screw Fortran, it's 2016!

<comment><USER39> _cough_ That still amounts to "C is inefficient." You just gave different evidence that makes a lot more sense in 2016 than it would have in 2006.

<comment><USER38> The whole point is - it's not about a concrete language (CUDA-C syntax is almost the same as C, you know). It's about the way of thinking and completely different parallel algorithms like Blelloch scan.

There is no compiler for any language, which can turn an arbitrary sequental algorithm into parallel, and will never be.

</comment></comment></comment><comment><USER40> This is just a variation on the "what language is best" debate, like we see here. [https://news.ycombinator.com/item?id=11932675](https://news.ycombinator.com/item?id=11932675)

Whilst C based apps can be made to run efficiently, its not efficient to code with the usual tools compared to other languages and IDE's (think big customer development bills), and thats even before you get into the debate of whether some code runs faster with multiple threads on multicore cpu's whilst bearing in mind the ram & bus limits, cpu cache collisions, available cpu instruction sets and so on.

More people really should get over the "awesomeness" of the clockwork turk machine, unless they are easily pleased?

</comment><comment><USER41> I believe modern C/C++ has the restrict keyword which tells the compiler that pointers don't overlap. I'm not sure if this applies to arrays, but if so, it should be fairly trivial for a compiler to optimise parallel execution so that it matches Fortran or OCaml.

What am I missing?

</comment><comment><USER42> While I agree with the author about the main topic, the actual content does not illustrate the point well.

The author wrote a naive loop in a bunch of languages, and, got a faster result in OCaml (0.3 sec) than in C(0.8 sec). Right, but the code was a totally naive loop:

for (int i=0; i < 20000) { for (int j=0; j < 20000) { x[i][j] = y[i-2][j+1] * y[i+1][j-2]; } }

The fact that C has a bunch of libraries for vectorization of such code, or that there are million compiler directives that can speed that code a lot is somehow missing from the article.

So, the conclusion is that a compiled OCaml code is twice as fast as a naively written unoptimized loop in C.

<comment><USER43> That's right: the author wrote a bad argument because they made an invalid performance comparison: naive vs. sophisticated. Is there a C code that can be written that compiles to assembly which is as efficient (or more) than OCaml? _Likely_ so, and if the author doesn't even attempt to do that, I don't find their argument convincing.

I recently learned that Knuth created a virtual instruction set (MMIX) specifically because he found that comparing algorithms with high level languages was too prone to problems. Using a direct instruction set that executes on a virtual machine means that algorithm analysis can focus on the algorithm.

Now, there is something to be said for languages which produce code that is nearly as efficient as the most efficient code you can produce with C, especially if it's easier to write it. But that's not the author's point.

<comment><USER44> _Is there a C code that can be written that compiles to assembly which is as efficient (or more) than OCaml? Likely so, and if the author doesn 't even attempt to do that, I don't find their argument convincing._

That is the point of the author, you certainly can achieve the same speed in C but you have to do extra work because the compiler is unable to do things like discovering the absence of aliasing. So you have to manually build this into the code which in turn makes it more complex.

<comment><USER43> OK, so let's turn it around: are there cases where an expert user who wants to get the most out of the machine can generate faster code in OCaml than the fastest implementation an experienced C author can write?

I don't care about the average case- I care about the maximum case (because I work on performance sensitive code).

<comment><USER44> The fastest code possible is determined by the underlying hardware and is ultimately independent of the language you use. The only reasons that you would not attain this speed limit are that you are either not aware of the best machine specific implementation, that you don't know how to express it in a given language or that a given language does not allow expressing it.

Chances are good that even really good developers are not aware of the best implementation because it might depend on pipeline latencies, constraints on which instructions can execute in parallel and so forth. That is what compilers are good at and I don't think there is an in principle difference here between C and OCaml.

The other two points are more involved. On the one hand C offers, for example, things like intrinsics and seems generally more apt to control what machine code will get generated as compared to OCaml. On the other hand C is less expressive when it comes to conveying higher level semantics to the compiler to inform its optimization decisions.

So if you are aware of the optimal implementation - which, as mentioned above, you may not be - you are in a good position with C to get (close to) the desired result if you put in the extra effort to express the semantic details. On the other hand OCaml will certainly make it harder or even impossible to express your desired machine specific implementation. But then again the OCaml compiler can use information not (easily) available from C programs to guide its optimization and could come up with a sequence of instructions better than what the developer could think of.

In principle OCaml seems to have an edge over C because of the stronger semantics making it easier for the compiler to find a good implementation possibly beating the human. On the other hand, if the compiler does a bad job and you realize it, then C probably offers you the better tools to force the compiler to honor your will.

To finally answer your question, it seems highly unlikely that there are cases where one could in principle not write a C program as fast or faster than the equivalent OCaml program. On the other hand it seems not so unlikely that a developer might not be aware of what that C code would have to look like.

</comment></comment></comment></comment></comment><comment> Const-me I don't do numeric methods, but I occasionally solve performance-sensitive CPU-bound problems, in areas like CAD/CAM, 3D graphics, and GIS.

An average C++ CPU-bound code isn't radically faster compared to e.g. C#. I suppose that's true for other higher-level compiled languages.

But the good thing about C++, after I've profiled which parts of the algorithm take most time, I can gradually replace my average C++ code with SSE or AVX intrinsics. Sure it's harder to read, write and understand, but if done right, on modern hardware it can become several times faster.

</comment><comment><USER45> The gist if this article is that in 2006 some C compilers would produce less efficient machine code than some other language compilers in certain specific type of problems.

</comment><comment><USER46> [https://news.ycombinator.com/item?id=3442172](https://news.ycombinator.com/item?id=3442172)

</comment><comment><USER47> Curious; how close do things like Rust, now or in the near future, come to "replacing" C/C++? Are they even meant to?

<comment><USER48> I don't think languages that are used as much as C and C++ are ever truly get "replaced".

But as for "being a viable choice instead", we intend for Rust to be so. The devil, of course, is in the details: for example, if you're on an obscure or proprietary platform, a C compiler may be your only choice. In the context of this article, while LLVM will autovectorize some things, direct control of SIMD hasn't made it into stable yet, and requires a nightly today. But as far as the language and its power goes, our goal is to be, yes.

</comment></comment><comment><USER30> Against this article, we have the results from the computer language shootout:

[https://benchmarksgame.alioth.debian.org/u64q/performance.ph...](https://benchmarksgame.alioth.debian.org/u64q/performance.php?test=nbody)

I like this because anyone can try to prove their assertion that their language is faster!

<comment><USER49> Unfortunately that site is very questionable although still might be the best resource. The person who runs it changed the interface to make it difficult to compare languages as well as it used to, and he purposely doesn't include many implementations and languages that people send him.

There is no ISPC or Julia, the C and C++ benchmarks use SIMD intrinsics specific to gcc, the benchmarks are run on core 2 processors... etc. etc.

<comment><USER50> _The person who runs it changed the interface to make it_ usable with touch screens.

"Why don't you include language X? … my favorite language implementation? … Microsoft® Windows® ?

Because I know it will take more time than I choose. Been there; done that."

[http://benchmarksgame.alioth.debian.org/play.html#languagex](http://benchmarksgame.alioth.debian.org/play.html#languagex)

<comment><USER49> Either way the interface is not even close to being as useful as it was, and he has removed languages and implementations that were already up and running.

<comment><USER50> More useful for the vast majority who only make one or two comparisons -- the choices they never used are no longer in the way.

I've added new language implementations and programs!

</comment></comment></comment></comment></comment><comment><USER51> It's true that for extremely simple programs like the author's sole microbenchmark, non-C languages can get a speed boost over C from reasoning about aliasing. But this doesn't scale. The author's argument breaks down as soon as you ask for an example benchmark that is larger and smells like something real.

</comment><comment><USER52> It's hard to take any post on efficiency seriously if the author doesn't indicate that they understand the difference between a cache miss and a branch-prediction. Things like that are so huge in modern processors that you will have trouble understanding how things work without it.

</comment><comment><USER53> This article's formatting is awful and very distracting.

</comment><comment><USER54> The formatting on that blog is distractingly poor.

</comment><comment><USER55> as much as it's not part of the standard, __restrict__ allows limiting the effects of pointer aliasing in C/C++. [1]

[1] : [https://en.wikipedia.org/wiki/Restrict](https://en.wikipedia.org/wiki/Restrict)

<comment><USER56> The first line of your reference says that restrict __is__ part of the C99 standard.</comment></comment>