Caches and Lisp: faster list processing via auto-rearranging memory (2003) [pdf] - <AUTHOR> http://www.cs.berkeley.edu/~fateman/papers/cachelisp.pdf

<comment><USER2> In section 4 they found that traversing an organized list is FASTER than traversing an array of the same size when both fit into the L1 cache. I find this highly implausible [0] because 1) traversing a list needs more instructions, 2) both L1 and L2 cache miss numbers are smaller for the array. The authors provide no explanation for the result either.

[0] I believe they got these numbers, but they should have dug more into the cause instead of accepting the conclusion at face value. Given this is a LISP benchmark, maybe array bounds checks are costly? Maybe a different result would have been obtained had they instructed the compiler to generate unsafe code?

<comment><USER3> It's to do with branch prediction.

[http://stackoverflow.com/questions/11227809/why-is- processin...](http://stackoverflow.com/questions/11227809/why-is-processing-a- sorted-array-faster-than-an-unsorted-array)

<comment><USER4> Seems unlikely: in the experiments here the lists are not sorted in the sense of the stack overflow question (i.e. they are not in numerical or alphabetical order). They are sorted (in memory) by their index in the linked list. By definition the array is already in this order.

Unless I've missed something?

</comment></comment></comment><comment><USER5> Interesting that it doesn't mention CDR-coding - I guess that's considered competitive only with hardware support. There's been some work by Appel on trying to improve performance by unrolling (in his case, immutable) cons lists, too.

</comment><comment><USER6> A fascinating idea.

I haven't digest the entire paper yet.

I am wondering already whether this idea would be combined with the schemes which yield cache-oblivious algorithms.</comment>


Tesla's Roadrunner project: new battery production at $100 per kWh - <AUTHOR> https://electrek.co/2020/02/26/tesla-secret-roadrunner-project-battery-production-massive-scale/ <comment><USER2> The important point is that if cells get down to $100/kWh, then ev's could have sticker prices equal to ice's. That would cause ev sales to skyrocket, which is absolutely essential for saving the climate.

<comment><USER3> This is important, but also note that the Model 3 cost ($40k-$60k) is above the average new vehicle price ($35k) and Tesla still can't build enough to meet demand.

If Tesla hits $100/kWh, combustion vehicles are done. They will never have a lack of buyers while legacy automakers collapse under their exciting capital cost structure (Ford is already in junk territory, and other automakers aren't far behind).

<comment><USER4> It's even worse than that, the price to beat is the Honda Fit and Toyota Corolla at about $20k

</comment></comment></comment><comment><USER5> The article doesn't say what the current cost per kWh is? I'm just curious to contextualize how big of a leap a $100 per kWh cell would be.

Additional question: Does a Lithium-Ion cell for a moving vehicle need to be higher quality/greater tolerance, relative to stationary? Just seems like a Lithium-Ion cell built for a vehicle needs to survive low/high temps, and also a lot of jostling or risk fire/venting.

<comment><USER6> There is a difference between Battery Cell and Battery Pack, which the article wasn't very clear although I seriously doubt $100/kwh is for Battery Pack.

Current estimate of Battery Pack is roughly ~$150 / Kwh.

<comment><USER7> Commenting on your comment, the cost of the battery pack is roughly 70% cells and 30% battery pack. Using those numbers, cells at $100/kwh would result in packs costing roughly $143/kwh, vs the current situation where cells are costing them roughly $105/kwh.

It doesn't seem like enough of a drop to warrant the hyperbole in the article if is a ~5% drop in cell price. Maybe they are talking about making packs for $100/kwh, which would be more impressive.

Given that pack prices have been dropping 18%/yr, pack prices dropping to $100/kwh is about 2.5 years away. Hiring for a production process that matures ~2 years in the future seems feasible. Maybe they are talking about pack prices?

Educated guesses of pack vs cell cost: [https://about.bnef.com/blog/behind- scenes-take-lithium-ion-b...](https://about.bnef.com/blog/behind-scenes-take- lithium-ion-battery-prices/)</comment></comment></comment>


Cold Comforts: Antarctic Research Bases Are Seriously Self-Sustaining - <AUTHOR> http://www.wired.com/magazine/2010/04/ff_antarctica

<comment><USER2> Very keen, but the line about Amundsen-Scott Station's supply chain points out the obvious: as efficient as they must be, they fail to be "self-sustaining" in the most glaring ways.</comment>


Steve Jobs preparing for his first TV appearance - <AUTHOR> http://www.youtube.com/watch?v=FzDBiUemCSY

<comment><USER2> Wow that is very early on for sure. Funny how they talk about using the bathroom and him saying he is deathly ill and ready to throw up

</comment><comment><USER3> so good to see him healthy.</comment>


A quote on Startup - <AUTHOR>

Startups win by doing something small, really really well. Usually 10 features does not sell better than one good feature! <comment><AUTHOR> I strongly believe on this!

<comment><USER2> Im currently working on this. I guess we are pretty much on the same track! But its pretty rare to see startups with brilliant ideas! Good point though.

</comment></comment><comment><USER3> True.</comment>


Let's build Babbage's ultimate mechanical computer - <AUTHOR> http://www.newscientist.com/article/mg20827915.500-lets-build-babbages-ultimate-mechanical-computer.html

<comment><USER2> Not that I wish the author anything but the best, but he's not alone in this.

Tim Robinson, of Meccano/Erector Difference Engine fame, has been working on building the Analytical Engine in Meccano for some time. His website (<http://meccano.us/analytical_engine/index.html>) shows a working control barrel from three years ago; I believe from memory (I'm on a mailing list with him) that the design and construction has progressed since then, though I don't have the details to hand.

I wish them both luck, I'd love to see both designs up and working at the first possible opportunity. Tim's Difference Engine was a thing of beauty to watch (there certainly were videos online) and had a fantastic sound in operation.

<comment><USER3> I have talked to Tim about this so he is aware. And recently Doron Swade (who built the Difference Engine No. 2) has joined the project: <http://plan28.org/> for details.

<comment><USER2> Cool, glad to hear it :-) Best wishes to you all. Been very impressed with what I've seen of Tim's work, hope you can all learn from each other.

</comment></comment></comment><comment><USER4> I want to know - WHAT di Babbage's analytical engine SOUND LIKE? Wouldn't this make an awesome sound effect for modern computers?

</comment><comment><USER5> Can we say that Babbage's projects failed because he was unable to create a team to help him complete it? He was also not very driven toward one particular design, stopping half-way and starting a yet more complex project.

<comment><USER6> Well, Babbage never wanted to build the Analytical Engine, he said so explicitly. He just wanted to design it, and let someone else build it. The reason was his experience with the Difference Engine. Because the D.E. would be useful for accurate, error-free charts for the British Navy, he felt that the Government should pay for it. And he did get an agreement with the Gov, but getting the money out of them was like getting water out of a stone. He was so disgusted by the whole process, that he resolved neither to finish the D.E. nor to even start the A.E. Other than that, I think there is no question that the Analytical Engine could have been built in the 19th century.

</comment></comment><comment><USER7> It would be interesting to build this, then run Ada Lovelace's program on it. Imagine that - running software that was written in Victorian times on a computer that should have been built in Victorian times.

</comment><comment><USER8> If I had the money I would absolutely commission some watchmakers to make an analytical engine with modern, tiny ass clock work.</comment>


The US Air Force's radical plan for a future fighter could field jet in 5 years - <AUTHOR> https://www.defensenews.com/digital-show-dailies/2019/09/16/the-us-air-forces-radical-plan-for-a-future-fighter-could-field-a-jet-in-5-years/ <comment><USER2> "developing small batches of fighters with multiple companies, much like the Century Series of aircraft built in the 1950s, Roper said."

Do we even have multiple companies anymore? Sure, there's Boeing (Boeing, McDonnell, Douglas, maybe part of Convair?), Northrup/Grumman (Northrup, Grumman), Raytheon (I don't even know how many rolled up in this one), LockMart (Lockheed, Martin, part of Convair). That's 4, out of the multitude of 50s-era aviation companies: Consolidated, Vultee, Martin, McDonnel, Grumman, Northrup, North American, Fairchild, Republic, Vought, others I'm forgetting. There's 4 companies, optimized for extracting large dollars from DoD and CIA contracts, instead of 10-20 companies, optimized for getting airplanes on runways. I kind of doubt this will work. The DoD, and Air Force in particular, have quit getting things done in the last 30 years, in favor of spending money in all the congressional districts in the US.

</comment><comment><USER3> But, but, but ..... wasn't the F-35 supposed to be the be-all, end-all plane that does everything, even washing the dishes?</comment>


How would the rejection of the Daily Stormer play out on IPFS? - <AUTHOR> https://discuss.ipfs.io/t/how-would-the-rejection-of-the-daily-stormer-play-out-on-ipfs/964 <comment><AUTHOR> I already got some good answers, but I'm posting here in hopes of attracting a larger crowd to this question, because I think it is an important one.</comment>


x86/doc: add PTI description - <AUTHOR> https://lkml.org/lkml/2018/1/4/775 <comment><USER2> This nicely summarizes the work that had to be done.

Hats off to the developers that got all this done on a tight deadline. These are fairly substantial changes, akin to implementing the code for a significant new Intel microarchitecture.

But, the amazing contortions necessary are quite disappointing. Intel isn't going to live this down for a while.</comment>


The UI Design Interview: A Primer - <AUTHOR> http://blog.palantir.com/2011/12/01/the-ui-design-interview/

<comment><USER2> I used Palantir software at a previous job. After that, I'm not so sure I'd go to Palantir for UI advice.

They're cool, they're hip -- they've even got the obligatory XKCD comic in the post, but their products are awkward (albeit improving slowly).

Here's some screenshots straight from the horse's mouth:

[http://blog.palantirtech.com/2008/07/04/palantir- screenshots...](http://blog.palantirtech.com/2008/07/04/palantir-screenshots- round-two/)

And yeah, using it is about as clunky as it looks. For better or worst, the software is somewhat open-ended, so there were differences in how analysts visualized entities, which sometimes caused problems when you needed to get actionable intelligence out the door quickly. It's not really suitable for quick.</comment>


Kodachrome was the Instagram of its time - <AUTHOR> https://edition.cnn.com/style/article/mid-century-memories-kodachrome/index.html <comment><USER2> I feel like, if you wanted to pick the film-photography version of Instagram, a better choice would be Polaroid: [https://en.wikipedia.org/wiki/Polaroid_Corporation](https://en.wikipedia.org/wiki/Polaroid_Corporation)

Kodachrome was always preferred by serious photographers (at least until the Japanese film companies started muscling their way into the US in the mid-'80s), but Polaroid was the peoples' choice -- cheap, social and fun. People would crowd around a Polaroid exposure taken at a party, watching the image develop and commenting to each other on it. Nobody ever confused it for high quality film, but for the purposes it was put to, that didn't matter.

<comment><USER3> I agree. Even more directly, Instagram's early icons were pretty clearly inspired by Polaroid, with one of them even a straight up illustration of a specific Polaroid camera.

See the second icon here: [https://images.squarespace- cdn.com/content/54bb4cfce4b045585...](https://images.squarespace- cdn.com/content/54bb4cfce4b045585ada36f7/1464087195265-470BIQAP3J8KAOXQ878J/)

It's a Polaroid OneStep with Q-Light: [https://i.pinimg.com/736x/bb/16/a4/bb16a4b51206b9fdbc73135c6...](https://i.pinimg.com/736x/bb/16/a4/bb16a4b51206b9fdbc73135c6f6a5af0.jpg)

</comment></comment><comment><USER4> I only ever shot one roll of it. In 2006 film was well and truly on its way out. I had just graduated college, and was bicycle touring the United States. In or around Yosemite, I picked up a roll at (I believe) a gas station of all places.

Even in 2006, the last place you could get it developed was Dwayne's Photo in Kansas. Kodachrome is a completely different process than any other film. The crazy thing about the film is that the development process actually removes layers of emulsion depending on what color light the film was exposed to. You can actually see this with the naked eye looking at the emulsion side of the film; it looks a little bit like a relief carving.

I'd like to say the results were magical, but honestly, my inexperience with the film meant they looked pretty normal. The color rendering really is different from other emulsions, and getting the most out of it probably requires some practice.

[https://en.wikipedia.org/wiki/Kodachrome](https://en.wikipedia.org/wiki/Kodachrome)

</comment><comment><USER5> Man I miss Kodachrome. Of course, I hadn't shot it in decades. I miss the idea of it more, I guess.

<comment><USER6> I never shot kodachrome, but did shoot a lot of medium format provia and velvia with a Rolleiflex TLR in 2004-2015.

I made prints from the slides digitally, because direct printing with cibachrome/ilfochrome was virtually unobtainable. Stopped shooting film completely when i got my 1st gen Sony A7R. Somehow the colors out of this camera were as pleasing as a roll of provia, with just a little exposure/contrast adjustment.

<comment><USER7> +1 on Velvia. Ektachrome was also pretty, especially for landscapes and green eyed people.

<comment><USER8> And it still is--- it was reintroduced recently.

<comment><USER7> I think that famous National Geographic photo of the Afghanastani girl was compelling mainly because of the choice of film. (Not sure which one, but most films preferred greens and blues)

<comment><USER9> That was very famously Kodachrome, so much so that the photographer, Steve McCurry was given the last roll ever produced.

</comment></comment></comment></comment></comment></comment><comment><USER10> I think the value that film gives us lies more in the journey than in the final result. With vintage lenses and enough digital processing you approximate 90% of the aesthetic qualities of most old film stocks. Nothing can replace the romanticism of shooting with limited exposures, having to develop and scan and print, and embracing all the imperfections that come with the process.

I've been a "dedicated amateur" street / event photographer and photojournalist for a few years, and the decisions that have helped me improve the most have been 1) shooting with a wide prime (28-35mm) and 2) learning to shoot and develop film.

</comment><comment><USER11> They give us these nice green colors...

<comment><USER12> That song is in constant rotation on Rochester radio. The annoying part is how he decries "all the crap he learned in high school" while lauding a technology that very much depends on people being educated enough to conceive of and produce it.

</comment></comment><comment><USER13> About a decade ago, I bought a slide scanner and scanned a bunch of old family slides. Some of it's the era, some the Kodachrome, but the colors look exactly like those photos. Here's a few: [https://imgur.com/a/PkJiGGx](https://imgur.com/a/PkJiGGx)

As for it being "archival," no, it's not. Some had pretty bad color shifts and started looking blue.

</comment><comment><USER14> Bad article. Confuses technology to take pictures with technology to share pictures with strangers online.

If anything, Kodachrome was the phone camera of its time, but even that is comparing a horse-drawn carriage to a Mack truck.

<comment><USER15> ... technology to share pictures. Online wasn't a thing then. Sharing pictures with strangers wasn't a thing then either. You'd be sharing the pictures with family and friends.

</comment></comment><comment><USER16> web2.0 jaded me would say 'instagram is a pale copy of <past tech that was inspiring and joyful>'

</comment><comment><USER17> Not just of its time: Kodachrome works remarkably well for archival. Or "worked", since it's not made anymore and you can't develop it.

<comment><USER18> ...and it was horrible for projection, fading quickly even as late as K14. You were generally good with slide viewers (Kodachrome can be found in most scenic Viewmaster discs), but light bright enough for projection would kill Kodachromes in no time at all, so you'd generally dupe to an Ektachrome or to a cine print film using an internegative.

<comment><USER17> Not very good for "traditional", dry scanning either. IR-based scratch/dust removal stuff in Nikon scanners didn't work on it. But at least if you don't project too often, the colors would last for a very long time.

</comment></comment></comment><comment><USER19> not new article, from ~3 months ago, mostly glorified ad for a photography book.

A few other kodachrome articles and discussion back from maybe when it was more newsworthy - when it was being retired [https://news.ycombinator.com/item?id=669585](https://news.ycombinator.com/item?id=669585) or an unseen archive of 1960s East London pics [https://news.ycombinator.com/item?id=16363283](https://news.ycombinator.com/item?id=16363283)

</comment><comment><USER7> Any surprise that its original icon seemed to be a hybrid of the Instamatic Brownie and the Polaroid logo?

</comment><comment><USER20> Can we effectively synthesize it with digital methods?

<comment><USER21> Yes, however one needs to take into account the difference in viewing devices (screens are self illuminated and film/paper subtract colours from a light source) as well as the limited colour depth on modern monitors. After all, 10-bit colour wasn't available on mainstream GPUs until the last couple of years.</comment></comment>


RTB: Where Erlang blooms - <AUTHOR> http://ferd.ca/rtb-where-erlang-blooms.html

<comment><USER2> Nifty, w' the point about queues (sometimes you don't need 'em, and can "safely" discard requests) being both blindingly obvious and most often neglected...</comment>


A Ray-Box Intersection Algorithm and Efficient Dynamic Voxel Rendering - <AUTHOR> http://www.jcgt.org/published/0007/03/04/ <comment><USER2> From the paper: "We use the rasterizer as a potentially-visible set optimization to iterate only over pixels for which rays might intersect a voxel, and then execute a small ray tracer in the pixel shader. That is, we "splat" billboards that give coarse visibility and compute exact visibility in a pixel shader. This works for any pinhole perspective projection, including eye rays and shadow rays, so we use it for the shadow map rendering pass as well"

Neat - I used a similar technique [1] in the last iteration of Voxel Quest, but I only rasterized single points/pixels then raytraced them into filled cubes (was not efficient other than on fill rate). However I did not invent the technique (I'm not sure who did, but I first got the idea from talking to Florian Boesch [2])

[1] [https://twitter.com/gavanw/status/717265068086308865](https://twitter.com/gavanw/status/717265068086308865) [2] [http://codeflow.org](http://codeflow.org)

<comment><USER3> > We use the rasterizer as a potentially-visible set optimization to iterate > only over pixels for which rays might intersect a voxel, and then execute a > small ray tracer in the pixel shader.

I was _just_ thinking about doing this last weekend (in the context of drawing exact reflections off reflective surfaces defined by arbitrary meshes). Nice to see it actually works, at least for voxels!

The fact that raytracing (onto a 2D plane) and rasterization are essentially isomorphic to one another is a powerful observation.

<comment><USER4> > The fact that raytracing (onto a 2D plane) and rasterization are essentially > isomorphic to one another is a powerful observation.

Hasn't this been a pretty well-known perspective (no pun intended) for a couple of decades now?

<comment><USER3> Of course, but it's only relatively recently that we've had the graphics APIs to make full use of that observation.

<comment><USER5> _< notices username>_

Will this somehow turn into faster CSS and/or SVG render times for Firefox? :)

</comment></comment></comment></comment></comment><comment><USER6> This is more useful than it might seem because many rendering techniques are sometimes implemented using voxel representations behind the scenes, like global illumination (indirect bounced light, so light bounces off a red wall to create a pale red glow on the ground, that sort of thing), shadowing, fluids, etc. The examples of Minecraft and LEGO given are direct applications but I suspect this will improve a lot of stuff.

<comment><USER7> This is exactly what we do at Roblox.

<comment><USER8> Just wanted to say that Roblox is a masterful system. My daughter plays it and its an amazing piece of software. Your map editor is great too. My daughter is 8 and she picked it right up.

Curious, how big is the team for the game and editor?

</comment></comment></comment><comment><USER9> Anyone hiring engineers to implement stuff like this? That would be my dream job.

<comment><USER10> We do heavy duty graphics/CUDA, feel free to ping firstname.lastname@example.com. We are currently looking.

</comment></comment><comment><USER11> Be sure to check out the video in the "sample code" supplement.

<comment><USER5> Thanks for the heads-up! Oddly enough the dynamic scene near the end reminds me of a recent video by Jon Burton where he explains how his game company managed to get an incredibly high number of particle effects on the Playstation 2. It looks like in both cases, the idea is to make the implementation functional and fully computable on a VPU/GPU with as little overhead of moving data between RAM, CPU and GPU as possible.

[0] [https://www.youtube.com/watch?v=JK1aV_mzH3A](https://www.youtube.com/watch?v=JK1aV_mzH3A)</comment></comment>


"Malamanteau", a self-powered notability? - <AUTHOR> http://en.wikipedia.org/wiki/Talk:Malamanteau

<comment><USER2> Wikipedia is a serious website, for the most part.

But at times I think some people take it a bit too serious and that's a net loss.

If nobody gets hurt or slandered and the content is legal you might simply tag it as 'non-noteworthy' but leave it anyway, as long as it is true and might be of use to someone some day.

Digest users of WP could simply strip such articles automatically.

<comment><USER3> I was disappointed to learn that the history of deleted wikipedia pages is hidden from public view.

</comment></comment><comment><AUTHOR> I find it increasingly funny how the fighting between wikipedia editors is actually increasing the notability of this (not so) fictitious neology created by Randall Munroe from XKCD.

</comment><comment><USER4> Just wait until the blogs pick it up, then "serious" newspapers, magazines, etc... Then it'll get it's own Wikipedia page. See: Santorum <http://en.wikipedia.org/wiki/Santorum_(sexual_neologism)>

</comment><comment><USER5> Trolling is a art.</comment>


Data structure and algorithms problems in C++ - <AUTHOR> https://github.com/mandliya/algorithms_and_data_structures <comment> brian-armstrong The bit twiddling in this looks fairly naive/slow.

There's some really great, fantastically fast methods in [https://graphics.stanford.edu/~seander/bithacks.html](https://graphics.stanford.edu/~seander/bithacks.html) that require some interesting insights in finite fields. These are usually the sorts of techniques you'd want to turn to for bit twiddles

<comment><USER2> And the fine book, which may cover a lot of the same ground. [http://www.hackersdelight.org/](http://www.hackersdelight.org/)

</comment></comment><comment><USER3> Sick job, that's nearly 4 years of hard work there.

</comment><comment><USER4> Some of the solutions are not safe (and should be indicated as such). For example MIN/MAX in generic.h are macros so shouldn't be used with functions (and mustn't be used with functions that have side-effects). Macro parameters aren't bracketed either so operator precedence can bite there too.

There's also potential for undefined or implementation-defined behavior (e.g. multiply_by_7() is left shifting signed int). Author's heart is in the right place it's just that his skills aren't necessarily up to the task, I'm afraid.

<comment><AUTHOR> edit: Spellings.

Thank you for the feedback. Some of the programs here are coded 4 years ago, when I had just started learning C++ and I never revisited them. At that time, I was in university, and this repo was to attract recruiters for internship. I got the job later, and kind of ignored to correct problems. I am sure if these are ran with strict compiler flags there would be many more issues. I will get around and fix it sometime soon.

<comment><USER4> Sure thing, reading through the repo I understood it was used to learn stuff. But it may not be immediately understood by a novice. And thing with the Internet is: people will start using code from your repo and get into all sorts of issues. It's unlikely you'll smooth this code up. Nobody's got the time, let's be honest here. ;) So I'd recommend that you at least state explicitly that these functions shouldn't be lifted directly unless someone knows what he/she is doing.

<comment><AUTHOR> Thanks again for the response, I understand it might be better to put a warning note at the top of readme. Thank you again for the advice.

</comment></comment></comment></comment><comment><USER5> Are these from somewhere? I might try to recreate this in Rust as a way to learn more of that language.

<comment><USER6> Just to caution you, data structures aren't a great way to learn Rust. See this recent discussion, for example: [https://news.ycombinator.com/item?id=16072630](https://news.ycombinator.com/item?id=16072630)

Algorithms are fine, though.</comment></comment>


$3 Million Health Care Analytics Challenge - <AUTHOR> http://www.readwriteweb.com/hack/2010/12/3-million-prize-for-health-car.php <comment><USER2> If you read the actual contest summary page (<https://www.theheritagehealthprize.com/Page/MediaCenter>) it states "A total of three million dollars will be awarded to Teams who achieve the benchmark targets set out for this competition."

This is quite different from a single $3MM award to one team. Also why it's better to post the original source than a blog piece that's just an (inaccurate) rehash.

</comment><comment><USER3> Curiously, nothing more specific about data availability than "registration will open in 2011, after the launch of the Prize."

Also, the competition is expected to run for approximately two years. They will require teams " . . . to be bound by final competition rules, sign a Master Team Agreement and pay a modest registration fee."

I work in clinical informatics and am eager to see a sample of their datasets though.

</comment><comment><USER4> Requiring insurance is a significant negative. I have difficulty understanding what liability a competitor in an algorithm design contest could have. Shows how much the medical profession is handtied by excessive liability.

<comment><USER5> I suspect that this is to cover privacy issues. They don't want a repeat of what happened with the Netflix Prize. The Netflix Prize 2 was canceled because of the privacy issues involved with the data set ([http://blog.netflix.com/2010/03/this-is-neil-hunt-chief- prod...](http://blog.netflix.com/2010/03/this-is-neil-hunt-chief-product- officer.html)).

The FAQ states that participants will be randomly audited to ensure that they are satisfying data privacy standards.

</comment></comment><comment><USER6> Someone should mention the $3m prizepool to Microsoft.</comment>


The success of drug decriminalization in Portugal - <AUTHOR> http://www.salon.com/opinion/greenwald/2009/03/14/portugal/index.html <comment><USER2> Anyone else believed drug consumption, possession, trading and the like were legal in portugal after reading this article?

Well it is not. Only consumption and possession of small amounts is. And that is the case in a lot of European countries.

Badly written article for leaving out important facts. Author obviously biased.

<http://www.encod.org/info/PORTUGAL.html>

<comment><USER3> That would be legalization, he uses the word decriminalization.

<comment><USER2> I used "Anyone else believed", not "The article states". Yes, the error is on my side here, but the author was not very clear.

</comment></comment></comment><comment><USER4> Upvoted for the possibilities.

It's a (almost confusingly) controversial topic, but there is a lot of good that could come out decriminalization. I've never done any drugs myself, but I can think of a number of business opportunities not only in legal narcotics trade, but also supporting industries. If the objective, empirical evidence suggests that it does not present society with an immediate nor a long-term health threat, then the economic and cultural benefits would be exciting.

That said, it will be a considerably long time before metaphors like "The war on drugs" are dropped in the US. People are passionate about it here.

<comment><USER2> > but I can think of a number of business opportunities not only in legal > narcotics trade, but also supporting industries

Like making children addicted and profiting afterwards? Legalizing drugs like heroin, crack or coke is just sick. Just have a look at what meth has done.

> I've never done any drugs myself,

Talk to one who has, then. It's going to alter your thoughts.

<comment><USER5> The assumption most people seem to base their conclusions on is that making drugs illegal reduces consumption. All empirical evidence so far - even from less biased sources than the wildly free-market Cato institute - shows that this is not the case. In countries where more drugs are legal, consumption actually goes down.

Drug use is a facet of human existence that needs to be managed, not a virus to be exterminated...

<comment><USER2> > The assumption most people seem to base their conclusions on is that making > drugs illegal reduces consumption. All empirical evidence so far - even from > less biased sources than the wildly free-market Cato institute - shows that > this is not the case. In countries where more drugs are legal, consumption > actually goes down.

We have empirical evidence on cannabis, true. In that case (Netherlands) statistics say that "everything is roughly better", but differences are so small that it's difficult to say it's due to the drug laws.

We do not have empirical evidence on drugs that make you physically and/or psychologically addicted after one shot.

And there should be no toying with this. Getting of cannabis after years of using it is a piece of cake compared to getting of heroin after less than five shots. It's a totally different story.

<comment><USER6> No it has actually, rather clearly, been shown that in countries like the Netherlands where drug tolerance is much higher, it has become much more acceptable for people using opiates to enter rehab.

The process of getting off heroin is extremely hard, in fact going cold turkey can potentially kill most addicts. So without acceptance the only way off heroin is strength of will (and a lot of luck) or death. With expansive and non-judgmental rehabilitation the process becomes easier, and you'd be surprised that the sheer disgust most heroin users have at themselves when they've got dry is enough to keep them away from the drug for life.

Most heroin addicts, between the time the high dies and before the low begins, hate themselves for doing the drug. They aren't corrupt evil people, they're people who were stupid and didn't necessarily have to be weak willed to get addicted to it. Many people get much more help for being much more stupid and much weaker, so why don't we help the people who really need the help?

<comment><USER7> Kicking heroin addiction is nowhere near as hard or as dangerous as kicking alcohol addiction.

<comment><USER6> If it's true alcohol addiction, but with alcohol psychological addiction usually presents long before physical addiction has completely taken over. Essentially heroin might kill you if you're a long time addict, but alcohol is pretty much guaranteed to kill you if you're a long time addict.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> Americans and their obsession with drugs is so strange. We're 5% of the population, yet we consume over 50% of all illegal drugs. Something tells me that this cultural inclination for drug usage makes it much more difficult to justify legalization. We're a country that can barely handle a free-market debt system -- I don't even want to know what we would do with a liberalized drug market.

Then again, I am one of those people who doesn't do drugs, drink, smoke, or eat processed foods, because I understand that it's extremely expensive (in that you lose long-term quality of life / life expectancy - the single most important non-renewable resource for a living creature); I am obviously a minority...

<comment><USER9> >"We're a country that can barely handle a free-market debt system -- I don't even want to know what we would do with a liberalized drug market."

There are a ton of packaged assumptions and a nice non-sequitir in just this one little sentence. That's sloppy rhetoric. It would take a page to unpack and discuss them all with clear thinking.

>"Then again, I am one of those people who doesn't do drugs, drink, smoke, or eat processed foods"

Probably most of the people that advocate drug legalization aren't regular users, and many of them have never used drugs. Rather, they support legalization because of the effects of prohibition on society. I wouldn't impute selfish motives to people that disagree with you without understanding them first.</comment></comment>


Ask HN: How to display experience on portfolio when work was in Porn? - <AUTHOR> Hi,
I'm trying to apply for jobs and my experience for the last 5 years has been primarily working/developing features/maintaining a pornographic website. In addition to other random freelance gigs.
How do I put this on my resume?
It seems pretty vulgar to just list the site and time there. <comment><USER2> IIRC, porn companies usually have an LLC behind them that has a more normal name. If possible, use that. When it comes to descriptions, focus on the technical nature of the job, rather than the content. For example, I used to work for a real estate company, but when I discuss my time there with potential employers, we never talk about real estate. Instead, we talk about databases, servers, and code. You can probably do the same. If asked for details, you can fudge a bit and just say video streaming services or something along those lines.

If they really dig deep, you can just take a risk and admit that it was pornographic in nature. I personally wouldn't care, and I don't know many people that would, but it's definitely a risk. If they press you for details about the business/content side of things, you may just have to come clean.

But if at all possible, I'd focus as much on the technical accomplishments as you can. Good luck... :)</comment>


Canonical writing their own X/Wayland replacement - <AUTHOR> https://lists.ubuntu.com/archives/ubuntu-devel/2013-March/036776.html

<comment><USER2> Ha, I bet this one will crash and burn over Fiji too.

More information here: <https://wiki.ubuntu.com/MirSpec>

They don't like X for all the normal reasons, they think Wayland might be like X but haven't investigated (I think you can replace the input handling in the "compositor" plugin; maybe I'm wrong).

They don't specify how they'll handle hard problems, such as:

\- Client vblank sync (as well as compositor vblank sync). Compiz, unity, etc, don't do this now; clients run unsynced and tear.

\- Switching unoccluded windows to render directly to the framebuffer instead of going through the compositor. Mandatory for good performance on something like a Nexus 10 (which has a 16MB framebuffer, so nearly 1GB/s bandwidth to deliver glClear at 60fps with no extra copies...).

\- Handling multiple clients that want to read gestures. They actually say in their "INPUT" section that they will handle "shell" gestures "in the server", which sounds like they're going to bake their UI into their display server.

They want it to be compatible with all of the old apps too, which Wayland already does. I'm trying not to be judgemental, but it sounds like they want to write something like Wayland but without designing it first. Good luck with re-entry, guys.

<comment><USER3> Bigger burn than reentry.

You're right though, this feels a bit like not developed here syndrome. X is horrible to hack on top, and has had a good run but Wayland could be a fantastic base for them, especially given Qt has spent time working on it.

<comment><USER4> Is it possible that once toolkits like qt and gtk have been ported to simpler/more direct APIs like wayland (w.r.t X11), porting them to other APIs like this mir would be easier than starting from scratch?

<comment><USER3> Possibly, but I've followed some of the Qt Wayland development and it looks like they've hit problems on a fairly regular basis. When you're building GUI toolkits you're also including a certain amount of understanding of the weird corner cases from the windowing system, so you're ending up porting a large chunk of code and then working out what's broken.

Although, Qt might have it a bit easier as they've switched to drawing their GUI components with OpenGL.

</comment></comment></comment></comment><comment><USER5> Open source is wonderful. Good luck to Ubuntu, though I love X. As an end user of X, fluxbox, and X-forwarding over ssh, X just works. I'm not looking forward to compatibility troubles when the new fleet of X replacements arrives.

Debian, please continue to maintain old stalwarts as Ubuntu and other distros flirt with the new.

<comment><USER6> What happens to the X windows forwarded over ssh when the connection drops? Can you reattach afterwards?

<comment><USER7> No, I think the app gets killed.

<comment><USER8> Depends on the app. Emacs can survive and re-attach later. A single emacs session can be connected to multiple X (and tty) sessions at a time. I use this all day, my emacs session runs for weeks or months between restarts and I just connect/disconnect to X servers or ttys as needed.

</comment></comment></comment></comment><comment><USER9> You may want to go to <https://wiki.ubuntu.com/MirSpec> if you want to read about their motivations. I don't enough about Wayland to say if their argumentation is justified or not.

</comment><comment><USER10> Somehow, I'm less than surprised. I'm waiting for them to decide to roll their own browser, email client, and office suite...

<comment><USER11> Don't forget about their own programming language. :)

<comment><USER12> Rubunthy on Mails?

[edit: Oops, I've jumped all the way to their own web development framework. At least this way they can also build their own DSL's.]

</comment></comment></comment><comment><USER13> They ought to fix all the pretty basic bugs and commit to actually supporting their customers before they bite this off.

LaunchPad is a testament to the shit state Ubuntu is in. All the windows are broken so we'll knock half the building down and rebuild it. Doesn't work.

</comment><comment><USER14> ...really?

X is an abomination. But why not Wayland? Anyone care to present Canonical's argument, with concrete examples of what they find so offensive about it?

<comment><USER15> [https://wiki.ubuntu.com/MirSpec#Why_Not_Wayland_.2BAC8_Westo...](https://wiki.ubuntu.com/MirSpec#Why_Not_Wayland_.2BAC8_Weston.3F)

<comment><USER7> If Wayland's input handling is not good enough, why didn't they raise that on the mailing list any time in the last few years?

What does "The shell integration parts of the protocol are considered privileged from our perspective" even mean?

</comment></comment></comment><comment><USER16> I think it's just as interesting (if not more so) that they are moving away from Gnome to Qt!

<comment> dylan-m That bit is nice to see, but I'm pretty suspicious of it. They're going to have to write _a lot_ of software to fill the gaps, now, if they mean to have Qt everywhere. GNOME provides a lot of functionality beyond the UI toolkit and the shell, including myriads of small applications we take for granted. I'm curious where Canonical is going to find the engineering resources to handle all of this.

<comment><USER17> > including myriads of small applications we take for granted

As someone who runs Arch with KDE and an intentionally different Gnome theme just so I can notice the apps I use running Gnome, there aren't _too_ many. In Ubuntu-space, definitely, since their entire ecosystem was built on gnome, but in general you can easily get a gnome-less Linux.

<comment><USER18> Gnome-less debian user right here!

</comment></comment></comment></comment><comment><USER19> Another item of note in this post is that they are moving Unity to Qt.

</comment><comment><USER20> Did they give any rationale why they can't use Wayland?

<comment><USER21> Yes. Canonical clearly lays out the project motivations, including not choosing to use Wayland, on the Mir wiki page[1].

[1] [https://wiki.ubuntu.com/MirSpec#Why_Not_Wayland_.2BAC8_Westo...](https://wiki.ubuntu.com/MirSpec#Why_Not_Wayland_.2BAC8_Weston.3F)

<comment><USER20> What worries me is fragmentation of drivers. It can be a very serious problem.

<comment><USER22> They'll probably use the same KMS and EGL drivers that Wayland and others use.

<comment><USER23> Yes, this is covered in the wiki page linked to in the email:

      https://wiki.ubuntu.com/MirSpec

</comment></comment></comment></comment></comment><comment><USER24> I'm surprised this hasn't been mentioned yet: Ubuntu was able to cobble together Ubuntu Touch running on Surface Flinger as a compositor.

There are limitations to that, including lack of really working support for multiple monitors.

Mir may be designed to fill in the deficiencies of Surface Flinger for use as a compositor for Ubuntu. That might make project scope controllable enough to think it could succeed.</comment>


Hey HackerNews: I'm looking for technical people to Beta test SuperTCP - <AUTHOR> https://docs.google.com/a/liveqos.com/forms/d/1MzUEEbOEnSjvBAamYVA0bNgl8vTH5LhTmXIO-mFJwGo/viewform

<comment><USER2> Any guess on the how? Part of me is just betting they flip on the Priority Flag (if that hack still even works today).

<comment><AUTHOR> Most detailed description is here: [http://www.supertcp.com/the-fundamentals- of-a-killer-reliabl...](http://www.supertcp.com/the-fundamentals-of-a-killer- reliable-transport-protocol-for-the-internet/)</comment></comment>


Validate your startup idea - <AUTHOR> Came across excube360, a tool to validate your startup idea before you start, to save time and money. It seems to look through patents, platforms like crunchbase and angellist, and social media to produce quantifiable results in terms of market activity, investor interest, growth and crowd.<p>http://www.excube360.com <comment><AUTHOR> Link: [http://www.excube360.com](http://www.excube360.com)</comment>


Rise of the Scientific Programmer - <AUTHOR> http://byterot.blogspot.com/2015/01/future-of-programming-rise-of-the-scientific-developer-bigdata-datascience-machine-learning-and-fall-of-the-craftsman.html <comment><USER3> As other people have mentioned - the problem is that academia rewards producing papers, not stable software libraries.

For example - scikit-learn is an amazing project, lead mostly by a small group at INRIA with Gael at the helm - and in terms of academic prestige, scikit- learn is probably 'worth less' on your CV than a couple of Nature papers.

This is of course ridiculous - scikit learn is used by a huge amount of people, it takes an insane amount of work to run the project, yet the incentives are what they are.

<comment><USER4> > the problem is that academia rewards producing papers, not stable software > libraries.

isn't this the good thing about academia? aren't you kids getting paid the big bucks to write and maintain the software libraries, while we work on novel problems for pennies?

<comment><USER3> I am an academic myself! Aside from that - it's actually a bad thing, poor software quality is incredibly harmful when trying to create reproducible research.

A few people are fighting this (Titus Brown, etc) but it's mostly swimming against the tide of bad incentives.

<comment><USER5> i have started using Docker for this kind of stuff. you can build an isolated environment for your software and experiments, where you can absolutely guarantee that anyone who wants to can easily replicate your experiments, since they don't need to create the environment themselves - just pull the docker image for conference-paper# and run the scripts.

if the experimental data is proprietary, or you want to keep it separate, you can set a mount point for it in the lxc.

<comment><USER6> " _anyone who wants to can easily replicate your experiments_ "

Replicate the experiments, or just repeat the results?

<comment><USER5> the stuff i work on is in the area of machine learning, so most published work involves one or more well-known data sets.

i would argue that the two are the same in this case.

the lxcs provide all the source code i write [plus of course the compiled version], all third-party libraries, and all scripts used to run and evaluate the experiments, and the data as well, where that is permitted.

it's still not perfect, but for my area, i honestly think it is the best, and most accountable way to do things that i have seen.

<comment><USER6> And hopefully one or more not-so-well-known, local data sets to check that the results are actually as claimed?

<comment><USER5> well, the idea is that you should be able to run _any_ data set you have, and get good results relative to other solutions. but that is an open question with any research.

the point of the docker/lxc aspect is to provide a simple working environment to facilitate replication and validation.

so in comparison to the status quo, which is basically 'write a paper, include some high level equations, and results', i think this is a step forward in a better direction.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> The other night I was searching for Python science books and stumbled across this one titled "Python for Biologists: A complete programming course for beginners"...the homepage for the book is here: [http://pythonforbiologists.com/index.php/introduction-to- pyt...](http://pythonforbiologists.com/index.php/introduction-to-python-for- biologists/)

Admittedly, it only has two reviews on Amazon, but they're both five stars, and they both seem to come from biologists who are apparently thrilled at being able to leverage code for their work...the funny thing is, the book itself is not "advanced" as far as what most professional programmers would consider "advanced"...the Beginners' book ends with "Files, programs, and user input" and the Advanced book ends with Comprehensions and Exceptions...

I think we as programmers vastly understimate how useful even basic programming would be to virtually anyone today. I work at Stanford and it continually astounds me when I run into non-programmers who are otherwise doing data-intensive research, who fail to see how their incredibly repetitive task could be digitized and implemented as a for-loop. It's not that they are _dumb_ , it's that they've never been exposed to the opportunity. And conversely, it's not because I'm smart, but I literally can't remember what it was like _not_ to break things down into computable patterns. And I've been the better for it (especially because I'm generally able to recognize when things _aren 't_ easy patterns)

Sometime ago, I believe it was Stephen Hawking who speculated that the realm of human knowledge was becoming so vast that genetic engineering of intelligence might be required to continue our progression...that may be so, but I wonder if we could achieve the same growth in capacity of intellect by teaching more computational thinking (and implementation), as we do with general literacy and math. As Garry Kasparov said, "We might not be able to change our hardware, but we can definitely upgrade our software."

[http://www.nybooks.com/articles/archives/2010/feb/11/the- che...](http://www.nybooks.com/articles/archives/2010/feb/11/the-chess-master- and-the-computer/)

<comment><USER8> From my experience, I've come across many individuals in academia who I've tried to suggest particular approaches to their choice of problem and be rejected because they are too proud or too afraid to learn something new. I'm not trying to be an ass when I do, because that would undermine my attempt to share information, but something as simple as suggesting "for i in list:" instead of "for i in range(len(list))):" is so offensive to them since they didn't learn it first when they learned how to program (and god-forbid they learn it from a second year graduate student).

May be biologists or people at Stanford (or biologists at Stanford) are less proud, but my experience here has made me stop trying to relate basic programming concepts to fellow academics.

<comment><USER9> I had the same experience in grad school, and I can tell you that giving people (good!) style advice when they don't ask for it is pretty pointless. It's kind of like giving grammar advice that wasn't asked for.

People need to be in the right frame of mind to learn new things. Otherwise, if they're getting their point across, just let them keep talking (or coding).

<comment><USER8> You're probably right about it seeming rude, comparing it to grammar advice makes is a good analogy--I didn't think about it like that.

<comment><USER10> Programmers are expected to give and receive style, performance, and idiomatic advice in every code review. Is there some way this sort of peer code review could be integrated into the academic process?

<comment><USER9> Principle Investigators (i.e. professors that run labs) could establish a code review process for code that their group produces, just like any other manager can establish such a process.

Many PIs don't have the expertise to do that well, and many of them don't especially value style, performance, or idiomatic code. You have to remember that most academic code gets used by 1-5 people, and is run something like 1-50 times total. In those cases, it's actually kind of ok that that code doesn't end up being maintainable.

The important risk, of course, is erroneous results. But good researchers generally find many independent ways to check their results, so ideally, bugs that affect results should get caught in that process.

I personally love writing high quality code, and I found the academic science attitude about code to be frustrating. But there are structural reasons for these attitudes, and it's wrong to imagine that you can change them by arguing in terms of things that software engineers find valuable. You'd need to make your case in terms of things that science PIs find valuable, and in the process of trying to do that, you might actually discover that what you wanted to argue for isn't so critical after all.

<comment><USER11> > You'd need to make your case in terms of things that science PIs find > valuable, and in the process of trying to do that, you might actually > discover that what you wanted to argue for isn't so critical after all.

This is a great sentence - thanks for contributing.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> I have noticed a general trend upwards in the interest of scientific programming for a few months now, and the community (most specifically Hacker News) has driven my interest in that area as well. The idea of functional programming and thinking in mathematically sound ways really appeals to me, but my lack of math and comp sci background is holding me back from going full-speed learning and getting better at it.

I feel many of us are lost swimming in a sea of opinions and juggling frameworks du jour, development methods, and business strategies, that it keeps us from focusing on improving our skills in areas that matter. This frustrates me and I've been looking for ways to get out of it. There is also this fear of another bubble mixed with trying to keep up with the trends and hipness of the industry, to remain gainfully employed.

I realize I am sort of just reiterating the authors point, so I guess what I'm saying is I agree.

<comment><USER13> HN tends to gives this perception of rapid change because you're always hearing about hot new tech and how great it is from the various tech evangelists. It's a side effect of their faith/vision in their own tech that you feel like you're missing the train.

However, I've been following HN for 7.5+ years now, and change is hardly as fast as you think. There really is nothing new under the sun, and you see the same core principles (and human needs) being expressed as the underlying technology changes. The core principles of functional programming have been around since the 30s as lambda calculus, and since the 50s with their initial expression as LISP. It's better to learn FP to add a new way of thinking about problems to your toolbox, rather than treating it as a panacea of programming.

If you're feeling lost in the sea, define your lighthouse. HN will most certainly have you paralyzed by the many waves of every new choice if you haven't defined a clear vision for how you want to harness your finite energy.

</comment></comment><comment><USER14> Great perspective on the future, we would need many more similar discussions. I love the main message of the post, i.e. "we need to grow up". I have the impression that while we are all optimistic that the future is owned by software developers we don't realise that not for all. There will certainly be more segmentation in our profession and there will be great demand for high- end developers. This requires a lot of learning, and I personally feel it's a tough challenge.

The post also made me realise how much we still think in terms of disciplines. E.g. we think a developer should learn more mathematics. If we were thinking in terms of problem solving, or "modelling reality" (at least in part with software) we couldn't separate these so easily. E.g. if you are writing a software for vehicle condition monitoring you use a combination of engineering, physics, mathematics, computer science - the less you try to - <USER2> need to - separate them the better you do.

I can't quite put it simple, but in my mind I can see the future "developer", how got a BSc in Physics, went on to work as a software developer for a couple of years and then continued to learn every day maths, physics, biochemistry, worked in various projects where she could use all these. She is neither a physicist, nor a software developer or mathematician.

<comment><USER15> > you use a combination of engineering, physics, mathematics, computer science > - the less you try to - or need to - separate them the better you do.

I disagree that mentally separating math, physics, and computer science is a good thing. I think a good scientific programmer should understand the science on its own, and then figure out how to model/approximate the science to do something useful on a computer. For instance, if you want to implement Reed- Solomon codes efficiently, you'll realize that understanding the algebra required to understand the code is more or less an orthogonal skill to designing an efficient encoders and decoders.

As a personal anecdote, I had much better luck learning about waves and quantum mechanics after I knew about differential equations, orthogonal decompositions, and a fair amount of linear algebra than I did before I knew this math, even though the physics classes included all of the necessary math. I attribute this better understanding to have a cleaner mental map, because I knew which statements were true because of some sort of physical fact, and which statements were just mathematical results.

In line with the generally smiled upon principle of decomposing problems, I think its particularly critical that the scientific programmer can decompose these interdisciplinary problems. A scientific programmer should understand the science (at least mostly), understand the hardware, and figure out how to efficiently and cleanly map the scientific problem to something that to be solved by a program.

</comment></comment><comment><USER16> I started on the science/engineering side at BHRA (on campus at CIT) only problem with technical/scientific vs commercial is the pay is so poor.

<comment><USER17> And academically the prestige is poor. One is not granted 'research time' to develop software, but to 'get things done' (see u/Danso's comment). As such there's no one to take the first step in actually making software that would benefit anyone. And in the generous circumstance that one can be allotted time to write the software, the result is a pat on the head - 'good job' \- for reducing everyone's workflow from weeks to minutes. Sometimes you get an acknowledgement. And no one will ever support/read your software when you leave - it will be used ritualistically until the lab's last computer's OS no longer supports it.

On the topic I see two other significant problems:

1) In basic research there is often a need for 'Every Option' style software - you're doing something that's never been done before and you need to be able to tweak it exactly how you need (but also be able to 'just hit run' for a first pass when coming from your native field). And those types of software are inherently a mess to design and build (ie. photoshop, CAD, 3D, programming languages).

2) Some of this software can only be written by those who directly do the research - or someone who very closely collaborates with them. Scientific software contains _scientific_ assumptions in it that are very hard to evaluate if you're not part of the field. Deciding to go right-way-round, or rounding up, or leaving off the last element in an array, or any other of those programming tricks can really mess up scientific work. Or conversely, using the entire array, using a non-weighted, 'avg' or treating the red channel mathematically the same as the blue channel is a very different way of designing software than other industries - and is not common and rarely given much thought.

<comment><USER18> There's a modest living to be made at the interface between "scientific programming" and "commercial implementation of programs scientists write". This is an under-appreciated niche because it takes a lot of work to get into it: you need to be a good developer and a good scientist with diverse experience. My way in was through experimental and computational physics, but there are certainly other avenues these days.

Modern statistics is the biggest piece of the picture that every interesting area has in common. If you're interested in scientific programming you need to understand Bayes as well as algorithms etc. I have friends in psychology, biology, etc and we can communicate surprisingly well because we all speak the same statistical language.

But more importantly you need to understand how scientists think. They are amazingly hard to pin down to the kind of specs developers need.

For example, a guy on my team once said after talking to one of the scientists we were working with for a couple of days, "I now have a much better grasp of the problem, but I still don't know what the default value of this parameter should be." I spent ten minutes talking to the scientist and came back and told the developer "5", because I could tell from the way the scientist was talking that he had no clue if the number should be 3 or 10, but seemed to be favouring the lower values. I didn't need to understand the problem domain in detail to make that judgement, but to have a reasonable grasp of the psychology of working scientists. So far as I know, there's no way to get that without working as a scientist yourself.

</comment></comment></comment><comment><USER19> Shameless plug: In January I'll be focusing on a project which deals specifically with scientific software: [http://sciencetoolbox.org/](http://sciencetoolbox.org/) This current version is a product of a hackathon, but this month will be improving it and adding functionality which brings the scientific software developer and her efforts into focus. Scientific software is gaining importance, but the recognition its developers get is trailing behind - I want to raise the level of associated recognition/prestige (among other related things). Some other projects rely on data collected here, e.g. a recommendation engine for said software that enhances GitHub: [http://juretriglav.si/discovery-of-scientific- software/](http://juretriglav.si/discovery-of-scientific-software/)

Shameless plug continues: if you'd like to keep track of what I'm doing I suggest you either follow the project on GitHub ([https://github.com/ScienceToolbox/sciencetoolbox](https://github.com/ScienceToolbox/sciencetoolbox)) or Twitter ([https://twitter.com/sciencetoolbox](https://twitter.com/sciencetoolbox)).

<comment><USER17> Units. I'd love so much for a standard fileformat/interpreter/concept that contained SI units as a requirement for most datatypes. Much to be learned from my TI-89.

<comment><USER20> I don't know what you mean by "as a requirement for most datatypes". I work with molecules, where distances are typically measured in Angstroms and masses in amu. I don't want to have factors of 1E−10 and 1.660538922E−27 hanging around my code.

<comment><USER21> I assume that he means that the type system knows that these are in useful units, and tracks them, so you can have it check that your calculation actually results in a value of the unit you expected.

<comment><USER22> Yeah. There's really no reason why our hot new scientific computing languages and libraries should all be lacking capabilities that graphing calculators had in the 1980s when they only had 2k of RAM. HP demonstrated that it doesn't even require a CAS to be extremely useful.

<comment><USER20> I know what unit libraries are, and why they can be useful. There are several units libraries for Python, the Boost C++ library includes one, etc.

I don't know why they should be a 'requirement for most datatypes'. Could someone please explain the requirement part?

As a further clarification, why should people in scientific computing, in fields which use non-SI units like eV, amu, Angstrom, barn, light year, and megaparsec, use a programming language which requires SI units? Quasar 3C 273 is 749 megaparsecs from us, or 2.31E25 meters away. I don't see why SI should be preferred.

<comment><USER22> Any programming language that makes units a first class part of its type system would allow for defining custom units just as easily as defining other data types. Nobody's saying that SI has to be the only units expressible, just that it has to be the foundation of the unit system. Likewise, unitless numerical quantities will necessarily still be expressible, but using those types for variables that represent a value with a unit should be considered extremely poor practice, just like when communicating those values on paper.

There's really not a good reason to argue against using only SI units for data interchange formats. It's trivial to map to the preferred units on import or display, if and only if you know what units the input data is in. I've dealt with too many bugs where interacting programs have differing assumptions about meters, centimeters, and millimeters to believe that the flexibility of storing different units on disk is ever worth the trouble.

<comment><USER20> I use Python. If I use one of the third-party packages for units then I can do what you say I can, but at extreme cost. Every single operation checks for unit conversion, on the off-chance that the values aren't compatible. The system, in trying to be nice to me, ends up making things invisibly slow.

(In practice, the performance code runs in C, so the Python/C boundary would have to negotiate the array types for full unit safety.)

In my work the base unit of length is angstroms. I've used nanometers a few times, and never used any other length unit, though I know that GROMAC's xtc format uses picometers. Saying something has a volume of 600 cubic angstroms is much more useful than 6E-28 cubic meters. While I can appreciate that other fields closer to human scale use may like to standardize through SI, I don't want your preferences enforced on my field. All I see is the chance to make things worse, and slower, and don't see any advantages.

One of my data formats has coordinates in angstroms, like "8.420 50.899 85.486". How would you suggest that I write that in an exchange format? As "8.420E-10 50.899E-10 85.486E-10"? (Or the last two normalized to E-11.) At the very least that's a lot of data for very little gain. It gets worse for trajectories, which might save 1 million time steps x 10,000 atoms/time step x 3 coordinates/atom = 3 billion coordinates to an exchange file. I see no advantage to doing that in SI units.

In practice those distance coordinates will likely internally represented in angstroms. Consider that the Lennard-Jones potential is sometimes written as A/r^12 - B/r^6 , with expected values of r around 1E-10m. The denominator of the first will go to 1E-120 in intermediate form, and not be representable in 32-bit float. While not relevant for Python, which uses 64 bit floats, some molecular dynamics programs will use 32 bit float. (Eg, for older GPU machines, or to save space.)

My other example was the atomic mass unit, another non-SI unit. I have only used amu (for chemistry) or dalton (for biology) in my work, not kilograms. It seems pointless to require that I store the mass of a carbon as 1.9926467051999998e-26 kg instead of 12 amu.

I therefore disagree, and believe there are good reasons to argue against SI units for some data interchange formats. I agree that I want to store a single distance unit on disk, only that unit is the non-SI unit angstrom and amu, and not the tremendously huge meter or kg.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> Computational and biological sciences will likely meet on a financial equivalent to commercial software applications at the intersection of epigenetics and pharmaceuticals in the new few decades.

When scientists begin to discover feasible methods to cure or manage previously incurable diseases (a more recent example of this has been attempts to cure Cystic Fibrosis), or more specifically reversing some of the diseases that our older baby boomer populations are suffering from via epigenetic methods, you can bet your bottoms that there will be a huge influx of capital in the sector and a subsequent increase in demand for computational biologists.

Of course we could end up in a sort of quasi-understanding parallel to that of quantum mechanics and end up in a epigenetic limbo, but the general feeling is that of high hopes.

</comment><comment><USER24> Pair programming keeps it human, and transfers knowledge very well.

TDD is about reproducibility of results, which is very in line with the scientific method. Benchmark tests will show you when your solutions are getting out of hand on performance.

The sunk cost fallacy is a big problem. Moving to a new platform like HTML5/iOS/Android gives a short reprieve, but soon those proprietary code bases will age.

The other big problem is that usually a smaller portion jobs goes towards management in flatter organizations. Managers want lots of layers for job security.

Eric Meijer is right that small teams which are given narrow mission objectives instead of detailed requirements, and measure their problem domain instead of guessing, will be effective.

I'm curious if a Fat-Tree model of management will take hold, [http://en.wikipedia.org/wiki/Fat_tree](http://en.wikipedia.org/wiki/Fat_tree) You get a flatness that improves communication latency, lots of bandwidth, and managers are happy because there are a lot of jobs at the top.

</comment><comment><USER25> I think many in the comments are misunderstanding what the author mean to say in his post. He is not talking about working in academia or making scientific software. He talks about improving one's skills in basic science and such fields in computer science as A.I. which have historically been entrenched more in academia than industry.

<comment><USER26> The author of the post started by getting into programming/CS by wanting to earn more/better money, by picking up an Access book and working it out from there, and now that he's more established, he looks down at the young hungry people who picked up an Access book in the hopes of more/better money, running through all the tropes that people who got where they are through knowledge will use.

Yes, it's useful to understand things better, and to know math. And as always since the first CS degree started, CS people gripe that people should Know More Math. Sure it helps. Other, less prestigious, things also help but you don't hear people griping about it. TDD allows the idiots in. Yes, that's effectively why you want TDD, you want to get more mileage and solving more complex or more bug-sensitive problems using the same people. Building software is not about being smart (although that helps on occasions), it's about getting stuff done.

Yes, machine learning and AI are the new kids on the block, and like Web programming, they will see a bloom of increased customer demand, and like Web programming, we'll get a progression from bespoke boutique software to frameworks that make people's lives easier to frameworks that allow any person with the intelligence of a pet rock to do simple stuff productively. Why is that? Because building frameworks is the only way that the smartest people can earn money faster than programming the (N+1)th variation on that theme everyone follows -- frameworks are what make people more productive, or allow you to use a workforce that's more accessible.

As a Wizard With a Pointy Head (aka academic), I'd say that the need for Wizards With Pointy Heads in production work is often overestimated and/or idealized. There is a large number of PhD graduates, and the market happily gobbles them up (indeed, realizing that you can hire PhDs and have them do productive work is one of the things that made Google successful as a company back in the early 2000s).

</comment></comment><comment><USER27> Mostly agree with the article. Being myself fascinated with machine learning and in the process of refreshing mathematic knowledge I haven't used since university (too many years ago) in order to dive deeper into it, I can definitely relate.

However, I think the main point is not that software developers should all hone their academic math skills (that would be probably be pointless for many if not most software developers), but rather that it would be best if software developers would strive to follow the scientific _mindset_ when developing software - In my experience, occam's razor is just as important in software development (design, architecture, algorithms, testing, you name it) as it is in physics, chemistry or other sciences and it is this aspect (which I feel is the most basic and most important) that gets lost sometimes in the noise of software development trends and fashions.

</comment><comment><USER28> The problem with scientific software is that the market is so small.

It is far more profitable to just write mainstream software.

<comment><USER29> It's more profitable to do scientific research slowly and tediously than it is to automate it?

<comment><USER30> Disclaimer: I'm in the scientific equipment and software business. There are some issues to overcome. These may just be excuses from people who don't want to change their game, but nonetheless:

1\. Sometimes, wages and equipment / software come from different pots of money. No matter how much sense it makes to replace labor with automation in the grand scheme of things, if you can't move money from one pot to the other, then you're stuck with the status quo.

2\. I think that people sometimes underestimate the degree of customization and effort required to automate a specific process. Or sometimes overestimate, as in, "our process is so special that no commercial tools will work for it."

3\. If anybody is going to work on automation, it will be the students themselves, as it's a way to learn a valuable skill. They may be doing it with minimal fanfare, and there is a strong movement towards open source tools. Students realize that the generous budgets and site licenses will vanish when they leave the academe, and are interested in preparing themselves for freelance work, startups, etc. This may also favor general purpose tools, rather than those designed specifically for science.

My anecdote, from 25+ years ago: I taught myself electronics and programming by automating my student projects, culminating in my thesis experiment.

<comment><USER29> I really think we need more required (good) computing courses in both math and science curricula.

<comment><USER30> I was really lucky that I took to programming pretty easily while still in high school. (graduated '82). Likewise with math. As a result, I was able to integrating computing into my work with minimal guidance.

But it's my view that the teaching of math and science should involve computation, starting as early as possible. It still amazes me that a kid can go through high school without learning about something that has had so much impact on our society. In my utopian world, there would be at least one question in each physics homework assignment with the instructions: Solve this with computation. And it wouldn't be a big huge deal.

</comment></comment></comment></comment></comment><comment><USER31> I'm having trouble understanding the definitions of these roles. I see the chart, but the terms are all vague to me. What does a data scientist do that a mathematician or scientist doesn't do, and what does a scientific programmer do that a data scientist doesn't do?

My impression was that "data scientist" was a colloquialism for "statistician that knows how to program." Is a scientific programmer just a programmer that knows some statistics? Why is the direction important? The author says he/she feels that a programmer that knows statistics can make "more robust software" than the other way around, but what exactly does that mean? Do they mean "doesn't crash as much", or do they mean "gives the right answer more often?"

<comment><USER25> for one thing statistics don't really encompass A.I., computer vision etc.

<comment><USER32> Sorry but this is wrong. Statistics and probability theory underpin most of AI and machine vision.

<comment><USER25> One thing can underpin another without encompassing it.

Mathematics don't encompass physics and finance.

</comment></comment></comment></comment><comment><USER33> Here's my attempt at a TL;DR:

* The '90s "Access in 24 hours" programmer has been replaced by the latest anecdote-based technique/toolset preacher; e.g., TDD.

* Becase deep learning is better than humans at finding useful patterns in data (whether concerning biochemistry or web site interaction) it is the best technique.

* Aesthetic (e.g., language) and social justice (e.g., feminisim) issues distract from utilitarian effectiveness.

* Utility is only furthered by math and science (where for "science" read "patterns inferred from data"), and we should aspire to be "scientific programmers" who apply only math and science.

<comment><USER33> My biases may be guessed at in my summarization, but let me make them more explicit.

I think it odd that some of the techniques he rails against are inspired by mathematics: TDD tries to preserve invariants; RESTful design tries to impose the invariance of idempotency. If the question is whether those techniques make those who use computers more productive, then a scientific answer would involve a stunningly expensive human subjects experiment involving large numbers of people and complex problems. The likely result would be this: [https://xkcd.com/1445/](https://xkcd.com/1445/)

I'm a sysadmin doing my best to automate (e.g., puppet); I rarely have the luxury of collecting data sufficient to the immediate problem, so I rely on math and (unreliable) heuristics. I write perl/shell/puppet/ruby/anything in small fizz-buzz complexity chunks; an "artisan" if you will. I support CAE environments with low-latency and poor parallelism opportunities, and until that changes (e.g., becomes cloud compatible) I don't see my tactics changing significantly.

</comment></comment><comment><USER34> A post about data science and scientific programming featuring a set of graphs with no y-axis scale and labels. At my gaff this kind of presentation of data leads to "scrap the whole analysis and start again".

<comment><USER35> dude, have you ever used Google Trends? It is a cut and paste from the source. The y axis is the popularity as the title says. It is so obvious even Google has omitted it.

<comment><USER36> should have labelled it "Magic Google Unitless Dimension"

<comment><USER34> You can put all of the terms on the same graph; that would allow people to compare these on a like for like basis as opposed to "look at the shapes, here is an argument".

</comment></comment></comment></comment><comment><USER37> Deeplearning4j and ND4J contributor here: We've created a distributed framework for scientific computing on the JVM, ND4J, which is the linalg library behind Deeplearning4j, which includes ConvNets and other deep-learning algorithms. Contrary to the author, we believe Scala is the future of scientific computing. While Python is a wonderful language, the optimization and maintenance of scalable programs often happens in C++, which not a lot of people enjoy.

<comment><USER35> Thanks guys for your hard work.

</comment></comment><comment><USER38> A look at the trends for "R programming", compared to "Python programming", is quite interesting too: [http://www.google.com/trends/explore#q=R%20programming%2C%20...](http://www.google.com/trends/explore#q=R%20programming%2C%20python%20programming&cmpt=q)

(Their curves are more or less parallel since 2011)

</comment><comment><USER39> I believe the author is right - This is the reason why I'm spending my final ECTS points on statistics and machine learning.

</comment><comment><USER40> What if we define intelligence, not from an anthropomorphic view, but from a systemic view; as all systems have intelligence.

What is "Artificial Intelligence"? The opposite of "Natural Intelligence"?

<comment><USER41> Saying that all systems have intelligence == anthropomorphizing all systems IF AND ONLY IF you consider intelligence to be a uniquely human trait.

A more useful distinction for the realm of intelligence could be 'designed intelligence vs. grown/evolved intelligence' instead of 'artificial intelligence vs. natural intelligence'; however, stuff like reinforcement machine learning is then neither, or perhaps a hybrid form of intelligence. In the end the pragmatic value of the concept intelligence is low, both for systems as for humans.

</comment></comment><comment><USER42> What is deep learning currently applied to besides object recognition in images?

<comment><USER43> [https://www.quantamagazine.org/20141218-machine- intelligence...](https://www.quantamagazine.org/20141218-machine-intelligence- cracks-genetic-controls/)

</comment></comment><comment><USER44> So Machine Learning in general is an almost solved problem?

<comment><USER14> I think the author claims more like "it works", not that "it is solved".

</comment></comment><comment><USER45> Plug: I made a list of software that is useful for scientists: [https://gist.github.com/stared/9130888](https://gist.github.com/stared/9130888).

</comment><comment><USER46> First of all, I hate this "Agile" nonsense. I've seen it kill companies. It's truly awful, because it gives legitimacy to the anti-intellectualism that has infected this industry. It's that anti-intellectualism that, if you let it, will cause a rot of your mathematical and technical skills. Before you know it, you've spent five years reacting to Scrum tickets and haven't written any serious code, and your math has gone to the birds as well. It's insidious but dangerous, this culture of business-driven engineering mediocrity.

I hope that it'll be the fakes and the brogrammers who get flushed out in the next crash. Who knows, though? Obviously I can't predict the future better than anyone else.

To me, Python doesn't feel like a "scientific" language. Python's a great exploratory tool, and it's got some great libraries for feeling out a concept or exploring a type of model (e.g. off-the-shelf machine learning tools). That said, science values reproducibility and precision, which brings us around to functional programming and static typing... and suddenly we're at Haskell. (Of course, for a wide variety of purposes, Python is just fine, and may be a better choice because of its library ecosystem.) I do think that, as we use more machine learning, we're going to have a high demand for people who can apply rigor to the sorts of engineering that are currently done very quickly (resulting in "magic" algorithms that seem to work but that no one understands). I also agree that "deep learning" and machine learning in general are carrying some substance, even if 90% of what is being called "data science" is watered-down bullshit.

I still don't feel like I know what a "scientific programmer" is, or should be. And I'd love to see the death of business-driven engineering and "Agile" and all the mediocrity of user stories and backlog grooming meetings, but there's nothing yet that has convinced me that it's imminent just yet. Sadly, I think it may be around for a while.

<comment><USER47> Curious, what are the software development processes that you do not consider nonsense?

<comment><USER46> I'm negative on "Agile" because it's an attempt to patch closed-allocation, business-driven development. If you have a high level of talent and you're solving interesting problems, you can do open allocation. If those aren't the case, and for some reason can't be, then you need to take different approaches entirely (but you should seriously question whether what you're working on is worth doing in the first place).

<comment><USER48> "Agile" in many shops means letting the passengers fly the plane (to borrow a phrase from one of your blog posts).</comment></comment></comment></comment>


Zstandard v1.4.0 - <AUTHOR> https://github.com/facebook/zstd/releases/tag/v1.4.0 <comment><USER5> Zstandard is awesome, there is nothing else close to it that I've seen. There are plenty of codecs that give you either obscene ratios or low CPU usage, but none that support both in combination. Zstandard on its highest setting easily competes with xzip for anything I've thrown at it. Decompression throughput is unaffected by compression setting -- it just burns more memory. So you can get xzip-like ratios with decompression throughput approaching 1GB/sec

It allows trading lower CPU for gzip-or-worse compression, and you can mix the settings within a single file. This means you can e.g. use the lowest setting (or no compression at all - it supports that too) to append to a file, while occasionally recompressing recent appends into a single block using the highest setting -- so the cost of compression can be amortized

The only petty annoyance with it is ecosystem support - e.g. GNU tar has no option for it, so it's slightly more painful to work with

<comment><USER6> In my experience, for the best balance between compression speed and compression ratio, nothing beats 7zip with the right options:

      -mmt=$(nproc) # use all available cores
      -ms=off       # disable solid archives (compress each file separately)
      -m0=lzma2     # lzma2 has better threading than lzma1
      -md=64m       # dictionary size
      -ma=0         # "fast" mode
      -mmf=hc4      # hash chain match finder
      -mfb=64       # number of "fast bits"
      -mf=off       # disable filters

The biggest gains are: 1) using all available cores, 2) setting the match finder (the binary tree match finders are terribly slow; I haven't played much with the newer patricia tree match finders), 3) disabling solid archives (this seems to cause 7zip to distribute the work more evenly between cores, though it still may only use a few cores if there are many small files), 4) using "fast" mode (whatever that is, it gives a noticeable performance boost and doesn't seem to affect compression ratio much).

Every few years I try zstd and others, and for the data I work with (primarily a mix of json and fixed-width-field binary data), lots of tools beat 7zip out of the box, but they fall short of 7zip with the above command-line options.

<comment><AUTHOR> A comparable zstd call that uses a 64 MB window size and all cores is:

        zstd --long=26 -T0

From there you can tune the compression level, or increase the window size up to 2 GB (--long=31). zstd won't beat the compression of xz, but it can compress much faster if you trade off some space.

</comment></comment></comment><comment><USER7> There are basically three scenarios where I choose various compression algorithms (a few exceptions excluded):

\- maximum compatibility (while tolerating low performance) - <USER2>

\- great performance (while tolerating larger files) - <USER3>

\- very good performance with good (not best) compression ratios - <USER4>

I don't really want to use Any New Shiny Algo to compress some data that might outlive this piece of software, that's why I use gzip very often, because I know I'll always be able to decompress it. But I've been increasingly adopting zstd and snappy for one single reason - they are becoming widely supported within the ecosystems I work in (data processing).

That, to me, is more important than compression ratios and decompression speeds.

<comment><USER8> How does snappy compare to lz4?

Also, for gzip, you might want to consider using its multithreaded version, called "pigz".

<comment><AUTHOR> lz4 compresses and decompresses faster than snappy, and compresses similarly. You can see some comparisons on the GitHub's readme [https://github.com/lz4/lz4](https://github.com/lz4/lz4).

</comment></comment></comment><comment><USER9> I've been using zstd compression on btrfs for a while now and it's excellent, most of my stuff is already compressed (movies, music) but my home directory (which is mostly comprised of text files) has shrunk greatly.

<comment><AUTHOR> The next GRUB release (grub-2.04) includes my patch to add support for zstd compressed BtrFS filesystems, which should solve one of the major pain points of Zstd BtrFS compression.

<comment><USER10> Cool! Looking forward to that.

A temporary work around in the meantime, I've used `chattr +C` on the directories I want to be exempt from zstd compression, so that grub can read those files.

</comment></comment></comment><comment><USER11> We have many terabytes of large files that are currently compressed using xz (well, pixz actually). In terms of compression speed and ratio, zstd is pretty comparable, and for single-threaded decompression it's faster. At this point the only thing stopping us from using it instead of xz/pixz is the fact that multi-threaded pixz decompression is faster. Are there any plans to add MT decompression to zstd?

<comment><AUTHOR> There aren't any technical limitations to adding multithreaded decompression zstd. We just need a compelling enough use case to justify the work it would take to add it.

pzstd is now obsoleted by zstd -T0, but it offers multithreaded decompression for files compressed by pzstd (it will still be single threaded for files compressed by zstd).

<comment><USER11> Thanks, I didn't know about pzstd, will check it out. See my reply below for more details of my particular use case.

</comment></comment></comment><comment><USER12> How widely is this being adopted compared to (say) Brotli?

<comment><AUTHOR> Disclaimer: I'm a maintainer of zstd, so I'm biased.

Brotli dominates HTTP compression. Zstd just got its RFC approved a few months ago, but Brotli has been present in browsers for years.

However, zstd is more widely adopted everywhere else, especially in lower level systems. Zstd is present in compressed file systems (BtrFS, SquashFS, and ZFS), Mercurial, databases, caches, tar, libarchive, package managers (rpm and soon pacman). There is a pretty complete list here [https://facebook.github.io/zstd/](https://facebook.github.io/zstd/).

Again, I'm biased because I know almost everywhere where zstd is deployed, but not everywhere that Brotli is.

<comment><USER13> With the zstd support in pacman, arch will soon switch all packages over to zstd.

<comment><USER12> Out of curiosity: Why doesn't pacman just use HTTP's built-in compression? It could cache packages in gzipped form, but there's no reason to re-compress them over the wire.

<comment><USER13> Because mirrors aren't guaranteed to use HTTP gzip compression and also gzip compresses much worse than xz.

</comment></comment></comment></comment></comment><comment><USER14> Older discussion: [https://news.ycombinator.com/item?id=16226923](https://news.ycombinator.com/item?id=16226923)

Some of the obvious competitors are brotli and snappy.

Here some comparison:

[https://quixdb.github.io/squash-benchmark/](https://quixdb.github.io/squash- benchmark/)

[http://www.mattmahoney.net/dc/text.html](http://www.mattmahoney.net/dc/text.html) (I wonder if anyone has an updated picture with Pareto front for these numbers.)

<comment><USER15> Mahoney's benchmark is missing the large-window brotli numbers, which are about 5 % better than those of zstd and 10 % better than those of small-window brotli.

Brotli with large window can do around 199M for the 1G text corpus: [https://groups.google.com/forum/m/#!topic/brotli/aq9f-x_fSY4](https://groups.google.com/forum/m/#!topic/brotli/aq9f-x_fSY4)

Here a large window aggregate result view: [https://encode.ru/threads/2947-large-window-brotli- results-a...](https://encode.ru/threads/2947-large-window-brotli-results- available-for-a-several-corpora)

</comment></comment><comment><USER16> Anyone know a decent Windows implementation?

There's a 7zip fork that includes Zstd support, but it can only put Zstd inside a .7z container, which doesn't appear to work with any other tools.

<comment><USER17> The linked site has windows builds...

<comment><USER16> True! I suppose what I actually want is a Windows utility that can make a .tar.zst archive, ideally from a GUI.

In the Windows world, archiving and compression are usually in a single file type (.zip, .rar, .7z). Zstd follows the unix style where it can't directly compress folders of files, they need to be in a tar (or other archive format) first. This isn't really an issue on Linux, since Zstd support is built into tar, which ships on pretty much every system.

<comment><USER18> There haven't yet been any extensions to zip or 7z for zstd support. There is a branch of wimlib that has experimental zstd support, though it's unlikely it will ever be merged into the master branch.

You could make uncompressed zip or 7z files and compress that independently as a zst file, but that's a bit baroque compared to just using tar. :)

7-Zip does often seem bent on supporting everything, I imagine some day in the future it'll support zstd at least as an independent archive, if not extending the Zip and 7z formats as well.

<comment><USER19> GP mentioned this fork of 7-Zip with zstd support:

\- [https://github.com/mcmilk/7-Zip-zstd](https://github.com/mcmilk/7-Zip- zstd)

The changes to the 7-Zip file format were discussed upstream, and upstream agreed to not tread on the magic values:

\- [https://sourceforge.net/p/sevenzip/discussion/45797/thread/a...](https://sourceforge.net/p/sevenzip/discussion/45797/thread/a7e4f3f3/?limit=25&page=2)

But ultimately the patches were not upstreamed. It's now onto its second developer:

\- [https://sourceforge.net/p/sevenzip/discussion/45797/thread/6...](https://sourceforge.net/p/sevenzip/discussion/45797/thread/6db98beb/?limit=25&page=0)

</comment></comment></comment></comment></comment><comment><USER20> Posted my zstd 1.4.0 compression benchmarks [https://community.centminmod.com/threads/round-3-compression...](https://community.centminmod.com/threads/round-3-compression- comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.17259/) \- looking good :)

</comment><comment><USER21> HTTP compression should be disabled by default, 99% of web content downloaded is already compressed.

<comment><USER22> But this is flatly untrue.

Images and video sure, but everything from html, just, css, svg isn't compressed.

In fact compression is critical to modern SPA frameworks to keep initial download times lower.

Also this post has little to do with http compression. ZSTD is used it many other circumstances.

<comment><USER23> And for resources that don't compress well, web servers like nginx (and I presume others) support listing what mimetypes to compress, so they won't double-compress those things.</comment></comment></comment>