Happy new year - <AUTHOR> <comment><USER2> Don't forget, there's always something (new) for our lives ... Just like Charles Bukowski wrote:

      don't forget
      there is always somebody or something
      waiting for you,
      something stronger, more intelligent,
      more evil, more kind, more durable,
      something bigger, something better,
      something worse, something with
      eyes like the tiger, jaws like the shark,
      something crazier than crazy,
      saner than sane,
      there is always something or somebody
      waiting for you
      as you put on your shoes
      or as you sleep
      or as you empty a garbage can
      or pet your cat
      or brush your teeth
      or celebrate a holiday
      there is always somebody or something
      waiting for you.

      keep this fully in mind
      so that when it happens
      you will be as ready as possible.

      meanwhile, a good day to
      you
      if you are still there.
      I think that I am---
      I just burnt my fingers on
      this
      cigarette.

      By Charles Bukowski

Happy New Year folks. Here's to "being prepared".

<comment><USER3> Happy new year from Barcelona!

</comment></comment><comment><USER4> Happy New Year. Let's work towards the best version of ourselves, as long as it isn't taking advantage of others.

<comment><USER5> Take advantage of others. Take advantage of yourself. Let this not be a zero sum game.

</comment></comment><comment><USER6> May this year see you writing many many negative lines of code :)

Seriously, simplifying any code I write is probably gonna be my resolution for this year.

<comment><USER2> The year of the refactor? Drink to that.

</comment></comment><comment><USER7> Happy new year from Seattle: [https://www.flickr.com/photos/aaronbrethorst/24017817061/in/...](https://www.flickr.com/photos/aaronbrethorst/24017817061/in/dateposted- public/)

</comment><comment><USER8> This year was really trying for me.

To everyone else that had a rough time, heres' to the fantastic opportunities of this _new_ year.

</comment><comment><USER9> Happy New Year from Los Angeles! May your code have no errors and your servers experience no down time!

</comment><comment><USER10> May your types be dependent and your functions pure.

#Idris2016

</comment><comment><USER11> Happy new year from Lagos, Nigeria, West-Africa, Earth

</comment><comment><USER12> } // This was a good one

</comment><comment><USER13> </year> <year>

<comment><USER14> you must mean </2015> <2016> or do I need an XML course this year?

<comment><USER15> IIRC an int is not a valid xml tag name. I think the best option would be like this:

</year><year ad="2016">

<comment><USER16> the original one is correct ...</year><year>..., its an succession of years with different content in them

</comment></comment></comment></comment><comment><USER16> Happy new year from Los Angeles, I hope your main projects become a reality

</comment><comment><USER17> Here's to a sweet sixteen ;)

<comment><USER18> If '99 is his birth year it's good for him to read HN :)

</comment></comment><comment><USER19> Happy New Year everyone and may you win this 2016!!

</comment><comment><USER20> Happy new year from Sarajevo

</comment><comment><USER21> Happy new year from Straya!

</comment><comment><USER22> Cheers! Let's make 2016 even better!

</comment><comment><USER23> Best wishes! I truly hope you make the most of this new year and I aspire to do the same. Good luck to us all.

</comment><comment><USER24> Happy new year! Best wishes. May this year will turn out to be your best year of the life.

</comment><comment><USER25> Happy new year from Copenhagen!

</comment><comment><USER26> Happy new year from Brooklyn!

</comment><comment><USER27> Happy New Year from Nepal..

</comment><comment><USER28> Happy New Year from Lisbon y'all! May you be blessed to many moons.

</comment><comment><USER29> Happy new year from China!

</comment><comment><USER30> Happy new year from Silicon Valley! Here's to new beginnings :)

</comment><comment><USER31> Happy New Year from Mars!

<comment><USER32> Wait, what?

</comment></comment><comment><USER33> Happy new years guys,let the new year be a good one!

</comment><comment><USER15> Happy new year from Manchester, England!

</comment><comment><USER34> Happy New Year.

</comment><comment><USER35> Happy new year my favorite community

</comment><comment><USER32> Happy new year from Rio de Janeiro!

</comment><comment> max-si-m Happy new year from Ukraine :)

</comment><comment><USER36> Happy new year from Dublin!

</comment><comment><USER37> Happy new year!

</comment><comment><USER38> Happy New Year!

</comment><comment><USER39> Happy new year;

</comment><comment><USER40> Happy new year from Kerala, India

</comment><comment><USER41> Happy new year from Canada +)

</comment><comment><USER42> Happy new year from Guatemala

</comment><comment><USER43> Happy new year from Paraguay

</comment><comment><USER44> Happy new year from Madrid!

</comment><comment><USER45> Happy new year HN folks.

</comment><comment><USER46> Happy new year everyone

</comment><comment><USER47> Happy new year! (:

</comment><comment><USER48> Happy New Year :)

</comment><comment><USER49> Happy new year!

</comment><comment><USER50> Happy new year!

</comment><comment><USER51> Happy new year!

</comment><comment><USER52> Happy 2016!

</comment><comment><USER53> happy new year</comment>


"Breathability" Comparison of Commercial Outerwear Shell Layers (2018) [pdf] - <AUTHOR> https://www.shelbyoutdoor.com/images/support/breathability.pdf <comment><USER2> This study is older than 2018. I used it to decide on eVENT (terrible unsearchable name) fabric when I bought waterproof clothing over 7 years ago.

I need breathable waterproof clothing because I cycle as my main form of transportation, including in bad weather. I find eVENT breathability adequate for staying dry while cycling at a relaxed pace (12mph average over fairly flat terrain on a touring bike), but insufficient at my more usual pace of about 15mph (same conditions). If there's only light rain I'll open pockets to get unimpeded air flow, which makes a big difference. If I was buying again I'd get something with specifically designed vents for more airflow. There's probably no fabric that can keep you dry during fast cycling while fully enclosed. If you only ride slowly or use electrical assistance then eVENT is likely good enough.

<comment><USER3> I also referenced this study when I bought ski wear. eVENT indeed seems pretty good, although they used to have an obnoxious "direct venting" marketing scheme in which they didn't allow vent zippers because the fabric breathes so well. This is, of course, nonsense.

It's worth noting that, in cold dry weather, you don't necessarily want maximum breathability. If the humidity inside your jacket drops too much, then your skins loses a lot of moisture, which dries you out and also means you lose a lot of heat to evaporative cooling.

So the Gore-Tex XCR curve might actually be good: it transmits vapor better when humidity is high.

<comment><USER4> In cold, dry weather you shouldn't wear a waterproof jacket.

<comment><USER5> But you should usually wear a windproof layer, and a waterproof layer serves as such, even if it's overkill.

<comment><USER6> If you are out in cold, dry and windy conditions, then you can wear a jacket made from a highly breathable but windproof material. The one I have is made from Gore Windstopper.

<comment><USER5> I prefer to layer a very thin windshirt under my insulating layer.

</comment></comment></comment></comment></comment></comment><comment><USER7> Combine this with measured water resistance, to help me find 'the most breathable material that will stand up to X minutes of rain', add data for 'crinkliness' and stretchability (even just your own qualitative scale), then add affiliate links to actual products that use these materials and you've got yourself a nice little side project to maybe generate a bit of passive income.

<comment><USER8> So "just" three more experiments.

How much would the kit for each experiment cost? And how long would each test last?

Could be a big investment for a bit of passive income.

<comment><USER9> based on the sheer number of SEO hacking coupon sites and huge sponsorships for coupon extensions, I'm going to wager a guess that you could earn a significant amount of money if you could position yourself as the definitive source on outerwear information.

</comment></comment></comment><comment> et-al I once undertook the task of searching for the mythical fabric that offered breathability and water-resistance.

A big issue is that as your exertion increases, you sweat more and need more breathability, compared to when you're sitting around. In addition, breathability is hampered by the fabric's pores being clogged by dirt, body oils, and just the fabric wetting out. As such, most fabrics are treated with some DWR coating. But the DWR coating wears off over time and at-home treatments never seemed to match what came from the factory.

Roughly a decade ago, eVent was supposed to be the wünderfabrik; and it did offer more breathability than Gore-Tex by omitting a PU layer. However the PU layer also prevented body oils from logging up the breathable membrane, so ultimate eVent jackets would stop being breathable if you didn't keep them clean.

Polartec Neoshell was also a runner up, but it never caught on big time; I think due to durability issues.

Right now, Gore-Tex Shakedry is the talk around town, but I haven't worn anything using it and stopped caring. The outdoor industry will always be selling folks a golden fleece.

\- - -

Addendum: I am super glad about how breathability is now being addressed beyond rainshells and fleeces. Patagonia's Nano-Air and Arc'tery's Atom hoodies are two examples of light and breathable insulation layers that didn't exist 10 years ago. They're great for moderate exertion activities.

If anyone wants to geek out more, I recommend reading:

Mark Verber's _What is Percent of Nakedness?_ \- [https://verber.com/mark/outdoors/stash/patagonia- testing.htm...](https://verber.com/mark/outdoors/stash/patagonia-testing.html)

Andy Kirkpatrick's (formerly psycho vertical) writings; mainly focused on mountaineering. As such he has a lot of good advice for dressing for bad conditions - [https://andy-kirkpatrick.com/articles](https://andy- kirkpatrick.com/articles)

Andrew Skurka's _Breathability: an explanation of its importance, mechanisms, and limitations_ \- [https://andrewskurka.com/breathability-its-importance- mechan...](https://andrewskurka.com/breathability-its-importance-mechanisms- and-limitations/)

<comment><USER10> The design of the garment also comes into play in a big way. For things like hiking and working in the rain, Pit Zips are vital and allow a greater level of control than any magic fabric can offer.

</comment></comment><comment><USER11> For something completely different, here is the non-breathable Stephenson's Warmlite Vapor Barrier "No Sweat" Shirt:

[https://www.warmlite.com/product/no-sweat- shirt/](https://www.warmlite.com/product/no-sweat-shirt/)

I bought one when I was into backpacking in the '70s but never really liked it that much. I would be curious if anyone knows anything about these and their "vapor barrier" idea.

I do have fond memories of their 1974 catalog (NSFW PDF):

[https://www.outinunder.com/sites/default/files/Warmlite%20Ca...](https://www.outinunder.com/sites/default/files/Warmlite%20Catalog%201974.pdf)

<comment><USER12> Who makes a NSFW clothing catalogue?

<comment><USER13> Naturism was pretty SFW before the resurgence of prudism.

<comment><USER11> The nude photos in Stephenson's catalog would have been NSFW in 1974 too. You wouldn't put one in a frame and keep it on your desk at work!

Not even if your boss was one of the cool people you might invite over for a hot tub party.

</comment></comment></comment></comment><comment><USER14> Weird that no Scandinavian brands made it on this list, do Americans not buy those? Stuff like the North Face is genuinely considered a pretty poor quality brand here in Denmark. Typically North American brands lose these types of tests by our consumer focused NGOs as well.

My inherited 150 year old Icelandic wool sweater is more water/wind resistant than my Fjällraven Hard shell though. So maybe quality just isn't really what it was anywhere.

<comment> hdrujvw-4579 I bought a NF jacket thinking it would be something that could last me a good 5+ years. The inner padding around the collar just desintegrated after a few weeks and looks like a white plastic band. The outer shell absorbs moisture and is far from the water repellent/absorver it's advertized. Maybe I'm chasing old unicorns but these "technical" brands are not what they used to be say 20 years ago, despite their higher prices.

<comment><USER15> NorthFace has become a lifestyle brand, so they don't have a lot of incentive to make things technically good anymore.

If you want good technical clothes you need to pick brands that average people don't recognize, because they have an incentive.

<comment><USER16> I think it's fairer to say that TNF has expanded also to include a whole lot of lifestyle clothing.

They continue to offer quite a lot of expedition-ready gear (tents, waterproof shells, etc.) that's definitely not lifestyle stuff - mainly under the "Summit" sub-brand.

e.g. does the shell have a hood that's sized to fit over a climbing helmet? Probably "serious".

</comment></comment></comment></comment><comment><USER17> Some of these don't exist anymore. Nextec from Patagonia, huh? I've given up finding a breathable waterproof layer. If it's heavy rain then I accept I'll get sweaty underneath, if it's cold then water resistant soft shells are awesome.

<comment><USER18> I experimented with different waterproof membranes a lot, and the one that seems to really breathe the best not just on paper is Polartec NeoShell. It also has a correspondingly lower waterproof rating, but still high enough to be fully rainproof. The downside (or the upside, depending on your perspective) is that it's not windproof - you can very much feel the gusts going _through_ it. It's also not as sturdy as Gore-Tex.

<comment><USER19> I've heard reports [0] that Polartec NeoShell is not very durable. After a few months of light usage, apparently the fabric develops cracks at the shoulders and wrists.

[0] [https://www.bergfreunde.de/basislager/polartec-neoshell- das-...](https://www.bergfreunde.de/basislager/polartec-neoshell-das-beste- aus-zwei-welten/)

<comment><USER18> My NeoShell items have 1-3 years of usage on them, without any cracks. This looks like a single comment - more likely to be an issue with that particular manufacturer or product.

</comment></comment></comment></comment><comment><USER20> Wow, this actually confirms my thoughts about my Columbia Omni-Tech, that thing seems like the _least breathable_ jacket I've ever owned. Good thing it has pit zips. It's like a cold rain jacket.

There seems to be 2 distinct categories: some fabric is asymmetric when it comes to relative humidity, others stay constant (like Omni-Tech). The asymmetrics become more breathable as humidity increases. Schoeller WB-Formula starts off half as breathable as Omni-Tech at low humidity and at high humidity comes down to match. But you've also got Gore-Tex XCR, which is more breathable than O-T at even low humidity, and gets even more so as humidity increases.

I've had a Columbia 3-in-1 with a softshell liner that I've really loved. It can take about ¼" of rain, but isn't quite as warm as a fleece. I washed the liner a bunch, but never washed the shell (still looks new, waterproof AFAIK). I just bought myself a Marmot 3-in-1 for Christmas (unaware of all this research). Shell is apparently Membrain, which is both _more and less breathable_ than Omni-Tech (less when dry, more when humid).

"Asymmetric" breathability fabrics, most->least breathable: Gore-Tex XCR, Entrant Dermizax, Gore-Tex, Membrain, Hydroseal, Sympatex, Conduit, Schoeller WB-Formula. All these are more breathable when humid (opposite probably not very useful).

"Symmetric" breathability fabric, most->least breathable: Expanded PTFE, EVENT, Schoeller Dryskin Extreme, Entrant GII XT Laminate, Nextec, Lowe Alpine Triplepoint, Omni-Tech.

The groups overlap, but symmetric fabrics are generally more breathable overall (especially when dry, but even still when humid). Interesting.

</comment><comment><USER21> Besides reading about them has anybody had first hand experience with "breathable" stuff that actually keeps you dry in real rain? To me this stuff seems like a mythical beast that is supposed to exist but I can never find.

<comment><USER22> I backpack in the Pacific Northwest and have hiked for hours in pouring rain at temperatures of berween 40 and 80 degrees and have found a standard gortex jacket by Marmot to be totally acceptable for this task.

On warm days I were just a base layer (the wickier the better) under the jacket with all the air vents open. I sweat a lot so I need good airflow. I imagine if you bike in similar conditions you'd need a better solution but aside from a lot of gimmicks I'm not sure what would be better.

<comment><USER23> > I backpack in the Pacific Northwest and have hiked for hours in pouring rain > at temperatures of berween 40 and 80 degrees and have found a standard > gortex jacket by Marmot to be totally acceptable for this task.

Aren't you sweating inside the jacket anyway due to hiking? So you're wet inside whether you're wearing a jacket or not? So what's the point in a waterproof jacket?

I think it's best to just accept you're going to get wet, wear basic fabrics, and practice wet-dry drills.

<comment><USER24> No, you get humid, but not wet when everything is working right. Your sweat evaporates and you are not wet.

<comment><USER23> Depends how fast you're moving, what kind of climb you're hiking up and what kind of load, I guess, but I've been sweating out in a snow storm in Wales when carrying a heavy pack moving at speed and I would not have wanted a waterproof on.

<comment><USER24> Sure, you can sweat more than your jacket can transpire. In general though I think you should strive to avoid liquid sweat in winter conditions, not just "accept" it. Improve your layers, or your fitness, or reduce your load or speed. Getting wet in winter can be dangerous.

<comment><USER23> > reduce your load or speed

Not really an option if you need to be at a given place at a given time with given equipment! That's why the military don't fight in waterproofs. They just accept they're going to get wet and then practice disciplined wet-dry drills.

<comment><USER24> I thought we were talking about backpacking in the pacific northwest (or at least other recreation). Obviously the military is a completely different animal, and they do not care what I think.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER25> What's a reasonable expectation for durability of these fabrics?

My most recent hydrophobic jackets were Rab (6 years) and Mountain Hard Gear (2 years). While initially awesome, they've both eventually lost their hydrophobic properties. The fabric treatments (NikWax Tech Wash), fastidiously applied, don't work.

Should I just plan on buying a new jacket after X hours of use?

<comment><USER24> I'll note that I used to line dry hydrophobic jackets after washing, but that was actually a mistake. The warmth in the dryer somehow anneals the DWR, refreshing it. (This will no work forever, of course)

</comment></comment><comment><USER26> I highly recommend spending a bunch of money on a well-constructed waterproof shell. My last Gore-Tex jacket lasted 13 years before finally breaking down in one of the pit zips. And pretty much every time I put it on, I was really glad to have it for the comfort it provided in cold, wet, or windy conditions.

You just need to be realistic about what a "breathable" shell can do. If you're moving enough that you would be sweating without a shell, then of course adding a shell is just going to make the situation much worse. In practice this often means that I'll be wearing just a thin base layer when I'm actually moving, and often prefer to get rained on than drenched in my own sweat. And then as soon as I stop and my body temperature starts to drop, the shell and extra thermal layers come out.

To me breathability is less important than a number of other factors like hood design, size, weight, durability, looks, how it feels against bare skin, venting options, zippers, etc.

<comment><USER23> Right - waterproof layers are for when you are _stationary_ only.

</comment></comment><comment><USER27> Buffalo Systems kit has always been good for me. Wear it properly (next to your skin) and even though it's not fully waterproof you will not get cold which is the main thing. Bring a dry top to put on when you've finished and done. Dries out really quickly and you can put it on wet without it feeling cold.

<comment><USER28> I feel like this secret is being kept alive by a dwindling community unfortunately. I have a great Montane Extreme Jacket, similar to the Buffalo one but with a better hood and not a pullover smock. I had a Buffalo about 15 years ago too.

This move to Gore-Tex et al reminds me of what happened after the invention of the trainer.

Trainers (sneakers) are everywhere now, and they get more and more complex each year yet people get injured running in them just as easily.

A small sommunity kept alive the idea of running in minimal shoes, zero drop, flat unpadded soles, and running like we did in the past.

Similarly, Gore-Tex came along and now we're obsessed with shedding the water on the outer layer. The modern mantra is water must not penetrate under any circumstance. There's a time and a place for that but in those conditions a truly impermeable layer like a PU shell is better.

And so there's just a few of us who know that it's not necessarily preventing all water ingress that's the goal - it's maintaining warmth, breathability, comfort and having material that will dry quickly.

Maybe it's because of our climate in the UK but it's no accident Buffalo, Montane, Mountain Equipment and Rab (all UK based companies) make 'Pile and Pertex' colthing and now some exceptional Primaloft based synthetic shells - The Mountain Equipment Citadel and Fitaroy Jackets, and the Montane Prism are two I own and are every bit as good as Pile and Pertex but without the bulk.

It's good to know there are a few of us out there keeping the wisdom alive!

<comment><USER29> I'm in the UK too and it's been great to have so many local contributions in this thread. I'm an enthusiastic walker (rambler?) and am always looking for suitable clothing - not just for walking holidays, but for everyday, keeping warm, dry and comfortable in variable weather and rain (lots of it this Christmas!). Whether a stroll, hill climb, birdwatching or taking a dog for a walk, getting sweaty or wet is what I'm always aiming to avoid.

> Maybe it's because of our climate in the UK

I do wonder if our climate is a bit "different" when it comes to outdoor clothing. Are we more humid and damp than most of the USA and Europe? They do great gear, but it often seems to assume a different environment (the alps) or different climate (sun for X days, then rain for Y days). Gear for us, given the small range of temperature variations and fast changes in the weather, seems to have different requirements.

Thanks for recommending EDZ in another comment. I like the performance of outdoor clothing but often don't want to look like an outdoor enthusiast (because I merely walk a lot, and also want to blend in in towns).

Can you recommend any trousers that keep you dry(er) in showers, or dry quickly once wet? I got caught in a squall on Exmoor today and soaked through. Stayed warm in no small part thanks to a Trespass AT200 fleece (ridiculously cheap for the performance) but warmer legs would be nice. For torrential rain there's overtrousers, but I'm not a fan outside winter (too hot and sweaty).

<comment><USER28> I use Rohan Overlanders. They are pretty lightweight but warm enough when moving. 6 pockets that fit an OS map :-)

They dry extremely quickly and you can give them a wash in tech wash to give them a bit dwr. I think they hit a good balance of fast drying and warm enough when moving.

But if it's going to be very wet and you know it then some simple baselayer trousers with a brethable (read: air permeable, not just water vapour) shell trouser, preferably with braces, is the way I go. eVent is great but so is NeoShell I think it's called.

</comment></comment></comment></comment><comment><USER30> Exactly why I love my lightweight extra large packable poncho.

It's completely water proof, but it's got a shit ton of ventilation, I can wear exactly what I need to stay warm and if it starts raining, sleeting, snowing, etc. on with the poncho, I stay dry and warm, all of me stays dry, and not just my upper body.

I've actually done trail running in my poncho, but it's a pain, frankly most people don't need all this "technology" to stay dry and warm. I use a cotton tee, a wool sweater and a cheap insulated jacket with pit zips, and a poncho. It works for anything down to about -20F, of course you still need to protect your legs for warmth, because the poncho protects them from the water.

My poncho is so old it could vote, I don't even know who made it, I do remember it was cheap, but not flimsy, it has some body so that it won't act like it's still packed. I may not meet today's standard for packable, but eh, what's an extra couple of ounces in the scheme of things.

</comment><comment><USER31> I've just come to accept that I'll get a bit wet one way or another; either sweat or rain ingress through hood or cuffs (this will happen eventually as you'll need to stop for lunch or to put up shelter). Keeping warm is more important than keeping dry.

<comment><USER23> Yes this is what I think - if you're outdoors and doing something you're going to be getting wet, inside and out. Might as well accept that and work with it (fast drying clothing, wet dry drills) rather than fight it.

</comment></comment><comment><USER32> As somebody who spends most of their time on the Celtic and Nordic fringes, Im a big fan of non-breathable PU (fishermans) waterproofs.

I think breathability is an overrated quality for waterproof fabrics. Zips and vents combined with waterproofness and robustness is the way to go.

<comment><USER18> As somebody who lives in PNW, my take is the exact opposite: waterproofness is the overrated quality, because the most commonly used fabrics are _way_ more waterproof than necessary to completely block out even extremely heavy rain. On the other hand, breathability means a lot less sweat on the move. The difference between e.g. classic Gore-Tex and NeoShell is night and day.

<comment><USER33> Based on my PNW experience having a hard shell is rarely worth it, unless you are static. Otherwise softshell jackets is the way to go, much more breathable and fast drying.

<comment><USER18> I'd say it's very much worth it when hiking out in a downpour in winter.

</comment></comment></comment></comment><comment><USER34> Generally, it's safest to assume that any waterproof material is non- breathable. If you exert yourself in them, you sweat and trap that sweat inside the membrane, often guaranteeing to soak yourself even if the rain lets up. Windbreakers with some basic spray on water resistance just work better (which is to say, not great).

Best thing overall is to just wear synthetics that dry very quickly (pair with waterproof bag for your backpack contents - trash compactor bags are amazing for weight efficiency)

<comment><USER35> I've had a perfect experience with my Arc'teryx Alpha FL from 2013.

The marketing said Goretex Active, and it worked like a charm.

I've worn it on three hour bike rides, snowboarding all day, 5-10 hour winter and summer hikes. I took it to Iceland and Greenland (and was quite active in the rain) and it just worked.

I live in Seattle and only maybe two days a year did I wish that it had pit vents.

It's expensive[0], but the technology works!

[0] Given it's lasted 6 years, and still looks good, I now categorize it as an investment rather than an expense. I love it so much I want to give them more money, so this spring I'll be buying a new one.

</comment></comment><comment><USER36> I've switched almost completely to breathable clothes over last couple of years. Huge fan of Icebreaker merino stuff (underwear and hoodies). Using breathable shorts from Kathmandu.

Big problem for me is that most of the stuff is way overoptimized for hiking and trekking while I just wish something simple and socially acceptable.

Also one thing I haven't figured out is t-shirts. Sticking to cotton for now but do want something I can keep on for days or even a week.

<comment><USER37> Yes, "technical" clothing for regular use seems to be a unfilled niche. You either get stuff for trekking, or that awful "techwear" trend that makes you look like a ninja who's really into Nine Inch Nails, with a bit of tacticool prepper mixed in.

Uniqlo has some "techy" fabrics that don't look too out of place, but a slightly better quality and slightly higher age average wouldn't hurt (skinny fits, friggin' 7/8 pants).

I've experimented with Merino shirts and socks for traveling with carry on luggage, but they're just too damn fragile and develop small holes way too soon. Yes, maybe I didn't take care of them the right way, but not having to take care of them too much is pretty much the point...

<comment><USER36> I agree merino doesn't last long. Worst were Allbirds. Super comfy for light use, but wears out in like 6 months.

"Techy" style is subtle tho. I wouldn't mind a premium lot2046 line. You'd have to kill me to wear anything star wars. That is just dweeb.

IMHO Some of normal stuff is just too normcore dad clothing.

I guess agreement is wet need more of every tribe.

</comment></comment></comment><comment><USER38> A bit off topic but I'll go up to Scotland in February and still need a hard shell. My old Mountain Hardwear won't make the cut anymore. Any recommendations for good deals? I only need it for a week every other year for some Alpine climbing so I don't want to spend crazy money but also don't want to get soaked when the weather is bad.

</comment><comment><USER39> As a frugal person, I'm as concerned with durability as performance. Anecdotally, after trying cheaper membranes that disintegrated or lost their waterproof-ness after frequent use (daily bike commuting, being stuffed in a bag), I have come to prefer Gore-Tex for how long it lasts.

</comment><comment><USER40> This is cool. I have purchased several pairs of pants that claimed to be "breathable" but in fact were anything but. I'd basically given up on breathable waterproof fabrics but this study seems to indicate that some of them may be OK.

</comment><comment><USER41> Theoretically, could a jacket made of a hard material ( _resistant to water for 50 years minimum_ ) perforated with nano-particle size holes be both breathable and highly water-resistant?

<comment><USER3> The surface needs to be hydrophobic, too. Otherwise, when the outside gets soaked, holes will be blocked. Presumably it also needs to be thin so the holes aren't too deep.

At this point, you more or less have a textile. A piece of aluminum foil isn't going to make a great jacket.

</comment></comment><comment><USER42> It's fascinating that "breathability" appears to increase as mean humidity increases.

In the end, the numbers are great until you accidentally wash your coat with detergent!

<comment><USER43> You can re-activate the fabric by washing it with a special spray that you spray on and tumble dry it with.

</comment></comment><comment><USER44> I had an idea for a 3d printed layer that could vent on the shoulders and would be rain proof if you walk upright..</comment>


Ask HN: What's the big deal about Deep Learning? - <AUTHOR>

<comment><USER2> It's a big deal because it works.

Back in the 1960's people only knew how to train one-layer neural networks. By the late 1970's and 1980's people were training "multi-layer perceptrons" but it wasn't practical to train networks with a lot of layers.

In the 1990's and early 2000's Neural Networks were out of favor for academic work, not so much that they were bad but for almost any learning problem there was some other kind of algorithm that did better. Neural Networks were still used a lot in commercial applications that weren't so talked about since they thrive given the kind of intensive attention that they sometimes get in such environments.

Geoff Hinton then figured out a way to quickly train deep networks to learn the statistical character of the input data. This works as an unsupervised process. It's kindof like the first layer learns very simple features that recur in the data and then the second layer learns relationships between theem and so forth.

After you've done that, you can teach the network to do some specific task using a supervised training algorithm such as back propagation. Because the network is close to where it needs to be, you can now apply back propagation to the whole network with excellent results.

Deep learning is such an advance that neural networks are now often the leading algorithm for many learning problems.

</comment><comment><USER3> Simple: Data Scientists spend a great deal of time doing feature engineering.

That is, finding observations about their data that are indicative of a prediction they are trying to make or a piece of data (say: a user or an image, perhaps a document) they are trying to label.

Deep Learning handles this for us. Typically machine learning takes a lot of domain knowledge. Deep Learning is capable of automatically finding correlations in the data and allowing us to focus on solving problems rather than worrying about what good indicators of data are.

Re: quick pretraining. Try again: deep learning pretraining takes forever to train due to how many passes through the dataset you need to make in order for it to make good approximations of the data.

That being said: deep learning is very receptive to map reduce type algorithms and online learning (streaming of data in mini batches)

That's why bigger companies tend to use it at scale.

That being said: it's astonishingly accurate despite being hard to train.

I will be more than glad to answer specific questions if you'd like.

I'll give you a few use cases:

Google uses it in their voice recognition on android.

IBM uses it in their Watson technology.

Google uses it in their Google+ image search.

It's being used for sentiment analysis at stanford.

Source:

I'm the author of a newer deep learning framework called deeplearning4j [1].

My goal with this is to make it easier to train these neural networks by just say: feeding it images and all you need to do is specify the labels: or text.

I'm also a collaborator/committer on another effort called metronome which is deeplearning on hadoop [2].

If you'd like an example of what deep learning is capable of, please take a look at [3]. It's a very different way of doing machine learning.

[1] [http://deeplearning4j.org](http://deeplearning4j.org)

[2] [https://github.com/jpatanooga/Metronome](https://github.com/jpatanooga/Metronome)

[3] [https://jpatanooga.github.io/Metronome/rbm20140306.html](https://jpatanooga.github.io/Metronome/rbm20140306.html)

</comment><comment><USER4> Current "machine learning" algorithms such as decision trees, SVM, regression etc are in a way cheating. In all these algorithms you need to design features and they contain much of the smarts that you see from "machine learning" algorithms. In other words, an impressive machine learned model is typically a result of an impressive features gymnastic manually programmed by humans after lot of sweat and blood.

A true machine learning algorithms shouldn't require humans to encode their smarts and knowledge. If these algorithms existed then they would just look at stream of symbols or sensory inputs that occurs in the nature and extract patterns from it. Deep learning gets closer to this by eliminating need for developing features (to some extent so far). I think it is first of many steps towards true machine learning.

</comment><comment><USER5> Are there any resources for someone wanting to get started looking into coding/understanding deep learning algorithms?

By which i mean, i'm already a data scientist. I'm not looking for software that runs it for me per se, i'm looking for resources that could let me implement/research it myself.

I've seen a lot of hype. Some of the claims I've seen people make for deep learning I know must be objectively false and sound like a lot of the hype/marketing for other machine learning techniques, but the ideas also look somewhat promising, so i'm really curious to get involved and at least be able to make an educated decision...

<comment><USER4> The [http://deeplearning.net/](http://deeplearning.net/) is a go to website for beginners. You would find everything from datasets to experiment to tutorials. This subject requires a lot of match and ML background though and would need sometime to catch up to the latest research. Coding part will involve GPU code if you want to go truely in with large data sets. If you want to just apply the algorithms to specific problem without diving deep in to its theory there are few kits available in Python and C++. There a tech talks and other videos on tutorial page that might be good starting point.</comment></comment>


Why events are a bad idea for high-concurrency servers (2003) [pdf] - <AUTHOR> https://people.eecs.berkeley.edu/~brewer/papers/threads-hotos-2003.pdf <comment><USER2> Worth noting this is from 2003. The performance concerns of event-based servers have been greatly alleviated by both hardware and software advancements.

The test setup used for this paper was a "2x2000 MHz Xeon SMP with 1 GB of RAM running Linux 2.4.20". Solid PC server iron for 2003, but basically equivalent to a $5/month server from Digital Ocean today.

If you're looking to squeeze 100,000 concurrent tasks from that $5 server, this paper is relevant to you.

<comment><USER3> C10K is another name for evented I/O, and it's from the _90s_. By 2003 thread- per-client was already obsolete and known to be very bad.

It's really quite simple: threads encourage the use of implicit program state via function calls, with attendant state expansion (stack allocation), whereas evented I/O encourages explicit program state, which means the programmer can make it as small as possible.

Smaller server program state == more concurrent clients for any given amount of memory. Evented I/O wins on this score.

But it gets better too! Smaller program state == less memory, L1/2/3 cache, and TLB pressure, which means the server can take care of each client in less time than an equivalent thread-per-client server.

So evented I/O also wins in terms of performance.

Can you write high-performance thread-per-client code? Probably, but mostly by allocating small stacks and making program state explicit just as in evented I/O, so then you might as well have done that. Indeed, async/await is a mechanism for getting thread-per-client-like sequential programming with less overhead: "context switching" becomes as cheap as a function call, while thread-per-client's context switches can never be that cheap.

The only real questions are:

      - async/await, or hand-coded CPS callback hell?
      - for non-essential services, do you start with
        thread-per-client because it's simpler?

The answer to the first question is utterly dependent on the language ecosystem you choose. The answer to the second should be context-dependent: if you can use async/await, then always use async/await, and if not, it depends on how good you are at hand-coded CPS callback hell, and how well you can predict the future demand for the service in question.

<comment><USER4> A context switch in a modern CPU takes only a few microseconds. A GB of RAM costs less than $10. So those concerns, although valid in theory, are usually irrelevant for most web applications.

On the other hand, simplicity in a code base usually matter. Code written with an evented API, littered of callbacks, is usually harder to read and maintain than that written in a sequential way with a blocking I/O API.

You can recreate a sync API on top of an evented architecture using async/await, but then you have the same performance characteristics of a blocking API, but with all the evented complexity lurking underneath and leaking here and there. Seems to me a very convoluted way to arrive to the point from where we started.

<comment><USER5> A GB of RAM only costs less than $10 if you are buying for your unpretentious gaming rig.

A GB of ECC server RAM costs more. An extra GB of RAM _in the cloud_ can even cost you $10/mo if you have to switch to a beefier instance type.

<comment><USER6> $10/mo is far less than the cost of thinking about the issue at all.

<comment><USER3> Yes, but. Suppose you build a thread-per-client service before you realized how much you'd have to scale it. Now you can throw more money at hardware, or... much more money at a rewrite. Writing a CPS version to begin with would have been prohibitive (unless you -or your programmers- are very good at that), but writing an async/await version to begin with would not have been much more expensive than a thread-per-client one, if at all -- that's because async/await is intended to look and feel like thread-per-client while not being that.

One lesson I've learned is: a) make a library from the get-go, b) make it async/evented from the get-go. This will save you a lot of trouble down the line.

</comment></comment></comment></comment></comment></comment><comment><USER7> A thread from 2017: [https://news.ycombinator.com/item?id=14548487](https://news.ycombinator.com/item?id=14548487)

2014: [https://news.ycombinator.com/item?id=7684163](https://news.ycombinator.com/item?id=7684163)

2011: [https://news.ycombinator.com/item?id=2907415](https://news.ycombinator.com/item?id=2907415)

Smaller but interesting threads from 2010-12:

[https://news.ycombinator.com/item?id=3482002](https://news.ycombinator.com/item?id=3482002)

[https://news.ycombinator.com/item?id=3101451](https://news.ycombinator.com/item?id=3101451)

[https://news.ycombinator.com/item?id=2910849](https://news.ycombinator.com/item?id=2910849)

[https://news.ycombinator.com/item?id=1547353](https://news.ycombinator.com/item?id=1547353)

[https://news.ycombinator.com/item?id=1355309](https://news.ycombinator.com/item?id=1355309)

[https://news.ycombinator.com/item?id=1269090](https://news.ycombinator.com/item?id=1269090)

</comment><comment><USER8> The results of our to be published paper clearly confirm the claims of this paper and shows that if implemented well, threads can perform and scale as good as events with not much memory overhead.

I worked on this subject during my Phd, and the result is a paper that will be published in sigmetrics2020.

We developed an M:N user-level threading library and exhaustively tested it against event-based alternatives and pthread based solutions.

We used both memcached and webservers to test it on 32 core and 64 core servers.

Even connection/pthread looks promising in terms of performance.

You can find the paper and source files here: [https://cs.uwaterloo.ca/~mkarsten/papers/sigmetrics2020.html](https://cs.uwaterloo.ca/~mkarsten/papers/sigmetrics2020.html)

<comment><USER9> Well, no, you are just hacking it to support your claims. A split-stack work- stealing implementation is ok, nothing special, basically goroutines. But you are not addressing the most important difference between events and shared memory multithreading - synchronizing access to shared memory. Idiomatic event driven applications don't do synchronization and have to be sharded to scale to multiple cores. Choosing memcached and running it multithreaded is particularly bad, as memcached is not a decent event driven applications, it mixes threads and events and suffers from all that synchronization overhead. At least you should run it in one process per core configuration [1]. But it's much worse than that, there is some serious research in this area that addresses those problems, in particular the Anna paper [2], it kills any possibility for shared memory multithreading as a concurrency model to compete with anything, it's just too broken on a fundamental level.

[1] [https://github.com/scylladb/seastar/wiki/Memcached- Benchmark](https://github.com/scylladb/seastar/wiki/Memcached-Benchmark)

[2] [https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf](https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf)

<comment><USER8> I believe one should not confine event-driven only to applications that don't do synchronisation, that's part of the misconception that leads to thinking event-driven has higher performance. This is due to the fact that part of the problem with threads (as you mentioned) is synchronisation, but this is the same problem with event-driven applications. We have a web server experiment that shows comparable performance to an event-driven web server (ulib) with its various hacks to make it faster and is always on the top list on tech- empower.

Regarding memcached, the first reference you posted is from 2015, yes in 2015 memcached was in a very bad shape in terms of synchronisation and things have significantly changed from that version with locks per hash bucket rather than a global lock, avoiding try_lock and .... So those results are too old to rely on. Seastar moves network stack to user-level and if I remember correctly the scheduler consisted of multiple ever looping threads even if there was no work to do. Considering in our experiments 40-60% of the memcached overhead were coming from network I/O, there is no surprise about their results. I would call this a hack for sure.

I have not read the Anna paper to be able to comment, but it seems that they are creating a key-value store using the actor model. I briefly schemed through the paper, and I did not find anything that points to "killing any possibility for shared memory multi-threading as a concurrency model"; this is a very bold claim and if they do claim that, they should have really strong results.

But my guess is anna's whole actor model was implemented on top of threads and shared memory multi-threading? Which will be in contrast of that bold claim. I worked with actor models and implemented one as well, it is a perfect fit for many use cases but threads at least for now are the bread and butter of multicore programming.

Having said that, all these models have their respective place in the software world. What we are trying to show in our paper , through thorough experiments, is that the misconception that event-driven has higher performance than thread programming is not fundamentally true. Therefore, falling into asynchronous programming and create hard to maintain applications [1] only due to better performance has no merit.

[1] [https://cacm.acm.org/magazines/2017/4/215032-attack-of- the-k...](https://cacm.acm.org/magazines/2017/4/215032-attack-of-the-killer- microseconds/fulltext)

</comment></comment></comment><comment><USER10> Events can handle much more connections than a thread based approach. For example nginx is implemented with event-driven architecture: [https://www.nginx.com/blog/inside-nginx-how-we-designed- for-...](https://www.nginx.com/blog/inside-nginx-how-we-designed-for- performance-scale/)

> NGINX scales very well to support hundreds of thousands of connections per > worker process. Each new connection creates another file descriptor and > consumes a small amount of additional memory in the worker process. There is > very little additional overhead per connection. NGINX processes can remain > pinned to CPUs. Context switches are relatively infrequent and occur when > there is no work to be done.

> In the blocking, connection‑per‑process approach, each connection requires a > large amount of additional resources and overhead, and context switches > (swapping from one process to another) are very frequent.

This is their explanation on events vs threading approach. Still, a lot of web servers today use a thread-per-connection which is acceptable since a database(e.g. postgres) performance degrades slowly as more active connections are introduced.[1]

[1] [https://brandur.org/postgres-connections](https://brandur.org/postgres- connections)

</comment><comment><USER11> I guess it depends.

I guess I'm somewhat used to the asynchronous I/O model that things like Win16 and JavaScript use. You have to think a bit more about the unpredictable order that things happen in, but not about preemption race conditions and data corruption.

The only threading model I have seen that I care for is Erlang's. The C ... Java model is a data corruption death trap.

At that, Erlang kind of straddles the gap between independent execution and communicating sequential processes.

<comment><USER11> Oh wait, the article is about performance, not safety. Real programmers don't care about reliability, only if it's performant at interweb scale.

Meh.

</comment></comment><comment><USER12> and then there was that just two days ago: [https://news.ycombinator.com/item?id=22165193](https://news.ycombinator.com/item?id=22165193)

<comment><USER13> Haha I think that's the point of OP's post IMO, and people were even discussing this dichotomy in that thread as well.

Total gold, nothing in this world is clean or definitive.

<comment><AUTHOR> Yes :). I do not actually have a strong opinion on the Event/Threads/Coroutines debate. I think they should all be considered on a case by case basis.

<comment><USER14> I'll restate the same thing I commented on that post:

[https://news.ycombinator.com/item?id=22174201](https://news.ycombinator.com/item?id=22174201)

People only now start to understand that the problem with multi-core is memory speed and the only way to solve that is by using a virtual machine with a concurrent capable memory model = Java.

In-lined memory does not play well with multiple threads. Cache-misses is the way to parallelize code!

</comment></comment></comment></comment><comment><USER15> Threads and events are duals. So it is possible to do both well if, in the threads case, the implementation is done right. The author makes the case he's done implementations well. The salient question then is: can the average programmer do threads based concurrent programming well? As dual, I think we can all agree that more effort/intelligence will lead to better solutions for threads or events.

</comment><comment><USER16> This paper is a followup to:

* J. K. Ousterhout. "Why Threads Are A Bad Idea (for mostpurposes)". ([http://web.stanford.edu/~ouster/cgi-bin/papers/threads.pdf](http://web.stanford.edu/~ouster/cgi-bin/papers/threads.pdf))

* V. S. Pai, P. Druschel, and W. Zwaenepoel. "Flash: An Efficientand Portable Web Server". ([https://www.usenix.org/legacy/events/usenix99/full_papers/pa...](https://www.usenix.org/legacy/events/usenix99/full_papers/pai/pai.pdf))

* M. Welsh, D. E. Culler, and E. A. Brewer. "SEDA: An architecturefor well-conditioned, scalable Internet services". ([http://www.sosp.org/2001/papers/welsh.pdf](http://www.sosp.org/2001/papers/welsh.pdf))

Those are some of the more influential papers in the development of event- driven designs.

</comment><comment><USER17> Circa the time that paper was written there was a lot of talk about "single- process" web servers. I was working on a web site from which people downloaded files and the founder was concerned that Apache used too much memory for that kind of request (we had many dial-up users) and we tried a few of the open source event-based web servers and at the time it seemed that many of them would corrupt data -- we would get all sorts of reports about it from users.

</comment><comment><USER18> Didn't get past the opening paragraph. Sure, this is from 2003 but they're wrong. Look at redis.

Thread-per-connection is just plain dumb. Threads aren't free. Context- switching is a thing. Modern advice is to try not have more threads than cores. Thread-local storage is a very useful, having 1000's of thread may make TLS unfeasible.

</comment><comment><AUTHOR> The paper is arguing in favour of userspace threads or M:N threading, which were fairly popular in the late '90s, before falling out of favour. Then got popular again with go.

</comment><comment><USER19> NodeJS seems to be doing alright with its event loop.

</comment><comment><USER20> I might add the caveat from the title to this HN submission: "(for high- concurrency servers)"

And mention 2003 :)

<comment><USER7> Added. Thanks!

</comment></comment><comment><USER21> No, events are actually a spectacular idea.</comment>


USPS Secure Voting System Patent Application - <AUTHOR> https://pdfaiw.uspto.gov/.aiw?docid=20200258338&SectionNum=1 <comment><USER2> Has blockchain based voting been tried anywhere else in the world? This is a very neat idea but would it's application even make it to this year's general elections?

With the attacks USPS is under, I hope something like this comes to life.</comment>


Qualcomm Now Sampling Next-Gen 7nm Snapdragon SoC For 2019 Android Smartphones - <AUTHOR> https://hothardware.com/news/qualcomm-samples-7nm-next-gen-snapdragon-soc <comment><USER2> Several people are about to remind us that node measures are just marketing. Having got that out of the way, is Intel behind or still ahead? I understand their 14nm process has been improving over the years, but with the Intel 10nm so long delayed, have other fabs taken the lead with their 7nm processes?

<comment><USER3> All that can really be said is that Intel is no longer substantially ahead.

It's really hard to compare processes, but Intel 10nm and TSMC 7nm are roughly comparable, with _perhaps_ a _very_ slight edge to TSMC 7nm. OTOH, Intel 10nm is shipping in low volume but TSMC is still sampling.

</comment></comment><comment><USER4> I'm no expert on this with nodes etc, and I know it's really hard stuff, but is Intel dropping the ball here? Considering they apparently have trouble getting 10nm to work reliably?

<comment><USER5> Yes. "Cannon Lake was initially expected to be released in 2016" but so far all we have is low-power "U" laptop CPUs in very limited quantities.</comment></comment>


Barack Obama knows his sort algorithms... jump to 6:40 in this video - <AUTHOR> http://www.youtube.com/watch?v=1nnj7r1wCD4

<comment><USER2> How appropriate that a topic of sorting algorithms comes up in a discussion about politics. Both spheres of discourse contain some of the most useless debate about solved problems.

</comment><comment><USER3> A question to any Americans:

What are your thoughts on Obama? From an outsiders perspective, he seems like the ideal candidate. He is charismatic, intelligent, and quick thinking. He also seems to have the 'right' approach on every subject he speaks about. How is someone like this not going to win by landslide?

On a side note, how are people like Rudy even remotely considered a candidate?

<comment><USER4> Please, let's not start talking about politics here...

<comment><USER3> Sorry, didn't realize this was against the rules...

I was just talking about a subject related to the video posted.

</comment></comment></comment><comment><USER5> So, what's the right answer, guys? Quicksort? Radix?

<comment><USER6> I don't think you can recommend Quicksort unless you know something about the distribution of the elements to be sorted, or am I mistaken. After all, Quicksort can be very slow under certain conditions.

<comment><USER5> Quicksort's worst case involves bad pivot picking. The randomized version pretty much makes that a non-issue unless you're incredibly unlucky. Even still, with median-of-medians you can guarantee a deterministic worst case of _nlog(n)_ , so that's not true.</comment></comment></comment>


A Short History of the BTree (2011) - <AUTHOR> https://www.perforce.com/blog/110928/short-history-btree <comment><USER2> CouchDB uses B-trees in an interesting append-only fashion:

[http://guide.couchdb.org/draft/btree.html](http://guide.couchdb.org/draft/btree.html)

That makes it work well for reliability and data durability. I have never had it corrupt data over the years. Can always pull the plug on the server (or kill -9 the process) knowing it will recover to a consistent state, or you can also take filesystem snapshot at any time and get a constent database image (I've used that for backups).

</comment><comment><USER3> Good article. As a side note:

"and you know how high-performing and scalable Perforce servers are"

Sure, their performance is only surpassed by that of the perforce client.

</comment><comment><USER4> Very interesting and links are gems.</comment>


Deep Learning Is Easy – Learn Something Harder - <AUTHOR> http://www.inference.vc/deep-learning-is-easy/ <comment><USER2> I think the best part of this article is the approach to learning new things he presents--going beneath the surface, learning the fundamentals below the "API" layer (abstractly speaking). Really great explanation of that insight!

The part I am having trouble with, with ~5 years of real world data science experience in industry, is the _implicit assumption that we all need or want to become good at deep learning._

In my experience, most businesses, at best, are still struggling with _overfitting logistic regression in Excel_ let alone implementing/integrating it with a production code base. And we all know that toy ML models that sit on laptops create ZERO value beyond fodder for Board presentations or moving the CMO's agenda forward.

The fact of the matter is that the vast majority of businesses, with respect to statistics/ML, aren't doing super duper basic shit (like a random forest microservice that scores some sort of transaction) that might increase some metric 10%. This is due to lack of sophisticated analytics infrastructure/bureaucracy/ lack of talent/ being too scared of statistics. Ultimately when you're rolling out a machine learning product internally (I'm not talking about Azure/ other aws-model-training-as-a-service type things), the hardest part isn't: "We need to increase our accuracy by 2% by using Restricted Convolutionallly Recurrent Bayesian Machines!" The hardest part is _convincing people you need to integrate a new process into a "production" workflow, and then maintaining that process._

<comment><USER3> Totally agree! If I had a dollar bill every time I had a senior exec (VP- level) for large orgs ask me what 'probability' means, I wouldn't need to be a data scientist anymore! The issue is deeper than what you write I think. There is technical debt of course in implementing complex ML/DS pipelines that should be compensated by the increase in lift/revenue. Outside of large companies like Google/FB/Apple etc who have incorporated ML in their products, many outfits that want to use their 'data' to 'address business problems' don't really need sophisticated ML or can justify the technical and human debt. Having worked in the industry as a data scientist, I'm not too hopeful about the prospects of many DS-as-a-service companies, not because there isn't solid technical content to offer, but their clients are routinely idiots.

<comment><USER4> Are you saying everyone is an idiot, including the customers of data science companies, or that people who pay for data science are especially idiotic?

</comment></comment></comment><comment><USER5> I think the article overestimates how much we understand about neural networks, and the novel ways we might use them.

Take word2vec for example. It's a two-layer (i.e. not deep) neural network that uses a relatively simple training algorithm. Yet, after 30 minutes it can learn impressive relationships between English words. The method is only 3 years old and businesses continue to find novel ways to apply word2vec. Check out this article:

[http://multithreaded.stitchfix.com/blog/2015/03/11/word- is-w...](http://multithreaded.stitchfix.com/blog/2015/03/11/word-is-worth-a- thousand-vectors/)

They are using word2vec to find clothing items that resemble other products by adding or subtracting descriptors such as "stripes", "blue", or even "pregnant."

For my company, we are attempting to normalize medical terminology using word2vec. Here are a couple examples:

1\. "myocardial" \+ "infarction" = "MI"

2\. "ring" \+ "finger" = "fourth" \+ "metacarpal"

We never would have thought to normalize the names of different body parts, but with word2vec, it does it for us. There's a great deal yet to learn about how this model alone works, let alone deeper networks.

<comment><USER6> I think I have an intuitive explanation why word2vec works. I tried to write it down, but I'm lazy.

Don't think of it as a 2 layer neural network. what it really is is a physics system. like a universe where each word is a planet. when words appear in the same context, that will assign some gravitational pull to the "planets". eventually, words with with same meaning will clustered as a galaxy.

for example, look at the following 2 sentences:

1\. I pat a dog.

2\. I pat a cat.

dog and cat, they appear in the same context. when the first time we see the word dog, we calculate center of mass of "I", "pat" and "a". Think of them as little planets.

then you give the planet "dog" a little force to pull it to the center of mass of the previous three words.

when you see the word cat, you do the same as it's in the same context.

Eventually, cat and dog will be clustered and form a galaxy, with other similar words. and the galaxy will be called "pet".

<comment><USER7> I don't know that they physics analogy works so well, or at least, it's definitely missing something. What prevents the whole word "universe" from collapsing on itself, forming a black hole? That is, if there are only attractive forces, the global optimum is to co-locate everything in the same point, which doesn't give you a useful model. There needs to be something in the model that keeps different words apart from each other.

This page has the clearest explanation of word embeddings and the relationship between the objective function and why vector translation captures meaning.

[http://nlp.stanford.edu/projects/glove/](http://nlp.stanford.edu/projects/glove/)

<comment><USER6> it works because the gravity of word2vec isn't the gravity of real life.

notice that I only pull the word "dog" to the center of gravity of the rest words, instead of pulling all of them together. I think the full version even push the rest of the words away from the center of gravity.

but I need to double check the math.

this is not just an analogy. this is what word2vec's math says.

the only analogy part is that word2vec is in high dimension, my analogy is in 3 dimension.

</comment></comment></comment></comment><comment><USER8> This article would be much better if it would specify whom is it addressed to.

My wife works in internet advertising, managing AdSense campaigns and stuff. The fact that she's able to write simple scripts in Javascript to automate some of her tasks already puts her ahead of her colleagues. Her whole company doesn't seem to have even a freshman-level of understanding of mathematical statistics, judging by how they operate and organize things internally. (Unfortunately, she has no power to change that). A competitor that would have just these two obvious things would already easily surpass them — but for now, they are bring more profit than a lot of SF-based startups could hope for.

Big data, deep learning? This stuff may be "obvious" for a computer scientist or a competent developer, but it sure as hell is not "obvious" to a lot of real world businesses out there, and there's a lot of value to be created by applying these things.

<comment><USER9> I interpreted the post as saying, "Don't try to get a Ph.D in deep learning, and don't learn it for it's own sake -- have an application in mind."

<comment><USER10> Adding to other commenter, he says that learning the recent tech will just give you a temporary advantage as a first-mover followed by all kinds of people having same skills. That's misleading as a first-mover will have many successful projects under their belt and a network of referrals if they played it smart. That advantage will be a lasting, selling point vs cheap labor of "I know the dark corners and time-wasters. I'll do it right."

<comment><USER11> >That advantage will be a lasting, selling point vs cheap labor of "I know the dark corners and time-wasters. I'll do it right."<

This very much still holds true...excellent point...

I've been in the profession for a number of years, paid attention to detail, and put all my efforts into nailing the skills I've observed to be relevant...

Much to my amusement, I learned a few years ago that some in my network of friends had given me the nickname of "the fixer"...which led to more referrals than I could possibly handle...

I know what I know, and if you call me in I may take a bit more time than you'd like to fix your problem(s), but when I'm done it will be "fixed"....

That's still worth quite a bit, even though the pace has picked up dramatically over the years...your "core" needs to be rock solid for long-term viability...

My advice to beginners is master something, then master something else...before long your reputation will precede you in ways that you'll be delighted by...

<comment><USER10> Glad to see you're benefiting from the effect. And this...

"My advice to beginners is master something, then master something else...before long your reputation will precede you in ways that you'll be delighted by..."

...is totally true. It's why I tried to wear many hats. At some point, wearing too many makes you look like you might not be that good at any one. So, the next trick I learned is to make different versions of a resume that leave off stuff and target specific audiences. Each one might think you only have 2-3 skills. High skill or mastery in them is still believable at that point. So, let them believe. :)

<comment><USER11> Agreed...targeted resumes are the way to go, especially with the current differentiated market, which is accelerating geometrically...

I'm full stack in 3, often 4 (depending), discreet environments...that took years, and (still takes) 50-some-odd active bookmarks and rss feeds...my traveling "hotshot" case has 34 thumbs loaded with goodies...a good memory helps a great deal...

<comment><USER10> You got me thinking. A friend several years back got a small box full of 60+MB thumb drives for almost nothing. We were thinking what could be done with those in terms of attacks, distribution of sample software, various ideas. Your comment makes me wonder if they could also be used as business cards of sorts showcasing a portfolio w/ code. Is that was you were referring to or something else like your software stacks for various roles you take on?

Regardless, I never considered that use case.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> That's a little bit like saying: don't learn web development, it's too easy, invest in learning fundamentals of computer science instead.

Sure, if you want to innovate in the field of academic AI, you probably need to know much more than how to throw some data at a neural net. But there are SO MANY problems supervised learning can solve out there - much like there are so many people and businesses who need a website done.

It's a tool, like any other - and right now it seems very hard to beat. So please go ahead, learn how to use it, and apply it to your own problem domain.

<comment><USER13> I guess one problem is that deep learning requires a different mindset than most people who are currently in computing are willing to have. I'm not a deep learning expert, but I guess that working in this field requires a lot of trial-and-error: "hmm this doesn't work, let's see what happens if we change this parameter". This is directly in contrast with the more "engineering" or "mathematical" mindset that computer science requires.

(Of course, deep learning certainly requires mathematics and engineering skills, but the driving force behind their use remains vague and difficult to grasp.)

<comment><USER14> I don't think the mental process is necessarily all that different in pure mathematics. The process of creating new structures and proofs in mathematics definitely involves a lot of trial-and-error and vague intuition. When the evidence mounts that a particular approach is likely to succeed, more effort goes into refining the intuitive approach into a valid proof. Further iterations often build on the original proof by tweaking the parameters to make it more general or change the context, just to see what happens.

<comment><USER13> You have a good point. But at least in pure mathematics, you end up with a rigorous proof, whereas in deep learning, you end up with a system that may- or-may-not work.

This means that in pure mathematics you can build on your previous results, but in deep learning you cannot (without danger).

<comment><USER15> Rigorous proofs are a formality that gets less attention than you'd guess at first.

</comment></comment></comment></comment></comment><comment><USER16> My takeaway is something I've observed and commented about for some time: understanding how any important/relevant technology works gives you an unbeatable edge over most of your peers. Andrew Ng makes a similar comment about 2/3 of the way through his Coursera course on machine learning.

With respect to deep learning, most of the work I see involves pulling existing networks off of the Caffe Zoo and retraining them with proprietary datasets or fiddling with Theano to reproduce the works of others. These efforts mostly end in failure, usually after a blind hyperparameter search.

The easiest way to separate the winners from the losers in deep learning and elsewhere is to ask them to derive the method they are using and explain what it's doing. For the most part, I find the above people are incapable of doing so for backpropagation. Bonus points if they can write (relatively) efficient C++/CUDA code to implement backpropagation from the ground up.

Edit the above paragraph for PCA, ICA, GP, GMM, random forests or any other technology/algorithm. If one has a decent grasp of the underlying logic, the instincts for adapting it to new domains/datasets come at no additional charge.

<comment><USER17> Can you give some anecdotes of where "understanding how any important/relevant technology works gives you an unbeatable edge over most of your peers"? Besides being able to pass a job interview. Naively I would assume that being able to apply these techniques (understanding cross validation, overfitting, domain expertise, etc) would be far more important than knowing the underlying logic of how to write them from scratch.

<comment><USER18> I think that it's way easier to apply something when you understand how that something _actually_ works.

i.e. There is a difference between being able to re-implement an algorithm, and understanding why that algorithm has the behavior it has. You'll probably write cleaner and more maintainable code, quicker, in the latter, and be able to make modifications as necessary.

<comment><USER15> That's an easy claim, but putting more time into learning foundations means you have less time for applications. Moth pure mathematicians are poor web developers.

<comment><USER16> Quality over quantity: sounds like a winning strategy to me.

</comment></comment></comment></comment></comment><comment><USER19> Sounds like he's viewing things very much from a research-scientist perspective. From that point of view, sure, a lot of the groundbreaking work has already been done. From the point of view of someone who just has a job to do, though, it's a tool in the toolbox. If it's the most appropriate tool then you'd be stupid not to use it.

</comment><comment><USER20> I'm not in CS, just an enthusiastic person using software to solve his own projects. I personally want to learn more of these that make "hard thing easy", not to come up with new methods, but to apply these now-achievable techniques to different problems, the problems around me. "learning deep learning should not be the goal but a side effect" sounds useful in general, in the same time I wish there would be more, wider set of people actually using (as opposed to merely learning) the awesome methods and technology we have these days.

</comment><comment><USER21> I don't know. If you go that route then it's just: Study mathematics or theoretical physics and then learn everything you need.

The problem is pulling that off. Sure if you are clever and pursue a PhD it's worth going the abstract theoretical route but in my personal experience as a somewhat stupid student it's fine to look for a niche and get hands-on experience on that.

Mastering Hadoop is still far from easy and if you need to use it without burning tons of CPU cycles it's even harder to get right.

<comment><USER22> I am interested in this, mostly as a computational biologist. Any intro on Hadoop? What is it used for?

<comment><USER21> Hadoop itself consists of different parts: HDFS is a distributed filesystem that can span across lot's of machines and stores data in blobs of varying sizes.

MapReduce is the Google idea from before 2004 how to do calculations on lot's of data. Now there is also YARN that could be described as a general job scheduler.

At the moment a lot of people use software on YARK like Spark (does more in memory, is faster, can use the GPU on the cluster machines).

So if you have biological data you could feed that into a HDFS and have Spark or MapReduce jobs that process the data. The clue about Hadoop is that don't need to care about getting the clustering and distributed setup right. This is done by Hadoop for you. You program like you would program a single thread algorithm (at least in the simple cases).

E.g. this Python code counts words over as much data as you want in parallel: [http://www.michael-noll.com/tutorials/writing-an-hadoop- mapr...](http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce- program-in-python/)

If you google each of these projects you'll find a lot of information.

Here are the original papers that should give a good idea:

\- HFDS (Google Filesystem was the original idea - Hadoop was a free implementation by Yahoo - [http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en//archive/gfs- sosp2003.pdf)

\- MapReduce - [http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce- osdi04.pdf)

\- Spark: [http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark...](http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf)

If you want a good introduction read Hadoop: The Definitive Guide, 3rd Edition

We used it for building a search engine out of multiple terabytes of crawl data - something that fit's not good on a single computer.

You can do all kinds of computations but graph problems or things that not good to parallelize require often other solutions beyond MapReduce or more thought - MapReduce also only fits for a certain class of problems - it's great for counting and aggregation of stuff but beyond that it's often not usable.

</comment></comment></comment><comment><USER13> Also, it is boring, because 90% of the time, you will be waiting for your networks to be trained :)

</comment><comment><USER23> In short, don't go for it because it is a fad. Over time, like the _data scientist_ buzzword, this too shall become something everyone can do. It is what you can do with Deep Learning that matters, and where you can push beyond this.

</comment><comment><USER24> There's a bit of optimism about technology adoption here; I'm guessing Ferenc is part of a bleeding edge research community from how he talks about the ease of uptake and adoption for new technologies.

The reality is there are still companies looking for their first 'big data' data scientist years after this was a 'hot area'. While aggressive researchers, then aggressive companies, then fast follower researches / companies, then ... all do eventually adopt useful technology, it can take decades in some cases.

So, do learn deep learning. But from my own experience, he's right in that you'll think "that's it?" once you've constructed a few networks and read through some basic 2014+ literature on tuning and tips. But, that will put you ahead of the VAST majority of technologists inre: deep learning. Seriously.

I think he's 100% right that the search for usefulness / novel applications is where the game is going to be for deep networks, but it's seems silly to tell people not to learn how to work with them.

</comment><comment><USER25> What about reinforcement learning? It seems neglected to me. There are only one or two good reinforcement learning resources I'm aware of compared to the hundreds of books and blogs on supervised learning. Most machine learning books will mention supervised, unsupervised, and reinforcement learning in the first chapter, then never mention reinforcement learning again. This is my experience at least, and I've read a few introductory machine learning books.

Reinforcement learning can drive you car and mow your lawn. Supervised learning can tell you which of two pictures is a cat. Granted, supervised learning techniques can have an important place in the broader framework of reinforcement learning.

Reinforcement learning is one of the harder problems.

<comment><USER26> The hard thing here is reinforcing the right thing. I did a bunch of work on genetic programming and RL and it worked... uhh... too well. Once it succeeded in learning, near as I could tell, my hard disk geometry. There was a subtle timing difference between data sets and this seemed related to where they were on disk. The literature is loaded with anecdotes like that, some even more hilarious.

But you're right. Nature clearly does learn this way, at least for many things. Maybe the way forward is to combine it with supervised and unsupervised learning and let different forms of learning work together.

<comment><USER27> This kind of unintended learning is something that all machine learning practitioners should be careful with. The example of the satellite photo analysis algorithm that turned out to have learned to classify difference between night and day photos comes to mind.

This means that your particular reinforcement learning example needs something similar to regularization of cost functions or cross validation for checking for over or under fitting to correct these latency effects.

It may be cheaper and more effective to use SSDs instead but the magnitude of the latency effects on your case are not clear either to determine if your algorithm would benefit substantially from a modern SSD.

<comment><USER26> The fix was to (1) take steps to make sure the image was entirely pre-loaded into RAM before the test was given, and (2) randomize the latency of the load. That worked and I actually got some real results, which unfortunately were decent but not as impressive as the cheating was. :) When it cheated the thing got 100% right of course.

I also learned that rapid convergence on incredible accuracy is suspect.</comment></comment></comment></comment>


Stellar.js - Parallax has never been easier - <AUTHOR> http://markdalgleish.com/projects/stellar.js/

<comment><USER2> It'd be nice if my scroll wheel still worked on this site.</comment>


How to Break Free from Email Jail - Daniel Markovitz - <AUTHOR> http://blogs.hbr.org/cs/2012/08/how_to_break_free_from_email_j.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+harvardbusiness+%28HBR.org%29 <comment><USER2> _Many people are already doing this in their private lives: they use RSS feeds, webpage bookmarks, Instapaper, etc. to consume information when they're ready for it — they "pull" it when they want, rather than have it pushed on them. Pulling information allows them to smooth the flow of information they receive. They get to drink from a water fountain rather than a fire hose._

These services are creating a buffer between the existing push systems and the desired pull system - content is pushed here and cached, then pulled when needed. So the key to this is having a bucket to which you can send everything, and which you can access on demand. For me, my pocket computing device (aka "phone") is my communications bucket.</comment>