How we recovered $300k of Bitcoin - <AUTHOR> https://reperiendi.wordpress.com/2020/04/03/how-i-recovered-over-300k-of-bitcoin/ <comment><USER2> "I'm currently looking for work in a staff or senior staff engineering or data scientist role. If you've got interesting technical analysis or optimization problems, please reach out to me and let's talk."

That is probably the best CV I've ever read by accident.

You should broaden your job hunt criteria to technical content marketing - seriously!

Very rare to possess the skill to tell a story in an entertaining, approachable and detailed technical fashion.

<comment><USER3> Hahaha, thanks!

<comment><USER4> > If you've got interesting technical analysis...

FYI, not sure if english is your first language or not but this should be "If you have interesting technical analysis..."

<comment><USER5> "If you've got" may not be formal "correct" English, but everyone understands it and it's used quite commonly in daily speech (and writing). I would wager that "if you've got" may actually be preferable to "if you have," in order to engage the reader at a more comfortable, personal level.

<comment><USER6> FWIW "you've got" seems very US American to me (native en-gb).

<comment><USER7> "You've got mail!"

</comment></comment></comment></comment></comment></comment><comment><USER8> The author is a very talented applied cryptographer with a very impressive resume (he is looking for new projects).

The following CV line stands out however:

Google: Software Engineer, Ads Review. June 2014– March 2016.

Angular / Java developer on the internal tool used by contractors to review Google ads for policy violations.

How did that saying about "brightest minds working on ads" go?

I am not blaming the author as I would have done the same(and I imagine author was not told when hired that he'd be working on Java/Angular ad tool).

Again it is not that Java or Angular are bad per se, but working on ad CRUD seems completely orthogonal to author's talents.

<comment><USER9> > _How did that saying about "brightest minds working on ads" go?_

I think it goes something like the Open Source saying "that person doesn't owe anyone their intelligence to work on so-called important problems for free, just because it might be nice if they did".

Want smart people to solve important problems? Find a way to pay more for that, than for ads, or find a way they don't need to earn money to live at the standard they want.

<comment><USER10> Never going to happen. Look at other important jobs - doctors and nurses can have very low wages, while professional footballers earn millions. We all know which one is more important, but there's no way the medics will ever get paid comparably.

<comment><USER11> Some top doctors can earn at least nba coach level figures. Frontline workers testing people will never make that amount because the skill level is lower so more people can do the work. If everyone refused and only a handful of people were left they couldn't reproduce the same impact anyhow and the program would close instead of paying them millions. Together they have a big impact and one on one they have a big impact but a single first responder doesn't have the attention or connection of millions. Would you put a poster of Jill, Nurse 56 from Grandrapids who works in the covid wing cleaning sheets? Probably not, even though the job is extremely important.

<comment><USER6> >Some top doctors can earn at least nba coach level figures //

Ethically?

<comment><USER11> Sure everything from plastic surgeons to heart specialists. Not everyone has to be a doctor to the stars or go on shark tank to make millions. Does your local walkin clinic doctor make that? No, but would make 500k before expensives.

</comment></comment></comment></comment></comment></comment><comment><USER12> "Until a man is twenty-five, he still thinks, every so often, that under the right circumstances he could be the baddest motherfucker in the world. If I moved to a martial-arts monastery in China and studied real hard for ten years. if my family was wiped out by Colombian drug dealers and I swore myself to revenge. If I got a fatal disease, had one year to live, devoted it to wiping out street crime. If I just dropped out and devoted my life to being bad. Hiro used to feel that way, too, but then he ran into Raven. In a way, this is liberating. He no longer has to worry about trying to be the baddest motherfucker in the world. The position is taken."

<comment><USER13> Thanks for this quote. I've added Snow Crash to my want-to-read list.

</comment></comment><comment><USER14> > Recovering the key was usually instantaneous, but to help people feel like > they'd gotten their money's worth, we'd put on a little animated show like a > Hollywood hacking scene with lots of random characters that gradually > revealed the right password.

and later ...

> I'm currently looking for work in a staff or senior staff engineering or > data scientist role. If you've got interesting technical analysis or > optimization problems, please reach out to me and let's talk.

I can't help but wonder if this write-up (which is fascinating) may not be one of those little animated shows to help propspective employers feel like they'll get their money's worth.

<comment><USER15> That's an incredibly cynical take.

<comment><USER16> Right? Jesus...

</comment></comment></comment><comment><USER9> @metaweta: The technical side is interesting, but seems like the admin side would be too; how did the administration and contract side play out?

Someone needed to front an estimated $100k of GPU costs, without being sure it would work - and then pay for your work on top; who risked that? You had no proof the claimed Bitcoins inside were real, or as many as claimed. You're in New Zealand(?) and the customer is in Russia - you need the file to study it and if you crack it then you have the Bitcoins as well - how did they become comfortable that you wouldn't steal them and say you couldn't crack it? Did it worry you that the owner might not be able to convert them to cash, e.g. if the Exchange was shady and there was very little recourse?

How much work did it take to convince your partner to stop what they were doing, and write GPU code for a crack which might not work?

<comment><USER3> I'm in the USA. He paid us for some work up front and agreed to pay the rest on delivery of the key. He expected to spend most of the $100K he'd budgeted on GPU costs, so we got much less than that for the work; we took the job because it sounded like fun. We didn't need the information in the archived files, just the encryption headers, so he set most of the bytes to zero. I couldn't have spent the coins even if I wanted to.

<comment><USER9> Thank you, that is interesting; he's wealthy enough and technically capable enough to make that all go a lot easier. (I guessed from going back to check on the process ID on his laptop that you must have had his laptop, and didn't know you could do that with just headers).

</comment></comment></comment><comment><USER17> Not only are the cryptography skills of Mr. Mike Stay is obviously impressive, his presentation of things which happened 20 years ago in a vivid yet subtle manner seems extraordinary to me.

Especially since, I can't remember what I did 2 weeks back to write in my blog.

Can you give us the secret of your documentation/notes workflow Mr. Mike (@metaweta)? Please don't say that you recalled from your memory!

<comment><USER3> I kept a Cryptonomicon!

</comment></comment><comment><USER18> I always assumed that the encryption used on zip files was relatively trivial and could be broken given the right software and hardware.

Well, it looks like it's a lot harder than I thought. You still need the right software and hardware but you also need the right person to do it.

Most of what I read was above my understanding but it was good reading anyhow. Good job on the recovery and good job writing about it.

<comment><USER19> You're not alone, wikipedia for zip says

> ZIP supports a simple password-based symmetric encryption system generally > known as ZipCrypto. It is documented in the ZIP specification, and known to > be seriously flawed. In particular, it is vulnerable to known-plaintext > attacks, which are in some cases made worse by poor implementations of > random-number generators. > [https://en.wikipedia.org/wiki/Zip_(file_format)#Encryption](https://en.wikipedia.org/wiki/Zip_\(file_format\)#Encryption)

IMO the fact that the author was able to recover the password at all indicates weakness. Something encrypted with AES-GCM would presumably be all but impossible.

</comment></comment><comment><USER20> This may be the plutonic ideal of a Hacker News story. It's also sort of a Travis McGee story but the salvage consultant is a cryptographer.

<comment><USER21> It got me with SoftIce. Although I never thought to print them out; I needed to watch all the registers change as I was doing similar work or reverse engineering encryption schemes.

<comment><USER22> I did not have a printer at that time. Notepad with a lot of addresses and registers' values... Also, did not realize one can print from the SoftIce directly, as I assumed it run at the lowest level before all the printer drivers. Certainly, it would be super useful!

</comment></comment></comment><comment><USER23> I've always wondered what the development process looks like for these type of algorithms. If you have to run the program for a year to know if it will work, how can you have any confidence that what you've written is going to do the trick?

<comment><USER24> Perhaps you try it on smaller, "test" data to see if it works?

<comment><USER3> Exactly. We created some zip files we knew the password to and then checked that our code found the right one. Each stage would generate a bunch of files with different candidate ranges, so when testing the next stage, we'd choose the one file we knew had the correct key in it.

</comment></comment></comment><comment><USER25> Does anyone have links to stories that are as good as this? I want to binge blog posts that teach and entertain at the same time.

<comment><USER26> I've been learning Rust. Reading "The Rust Book" and "Rust By Example" gave me a similar feeling as reading this blog post.

</comment></comment><comment><USER27> I kept expecting to read that the password on the zip file turned out to be "password123" or something like that.

<comment><USER28> I want to know the password too. If it was machine generated looking at that algo might have been useful.

</comment></comment><comment><USER29> NEARProtocol.com is hiring. Please have a look. [https://nearprotocol.com/careers/#openings](https://nearprotocol.com/careers/#openings) and I can connect you with our recruiting team. Thanks :)

<comment><USER3> Looking, thanks!

<comment><USER30> You are smart beyond my comprehension. Your article was amazing.

Although blockchain technology fits your skillset, I hope you will choose to do something [actually|more] worthwhile to our societies.

</comment></comment></comment><comment><USER31> "I knew of one place that ran the software for nine months before finally getting in."

And what a beautiful new baby password crack it was.

</comment><comment><USER32> Attacking the wetware would have been worthy try :) Would hypnosis work to recover the forgotten password?

<comment><USER3> He actually tried that without success.

<comment> ur-whale Any reason why a dictionary based attack wasn't tried first?

Or did your client remember selecting something really hard in the first place?

<comment><USER3> Yeah, he knew it was a long passphrase.

</comment></comment></comment></comment><comment><USER33> Collected the biggest archive of Bitcoin wallet.dat files with balance and lost passwords. [https://allprivatekeys.com/wallet.dat](https://allprivatekeys.com/wallet.dat) The collection consists of 32 files total for 2500+ BTC. The biggest wallet 576 BTC, the most interesting wallet.dat files with pre-mined coins from 2009-2010. Let's try other wallets for a share?

</comment><comment><USER34> Great story and impressive work at Pyrofex. Can you share your perspective on the crypto industry right now?

<comment><USER3> There's no way that the AccessData job I had would exist today. Most services are online, with data encrypted in transit and stored in the cloud. TLS security has improved dramatically over the last decade in response to attacks like BEAST and BREACH and CRIME and POODLE. Google drops (? is going to drop?) your SEO ranking if you don't have proper certs installed. Nowadays, it's rare to find an attack on the connection from the browser to the server. Instead, it's either malware on the client or hacking into the servers, where the operators have terrible security practices like storing data unencrypted at rest.

The field of cryptography has grown tremendously, and there's still a lot of research being done. There have been many competitions for developing strong cryptographic primitives. There's a lot of work being done on zero knowledge proofs and verifiable computation. Cryptocurrency has encouraged lots of bright young minds to get involved.

One thing we learned from the Snowden revelations is that crypto works where it's applied, so every little bit of crypto helps. Run a Tor node if you can.

<comment><USER34> Thanks. Is an implementation of the Casanova and c delta blockchain papers in the works? I don't see it on GitHub.

</comment></comment></comment><comment><USER35> The technical proficiency on display here makes me swoon.

</comment><comment><USER36> No one said TLDR this article, ever. This account has all the components of great book. It reminds me of 'The Soul of a New Machine'.

</comment><comment><USER37> How off was the 100k compute time estimation?

<comment><USER3> I think he ended up only spending around $5-10K on compute. I'd have to check with Nash.

</comment></comment><comment><USER38> What am amazingly fun read. Beyond the obvious technical skills, OP is a great writer.

</comment><comment><USER39> Great story. I just wish the title was in BTC rather than USD.

<comment><USER40> The story would be less thrilling, I think: someone bought around 35 BTC, and then later he desperately wanted access to them, because they were worth 35 BTC!

<comment><USER3> lol yeah

</comment></comment></comment><comment><USER41> Resume of the decade!

</comment><comment><USER42> excellent story. can you share the number of bitcoins that were retrieved and how the customer lost the password in the first place?

<comment><USER24> > Back in January of 2016, he had bought around $10K or $15K of Bitcoin and > put the keys in an encrypted zip file.

Based on the price of Bitcoin back then, it seems like this was about 30-40 Bitcoin?

<comment><USER43> Was the computer stolen or did he actually forget the password?

<comment><USER44> From TFA:

> Luckily, he still had the original laptop and knew exactly when the > encryption took place.

</comment></comment></comment></comment><comment><USER45> This is such a cool story. Thank you, Mike!

</comment><comment><USER46> That was an amazing read!

</comment><comment><USER47> This is pretty hardcore

</comment><comment><USER48> What a great story - a surefire cure for Dunning-Kruger. I'm in awe of this guy.

</comment><comment><USER49> How could you be sure you didn't just help heist someone's bitcoin that this Russian snatched with some malware?

<comment><USER50> > Luckily, he still had the original laptop and knew exactly when the > encryption took place.

<comment><USER49> But how do you prove that? It's trivial to replicate the hardware the file was lifted from if the malware grabbed the standard sys info. File create/modify times could also give you a pretty damn good guess as to when the encryption happened.

</comment></comment></comment><comment><USER51> I don't know why, but these headlines disgust me. There is no insight. Just. Headline of greed.

<comment><USER52> "How we cracked a zip file" would have been pretty shitty though, right?

</comment></comment><comment><USER53> If some random Russian dude shows up and offers to pay $100k to crack something with untraceable digital currency inside, don't you worry about getting killed if you fail to do it? Maybe I watch too much TV.

<comment><USER54> This is racism.</comment></comment>


Five Memorable Books About Programming - <AUTHOR> http://prog21.dadgum.com/19.html

<comment><USER2> Hmmm... my own list of memorable programming books would look like (in no specific order):

      1. The Mythical Man Month, Brooks
      2. Peopleware, DeMarco and Lister
      3. Paradigms of Artificial Intelligence Programming, Norvig
      4. The C Programming Language, Kernighan and Ritchie
      5. The Ruby Way, Fulton

I had a hard time deciding whether or not to include the Pragmatic Programmer, in the end I decided to go with The Ruby Way because it was an unusually enjoyable read for a language book.

I know the original list was supposed to be more esoteric books, not classics, but I'm anticipating this becoming a thread of everyone's favorites. : ))

<comment><USER3> I'm just beginning PAIP - I finally broke down and bought it when borders emailed me a 40% off coupon. CS books aren't supposed to be readable, but so far this one seems to break that rule.

<comment><USER4> I think PAIP is underrated (or maybe just unknown) because a) AIMA replaced it as the go-to AI text, b) there's no free online version (unlike SICP, PCL, OnLisp, etc), c) it's expensive. But it's probably my favorite of all the Lisp books I'm looking at now.

</comment></comment></comment><comment><USER5> Joel Spolsky's list: <http://www.joelonsoftware.com/navLinks/fog0000000262.html>

</comment><comment><USER6> I always have to plug Managing Gigabytes whenever a discussion of computer books comes up. Great reference for anyone dealing with searching or compressing large amounts of information:

[http://www.amazon.com/Managing-Gigabytes-Compressing- Multime...](http://www.amazon.com/Managing-Gigabytes-Compressing-Multimedia- Information/dp/1558605703)

</comment><comment><USER7> \- _Introduction to Algorithms_ (MIT Press). This is the only one of my computer science textbooks that I consistently refer back to. It's elegant and timeless and covers a lot of ground.

\- _Design Patterns_. Dry, boring, useful. You can skim it and get most of the ideas. The best part about it is building up a vocabulary for discussing common patterns with other software engineers.

\- _Programming Perl_. Ah, the camel book. It's possibly the only written-by- the-language-designer reference that's fun to read. Regardless of what your favorite scripting language is, you have to love Larry Wall.

\- _The Mythical Man Month_. Another classic. Easy to read. Stuff people should know, but don't.

\- _The Design of Everyday Things_. Not really a programming book strictly, but a nice introduction to the principles of usability. If you can get over the slight smugness, it can shake up the way you think a bit.

</comment><comment><USER8> 1\. _Programming Pearls_ & _More Programming Pearls_ , Jon Bentley.

2\. _SICP_ and _The Litle Schemer_.

3\. _K &R_

4\. _Find the Bug_ by Adam Barr (I read it after reading a Guy Steele recommendation for this one and it's awesome. Author is a cool hacker, writer and a PM at Microsoft (his other book _Serving my corporate Masters_ was available online. Not sure about now):-)

5\. _Hackers Delight_

</comment><comment><USER9> Here are 5 of my favorites nobody has mentioned here yet:

\- The Art of Computer Programming, Knuth

\- The Algorithm Design Manual, Skiena

\- The Practice of Programming, Kernighan and Pike

\- ACM Turing Award Lectures

\- Applied Cryptography, Schneier

</comment><comment><USER10> The best book I ever read about programming was Writing Solid Code, by Steve Macquire.

<comment><USER6> That was a great one. His malloc wrapper was a life saver, and getting into the habit of adding lots of assertions was really helpful for me.

</comment></comment><comment><USER11> I would think Design Patterns: Elements of Reusable Object-Oriented Software by the gang of four is the most famous -- I can't say I've read any books on programming that I felt were "memorable"

</comment><comment><USER12> _Expert C Programming_ by Peter van der Linden

<comment><USER3> Yeah, that's a good one. Well written, and a great book to read after you've read K&R and have been programming C for a while. It gets into the corner cases, and has a nice explanation for why certain declarations cause errors, etc.</comment></comment>


Yale Discovers a Fungus That Eats Plastic  - <AUTHOR> http://www.pcworld.com/article/249216/yale_discovers_a_fungus_that_eats_plastic.html <comment><USER3> As programmers we are always attuned to thinking about the edge cases and endgames, so I wonder:

Would be possible that such a fungus proliferated into a sort of "termite for plastic", feeding on plastic piping (in houses or cars maybe) and the like.

Of course house owners already deal with mold so I suppose this would just be another one.

The article suggests introducing it into landfills to eat the plastic. Kudzu was introduced to America to control soil erosion. The invasive vine now spreads at a rate of 150,000 acres a year, so it certainly accomplished goal A.

<comment><USER4> >The invasive vine now spreads at a rate 150,000 acres [per what]

150K acres [per year]? or per what metric.

<comment><USER5> 150,000 acres annually.

</comment></comment></comment><comment><USER6> One interesting thing that this article fails to mention is that the fungus was discovered by an undergraduate student. Jon Russell, the lead author of the paper, graduated last spring, and he originally discovered the fungus in Spring 2008. Yale offers a class that pays for students to spend Spring Break in the Ecuadorian rainforest collecting samples. The class also includes a stipend for students to continue their research projects during the summer following the course.

<comment><USER7> One of my greatest regrets from undergrad days was having graduated a year too early to enroll in this course. Scott Strobel was something of a visionary for setting it up, and it seems to have paid off with at least one cool result -- not to mention, a handful of students who now know how it feels to find something new.

I wish more professors would actively engage their students in "real" research projects, i.e. beyond the usual cookie-cutter laboratory lessons or follow-a- grad-student internships. Sure, it's more work for everyone involved, but experiences like this are the best way to convert bright students into real scientists, IMO.

</comment></comment><comment><USER8> This reminds me of the book Ringworld and the "Fall of Cities" - <USER2> civilization had become dependent on a particular superconductor, and wired it into everything. A microorganism that fed off of the superconductor came along and spread like wildfire. The side-effect was that it essentially wiped out the civilization because their entire energy infrastructure was built around this stuff.

Hopefully these guys know what they are doing :)

<comment><USER9> I was thinking the same thing. Getting a colony of this stuff established in the voids and nether regions of say a wide body jet would not be a very good thing at all.

<comment><USER10> I imagine that jets are inspected often enough for something like this to be caught. I imagine that a bigger problem would be an infection getting loose in a manufacturing factory. How much of my laptop is edible?

<comment><USER11> This fungus would still need water, so all they have to do is keep the jet dry, which they do anyway.

</comment></comment></comment></comment><comment><USER12> I'm confused by the "exclusively subsists on polyurethane" and "scientists recently found a fungus in the Amazonian rainforest"... TIL there's a bunch of polyurethane in the Amazonian rainforest?

<comment><USER13> That caught my eye, too. Rather than having evolved to eat polyurethane only, and then waiting patiently for us to arrive with dinner, I took this to be an awkward way of reiterating that the fungus is the only organism we know of that can eat PU. Which does make me wonder a bit about what it normally eats - resins?

<comment><USER14> I read it as "capable of subsisting on polyurethane only" in the lab.

</comment></comment></comment><comment><USER15> Any one know what the by-products of the fungus' metabolism are?

<comment><USER16> This seems like the most important question... Since the paper is paywalled, and science reporting is terrible, can someone with access inform us just what the hell the fungus turns the plastic into?

<comment><USER17> <http://aem.asm.org/cgi/reprint/AEM.00521-11v1.pdf>

</comment></comment></comment><comment><USER18> There are two competing methods, as far as I see: use the bacteria themselves to digest plastic, or use extract the enzyme responsible for plastic dissolution and manufacture it the enzyme responsible for plastic dissolution.

1\. Bacteria:

      - Can they adapt to the climate and the ecology of their target environment?
      - Can they adapt, in particular, to oceanic salt water to dissolve dumped plastic waste?
      - Overpopulation: what are the consequences? Do natural predators of this bacteria exist?
      - Underpopulation: can the bacteria be genetically modified to survive in landfills or oceans? Must they be isolated in a controlled environment with plastic?

2\. Manufacturing

      - If the enzyme/manufacturing process is controlled by a profit-seeking corporation, would this mean unequal pollution capabilities between the developed and developing worlds? I suspect more plastic waste is improperly disposed by developing countries - thus further exacerbating the problem.
      - Must the bacteria manufacture the enzyme necessary, or can an enzyme be chemically manufactured?
      - Does the enzyme have an optimal/useful operating temperature? The Amazon rainforest is not only a freshwater environment, but also a relatively warm climate. Our waste may be captured by cold ocean currents or be present in countries simply far too distant from the equator for this to be a feasible option.

3\. Process consequences

      -What are the products of plastic dissolution by this bacteria?
      -If the process is performed inefficiently/incompletely due to some environmental factors (water salinity, pH, temperature), are there any harmful byproducts?
      -If the process produces simple chemicals - do these harm other organisms in the environment surrounding the bacteria?
      -Would the accumulation of the products (CO2 gas for example) further global warming or pollution?

This article leaves me yearning for more details.

</comment><comment><USER19> I always thought that some day in the future we would start digging up the dumped plastic for turning it back into oil or fuel.

There is another fungus which is capable of turning cellulose from wood into diesel. A fungus which could digest plastic into fuel would be great. <http://www.ens-newswire.com/ens/nov2008/2008-11-04-02.asp>

Of course this fungus should not be injected into the dump sites. Better dig up the plastic and have it converted into fuel in a chemical plant.

<comment><USER20> Yeah. A lot of plastics are 100 percent recyclable. Eventually you just mine the landfill if it is worthwhile. No? Destroying the plastic seems a waste.

Possible selection and confirmation bias as I design plastic products.

<comment><USER21> A ton of them are recyclable in their pure state, but I get the impression that a lot of seemingly mundane plastic products these days are actually surprisingly complicated mixes/composite materials: I know something simple like a potato chip bag is at least a 3-layer composite structure, and you've got inks from the label, metal from the barrier layer, polypropylene from the film, and polyethylene from the adhesive layer.

On top of that, the polypropylene film itself is likely a multilayer structure of different copolymers and crystallinities, and it all turns a crappy little thing like a chip bag in to something damn hard to recycle.

</comment></comment></comment><comment><USER22> Wasn't this already done? [http://www.mnn.com/green-tech/research- innovations/blogs/boy...](http://www.mnn.com/green-tech/research- innovations/blogs/boy-discovers-microbe-that-eats-plastic)

<comment><USER23> Reading the paper (paywalled, unfortunately), it seems that the big breakthrough was not that a microbe can degrade polyurethane (there's almost a dozen other citations), but that this one can do it both aerobically and anaerobically. That means that you can introduce it to a dump, and it will work in both the deep and surface layers.

[1]<http://aem.asm.org/content/77/17/6076.abstract>

<comment><USER24> Yep --

The discovery here is an endophytic (living in a plant) fungus that can grow anaerobically, using polyurethane as its sole carbon source.

The authors suggest fungi that digest polyurethane have been known for decades. ("""Enzymatic degradation of PUR has been demonstrated by both fungi (4, 5, 6, 19) and bacteria (14, 17, 23).""")

They cite one reference from 1968, which itself implies polyurethane-degrading fungi had been known long before then:

"Fungal susceptibility of polyurethanes." Darby, R.T, and A.T. Kaplan. 1968. Appl. Microbiol. 16:900–905. <http://www.ncbi.nlm.nih.gov/pubmed/16349806>

</comment></comment></comment><comment><USER4> I'd love to see if it could be waterborn.

It would be great to have something eating all the plastic in the oceans other than the fish. And we really REALLY need to start doing something about the trash gyres in the seas.

<comment><USER25> I'm definitely for cleaning up the gyres, but how do we know that this is something that can be controlled? It's not just plastic that's floating around, either. I'll note that this is a tremendous discovery and I hope we can see it through..

</comment></comment><comment><USER26> I had hoped to craft a joke about a scientist's iPad disintegrating halfway through his presentation, but that topic has been covered. So instead I'll object to "Yale discovers ...". Really? Yale did it? Imagine a world in which that's the rule -- imagine a headline in 1905 saying "Swiss patent office discovers new theory about the universe."

<comment><USER27> Seems reasonable enough to me. It would be different if the ex student had completed their studies at Yale and went on to research something different independent of Yale. Here though it sounds like they were funding it and that it was a continuation of research being done while a student.

</comment></comment><comment><USER28> This could be great news for the 3D printer industry.

</comment><comment><USER29> This is the full paper, as linked from reddit: [http://view.samurajdata.se/psview.php?id=f1de9924&page=1...](http://view.samurajdata.se/psview.php?id=f1de9924&page=1&size=full)

The pageturn navigation is in the upper left hand corner.

</comment><comment><USER30> How would they have identified this fungus for it's properties in the Amazon rainforest? Do they just leave a piece of plastic near a tree and come back to see if anything's growing on it?

<comment><USER23> Scanning through the abstract[1], it seems pretty likely to me that they did the actual screening in a lab, and it's merely that the fungus was originally from the rainforest. In fact, one of the 21 authors on the paper is from a Peruvian university, and he's likely the one who's group originally isolated the fungus.

[1] <http://www.ncbi.nlm.nih.gov/pubmed/21764951>

</comment></comment><comment><USER31> This reminds me of a YouTube video, I saw within the last ten days here, where a stand-up comedian talks about, how the planet does not need saving, it will find out ways to save itself. I can't find it now though.

Scientists cannot find a way to degrade plastic does not necessarily mean plastic is not bio-degradable.

I have found time and time again, that stand-up comedians are the ones who make the most serious points.

EDIT: Found it - <http://www.youtube.com/watch?v=eScDfYzMEEw>

<comment><USER32> Was it George Carlin, by any chance?

<comment><USER31> Yes indeed it was him.

</comment></comment></comment><comment><USER33> This topic was rather fully explored in a 1971 book: Mutant 59: The plastic eaters by <http://en.wikipedia.org/wiki/Kit_Pedler> and Gerry Davis (both affiliated with Dr. Who)

As a kid that book seemed captivating(scenes of disintegrating airplane, sewage exploding etc.), upon rereading it some years later the plotting and characters seem hackneyed.

</comment><comment><USER34> Do you know why we have oil and coal?

When all those carbon-based life forms died all those years ago, there was nothing to break them down. So they persisted, and the result is oil and coal.

I've often wondered whether a similar thing would happen to all this plastic lying around. I suppose it's still sort of a toss-up, but now I'm at least sort of convinced that it's vaguely possible.

<comment><USER35> > When all those carbon-based life forms died all those years ago, there was > nothing to break them down.

There was nothing to break them down not because such a thing did not exist but because they were protected from those organisms: they were buried under anoxic conditions (complete lack of oxygen) or beneath acidic waters, shielding the organic matter from that which would normally degrade it by feeding on it.

</comment></comment><comment><USER36> There was an article about something similar on Slashdot a decade ago about CD/DVD-eating fungi.

[http://science.slashdot.org/story/01/06/18/1317218/cd- eating...](http://science.slashdot.org/story/01/06/18/1317218/cd-eating- fungus-among-us)

</comment><comment><USER37> Landfills are not the problem.

The plastic waste that _isn't_ on landfills is.

</comment><comment><USER38> What is the byproduct after consumption of polyurethane?

</comment><comment><USER39> This is actually huge!! However, I wonder if uncontrolled growth would lead to something even more terrible.

</comment><comment><USER40> So if life started 6,000 years ago when there were no plastic, does this mean that this fungus was a recent creation?

<comment><USER40> Whoever voted negative on this comment does not fully understand my point. I am pointing out how ridiculous it is to believe that all life started 6,000 years ago and all species were already made exactly as they are. The fact that this particular fungus is surviving by eating plastic simply states that evolution happened and is still happening. Or maybe he did fully understand but just got offended by my mockery of the Judeo-Christian-Islamic version of how the universe, the world, and life began.

<comment><USER41> Or maybe they understood you and thought that your hate mongering vitriol doesn't belong here. No one brought up evolution or creationism here but you.

<comment><USER40> Hate? is telling the truth hate? Religion is an idea. Like any other idea it should be subjected to scrutiny and criticism. Unfortunately in our culture criticizing religion is considered wrong. I did not even criticize any religion in particular but criticized a specific idea, an idea contradicted by scientific evidence. Religion should not be untouchable when its teaching contradict science. No Hate mongering or vitriol here. Implying that the doctrine of special creation is flat wrong is not hate mongering, it is simply telling the truth.

<comment><USER41> Except no one brought the subject up. You're blurting out your off-topic opinion/problems/disagreements just because you felt like it.

<comment><USER40> It is not off subject. This is very strong evidence for speciation. The discovery of such fungus has wide implications, from industrial applications to proving the theory that life adapts and continue to evolve. It is related because the very existence of this life form proves a scientific theory.

</comment></comment></comment></comment></comment></comment><comment><USER42> is it edible too? Two birds with one stone(effect plastic garden ornament)

</comment><comment><USER43> Now that's great news! I would be interested in the environmental impact of this fungus though. It would not be a good thing if the fungus turns out to cause more damage than the plastic itself.

One of the most important things we learned from early 20th century conservationist movement was that to "preserve" something the best thing you can do is leave it alone. Every time they tried to correct something in a ecosystem something else would break. They messed up a lot of Stuff in yellowstone until they learned to just let it be.

<comment><USER44> this fungus could end up contributing to global warming, by taking the carbon that is trapped as plastic on the ground and releasing it eventually into the atmosphere.

</comment></comment><comment><USER45> Sooooo Andromeda Strain.

</comment><comment><USER46> the last paragraph is the most depressing part, just as weve found a sustainable ecological solution to problem of plastic, we learn that someone has commoditized it into a chemical solution, which no doubt has the accompanying legal patent absurdities,few if any positive environmental interactions and actually endangers a very special species due to corporate anti competative behaviour. Irony anyone?

<comment><USER47> What on earth are you talking about? How does isolating the enzyme hurt the environment (more than the creature itself would) or endanger the species?

<comment><USER46> the environment is a holistic function of living symbiotic relationships. Isolating an enzyme and spraying it on a field is about as environmentally friendly as covering a field in pus and saying its ok because it comes from nature.

<comment><USER48> As opposed to covering a field with fecal matter?

<comment><USER46> yes because shit happens to be something that would occur naturally in that environment, whereas isolating some of the things that are in shit and spraying it on a field (chemical fertilizers) actually do harm in the long run.

</comment></comment></comment></comment></comment><comment><USER49> Andromeda Strain!

</comment><comment><USER50> Having Andromeda Strain flashbacks.</comment>


"Science as a Vocation" by Max Weber (1918) [pdf] - <AUTHOR> http://anthropos-lab.net/wp/wp-content/uploads/2011/12/Weber-Science-as-a-Vocation.pdf

<comment><USER2> Here's the original (german): [https://archive.org/stream/gesammelteaufs00webeuoft#page/524...](https://archive.org/stream/gesammelteaufs00webeuoft#page/524/mode/2up)

<comment><USER3> Thanks for the source. I got intrigued by the first sentence of this phrase:

"Democracy should be used only where it is in place. Scientific training, as we are held to practice it in accordance with the tradition of German universities, is the affair of an intellectual aristocracy, and we should not hide this from ourselves."

Turns out the original states "Die Demokratie da, wo sie hingehört", which seems to translate to "The democracy there, where it belongs." So the translation should probably read "Democracy should be used where it _has its_ place."

Also noticeable: the German original features page-long walls of text, which have been broken into additional paragraphs in the translation.

</comment></comment><comment><USER4> I disagree with many of the authors conclusions, but it is a wonderfully challenging article. Thank you!</comment>


What Every Computer Scientist Should Know About Floating-Point Arithmetic (1991) - <AUTHOR> https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html <comment><USER4> Posted 30 times, but most of the comments are about how many times it has been posted. Here are the on-topic threads:

2017 (just a bit): [https://news.ycombinator.com/item?id=13431299](https://news.ycombinator.com/item?id=13431299)

2012: [https://news.ycombinator.com/item?id=4815399](https://news.ycombinator.com/item?id=4815399)

2010 (not a good year): [https://news.ycombinator.com/item?id=1982332](https://news.ycombinator.com/item?id=1982332)

2009 (also pretty bad): [https://news.ycombinator.com/item?id=687604](https://news.ycombinator.com/item?id=687604)

[https://hn.algolia.com/?query=What%20Every%20Computer%20Scie...](https://hn.algolia.com/?query=What%20Every%20Computer%20Scientist%20Should%20Know%20About%20Floating- Point%20Arithmetic&&sort=byDate&dateRange=all&type=story&storyText=none&prefix)

</comment><comment><USER5> That's quite a lot. The by far most important things for me: Floats guarantee at least 7, and doubles at least 15 significant decimal digits. That's regularly surprisingly low to many devs.

<comment><USER6> It may seem low, but it's really not. At 6 significant digits, you'd have a margin of error of a thousandth of an inch when measuring a football field.

<comment> T-hawk Sure, that's low, depending on context. At 6 significant digits, a million- dollar bank account would have a margin of error of up to +/\- 5 dollars.

And if you add up a million of any item with 6 significant digits, that's enough for inaccuracy to propagate all the way into your _first_ significant digit. (It would require systematic inaccuracy in one direction, but there are certainly data sets and contexts where that would happen.)

<comment><USER6> [https://dzone.com/articles/never-use-float-and-double-for- mo...](https://dzone.com/articles/never-use-float-and-double-for-monetary- calculatio)

</comment></comment></comment></comment><comment><USER7> Some historical context... IEEE 754 (the current floating point standard in H/W and S/W) arose in 1985, only 6 years before this paper. Before that, floating point representations were often proprietary (e.g. Cray or IBM). Back then, most microprocessors did not have floating point hardware, so most non- scientific apps avoided floating point operations and booleans using them. Few coders had much experience with floating point unless they were engineers or scientists.

This paper broke new-ish ground in introducing 'the rest of us' to the nonlinear representation space of digital scientific numbers and subtleties of roundoff & inaccuracy inherent in math-intensive libraries and scientific math functions. It still usefully covers more ground in that space than most of today's BSCS degree programs.

</comment><comment><USER8> Oddly enough I got an issue with float this week, 2 systems are communicating by transmitting float (rather needlessly I would add) and the conversion from float seems to be implemented differently on both systems, resulting in discrepancy sometimes.

This is kind of textbook example of an obvious bug but still surprising when you find it.

<comment><USER9> I do not understand what you mean. If you "convert" a float to a float it shouldn't change anything? As long as you keep to floats there should be no problem.

<comment><USER10> You could run into problems with little endians and big endians if you do a binary tranfer or, more likely, have a lossy float->string->float conversion.

<comment><USER11> Unless you are using an IBM mainframe, Big Endian is all but extinct. Besides, it is about ~10 lines of C to detect endianess and reorder bytes if necessary.

<comment><USER12> > Besides, it is about ~10 lines of C to detect endianess and reorder bytes if > necessary.

Proper code to do this doesn't do any tests. Just grab the bytes from the order of the transfer format (in this case network order) into the order of the abstract machine in C. The compiler will turn that into whatever is needed depending on if it's compiling to a big-endian or little-endian machine and may use byteswap instructions if needed. Having tests in the code is a code smell that creates unchecked paths to the code that are much more likely to be wrong as they are not used most of the time.

<comment><USER13> > and may use byteswap instructions if needed

Or it may not because often compilers fail to make simple optimizations:

[https://godbolt.org/z/NAt3uX](https://godbolt.org/z/NAt3uX)

<comment><USER12> It does fine if you actually write the correct code :)

[https://godbolt.org/z/qpLeWP](https://godbolt.org/z/qpLeWP)

<comment><USER13> There is nothing incorrect about the code I posted - clang will compile it to a single read + bswap just fine. And you don't need to go that far back for your code to not be recognized - GCC 4.9 will produce individual loads and shifts for that too.

The point is that you can't rely on compilers consistently to recognize complex patterns.

<comment><USER12> Incorrect was too strong but it's a weird pattern to do this with sums. The OR pattern is what is used pretty much everywhere and conveys the intention of the code much more clearly.

And even if the compiler doesn't always optimize ideally my original point still stands. Delegating this to the compiler instead of trying to manually call swap instructions is a much better way of going about it.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> Original paper (PDF): [https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf](https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf)

</comment><comment><USER15> I think what every programmer should know is probably only 4 words:

They're approximations. Use accordingly.

</comment><comment><USER16> I'm still hoping posits make it one of these days.

</comment><comment><USER17> This is gold. I've been forwarding this link for everyone who says they are using float for storing currency values (money, $$$) since forever.

<comment><USER18> I've often used floating point for currency. What can go wrong? Actually, a number of things can go wrong when using floating point as the article pointed out. The main concern nowadays is that floating point doesn't represent values with infinite precision and consequently is not capable of exactly representing every possible value.

In the past, number formats were often a concern. The first assembly language program I ever wrote was for an IBM mainframe that required a conversion of some numbers in zoned decimal format to be converted to packed decimal format. There were actually native assembly instructions for doing calculations in these different decimal formats. The IEEE 754 standard for floating point that we use today came about in 1985. Before that, floating point might involve 60 bit values (as on CDC mainframes) or hexidecimal based floating point (as on the IBM 360/370 mainframes). Double precision was not widely used because of memory limitations. Floating point was much slower. Programming languages didn't provide good or consistent facilities for non-integer values (it was virtually impossible to predict which implicit conversions were being done by PL/1 when doing mixed calculations, COBOL and FORTRAN handled calculations wildly differently). I believe that some of the current general advice about handling financial calculations stems from considerations that made sense during these Middle Ages of programming.

Now, with double precision available everywhere and standard, specified rounding modes, and the other benefits of IEEE 754, I think it's safe to consider using floating point for currency calculations. _The most widely used software for financial calculations uses floating point_ (Microsoft Excel).

If Excel uses floating point, why is there a widely promelgated admonition to avoid it for currency? I believe that it made sense in the Middle Ages of computing, but now is not as relevant.

While it is true that some quantities cannot be represented exactly in floating point, for example 0.01, the same is true about decimal fixed point where 1/3 cannot be represented exactly. Common financial calculations can fail to be exact no matter the number format.

Consider calculating the payments for a fixed-rate 30 year mortgage. At a 5% interest rate a $200,000 loan will have a monthly interest rate of 0.05/12 so there will be 12 * 30 payments the amount of each of these payments will be:

        200000 * (0.05/12) / (1 - (1 + (0.05/12))^(-(12*30)))

This formula cannot be calculated exactly in decimal floating point for two reasons. First, the fraction (0.05/12) is not exact in decimal and secondly, there is unlikely be be a direct way to do exponentiation of decimal values.

Some languages (like Common Lisp) support exact rational numbers. This allows exact calculations with any rational fraction, but this still doesn't allow calculations involving irrational numbers, like sqrt(2) to be represented exactly. Consider calculating the cost of a circular plate when priced in dollars per gram. This involves using pi.

Care must always be exercised when doing financial calculations if they need to match how financial institutions are doing their calculations. Reports must round fractional values using the same rounding methods (is 1.005 rounded to 1.00 or 1.01? i.e. round-up vs round-to-even). Values should usually be stored after rounding to the same units used in the currency. These problems are not caused by the inaccuracy of the 17th digit of a double precision floating point being off by one.

For further information on the kinds of considerations that need to be made take a look at the design discussions that have been documented for the beancount project [1].

[1] [https://beancount.github.io/docs/31_rounding_precision_in_be...](https://beancount.github.io/docs/31_rounding_precision_in_beancount.html)

<comment><USER19> > I've often used floating point for currency

The question is WHY? Because decimal types are barely second class citizens across the stack. Is similar with dates.

Bad defaults are bad, and requiere stupid amounts of workarounds.

And just because excel do it?

[https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft...](https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft_Excel)

Excel is amalgamation of surprises that are not fixed.And when you put the CORRECT results, the users demand that we give the same wrong results as excel!

</comment></comment></comment><comment><USER20> Hello imposter syndrome my old friend

<comment><AUTHOR> Computer science is hard and even Knuth is humbled by it. And he's been writing several programs a week for about sixty years.

</comment></comment><comment><USER21> A follow up to this classic article could be titled "What every language designer should know about humans." :) Floats are among the least user- friendly ways to do decimal arithmetic. They aren't even that fast anymore given the resurgence of fixed point arithmetic in hpc.

Imo, in 2020 we should have had mainstream languages in which (2^(1/2))^2 == 2. It's not rocket science. :) Float truncation is a little like an OS silently converting a user's flac collection to mp3 because it was running low on disk space.

<comment><USER22> > Imo, in 2020 we should have had mainstream languages in which (2^(1/2))^2 == > 2

Such things are either incredibly inefficient or mathematically unsound. And there's plenty of provable impossible computational things involved.

Start at the tablemaker's dilemma, then learn the difficulty of deciding if some basic expression is actually zero, and so on. There's a massive literature on doing such computations, and many, many impossibility theorems.

There is no getting around some seemingly simple problems require arbitrarily large computational resources to resolve - floating point picks the very useful path of doing approximations in predictable time over attempting exact answers in unbounded time and space.

<comment><USER21> "Such things" are the foundation of symbolic math libraries, deep learning libraries, most optimizing compilers and Wolfram Alpha. I.e it is not true that they would be incredibly inefficient or mathematically unsound.

I don't understand your point about the Tablemaker's dilemma because it is a dilemma only if precision is fixed. With the correct numeric type (rationals) precision is not fixed.

<comment><USER22> Let's unpack:

> With the correct numeric type (rationals) precision is not fixed.

Almost no problems can be represented exactly by rationals. Even the sqrt(2) you started with is not rational.

And rationals are unusable for almost any type of work, because they grow exponentially (in time and space) for most problems.

For example, if you tried to compute a simple mandelbrot set with rationals, you'd need more bytes than there are subatomic particles in the universe (~10^80).

Proof: Mandelbrot takes a complex number c = x + i y, which you want to represent as a rational, and iterates z <\- z^2 + c. Squaring doubles the number of digits for each of the numerator and denominator, so each iteration requires ~2 times the storage of the previous iteration. Thus you'll need over 2^300 bytes for 300 iterations, which is > 10^90. A common mandelbrot uses over 1000 iterations.

So you see rationals are computationally unusable for even simple tasks.

> symbolic math libraries

These can be arbitrarily slow and use arbitrarily large memory for simple problems - which I stated above when I said "Such things are either incredibly inefficient or mathematically unsound." For example simple converging infinite sums that don't have closed forms known to your tools would never evaluate - the only solution is some fixed, finite representation if you want to use the value in later computations. The same thing for integrals, root finding, pdes, and thousands of other problems.

> deep learning libraries

Deep learning libraries have led to a reduction in precision specifically for the reasons I mentioned: lower memory requirements and higher speed. Hardware vendors from Intel to NVidia to AMD to ARM have introduced lower precision math, including the new bfloat16 (and some have a bfloat8) for these reasons. I think this is the opposite of what you claimed, but supports my claim.

Care to show me where deep learning libraries do perfect symbolic computation in the process of deep learning? I'm aware people try to train them to do symbolic computation, but the libraries themselves aren't using the types of reductions you posted.

> most optimizing compilers

Most compilers would reduce sqrt(2) * sqrt(2) to 2? List them. (Heck - <USER3> one!) It's simply not true (as can be checked trivially for all common C++ compilers on godbolt).

I've seen none, despite following optimizing compiler theory and practice for decades.

>Wolfram Alpha

Redundant with the symbolic stuff above - and I've used Mathematica since v1.0 in 1987 quite extensively. It most certainly is slower doing anything symbolically that can be done numerically, and there are lots of problems it cannot do symbolically without puking, but those same problems run instantly when approximated with floating point.

There's ample pages listing things symbolic engines get wrong or cannot do, but that can be done elsewhere.

>I don't understand your point about the Tablemaker's dilemma because it is a dilemma only if precision is fixed

????

The dilemma is not about fixed precision.

The problem is correctly rounding a function as if it had an _infinite_ expansion first. For some functions this means arbitrarily large computations. Ahead of time mankind does not know how many digits are needed for many common uses. So the problem is not about fixed precision - it's about possibly unbounded computation, which is vastly different.

For example, no one has a bound on the number of digits needed to compute correctly rounded a^b for any two floating point values a and b.

And there exist computable numbers for which the rounded value can never be computed with any amount of digits [3].

So it's not about fixed precision. It's far more subtle.

If you're willing to run unbounded in time and space computations, then you can more and more precise answers, but at every point you _still_ don't know if you've escaped the Tablemaker's dilemma. This is the point.

Here's a guy I've followed for decades, one of many researchers on the topic [1]. Note that his "hard to round cases" in 2013 took 1576 _years_ of computer time to determine how many digits are needed for double precision values for some elementary functions, precisely because this is not the trivial problem you think it is.

Here's a 2016 paper doing some work on GPUs [2]: " For example, the Nesterenko and Waldschmidt [1996] bound for the exponential in double precision states that 7,290,678 bits of intermediate precision suffice to provide a correctly rounded result."

To get 64 bits of accuracy, correctly rounded, you need 1 megabyte and an incredible amount of computation. My point exactly. And these bounds tend to increase exponentially, not linearly, soon pushing such issues beyond currently computable (which is also why there are still research papers being published on getting results for 64 bit doubles in various ways and for various elementary functions).

Again you have the choice I presented: you either accept unbounded in time and space behavior, or you fix time and space to make things usable with approximations. Floating-point is an incredibly good solution to making numerics as good as possible under fixed space and time requirements.

[1] [https://www.vinc17.net/research/index.en.html](https://www.vinc17.net/research/index.en.html) [2] [https://dl.acm.org/doi/pdf/10.1145/2935746](https://dl.acm.org/doi/pdf/10.1145/2935746) [3] [https://en.wikipedia.org/wiki/Rounding#Table- maker's_dilemma](https://en.wikipedia.org/wiki/Rounding#Table-maker's_dilemma)

<comment><USER21> > Almost no problems can be represented exactly by rationals. Even the sqrt(2) > you started with is not rational.

Almost all numerical problems software developers deal with can be represented exactly using rationals. Only a small fraction of all problems involve transcendentals. And those can't be represented exactly by floats either so I don't know what point you're making?

> For example, if you tried to compute a simple mandelbrot set with rationals, > you'd need more bytes than there are subatomic particles in the universe > (~10^80).

Well, yes. You'd run into the same problem if you tried to compute all the decimals of pi too. Using rationals does not mean that you can't round results.

> So you see rationals are computationally unusable for even simple tasks.

I see that rationals cannot represent all real numbers.

> These can be arbitrarily slow and use arbitrarily large memory for simple > problems - which I stated above when I said "Such things are either > incredibly inefficient or mathematically unsound." For example simple > converging infinite sums that don't have closed forms known to your tools > would never evaluate - the only solution is some fixed, finite > representation if you want to use the value in later computations.

I wrote that in mainstream languages should be able to evaluate (2^(1/2))^2 == 2. How do you go from there to requiring them to be able to evaluate infinite sums?!

> Care to show me where deep learning libraries do perfect symbolic > computation in the process of deep learning?

I didn't claim that.

> Most compilers would reduce sqrt(2) * sqrt(2) to 2? List them. (Heck - <USER3> > one!) It's simply not true (as can be checked trivially for all common C++ > compilers on godbolt).

I didn't claim that. Compilers are stuck with IEEE 754 fp semantics and therefore won't simplify (2^(1/2))^2. But there is no technical reason why they couldn't.

To paraphrase your reply: "Rational arithmetic and symbolic computation can't solve every problem. Therefore they are useless/no better than floating point arithmetic." From a theoretical perspective that is perhaps true but in practice both are very useful tools.

<comment><USER22> >Almost all numerical problems software developers deal with can be represented exactly using rationals. Only a small fraction of all problems involve transcendentals.

There's more classes of real numbers than rationals and transcendentals. Hint: your example of sqrt(2) is neither.

As to doing things _exactly_ with rationals, anytime you take a sin, cos, exp, log of any non-zero rational number the result is not rational. Any time you take a root of a non-perfect power rational you don't get a rational number (you get an algebraic number, which is neither rational nor transcendental).

So all you can do is simple arithmetic: + * / \- (actually, that's from the definition of the field Q of rational numbers).

And you still run into the problem that rational number arithmetic gets arbitrarily slow and runs out of memory. Each multiply on average results in requiring the sum of the storage sizes of the multiplicands, making more than about 40 multiplications infeasible before you're out of RAM.

So your view of "almost all numerical problems software developers deal with" must be limited to only those using simple arithmetic and no more than around 40 operations. That's amazingly constraining.

What problems do you call numerical problems that fit your constraints?

Things that cannot be done exactly with rational numbers includes pretty much all of gaming (3D or otherwise), deep learning (probably every model has an exp in it), scientific computing, audio and video processing (cos and sin all over the place), and on and on.

>Using rationals does not mean that you can't round results.

Wait, you just claimed "...can be represented _exactly_ using rationals. Now you want to round, throwing out the exactness? Which is it?

And congrats - you just re-invented floating-point math which is simply approximating values using rational numbers in a clever manner to handle larger ranges than fixed point. But they're still always rational approximations to values.

>I wrote that in mainstream languages should be able to evaluate (2^(1/2))^2 == 2. How do you go from there to requiring them to be able to evaluate infinite sums?!

Ah, so you mean languages should be able to simply solve this one specific problem, not all relatively simple symbolic problems? That seems like an even bigger mess than using floating point, which is well-defined and covers a large class of problems.

Does your mythical language know (A^(1/B))^B==A for all rationals A and B also representable in your language? Or are you limited to the value A=B=2 only?

(Hint: if you claim it should hold, you're gonna hit problems again :)

<comment><USER21> > As to doing things exactly with rationals, anytime you take a sin, cos, exp, > log of any non-zero rational number the result is not rational.

I fail to see your point. I never claimed that you can do real arithmetic with rationals.

> So all you can do is simple arithmetic: + * / \- (actually, that's from the > definition of the field Q of rational numbers).

Same as with floats. Except with floats you don't even get division.

> And you still run into the problem that rational number arithmetic gets > arbitrarily slow and runs out of memory. Each multiply on average results in > requiring the sum of the storage sizes of the multiplicands, making more > than about 40 multiplications infeasible before you're out of RAM.

What?

        >>> from fractions import Fraction
        >>> Fraction(7, 8)**40
        Fraction(6366805760909027985741435139224001, 1329227995784915872903807060280344576)

In other words, no, you don't run out of memory after 40 multiplications...

> Things that cannot be done exactly with rational numbers includes pretty > much all of gaming (3D or otherwise), deep learning (probably every model > has an exp in it), scientific computing, audio and video processing (cos and > sin all over the place), and on and on.

You sure can do all of those things with rational arithmetic. What you can't do is do them _exactly_ but I never claimed you could.

> > Using rationals does not mean that you can't round results.

> Wait, you just claimed "...can be represented exactly using rationals. Now > you want to round, throwing out the exactness? Which is it?

I wrote "ALMOST ALL numerical problems software developers DEAL WITH can be represented exactly using rationals." The key word in that sentence is "ALMOST."

> And congrats - you just re-invented floating-point math which is simply > approximating values using rational numbers in a clever manner to handle > larger ranges than fixed point. But they're still always rational > approximations to values.

Yes, but I have also solved the freakishly annoying issues detailed in the article. Which one do you prefer:

        >>> print(sum(0.1 for _ in range(10)))
        0.9999999999999999

or

        >>> print(sum(Fraction(1, 10) for _ in range(10)))
        1

? I know which one _users_ prefer.

> >I wrote that in mainstream languages should be able to evaluate (2^(1/2))^2 > == 2. How do you go from there to requiring them to be able to evaluate > infinite sums?!

> Ah, so you mean languages should be able to simply solve this one specific > problem, not all relatively simple symbolic problems? That seems like an > even bigger mess than using floating point, which is well-defined and covers > a large class of problems.

I don't think Mathematica or the Python symbolic math packages I've worked with are particularily messy.

> Does your mythical language know (A^(1/B))^B==A for all rationals A and B > also representable in your language? Or are you limited to the value A=B=2 > only?

Yes, assuming A and B are positive.

<comment><USER22> > What? ... > In other words, no, you don't run out of memory after 40 > multiplications...

So you think one case proves there is no set of 40 multiplications that overflows?

Since we're using a small starting value and only one operation per iteration, let's try 50 mults:

        x = frac(7,8)
        for i = 1 to 50
            x = x * x
        print(x) # as exact decimal fraction

Tell me how much ram and time this took you to evaluate with exact rational values. I'll wait :)

(Hint: you'll need approx 500 terabytes of ram to store these integers. I guess I didn't really need all 50 iterations! )

If you want only 40 mults, then using different starting values and adding a few additions will cause the same overflow. In fact, all sorts of common algorithms will overflow in very few operations using rationals. This is why numerical algorithms textbooks don't even cover it as a possibility - it's a terrible idea thrown out decades ago for precisely these reasons.

Your exact rational number idea has replaced predictable behavior on reasonable inputs with highly finicky unstable behavior, and a programmer will have no idea how to ensure things don't go off the rails for common uses.

In each reply you make fundamental math errors, provably incorrect claims, and don't stop to learn. I already gave a proof that such simple things will overflow - you ignored it, then tried to prove the converse with one toy example. It's not worth showing you more errors when you ignore evidence and continue disproven claims.

We're done. You're too stubborn to absorb relevant material.

<comment><USER21> > So you think one case proves there is no set of 40 multiplications that > overflows?

> Since we're using a small starting value and only one operation per > iteration, let's try 50 mults:

That is repeated exponentiation, not repeated multiplication and you are grasping for straws. Try the same code using an integer or a float greater than 1. Since the same thing will happen as when using rationals (you get an error) are you going to tell me that those number types are useless too?

> In each reply you make fundamental math errors, provably incorrect claims, > and don't stop to learn.

Now I realize that you are most likely trolling me. Fine. Have a nice day! Bye!</comment></comment></comment></comment></comment></comment></comment></comment></comment>


Why Does It Take So Long to Connect to a WiFi Access Point? - <AUTHOR> https://arxiv.org/abs/1701.02528 <comment><USER3> And why is there no proper 'invalid password' response? It always seems to me like if you have the correct password the connection succeeds and if you have the wrong password the connection just mysteriously fails and the computer has to guess that maybe it was because the password was wrong? In fact I've had a Macbook tell me the password was wrong when it was _definitely_ correct. I imagine it's because they have to guess the failure reason.

<comment><USER4> If you're using a pre-shared key, the password is verified during the 4-way handshake. The thing is, if your password is wrong, then the Message Authentication Code (MAC) of the messages your are sending is wrong. The AP will simply drop frames with a wrong MAC, and will not respond to them. The problem is that as a client you do not know whether the AP is not responding because (1) the MAC was wrong, and hence the password was wrong; or (2) the message did not arrive at the AP (or you did not receive the response of the AP).

tl;dr: can't tell the difference between dropped messages due to wrong authentication check (i.e. wrong password), or dropped messages due to bad connection.

<comment><USER5> I think this is half right. There is still an L1 ACK, so the STA doesn't have to retry sending the packet, it knows it was received.

I believe what happens is AP sends Nonce to the STA, STA uses the PSK to send Message 2 back to the AP. It will receive a '802.11 Ack' but then no Message 3 of the 4-way handshake will ever come from the AP.

Good drivers see this and flag an invalid password warning back to the user within milliseconds. But bad drivers... sure, they will just keep assuming magic dust got in the way and if they just keep retrying the handshake enough maybe they will see a Message 3.

I'm not sure why from a security hardening perspective it's better not to specify the AP should send '802.11 Disassoc' immediately after receiving an invalid Message 2 with a proper error code so that the driver can message the UI that the password is wrong instantly.

<comment><USER4> Not really. The STA may know that its message was received, but can never be sure whether the AP replied. The reply from the AP could have been missed due to noise, or maybe it didn't reply at all. You cannot be sure. There are only heuristics.

Good drivers indeed tell you whether the message arrived or not. But it's up to the client to decide what to do with that information. And again, it's just a heuristic. I've read and messed with the code of four different Wi-Fi clients, and none of them attempt to detect a bad password this way. Most simply report an error after trying to retransmit message 2 multiple times (e.g. if wpa_supplicant got message 1 from the AP, but didn't get a reply to message 2, it warns that _maybe_ the password was wrong).

</comment></comment></comment></comment><comment><USER6> This is great work but I think the machine learning part is just thrown in for trendiness reasons. More rigorous statistical analysis should lead to a cleaner algorithm without relying on pretrained classifiers.

<comment><USER7> In general, any opinions about where we are on the hype curve for machine learning (as a stand-alone concept separate from traditional statistical inference)?

[https://en.wikipedia.org/wiki/Hype_cycle](https://en.wikipedia.org/wiki/Hype_cycle)

<comment><USER8> It's not the first hype cycle, so we've already been through all phases, but now I guess we are still raising towards peak hype. The fact is that this time AI has real world applications and benefits, so it's not just a bunch of hot air. It will not have a crush at this level of hype yet because we now have lots of results that were unimaginable even 5 years ago.

We're breaking human level accuracy in vision, speech, text and behavior - <USER2> specific tasks, not in general yet. In the last 3 years neural nets have become creative - now they can create paintings (neural style transfer), images (GANs), sounds (WaveNet), text (seq2seq, translation, image captioning) and gameplay (Atari, AlphaGo).

All these are complex forms of creativity, as opposed to simple classifiers. So we have recent progress, there is no period of lacking results behind us. That's why we're still on the rise.

<comment><USER6> Well said, I would agree that we're just to the left of the peak but that applies more to the areas adjacent to the problems where deep learning has shown some real value.

I remember the previous AI peaks in the 80s and 90s as well the neural net and fuzzy logic euphorias. The problem back then was that the results were not that exciting. Now we have some really impressive applications searching vast datasets and recognizing useful features. However, I notice everyone is trying to apply the same algorithms to problems not well suited for that type of approach.

</comment></comment></comment></comment><comment><USER9> This reminded me of this explanation of how MacBooks (and presumably other devices) manage to reconnect to Wifi so quickly after waking from sleep: [http://cafbit.com/entry/rapid_dhcp_or_how_do](http://cafbit.com/entry/rapid_dhcp_or_how_do)

<comment><USER10> And in the process walking all over various security best practices?

And was there not a similar security snafu involving iPhones broadcasting past ssids every time they tried to connect to a access point?

<comment><USER11> Don't most (mobile) devices broadcast all known SSIDs just in case a network is available and hidden? I don't really know a lot about the subject matter, but as I recall this does not apply to _just iPhones_.

I also remember an installation at the Datenspuren in Dresden with a monitor showing all of these SSIDs it intercepted with people walking past and being astounded how the device knew their home network name^^

<comment><USER12> Almost all Android devices do the same, they just send probe requests with all known networks. That is why you can automatically connect to "hidden" networks. Just run kismet or wireshark and see for yourself.

<comment><USER13> And it can be used to uniquely identify devices and track them with fairly standard hardware

<comment><USER14> Doesn't a MAC address get broadcast anyway? Unless wifi devices are randomizing their MAC addresses that seems like a fairly trackable thing.

<comment><USER15> IIRC newer iOS devices do randomize the MAC address used for probes.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16>
        > we develop a machine learning based AP selection strategy that can
        > significantly improve WiFi connection set-up performance, against the
        > conventional strategy purely based on signal strength, by reducing the
        > connection set-up failures from 33% to 3.6% and reducing 80% time costs of the
        > connection set-up processes by more than 10 times.

        > The correlation analysis finds that though the signal strength is important,
        > knowing the AP model and mobile device model has great help to predict the
        > connection set-up time cost.

Neat!

<comment><USER17> > [Abstract] "we develop a machine learning based AP selection strategy that > can significantly improve WiFi connection set-up performance, against the > conventional strategy purely based on signal strength, by reducing the > connection set-up failures from 33% to 3.6% and reducing 80% time costs of > the connection set-up processes by more than 10 times."

> [Conclusion] "The correlation analysis finds that though the signal strength > is important, knowing the AP model and mobile device model has great help to > predict the connection set-up time cost. To the best of our knowledge, we > are the first to add AP model and mobile device model as features which > greatly increases the accuracy to predict the connection set-up time cost."

_____

(NOTE: Please do not use block quote formatting, it's unreadable on mobile.)

<comment><USER18> @HN please add wrappable quoting markup.

<comment><USER19> I requested this a while ago and was told that, although they like the idea, adding standardized blockquote markup would somehow break HN's spam filter.

<comment><USER15> It's not uncommon to use italics for quoted paragraphs, optionally prefixed by a single greater than, or surrounded by quotes.

 _Like this._

 _> Or this._

</comment></comment></comment></comment></comment><comment><USER20> Was hoping for an explanation to: Why doesn't it take a fraction of a second to connect?

<comment><USER21> The paper says that the biggest delay is in the "scan" phase, just getting a list of all the available APs in the area. This would be the same problem addressed by Apple's "try to associate to all known SSIDs on powerup" approach.

Maybe I'm missing something, but their actual machine learning model seems to address a different problem:

      The final features we choose to train: the connection
      time cost includes hour of day, RSSI, mobile device
      model, AP model, Encrypted

Given this, I they're basically giving lower priority to known incompatible device & AP pairs based on their BSS IDs.

Not sure I like the approach that much, seems an AP running a recent OpenWRT and very reliable would be penalized for having buggy factory firmware.

<comment><USER22> Seems like something simple like "look for the last N and the most common {M0, M1, M2} in the past {week, month, year} before doing a full scan" should hit the vast majority of use cases, no?

</comment></comment></comment><comment><USER23> Apple devices seem to aggressively cache the last known IP address on each wireless network, rather than issue a new DHCP request.

At home this results in duplicate IP addresses when the kid with the iPhone gets home after being away and meanwhile another device has started using that IP address. This tends to bork up the entire network on my cheap Netgear router and I usually have to reset it at that point.

<comment><USER24> Naive question...why does that mess with the network? Your router has the correct MAC <-> IP mapping, and iPhone kid is the only one losing out.

<comment><USER23> Not sure but the correlation of Apple device walking in the door and entire home network hanging is pretty consistent.

I presume the router periodically issues "who has 192.168.1.10" or whatever and upon getting responses from two different MAC address just gives up.

</comment></comment></comment><comment><USER25> I would be very interested in seeing the correlations between set-up time and total devices in the area (or, to be more precise, total channel utilization). This paper studied devices associated per access, point, which is a separate metric. They mentioned the issue in a paragraph at the end of a section, probably dug up in post-study literature review.

From work in wi-fi router development, high channel utilization is often a much bigger determinant of packet loss than either router overload or RSSI. Hour of day is probably just a good proxy for this.

802.11 is pretty good at handling the shared medium when a single access point can do traffic control, but multiple access points in a crowded city or office building gets you into all kinds of problems. Usually you can get big performance gains in large deployments just from making sure nearby access points are on different channels.

Between the other two major factors the paper looked at:

1) The number of clients a single router can handle is mostly limited by CPU power (and the failure mode there is typically not association request drops, since those are usually processed pretty early in the packet pipeline, without much queuing). So I'm not surprised that they saw very little effect of number of associated clients with connection time.

2) RSSI is more important in low-interference environments, where the ability to hear packets over the noise floor is a big limiter. In dense, high- interference environments, it helps a bit in terms of being able to shout over the noise of very distant interference sources, but for the most part a collision is a collision even with substantial magnitude differences between the colliding packets.

<comment><USER26> Since you work in WiFi, I was wondering if you could comment the linked chart of SNR to Modulation rates [0]. Obviously the exactly values are going to depend on hardware to some degree.

[0][https://dl.dropboxusercontent.com/u/8644251/Revolution%20Wi-...](https://dl.dropboxusercontent.com/u/8644251/Revolution%20Wi- Fi%20MCS%20to%20SNR.pdf)

<comment><USER25> What exactly do you mean by comment? Explanation of what it represents?

Basic overview, with the caveat that I haven't done wifi stuff in a couple of years, and was not doing straight up RF stuff like the hardware folks. It's basically saying that as signal strength above the noise floor (in dB) (ie RSSI - kind of) increases, you can get to higher bit rates. Let me know if you want more info on what exactly is being described.

</comment></comment></comment><comment><USER27> There's a new tech called MulteFire, which is LTE over unlicensed spectrum. Everybody could buy an access point(altough those will be more expensive for than wifi for some time, at the very least).

One of the biggest selling points for it is a simpler faster handoff than WIFI. Is that a big enough problem that people will be willing to buy access points ?

</comment><comment><USER28> Interesting to see this researched, but I think this isn't really applicable to most people's situation at home, where there's 1-3 access points, and most often you just want to connect to a specific one. Then you don't really need software telling you that connecting will be slow, because you know, but you just want to connect.

Besides, in my experience, having a lot of wifi interference is a huge PITA when connecting or communicating with an AP. Maybe they were not able to include that factor in their dataset for some reason, but I think you'd find a strong (negative) correlation.

<comment><USER29> Probably not at home, but that's not when I want to connect faster. I use the Fonera network when I'm in the street, and there's often three or four APs near me, so this could be helpful. Another case is campus wifi, when I was in college. In both of these cases, I want to connect to a specific SSID, but not to a specific AP.

</comment></comment><comment><USER17> Here's an MIT Technology Review article covering this reach, "Data Mining Solves the Mystery of Your Slow Wi-Fi Connection":

[https://www.technologyreview.com/s/603414/data-mining- solves...](https://www.technologyreview.com/s/603414/data-mining-solves-the- mystery-of-your-slow-wi-fi-connection/)

<comment><USER30> That article is crap because it never says why connections are slow. Lots of words and no substance.

<comment><USER17> I would ask if you're missing a bit, but that would be like saying something is "crap" and materially misrepresenting its value and substance.

</comment></comment></comment><comment><USER10> Seems to me like a "problem" only in the sense that one wants to leech data connectivity while out and about.

<comment><USER29> There are many cases where one wants to use wifi while out and about without "leeching". In my city alone, we have areas with paid networks, tit-for-tat networks (Fonera) and public networks (as in paid by taxes).

<comment><USER31> Interesting, I'd never heard of Fonera. Trying to find hotspots in my area has simply emphasized to me how dusty my laptop screen is (the closest hotspot seems to be several hours' drive away).

<comment><USER29> Fonera is nice in my country (and a few other European ones) because one of our largest ISPs made a deal with them, so every client of theirs is also a Fonera member, and therefore a hotspot I can use.

</comment></comment></comment></comment><comment><USER32> Could someone with permission to do so please fix the title? It either needs the question mark removed, or to be written as a question (e.g. "Why Does It Take So Long…")

<comment><USER33> Only the paper authors can fix their PDF, I guess.

<comment><USER32> Is there some policy on HN that the title of the submission always matches the title of the thing that is linked to? I'm sure I've seen situations where it doesn't.

<comment><USER34> The verbatim title is preferred:

"... _please use the original title, unless it is misleading or linkbait._ "

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

<comment><USER35> Though it's more like the verbatim title is preferred except when it isn't and there's a good chance an editor will change it to the verbatim title, or something worse, or maybe better and an equally good chance people will complain and it'll be changed [back].

I've always said we should have the verbatim title and an editorialised one and let people choose in config which one[s] they want.

Current system mostly works though.</comment></comment></comment></comment></comment>