The Internet is running in debug mode (2014) - <AUTHOR> http://java-is-the-new-c.blogspot.com/2014/10/why-protocols-are-messy-concept.html <comment><USER2> OK, we're going to see a lot of rehashing of the old arguments:

Pro-text: Human-readability saves immeasurable human developer time!

Pro-binary: Text parsing wastes a lot of _very measurable_ machine time!

Problem is, half of this argument is based entirely on anecdotal evidence and gut feelings. We really have _no idea_ how much developer time is saved by having messages be human-readable. You will find smart people who believe _very strongly_ both ways. In a seemingly very high fraction of these cases, it seems that people are really just taking the practice they are more comfortable with (because they've used it more) and rationalizing an argument for it because it makes them feel good about their choices. When hard evidence is lacking, confirmation bias, unfortunately, takes over.

As the author of Cap'n Proto and former maintainer of Protobufs, I obviously come down pro-binary... but I won't bore you with my argument as I don't really have any hard facts either.

<comment><USER3> You're missing a major feature. Text formats don't have any natural integer size limits.

That's been a big win for HTTP. It's easy to imagine the original HTTP authors thinking that 24 bits is plenty of space for file size (anything bigger should use FTP) or using seconds since 1970 in 32 bits instead of text dates.

Those are a bit obvious, but a developer could easily put in something like a 255 chunk limit without thinking too hard about it.

Even John Carmack accidentally screwed over OpenGL driver developers by copying the GL_EXTENSIONS list into a 1000 char buffer in GL Quake (that's way too small).

A binary format makes hard limits too easy.

<comment><USER2> All good formats -- text or binary -- are capable of being extended over time in a backwards-compatible way, in order to fix mistakes. Formats that don't allow this are _bad_ and should not be used. Protobufs and Cap'n Proto in particular have very strong support for forwards- and backwards-compatibility.

Choosing a too-small fixed integer size is one way that a format can screw up and need correction -- one which happens to be (somewhat) exclusive to binary formats. But there are plenty of ways text formats can screw up too, like trying to pack fundamentally structured data into an ad-hoc hard-to-parse string format in order to make it look more pleasant to humans (I'm looking at you, MIME type specifications), or failing to use consistent parsing rules (cookie and user-agent headers totally flout otherwise-consistent HTTP grammar rules, confounding well-written parsers).

<comment><USER4> Is this an unbiased argument for binary formats in general, or is it a sales pitch for Protobufs and Cap'n Proto?

If a fairly recent tool is the only way to get binary formats right, then it's entirely understandable that they've fallen out of use.

<comment><USER2> Obviously I am biased, but my intent is to make a technical argument, not a sales pitch. I don't make any money from these (and I no longer have anything to do with protobufs).

Protobufs was open sourced in 2008 (after being in use inside Google since around 2001). I suspect other binary formats prior to that had a solution for compatibility as well, but I haven't done a survey.

</comment></comment></comment></comment></comment><comment><USER5> The problem I see with binary protocols is the same one that plagues XML: as soon as you build something to be "not human-readable", any potential efficiencies gained will quickly be overshadowed by increasing bloat.

When you as a developer look at JSON messages, and you see endless walls of irrelevant text scroll by, you're disgusted. This disgust drives minimalism. With a machine-centric format, the excuse quickly becomes "oh, this is intended for machines anyway, so who cares". If you can access something through tools only, the bloat becomes hidden and is encouraged to grow.

At that point, you'll start seeing articles asserting that, sure, the new super-efficient binblob messages are 10x-20x as large as JSON used to be, but _look at all the things we gained_ , like automatic protocol negotiation, contracts, actual serialized objects. Any of these sounds reasonable at first but in reality will only benefit tool vendors in a vicious feedback cycle where the format slowly evolves itself to death.

I take that 3-5x overhead of parsing JSON any time over the non-human-readable alternatives. That doesn't mean it's the right choice for all protocols. But it's a reasonable default for a lot of systems.

<comment><USER6> XML was never intended as a memory-efficient format (or if it was, it was an utter failure before a single tag had been transmitted over the wire). I don't understand why you believe that binary formats naturally lead to increased bloat; while it may indeed be the case, your post largely reads like speculation to me. One could just as easily posit that the rigidity required of such a format makes it much harder to change (and examples supporting this viewpoint abound, including things like IPv4). Can you provide some examples of actual binary protocols that underwent this transformation?

<comment><USER5> _> XML was never intended as a memory-efficient format_

I never said it was. I apologize if it wasn't clear from my comment, but it was centered solely on the aspect of human readability. You're not wrong about it being speculation though, I just don't see the harm since we're talking about a speculative source article in the first place.

 _> Can you provide some examples of actual binary protocols that underwent this transformation?_

My entire comment was addressing an example where I felt a format had degenerated because it transitioned from a human-readable to a decidedly machine-readable form. My criticism is not based on the notion that binaries are bad in of themselves.

</comment></comment></comment><comment><USER7> I wouldn't be so quick to judge the energy lost by using JSON over some binary format as "waste". Standardization on JSON has saved countless developer hours by being so easy to read, write, implement, and debug* . This translates directly and indirectly into economic benefits for everyone. Similarly, I would have taken much longer to start learning HTTP if it was not inspectable with near-zero friction. People "waste" electricity for a reason.

* Is it perfect? No. Is any protocol? No. Would a binary format be better? Very likely, no.

<comment><USER8> Not to mention most importantly JSON is firmly on one side of the undecidability cliff in terms of parsing limits. The nesting limits and length limits make parsing it safe, whereas a given binary protocol may have recursion or buffer size definitions which make for ripe security holes. (Even XML has parser bombs)

</comment></comment><comment><USER9> The Internet _is_ running in debug mode. Because it's large and complex and humans have to _debug_ the problems. Humans need to be able to read the data as it whizzes by to spot the problem[1]. Often, this needs to happen in environments with minimal tooling; e.g. you're staring down a problem on a production server, and you're forbidden to install the newest analysis script and its requirement of the latest version of Python.

We've already agreed on a binary protocol: UTF-8 (previously it was ASCII.) But we've also built redundancy into it for the humans to make sense of it with their high-level brains. Instead of a single byte representing an HTTP header, we use a string of bytes. Now the human involved can tap the wire and watch the request in real-time without processing anything.

Now, if you'd like to remove redundancy without the need for a compression library, we'll just need to agree on shortening those strings. And we'll need a new diagnostic/parsing tool for each [binary] protocol that's invented -- unless you can convince the grep/sed/awk developers to add every protocol to their tools. Or maybe we could all agree on a single binary encoding for every potential combination of strings; something like an index into a dictionary. It might be better (i.e. higher compression ratios) if we let the computer decide on the dictionary for each message.

Do you see where this is headed?

1 - This, of course, is only the case until the machines can accurately gauge human intent and respond appropriately, preventing us from making mistakes to begin with.

</comment><comment><USER10> Let's not forget about TCP, IP, and all the other protocols involved in the internet that are binary by design. It's more than just HTML, JSON, and JavaScript over HTTP.

> Standardize on some simple encodings (pure binary, self describing binary > ("binary json"), textual)

Maybe like gzip [1], hpack [2], bson, or others?

I realize the point he's making about doing unnecessary work, but there's also a reason we haven't expanded human language past written characters or spoken syllables. It's efficient for our brains, and for preserving and transmitting knowledge.

There's just no way to create a binary format (character encodings aside) that can encompass all the possible ideas that can be communicated. Instead, the common text protocols eventually get optimized into binary (HTTP/2) without compromising the ability to express the rest.

[1] [https://www.ietf.org/rfc/rfc1952.txt](https://www.ietf.org/rfc/rfc1952.txt)

[2] [https://tools.ietf.org/html/draft-ietf-httpbis-header- compre...](https://tools.ietf.org/html/draft-ietf-httpbis-header- compression-02)

<comment><USER11> > Let's not forget about TCP, IP, and all the other protocols involved in the > internet that are binary by design.

It's funny that you mention IP and TCP. I's sad that we've degraded the end- to-end principle those protocols embodied. Today, it's not really practical to use IP protocol numbers other than 6 (TCP), 17 (UDP), and 1 (ICMP) and their IPv6 equivalents. Middleboxes, out of misplaced caution, reject packets that look unusual. We can't even use ECN. To work around this problem, we run everything over TCP and UDP.

That doesn't even work, though, because other middleboxes, working one level up the stack, reject TCP on anything but ports 80 and 433.

So now we run everything over multiplexed encrypted connections on one TCP port on one IP protocol. That's just silly.

The low-level protocol situation is like a fossilized idiom in human language. In English, we can that something "wreaks havoc", but what about "wreaks happiness"? Nobody understands "wreaks" anymore. It's no longer a productive rule. Likewise, IP and (arguably) TCP protocol details aren't productive either, except in specialized cases, or on closed networks.

The internet isn't just running in debug mode. It's also flinging around lots of bytes that do nothing other than reflect past aspects of its evolution, like an embryo's tail and gills.

</comment></comment><comment><USER12> The fact that the Web (not the "internet") runs in debug mode is directly responsible for its popularity and accessibility.

Anyone can look at a web page and learn how it works. Anyone can copy bits and pieces of code from various places and put it together into a web page of their own. Most "web programmers" manage to make a living without ever having to learn a complicated protocol or trying to figure out what a long string of hexadecimal digits means. Easy to learn = more casual tinkerers = more people learning how to code, at least at a basic level.

You could design perfectly efficient protocols and encoding formats, but if people don't use it, what's the point?

HTTP/2 is a nice compromise. The web server and browser abstract away all of the binary layers, so most programmers only need to care about human-readable text.

<comment><USER13> > Anyone can look at a web page and learn how it works.

Not so true anymore. It's true that HTML/CSS are easy to inspect on a browser, but the meat of most websites today (JavaScript) is heavily minified to the point that you'd be hard-pressed to call it plain text.

<comment><USER12> True, but minified scripts are often accompanied by sourcemaps. Technically sourcemaps don't need to be public, but they often are because they make debugging easier for in-house developers.

So even if not all websites are in debug mode, there are incentives to make it easy to get them into debug mode.

</comment></comment></comment><comment><USER14> I know I'm not making any big revelations here, but the funny thing is there's no "plain text" at all. It's just that our tools all know how to decode ascii. ascii bytes are just as non-human-readable as any other until they're decoded as ascii and displayed on screen. Another encoding could be just as transparent if all our packet sniffers, editors, etc, spoke it.

<comment><USER15> This here is really the only revelation that's ever made me think twice about text-encoded protocols. Someone else here mentions TCP, IP, etc, and it's true that it's pretty readable in Wireshark, when you spend a little time learning where to look for the information. BUT... You still essentially need Wireshark to read it.

</comment></comment><comment><USER16> The bit about global warming - come on. Reminds me of HBO's Silicon Valley. "And we're making the world a better place, through standardized binary encoded web protocols"

<comment><USER17> Especially since most request will originate from a Desktop or mobile client. For each requests, the client will surely display the result for a few seconds or even minutes, using way more energy than parsing those text protocols. Therefore, switching to a binary protocol can't make a big difference, not even for the comparatively small part of human energy consumption that's used for computing devices.

</comment></comment><comment><USER18> From last year (Oct 2014) which describes textual protocols as "debug mode" since they consume cycles to parse.

Its an interesting claim, and it is certainly true that encoding numeric information into UTF8 consumes CPU cycles. But what isn't quite so clear is "What percentage of packet latency is dedicated to encoding and decoding packets?"

Back in the old days I was the ONC RPC architect for Sun and we spent a lot of time on "RPCL" (RPC Language) which was a way to describe a protocol textually and then compile that description into library calls into XDR (the eXtensible Data Representation). We did that because you burned a lot of CPU time trying to parse a C structure out of the network, and more importantly the way in which it was represented in memory was an artifact of the computer architecture (bit endianness, did structures get packed or were they word addressable, etc) XDR solved all of those problems by putting data on the network in a canonical format, and local libraries could always convert from the canonical format into the local format correctly.

That actually works quite well. It almost became the standard way to do things on the Internet, but politics got in the way. The big argument was that if you converted things into big-endian form on the network, then a little-endian processor had to convert to send and convert to receive, but a big-endian processor got a free pass without "painful" conversion steps.

Later, rather than converting big endian to little endian people just convert to text (which has the same effect of a canonical form) but it hides the religious argument behind the "hey its just text, we all know how to parse text right?" sort of abstraction. At least then it penalized everyone equally.

But the truth, which came out in the RPC wars, and is even more true today, is that you have to burn a few billion CPU instructions to have any impact at all on latency. That is because computers are so much faster and the network? While faster isn't a million times faster, it isn't really is just barely, and only on a good day a hundred times faster than it was back in 1985. What that means in principle is that if it takes 1 uS or 30 uS to queue your packet for the internet it doesn't even show up on the carve out of the 5,000 uS it takes to send a small packet from here to there, or the 200,000 uS it more typically takes.

If you're a supercomputer sending data around to simulate fluid dynamics, that stuff adds up. If you're sending ajax calls from here to there, not so much.

<comment><USER19> > Back in the old days I was the ONC RPC architect

I just recently used ONC+ for controlling a low spec, low-as-possible-latency, embedded Linux controller. The ridiculously low memory footprint and CPU usage, compared to all of the others, along with the whole 5 minutes it took me to write the server and client stubs with rpcgen made it an easy winner. So I thank you. :)

</comment></comment><comment><USER20> We have realized, long ago, that computers are cheap and programmers (the humans who most often read "human readable" stuff) are expensive.

We also tend to forget, or never learn, what was discovered before our careers began.

<comment><USER21> Could not agree more.

Not only hardware is cheaper, but is much more easily replaceable, have a well defined behavior and usually don't go nuts and need to take prescription drugs. Also they mostly don't have a life and family outside work.

This is why technology usually converges to serve the humans, not the machines.

Or else we would all be writing assembly code instead of garbage collected, duck typed and compiler aided languages.

</comment></comment><comment><USER22> I used to use Sun XDR [1] for this reason. Pushed Juice [2] applets briefly while they existed with efficiency neither JS nor Java delivered. Other benefits authors didn't see, too. Plus, client-server or P2P architecture with native code gave me performance, portability, and security benefits Web can't deliver.

Whole Web is ridiculously inefficient. Even in 90's, there were better architectures [3] to choose from. It's unsurprising that "Web" companies such as Facebook have moved back to Internet apps where possible (esp mobile) and often avoid vanilla Internet/Web protocols within their datacenters. There's better stuff in every area. Here's to hoping that more of it gets mainstreamed.

[1] [https://en.wikipedia.org/wiki/External_Data_Representation](https://en.wikipedia.org/wiki/External_Data_Representation)

[2] ftp://ftp.cis.upenn.edu/pub/cis700/public_html/papers/Franz97b.pdf

[3] [http://www.cs.vu.nl/~philip/globe/](http://www.cs.vu.nl/~philip/globe/)

<comment><USER23> [http://cr.yp.to/sarcasm/modest-proposal.txt](http://cr.yp.to/sarcasm/modest- proposal.txt)

<comment><USER22> That's hilarious. Strange enough, there's people in the country without access to fast Internet that would've appreciate proposals like this to shave much time off their email downloads. So long as the client automatically decompresses the message during viewing. I think we can settle for a more conventional algorithm for that, though. ;)

</comment></comment></comment><comment><USER24> Actually, the internet is dominated by binary. The wire formats are binary, and a huge hunk of all bandwidth is being eaten up by binary formats: video, bittorrent, voice, JPG images, etc.

Sure, HTML, CSS and JS are "text", but it's usually optimized, compressed text, and pretty soon we'll be shipping that around in HTTP2.

People optimize what matters. Look at the HTML source for this page. It's mostly user content. More efficient packaging for the HTML structure wouldn't make a measurable difference in load time.

</comment><comment><USER25> Google has an elegant solution for this in the form of protocol buffers, which are human-readable when expanded but compress down to a very efficient binary format:

[https://developers.google.com/protocol- buffers/](https://developers.google.com/protocol-buffers/)

Unfortunately it was made public only after JSON had already become "the XML replacement".

</comment><comment><USER26> The more readable and accessible data is, the more "future-proof" it will be. Unless you have a specific requirement for performance, it is never a good idea to sacrifice accessibility. JSON has won the developer community primarily because it is simple, readable and easy to implement.

</comment><comment><USER27> Accessibility and ease of use contribute to the advancement of technology. The easier something is to work with the more people will use it.

I used to be obsessed with performance and refused to write in anything other than assembly. When I started working with teams of other people I realize there is efficiency for the computer and there is efficiency for the developer and the company. The latter 2 trump efficiency of the computer pretty much every time.

I'm more than happy to sacrifice some efficiency if it means getting the job done faster, cranking out more features, greater compatibility, more leverage with existing tools / standards, and so on.

If you want to talk about efficiency though. Why don't we get rid of time zones and daylight savings time?

<comment><USER28> Also, I like the idea of someone getting interested in coding because made something that is interesting to them and in unminimized javascript, for example. I know when I started digging into things it was mostly inspired by random things like that; they didn't even have to be great, they just had to be _there_ at the moment. First I experimented with stuff others made, then I learned, and if playing around with stuff hadn't been that fun, I might never have gotten to the learning more part.

</comment></comment><comment><USER29> > 99% of webservice and webapp communication is done using textual protocols.

Yep. <img src="whatever.jpg"> is indeed all text. and zip would have turned those 25 bytes into 191. And then a 300KB highly-compressed binary data file would be transmitted.

aka "Penny-wise and pound-foolish."

</comment><comment><USER30> One of the many nice things about canonical s-expressions[1] is that they combined many of the features of binary and textual protocols. Fast to parse, easy to write a parser for, easy to examine by hand, they were ahead of their time.

[1] [http://people.csail.mit.edu/rivest/Sexp.txt](http://people.csail.mit.edu/rivest/Sexp.txt)

<comment><USER22> Good points. I used them as a text data structure in the past. Saying the parser was easy to write is an understatement: years of papers and code from LISP/Scheme community had done it dozens of times. Even in hardware. We used it for architecture-neutral data format and mobile code for agent-oriented programming. Fun times.

I've considered going back to that in custom designs. It's just that Galois built a high assurance ASN.1 parser, INRIA has a verified parser generator, and ZeroMQ is pretty solid. Too many neat things to consider these days... haha

</comment></comment><comment><USER27> To counter my earlier statement though I think there are scenarios where you can have your cake and eat it too.

JSON is easier to use, debug, and is more efficient than XML.

WebAssembly is going to reduce data transfer sizes and load times while increasing developer productivity as the ecosystem and tools surrounding it expand.

<comment><USER13> > WebAssembly is going to reduce data transfer sizes and load times while > increasing developer productivity as the ecosystem and tools surrounding it > expand.

That's being a little too optimistic. And assuming that does happen, it's going to take years before it surpasses the current toolset.

</comment></comment><comment><USER31> I just recently implemented something using binary websockets, so it's easily possible now.

</comment><comment><USER32> So he wants HTTP2?

</comment><comment><USER4> Isn't it the entire fact that the Web uses plain text for everything from html to css, to javascript and json, that contributed so enormously to its success?

Suppose HTML had been a binary format. Would it have gotten anywhere near as far?

</comment><comment><USER33> JSON+UTF-8 is just another binary encoding...

CBOR is great, it's similar to JSON & already an RFC [http://cbor.io](http://cbor.io)

</comment><comment><USER34> "Human society is running in debug mode!"

</comment><comment><USER35> > textual encodings like xml...have become very popular

Written by someone with "Java" in the URL. Why did I even click.

</comment><comment><USER15> You can read the author's inexperience between every single line.

<comment><USER36> Perhaps. But if you know more, why not teach us? A good HN comment would convey relevant information and not be snarky.

<comment><USER15> What's there to say to this person that clearly has made a decision that they're going to stand by forever? People don't change their minds even in the face of reason, let alone their religious convictions. I'll be downvoted for this comment, but there is absolutely no point is trying to reason here, because of the barrier of human nature, and what I'm saying is the truth, unfortunately.

<comment><USER36> It's not only a question of convincing the other person but of providing clear information to everybody else.

Perhaps you feel there's no point to doing that either, but then the only HN- appropriate thing to do would be to just not comment. Posting empty swipes doesn't help anyone, and breaks the site guidelines.

<comment><USER15> I've posted two other comments in this thread with the information you're looking for. I don't think it's an empty swipe. It's how I feel about the content in the link. Part of discussing technology is using social cues to reinforce or reprimand certain kinds of behavior, just as you're doing now. If some new person came to this thread and saw nothing but chipper, positive, glowing comments about this guy's wonderful insight, there would be nothing here indicating the enormous friction around this very topic. Comments like mine reinforce that the community needs to inform themselves about a topic before deciding for themselves what is the best way forward. If we stumble blindly into the future, coddled by the happy comments we are allowed to post, innovation suffers and ultimately dies. This is about much more than just negativity in posts. I hope what I've said makes some kind of sense.</comment></comment></comment></comment></comment>


Laboratory Notebook Skills [pdf] - <AUTHOR> https://www.dur.ac.uk/resources/physics/students/labs/skills/notebookskills.pdf <comment><USER2> I started using Org mode in emacs. I use it to record the next tasks (Todos), expected results, and actual results. Its damn convinent that I can run code inside org file and say "cat" out the result of an experiment output as JSON by another script into my notebook. I can even embed and run multiple languages using org-babel.

I think I will be coming up with workflows and checkboxes soon for organizing a experiment and easily integrating results of the experiments I am running from disparate software.

I think its very easy to incorporate the stuff in this post, and create a new template in org mode for this. And its so flexible that it will fit all usecases.

Emacs is too awesome.

</comment><comment><USER3> Have we reach a point where there are similar guides for Jupyter notebooks? Or just for the domain of data science / exploratory statistics

Like, conceptually, what should I keep, and how should I organise my notebook(s)?

Most guides are focused on what you can do (Here's how you draw a graph, here's how you load a kernel), I'd rather read about what I should store in my notebooks, and how many I should have.

<comment><USER4> This is something we think a lot about at Kyso ([https://kyso.io/for- teams](https://kyso.io/for-teams)) - my startup - we let teams manage analytics knowledge basically by letting teams share Jupyter notebooks - we've spoken to lots of people about how the organise them and are writing a guide really soon that sounds like what you want.

\- Keep all your teams notebooks in one place (not on laptops)

\- Reuse them liberally (the %run command is excellent and often better than creating python libraries)

\- Most important of all is make them readable documents not just list of coding commands, that means you should title things clearly, explain what you going to do.

\- Label everything, axis labels are needed on almost every chart, and also make sure you put a title and description for every chart.

\- Separating your project into processing and presentation notebooks is really helpful. So one notebook for data prep and another for plots and explanations.

<comment><USER3> > Reuse them liberally (the %run command is excellent and often better than > creating python libraries)

Does that mean you run other notebooks from within notebooks?

Can you ping me or something when the guide is out?

<comment><USER4> Happy to, but I've no way to ping you?

</comment></comment></comment></comment><comment><USER5> Everyone's focusing on charts and notebook types; I'd like to highlight something I find particularly important: reporting uncertainties. I strongly agree with recommendations in this article, and personally I don't think a work qualifies as professional if it reports measurements without providing uncertainties. And no, I don't trust it's correctly rounded off (no more digits than justified by certainty level) - too many people don't care or know they should care, and too many areas of endeavor are poisoned by marketing.

A trivial every-day example of why that matters: a person steps on scales twice in a span of few days, and notices 1 kg of gain, and panics. The reaction is completely unwarranted, given that a cheap household electronic scale will be accurate to +/\- 0.5kg in this range, and random daily variability of weight is somewhere around +/\- 2 kg. They may as well have lost their average weight over that span, but it's hidden under random and systemic errors.

</comment><comment><USER6> My version of this, which i maintain to this day, is a git repo of org-mode documents with excecutable inline snippets of code and figures.

It is append-only; i don't delete stuff, usually, i just modify the heading if it's obsolete. (it's easier to read ~text~ than it is to dig through commits)

</comment><comment><USER7> "Draw your figures in Excel" No no no no no no no! Do not draw your figures in Excel. Yes, they can be drawn quicly, but Excel figures suck, are a pain to customize and will introduce bad habits. Learn to draw figures in R using ggplot2.

<comment><USER8> Pure opinion until you explain why Excel figures suck and which bad habits it introduces?

<comment><USER9> Not the person to whom you were responding.

I think Excel charts can be quite good. But my impression is that ggplot2 is much more flexible in what you can do with it. It's also more difficult to learn, but that's to be expected.

</comment></comment></comment><comment><USER10> > The Level 1 Lab book is a bound A4 notebook

I would add that it should be acid-free paper, very strong binding (sewed+glued) and have graph paper. The acid-free paper can give notebook 20+ year life span, strong binding is important as you may frequently force notebook open with heavy objects, and finally graph paper style allows you to draw complex diagrams easily. Here's notebook I've liked: [https://www.amazon.com/gp/product/B071NWSB7P](https://www.amazon.com/gp/product/B071NWSB7P)

</comment><comment><USER11> Why do they recommend a paper record that can't be backed up and nobody can verify what it said when? Why not a digital file that can be backed up and with a published checksum that can be used to verify what the state was at a given point in time.

<comment><USER12> Digital notebooks (Electronic Lab Notebooks --- ELNs) exist, but haven't caught on yet.

Many researchers often need to be able to note down arbitrary diagrams, not just text, in real time, which pretty much means low-latency tablet with stylus. This became feasible in just the last decade or so.

Many ELNs do implement published hashes for verification ("trusted timestamping").

<comment><USER13> I personally understand better my own ideas (it souds weird i know) when i draw graphics and diagrams, that can be done quickly and easily on paper.

</comment></comment></comment><comment><USER14> would love to hear input from people with both lab and software dev experience to compare and contrast the two notetaking experiences

<comment><USER7> These are excellent suggestions and are taught in science classes. Except for the use of Excel. Yes, Excel is easy, but its flexibility will lead to sloppiness, and drawing figures sucks. Excels statistical functions are also wrong in some cases. For data analysis, learn R or Python, period. If you have lots of data, learn to use SQLite in addition. The learning curve is steep, but well worth it.

Source: I have a decade of experience in science, and some 5 years in software development.

<comment><USER15> So I had the opposite experience-- learned to do all my data processing in undergrad+grad school for physics using Python. Moved out to my first industry job developing simulations and learned that everyone (other scientists, management, etc) would rather me process results in excel (unless we were working on a database scale, in which case we used postgres). I actually had to learn excel properly for the first time for this job.

I'm now at another large university-affiliated research lab and excel is king here as well, though I can get away with using Matlab generated plots in my slides when I'm working solo. People still don't like python for some reason.

I had a similar experience with paper writing-- in academics it was conventional to do everything in LaTeX from my first lab courses in freshman year. In both workplaces, we've just been using word.

And it's not that people don't know python/latex here-- we've just apparently developed a culture of using these matlab/excel/word tools instead.

</comment></comment></comment><comment><USER16> I appreciate codifying it in a document, but most of the rules listed are taught (at least implicitly) in middle school.</comment>


EFSA updates Glyphosate toxicological profile - <AUTHOR> http://www.efsa.europa.eu/en/press/news/151112 <comment><USER2> In summary; this was a far less rigorous review than IARC which found the opposite.

In arriving at its conclusion, EFSA relied heavily on the glyphosate assessment conducted by German pesticide regulators, which was based on egregious violations of accepted standards for interpreting studies. For instance, faulty statistical methods were used to reject clear evidence that glyphosate causes cancer in experimental animals.

Second, EFSA only considered studies on the active ingredient, glyphosate, whereas IARC also considered studies that were done with various formulations of the herbicide, in which glyphosate is mixed with other substances that can be toxic in their own right, or make glyphosate more toxic. Such formulations are what farmers and consumers actually use.

Third, EFSA assessed some studies provided by the pesticide industry that IARC rejected because they lacked critical information required for proper interpretation, and/or were not publically available.</comment>


Show HN: A bot that notifies you when your limits for ETH, BTC are reached - ro-_-b https://telegram.me/cryptocapbot <comment><AUTHOR> Nice! Looks useful</comment>


Making 30-Year-Old Pascal Code Run Again - <AUTHOR> http://troydm.github.io/blog/2014/01/26/making-30-years-old-pascal-code-run-again/

<comment><USER2> "Well, apparently back in the 80's you could jump between procedures in Pascal using goto statements, some primitive try/catch mechanism."

You can jump between procedures in GNU Pascal. [http://www.gnu- pascal.de/](http://www.gnu-pascal.de/)</comment>


Is this the wallet of the future? (kickstarter) - <AUTHOR> http://www.kickstarter.com/projects/1404403369/geode-from-icache?ref=live

<comment><AUTHOR> I've been looking at this for a few days and I really like it, it's a bit of a risk for me as it seems to be a US product and I'm in the UK (not sure if it works with chip and pin)</comment>


Apple to EU: Set FRAND Licensing Standards - <AUTHOR> http://www.macobserver.com/tmo/article/apple_to_eu_set_frand_licensing_standards/ <comment><USER2> This article, and the Florian Mueller source, both seem to be making the case that Apple's proposed solution is morally superior. In fact, like many things related to patents and standards it just seems to be tailored to the interests of the large incumbent firms. In this case limiting fees to component prices so high margin producers like Apple don't have to pay more. It's the same game as capping yearly H.264 fees so big players pay less per unit.

Standards shouldn't be allowed to have patents on them period. That's the core issue that leads to this political bickering. Standards get their power from being standard and everybody using them, not from being technically advanced. Patents are all about profiting by stopping people from using the technology unless they pay. There's a fundamental disconnect there.</comment>


Show HN: Track and Control Your PDFs - <AUTHOR> https://www.helprange.com <comment><USER2> The only two questions that matter: 1\. Why is the PDF reader running active or remote content without prompting? 2\. What PDF readers do not respect my privacy?

<comment><USER3> Yeah, why does a document format have the ability to run code and contact the network without the user's knowledge or permission? (glares at the web)

</comment></comment><comment><USER4> I was CEO and co-founder of doxIQ ([https://www.crunchbase.com/organization/doxiq](https://www.crunchbase.com/organization/doxiq)), a similar product. We were acquired by Nitro, another PDF software company, after a couple of years. DocSend started around the same time as us and has become the standard for final format document tracking.

My suggestions:

\- read analytics are valuable, but it's hard to get people to pay for that alone given how many other products offer it for free

\- pick your target user persona (SMB vs enterprise) and focus on making them happy

\- focus on integrations with other systems that your target persona uses, like CRM/marketing automation/email/etc

\- ignore shiny object syndrome. you'll be tempted to add more analytics tools or insights to "add more value." while YMMV, i've found this to be a waste of time.

<comment><AUTHOR> Thanks, quite interesting tips!

</comment></comment><comment><USER5> Can't seem to find any information which PDF readers are supported (read - that fetch external resources without explicit user's consent). I suspect that not that many...

Edit - in fact, the site badly needs "How it works" section, because otherwise it reads like yet another "self-destructing emails" type of product.

<comment><USER6> Pretty sure you don't open the PDF in a reader - in fact it looks like there is no "resulting PDF". Instead the PDF is hosted on the site, which you link to, and the analytics are run on the page.

<comment><USER7> At which point, why even bother with the PDF... just host your content on any web site, and run Google Analytics.

</comment></comment></comment><comment><USER8> Seems like a good tool for professors at universities. For some classes my professors send out quizzes along with (mandatory watch) lecture videos (3-4 min) long and they track how much of it have you actually watched. Mostly to see if a student is actually putting up effort or simply copying.

</comment><comment><AUTHOR> Hi,

my name is Wiktor. Together with my friend Tom, we built HelpRange, tool to help you track and control your PDFs. What do you think about the idea? I would love to get feedback!

Best, Wiktor

<comment><USER9> Can you confirm that the end result is no longer a PDF, but content hosted on your site?

Or rather a PDF hosted on your site that the user doesn't copy to their computer but views on your site in their browser?

<comment><AUTHOR> Yes, but there is optional possibility to allow for downloading the file, which we can't track in this case

<comment><USER10> So people will screenshot it if they want a copy?

<comment><USER10> Sorry didn't read download option :/</comment></comment></comment></comment></comment>


Ask HN: DNS service with an api - <AUTHOR>

What's a good DNS hosting solution that has a good, straightforward api. (I find route53 to be an api headache) <comment><USER2> I use <http://www.rackspace.com/cloud/dns/> Free RESTful API with language bindings.

Api docs: [http://docs.rackspace.com/cdns/api/v1.0/cdns- devguide/conten...](http://docs.rackspace.com/cdns/api/v1.0/cdns- devguide/content/index.html)

Python examples: [https://github.com/rackspace/pyrax/tree/master/samples/cloud...](https://github.com/rackspace/pyrax/tree/master/samples/cloud_dns)

</comment><comment><USER3> www.dnsimple.com</comment>


Faster Fene: A New Dawn for Marathi Cinema - <AUTHOR> https://medium.com/@ashnehete/faster-fene-a-new-dawn-for-marathi-cinema-534b301aab87 <comment><AUTHOR> A recommendation to all those film aficionados out there who love to explore foreign language films.</comment>


Password generation and safe offline storage - <AUTHOR> https://www.hackthis.co.uk/articles/password-generation-and-safe-offline-storage

<comment><USER2> Nice to see some information that is clearly from a security expert, who has real world experience of using such tools to try and break encryption.</comment>


What Was the Greatest Era for Innovation? - <AUTHOR> http://www.nytimes.com/2016/05/15/upshot/what-was-the-greatest-era-for-american-innovation-a-brief-guided-tour.html?_r=0 <comment><USER2> 1870-1920. Electric light, automobiles, aircraft, radio, paved roads. Railroads everywhere in the developed world. Electric mass transit (trolleys, subways) in major cities. Tractors and combine harvesters on farms. Refrigeration. X-rays.

In computing, the Computing-Tabulating-Recording company (later IBM) produced the first printing tabulator, the beginning of commercial data processing.

Those 50 years held the greatest lifestyle changes in history.

<comment><USER3> I'm constantly amazed how many not well-known things were invented back then that sound like something that came only recently, for example sending photos over the wire - some time ago I saw a video [1] about wirephoto [2] that blew my mind a little, I would never guess that the first photograph sent over the wire was in as early as 1921!

[1] [https://www.youtube.com/watch?v=LetlcmqZFyA](https://www.youtube.com/watch?v=LetlcmqZFyA) [2] [https://en.wikipedia.org/wiki/Wirephoto](https://en.wikipedia.org/wiki/Wirephoto)

<comment><USER4> Any reason homes with phones back then couldn't have "emails", in the sense of being able to receive text messages from other phone numbers and automatically print them to a paper feed?

<comment><USER5> Well you had things like ticker tape.

but there was not really any concept of store and forward like you have with texts and email.

Notice how they had to tell the operator to make sure there was a clean line, then manually sync the two devices.

I guess you could claim the telegraph network had something akin to store and forward, as each operator would note the message, and then relay it down the chain.

<comment><USER2> Edison's first profitable invention, in 1869, was a way to get stock tickers to sync up automatically. It was a hack (send lots of idles every once in a while to force the printing wheel to hit a stop) but it was good enough to deploy. Previous printing telegraph technology, back to 1852, required a skilled operator at the receiving end to manually sync. That was used only for main lines with heavy traffic.

Howard Krum finally figured out how to do sync right in 1919, when he invented serial stop-start async as we know it today. In 1924, he came out with the Teletype Model 14 tape printer, with the first reliable mechanical async decoder. That design was used into the 1960s, and most computer serial ports are compatible with it. (45.45 baud, 5 bits, no parity, 1.5 stop bits.)

Telegraphy was store and forward, but manual at first. By the late 1920s, punched tape was used, but someone had to manually look at tape, tear off each message, and put it in the proper outgoing reader. This was manual Sendmail. After WWII, automated forwarding centers were built, with paper tape punches connected to readers via a bin as the storage mechanism and telephone-like switches to change connections. These were hugely expensive to operate for the amount of data transferred, but it was essentially email, with electromechanical technology.

(I restore Teletypes as a hobby. Here's a Model 14 working.[1]. If you want to see some in action, they'll be at the Clockwork Alchemy Steampunk Convention, Memorial Day weekend, in San Jose.)

[1] [https://archive.org/details/Aethericmachine14](https://archive.org/details/Aethericmachine14)

<comment><USER5> Thinking about it, i guess magnetic tape replaced those paper tapes for a while. Before various "solid state" solutions were worked out with enough capacity to handle the same job.

</comment></comment></comment></comment></comment></comment><comment><USER6> In terms of pure impactful science I'd say 1920-1960 or so. Quantum mechanics and everything it led to (modern chemistry, in particular) is the big one, but there was also relativity, the modern evolutionary synthesis, huge advances in statistics, the math of computing, and so much more.

<comment><USER7> I agree. The important advance in the first half of the 20th century was understanding how atoms work, and that knowledge lead __directly __to semiconductors, the nuclear industry, and laid the foundation for modern chemistry, materials science and all the analytical technologies than enable chemistry, physics etc (i.e. x-ray analytical techniques, mass spectrometers).

Something that I did not know until recently was that the "mineralogy" of steel was not understood until the 20th century. If you do not understand the phases comprising a material, it is difficult to relate bulk material properties to bulk composition.

</comment></comment><comment><USER8> The push for innovation is probably the greatest right now. However, I like to think the greatest innovations are those that make my own life simpler. There's: airplanes, cars, electricity, GPS and others. Innovation today, while technically impressive, is mostly making my life more complex, not more simple.

<comment><USER9> The average Joe see innovation as stuff like Facebook and snapchat...meanwhile research is losing money...

</comment></comment><comment><USER10> The printing press, and its proliferation a few hundred years ago.

Without it, the cost of copying information is enormous. When you can't copy stuff cheaply, you end up copying only a few things. Like what whoever is in power likes. Or whatever the accepted wisdom of the day is.

With it, you can have ideas, spread them, and get feedback from a lot of people. That goes for social ideas like liberalism, religious ideas like Protestantism, scientific ideas like gravity, and technological ideas like all the ones on HN.

You also get away from authority. If you have some new thoughts about how government should be organised, it's a lot harder to stop you telling everyone.

<comment><USER11> I agree that information (communication) is fundamental to human society. But I don't agree that information technology is a force for equality and decentralization of power. Over the course of history it's been the opposite - papyrus enabled the the expansion of the Roman empire, the nation state could not exist without paper and printing, and digitalization is creating larger multinational corporations than was possible before, and we will soon see states evolve to a version where it has even more control over citizens.

<comment><USER12> This is an excellent oberservation but I would point out that all of your examples are of one group having a large information technology advantage over another. I think the situation _could_ be different as very powerful IT becomes as widespread and inexpensive as seems to be the trend. Perhaps there is some point where the worst IT is good enough that it's hard to parlay a tech advantage into any form of oppression and at that point it's a tool for equality and decentralization.

</comment></comment></comment><comment><USER13> In practical terms of impacting the greatest % of the population, some data that I'd like to have seen here are around life expectancy:

\- Life expectancy, men/women

\- Infant mortality

\- Accidental death

\- Death by illness/disease

\- Death by military conflict

For example:

\- Life expectancy (US, men & women average)

1870: 35.1 years

1920: 54.1

1970: 70.9

2012: 78.7

The most years gained were 1870 - 1920.

As for military conflict, innovation happens in politics and diplomacy as well as technology. I'd like to see how innovations in politics and diplomacy have impacted death by military conflict as military technology become increasingly lethal.

<comment><USER14> Gordon actually has some _really_ good chapters on healthcare and medicine, among the very best parts of his book. He's also absolutely nuts with graphs and tables, and covers much of this in detail.

TL;DR: 2x the improvement of life expectency came _before_ 1950 as after. Medicine since 1970 has been expensive, high-tech, changed rapidly, and delivered very little. Much the improved outcomes arises from improved access for the underserved, not radical treatments. Genomics and stem cells are specifically and pointedly disappointing.

</comment></comment><comment><USER15> You're living it.

</comment><comment><USER16> The future of innovation is micro, not macro - genomics and biotechnology, apps and on-demand entertianment, miniaturization and nano technology, etc.

The past always seems better than the present, probably we're not good at assessing our progress, but only retrospection can we.

</comment><comment><USER17> Probably depends on what constitutes innovation and the time scale. It could be argued that language, agriculture, money, etc. have changed people lives than more dramatically than recent technology.

</comment><comment><USER18> Hopefully getting the Internet in the hands of many non-technical people with Smartphones and tablets will get us to another Era of Innovation. Many people started using the Internet that wouldn't have it it was only available on PCs.

The "born after the iPhone" children will get old enough next decade to do their own thing and new ideas will spread :)

<comment><USER19> >Hopefully getting the Internet in the hands of many non-technical people with Smartphones and tablets will get us to another Era of Innovation...

This seems to be a fallacy. Give internet to 10 random people and 9 of them will use it for Porn and Social media. You know, instant gratification stuff.

The one guy who uses it to "expand their horizons" will find himself too alone and powerless in a culture that has gone south by the the aforementioned 9 people who wouldn't take their heads up from the social media in their mobile phones....

<comment><USER18> Well, but if one of 1000000 people does something on the scale of google, facebook or wikipedia, this would be enough.

With the internet, ideas scale much better.

Before smartphones, we nerdy white males made stuff.

Now the rest of the world can take over :)

<comment><USER19> Well, is it possible that problems solved by those things were not the real bottlenecks of real innovation.

But they took away something that had brought about innovation in the past and had a very good chance of brining it in the future....idle human thought.

</comment></comment></comment></comment><comment><USER20> Across human history, it's probably 10,000 BC to the present era. Within that period, the sliver of time occupied by the Second Industrial Revolution, as a catalyst for accelerating growth and progress henceforth, hasn't yet been bettered by any other period for the scale of its impact.

</comment><comment><USER21> 750–950 The Islamic Golden Age. Most relatable to HN was the work of _al- Khwarizmi_ who was one of the fathers of algebra and whose name rendered as _Algoritmi_ in Latin led to the term " _algorithm_ ".

<comment><USER22> That's 200 years. Consider how US society has changed in the last 200 years.

<comment><USER23> The last 200 years are special. We've gone from 1 billion to 7 billion people. It's hard to compare that with a period of time when there were only ~200 million people.

</comment></comment></comment><comment><USER24> I didn't rtfa but if I recall correctly, patents per capita peaked in the 1860s

<comment><USER25> Patent lawsuits per capita today probably exceed that.

</comment></comment><comment><USER26> 200000 years ago? The invention of language.

</comment><comment><USER14> As I should have suspected from the context, this concerns Robert Gordon's epic _The Rise and Fall of American Growth_ , which I've read and am trying to synthesize a review.

The article here suffers from much one of the book's major faults: it's an epic romp through the _details_ of technological advance, and their impacts, _both_ of which are admittedly interesting, but lacking in the backbone of theory and a model to explain both _what_ technology is, specifically (one definition, from Gordon, is "the sum of our ignorance", though that requires further explanation ;-), and _how_ it impacts everyday life.

Don't get me wrong: Gordon's grasp and exposition of history is masterful, detailed, and extensively researched. Unlike much of economics, he looks _pointedly_ at every-day impacts, and strongly questions many (though not all) the often-questionable interpretations of economists.

I also agree strongly with Gordon's conclusion that _technological progress_ hit an apex _prior_ to 1920. More specifically, and Neil Irwin does a huge disservice in not mentioning this, Gordon specifically distinguishes _rate of change_ and _impacts_. While _technology_ has changed markedly in his third era (the book is divided in three parts: 1870 - 1920, 1920-1970, and 1970-2015, nearly 50 years each), the _impacts_ have been quite limited. Little in household appurtenances, particularly kitchens, has changed since 1970, other than introduction of the microwave. Transport's pretty constant. Medicine, for all the expense poured into it, has delivered virtually nothing, particularly contrasted with 1900-1950 (and, for what it's worth, earlier). Work and manufacture has changed, but only modestly, and _productivity_ as measured by economists (dollar output per worker hour) has increased at best modestly.

And, sorry, HN, but Gordon's quite harsh on information technology -- it delivered, somewhat, from 1994 - 2004, but little 2005 - 2015.

That's all covered in the book, and missing from Irwin's article.

What Gordon himself misses is IMO even more interesting.

His core thesis is what's called "the Solow Residual" \-- what's left over in regression analysis after accounting for economic growth using capital and labour. (Gordon was a student of Robert Solow's). What bothers me, _greatly_ , is that Gordon all-but-answers his own question -- he points at the 99% growth in worker productivity 1920-1950, and the 133% increase in horsepower and kW per worker over the same period. On numerous grounds, power makes sense to consider in transformational activity.

Gordon completely dismisses it.

Several other researchers have dug into that in depth. Robert A. Ayres (apparently it's a rule for all economists in this saga to be named "Robert") is one, there've been other analysis, including Hall and Klitgaard, and a recent item I've seen. Solow and Ayres had a correspondence according to Solow's archive at Duke University, though I don't know the upshot of it.

Another element missing from Gordon's analysis is some sort of theory of technology -- what it is, how it operates. He describes this to some extent, but it's quite insufficient. He mentions the real-world, everyday-life impacts of many early technologies, but doesn't think to reference Maslow's Hierarchy of Needs, another lacking point.

Along with that is a _metric_ of technological impact. Gordon's is GDP and especially GDP/worker. This is actually and area of some disagreement in various circles, but a concept of energy throughput has been proposed by Leslie White, known as White's Law.

I could also quibble with some economic discussions and considerations, though most are quite secondary to his point and if anything would, corrected, _strengthen_ his basic thesis.

But yeah, today's got nothing on the 1880s, progress-wise.

</comment><comment><USER27> Now.

<comment><USER28> I guess that means innovation ends now too.

</comment></comment><comment><USER29> The golden quarter..

[https://aeon.co/essays/has-progress-in-science-and- technolog...](https://aeon.co/essays/has-progress-in-science-and-technology- come-to-a-halt)

<comment><USER30> Maybe I'm being overly cynical, but I see the choice of years as another effort to put the 1960s on a pedestal. If I was being even more cynical, it wouldn't surprise me if the article was written by a Baby Boomer.

You could pick any era in the 20th century and find multiple examples of world changing innovation, and there were other periods of rapid change further back in our past as well (the Industrial Revolution springs to mind).</comment></comment>


Experts recommend Machine Learning books - <AUTHOR> https://mentorcruise.com/books/ml/ <comment><USER2> For a survey of AI and ML I really liked "Artificial Intelligence – A Guide for Thinking Humans" by Melanie Mitchell. I've written a summary of it here: [https://henrikwarne.com/2020/05/19/artificial- intelligence-a...](https://henrikwarne.com/2020/05/19/artificial-intelligence- a-guide-for-thinking-humans/)

</comment><comment><USER3> The list is decent, but not exactly original.

For people w/ a physics background, I would still recommend [https://www.inference.org.uk/itprnn/book.pdf](https://www.inference.org.uk/itprnn/book.pdf). Some of it is a bit obsolete, but then DL made a lot of stuff around generalization/overfitting somehow obsolete. It makes a lot of connection between different kind of approaches in ML, information theory, (Bayesian) statistics, and physics.

It is not a very good book if you only care about applications (in which case the Keras book, for beginner, or fastai/etc. are much more appropriate).

<comment><USER4> David MacKay was an absolute rockstar and this book is grossly underrated among beginners in machine learning. This should be THE complementary reference for anyone who uses the Bishop book. Both these books follow a philosophy which some people may not completely agree with, that from the "church of Bayes".

My guess is that information theory went through its hype phase and much of the ideas are so pervasive across real systems that people forget how important those connections are.

To tell you its importance, skim this work on information-theoretic probing [1]. I find this so satisfying. Its most famous alternative, linear probing, always felt inelegant. This paper experimentally shows how terribly linear probing fails.

[1] [https://arxiv.org/abs/2003.12298](https://arxiv.org/abs/2003.12298)

</comment></comment><comment><USER5> For NLP I would maybe add

Speech and Language Processing From Dan Jurafsky,

Available at: [https://web.stanford.edu/~jurafsky/slp3/](https://web.stanford.edu/~jurafsky/slp3/)

</comment><comment><USER6> Slightly crestfallen that the "ML" here is machine learning and not the programming language. I still refer to my vintage paperback of L. C. Paulson's _ML for the Working Programmer_ from time to time.

<comment><USER7> I still read NLP as Neuro-linguistic programming. Every. Time.

<comment><USER8> So I have a little disambiguation heuristic:

Neuro-linguistic programming definitely had its day. It seems to have fallen out of fashion in the last 3 decades (it was really just a phenomenon in the 80s -- I remember those days) so it's quite likely that modern references to NLP don't refer to it.

I usually read NLP as "Nonlinear Programming" (nonlinear optimization) which is the community I come from. This acronym is not widely used outside the community so if I'm not reading the optimization literature, I'm pretty sure NLP doesn't refer to it.

Natural language processing is more in vogue these days, so that tends to be my default reading. The term itself seems to have existed for decades, so it's not like it came after the others but this is the most likely reading today.

<comment><USER9> Oh, for a second I thought the GP _was_ referring to nonlinear programming. Then I remembered this is an ML thread.

There are some nonzero number of papers referring to NLP (as in nonlinear programming) in ML, mostly for the purposes of constrained optimization, but I agree with your current breakdown.

</comment></comment></comment></comment><comment><USER10> As a new grad (bachelors swe), is it worth it to jump on the ML hype train ? I see modelling is almost always only open to phds/masters.

So is studying all that stuff just for being a MLE/ data engineer worth it, if you are already a software developer (full stack)?

<comment><USER11> > I see modelling is almost always only open to phds/masters.

I think this varies pretty widely based on employer, e.g. if you are at a smaller company (and you show interest and have the necessary skills) then you're much more likely to be able to contribute on the modeling side. It's easier to get there if you have an advanced degree, but definitely not necessary.

That being said, IMHO book lists like this aren't very useful because there's no incentive to keep them short and realistic. Reading seminal papers and implementing them is a different learning philosophy, maybe, but lists like this are probably more feasible to complete: [https://dennybritz.com/blog/deep-learning-most-important- ide...](https://dennybritz.com/blog/deep-learning-most-important-ideas/)

<comment><USER12> This is a great approach to learning. It explains how the field got to where it is and allows the reader to go as deep as they want.

</comment></comment></comment><comment><USER13> State and Revolution is where I'd recommend people start for ML ;)

<comment><USER14> Slightly Meta:

I wish HN had a parallel community for these kind of silly posts. I use it as a worksafe site to take a break at work and sometimes jokes like this, while not being in the spirit of HN, are things I very much appreciate.

<comment><USER15> I believe its called reddit

<comment><USER16> The only difference I find between reddits and HN are the amount of memes and the delusions of grandeur.</comment></comment></comment></comment>


New non-profit TechShop replacement on SF peninsula - <AUTHOR> http://makernexus.com <comment><USER2> I've been communicating with the MakerNexus board for the past few weeks, and attended an info session at Hacker Dojo last Friday. The principals have a very well-thought-out business plan, and solid credentials in both business and the maker movement (all are former TS DC's or multi-year members, and/or have run professional fabrication businesses).

The board is pledging total transparency on income, costs and roadmap -- no middle-of-the-night shutdowns.

Ultimately, I think this formula (reasonable aspirations, frugality, volunteer contributions and charitable donations, transparency, driven by and for makers) is the key to success where TS failed.

<comment><AUTHOR> I agree. Thanks for putting it so well.

</comment></comment><comment><AUTHOR> A non-profit creating a new makerspace on the SF peninsula similar to what TechShop was.</comment>


Watch what happens when a drone smashes into an airplane wing - <AUTHOR> https://www.nbcnews.com/mach/science/watch-what-happens-when-drone-smashes-airplane-wing-ncna920346 <comment><USER2>
      "simulated a collision by launching a small quadcopter out of a cannon at 238 mph (383 km/h)"

Correct me if I'm wrong, but that's faster than the maximum structural speed of the M20, 195 KCAS (knots indicated airspeed). [https://www.aopa.org/go- fly/aircraft-and-ownership/aircraft-...](https://www.aopa.org/go-fly/aircraft- and-ownership/aircraft-fact-sheets/mooney-m20)

Just more drone FUD. Anyone flying a drone intentionally or recklessly in the path of any passenger aircraft should be punished, and in many (most?) cases can already be punished. If you're flying a drone high enough that the impact is comparable to that at sea level, you're probably per se flying recklessly unless you're following standard procedure and rules for normal aircraft, including professional certification.</comment>


The Pirate Bay Now Accepts Bitcoin Donations - <AUTHOR> http://torrentfreak.com/the-pirate-bay-bitcoin-donations-130423/ <comment><USER2> > One of the advantages for the operators of the infamous BitTorrent site is > that Bitcoin funds can't easily be seized or traced back to a person.

But that last bit isn't true at all. It's trivial to trace Bitcoin transactions.

<comment><USER3> You should read up about Zerocoin here: [http://blog.cryptographyengineering.com/2013/04/zerocoin- mak...](http://blog.cryptographyengineering.com/2013/04/zerocoin-making- bitcoin-anonymous.html)

It's going to be presented at the ieee conference next month.

</comment></comment><comment><USER4> Sigh.. is any publicity better than none? I'm not sure. I am a little bit disturbed that sites that promote illegal or NSFW material are the ones that are quickest to adopt Bitcoin. It's already well-known for being associated with the Silkroad.

We need local businesses, or even corporations to start adopting Bitcoin! When this starts happening at a large scale, I think we'll see a tipping point in Bitcoin adoption.

<comment><USER5> IF that happens. There is still a chicken-egg dynamic at play (businesses don't plan on holding BTC and no one seems to reliably hedge against USDBTC volatility at a scale that justifies using it for business. Expenses are still in dollars so revenue needs to reliably exceed expenses. If that means grossly unfavorable prices, people will just convert BTC to USD first.)

<comment><USER6> i guess you haven't heard of <https://bitpay.com/>.

<comment><USER5> You clearly haven't bothered to learn about their service:

"To calculate the exchange rate for US Dollars, we pull up-to-the-minute BID prices from three exchanges. We take the 2 that are closest together and toss out the third, so that a bad feed from one exchange will not affect our calculation. Of the 2 rates that remain, Bit-Pay uses the highest BID price as our exchange rate. "

Companies don't end up changing prices every minute. Gas stations, for example, are prohibited from changing prices too frequently in many states.

A proper solution has to provide more than one minute of price stability. BitPay doesn't provide longer term price stability.

</comment></comment></comment></comment><comment><USER7> When it comes to donations, people will take boxes of used clothing. Better than nothing, right? Post something when Boeing will take Bitcoins for a 737.

<comment><USER8> > When it comes to donations, people will take boxes of used clothing.

People including the Pirate Bay?</comment></comment>


Ford is investing $1B into a self-driving car startup argo.ai - <AUTHOR> https://www.recode.net/2017/2/10/14576730/ford-investment-uber-google-self-driving-cars-argo-ai <comment><USER2> ok so what do they have exactly that is worth 1B? I mean sure, sounds cool but what is so unique about them?

<comment><AUTHOR> Probably the team? Tech leads:

BRYAN SALESKY, CEO - ex. Director, Hardware Development, Self-Driving Cars at Google ([http://google.academia.edu/BryanSalesky](http://google.academia.edu/BryanSalesky))

DR. PETER RANDER, COO - ex. Engineering Lead at Uber ([https://www.researchgate.net/profile/Brett_Browning/publicat...](https://www.researchgate.net/profile/Brett_Browning/publications))

DR. BRETT BROWNING, VP ROBOTICS – Engineering Lead at Uber ([https://scholar.google.com/citations?user=UKUngBoAAAAJ](https://scholar.google.com/citations?user=UKUngBoAAAAJ))

"$1 billion over a five year schedule" seems to be a safe bet for Ford to secure themselves from missing the SDC future.

</comment></comment><comment><USER3> Congrats to argo.ai!</comment>


Sales Have Slowed To A Trickle On Google's Chrome Web Store - <AUTHOR> http://techcrunch.com/2011/01/04/sales-have-slowed-to-a-trickle-on-googles-chrome-web-store/ <comment><USER2> The title of the article seems misleading. Apparently the Chrome web store was first launched less than a month ago, so it is a product very close to the origin point of its growth curve and a phrase like "sales have slowed to a trickle" implies a substantial past history of higher volume sales. I think a better description of the data would be "chrome web store has yet to gain sales momentum".

<comment><USER3> Good point. I certainly wasn't trying to be misleading — the "slowed to" was based on my observation immediately after the store launched, when I believe some of the paid apps had higher numbers of installs. However, I don't have the sales figures on those handy (I'd have to ask Google) and it may not be fair to base sales data off the initial launch anyway.

I've changed the title to be less ambiguous/misleading.</comment></comment>


Long Island Iced Tea Soars 500% After Changing Its Name to Long Blockchain - <AUTHOR> https://www.bloomberg.com/news/articles/2017-12-21/crypto-craze-sees-long-island-iced-tea-rename-as-long-blockchain <comment><USER2> When I first read the title I thought, "How did an Onion article make it to the front page of HN?"

<comment><USER3> There is a serious point here - as someone more familiar with the alcoholic beverage of the company's original name than its wholesome products, I think we need some blockchain-themed cocktails for the holidays (it's probably been done... does anyone have recipes?)

<comment><USER4> Something with a lot of fizz that leaves you with a bitter aftertaste and a bad hangover?

Looks like there's already a bitcoin cocktail floating about: [http://www.barmeister.com/drinks/recipe/7511/?o=by_date](http://www.barmeister.com/drinks/recipe/7511/?o=by_date)

<comment><USER5> Coke and Mentos, resulting in a briefly frothy mixture followed by a mostly empty glass.

<comment><USER6> That only works with diet coke.

<comment><USER7> It works with any carbonated drink, diet coke just happens to be the frothiest of the ones tested.

</comment></comment></comment></comment></comment></comment><comment><USER8> A tulip of our times...

History never repeats but it rhymes

For people who missed out on the .com bubble don't miss this out. It's a good time to experience what a bubble looks like so you can detect one in the future...

Remember this buffet quote It's better to be fearful when others are greedy and greedy when others are fearful

<comment><USER9> Respectfully, what drives people to post tulip and dot-com bubble comparisons on literally every cryptocurrency / blockchain post? We saw it yesterday. We saw it last week. And a few months ago. And in 2016, and 2014, and 2012.

It's unfalsifiable and meaningless. We don't need to have this same "point" made on every post.

<comment><USER10> If just one person is contemplating betting their mortgage on bitcoin and they are made to think differently because of a bubble post then it's worth it to me!

<comment><USER11> Not that I think you're wrong, but every person who did that in 2014, 2015 or 2016 probably begs to differ.

In fact, if they sell today, they'd probably be set for life.

<comment><USER12> Guaranteed anyone foolish enough to bet their mortgage at those times, bought high and sold low in fear of losing it all. They are not the rational type.

<comment><USER11> Maybe. I don't necessarily think taking a huge risk with substantial upside is always an idiotic thing to do.

Yes it might be a gamble and speculation, but get it right in a bubble and you're (sort of) set for life. You can make a good calculated bet and see it pay off. Any engineer on $100k to 200k a year can probably afford to put $200k in crypto and recover if it evaporates.

If you compare to certain high risk shares trading on the stock market (i.e. some tech and biotech shares), even though crypto might have a 10x upside, I don't think investing in crypto is 10 times as risky. If you're aggressive, you're probably better off investing in crypto today than in the market.

<comment><USER12> Ok, so for majority of the rest of the population, without access to six figure incomes. Bitcoin is completely irrational. There is a guy in my shop right now, a shop full of guys making maybe $15 an hr at the high end, handing out his "bitcoin consultant" business card. These are the people making irrational bets on this, while those invested years ago are profiting on their foolish dreams. People with no money to be wasting, are wasting their money.

<comment><USER11> Yes, of course. And since they don't put in a significant amount of money, even if their investment were to go 50x, they probably wouldn't cash out and keep holding. That is a clear mistake, as it can be dangerous not to exit.

I was just pointing out that it can be profitable riding a bubble, as long as you're careful about it. I have an old money friend who put in a few million last year. That's exciting stuff.

One thing I wonder: since they own so little, do they even move the market?

<comment><USER12> Alone no, but the combined yes. There are a few big players that manipulate this market. The Winklevoss did it, and caused the first bubble in 2013. Then the market fell off the cliff, and sat a $200 for the next 3 years, until someone bought up tons at around $250-300 mark, and here we are again.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> I think this is likely not "small companies chasing a fad" but rather "penny stock promoters doing their usual thing: grab a listed entity for five figures then run a boiler room, citing the most recent successful boiler room."

<comment><USER4> That's along the lines of what I was thinking was going on here. This is pretty brazen.

</comment></comment><comment><USER14> My comment from: [https://news.ycombinator.com/item?id=15978994](https://news.ycombinator.com/item?id=15978994)

\----

Looks like it [is a joke!]. I mean, the new website is only a day old, hosted with "Hostgator", uses a free theme ([http://websites.simplesphere.net/piupiu/](http://websites.simplesphere.net/piupiu/)) and NGINX has directory listing switched on! That and all the news reports are syndicated from the same place, so its just propagating all over the place.

That - or I am mistaken, in which case - "All Hail our new Blockchain overlords" :)

<comment><USER5> Hard to fake a NASDAQ listing:

[https://finance.google.com/finance?q=NASDAQ%3ALTEA](https://finance.google.com/finance?q=NASDAQ%3ALTEA)

Whether or not the press release is a fake, the spike is real.

<comment><USER15> Maybe I'm blind. Where does that page show their new name (other than the news ticker on the side)?

<comment><USER16> Why would their stock ticket show anything not related to the financials of the stock?

<comment><USER15> This person is claiming that the NASDAQ listing proves that they've changed their name. Yet the link itself clearly says "Long Island Iced Tea Corp", not "Long Blockchain".

So, it would show it, because it shows the company name. And the company name is not "Long Blockchain", so the link proves nothing and the gp is full of shit.

<comment><USER17> The price spiked after the press release announcing the name change. The press release caused a real spike.

It seems far more likely that the name change hasn't been made formal, or if it has, it hasn't trickled into Yahoo's database yet. The press release says:

> The Company intends to request Nasdaq to change its trading symbol in > connection with the name change.

It's an intention. It hasn't formally happened yet.

</comment></comment></comment></comment></comment></comment><comment><USER18> I think this might be block chain's pets.com moment. If this weren't from Bloomberg, I would honestly think this was satire.

<comment><USER19> This is the fourth or fifth of these I've seen in the last year. Riot Blockchain for example, which used to be a worthless, failing, tiny biotech company. Overstock.com pulled a version of this fraud (I'm calling it a fraud, they may be technically legally in the clear; I believe these companies are commiting intentional fraud however, knowingly misrepresenting critical facts about their business to shareholders).

</comment></comment><comment><USER20> I've only known Long Island Iced Tea as Vodka, Rum, Tequilla, triple sec, gin and coca-cola.

<comment><USER21> Yeah, go figure.

</comment></comment><comment><USER22> The only reason this made any news is because it's tied to blockchain. Penny type stocks do this pump and dump scheme all the time with whatever is hot. Biotech was big for awhile, then pot, and now blockchain (and I'm sure I missed a few in there).

</comment><comment><USER23> Is this the result of algorithm trading?

<comment><USER24> Also wondering that same question. If it is, it would be interesting to learn how the conclusion to buy was reached.

<comment><USER25> buylist_keywords = ['blockchain', 'snake oil', 'tulip']

</comment></comment></comment><comment><USER26> It's not a bubble -- it's carbonated!

</comment><comment><USER27> Don't be surprised if in the next edition of A Random Walk Down Wall Street cryptos get appended to chapter 2, The Madness of Crowds. Events like these are embarrassingly similar to those that have occurred in past bubbles. If you have any skin in this game (as I do) you'd do well to read the aforementioned chapter and start planning your exit strategy.

</comment><comment><USER28> For those who think this is a spoof, check the 5 day view on Google Finance: [https://www.google.com/search?q=NASDAQ:LTEA](https://www.google.com/search?q=NASDAQ:LTEA)

<comment><USER29> Look at the 5 year chart, there have been a few large jumps in the company's stock price in the past. Don't think that "blame the Bitcoin" works for those.

</comment></comment><comment><USER30> definitely not a bubble

<comment><USER31> We don't even see this kind of stuff in a regular bubble. I think this is something new, either Bubble 2.0 or The New Dawn.

I lived through the IT bubble and it mostly went unnoticed among me, my friends and family. Literally everyone talks about Bitcoin though. I'm seriously thinking this is a first for humanity and the consequences will be extreme regardless of crash or paradigm shift. It's one for the history books for sure.

<comment><USER32> This probably isn't even 1/10 as bad as internet and tech bubble. That had common people throwing their entire 401k into tech stocks.

My dad ruined his retirement fund buying Nextel options that ended up worthless.

Bitcoin is a dumber bubble though.

<comment><USER33> There are people putting their life savings into bitcoin. You'll see those stories pop up from time to time on the bitcoin subreddit.

I've only got in it what I can afford to lose, personally, which isn't a whole lot, but it's still some skin in the game, at least.

</comment></comment></comment></comment><comment><USER34> Is this real? It's not April 1st.

</comment><comment><USER35> This reminds me of during the dot com boom. An analyst said that business to business sector is worth some huge amount of money, and B2B stocks started booming.

So a janitorial supply company changed their name and their symbol to BTOB and exploded in value.....

</comment><comment><USER36> Bubbles can be a fantastic way to generate wealth. Amass as much cash as possible, wait for the inevitable crash (you can't time this so just be prepared for when it does happen), and then buy tons of assets for pennies on the dollar. If you truly believe in the fundamental value of the frothy asset, this approach can be a huge win. Obviously, this should not be your primary investment strategy but it worked well for me (e.g. doubled the value of my stock portfolio in 2009 and doubled the square footage of my bay area home in 2010).

<comment><USER37> That's assuming you can correctly call the bottom, otherwise you could be waiting for a long time just to break even.

</comment></comment><comment><USER38> Reminds me of Zapata the fish oil company launching zap.com an internet portal and then spinning out and IPOing the subsidiary. [0]

[0] [https://mobile.nytimes.com/1999/04/14/business/company- news-...](https://mobile.nytimes.com/1999/04/14/business/company-news-zapata- plans-to-spin-off-its-zapcom-internet-business.html)

</comment><comment><USER39> Efficient markets, price contains all available information, rational expectations and whatnot there were. Oh, yeah, almost forgot,

 _I don 't even know what a bubble means. These words have become popular. I don't think they have any meaning._ \- Eugene Fama, Nobel Memorial Prize laureate in Economics

</comment><comment><USER40> So for those who remind us that the cryptocoin world is full of tulip-like behavior with no underlying asset value... while stocks provide dividends and represent growth and so forth: this article is a gentle reminder that stocks are also subject to tulipy speculations.

<comment><USER41> Because people thought the stock was a tulip.

<comment><USER40> Sure, but as the prevalence of day trading and even automated trading that buys and sells in short periods of time (which you can tell if you look at the price movements of a stock in a small, say daily interval and notice repetitive price pattern movements), a lot if not MOST stock trading is entirely NOT based on the expectation of underlying value of the stock. Not everyone has a Warren Buffet or Bogle-like approach to long-term investing of stocks.

</comment></comment></comment><comment><USER42> WTF. Clickbait title. Even bloomberg knows it's not accurate. From the story itself: "Long Island Iced Tea Corp. shares rose 238 percent after the company rebranded itself Long Blockchain Corp."

238 != 500

Anyways, I think it's still crazy what blockchain mania is doing/causing.

<comment><USER43> Check the chart at the bottom of the article. It went up 500%, then dropped. Since the article, it seems to only be up 190%.

I imagine that over the intervening period, some value investors independently investigated the exact nature of Long Blockchain's technical innovations in the distributed thirst-quenching space.

</comment></comment><comment><USER44> Is this what Karl Marx had in mind when he said that capitalism would destroy itself?

<comment><USER45> No, that would be the republican party.

</comment></comment><comment><USER46> Can't wait for books about the crash, like boo.com. That was a good read.

</comment><comment><USER47> Did satirists just give up? This can't be real life.

</comment><comment><USER48> Anyone notices the screenshot? The price was at 13.37.

</comment><comment><USER49> "money for nothing and chicks for free"...

</comment><comment><USER50> This happens all the time whenever there is a "craze". When marijuana companies were the craze, scammers who peddle stocks that sounded similar to high flying stocks.

It happens during big IPOs.

[https://www.bloomberg.com/news/articles/2017-02-08/mistaken-...](https://www.bloomberg.com/news/articles/2017-02-08/mistaken- identity-sends-snap-sound-alike-up-140-after-ipo-news)

It's pretty much a non-story but bloomberg has to make money with clickbait.

</comment><comment><USER51> Lots of opportunity to make quick bucks with shorting this. And riot blockchain.

</comment><comment><USER52> gold rush at its finest

</comment><comment><USER53> This does show how most share brokers/buyers trade.

</comment><comment><USER54> For a moment I thought today is the first of April.

</comment><comment><USER55> totally need to reverse merger a penny stock company and change it's name to some blockchain company name.

I'm thinking:

Ageless, blockchain, AI VR company

<comment><USER56> Visit my new startup's website, [http://serverless.blockchain.ai/mongodb](http://serverless.blockchain.ai/mongodb)!

<comment><USER57> It's Bitcoin Scale!

<comment><USER58> So, 4 requests per second?

</comment></comment></comment></comment><comment><USER59> They should've named it Long JavaScript !

</comment><comment><USER60> Is there an 8-K filing supporting this?

<comment><USER61> None that I could see:

[https://www.sec.gov/cgi-bin/browse- edgar?CIK=LTEA&owner=excl...](https://www.sec.gov/cgi-bin/browse- edgar?CIK=LTEA&owner=exclude&action=getcompany)

<comment><USER60> Might come after market close, but I think this is real judging from the other things I've read

</comment></comment></comment><comment><USER62> Huh

</comment><comment><USER63> I can see the shark below...

</comment><comment><USER64> Except the industrial value of gold which is around 10% of its use and not the quality/pure one, how is gold better than bitcoin? The favorite argument and the only argument is its history, yet when someone tries to use the same historical argument of bitcoin going up and up for 7 years, the elitist, NY folk will quickly follow with "Past performance is not an indicator of future outcomes".

I think people here are just salty because they think they are smart yet didn't saw the value in the crypto/blockchain, especially that is up their geeky alley.

<comment><USER5> Good luck stealing a ton of gold using a keylogger.

<comment><USER64> if you keep a ton of gold in your house, good luck.. keeping it safe, there are safe ways to keep bitcoin, cold storage, etc. is not like is impossible.

<comment><USER5> It appears to be substantially easier to secure a ton of gold than the equivalent amount of Bitcoin.

It's also a lot harder to accidentally lose.

<comment><USER64> right is so hard to buy a device from amazon or print some words on plasticized paper and keep it safe, hell you can even memorize 12 words.

<comment><USER5> Sure, assuming the Amazon device isn't a compromised counterfeit, the printer isn't rooted, and the paper doesn't get burned up in a house fire.

Case in point: [https://www.wired.com/story/i-forgot-my-pin-an-epic-tale- of-...](https://www.wired.com/story/i-forgot-my-pin-an-epic-tale-of-losing- dollar30000-in-bitcoin/)

<comment><USER64> yes, this is what hold bitcoin back but is getting better, the amazon device can be software signed, also you could encrypt your keys and upload to all cloud hosts.

</comment></comment></comment></comment></comment></comment></comment><comment><USER65> Just dont let the funny-facts divert you from the important concepts: bitcoin = money-without-banks, first time in human history

<comment><USER66> > money-without-banks, first time in human history

... please google 'salt as currency'

</comment></comment><comment><USER67> Unqualified Americans can't invest in startups or ICOs. Blockchaina are the only assets that are growing that they can invest in. That deosn't even include the rest of the world.

<comment><USER68> The S&P 500 is up 86.95% over the last 5 years. Is that "not growing" in your mind?</comment></comment>