Why Jupyter is data scientists' computational notebook of choice - <AUTHOR> https://www.nature.com/articles/d41586-018-07196-1 <comment><USER2> The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both:

a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and experiment with the examples you're giving to improve their understanding or test their hypotheses. If I change this variable, how will the system react? You can just try it!

b) Just like with a REPL, you can type in and execute commands step by step, viewing the output of the previous command instead of running the whole thing at once. From the document side, as a bonus, you get nicer presentation (charts, interactivity, nice and wide sortable tables, etc) than you would in a shell, which comes in handy when doing things like data exploration or mathematical simulation.

It's decidedly NOT there for you to type all your code in like an editor and make a huge mess. It's apples and oranges w.r.t and a poor substitute for something like PyCharm or VS Code or vim. It is there for you to a) try things out yourself, and whatever you discover hopefully eventually make it into proper python modules b) make interesting ideas presentable and explorable for others. That's all!

When I see stuff like "out of order execution is confusing", I don't disagree, but it does make me wonder how long and convoluted the notebooks these people work with are - probably a ripe candidate to refactor stuff out into python modules as functions. When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would. In that regard I think it's barely an improvement over .py files lying around. When I hear "how do I import a notebook like a python module", I'm very very scared.

Granted, I've seen huge notebooks that are a mess, so I understand the frustration, but it's not like we all haven't seen the single file of code with 5000 lines and 10 nested layers of conditionals at some point in our lives.

<comment><USER3> If you ever have used an R Notebook written in R-Markdown, then its pretty easy to see why Jupyter Notebooks putting everything in JSON is just... infuriatingly wrong-headed. In an R Notebook, I can see my code, I can see my text, everything is exceedingly simple to understand, and I can edit it in any of the fantastic text editors out there (Jupyter's editor is not among them)

<comment><USER4> The main reason for json, I believe, is that the Jupyter client is separate from the backend. It's actually pretty trivial to run the engine on a beefy box while interacting on a light laptop (on the same subnet). With Jupyter Lab and some fiddling, you can put the server anywhere.

It's also trivial to export notebooks to .py files.

That said, my goodness do notebooks wreak havoc on git. I hope this in particular gets fixed as popularity grows.

<comment><USER5> Having a proper server available is even more reason to use a proper fileformat. The client doesn't care what the server handles and the server doesn't need to send raw datastructures directly from the storage.

Actually, fixing the fileformat-mess should be very simple. Just change the file-load/save-functions. Use a folder-structure with every cell being a seperate file. Or switch to XML. Or make a generic interface and allow to save in whatever the people want. Saving notebooks in Mongodb or some SQL-Database seems like a good goal for dedicated services.

</comment></comment></comment></comment><comment><USER6> Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly.

I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc.

[1] [https://reviewnb.com](https://reviewnb.com)

[2] [https://nbdime.readthedocs.io/en/latest/](https://nbdime.readthedocs.io/en/latest/)

<comment><USER7> I'm glad I'm not the only one. When I inherited some "production notebooks" (if that's a thing) I couldn't believe it was nearly impossible to do basic things such as test and review changes (via version control).

<comment><USER8> You don't use Jupyter notebooks in production; they are super useful for pitching ideas to clients/bosses and doing some early prototyping. I feel sorry for anyone that has to work with "pure data scientists" that have no clue about software engineering practices...

<comment><USER9> FWIW, Netflix uses Jupyter notebooks in production, using nteract UI:

[https://medium.com/netflix-techblog/notebook- innovation-591e...](https://medium.com/netflix-techblog/notebook- innovation-591ee3221233)

[https://nteract.io/](https://nteract.io/)

This approach seems promising, particularly as it facilitates cross- disciplinary collaboration.

</comment></comment></comment></comment><comment><USER10> As with so many things python related (including python itself), I am perplexed by how willing people seem to be to fall in love with solutions that have so many limitations and problems. I find Jupyter just barely usable. I constantly have issues with editing in the cells, diagrams not sizing correctly, cells accidentally displaying huge amounts of data and freezing my browser, complete failure of autocompletion in many languages, a very awkward security model involving manual cutting and pasting of auth tokens around, nearly impossible to get a reasonable rendering of the notebook into something reasonable like PDF (yes there attempts at solutions, they are full of problems). Many limitations derive directly from the architecture where the kernels are limited in what they can do because specific parts have to be interpreted in the browser that are language specific.

From my perspective, it's a dumpster fire - in 2018 there should be something so much better than this. RStudio is a thousand times better but only does R. I used to like Beaker Notebook but it gave up due to Jupyter's popularity and converted itself into a bunch of Jupyter extensions which now have all of Jupyter's limitations.

Yet despite all this I can see that there's this enormous community that loves this and keeps developing and contributing to it.

<comment><USER11> _it 's a dumpster fire - in 2018 there should be something so much better than this_

In 1998 I was using a tool called MathCAD that provided a notebook interface running as a plugin to MS Word. In 2018, Jupyter is still not as good as that. Some things are just not meant to be webpages.

<comment><USER12> > Some things are just not meant to be webpages.

This is how I feel about most of the single page apps I've worked on.

</comment></comment></comment><comment><USER13> Link to the deck by Joel Grus' talk that is mentioned in the article: [https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...](https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV- dkAIsUXP-AL4ffI/edit#slide=id.g362da58057_0_1)

<comment><USER6> Really good talk. Here's the video: [https://www.youtube.com/watch?v=7jiPeIFXb6U](https://www.youtube.com/watch?v=7jiPeIFXb6U)

And all JupyterCon 2018 talks if anyone is interested: [https://www.youtube.com/playlist?list=PL055Epbe6d5b572IRmYAH...](https://www.youtube.com/playlist?list=PL055Epbe6d5b572IRmYAHkUgcq3y6K3Ae)

</comment></comment><comment><USER14> I love Jupyter Notebook for experimenting and rapid creation of reports, but dislike it for not being able to use my editor and for intermingling inputs and outputs in a single file. So I'm working on an alternative frontend to Jupyter kernels, which is heavily inspired by KnitR: [https://github.com/azag0/knitj](https://github.com/azag0/knitj) It is still being developed, but it's functional and I use it every day.

<comment><USER15> This looks incredible! Does your project already support other language kernels than the Python kernel?

I use R for 90% of my work, but most of it has been happening in Jupyter notebooks (which I'm not a huge fan of, despite practically living in them for the past 4 years of my life).

Thanks for sharing!

<comment><USER14> I have not tested it with anything else than the Python kernel, but it uses Jupyter Client to communicate with the kernel, which is kernel agnostic. So you should be able to do just "knitj -k <kernel name> ...".

</comment></comment></comment><comment><USER16> I've found ob-ipython [1] within Org Mode to be one of the best options for interfacing with Jupyter. If you're sick of the limitations of working in a browser, it's worth checking out.

Scimax version: [https://github.com/jkitchin/scimax/blob/master/scimax- ipytho...](https://github.com/jkitchin/scimax/blob/master/scimax-ipython.org)

Video of Scimax version: [https://www.youtube.com/watch?v=dMira3QsUdg](https://www.youtube.com/watch?v=dMira3QsUdg)

Previous HN discussion highlighting key features: [https://news.ycombinator.com/item?id=17839926](https://news.ycombinator.com/item?id=17839926)

Some relevant blog posts:

\- [https://vxlabs.com/2017/11/24/getting-ob-ipython-to-show- doc...](https://vxlabs.com/2017/11/24/getting-ob-ipython-to-show- documentation-during-company-completion/)

\- [https://vxlabs.com/2017/11/30/run-code-on-remote-ipython- ker...](https://vxlabs.com/2017/11/30/run-code-on-remote-ipython-kernels-with- emacs-and-orgmode/)

\- [https://kozikow.com/2016/05/21/very-powerful-data- analysis-e...](https://kozikow.com/2016/05/21/very-powerful-data-analysis- environment-org-mode-with-ob-ipython/)

[1] [https://github.com/gregsexton/ob- ipython](https://github.com/gregsexton/ob-ipython)

</comment><comment><USER17> Here are the issues with Jupyter, and most other flavor, of notebook:

1\. variables have to be explicitly output

The most important tool for programming, for me, is that window that shows you the current state of all the variables. When I step through a program, I look at the state. 90% of my debugging solutions come from seeing that variable doesn't have the right state.

2\. Intellisense

For the love of god, I do not want to remember if it is len(), length(), .len(), .length(), .size(), size(1) or whatever.

That's it. But those two are so big that I have to code and debug in Spyder and then paste the code into notebook. I feel sorry for people who are new who think that all the debugging is happening in the notebook.

<comment><USER18> Hey There! I'm trying to solve the issue of IntelliSense.. I'm building/improving Jupyter Notebooks inside VSCode: [https://github.com/pavanagrawal123/VSNotebooks](https://github.com/pavanagrawal123/VSNotebooks) . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :)

Also planning to add some nice debug features, plus hopefully integration into the inbuilt VSCode debugger!

<comment><USER19> How do you see the idea of a vs code notebook comparing to or being different from the goals of the hydrogen editor?

<comment><USER20> To me, my favorite part of the design of Hydrogen is that it's entirely language agnostic, and can be used with _any_ Jupyter kernel.

<comment><USER18> I'm planning on adding better language support in a couple of weeks! Don't want to limited to Python and R

<comment><USER21> One thing you may want to be aware of is that the python language server used for vscode runs pylint which performs static analysis on the code. However, Jupyter Notebook uses autocomplete by actually introspecting the variables as they are defined. This creates large differences when doing things such as selecting a column in a pandas dataframe. In jupyter if you press tab on the column name, it can autocomplete and also assumes you are getting a series, which leads to autocomplete on things like .min, .max, etc... In pylint you don't get any of this autocomplete since pylint cannot statically determine the column names so you lose the intellisense.

<comment><USER18> yep! this is something I will be focussing on in VSNotebooks. I noticed this as a huge drawback to the current implementation, so I will be fixing it :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER22> For people who prefer to code in JS. There is a similar application called observable notebooks that has recently come out: [https://beta.observablehq.com/](https://beta.observablehq.com/)

It offers some nifty things including, _well_ , observables where cells of the scratchpad can automatically update by observing changes from other cells.

<comment><USER23> More of an online service, though.

<comment><USER22> true true.

</comment></comment></comment><comment><USER24> I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code _as well as_ stats, I suspect we'll start to see an exodus from pure R solutions.

The real question is when (and whether) new social scientist stats courses will start teaching Python stats toolchains, rather than R. That seemed to be an inflection point for R (as folks moved away from SAS), and could be for stats-centric Python too.

<comment><USER25> I'm not sure what about Jupyter makes Python more compelling in comparison to R. R is entirely usable in Jupyter Notebooks, and R Notebooks are, in my opinion, possibly superior to Jupyter notebooks in many ways.

> and as more people realize that there's a benefit to simultaneously training > researchers to run code as well as stats, I suspect we'll start to see an > exodus from pure R solutions

I'm not sure what you are saying here.

I would actually argue that most of the Python data science toolchain is years behind what is available in R.

<comment><USER26> > I would actually argue that most of the Python data science toolchain is > years behind what is available in R.

I do not want to litigate this on HN, but the problem with R is the toolchain _around_ your data science work.

You've fit a model in R, and that's great! Now how do you get it into a real- time system? Or how do you test the software you wrote to train the model?

</comment></comment></comment><comment><USER27> It's easy to grade student assignments in notebooks with [https://github.com/jupyter/nbgrader](https://github.com/jupyter/nbgrader), which also makes it great for teaching.

<comment><USER28> Meh. As long as you have defined deliverables between grader and student, grading programming based assignments are relatively easy. Coursera has been around longer than Jupyter has been popular, after all. (And they aren't all just multiple choice.)

Being interactive is what makes it good for teaching. But there are plenty of interactive options. And for a certain class of teaching, it is not "on rails" enough such that people will have to have a ramp up period first on Jupyter before they can really get into their topic.

</comment></comment><comment><USER29> The only thing that stops me from being able to use notebooks full time is their intellisense compared to IDEs is horrible. I like being able to use them for demos/presentations, but I can't imagine trying to code within one primarily. Especially when it comes to tracking results.

How do people cope with this? Do you supplement it with other tools? I spend a lot of my time in an IDE and then just paste some of the code in to cells. That seems easier.

<comment><USER30> I do the opposite, my job is kind of bad data engineer/scientist/etl minion so it's a lot of dataframes.

Work (and often debug) in jupyter -> open the notebook from pycharm when it's got some completed thoughts and write into a python module + test module, tidying up and adding type annotations.

Sometimes doing that multiple times so that the notebook is importing from modules which were originally pulled out of the notebook.

It sucks having to use two tools but I don't think there's any one tool that can do both as well as pycharm/jupyter, short of me getting a lot better at emacs or writing a lot of custom Atom extensions (I think).

<comment><USER31> I am very hopeful that JupyterLab will get support for the Language Server Protocol sometime soon. That would make all the difference in the world for me. I'd still have to use a terminal to build and run tests, but I wouldn't be surprised if a test runner comes along fairly quickly after that.

(Relevant issue: [https://github.com/jupyterlab/jupyterlab/issues/2163](https://github.com/jupyterlab/jupyterlab/issues/2163))

</comment></comment></comment><comment><USER27> I've switched largely to Jupyter / Python for computational linguistics / psycholinguistics because of the pandas / numpy /numba stack, decent off-the- shelf NLP (spacy and gensim), and the ease of moving data into an R kernel for specific analyses and plots. Also nice that any reasonably sized notebook will render on GitHub (and access can be controlled through the accounts system, until something is ready to be public).

One thing I haven't figure out how to do is to generate fully styled LaTeX manuscripts from notebooks (like papaja for RStudio). Is there a way to do this with pandoc?

<comment><USER32> Yes! Juypter notebook has an export to .tex file in its export menu. If you install the right stuff you can render the output on the server.

However, the output of this isn't nearly as well formed as a hand written latex document is.

<comment><USER33> Yeah but I believe you can customize the export template, right?

</comment></comment></comment><comment><USER34> I see Jupyter notebooks as the next step in spreadsheets with more code foundation and different media support. For me the interface doesn't work as well as I would like but I can see the potential.

My brother in law wants something like this for structural analysis reports where the code, data and report are all one thing that can be pulled out and examined.

Watching the new iPad announcement today I think this is something that would make an excellent iPad app as well.

<comment><USER35> yeah, i have a use case now where we pull data from a database, manipulate it, and then have a final table/csv/dataframe/whatever. the problem is then how to share this with non-technical users. in an ideal world, this would get inserted into a google sheet, and that sheet would just update daily after new data is loaded into the database.

i'm pretty sure this is a usecase which others have and curious what people use to solve it. i've heard, variously, that some options are to use tableau/similar or email a csv and ask the end user to import into google sheets/excel

<comment><USER36> Google Sheets has an API. So if putting the output there is the ideal, just use that API from Python?

<comment><USER35> yeah, issue is that if you're inputting more than ~500 rows, you'll be rate limited :|

</comment></comment></comment></comment><comment><USER37> RStudio with using Rmarkdown is also popular. Both workflows are language agnostic.

<comment><USER38> Putting aside the R vs Python question (as as noted in this thread, you can use R in a Jupyter notebook and Python in an RMarkdown notebook), I much prefer RMarkdown notebooks. RMarkdown notebooks are plain text, so you can read them easily in any text editor (which also means they play well with git, unlike Jupyter notebooks).

And it's meant to work with the RStudio IDE, so I get a much more seamless experience going between regular code and notebooks (although this is admittedly a more R-centric benefit, at least until and unless RStudio adds Python support outside of notebooks).

<comment><USER3> Using markdown for python notebooks makes so much sense. What was the thinking behind encoding everything inside JSON?

</comment></comment></comment><comment><USER39> IMO, Jupyter is nice for presenting the final results of research (like LaTeX), but it is often not the right tool to get there. It's good for professors who teach and publish but it's bad for students to learn and research.

Frankly I find that all programming environments for scientific computing are deficient in some way or another. If you look at the set of features in Visual Studio, R Studio and Jupyter notebooks, you will see that the Union of useful features is large, and the intersection is almost empty.

</comment><comment><USER40> IMO Jupyter notebooks are popular because it's open source, they are convenient and they help a lot illustrating an idea.

The results of a notebook can be shared more easily than a plain repository(via nbviwer or binder) and more importantly the science there it's reproducible.

</comment><comment><USER41> Question/idea: Could a notebook-model supplant bespoke photographing processing software such as the "darkroom" mode of Lightroom (or darktable). The extant programs essentially take a lot of data (camera's raw output) and apply a configurable recipe to produce intelligible output (an image). Each recipe (stored as an XMP sidecar) is essentially a list of math operations (increase brightness, wavelet decompose, change color model, etc.) and their parameters.

Obviously a great part of why we use Lightroom/darktable is because of the speed with which the recipe-processing occurs. Plus a smooth UI, a catalog- viewing feature, and a well vetted choice of image operations. The appeal of moving this work to a notebook would be that an actively maintained Jupyter ecosystem could supplant lock-in to a specific software, and open up the underlying math magic.

At the very least, this could be an interesting platform for experimenting with image processing methods. And the reordering of cells could become a virtue, to run an image processing pipeline out of the standard order.

I'm curious if anyone has already worked along these lines. I find through a quick web search that people are doing some image processing, but more in the face detection or ML for medical imaging aspects. I see as a basic toolkit that [http://scikit-image.org/docs/dev/auto_examples/](http://scikit- image.org/docs/dev/auto_examples/) is something, though this isn't the whole range of operations needed for, say, fine art image tuning.

<comment><USER42> Yes and no. I do computer vision and like photography as well. I use Jupyter notebooks extensively for computer vision, and they work pretty well for (semi) interactive manipulation of image data with code. But as a general purpose tool, it's too clunky for anything more than prototyping. I don't see them replacing darktable/lightroom anytime soon.

</comment></comment><comment><USER43> Zeppelin[1] is another great tool of a similar nature. It leans a little bit more towards the Scala / Spark world, for people who like that stack. That said, you can use Python, R, etc. with Zeppelin as well.

[1]: [https://zeppelin.apache.org/](https://zeppelin.apache.org/)

</comment><comment><USER44> Does anybody know of a good hosted solution of JupyterHub? I made a neat notebook that I needed to share with my non-technical team, it was using iPyWidgets to do some interactive modeling, but they each needed to be able to use it independently. It has private data so I couldn't use Binder. I've been following Zepl.com for a long time, but couldn't use them here because Zepplin doesn't support iPyWidgets. Pretty soon I found myself installing helm and trying to follow along a tutorial on how to deploy JupyterHub on a kubernetes cluster. That started to add an unmanageable level of complexity to own, especially to share a simple notebook. And while spinning up a GKE node per user is the whole point of Kubernetes, it got expensive quickly in my test. We cannot spend $75K a year on Domino. Any other options?

<comment><USER45> Do you have some server to host it on?

If so, I'd run the notebook on the remote server and just teach them whatever command they need to make a ssh tunnel there. Something like what is described here: [https://techtalktone.wordpress.com/2017/03/28/running- jupyte...](https://techtalktone.wordpress.com/2017/03/28/running-jupyter- notebooks-on-a-remote-server-via-ssh/)

So they would utter the unknowable incantation and then point their browser at localhost:8000 or whatever and then use their version of the notebook.

<comment><USER44> I think that would be within reason for most of our users, but we have a couple of Windows users. I'm not particularly keen on telling them to install putty or WSL.

<comment><USER45> Ah, well they can always borrow their less disadvantaged coworkers computer until _they_ get the sysadmins to install and configure a shortcut on their desktops via active directory I guess.

</comment></comment></comment></comment><comment><USER46> Notebooks are great for invoking existing functions and exploring data.

Notebooks aren't ideal for creating functions (standard text editor features are lacking and testing is impossible).

Notebooks encourage an "order dependent variable assignment" programming style without abstractions. Here's what you'll commonly see in a notebook:

val df = spark.read.csv("some_data")

df2 = df.withColumn("clean_name", trim("name"))

df3 = df2.filter("clean_name" === "Mark")

I've found that notebooks are very useful if you write all the complicated code in separate GitHub repos and attach binary executables to the cluster. If you try to write all your logic in notebooks, you'll quickly struggle with order dependent, messy code.

<comment><USER47> I had the same trouble with order dependence as notebooks got to a certain size, so my team and I created and open-sourced a library, Loman, to help with that. It allows you to interactively create a graph, where nodes represent inputs or functions, and then keeps track of state as you change or add inputs, intermediate functions and request recalculations. Our experience has been broadly positive with this way of working. As graphs get larger, it's easy to lift them into code files in libraries, while continuing to modify or extend them in notebooks. The graph structure and visualization make it easy to return to loman graphs with up to low hundreds of nodes, which would make for a fearsome notebook otherwise. It also makes it easy to bolt Qt or Bokeh UIs onto them for interactive dashboards - just bind UI widgets and events to the inputs and widgets to the outputs. They can be serialized, which is useful for tracking exceptions in intermediate calculations when we put them in airflow to run periodically, as you can see all the inputs to the failing calculation, and its upstreams.

[1] Github: [https://github.com/janushendersonassetallocation/loman](https://github.com/janushendersonassetallocation/loman) [2] Quickstart/Docs: [https://loman.readthedocs.io/en/latest/user/quickstart.html](https://loman.readthedocs.io/en/latest/user/quickstart.html)

</comment></comment><comment><USER48> Having spent a decent amount of time learning to be a programmer while doing scientific image analysis in Matlab (shudders from the real programmers), and with a decent amount of time spent in Mathematica as well, I just can't seem to buy into the Jupyter/notebook based programming enthusiasm. The talk linked in the article explains it better than I ever could, but for me, when I am leaving data in memory, it is much more convenient for me to have a completely linear history, ordered by command execution time. In python I have found the best way to do this is writing standard python functions and scripts, and running them in an IPython environment with the %run magic. You have the linear history, git works well on standard .py files, and you can interactively work with the data in the IPython prompt without worrying that something is proceeding nonlinearly. What I find works best is to explore with the data in the live prompt, which gives you interactivity, and then build up slowly a master collection of functions and commands that when run, with a single command, can reproduce the results you got while exploring. Then to come back to the data at a later point in time, you have to run one script file on the raw data. Of course, this is kind of the point of the Jupyter notebook, but I find that often when I want to change parameters, the ability to jump around and redefine things means I do. By moving from IPython to a script/functions I run, I ensure that everything progresses linearly. Idk, just my two cents.

<comment><USER49> Which talk? I didn't see any link to a video.

<comment><USER48> Here they are - they're definitely irreverent, but I find myself strongly agreeing with everything he says.

[https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...](https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV- dkAIsUXP-AL4ffI/edit#slide=id.g3d168d2fd3_0_130)

[https://www.youtube.com/watch?v=7jiPeIFXb6U&feature=youtu.be](https://www.youtube.com/watch?v=7jiPeIFXb6U&feature=youtu.be)

</comment></comment></comment><comment><USER50> Many comments are about implementation details such as JSON format of Jupyter and comparing user experience with IDE and shell and fail to see the fundamental difference between Emacs and Jupyter, which is in this quote in the article:

"In many cases, it's much easier to move the computer to the data than the data to the computer," says Pérez of Jupyter's cloud-based capabilities. "What this architecture helps to do is to say, you tell me where your data is, and I'll give you a computer right there."

</comment><comment><USER51> Jupyter is lovely (and JupyterLab looks _delicious_ ), but the set up required to achieve a reproducible local server with the R kernel and versioned R packages is 100% not.

Installing R packages through anaconda is like pulling teeth and the docker images for my Jupyter notebooks push past 6GB and take multiple cups of tea to build.

Is there a good solution I'm missing? A good hosted solution perhaps?

<comment><USER52> I run [https://nextjournal.com](https://nextjournal.com)

We archive full-stack reproducibility by allowing you to install arbitrary software and version these environments using docker. You can reuse these in other articles or pull and use them locally. `xoxo` is a signup code you can you if you want to give it a try.

<comment><USER51> Looks cool, do you have an article that explains the stack from a technical perspective? Is this based on Jupyter?

<comment><USER52> Not yet. It isn't based on Jupyter but it's all written in Clojure. Been meaning to do a writeup on our stack for some time…

In addition to our own runtime protocol, we support Jupyter kernels and you can import Jupyter and (R)markdown documents.

</comment></comment></comment></comment><comment><USER53> I recently wrote on how to build/grow clean software out of Jupyter notebooks and on pitfalls to avoid when coding like that: [https://github.com/guillaume- chevalier/How-to-Grow-Neat-Soft...](https://github.com/guillaume- chevalier/How-to-Grow-Neat-Software-Architecture-out-of-Jupyter-Notebooks)

<comment><USER53> I submitted it to HN too, why not! Here: [https://news.ycombinator.com/item?id=18339703](https://news.ycombinator.com/item?id=18339703)

</comment></comment><comment><USER54> What was the earliest of these tools? Mathcad? Mathematica? Maple?

<comment><USER55> there's a relatively esoteric paradigm known as "literate programming" which has been around since Knuth (he wrote the book [0]) and that has some software tools associated, of which Jupyter is a particularly web-age example.

[0]: [https://en.m.wikipedia.org/wiki/Literate_programming](https://en.m.wikipedia.org/wiki/Literate_programming)

<comment><USER56> Literate programming _was_ esoteric, true, but the concept saw a huge renaissance in academia and data science with the advent of RMarkdown¹ which for many of my colleagues is the default way of preparing technical documents. Another area in which literate programming has become hugely popular is Emacs' Org-mode ecosystem which has fantastic support in the form of Org Babel². I use literate programming for almost everything. Research papers, tech reports, notes, experiments, teaching materials, letters, student evaluations, and so on. It's completely ridiculous how useful it is once you get the hang of it and make it your default document type.

[1] [https://rmarkdown.rstudio.com/](https://rmarkdown.rstudio.com/) [2] [https://orgmode.org/worg/org-contrib/babel/](https://orgmode.org/worg/org- contrib/babel/)

<comment><USER57> I'm still not sold on writing complete program this way (I did try, with various level of success), but even partially-literate approach is ridiculously convenient if you happen to live in Emacs.

I do my task management and note-taking in Org-mode, and recently I found myself doing things like jotting in the middle of my notes[0]:

      #+BEGIN_SRC http
        GET address.to.api:123/sth
      #+END_SRC

and tapping CTRL+C twice, to get the actual response of the API I was debugging.

Or, the other day I was making notes about gravity batteries, and was wondering how efficient is one startup's solution. I briefly thought about firing up Jupyter, but then simply wrote the following[1]:

       these guys power a LED (or three?) with a 0.1W, generated through dropping
       a 12kg weight down 1.8 meters over 20 minutes.

       Doing some basic math on that:
       #+BEGIN_SRC elisp
         (let* ((m 12)
                (g 9.81)
                (h 1.8)
                (_t (* 20 60))
                (E (* m g h))                    ; E = m*g*h
                (P (/ E _t))                     ; P = E/t
                (efficiency (/ 0.1 P))           ; efficiency = Pout/Pin
                )
           `("ideal power [W]" ,P
             "efficiency [1]" ,efficiency))
       #+END_SRC

Typing CTRL+C twice, out pops:

      #+RESULTS:
      | ideal power [W] | 0.17658000000000001 | efficiency [1] | 0.5663155510250312 |

(which is automatically rendered as an org-mode table I can operate on, or even reference in other code snippets).

Point being, note-taking in org mode makes it ridiculously easy to invoke any programming language you hooked up to Emacs without breaking your flow, and you get to edit the code in the mode specific to that programming language - so everything from autocomplete to linters work.

I know Emacs is niche, but I can't recommend it enough.

\--

[0] - BEGIN/END_SRC block is under convenient autocomplete of "<s TAB".

[1] - this is a real note, so if I got the physics wrong, I just made a fool of myself publicly -.-

</comment></comment></comment></comment><comment><USER30> I kind of find Jupyter an indictment of other coding tools really, it's 2018 and they're normally kind of weak or kind of unprogrammable. Feel like we're waiting for someone to really reinvent Emacs, preferably using web tech.

Most editors can't open a terminal that you can use VIM keybindings on to search/navigate history and treate like any other buffer.

VSCode -> not currently possible because they wrote it in a restrictive way with Panel as a special case very different to code window Atom -> probably possible but I don't think terminal-plus is quite it. Any IDE I've tried -> not possible. Emacs -> possible.

Not that this is the be all end all feature but it is useful as hell and kind of a litmus test for whether you can program your environment.

edit: LightTable seemed kind of cool but became abandonware like the author's other projects

<comment><USER58> I use Jupyter in emacs: ein-mode. The whole concept of programming in a browser sounds bizarre to me. I have a tool that's designed for programming (emacs) and a tool that's designed for streaming cat videos (firefox) and I use the latter for programming? Thanks but no thanks. Ime emacs works pretty perfect with jupyter too, there is no need to use firefox for something it's not designed to do for my full-time job.

<comment><USER30> I have also tried ein-mode, and sometimes use it. I haven't used it enough to have a super educated opinion but it didn't blow me away.

I don't think it supports cell folding (?) as an example missing feature

Minor point but it also can't/shouldn't support widgets, which we use at work. Any extension to jupyter is going to be written in javascript so there's an element I'd be locking myself out of the ecosystem.

I didn't mean this as a criticism of emacs, my post said that the best thing I know is emacs, just that it's (probably) not the future of editors imo so I'm reluctant to throw 100 hours into it.

<comment><USER57> > _I don 't think it supports cell folding (?) as an example missing feature_

If I understand what you mean correctly, that would be handled by built-in outline-minor-mode, or by a third-party Emacs module like yafolding or fold- this.el. Emacs packages tend to be made to compose well with other packages (it's a requirement given how everyone's Emacs is a special snowflake, unlike any other Emacs).

As for widgets/Jupyter extensions, then yes. Emacs can't really help you there AFAIK.

</comment></comment></comment></comment><comment><USER59> "Two additional tools have enhanced Jupyter's usability. One is JupyterHub, a service that allows institutions to provide Jupyter notebooks to large pools of users. The IT team at the University of California, Berkeley, where Pérez is a faculty member, has deployed one such hub, which Pérez uses to ensure that all students on his data-science course have identical computing environments. "We cannot possibly manage IT support for 800 students, helping them debug why the installation on their laptop is not working; that's simply infeasible," he says." I think this result is a real winner, I recall the problems of setup in university student labs. Good win for reducing teaching friction.

</comment><comment><USER60> I recently got a Jupyter Notebook, and found it's a large JSON document, with some sections in markdown and some in Python. A browser could omit the Python, and an interpreter could omit the markdown.

Would this work for other languages? Maybe JavaScript or Powershell instead of Python?

<comment><USER61> Can and does! Jupyter notebooks have a flexible architecture for running anything (called "kernels").

Notable examples...

R: [https://github.com/IRkernel/IRkernel](https://github.com/IRkernel/IRkernel)

node: [https://github.com/notablemind/jupyter- nodejs](https://github.com/notablemind/jupyter-nodejs)

In my mind, one of this big advantages of jupyter is its extendability. I was able to quickly modify notebooks to run unit tests, so we could use them for projects at DataCamp: [https://www.datacamp.com/projects](https://www.datacamp.com/projects).

</comment></comment><comment><USER62> Have people been finding that AWS SageMaker disconnects you and you have to restart after a couple hours?

I am curious people's thoughts on using Jupyter for long-running code. Having a totally self-contained experiment in one notebook, even if it long-running, is very useful for reproducibility. It works fine on my local laptop and a remote server, but not with SageMaker.

</comment><comment><USER63> The problem with Jupyter isn't with what it does. It's the people who use it.

My experience as a data engineer/architect/application developer attached to data science teams for a while now is that most really good data scientists are very good at what they do, write somewhat competent code, and do not--in any way--care about writing good software or good application code.

Jupyter is a bane of my existence because people who use it want to use it for everything. Oh, it can have a web interface? Okay. The app is done. DEPLOY TO WEB USERS! NOW!!

It's a great tool. A lot of the people who use it are not software engineers, and they don't want to be. For a lot of people it's the straight line from point a to point b.

But in my experience, legit data scientists are pretty smart and are willing to learn a little if you're willing to give a little. This is a good exercise because they are typically skeptical about everything. So you have to be really secure about why you want certain things done certain ways, and why you definitely don't want things done other ways.

It's a good exercise for everyone involved if you have the right team dynamic and mutual, healthy respect for each other.

If you don't . . . well, then Jupyter notebooks completely suck.

</comment><comment><USER64> I love notebooks as a way to present information, data, code and computations.

However, I cannot stand typing _any_ text into a web browser window. Is there any way to edit a jupyter notebook with a text editor and then run it in the browser? The native json is not really human-editable.

<comment><USER42> I have felt the same way in the past. There are some ways to do this, but none is great. Unfortunately using a text editor to put in the notebook text areas is not that straightforward because of security features in modern browsers. Since jupyter is actually a server (usually running locally) it's possible to communicate directly with it from a sufficiently advanced editor, but haven't seen any good execution of that idea. There's also the ipymd ([https://github.com/rossant/ipymd](https://github.com/rossant/ipymd)) format, which is just markdown, and seems more or less what you want, but you lose the interactivity and display of images/plots/HTML/etc. Personally, I've found the "jupyter-vim-binding" ([https://github.com/lambdalisue/jupyter-vim- binding](https://github.com/lambdalisue/jupyter-vim-binding)) to be a relatively acceptable emulation of vim keybindings for the code editing.

<comment><USER64> This is not what I mean. Mine is a problem of file formats, not of interactivity. I want to edit a text file alone, without need for any web browser in my computer. Then I push the notebook to git, and somewhere else it is opened by the browser.

This would be possible today if the notebook file was python code with comments, for example, instead of an uneditable json.

<comment><USER65> This is quite a neat idea. A couple of people have mentioned Jupytext around here. I found a guide with animations which appears like it might do what you want. [0]

I personally look forward to trying this out, as it means that I can use Jupyter in a way that doesn't mean adapting my workflow to the tool so much.

[0] [https://towardsdatascience.com/introducing- jupytext-9234fdff...](https://towardsdatascience.com/introducing- jupytext-9234fdff6c57)

</comment></comment></comment></comment><comment><USER66> There is no "why" because it is not. An awkward JSON format plus the inability to survive network interruptions. A WiFi router goes down during a long computation, and voila -- the results are lost, which is unimaginable with RStudio.

</comment><comment><USER67> Out of curiosity, what other options are there?

<comment><USER3> for python, org-mode in emacs will do it. but that ties you down to emacs.

<comment><USER56> Org-mode was a total game changer for my work life. I use it for almost everything. Yes, it ties me to Emacs. I don't see this as a problem.

</comment></comment></comment><comment><USER68> Link is dead?

</comment><comment><USER69> Jupyter could be viewed as a modern reminiscence of Lisp Machine UI, without the elegance of homoiconicity, of course.

Python is a good "glue" for optimized C++ or fortran libraries which the core of things like tensorflow or numpy. Everything fits together nicely.

<comment><USER70> > Jupyter could be viewed as a modern reminiscence of Lisp Machine UI

maybe some part of it, but the Lisp Machine UI has a full window system, many different applications based on it with different UIs (font editor, file system browser, process overview, chat program, terminal, Zmacs editor, debugger, documentation browser, documentation editor, drawing program, ...)

<comment><USER71> That is what Jupyter Lab is becoming.

<comment><USER70> great, but how is it Lisp Machine UI like?

</comment></comment></comment></comment><comment><USER72> I hate Jupyter notebooks; I seriously do. Maybe that's an irrational sentiment, but I find them nauseating to me in the same vein that I find country music nauseating.</comment>


A Raft implementation in Haskell - <AUTHOR> https://github.com/adjoint-io/raft <comment><USER2> There is an older Haskell implementation of Raft with additional changes for Byzantine failures: [https://github.com/kadena- io/juno](https://github.com/kadena-io/juno)

</comment><comment><USER3> A great README file, explaining things in detail, from the ground up.

Kudos.

<comment><USER4> Agreed, very nice docs, both for how it works and examples, and also for using the code.

For Haskell fans, you might also like tha Pact DLT contract language, implemented in Haskell with, Lisp syntax.

</comment></comment><comment><USER5> There are others: [https://github.com/NicolasT/kontiki](https://github.com/NicolasT/kontiki)

</comment><comment><USER6> Does anyone actually use raft these days

<comment><USER7> Is there another distributed consensus algorithm that you think is more prominent? Paxos is on the decline. AFAIK Raft underlies most new distributed systems. Kubernetes for example relies on etcd which is a Raft implementation.

<comment><USER6> Ah, I didn't know Paxos was on the decline or that etcd uses raft

<comment><USER8> I wouldn't call Paxos on the decline. It has several nice properties that Raft lacks[1]. However, implementing it correctly is more brutal than Raft, which is cut into easier-to-understand components.

Also, the paper introducing Paxos[2] is famously obfuscated, while the Raft website[3] trivializes the problem, and has ready-to-use libraries.

As a result, there are more recently-developed systems in Raft than Paxos, but I expect there is more volume hitting Paxos systems than Raft ones in the world.

[1]: [http://rystsov.info/2017/02/15/simple- consensus.html](http://rystsov.info/2017/02/15/simple-consensus.html)

[2]: [https://lamport.azurewebsites.net/pubs/lamport- paxos.pdf](https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf)

[3]: [https://raft.github.io/](https://raft.github.io/)

<comment><USER9> As a member of a team that has implemented Raft in production ([https://fauna.com](https://fauna.com)) and in conversation with others who have, the general consensus is that Raft, Paxos, and even Viewstamped Replication are variants of the same fundamental algorithm.

There is an interesting paper about the commonalities in consensus here: [https://arxiv.org/pdf/1309.5671.pdf](https://arxiv.org/pdf/1309.5671.pdf)

<comment><USER10> Thank you for sharing an interesting paper. The similarity between Raft and Viewstamped Replication is easy to gauge. The original Raft paper too acknowledges this similarity. OTOH, the similarity between Paxos and Raft is not very apparent (to me). The paper in the link seems to predate Raft, and only considers VR, Zab and multi-Paxos.</comment></comment></comment></comment></comment></comment>


McKinsey and Company: Capital's Willing Executioners - <AUTHOR> https://www.currentaffairs.org/2019/02/mckinsey-company-capitals-willing-executioners <comment><USER2> excellent article about the flaws of mckinsey as an establishment. it's true that their grift is inexcusable, amoral, and a burden to humanity -- but the OP covered that. the biggest myth surrounding mckinsey is that they're competent.

my favorite part about people from mckinsey has been that they're stunningly incapable. need some quick research done? after a lot of hand-holding, you'll be given an uninsightful and dubiously comprehensive summary of the mainstream view on the topic of your choice and a pivot table you didn't want or need, roughly 10 days after you needed it. you won't need to proofread the report, of course. but that isn't what you cared about anyway.

need something obscure and complex deeply analyzed and fully understood in an insightful way? again, you'll get some flashy combination of excel and powerpoint shit you don't need or ask for, maybe even ahead of schedule, but you'll also get an irrelevant speech about whatever tangentially topical prestigious person/institution the mckinsey person recently bumped elbows with. you'll also realize that the person working on the thing you wanted doesn't really have any understanding of the thing you wanted them to understand at all -- they're just regurgitating whatever they could find without analysis.

however, they've all been imminently presentable people who can articulate themselves clearly and appear to be very serious. this has been my experience with no fewer than 4 former employees -- all of whom were hired to perform tasks exactly like one would expect a business management consultant to be capable of -- but take it with a grain of salt anyway.

<comment><USER3> I think the article makes a pretty persuasive case why competence isn't a core selling point of McKinsey (or many other management consultancies) anyway. What many companies want is for an external party to tell them to do what they've already decided to do anyway, but would rather not take personal accountability for. They're paying for the amorality - if the suggestion comes from someone else, it's not _you_ being amoral, it's a consultancy with a prestigious brand that everyone else listens to, even if this unspecified "everyone else" is hiring McKinsey for the same reason you are: to defer blame for doing something abhorrent.

<comment><AUTHOR> Yes, just so. Criticizing McKinsey-type outfits for being amoral misses the point. For them, amorality _is the product._

</comment></comment></comment><comment><USER4> I worked at McKinsey a while ago. I'm willing to bet this person was an Analyst (the recent college grads). Analysts were most likely to believe the firm's marketing schpiel about how important they are in the world. In reality, McKinsey just kinda puts out power point decks. People past a certain level at McKinsey admitted that in quiet confidence.

</comment><comment><USER5> Excellent article, thanks for sharing.

I wonder, if one wants to disrupt management consulting, what would it be ? Massive open annonymized datasets ?

<comment><USER6> This whole "management consulting" thing is but a symptom of a deeper, underlying problem.

What really needs to be disrupted is the governing ideology of the political- economic system itself.

<comment><USER7> I'm sitting here in Tokyo, in a now rich country that was a developing nation when my dad was born, arguing on my magic pocket computer with someone who may we be in the opposite side of the world. All thanks to the current "governing ideology of the political-economic system."

Making the case that needs to be "disrupted" is a tough road you've chosen to hoe.

<comment><USER6> Yes, the gadgets are nifty and -- for the time being -- life is relatively swell (for some).

But there are downsides, too -- and the long-term prognosis of what this system holds in store for us... just doesn't look good. And may potentially end quite disastrously, in fact.

<comment><USER8> Can you elaborate what the long-term prognosis is?

<comment><USER6> The inevitable outcome of a system predicated on infinite growth on a planet with finite resources.

</comment></comment></comment></comment></comment></comment><comment><USER9> Just a reminder that Mayor Pete is an alum of this organization.

<comment><USER10> I did not know that!

</comment></comment><comment><USER11> who would have thought that capitalist realism wasn't the antidote to all of the world's problems? certainly not thatcherite or reaganite policy building ideologists!

<comment><USER12> > capitalist realism

I've never heard this term before, but it's perfect. The implicit assumption that certain capitalist economic theories _are literally the real world_ (rather than limited and imperfect models) is something I've been noticing more and more, but never had a term for.

<comment><USER13> highly recommend the book: [https://libcom.org/files/Capitalist%20Realism_%20Is%20There%...](https://libcom.org/files/Capitalist%20Realism_%20Is%20There%20No%20Alternat%20-%20Mark%20Fisher.pdf)</comment></comment></comment>


Testing Privacy-Preserving Telemetry with Prio - <AUTHOR> https://hacks.mozilla.org/2018/10/testing-privacy-preserving-telemetry-with-prio/ <comment><USER2> Great strategy here to aggregate results by splitting and distributing the data to different nodes.

As for them vetting third-parties as mentioned in the article, I'd actually prefer Mozilla to host different telemetry collection nodes itself and just route them round-robin anonymously within their own network.

IMO having to extend intermediary trust might be taking one step forward two steps back here if who collects the data != who analyzes it.

<comment><USER3> You don't need to trust all third-parties. _Crypto magic_ guarantees that what you need is at least ONE server being honest.

If you trust Mozilla, then you have nothing to worry about, except the proof or program may be wrong ;)

Third-parties are for someone who doesn't trust Mozilla. Or that Mozilla could be compromised by an unknown adversary.

</comment></comment><comment><USER4> It is probable me but I don't understand what are the advantages here or what that picture supposed to convey. Data is transmitted and aggregated, albeit in a fancier way. What does it have over Telemetry?

<comment><USER5> Here's a cool example of secret sharing from the paper[1]:

> App store. A mobile application platform (e.g., Apple's App Store or > Google's Play) can run one Prio server, and the developer of a mobile app > can run the second Prio server. This allows the app developer to collect > aggregate user data without having to bear the risks of holding these data > in the clear.

However I would like to know why Firefox doesn't want to just adapt RAPPOR instead. It seems like a better fit for general telemetry, and it's already used by Chromium.

I guess you can intersect and union Prio datasets in ways you can't with RAPPOR?

[1] [https://crypto.stanford.edu/prio/paper.pdf](https://crypto.stanford.edu/prio/paper.pdf)

<comment><USER3> Google RAPPOR technique adds noise to the report at client (user) side. The noisy reports are then collected to a central server for analysis.

Doing this guarantees differential privacy, a strong privacy protection.

However, the _major_ disadvantage of RAPPOR is that we get a collection of very noisy data points. Therefore, we need a very big collection to learn anything useful at all!

If the number of clients/users/records is small, RAPPOR is practically useless.

Prio, on the other hand, adds NO noise to client's reports. It uses _crypto magic_ to compute _some_ functions (mean/sum/...) on the collection of reports by a network of servers. Only outputs of these functions are known. No reports are leaked until at least one server is honest!

However, Prio has no guarantee that the computed output doesn't leak information about client's reports (in fact, it surely leaks some information). While, RAPPOR guarantees this protection statistically!

A current research direction is to add noise to the output of Prio as such it guarantees differential privacy.

</comment></comment></comment><comment><USER6> I worked on the Firefox integration (per the article) if anyone has specific questions.

If you want more detail on Prio itself, I'd suggest [http://blog.ezyang.com/2017/03/prio-private-robust-and- scala...](http://blog.ezyang.com/2017/03/prio-private-robust-and-scalable- computation-of-aggregate-statistics/) as a more gentle introduction than the research paper.</comment>


Open Source Developers are Rock Stars - <AUTHOR> http://www.kbedell.com/2011/04/28/open-source-developers-are-rock-stars/

<comment><USER2> The thing is, being a rockstar is a mostly about sleeping on buses, not washing, and using drugs to dull the loss of identity caused by never being in the same city for more than 6 hours.

What people want is to be successful and adored. That's all anyone ever talks about in these 'programmer as rockstar' articles. As much of a sense of martyrdom startup people cultivate about their work habits, none of it is as physically difficult or as psychologically degrading as actually being an truly internationally famous rock musician.

There's a reason why they all act so damn crazy after a certain point in their careers.

If you want to be celebrated and rewarded way out of proportion to your contribution to society, just say so, dont make some false analogy to a completely unrelated profession.

<comment><USER3> I agree with most of your comment, but I think there are a few inaccuracies worth pointing out.

 _> The thing is, being a rockstar is ... [about] using drugs to dull the loss of identity caused by never being in the same city for more than 6 hours._

I am not entirely sure that's the root cause for rock stars' drug use. It's certainly part of it, but in large part it's intimately tied to the socialization context that surrounds the entire rock scene.

When a fan is going to a show, it's a special event. A large population of those fans use drugs in an effort to enhance their enjoyment. They also want to share that enjoyment with the band, and they are quite willing to share their supply with the band. In other words, the rock band basically has a free, unlimited supply of drugs and a social expectaton to take advantage of that.

 _> There's a reason why they all act so damn crazy after a certain point in their careers._

I'm sure you didn't mean that literally, but there are several rock stars who don't end up crazy. Led Zeppelin, Brian May (a PhD!), BB King, Buddy Guy, Ben Folds, Nickelback, Darius Rucker (Hootie), I could go on forever... The reality is that it's a small subset of rock stars that go crazy, and I would argue that the proportion is actually lower than for the average population of somewhat-heavy drug users.

<comment><USER2> To be honest the whole comment was pretty flippant and ill-considered, I was just trying to combat the idea that the life of a 'rockstar' is really worth idealising outside of the material rewards, which aren't even that great.

Rockstars get paid just enough so that they'll be too busy spending it to realise that they're getting stiffed.

In the words of Chris Rock: _"Shaq is rich. The white guy who signs his cheque, is wealthy ... If Bill Gates woke up tomorrow with Oprah's money, he'd jump out a fucking window."_ \- <http://www.youtube.com/watch?v=4m37JkkGjAY>

_> there are several rock stars who don't end up crazy_

I'd be willing to bet though that the eventual craziness of rockstars is fairly closely related to what proportion of their career was spent on a hardcore, label-mandated touring schedule.

</comment></comment></comment><comment><USER4> Unless your job gets you laid as much as a rockstar, chances are you are not a rockstar. Successful entrepreneurs may be a little closer to the definition of a rockstar.

<comment><AUTHOR> I tried to search the first mentions of developers as rockstars. Indeed the "Developers are the new Rockstars" (beyond Open Source) is a motivation statement.

</comment></comment><comment><USER5> Developers will be rock stars only when people compare rock stars to developers rather than the other way around.

</comment><comment><USER6> More "mildly popular local pub musicians ... without groupies"

Stephen Tyler != Linus Torvalds

<comment><USER7> Maybe it should be called "stage technicians". Working hard in the background so others can become stars.

<comment><USER8> They're called "roadies"

</comment></comment></comment><comment><USER9> There's a typo in the title. Should be: "Open Source Developers Wish They Were Rock Stars"

</comment><comment><USER3> I'm going to be a cynic here and say that the author's real thrust seems to be to reconcile his childhood dreams of being a rock star with his current station in life. The definition of rock star precludes the vast majority from ever attaining it, and most of us have to "give up the dream" at some point.

A rock star is a public figure that a sizable portion of the general population respects, envies, appreciates, and to some degree emulates.

I agree with many commenters here that the almost no tech-related workers are rock stars. Steve Jobs (and even he is debatable), Bill Gates, and now Mark Zuckerberg are the only legitimate tech rock stars. If you claim anyone else, you're just kidding yourself.

Everyone wants to be adored and respected -- it's what makes us care about "meaningless" karma points. It's great to find from time to time that a total stranger appreciates you. That does not mean you're a rock star. You can come to grips with that, delude yourself, or actually become a true rock star. Changing the definition of _rock star_ is not an option.

</comment><comment><USER10> I like the analogy - it works to a degree.

I'd say more like it covers the whole spectrum from solo musicians to garage bands that suck, to good garage bands, to "rockstars".

Sure, those working on super high profile projects can be like rock starts to the software community, not becuase they get lots of chicks, but they attain a following, but because their work is out there for everyone to see and use, learn from, and appreciate. Other aspiring programmers can say "Wow! I want to do that someday."

Those with real talent (including the ability to execute!) can rise to the top, even if just for a while. Solo people can create game-changing technology (eg: Bram Cohen / BitTorrent protocol)

But just like rock bands - you can also sit in your garage making decent music forever, having fun, but never becoming famous. That doesn't make you a rock star, but it still gives you practice, experience, and confidence.

</comment><comment><USER11> Not answering the main thesis of the article, but I found this interesting:

I've made some googlefight-s. Turns out that Linus Torvalds has more google results than most of the absolute top pop/rockstars, (much more google results than John Lennon) Other famous open source developers are pretty 'weak' compared to famous musicians though, so Linus is an absolute outlier.

Ordered by 'num. of google-results popularity':

Britney Spears: 5730000 results

Michael Jackson: 5650000 results

Linus Torvalds: 3640000 results

James Hetfield: 1850000 results

Elton John: 1740000 results

John Lennon: 1560000 results

Jim Morrison: 698000 results

Jimmy Hendrix: 469000 results

Richard Stallman: 149000 results

John Resig: 43400 results

<comment><USER12> _Turns out that Linus Torvalds has more google results than most of the absolute top pop/rockstars, (much more google results than John Lennon)_

Though to be fair, John Lennon wasn't very active on mailing lists, and didn't even have a blog.

<comment><USER3> My thoughts exactly. And Linus's name was not misspelled like Jimi Hendrix's name was.

</comment></comment></comment><comment><USER13> Non-eye-burning: <https://www.readability.com/articles/9likw7md>

</comment><comment><USER7> After reading the article maybe it should be titled "Open source developers are like local musicians but maybe can make a bit more money".

In general I don't believe that open source developers are rocks stars but the people who use open source. Pretty much all well known web companies use open source but don't open source their flagship products. E.g. Google. Facebook, 37 Signals

</comment><comment><USER14> Now I'm confused, I thought they were ninjas. Or Pirates. One of those, I'm sure

<comment><USER15> They're ninja-rockstars. That's why they stay out of the limelight ;-)

</comment></comment><comment><USER16> Open Source Developers diminish importance of pay by giving things away; society as a whole suffers a bit as a result...

By contrast, rock stars are paid handsomely.

<comment><USER17> At the risk of feeding a troll, are you seriously saying that society is suffering because developers give stuff away? If so, you really need to take another look at the world get your priorities adjusted.</comment></comment>


25 Rules For Living From A (Semi-) Successful 26-Year-Old - <AUTHOR> http://news.rapgenius.com/Ryan-holiday-25-rules-for-living-from-a-semi-successful-26-year-old-annotated

<comment><USER2> Not hackerly, not notable, not cogent.

No thanks. Flagged.

</comment><comment><USER3> this article is a piece of shit</comment>


JavaScript: The Evil Parts from JSConf 2010 - <AUTHOR> http://blip.tv/file/3684946

<comment><USER2> Disappointing. Nothing but respect for Billy Hoffman, but this talk has very little to do with Javascript and almost everything to do with the browser security model. Being able to using request timing to sniff things out of someone's email spool is an "evil part" of the browser and of the application architecture of Gmail. It's not a facet of Javascript.

The only thing in this talk that seemed uniquely Javascript-y was his explanation of di Paola's Prototype Hijacking attack (where you override the Ajax calls to sniff requests). But this is an issue in virtually every dynamic language; it's not a specific flaw in Javascript.

I want to be careful here, because I've had to give this talk a bunch of times --- the one security talk at a generalist conference, which is always going to devolve into a survey talk. I'm sure his audience loved it. I'm not sniping at Hoffman. But on HN, when you say "Javascript: The Evil Parts", I'm really wanting to see something about the evil parts of Javascript; like, I don't know, maybe automatic semicolon insertion being exploitable.

<comment><USER3> Thomas,

Man, if only there was something negative and bad around semicolon insertion, besides JSLint yelling at you ;-)

I appreciate the props, especially coming from someone of your caliber. The talk was exactly what you said it was, a survey of nasty tricks that people have used. The goal was to review features of JavaScript, the DOM, plug-ins, etc, that seemed like a good idea (getting computed styles, rich error handling around the Image object, window.onerror as a massive TRY...CATCH that can recover syntax errors, runtime modification of code, etc) and how those have been twisted to create some nasty problems. More importantly, nasty problems that we really cannot patch away. If we know how we created design flaws in the past, hopefully we will not do them in the future.

Billy

</comment></comment><comment><USER4> This presentation was awesome. In particular, the hydration/dehydration explanation at 51:35 was deliciously evil: replacing ones and zeroes with tabs and spaces to not just obfuscate javascript, but actually hide it from human view altogether.

</comment><comment><USER5> The most disconcerting part for me was about attacking services inside a firewall using JS, starting around 45:00. He explains how somebody once wrote a page that, if you load it, would log in to your wireless router (if it had the default password) and hijack your DNS and all your requests.

<comment><USER2> But this has almost nothing to do with Javascript! The problem here is that the browser engine to which your Javascript context is attached will make virtually arbitrary HTTP requests based solely on the contents of the DOM.

If you think faking HTTP login and action requests from Javascript and the DOM is bad, Google "samy nat pinning", which you can do without JS, and which will trick many soho routers into directly allowing remote users onto your network.

<comment><USER5> It doesn't have to be a fault in the JavaScript language for it to be disconcerting or fascinating. That NAT pinning attack is also disconcerting and fascinating. An XSS to your router attack is more elementary and so it's easier to grok for the average JavaScript developer and less interesting to a security expert.

<comment><USER2> The fact that Hoffman's router attack _wasn't_ XSS is an example of why I'm disappointed in this talk. I grant you that the talk was probably fascinating, especially to generalist Javascript programmers. I'm only disappointed in a "Hacker News" sense. Of course, it's not Hoffman's job to be fascinating to (or appropriately headlined on) Hacker News.

</comment></comment></comment></comment><comment><USER6> tl;dw

Transcript?

<comment><USER7> JavaScript often runs in a trusted environment, same origin policy does not save you from accessing certain kinds of information so you can cause mayhem even on private networks, you can overwrite built-in functions to sniff things out, there are CSS exploits (the history exploit for example) which allow you to de-anonymize a user, obfuscating code is easy.

Also, don't implement public key cryptography in JS, because that's stupid.

<comment><USER2> Don't implement _any_ cryptography in JS, because that's stupid.</comment></comment></comment>


Ask HN: Best way to share large data table online? - <AUTHOR>

I have a large csv file (data table) I would like to share online in a nice sortable/filterable/searchable table. Are there any tools that do this - preferably would like a solution with less technical experience as I know friends that have a similar goal. <comment><USER2> Google fusion tables is somewhat useful. I think you can give them money if your data is too big for the free limit, but I could be wrong.

<comment><AUTHOR> This worked great! Didn't know this existed!

</comment></comment><comment><USER3> Look into DataTables javascript library to make the data presentable once you find a way to serve it.

<comment><AUTHOR> I've seen DataTables which I may be able to set up but for non-technical data to be able to share data its simply unmanageable.

</comment></comment><comment><USER4> Just give people the CSV file. Excel/OpenOffice will sort/filter/search it. Your friends are more likely to have used Excel than some web app or JS library.

</comment><comment><USER5> Google Docs Spreadsheet

<comment><AUTHOR> I reached their 400k cell limit :( Although Google Fusion works great as someone else pointed out below!

<comment><USER5> Didn't know there was a limit - interesting :)

</comment></comment></comment><comment><USER6> How large is "large"? Thousands of rows? Hundreds of thousands? Millions?

<comment><AUTHOR> Tens of thousands of rows for me. Though may be higher for some of my buddies.</comment></comment>


Credit card with a fingerprint sensor revealed by Mastercard - <AUTHOR> http://www.bbc.co.uk/news/technology-39643453 <comment><USER2> That's nice, when they steal your card, your fingerprint is always already there. That will be so easy to "hack". Not even a real hack needed</comment>


Verizon Launches Video Game Innovation Challenge - <AUTHOR> http://www.freebeeperksgaming.verizon.com/ <comment><AUTHOR> Verizon is asking developers and publishers, "How might you reward player achievements in your game play with free data from Verizon Wireless?" In addition to awarding campaign credits, there may be exposure for game titles through published profiles, videos, and interviews. Ideas can be submitted through April 28, 2017.</comment>


What software is made of - <AUTHOR> http://siderea.livejournal.com/1241996.html#cutid1 <comment><USER2> The basic idea, that software is a set of decisions, is insightful. I'm definitely going to borrow this concept when talking about software design to non-technical people.

However, a less insightful decision is to have yellow text on a purple background. I can't force myself to read the whole thing.

<comment><USER3> I did read the whole thing - it was a really well written and enjoyable article. I did have to slog through though, because of the yellow on purple.

My favourite bits:

Waterfall as a expression of power and hierarchy within an organisation, something like: "I tell you, lowly worker, what to do" (my words).

This characterisation of a good client: "These are people who will be a pleasure to work with: they are forward thinking, attentive to detail, serious about this project, and have some idea of what it is they want their program to do."

The idea of a cultural conflict between people who value good design and want to spend time on it, and other who hold it in in contempt. Not something I'd thought about before, beyond some idea that the market doesn't demand quality, and so worth a chew over.

<comment><USER4> Actually the "waterfall" effigy defined in various manifestos was the revolt against civil engineering project managers who entered the software engineering field and believed that "building software" was like building a bridge, a skyscraper, or a sewer system.

The correct solution space wasn't to abandon dependencies and long range planning, but to recognize that software engineering is a specific type of process engineering that involves partially or completely automated processes.

Software doesn't describe the state of a machine in so much as it describes a process. Often developers don't recognize this so much because many of the terms have been concretized into the operating system (ie, process) and get thought of as a container for software instead of the representation of an automated process working in concert on shared resources.

And consequently software is complete when the process is well-defined and repeatable, not when things are bolted together.

</comment></comment></comment><comment><USER5> On larger projects with many contractors and subcontractors, structuring and documenting the (iterative) communication between the "client" and the "contractor" is a challenge. This is largely the problem that the "V" model is designed to solve -- the client "owns" the top of the "V", and puts effort into understanding the problem that they want to solve (Systems Engineering and Modelling) -- the contractor integrates with that model to (iteratively) improve their understanding of the component that they are providing, particularly in terms of the yield curve for the KPIs that their component must deliver -- with that relationship recursing down (sub)contractual relationships until you get to the bottom of the "V". There really isn't that much difference in the fundamentals between software engineering and other disciplines, although the details of the interfaces and the impact of automation is, of course, more significant.

</comment><comment><USER6> A lot of it is also just boilerplate, if by software we mean code...

I've often heard coders talk about "logic" as a vaguely intimidating mass noun, as in "ugh, this module is full of old logic, who knows what's going on here?"

The word "logic" normally implies structure, coherence, and correctness. In coding, that's rarely the case. We still need to learn how to code logically.

<comment><USER7> According to the article's terms I think boilerplate can be referred to as "trivial decisions", e.g. "there's no elevator in this building so I gotta trudge up the stairs" \- there's no real decision there besides "do" or "don't" and a bunch of boring work you have to do in order to reach your goal.

<comment><USER6> I'm thinking also of stuff like complex hierarchies of "object orientation" that don't embody any actual logic, but only serve to impose a "structure" that may or may not have benefits.

Writing code that consists mostly of actual relevant decisions seems like an often-ignored art.

</comment></comment></comment><comment><USER8> Software is an idea morphed into a material configuration.

What the idea "hammer" is to wood and metal is the idea "GTA V" to a computer system.

The only difference is, that everyone can tell his computer to configure itself for running GTA with information from its installation package. But one can not simply tell its wood and metal to configure itself to be a hammer with a documentation of how to build a hammer. The documentation is more of a source code and needs a compiler, a craftsman or machine, that could make the hammer with this docs.

That's why I find these commercials about "stealing software" so funny.

"You wouldn't steal a car!" as if it was the same as downloading software.

It isn't.

Stealing a car is more like stealing a PC installed with some software.

Downloading software is more like stealing the documentation of how to build a car in some format that only the materials cars are build of can understand.

</comment><comment><USER9> Empathy looks like the key to making better design decisions, since defining behavior is basically communicating. (This is why incidental design leaves bad taste—it's noise.) Singling out software as being made of decisions is not entirely fair though, as this applies well generally to significant part of our conscious reality.

On a higher level there are decisions that do not directly define the behavior of the end thing under some specific conditions, but rather apply to the process by which that design happens. Those meta-decisions can also be incidental, and can be felt if you take the end thing as existing in longer time frame.

</comment><comment><USER10> Great software is hand crafted from only the finest ASCII

<comment><USER11> Ah yes, I hear Kuwaiti ASCII is the best, despite the international embargo.

</comment></comment><comment><USER12> Reminds me of a project I signed up for, where I said I could commit a couple of days a week to ensure they were on the right lines. I ended up as the full time product owner / business analyst in charge of 5 staff for 6 months.

</comment><comment><USER13> And yet there is mention of machine code and turing machines in there. TFA is not going to explain what software is to a lay person. It does offer some help for those trying to craft such an explanation though.

</comment><comment><USER14> C-f coffee

Phrase not found.

<comment><USER15> Good point. However, I write good software, but barely drink coffee. Where does that leave us?

<comment><USER11> >I write good software, but barely drink coffee.

I hate to be the one to tell you ...</comment></comment></comment>


Ask HN: What are some tech trends for 2019? - <AUTHOR> Past years, I've seen subscription based services, AI, privacy were top among others. I wonder if any solid trends building up for 2019 such as podcasts? Thank you for any clues. <comment><USER2> My money is on serverless. [https://trends.google.com/trends/explore?date=today%205-y&ge...](https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=serverless)</comment>


Five weights monospaced typeface for the benefit of charities - <AUTHOR> http://almamono.com <comment><USER2> _Really_ bad for _code_ though...

\- the difference between uppercase and lowercase is too low making camelCase very hard to read

\- the colon is aligned to the left of its space making ":=" look horrible

\- "$" is hard to distinguish from "S"

\- [] have funny vertical alignment

\- every char is a bit wider than it should, wasting a lot of horizontal screen estate - what you want to avoid when working with 3+ columns of code on one screen

Bad compared even to Ubuntu Mono. Very bad compared to Source Code Pro. Horrible compared to Consolas.

But looks good on titles of articles referencing charity :) Seems like the intended use for a monospace font... right?

<comment><USER3> And ^ results in a blank character.

<comment><USER4> That's strange! I'll take a look, I know I designed that character! Thanks for the heads up!

</comment></comment></comment><comment><USER5> It's disturbing that the landing page does not mention at all what charity procedes go to. You need to go to the very bottom of the About page to finally find it's for [http://www.unhcr.org](http://www.unhcr.org)

The donation is also done in a sketchy way. Also near the bottom of the About page:

> How does this charity thing work?

> So far I'm using a solution that means I get payments sent to me to my > Paypal and I personally make the donations to charities. I've chosen this > way because it's easy.

In other words, you're paypal'ing money to a stranger and hope that the money makes it's way to charity

To the author's defense, he does have a "Are you to be trusted?" section that acknowledges this. The idea of a pay-what-you-like-for-charity-on-my-side- project idea is neat, but you need some degree of credibility or third-party to make it work. The next side project should be integrating against something that guarantees the promise of the donations.

<comment><USER4> It's a leap of faith, most definitely, and maybe a bit shady when viewed from an outsiders perspective. Just give money to a stranger? Yes. Shady indeed, I agree.

But then again, if you chose to pay for it, which is your choice - you could download it for free - you actually get a font in five weights that has taken me a lot of time to craft.

It's your choice. But you can trust me. It's the Internet - people wouldn't lie on the internet, no?

And - I'll be the first to agree that this project needs more credibility and/or third-party involvement. I've already gotten the ball rolling on some partnerships with third parties for the future. Getting Hacker News'd wasn't even on my horizon. Superexciting! I was expecting friends and family to support this, like on my other design/charity project that I mention and link to on the about page.

</comment></comment><comment><USER6> Microsoft Windows ships with a very nice font called Consolas. It is fantastic for programming.

I use it everywhere, including Gnome Terminal on Ubuntu, usually in an 11 point size.

I settled on this font while trying numerous monospaced fonts: everything I could get my hands on. Boy was I surprised at how I overlooked Consolas.

</comment><comment><USER7> That's an interesting license, and very nice of course.

I got a bit confused, my initial reading and interpretation was that the intent is to have the actual font used in charity and relief efforts, I guess in signage. That's probably not what the author meant, though.

<comment><USER8> It doesn't appear so, but that would be a great 3rd option. Send in your charity registration, and get a free licence to use the font in your charities branding. That could serve to both help charities and build more awareness around the font, creating more sales, and more proceeds. You should mention it to them.

</comment></comment><comment><USER9> I read. I went elsewhere. Here's why.

The pages do not tell me what I would be paying for. For me, it's important to know what font file format is used, because I need to know what and how many hoops I would have to jump through in order to obtain vt fonts. (Best outcome is two vt fonts: one normal+bold, one light+demibold. This is why the multiple weights interested me in the first place.)

* [https://freebsd.org/cgi/man.cgi?query=vtfontcvt](https://freebsd.org/cgi/man.cgi?query=vtfontcvt)

* [https://wiki.freebsd.org/Newcons#Font_Support](https://wiki.freebsd.org/Newcons#Font_Support)

There's also no information about the character repertoire. Does it have everything in the WGL4? Or the W1G? Or less than either? What _does_ it have?

</comment><comment><USER10> If any font maker read this. I would like to have a "bitmap" monospace font as a true type font in the same size as Consolas 15px. With "bitmap" I mean high contrast super crisp, pixel perfect, that don't lose contrast without sub- pixel-antailias.

<comment><USER11> Crispness was one of my primary motivations for learning to make my own programming font, Luculent [1][2]. Consolas is wider so the proportions aren't quite the same. But mine is superhinted to look good with no or limited antialiasing at 15 ppem, among other sizes.

[1] [http://eastfarthing.com/luculent](http://eastfarthing.com/luculent) [2] [https://fontlibrary.org/en/font/luculent](https://fontlibrary.org/en/font/luculent)

<comment><USER10> It looks good on many sizes with anti-alias turned off. But with anti-alias it's very thin and the contrast is bad.

</comment></comment></comment><comment><USER12> Really nice and clean. Would prefer it if it included cyrillic characters and more latin ones to cover all European languages (e.g. Polish ł is missing, Czech š, č and probably several Hungarian letters).

</comment><comment><USER13> spoiler: [http://www.unhcr.org/](http://www.unhcr.org/) is the charity to whom the author intends to donate contributions.

</comment><comment><USER14> Well-intentioned, but needs work, especially from a coding perspective.

<comment><USER4> I totally agree to that! Never really thought this educational experiment would be featured on HN. It has kinda blown up! I'm glad you took a gander though!

</comment></comment><comment><USER15> Good news: the purchasing process works fine.

Bad news: you cannot download the font (the supplied URL, when clicked, first goes to gumroad.com then to cloudfront.net which gives the error: Safari cannot open the page).

</comment><comment><USER16> About page throws up a big scary phishing warning in Google Chrome.

<comment><USER4> No, really? Shit. The only fish I want is Swedish fish. I need to investigate that, can't really see what that would be...

<comment><USER17> I also don't see the warning. Toss your domains into Google webmaster central if you haven't, it'll give you info if it actually arises (and is just generally a good practice.)

<comment><USER4> Thanks, I'll do that right away!

I use Jekyll that generates a static html site. The only third party stuff I have on that page is a MailChimp-widget. So, I'll wait and see what Google webmaster tools will tell me.

</comment></comment></comment></comment><comment><USER18> There appears to be no carrot (^) character for the font

<comment><USER19> I assume you mean _caret_ instead of a vegetable.</comment></comment>