Ask HN: How much of your time is spent on classical algorithmic problems? - <AUTHOR>

Almost none of the time my team and I spend writing code is spent focussed on solving "classical algorithmic problems" - search / sort / comparison etc. It is all writing clear logic.
Our efforts are all around writing unambiguous code which does what it says on the tin (and says on the tin what it does). Code which other programmers can easily work with and code which interacts defensively with the rest of the system and only affects that which its told to.
I'm interested in how representative that is. How many of you have programming jobs which are genuinely algorithmically driven?
What fraction of your time is spent on "classical algorithmic problems"? <comment><USER2> 5% - but it's the very important 5% where I'm optimizing existing code. The "classical algorithmic problems" you reference are almost all optimization problems; there's a naive search/sort/comparison, but how can you do it better?

The algorithmic problems I deal with are things like:

\- Why the hell is this API call running O(n*m) SQL queries? Can I tweak my logic and use of the ORM get it down to O(1)? The ORM doesn't let you forget about the underlying algorithms, it just makes it easy for you to shoot yourself in the foot if you don't understand them.

\- Why is my single SQL query so slow? DB optimization is very much a "classic algorithmic problem", in that you need to understand the operations the DB performs and optimization is reducing its search space / # of operations.

\- We're using too many jQuery pseudoselectors like ":hidden" and they're causing framerate drop on a particular page; can we use some dynamic programming / memoization techniques to dramatically reduce use of pseudoselectors?

\- We need to figure out what font size will allow a piece of text to fit within a container on a PDF, but the bounds of the font (space consumed) don't scale perfectly linearly with the font's point-size. Finding the best fit is a search problem!

\- And that's not even getting into infrastructure scaling issues! Do you know why Vertica might be a better fit for your data set and end user needs than Hadoop? If you understand the difference between an ETL approach and a MapReduce approach to data analysis, you're thinking about algorithms!

I spend a meaningful amount of time on "classical algorithmic problems" while doing frontend and backend web development, even if I've never had to re- implement mergesort.

<comment><USER3> Apart from maybe the last one, none of them are classical algo problems.

They're basic math problems.

Query takes N milliseconds which is too long, mainly consisting of z & c when I run my profiler. It does x * y = z of these things and a * b = c of these things.

As z is the biggest number, can I reduce x or y?

Basic maths.

And even the last one you're better off actually trying out the technologies to trying to armchair theorize which tech is a better fit as without reading all their code you can't really see what they're _really_ doing compared to what they say they do. Heroku being a pertinent case in point.

<comment><USER2> What's "classical" in your book?

Search trees, dynamic programming, and reducing an algorithm's worst-case runtime are about as "classical" as you get; that's why they're in every beginning algorithms textbook.

</comment></comment></comment><comment><USER4> In the last two months I started deliberately increasing the amount of coding time that is classical algorithm time. Not by working through books and problem sets, but by redefining day-to-day engineering problems that have an easy way out - bash away at it with "industry best practices" and whatever comes to mind, until it doesn't break - into formalized versions that become first data-driven, and then a DSL, so that each "100 lines" solution becomes an effective 1-liner, in the fashion suggested by VPRI's work on new languages. This motivates studying at the algorithmic level and researching PL theory to synthesize that ideal solution.

As a result of this, I am writing very, very few lines of code, in a start- and-stop fashion. I believe it is over 70% algorithmic at this point. The reliance on deep insight is leading me to actively avoid the typical coder's flow state and spend a lot more time on "sit and think" instead. If I hit flow for a long stretch something is wrong.

And, certainly, I would have more "shipping code" at the moment without going this route. But it's still ultimately guided by problems I'm familiar with from doing them the "best practices" way - and each time I make progress, I think to myself "why did I not do it this way before?" (It helps that I have a lot of freedom, working alone.)

Also, it's quite scary doing things this way. The high uncertainty about what the solution is, even knowing the problems well going in, is absolutely terrifying.

<comment><USER5> Can you show some code that is the result from this process?

It seems really interesting, although I've always consciously fought with the tendency to overformalize problems I don't yet understand well.

<comment><USER4> I'm embarrassed to show it since I only have parts of a system right now, not a finished system. But as I implied in the OP, the code I'm writing is mostly conventional algorithms and data structures; the twist is in some subtle detail that ripples out towards the top level. I'll give some examples which will hopefully be more instructive than a code dump.

One of my targets was the entity system used to describe a scene in video games. Conventionally this involves an OO hierarchy of some kind, either compositional or inheritance based. The problem is that you don't know up front all of the sorts of properties an entity will need and what algorithms it will work with; the integration of the various subsystems acting upon an entity is a high-maintenance area and tends to involve messy boilerplate data tracking. I pursued the problem further, and came up with a way to move the integration process out of the subsystems code.

This was done by creating a customized set of collections(array, key-value, bag) that associate their data against an entity structure, can handle multiple instances of data associated against the same entity, can efficiently access data of a specific entity, and can remove all of the data related to that entity upon a despawn request. Once these constraints are satisfied, the only thing each subsystem has to do is work with those collections.

Also of interest, and still in progress, is an implementation of J. Paul Morrison's "Flow Based Programming" concepts to control overall program structure. I was looking around at ways to minimize coupling, and this particular method added a degree of structure and contractual behavior.

The first thing that happened after taking on this concept was to work through the ramifications for interactive programs: I had to start with a big data structure containing all the state, and then split it out and merge it together to attain synchronized behaviors. This in turn motivated the entity system idea. It's a very spatial model, so I am now doing some work on a graphical tool.

NoFlo is an example of a more finished FBP implementation than mine: <http://noflojs.org/>

I have some other ideas too, but they're in a very early stage so there's little to say. Having prior experience in the domain is a huge help, and I wouldn't be taking this approach to conquer arbitrary new problems. I'd try to make at least one mess first ;)

</comment></comment></comment><comment><USER6> My current job is to write software for those huge assembly-line robots you see in factories.

I entered this field (taking a significant pay cut) thinking that maybe I could put some of my algorithmic knowledge to use. Turns out 90% of my job is rote logic - rewriting process flows and the like. Nothing too smart. Really boring tedious stuff.

The 'cleverest' task I did was implementing a Kalman filter to smooth over some encoder readings. That was the most fun I had in a while.

</comment><comment><USER7> Averaged over my entire career, about 0%.

In isolation, I've had a few brief stretches where I spent some time doing low level algorithmic stuff. I wrote some sorting/searching stuff by hand once, because I was working on external files, not doing it in memory. There was, at the time, either no library that did exactly what I was trying to do, or I didn't know about it.

More recently I've done some graph theory based stuff (Floyd-Warshall algorithm for "all pairs - shortest paths"), as I was experimenting with some social network analysis stuff that I wanted to do. But even that probably won't go anywhere. As I explore the capabilities of existing off-the-shelf graph database products, I'm finding most - if not all - of what I need.. At one point I thought I might need to roll my own, but it's a burgeoning field and there are more and more OSS options popping up all the time.

</comment><comment><USER8> The most useful part of studying algorithms is knowing what problems they solve. If you've never heard of sorting algorithms, you'll waste time implementing a bad solution.

(Everyone knows about sorting algorithms, but many algorithms are not well- known. They're almost never needed, but when they're useful, they're _really_ useful.)

<comment><USER9> I would say the most useful part of studying algorithms is knowing how to analyze them and understanding the meaning of asymptotic analysis. Understanding that asymptotically fast algorithms might be slow for the problem sizes you are dealing with is the most important thing an algorithms course can teach.

<comment><USER10> This. I've been marking undergrad assignments, and you would not believe how many sets and sorted sets I've seen implemented by linear scan over an unsorted array (or linear scan over a sorted array with linear time insert, and a few other, worse things). This lack of understanding of algorithms leads to slow, unscalable solutions, and even writing business middleware you at least need to know which standard library collections to use for good performance.

</comment></comment></comment><comment><USER11> Leading up to my current job, virtually none. However, at my current job (Firebase), it now makes up a significant percentage of what I do. The way I've described it to friends is: "You know how most companies ask you tree traversal/sorting/search questions in an interview, then when you show up for work the first day, your job turns out to be 'add this button to this web form'? We actually work on those algorithm problems." It helps that we are an infrastructure company and that our user interface is an API.

If this is something you're looking for, my recommendation is to look at either large companies that have large infrastructure needs (Google, Facebook, Twitter, Amazon, LinkedIn, etc.) or small infrastructure-related startups, depending on your preferred working environment. If the hard parts of your job are offloaded to the database/language runtime/middleware, look for companies that work on databases, language runtimes or middleware. And in the interview, ask them for examples of hard problems that they have solved.

Also, I have absolutely nothing against the companies that don't do this kind of work. In fact, Firebase exists so others don't have to do this kind of work. This is just an observation that occasionally, the skillset tested in these interviews is not in line with the responsibilities a successful candidate will end up having, so make sure you ask.

</comment><comment><USER12> No one spends time on job solving "classical algorithmic problems", they are solved already ;) Unless you work in research and looking for more efficient solutions.

But understanding algorithms brings you to a new level and opens a wide scope of other problems you can solve by applying this knowledge.

If you are not using knowledge about algorithms in your programming job it doesn't mean such knowledge is useless. Maybe that means your problems are not hard enough for this knowledge to be applied.

</comment><comment><USER13> I dont spend a lot of time on these kind of problems. But the times I do, it's an exillerating experience. Knowing the theory of big O notation and time complexety of algorithms has allowed me to build amazing things, I've even used for GUI and JavaScript in the browser.

The most exciting stuff has been implementing things where I have been unsure if what I was trying to would be possible at all in a browser. The feeling you then get when it actually works, and you know that you've pushed a limit that you have never seen anyone do in the browser before, can' be beaten.

If you push yourself to the limit, think outside the box etc. You are going to need knowledge of algorithms.

</comment><comment><USER14> I did game development in a previous lifetime. Probably half my day was implementing algorithms and thinking about the correct data structure for a problem.

I spend less time these days implementing algorithms. However, you need a good understanding of algorithms to decide between using ActiveRecord or a service layer. You need a good understanding of algorithms to pick between a relational DB, a key-value store, or a document store.

Maybe you aren't picking difficult enough problems if you don't have to think about this kind of stuff.

<comment><USER15> I've mainly done web development and typical university style assignments for the bulk of my programming experience- barely encountering algorithmic problems that weren't well defined.

Recently, I started creating a throwaway physics/game engine for fun and it is an awesome way to learn some really cool algorithms. Also, I did a small image manipulation app that exposed me to some cool stuff as well.

As a new grad it definitely pushed me more towards game development- would you mind sharing your reasons for leaving?

<comment><USER2> Game development is hard. The average engineering role at a games company involves a lot of work on hard deadlines.

It's also tough for some engineers because there's usually a very clear product vs engineering divide -- you're implementing stuff other people come up with, and when they change their minds you throw it out and do it again a different way.

That said, making games is also a ton of fun and is a great way to be exposed to a lot of interesting engineering challenges. If you like games and you can put up with the issues above, you should definitely consider it.

</comment></comment></comment><comment><USER16> Pretty much 0%. Mostly because of libraries already solving the problem for me.

</comment><comment><USER17> Just about every project I've worked on contained one "chunk" of the project that was an algorithm, while the remaining 95% of the functionality was achieved by implemented by Postgres, Apache, Django, etc. This is why I (and other people that work this way) am able to get things done so fast. As I've gotten better at launching projects, my algorithm writing skills have not gotten much better, but my "open source understanding and researching" skills have multiplied hundred-fold.

</comment><comment><USER18> In 25 years of coding i needed to sort something about three times (not counting sql) in total. So when somebody asks me to write some sorter as like in job interview, i'll probably fail...

<comment><USER9> Well, here's something to think about: is it possible that sorting some data could have improved the performance of your code? Here is an example that my adviser used when he last taught algorithms at the undergrad level:

 _Given three arrays of integers, determine if it is possible to take a sum of three integers, one from each array, that equals a target value (say, 0)._

It is not immediately obvious, but sorting the arrays will allow you to solve the problem much faster (whether or not this is the _fastest_ solution possible is actually an open question). This problem has connections to several computational geometry problems, and I have heard that variations of it come up in certain real-world applications.

Also, there is more to algorithms than just sorting.

</comment></comment><comment><USER19> I work on a large commercial database mainly on optimizing joins and a lot of the work that has to be done is algorithmic.

Library functions normally are to generic and therefor we end up implementing lots of different variations of fast and memory efficient searching, sorting, graph optimizing, etc.

</comment><comment><USER20> My experience over about 20 years of writing software professionally is that I need to tackle one algorithmic problem a year. Probably amounts to a day or two of work, and usually it is a refinement or adaptation of an existing algorithm, rather than totally new code. Call it less than 1% of my time.

</comment><comment><USER21> At my last job, the coolest project i did was to create an image duplicate detection system (so that it would tell, given an image, that the image is new, or it's a version of what was uploaded before, or the image contains fragments of some known image, etc). It used opencv for feature points detection, so i had to maintain an index to search for those points (there were around 7M images, and load was up to 50 rps). I came up with a rather clever algorithm, which as i later discovered (thanks, wikipedia) is a kind of implicid k-d tree.

That task was the most algorithm-intense task I ever had, and that algorithmical part took 20% of the time. 80% was straight coding, glueing things together, testing and putting all that in production system.

</comment><comment><USER22> 100% of my time is spent on cache invalidation and naming things.

<comment><USER5> It's off-topic, but could anyone critique a rule for (not) naming things I came up with recently:

If something (an expression or function) is used more than once, give it a name. If it's used once, name it iff you would otherwise comment to explain it. Otherwise, inline it.

<comment><USER23> Sure, that's basically a rule for when to do the "introduce explaining variable" refactoring [1]. There's a bit of taste as to when to name things in this instance, but certainly if a name can replace a comment then it's better to use a name. You might additionally use a name for something you wouldn't comment, if the logic is straightforward but the code is long enough that it would otherwise be hard to read.

If you use something more than once, then you either have to give it a name or repeat it, and generally people like to avoid repeating things. Some people will accept two copies and get rid of the duplication when they get to three copies, though purists would dislike that.

[1] [http://www.refactoring.com/catalog/introduceExplainingVariab...](http://www.refactoring.com/catalog/introduceExplainingVariable.html)

</comment></comment></comment><comment><USER24> A lot of CS people complain about that. Especially in college and those who are just joining the work-force.

The classical problems were problems back in the day. There is no point in re- implementing them nowadays. But it is very important to know how they are implemented, and the logic behind them.

Why?

Because we are tackling much, much harder problems. Fields like data analysis, NLP, machine learning are yet to reach their potential. If we cannot master the easy classical problems, how are we expected to tackle the bigger fields? By master, I mean understand and not re-implement/re-invent the wheel every single time.

</comment><comment><USER25> Conversations like this make me wonder if there's some disruption waiting to happen. My understanding is that from a theoretical perspective software should be reducible to a combination of algorithms and business rules, but we seem to spend our time on something else. Is there some meta-solution to the rote stuff that we just haven't noticed? Doesn't it seem like the most tedious tasks - Extract-Transform-Load, for example - could get automated away? Is there something preventing that from happening? What is it?

<comment><USER26> You are assuming that you know both the algorithmic and the business rules when you start the project. That's rarely if ever true.

People in charge of setting those business rules also rarely know what they actually want in details. This is what is meant by the joke that writing an accounting software is order of magnitude harder than building a kernel, even if the latter is much more challenging technically speaking.

</comment></comment><comment><USER27> First, in education, sorting algorithms are used as prime examples of well- defined problems with well studied solution.. they are not taught because every student is expected to work on sorting problems for the rest of their life. It is usually the first part of a course on the _design and analysis_ of algorithms.

As for my answer: Most of my time is definitely in the usual grind of software engineering: designing, testing, maintaining, etc... However, at my lab we have been engaged in a number of projects involving network protocol engineering and the development of formal code/model verification tools. As a first step toward the development of systems like this, you must be able to construct a precise formal model, and then show that in such a model, then _so-and-so_ properties hold true (including memory usage, upper/lower bounds, stability [in cases of routing algorithms], intractability of certain values [in the case of secure protocols], etc...). If you are developing software for usual line-of-business type applications, you'll do much less basic algorithm development than if you are, say, developing the routing software for a wireless ad-hoc mesh network.

</comment><comment><USER28> When I was writing games, There was a lot of re-implementing nearest neighbor searches in python. Now that I'm working on a Saas web app, it's 90% UI busy work.

</comment><comment><USER29> A lot. I work with large amounts of data so relatively minor changes in algorithmic complexity can determine if a method is practical or not.

Recently I found a great new dataset but it didn't work well with our database - the naive method of doing lookups was at least 10x too slow to be practical. Studying the underlying algorithms of our database helped me find a solution that took up 10x as much space but was 100x faster.

</comment><comment><USER30> How much of my time is spent on algorithmic problems? Not as nearly as much as I would like!

I once wanted to stay away from web development because I knew a lot of it would be boring stuff like learning to work around browser bugs (which change with every generation of browsers, rendering your previous learning useless), rather than the mathematical/algorithmic mind puzzles which I really enjoy. I gave in because there's so much work available (and I found some clients who pay really well).

I think having experience working on both low-level embedded software and algorithmically intensive software (natural language processing, etc) gives me some advantage as a web developer. When the need to devise an algorithm or do something else hard-core comes up (which it does from time to time), I take it on with pleasure rather than shrinking back in fear. Or if I find a bug in my platform, I can drop down into the C source, figure it out, and submit a fix to the maintainers, rather than waiting for someone else to help.

</comment><comment><USER31> I'm writing my master thesis in computer science, so quite a bit (it's a lisp- like macro system for Java). But then it's not a real job.

<comment><USER32> That seems to be a popular topic. At least, I just came across this yesterday: <http://www.cs.utah.edu/~rafkind/papers/dissertation.pdf>

<comment><USER31> Thanks, this is relevant to my interests :)

</comment></comment></comment><comment><USER33> Directly writing? Almost none. Indirectly? A lot.

Queueing theory turns out to be a big part of the system I'm working on, as do graph traversals. I'm not proving anything new, but I am certainly transforming known solutions to be present implicitly in the way data flows through the device (a network gateway).

I've also done work on numerical solutions, implementing various formulae the researchers have come up with, and dealing with various optimization (in the mathematical sense) problems.

Lately there has been a lot of talk on "new classic" things, applying machine learning algorithms to data sets, and finding relevant classifiers and adjustments to the ML algorithms that provide better results. Hopefully I'll get into implementing and doing real work on that soon.

I do work in academia though, in a department focused on implementing research in ways that industry can actually use it.

</comment><comment><USER34> 0%, these are solved problems that you can easily use a library for.

<comment><USER30> I guess that depends on which "classical" algorithmic problems he is referring to. The field of "algorithms" is like an infinite sea which will never be fully explored; there will always be more algorithmic problems for which no library is available.

For example, last year I worked on a software package which was intended to match clothing automatically to form outfits. The client was a fashion consultant; the software had to duplicate what he did in his day-to-day work, algorithmically. You won't find that one in "The Art of Computer Programming"! (Or in any library, for that matter.)

</comment></comment><comment><USER35> At Mixrank we process a lot of data, so a lot of our time goes into designing efficient data systems. Algorithmic complexity really matters, and small tweaks can result in large improvements.

If you'd like to spend more time designing algorithms, then working on big data systems is a sure bet.

</comment><comment><USER36> Zero. It's all refactoring ugly, old, unmaintainable and unexpandable code to remove all those characteristics. It's an exercise in "engineering" that involves zero algorithmic work apart from the occasional replacing of a linear scan through an array with some more efficient data structure.

</comment><comment><USER37> Only a few times, but when I needed to solve something like that, it was really important. A few examples: processing a giant, highly-interconnected graph that was too big for memory, so figuring out how to split it up into parts small enough to handle; learning Bayesian statistics to make strong inferences from millions of weakly-indicating data points; making inferences about the nature of edges in a graph with tens of millions of nodes.

</comment><comment><USER38> Professionally: 0

My own project: At least once a week.

</comment><comment><USER39> ~50% - I'm in "big data" right now, partly because I get to worry about the algorithmic complexity of things all the time. As the scale of a task goes up and the feasibility of just throwing more machines at it decreases, you can suddenly justify spending a lot of programmer time on interesting things.

</comment><comment><USER40> Your job (much like your life) is what you make of it. If you seek out problems that are unsolved & require solutions that involve thinking up a clean/simple solution, you'll find yourself spending much more time solving algorithmic problems than just writing/copying rote logic.

</comment><comment> alok-g Zero percent. On the other hand, for as much as 50% of the problems I solve, it needs understanding the nature of the problem and its solution _in terms of the classical algorithms_ , which then are by definition already available somewhere at least as pseudo-code.

</comment><comment><USER41> Zero. I have not implemented any "classical algorithmic" code since college. Nobody writes search/sort/comparison code, or even implements data structure code such as balanced trees, skip lists, etc. because your framework or libraries provide those.

</comment><comment><USER42> About 1/4th I'd say, another 1/3rd is thinking about architectural decisions (or restructuring bits that are broken) and the rest is mundane work including meetings. I work on infrastructure stuff

</comment><comment><USER43> At BitGym we solve vision and sensor problems, and it has to be efficient because the result will run on an iPad. So yes, we spend a fair amount of time on it. Maybe 20% of our heads down time.

</comment><comment><USER44> I think it depends. I've been building GUI builders and there is a ton of tree manipulation and traversal. Beyond that though most operations are already available in the standard libs.

</comment><comment><USER45> roughly 50% of my current duties involve parsing and transforming an AST. As well as performing static analysis.

All of it is to support web developers by giving them tools to handle html and css well. But this is not the usual case for my career much of it has been exactly what you describe instead.

I'm just glad that I knew enough of this stuff to be able to tackle these problems when there was a need to.

</comment><comment><USER46> I would say less than 1 % . Most of the time its picking the best from an already existing solution to suit the need.

</comment><comment><USER47> Not much of my time but a lot of my value.

</comment><comment><USER48> Take a job involving map-reduce. Search/sort/comparison in about all you'll do.

</comment><comment><USER49> First, let's look at what is an 'algorithm': The examples that got famous were the sorting and searching algorithms in Knuth's TACP. So, people quickly coded up bubble sort which has running time proportional to n^2 for sorting n items, and in practice n^2 grew too darned fast. So, people were eager for Shell sort, quick sort, heap sort, merge sort, etc. which all had running times proportional or nearly so to (n)log(n) in average case or even worst case. Then there was radix sort which could beat (n)log(n). So, people got excited about 'algorithms'.

So, what did we get? We wanted to sort (or work with trees, etc.). We knew how to do that manually. We could write the code, but our first cut code tended to be too slow. Heap sort, etc. are really clever. Stare at the code (typically are doing an informal proof of correctness by mathematical induction based on the definition of the natural numbers) and can confirm that it sorts, and with analysis as in Knuth can confirm the running time, e.g., worst case (n)log(n). So, net, there's not much question about what it does, that it actually sorts, and how fast it is. Great -- for sorting and searching.

Do I ever do such things? Actually, early in my project I wrote a bunch of heap sort routines, polymorphic, etc. I wanted to write some 'stable' versions but didn't get around to it this time. Also can write versions that sort by 'surrogate', that is, move only pointers and not the data. For some of the details of how Microsoft's Visual Basic .NET works with strings and instances of objects, there is some question about the value of sorting by surrogate! Then with the results, can have an O(n) routine that will take the 'permutation' of the pointers and move the data. Etc. I didn't write everything I could think of in sorting, but some of what I wrote is good and used in my code. And I got some polymorphic code that is only 2-3 times slower than a non-polymorphic version. So, yes, I did some work in algorithms, even some quite old ones.

Next I needed a 'priority queue' so borrowed the heap data structure from heap sort and, presto, got a nice priority queue routine. It's in my production code.

So, yes, I worked in algorithms. And for my project, that work is crucial in the sense that without it my software wouldn't work or, if I had used code I knew about from others, not work as well.

But now there is another issue in computing and software: There are old examples; since they are more explicit, let's start with those. For a positive integer n, suppose A is an n x n matrix of real numbers and we seek x, n x 1, to solve Ax = b where b is also real numbers and n x 1.

Now we need an 'algorithm'? Not really! An 'algorithm' as in sorting is not enough! Instead we need some mathematics prior to any sense of an 'algorithm'! Usually the mathematics we use is Gauss elimination, and a good chunk of a course in linear algebra makes clear just what that is doing and why it works.

Here's much of the difference: Looking at a description of code for, say, heap sort, it's easy to see that it sorts. Looking at a description of code for Gauss elimination, it's totally obscure why it solves Ax = b, that Ax = b has none, one, or infinitely many solutions, and if infinitely many how to characterize them. Instead of just the code, just the 'algorithm' for Gauss elimination, we need the math from, say, linear algebra. Similarly for a numerical solution to an initial value problem for an ordinary differential equation.

Then more generally, for some software, it may be that some math, new or old, is crucial for knowing what the software will do, much as the linear algebra was crucial for knowing what some Gauss elimination code will do. E.g., in Gauss elimination, turns out have some choice about the 'pivot' elements. Well, typically want to select the pivot that is largest in absolute value. Why? Some math! Next, do a lot of 'inner product' arithmetic. Then want to accumulate inner products in double precision. Next, when get a solution, want to do a little more work with double precision and do some 'iterative improvement'. And would like to know the 'condition number' to get some error bounds -- all in the code and all needing some math to know why it works.

So, net, for many good solutions to real problems, an 'algorithm' alone has not enough to recommend it and need some prior logical support from, say, some applied math. So, in code, algorithms are not everything that is important; in such cases, an algorithm alone does not have enough to recommend it.

In my project, some original math was crucial, and got converted into code. But to have any confidence in what the code will do, can't just stare at the code and, instead, need the math. That is, just an algorithm was not enough.

So, in my project there has been work in both algorithms and applied math. That work is crucial in that without it the project would flop or nearly so.

But, right, also is working through a lot of routine logic. E.g., the code for yesterday was to be clear at the start of the code for a Web page just where the user came from. There were three cases, the first two legal and the third a symptom of an attempt at hacking. So, I needed to work out the logic. For that I needed some answers to some of how Microsoft's ASP.NET worked. E.g., if transfer to page B from page A via in page A code server.transfer("B"), then in the code of page B, what is the value of Request.UrlReferrer? Use the TIFO method -- try it and find out. So, right, such 'logic' is routine but takes time.

Then where most of the time went! Right, to none of the above! Most of the time, likely over 80%, went, may I have the envelope, please? Yes, here it is (drum roll): The nominees for where most of the time went are (1) working through poorly written technical documentation, (2) software installation, (3) software configuration, (4) data backup and recovery, especially for the boot partition, (5) system security, e.g., fighting viruses, and (6) fighting bugs. And the winner is, (1) working through poorly written technical documentation! Heck, reading Knuth, Ullman, Sedgewick, etc. was fast, fun, and easy, but reading the documentation on the 'platform' was painful and wasteful.

So, the crucial stuff unique to the project took the least time. The rest of the work unique to the project took much more time but still was fast, fun, and easy. Work on (1)-(6) essentially independent of the project took maybe 80% of the time.

In particular, the most valuable work, that gives the project some potential, took the least time; routine work still unique to the project took much more time but still was fast; nearly all the time went for work not unique to the project.

Why? There is good/bad news: Some of the good news is that there is now so much infrastructure software that write less of own code and use code written by others. The bad news is that when using code written by others, also need to use their documentation, and for the industry so far writing good documentation is, in practice, essentially too difficult. E.g., I had to use just the TIFO method to determine what would be the value of Request.UrlReferrer. As far as I could tell from my 4000+ Web pages of documentation, the only way to know was the TIFO method. Then, of course, in my code I did put some documentation. I wish I didn't have to write my own documentation of the tools I am using from others.

Net, what's crucial and unique to a project may not take the most time. That is, the work that takes the most time may not be the most important work. And, then, can't measure importance to a project of some work by the time the work took.

<comment><USER50> Interesting, but I noticed you write about your project, singular. Have you had any other projects?

<comment><USER49> Sure. At one time I was working to support my wife and myself through our Ph.D. programs. The work was in applied math and computing on mostly US Navy problems. At one time, there was an urgent request: Evaluate the survivability of the US SSBN fleet under a controversial scenario of global nuclear war limited to sea.

Point: A claim is that if find an SSBN, then can kill it with a nuke -- likely true. Another claim is that can't find an SSBN -- not really true but close to true since a promising effort to find the SSBNs would be 'provocative' and raise alarms. Also, if find and sink one, then that's the start of nuclear war, and the other SSBNs will be free to fire. So, what is tough is to find all the SSBNs at one time and sink them all at once. But if in the special scenario, then could find and sink the SSBNs one at a time. Then how long would they last?

The Navy wanted their answer quickly, in two weeks. That was about right, since my wife already had us scheduled for a vacation cabin in Shenandoah starting the day after the due date!

So, I derived some math, wrote some software, and both the Navy and my wife got what they wanted on time!

My understanding is that my work was later sold to another group interested in US national security. I could tell you what that group was, but then I'd have to ...!

There have been other projects!

</comment></comment></comment><comment><USER51> Zero.</comment>


Show HN: The Discographer – I built Songkick/Bandsintown for new album releases - <AUTHOR> http://www.thediscographer.com/

<comment><USER2> Love the idea. My first thought is that the icon looks like an alarm clock.

Currently, the way I track album releases is by following every conceivable artist on Spotify, which will then notify me. What I like about this is that I can then stream the whole work, for free. However, Spotify doesn't always have works when they're immediately released. Additionally, I have to follow the artist in the first place--there's no library scanning. There's also the chance that I'll have work by an artist in iTunes but won't have ever followed/listened to them on Spotify.

In other words, could you also make a Spotify plugin which scans listening history, and combines this with your iTunes library?

<comment><AUTHOR> Spotify/rdio support is one of the most requested feature so far, so it's definitely on my things-to-consider list.

<comment><USER3> May I kindly ask for Google Music support too, I use it and would love such a service. I was with rdio before, and getting notifications of new releases for artists in my library is what I miss the most.

</comment></comment></comment><comment><USER4> This is something I wanted to build for myself for a while, at least as an RSS feed or email notification or the like. That way I could track bigger releases the same way as my Bandcamp music feed emails. I'll give this a shot.

EDIT: I like the app, but my immediate thought is to offer a feature request to explicitly _not_ track a certain artist. There's at least one artist I spotted in my recent releases that I don't want to pay attention to, and receiving notifications for them would seem like a waste. Otherwise, it's great! Much better than last.fm's release tracking, which had a tendency to get confused by similarly-spelled artists more often than showing actual releases.

<comment><AUTHOR> Great feedback thanks, being able to untrack specific artists is coming in the next release!

</comment></comment><comment><USER5> Partly, because I'm disappointed I won't be able to try it, and partly out of curiosity, what was the reason for not just making this an HTML5 web app? Was there some functionality you couldn't get from Web APIs?

<comment><AUTHOR> Two reasons mainly: 1. being able to scan the user's iTunes library, and 2. push notifications (could have done email, but I find it higher friction).

<comment><USER6> The idea is sound, but what about those of us who listen to Spotify/Rdio/et al. rather than having an iTunes collection?

There's also Last.fm. Everything I listen to (or at least as much as reasonably possible) get's scrobbled. I believe it's this feed that my songkick account uses. It'd be nice to get integration with that!

<comment><AUTHOR> Support for Spotify/rdio/last.fm is definitely something I'm thinking about for the future, yes!

<comment> vhost- Spotify notifies me when artists I follow add new albums. Have you tried following an artist that you want updates from?

</comment></comment></comment></comment></comment><comment><USER7> Where do you get the release data from? A couple years ago I started to work on an app to do this exact thing but I couldn't figure out where to get information about releases in an automated way.

</comment><comment><USER8> I would like to be able to unfollow artists, or selectively follow from my library.

</comment><comment><USER9> Great concept, I actually wanted to do something like this for a while... Any plan for a EU release?

<comment><AUTHOR> Most probably in the next few months yes!

<comment><USER10> What's the reasoning behind only putting it on the US store? Luckily I have a US iTunes account to try it :)

</comment></comment></comment><comment><USER11> what's your release date source?

<comment><AUTHOR> for now, it's iTunes focused, so I'm using that data, looking into expanding to other services/sources</comment></comment>


Planned Parenthood's sexual health chatbot will answer teens' burning questions - <AUTHOR> https://mashable.com/article/planned-parenthood-sexual-health-chatbot/ <comment><AUTHOR> are we going to use a chat bot for everything ? or some human touch with kids health on the line will be nice ??????</comment>


When you need a CT scan ask for this - <AUTHOR> http://freepress.intel.com/community/news/blog/2012/03/12/computing-power-speeds-safer-ct-scans

<comment><USER2> _The answer to the question Baker posed lay in a set of mathematical rules called an algorithm._

I never cease to be amazed just how low the math level is in our culture. And how acceptable it is to be publicly that math ignorant.

 _It was difficult, for example, to change the algorithm, which worked best in a single-threaded environment, into one that could take advantage of multi- core processors._

Aha, so the parallelized it!

 _"We have been able to reduce X-ray doses to previously unthinkable levels," said Professor Johan de Mey, head of the radiology department of University Hospital in Brussels, Belgium. That is opening up the benefits of CT scans to a wider variety of patients._

Great!

The terrible way in which this article explains what was done, is doubly ironic considering how it is an article about math and programming literally _saving lives_.

<comment><USER3> Using the same phrases you consider "terrible", the writer was able to make this piece accessible to a wider audience. I would suggest that articles like this, which describe technological breakthroughs in language the lay public would understand, are good for the science and engineering communities.

By the way, the article isn't about "math and programming saving lives", it is about an advancement in technology that reduces the amount of radiation patients are exposed to (in CT scans) so that there's a lower risk of radiation poisoning.

</comment></comment><comment><USER4> My advisors work is in this - part of it is making better use of the statistical properties of the noise in the reconstruction of the image.

<comment><USER5> Stochastic Resonance

<http://en.wikipedia.org/wiki/Stochastic_resonance>

<comment><USER4> That was not what we were doing - It was more like weighting measurements proportional to their statistical variance, so that you penalize noisier measurements. In this process, you typically apply some sort of regularization to the objective function so that it converges to a smooth solution. There are some interactions between the weights chosen and standard regularization that creates non-uniform spatial resolution propeties, so much of the work was spent designing roughness penalties that played nice with the weightings.

<comment><USER5> Hmm. Pretty much the inverse of SM.</comment></comment></comment></comment>


Quire - A visual collaborative platform - <AUTHOR> https://quire.io/ <comment><USER2> The ability to have arbitrarily deep nesting along with the efficient and information-dense visual organization make this look pretty interesting.

Trello is just too simple to capture the complexity in any project of reasonable size, and JIRA is a maze where you really have to know what you're looking for.

We have been using Plectica[1] to plan together and keep track of progress. It's not perfect, and sometimes it's actually too flexible, but in the end has really worked for us.

[1]: [https://plectica.com/welcome/technologists](https://plectica.com/welcome/technologists)

</comment><comment><USER3> I used Trello to manage my projects, but a few months ago, I discovered Quire. It's pretty easy to use and it has a list view. Which is great for my teams to dump ideas, and i can moved them to board afterward for tracking the project's progress.

</comment><comment><USER4> One of the best task management app I've ever used. Not a fan of their theme choice though but overall, it helps me with my work. Simple enough to not create any confusions.</comment>


Launch: Basecamp Gets Personal - <AUTHOR> https://m.signalvnoise.com/launch-basecamp-gets-personal/ <comment><USER2> They must've done some analysis on the percentage of free users that'll convert to paid. That analysis is valuable. Anyone here care to guesstimate an analysis, replete with the cost free users add?</comment>


A 1,000-year-old road lost to time - <AUTHOR> http://www.bbc.com/travel/story/20181203-a-1000-year-old-road-lost-to-time <comment><USER2> This writing is so pretentious.

> As I trudged through the streets of Lucca on my first day, the sun shone hot > on my skin and the wind brushed my face. Without the protection of a car or > bus, I smelled every rubbish bin and felt the whoosh of passing cyclists. I > heard the gentle thud of my feet and noticed how the texture of the ground – > whether earth, grass, cobblestone or cement – changed my stride.

That is walking. He is literally just describing walking.

> On my third night, I was eating dinner with other pilgrims in a hostel > outside the vertiginous hill town of Gambassi Terme (I chose the small hotel > because when I arrived, my feet riddled with crippling blisters, I knew that > if I stayed there I would not have to climb the steep slope before I could > rest).

Here is the Italian version of the 7-11 in this "vertiginous hill town." [https://firstname.lastname@example.com,10.9519354,3a,71.1y,...](https://firstname.lastname@example.com,10.9519354,3a,71.1y,124.87h,92.46t/data=!3m6!1e1!3m4!1sxj54syDPdbLo3Xs6WARphA!2e0!7i13312!8i6656). He's not in the wilderness here. He's walking through semi-rural/suburban Italy between populated towns.

> As we dug into plates of pasta al pomodoro (pasta with tomato sauce)

Oh FFS.

<comment><USER3> > This writing is so pretentious.

What is it pretending?

<comment><USER4> pretentious[1] != pretending[2]

[1] [https://www.dictionary.com/browse/pretentious](https://www.dictionary.com/browse/pretentious)

[2] [https://www.dictionary.com/browse/pretending](https://www.dictionary.com/browse/pretending)

<comment><USER3> It's just a form of it. "Characterized by assumption [...]" that's pretending. "making an exaggerated outward show"; pretending. "full of pretense"; hey, what does the definition for that return? - [https://www.dictionary.com/browse/pretense](https://www.dictionary.com/browse/pretense)

If it can be characterized as such, then one ought to be able to elucidate why.

<comment><USER4> > "Characterized by assumption [...]" that's pretending

Maybe by a meaning of "assumption" or "pretending" that I'm not too familiar with. Normally, I would think that someone "assuming" thinks something is true and acts like so, and someone "pretending" thinks it's not true but acts like it is.

> "making an exaggerated outward show"; pretending

My English must be lacking... Exaggerated just means making something more apparent than it needs to be. That's also different to pretending.

> "full of pretense"; hey, what does the definition for that return?

Ok, you convince me there's related meanings with that.

</comment></comment></comment></comment></comment><comment><USER5> Here is a link to a map of the whole road as the article and official website don't seem to have one[1]. There are nice interactive maps for each country at the official site though[2].

[1][https://en.wikipedia.org/wiki/Via_Francigena#/media/File:VF_...](https://en.wikipedia.org/wiki/Via_Francigena#/media/File:VF_Ruta_completa_con_pricipales_poblaciones.svg)

[2][https://www.viefrancigene.org/en/resource/tour/la-via- franci...](https://www.viefrancigene.org/en/resource/tour/la-via-francigena- in-francia/)

</comment><comment><USER6> > Lucca had faded into semi-rural, semi-industrial outskirts that will likely > never be on any tour itinerary. It was not particularly impressive or photo- > worthy – it was a moment that would be hard to justify to someone else, to > explain why, out of all the things I could have done, I had chosen to be > there.

I beg to differ. My wife and I made a day trip there when we were staying in Florence. Lucca is a beautiful little town with some cool medieval architecture and history. It's a rare example of a town that still has its original walls intact. There were tourists but it wasn't crowded, and for under 10 euros you can climb a couple towers from the 14th century for a spectacular view of the town and rural countryside. Highly recommended if you're in the area.

<comment><USER7> I believe he is referring to the area he traveled through after he left Lucca, not Lucca itself, which is on many tour itineraries.

<comment><USER8> > he

her

<comment><USER9> > her

she

<comment><USER8> Werds. Better grammar than gender.

</comment></comment></comment></comment></comment><comment><USER10> Official site of the Via Francigena: [https://www.viefrancigene.org/en/](https://www.viefrancigene.org/en/)

Side note:

> _The saying 'all roads lead to Rome' has become a quaint and somewhat > clichéd turn-of-phrase these days. But when the Roman Empire ruled over > places such as England, present-day Spain, North Africa, and even modern-day > Israel and Turkey, it was true._

[https://sashat.me/2017/06/03/roman- roads/](https://sashat.me/2017/06/03/roman-roads/)

discussed here:

[https://news.ycombinator.com/item?id=14511627](https://news.ycombinator.com/item?id=14511627)

<comment><USER11> It seems we killed the site again, so here are some mirrors:

[https://sashat.me/wp- content/uploads/2017/06/Rome_III-01-1.p...](https://sashat.me/wp- content/uploads/2017/06/Rome_III-01-1.png)

[https://i.imgur.com/cun1MCJ.png](https://i.imgur.com/cun1MCJ.png)

</comment></comment><comment><USER12> Not far from where I live is a completely overgrown path/road known as the "Old North Road" \- looking at the landscape it takes a route that was probably the easiest way through the hills before more modern roads of recent centuries.

Maybe Agricola's armies marched that way from their camp on a nearby hill to help rescue the 9th Legion... who knows?

</comment><comment><USER13> For those who might prefer less story, and more 'meat and bones'.

A little known fact: Many of Australia's roads were originally mapped out by Aboriginals, who used songlines (/the stars) to navigate. This means many of Australia's main roads are 10's of thousands years old. (Admittedly, the roads didn't actually exist, even though the pathway did)

Research by Ray Norris (of CSIRO) is very interesting: [https://arxiv.org/abs/1607.02215](https://arxiv.org/abs/1607.02215)

His website on Aboriginal Astronomy: emudreaming.com

<comment><USER14> Thanks for this. I've recently been reading Bruce Pascoe's 'Dark Emu', which is on how Aboriginal Australians had agriculture too, even recorded in the journals of the original Euro explorers but now lost to public consciousness.

</comment></comment><comment><USER15> Fascinating.

I wish the article had actual pictures along the route, instead of bland standin pictures of anywhere (feet, streams, roads that aren't the Francigena etc)

</comment><comment><USER16> The BBC is awesome in many ways but this 'travel' show is such bollocks it's amazing. They tourist like a MF instead of doing the Bourdain and actually speaking to people who live there. EDIT: actually rather than interview I would like to stress

</comment><comment><USER17> I have traveled to Europe a handful of times and never heard of this until now! Time to add this to the list of things to check out next time. Very cool. :)

</comment><comment><USER18> > Walking over long distances, sometimes as far as 24km a day, was new to me.

24 km ain't a long distance to walk in a day</comment>


JPL Open-Source Rover Project Based on the Rovers on Mars - <AUTHOR> https://github.com/nasa-jpl/open-source-rover <comment><USER2> My coworker Liv designed that! I don't know the original purpose but it's a fun project.

I've also got my own open source Rover design if you like this kind of thing:

[https://youtu.be/DXPmqCd0r04](https://youtu.be/DXPmqCd0r04)

<comment><USER3> It's very confusing having the frame rate of the camera and the rotation of the wheels in sync so that the wheels appear completely stationary while the rover drives forward.

<comment><USER2> Right! It was weird to watch when editing.

</comment></comment></comment><comment><USER4> If you are interested in other open-source space technologies feel free to check out Libre Space Foundation's ([https://libre.space](https://libre.space)) code repositories ([https://gitlab.com/librespacefoundation/](https://gitlab.com/librespacefoundation/))

We aren't NASA or JPL but a non-profit based in Greece. We already run SatNOGS (a network of satellite ground-stations), UPSat (the 1st satellite with components mainly under the CERN Open Hardware License) and on the side we work on amateur high power rocketry telemetry and PocketQubes.

<comment><USER5> Awesome! Can I contribute? In other words, how can I, as an experienced backend developer (Java, Python, Scala, Ocaml, some C/C++) but a complete newbie in space, hardware and/or realtime systems, contribute?

<comment><USER4> Hey tomp. Of course you can. Most of our source code for SatNOGS is actually Python (and Django) and there is a lot you can do with C++ too. Check out our forums and don't hesitate to introduce yourself to the rest of the community [https://community.libre.space/t/new-users- welcome/29/278](https://community.libre.space/t/new-users-welcome/29/278) feel free to also hang-around over our IRC channels at the freenode server #satnogs Matrix bridged here [https://riot.im/app/#/room/#satnogs:matrix.org](https://riot.im/app/#/room/#satnogs:matrix.org) or #librespace Matrix bridge [https://riot.im/app/#/room/#librespace:matrix.org](https://riot.im/app/#/room/#librespace:matrix.org)

</comment></comment></comment><comment><USER5> if anyone has worked at JPL or similar, what's the best resource for learning how to make an _actual_ Mars/Moon rover?

I'm very curious about things like, what kind of wheels you need for regolith, how to manage extreme temperatures (electronics, motors, batteries, cameras), how communication works (could you have a remotely-operated lunar rover?), dealing with radiation (SpaceX works with consumer electronics AFAIK, but they mostly fly within Earth's magnetosphere, which ... changes things).

Any replies or emails would be extremely appreciated.

<comment><USER6> Haven't worked in this field, but here are some episodes discussing the topic:

[https://www.youtube.com/watch?v=nJLOZDPTp3I](https://www.youtube.com/watch?v=nJLOZDPTp3I)

[https://www.youtube.com/watch?v=ADJwChUtdDQ](https://www.youtube.com/watch?v=ADJwChUtdDQ)

[https://theamphour.com/334-an-interview-with-gerry- roston/](https://theamphour.com/334-an-interview-with-gerry-roston/)

[https://theamphour.com/220-an-interview-with-shaun-meehan- do...](https://theamphour.com/220-an-interview-with-shaun-meehan-doctiloquent- dove-deployer/)

[https://theamphour.com/401-an-interview-with-brent-and- bryce...](https://theamphour.com/401-an-interview-with-brent-and-bryce-salmi/)

[https://www.youtube.com/watch?v=FzRP1qdwPKw](https://www.youtube.com/watch?v=FzRP1qdwPKw)

I think what you are talking about is mostly just normal engineering in the sense that the process is similar. What really makes space unique isn't space as such but that there is no real-world (real-space?) testing, no inspections after the fact or do-overs.

<comment><USER7> >I think what you are talking about is mostly just normal engineering in the sense that the process is similar. What really makes space unique isn't space as such but that there is no real-world (real-space?) testing, no inspections after the fact or do-overs.

This is sort of true?

Spacecraft of all types are subject to a battery of physical tests before launch. This includes vacuum testing, thermal cycling and vibration (where many parts fail). There are normally duplicate components produced for different purposes. Some are just for testing the fit, others are to check that thermal expansion isn't a problem and so on. The actual flight hardware itself is usually tested as well, since it's designed to be repeatedly stressed within certain physical limits.

NASA has duplicate rovers in the lab that commands are tested on. Before a mission plan is uploaded, it's tested on the ground system to make sure that, for example, an inspection arm doesn't collide with the body of the rover. Modern rovers like curiosity are reasonably good at self-inspection, since it can take a 360 selfie. The rover is covered with calibration points, for example to measure how much dust is covering things and how far the wheels have worn down.

A lot of "testing" is gained through experience. Satellite manufacturers know what designs work based on prior designs. NASA learns a lot every time they put a new rover on Mars. Rover missions (like Curiosity, Spirit/Opportunity, etc) tend to outperform their original specs because the engineering is so conservative.

In very rare cases astronauts have done in-space inspections - for example the repairs to Hubble. The ISS is also used as a space-based testing laboratory for hundreds of different projects.

(I've worked in a space science laboratory where satellites were routinely tested, and indirectly on rover data).

<comment><USER6> Sure. I would put that sort of more in the high level engineering than "real- world testing" though. I mean if you are Waymo you can drive around for a year and see what happens. If you are Audi you can still repair some fault after delivery and redesign the next model year. It is when you can't do that you need high levels of quality assurance, integration, fault tolerance etc. If anyone could teleport small objects to Mars it would soon be littered with smart phones in aluminium foil regardless of how harsh the environment is ;)

</comment></comment></comment></comment><comment><USER8> This seems really awesome, but my God is it expensive. Maybe I am not reading correctly but it seems like just the nuts and bolts from McMaster cost $230 and that's still excluding the electronics and additional hardware! I guess I'll have to save up a bit before I can play with my own minituare rover.

edit: the motors also are going to cost around $500

<comment><USER9> Regarding the motors, they wouldn't be suitable for this particular project but you can buy broken hoverboards _really_ cheaply. Where I live you can get stripped ones (no battery, no logic board, but you can get hoverboard logic boards on AliExpress for under 20€) for 10€ and returned ones (99% of the time it's a dead battery) for 35€. The motors are fantastic and powerful (~350W each), and you can build lots of stuff with them. There's a great talk on how to use these things to build all kinds of moving objects (armchairs, bobbycars, motorised cargo carriers: [https://twitter.com/JanHenrikH/status/1023218344282021888](https://twitter.com/JanHenrikH/status/1023218344282021888), ...) at [https://www.youtube.com/watch?v=qnQSL9DBPaE](https://www.youtube.com/watch?v=qnQSL9DBPaE). You can motorise anything this way for around 100€. The only thing that's hard to come by cheaply is the battery.

<comment><USER10> > The only thing that's hard to come by cheaply is the battery.

Dead laptop and power tool battery packs usually have at least some useable 18650 cells in them. A friend has built more than one battery pack for electric scooter out of harvested cells.

You do have to know what you're doing when working with unprotected cells though.

<comment><USER9> Yes it's easy enough to get unprotected cells, but the hoverboard batteries usually have short circuit and undervoltage protection, both of which are very nice to have as a beginner. If you know what you are doing you can also get RC hobby batteries for not too much money.

</comment></comment></comment></comment><comment><USER11> This is great. But it's a little discouraging to see the project only use the US Customary units considering this is a science project. I was under the impression that NASA used SI.

<comment><USER12> I think it's aimed at students, schools etc. Its purpose should be to attract people to science so it has to be less intimidating, hence the choices of the Raspberry PI or the phone app or the Xbox gamepad to control it.

<comment><USER13> It's a weird world where kids are more scared of the metric system than they are of a credit card-sized Linux computer with its guts showing.

<comment><USER12> It probably is just a matter of being used to it. In the EU we are used to the metric system, but are ok with screen or hard disk sizes expressed in inches to the point if some brand advertising offered a 1.5 meter wide screen TV most of us would look for a calculator to make the conversion, including myself. If we changed to the metric system also for TV sizes, most people would find it uncomfortable at first, but probably would adapt in months.

</comment></comment></comment></comment><comment><USER14> The fact that this level of work is given away for free with the intent of making the world a better place makes my heart dance.

I am so optimistic for the future of humanity.

</comment><comment><USER15> It's interesting that the rover code itself is written in python. That makes it more approachable..

<comment><USER16> You can be 100% sure that no python code is on Mars. Not even Common Lisp which is much more secure and performant is on Mars.

The real rover code is all hardcore C. Erann Gat gave a presentation on the Lisp Conference about the JPL rover code and simulation.

<comment><USER17> I'm a bit disappointed they're not using Forth.

</comment></comment></comment><comment><USER18> This is beautiful, and a wonderful project to develop along with some friends. JPL nailed it.

</comment><comment><USER19> Man I hope someone like Massdrop can find a way to source these parts a little cheaper. It seems to me that there has to be a way to build this thing for at least half that price.

</comment><comment><USER20> How difficult would it be to add a self-leveling platform to a vehicle like this ?

I have seen open source self-leveling _two_ wheeled platforms (segway without the yoke) but I have not seen a self leveling platform on a 4-6 wheeled vehicle like this one ...

</comment><comment><USER21> A mix of Python and Java regarding the software stack.

</comment><comment><USER16> Python? God riddance! The real rover code was in C, and the simulator in lisp.

</comment><comment><USER22> Looks more like a toy rover project, rather than "an open source rover project". Hoped they will make open source substantial (even if simplified) part of the rover project, so space startups can reuse and improve it.

<comment><USER15> Why couldn't this be re-used to some degree for a rover in non-Earth with tougher, more resilient hardware?

<comment><USER23> The hardware we use for non-earth stuff is effectively arduino-level (ok, maybe not exactly - but on the order of 70-100Mhz) because of the constraints put upon it by the radiation hardened hardware. Writing python is a trade-off between computer resources and developer time; we've got a lot of developer time and not a lot of hardware resource, so we write in compiled languages.

There are also some safety implications centered around determinism etc which mean interpreted languages are a bit more gray area; with a well understood compiler and language you understand what the code is doing better, and you know it will run the same every time. That may be the case with python if you understand the whole shebang, but not that many people do... And if you have that level of knowledge of python, you may as well be writing C or C++.

<comment><USER5> Do you find C/C++ acceptable from a safety/reliability perspective, or do you write in safer languages (Fortran, Ada)?

AFAIK critical software coding standards forbid dynamic memory allocation, so things like stack overflows, null dereference or out-of-memory errors can't happen, but array-out-of-bounds exceptions can still get you. On the other hand, static verification tools catch most of that.

<comment><USER23> C/C++ is fine as long as you adhere to coding standards like MISRA and you _thoroughly test it_. Our code for example, is unit tested, then we have java tests running on a server-based simulator, then other independently written java tests running on test benches that more closely represent the spacecraft itself, then a replica of the spacecraft.

The industry used to use Fortran / Ada quite extensively, and depending on company/organisation and team, might still do so. Things like the F-35 are written in C++, which is where I can see these things heading.</comment></comment></comment></comment></comment>


Why Wolfram Tech Isn't Open Source - <AUTHOR> https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-open-source-a-dozen-reasons/ <comment><USER2> Feels like trolling to me. I suspect this was posted a day late.

As he mentions, there are lots of projects that counter his claims. Linux kernel. Every Open Source high-level language ever. Red hat.

Wolfram isn't open source because they don't want it to be open source. And you know what? They're allowed to do that! They don't need to justify that decision, it's their decision to make.

Without resorting to "they love money too much", I'll give them another, slightly more honest reason: It's certainly not impossible, but open sourcing an established project is hard. Managing a community of contributors AND a community of users is hard. Keeping a steady stream of revenue while you make the transition is haaaaarrrd. Maintaining all the values that he's mentioned while still keeping all your contributors and community happy: HARD.

And if the cost:value ratio is wrong, you don't do it. And if you have lots of other hard problems to solve, adding one more hard problem with fractional value is silly.

Yes, we would love it to be OSS. Maybe this was posted as a way of never having to answer the question again. Just sad that it isn't a little more honest an answer.

<comment><USER3> They are allowed to make it closed-source, and there's no shame in making well-maintained commercial software. That's how business works.

What about when business fails though? Open-source projects never die; see the forks and reused knowledge in projects such as ffmpeg, libav, iPodLinux, Rockbox. For archiving purposes, open-source is tremendous.

I wish another kind of license were available, where software remains commercial until the company closes down, but then becomes OSS rather than abandonware.

<comment><AUTHOR> Your last sentence reminded me of an interview with the Ton Rosendaal, the creator of Blender. He talks extensively about how Blender became OSS and not abandonware. I highly encourage to watch the whole interview (you need at least 5 mins to get used to his accent), but the important part starts around minute 14

[https://youtu.be/qJEWOTZnFeg](https://youtu.be/qJEWOTZnFeg)

</comment></comment></comment><comment><USER4> _The very concept of unified computation has been largely led by Wolfram._

For "unified", try sandboxed, walled harden or evolutionary dead end. Just look at this literal Hello World example:

GeoGraphics[Text[Style["Hello!", 150]], GeoRange -> World]

In any other language, if I wanted to write some text on a graphic, I would find a routine to load an image into memory and another to render text into that memory. That's genuinely easy.

But in Wolfram's case, I'm apparently just supposed to know that there's something called "GeoGraphics" that does this kind of thing, that the text goes in a Style[] block (what?), and "GeoRange -> World" is how you tell it to create a worldmap as a background. WTF. And don't tell me "read the docs", because even if I found this, it wouldn't help me much with the rest:

[http://reference.wolfram.com/language/ref/GeoGraphics.htm](http://reference.wolfram.com/language/ref/GeoGraphics.htm)

<comment><USER5> That's not really fair criticism. The Mathematica approach to things is very different to most other platforms. Have a look at some of the maths stuff and symbolic manipulation stuff, that's the core that it was built out from.

<comment><USER4> Sure, I'll buy that Mathematica has its niche in (surprise!) math, but that's very different from the friendly article's claim that Wolfram has a superior approach to, well, everything:

 _Wolfram's vision for computation is much more profound—to unify and automate computation across computational fields, application areas, user types, interfaces and deployments._

They even mention blockchain further down!

</comment></comment></comment><comment><USER6> For the greater good of the world he shouldn't OSS it, but make it free! I'm mostly a R/Python statistics type, but a multi-day course in Mathematica left me in awe. It is great software. The way you could make models dynamic in a user interface kind of way was way ahead of it's time. It's pricing was prohibitive though and we chose not to pursue it.

<comment><USER7> Honestly, it's time we all put our money where our mouths are. A hobbyist/individual license for Mathematica's desktop version is $170/year (or $335 for a forever license to the current version). If we really value our time, convenience and happiness, paying for software which we consider awesome should be a no-brainer. Heck, that's not even 1% of annual rent for someone living in the Bay Area. Of course, this comment is overly broad, and doesn't apply directly to people living in areas with lower cost of living or purchasing power (I grew up in India, so I understand the disparity).

I think Mathematica is quite fantastic (took me some time to understand it's patterns). I don't use it currently because 1) A lot of my current work is numerical and not symbolic -- for which Mathematica isn't great; so I use Python/Julia 2) From a long term perspective, I wouldn't want to invest a large fraction of my personal computing efforts into a proprietary single point of failure.

TBH, with regards to paying for software, my psychological motivation is still a work in progress (so the comments above are directed at myself as much as at others), but I'm trying to build a habit of doing it more -- especially to support free software. I care about computing too much -- I would hate to look back on this twenty years later and rue the loss of computing freedom because we were too stingy to invest a hundred bucks a year in the appropriate places.

<comment><USER6> Aren't there two seperate discussions? 1. Paying something for the contribution of others that offer their product for free (OSS) and 2. Paying for and using proprietary software according to the licenses as offered by the licensee. Mathematica falls under 2 and is obviously totally allowed to pursue any pricing scheme they find acceptable. We just happen to think that leads to a lower than optimal rate of acceptance, which is a societal loss but Wolframs good right. Given Wolframs love of I and there being two I's in prIcIng, he must have written about his motivations somewhere.

</comment></comment></comment><comment><USER8> This is just a PR piece to justify to their customers why they're staying closed source.

The obvious reason is that they'd make a lot less revenue if it were OSS/free. They're right that there's several business models around OSS that have negative side-effects and that just charging for software is the more transparent quid pro quo transaction.

It's ok to sell software, but they should just be transparent about it.

<comment><USER9> I have no skin in the game, but I'll drop in my 2 cents. This may be a PR piece or someone who's had this conversation a countless number of times and wanted to lay it out in a single place.

The points seem mostly fair, though I would argue what he argues is "not possible" with an free or open source model is that it's "more difficult" to coordinate.

As an example, R is my go-to language, but the function arguments and style can be vastly different from function to function or package to package. Here I have found MATLAB to be more consistent in this regard. Python's kind of a mix - the core of Python is like one language and Numpy, Pandas another, but it does reflect the semi-independent management of these projects.

</comment></comment><comment><USER10> Linux is an example of an open source project with centralized design & management and a unified vision.

Mathematica is a brilliant product, and more people would use it if it was open source but it's ok that's it not.

<comment><USER11> An OS kernel is a bounded problem, it abstracts hardware, enforces some security constraints, and allows user programs to run. So it's not really that hard to have a consensus of what Linux should do.

Linux also has the (unique to it) idea that the kernel and the rest of the OS are not the same thing. There is no other integration point than the ABI for userspace.

And OTOH, Linux userspace is ridiculously de-centralized. There's competing distros, init systems, C runtimes, memory allocators, filesystems, etc.

And that's not even taking into account the absolute chaos of the Linux desktop.

</comment></comment><comment><USER12> "fall madly in love!" with GAP (Computer Algebra System)

"9.0 Open source doesn't bring major tech innovation to market". You have to have some balls to say that. Major mathematical libraries used in HPC are OSS.

<comment><USER13> > _" 9.0 Open source doesn't bring major tech innovation to market". You have > to have some balls to say that. Major mathematical libraries used in HPC are > OSS._

And most of them are copies of age old Fortran stuff. Linux itself is a UNIX clone. Gnome/KDE started as Windows clones. Where's the "tech innovation" in that?

<comment><USER14> > Linux itself is a UNIX clone

I think Linux is a prime example of innovation. Yes, it first started as a UNIX clone, then basically took over and killed all competition, starting with x86 and then practically all other platforms, with very specific exceptions.

What used to be a UNIX clone decades ago is now a testbed of all new technologies you can imagine in an operating system. And if some cool tech cannot be implemented in Linux for design reasons (e.g. a microkernel or a unikernel), a separate project is started.

<comment><USER13> > _What used to be a UNIX clone decades ago is now a testbed of all new > technologies you can imagine in an operating system._

But most of them are still copies of FreeBSD, Solaris, and Plan9 stuff. Something particularly innovative in Linux?

<comment><USER14> The article admits:

> Open source often does create ecosystems that encourage many small-scale > innovations, but while bolder innovations do widely exist at the early > experimental stages, they often fail to be refined to the point of > usefulness in large-scale adoption.

You can argue that, for example, KVM is a clone of something else. However, it's because of not stopping at an early stage KVM is now a mature and insanely popular technology.

</comment></comment></comment></comment></comment><comment><USER15> It's me or a lot of those points don't explain anything? I totally get they don't want to open source it as a company but don't try to give false reason for it.

It's not because it's open source that you can't have quality, central design, or good documentation.

<comment><USER16> It's not like anyone can push code to your project once you open source it, you're still the gatekeeper that decides which PR gets accepted and which one does not.

I could understand the argument that sifting through the pull requests would take them too much time that they could spend doing something else, but at least 8/12 of these could be solved by... you know, properly maintaining an open source project.

The article leads me to the conclusion that they have no intention of letting the community drive the company, and I could totally understand that. I don't see how any of the listed reasons could prevent them from making their code source-available without indicating an open source license.

</comment></comment><comment><USER17> Isn't the blog post facilitating the types of open vs. closed source discussions we should stop having as a community? Very little (or nothing) of what McLoone introduces as "reasons" cannot be achieved with an OS approach. In the end, it's a business choice someone has made a long time ago and McLoone is now arguing backwards.

<comment><AUTHOR> I'm not even sure, why this blog post was necessary in the first place. No one argues that a company needs revenue to pay employees and WRI is trying to maximize its profit. That is absolutely fine. However, claiming that OSS cannot achieve great and well-designed software, just to cover the true reasons is simply wrong. Maybe someone should remind WRI that they use Qt, Cairo, Pango, OpenCV and many more OSS libraries under the hood. Qt, in particular, is almost as old as Mathematica itself when the information on Wikipedia is correct.

<comment><USER18> He acknowledges that: "As I said at the start, the open-source model can work very well in smaller, self-contained subsets of computation where small teams can focus on local design issues. Indeed, the Wolfram Technology stack makes use of and contributes to a number of excellent open-source libraries for specialized tasks, such as..." and lists some

</comment></comment></comment><comment><USER19> "It is difficult to get a man to understand something, when his salary depends upon his not understanding it!" \-- Upton Sinclair

Quite literally, in this case.

</comment><comment><USER20> It's because they want to sell if tor money. Not bad or good (although unreasonably expensive), just simple as this.

</comment><comment><USER21> I think control and making money are not bad reasons to keep something closed source. I would like to see products going open source when the product is discontinued or, something I have been thinking about, when you made enough money for past and future to fullfil goals. For Wolfram that seems a point reached years ago, but then again, Wolfram his goals are not clear for the future, so you don't know when that part is satisfied. So I see his point and think it's valid.

Like others have said here, if I were Wolfram, I would make it free for personal use and then price it depending on the company size and revenue (like some game engines do). I mean 345 euros for Home Edition is really too much imho; I understand the power and reach of the product, but wouldn't it be better to get more people dependent on it so they try to push it at their jobs/companies?

</comment><comment><USER22> Interesting that money is not even on this list

<comment><USER18> Its on the list twice - 10 Paid software offers an open quid pro quo 11 It takes steady income to sustain long-term R&D »

<comment><USER22> Those are not "because it allows us to get more profit." They are arguing that non-free Mathematica is completely in the best interest of their users. Of course it's not. At the end of the day, Sagemath is there for everybody to use and a person can't even use Mathematica unless they're connected to a university. They've sabotaged it with legal barriers and DRM.

</comment></comment></comment><comment><USER23> .NET and the languages C# and VisualBasic.NET are open source (definitely not free software) but tightly developed by one central R&D. I find the second argument especially weak. You can be completely open source (and probably even free) without giving up any control over your distribution.

<comment><USER8> They're only central R&D because MS employ most of the paid dev resources working on it and no one would use any forks except from MS (or possibly Google, but they wouldn't ratify MS's platforms by investing resources in maintaining a fork) so there's no risk of fragmentation.

That definitely doesn't apply to all projects, you can look at the successful forks of Oracle's OSS acquired products.

If the OSS'ed code-base is valuable, you've just given a open invitation to all your competitors to create their own competing forks.

<comment><USER24> > or possibly Google, but they wouldn't ratify MS's platforms by investing > resources in maintaining a fork.

While they're not maintaining a fork, it's worth noting that the _head_ of the ECMA C# standards committee is, in fact, a prominent Google employee.

<comment><USER23> "prominent" ... What a modest description :)

</comment></comment></comment></comment><comment><USER25> Atleast they could include the source, with a paid purchase. Many software like vxworks will give you the source, which you can yourself rebuild, if you pay for a license. More commercial software should be bundling source code with paid licenses.

</comment><comment><USER26> While using Mathematica, you don't need to go out to fetch some random 3rd party tools, because most components are already built into it. What makes it better is that combining those components is well streamlined, and there's little need for costly plumbing.

Paywall does suck, but someone gotta curate all those shits. Mathematica is a good platform not only because of some advanced algorithms built into it, but also because of the curation from Wolfram. Curating software component is one of the least fun job in the world, and any entities who do it properly should be paid for the sake longevity of their roles.

</comment><comment><USER27> tl;dr It's not open source because they like money

<comment><USER28> Thank you, you saved some of my time.

</comment></comment><comment><USER29> Because, open source is more about intellectual property rights, than about architecture and control. From the outside looking in, it appears that you have made some decisions made on some bad assumptions of what open source software has to be. I could argue your points one by one, but I think it's more important to address the key factor here: The market is inherently distrustful of closed technology systems.

</comment><comment><USER30> Does anyone remember his book "A New Kind of Science" which was a really discredited, unoriginal, hack-fraudish, non-scientifically vetted essay on cellular automata?

<comment><USER31> Still better than anything you or I will ever write.

<comment><USER32> Are you sure he wrote it? Rumor has it that Wolfram had his grad students/postdocs write Mathematica and is essentially trampling on their backs making money off of it now.</comment></comment></comment>