The Perilous World of Machine Learning for Fun and Profit - <AUTHOR> http://www.john-foreman.com/blog/the-perilous-world-of-machine-learning-for-fun-and-profit-pipeline-jungles-and-hidden-feedback-loops <comment><USER2> The OP wrote a great post some time ago (it was posted on HN and was probably how I discovered him): "The Forgotten Job of a Data Science: Editing" [http://www.john-foreman.com/blog/the-forgotten-job-of-a- data...](http://www.john-foreman.com/blog/the-forgotten-job-of-a-data- scientist-editing...the) theme of that post is similar to the one here:

> _In a business, predictive modeling and accuracy is a means, not an end. > What's better: A simple model that's used, updated, and kept running? Or a > complex model that works when you babysit it but the moment you move on to > another problem no one knows what the hell it's doing? Robert Holte argued > simplicity versus accuracy in his rather interesting and thoroughly-titled > paper "Very Simple Classification Rules Perform Well on Most Commonly Used > Datasets." In that paper, the "air quotes" AI model he used was a single > decision stump – one rule. In code, it'd basically be an IF statement. He > simply looked through a training dataset and found the single best feature > for splitting the data in reference to the response variable and used that > to form his rule. The scandalous part of Holte's approach: his results were > pretty good!_

The OP has a book (the link is on his site) that I've bought...it purports to teach non-technical people how to do machine learning via _Excel_...I haven't gone through all of the exercises, mostly because I shudder at having to open Excel to do any kind of data work, but the mental framework that he builds, demonstrating via examples in Excel, is pretty remarkable...I think it's admirable that a data scientist is doing such significant work in demystifying (and maybe even devaluing) his own profession.

<comment><USER3> I'll just step in and say that, as a non-coder who nevertheless has to understand how code works, Foreman's book "Data Smart" is an invaluable primer on the theory and practice of machine learning.

Certainly I'm no machine learning pro, but as a discipline it's a whole lot less mysterious to me now than it was before I read that book.

</comment></comment><comment><USER4> I feel like these are all reasonably obvious gigantic landmarks taught to anyone with a stats or pattern recognition background. It's sort of just the lay of the land with statistics, modeling, optimization.

That doesn't mean at all that more people shouldn't be aware of them and think critically about them. It also doesn't mean at all that the implications of these features of ML are understood by the entirety of the organization which is affected by them.

But I sort of feel like it's the (perhaps unstated) primary job of a data scientist/modeler/mathematician/statistician/what-have-you employed at any company investing in this kind of technology. It feels about akin to a company building their software without using version control. Sure some people do that, but... c'mon!

If you're failing to do it then you're probably not listening to your experts enough. If they're not screaming it's because they're wrapping blankets around their head trying to block out the noise.

<comment><USER4> After having skimmed the paper, I think it has a subtly different focus which is vastly more important to consider. In particular:

> At a system-level, a machine learning model may subtly erode abstraction > boundaries. It may be tempting to re-use input signals in ways that create > unintended tight coupling of otherwise disjoint systems. Machine learning > packages may often be treated as black boxes, resulting in large masses of > "glue code" or calibration layers that can lock in assumptions. Changes in > the external world may make models or input signals change behavior in > unintended ways, ratcheting up maintenance cost and the burden of any debt. > Even monitoring that the system as a whole is operating as intended may be > difficult without careful design.

In other words, due to either unavoidable or marketing-based complexities, ML projects have a tendency to blow up abstraction boundaries and induce loops leading to unexpected tight coupling. In other words, you can interpret this less as a feature of ML as a task and more as a feature of how ML algorithms tend to operate, especially when operated in "black box" mode.

ML may, when practiced in some ways, be antithetical to good software design.

Again, quoting the paper

> Indeed, arguably the most important reason for using a machine learning > system is precisely that _the desired behavior cannot be effectively > implemented in software logic without dependency on external data_. > (Emphasis mine)

This rings _very_ true in my experience building ML systems. It's an unexpected challenge in software terms. You're often used to many kinds of behavior as being undefined until runtime (e.g. needing user input before they can be understood), but typically the _function_ of a program is something you fully internalize statically—this is the cornerstone of understandable code.

OTOH, ML says that even the very behavior of the system must be deferred until "runtime" information can propagate back. This is especially pronounced in online decision-making tools, but plays out over much longer time periods even with batch algorithm design. This "phase distinction" problem is something you will feel immediately in building ML-based code.

It's probably the case that black box ML algorithms exacerbate this. I don't have a lot of experience there, though.

</comment></comment><comment><USER5> Definitely agree with this. I do see more folks out there (like myself) with one foot in both the data science and production engineering worlds, and my personal approach to the problem of black boxes would be to have more of us! Rather than some 'sucker dev' implementing a statistician's hack, a full-stack mindset means the data science is done with engineering considerations all along.

But even then, half the joy of data science is not really knowing what you're doing at first as you research and experiment, just as your first attempt at a production system for something complex will also not necessarily be the final thing. If you have both of these at once, you can end up with a 'production' system that looks pretty well-designed and, indeed, works well enough at first -- until the product outgrows it, leading to a somewhat uncomfortable transition later on.

My personal anecdote starts with my PhD research which initially used a combination of Perl scripts, WEKA and god-knows-what-else, on fixed pre- formatted corpora, and became a Python engine powering a Django web-app using the Twitter API - not exactly rocket surgery at Google scale, but I found the whole shift in mindset from research to execution extremely useful and totally recommend to any data scientists who have focused solely on research that you try the other side of the fence, at least a little.

Stripping things down and simplifying, getting that 80/20 balance right, that's also some of the investigative fun of data science and we shouldn't get carried away with our own cleverness to ignore the poor sods who have to use the models we build years from now. I was consulting on a project which had very complex visions of a layered AI system using machine learning to make key decisions, but given the stage of the project and the depth of ML knowledge of the team, it was way easier for the initial stages to use what was effectively a switch statement on pre-defined values instead. You can always add complexity!

<comment><USER6> I see lots of common points with the "lean" method here (at the expense of technical debt), but can't forget Knuth's advice that premature optimization is the root of all evil.

To me, the utmost important aspect to take into account is the "key" of the business. I had a similar story with my PhD research (AI+NLP): I wanted to focus it on adding value to commercial products. I worked on the core of the implementation, I built it with Java, with a configurable pipeline, plus an online app with servlets. I did attain getting noticed by some companies, but at the moment of closing future lines with them, my adviser told me this was not what he expected from me, and I was forced to get back to the non-useful- goal-centred research of academia. I am still glad I did what I did, regardless of how useless it was eventually, but my knowledge grew, and I learned how important it is to know your business model before you do anything at all. I learned how unbalanced is the academic market, always relying on public funds to survive instead of worrying about building useful appealing stuff. I realized where I wanted to be, so I dropped out and joined the private industry where I now feel very fulfilled.

My present approach goes from small to big, little by little, getting as much feedback as I can so I can fix mistakes asap and prevent them from getting bigger and more difficult to manage. My agreement with your words.

</comment></comment><comment><USER7> The google paper the OP is summarizing is much, much better than the summary. [http://research.google.com/pubs/pub43146.html](http://research.google.com/pubs/pub43146.html) leading to [http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/43146.pdf)

</comment><comment><USER8> My attempt at abridgment.

Do not use more data sources than necessary. You can incur technical debt. [1]

Metaphor about dead bodies at the top of hill.

Be cognizant of the fact that in some use cases by making use of the information output by the predictive model you can reduce the effectiveness of the original data model. This is called a feedback loop.

The goal of ML is business. Not some new ML algorithm/technique or programming language paradigm.

[1] [http://research.google.com/pubs/pub43146.html](http://research.google.com/pubs/pub43146.html)

</comment><comment><USER9> "the pedestrianization of data science techniques"

This is the most condescending phrase I've read in a long time.

Otherwise the article had some good points, but that attitude really ruins it for me.

<comment><USER10> Hey, author here. It does sound kinda bigoted, doesn't it? Lemme clarify. I would really love for more people to learn this stuff. Truly. That's why I wrote a book to teach it to folks in Excel. What freaks me out a bit is that in order to enlarge the ML pie, a lot of vendors are trying to democratize machine learning not by teaching it to more people but by putting the gun in hands of folks who don't actually understand how supervised machine learning works and where it can go wrong.

I don't think people need PhDs to learn or do this stuff. But I think they need more than a "machine learning made easy!" app and a gung-ho attitude.

I feel that I'm halfway between main street and an ivory tower...not sure where that puts me. The upstairs bar at a pizza parlor?

<comment><USER11> I just recently posted in a HN machine learning thread asking for beginners resources. This sounds right up my alley.

I'm teaching myself Ruby (and other stuff), but consider myself pretty advanced with Excel and web analytics in general. This seems like a great way for me to get my feet wet in the deeper science of things with tools I'm already very familiar with (moreso than Ruby at least).

John, can you clarify a bit on how much background is needed in various areas of math to get the most out of this book? Or do you feel you do a solid job of teaching that as one progresses through the chapters?

<comment><USER10> A semester of linear algebra (or just a willingness to Wikipedia a few things) plus Excel experience is all you need.

That said, the book does require a lot of effort, because the techniques are worked through step by step.

But once you learn all the guts of the algorithms, you never have to implement them again! The last chapter moves the reader into R package land with the confidence that you now know what those packages are basically doing and what to watch out for.

</comment></comment></comment></comment><comment><USER12> I'm curious about the real world implementation risk and if anyone has a methodology to proactively deal with external factors affecting the model performance. Such as if a feature is highly predictive of a certain outcome is there a framework to measure the volatility based on information outside of the dataset ie. product changes, marketing campaigns etc.</comment>


Show HN: An Easy Programming Language That Runs in the Browser - <AUTHOR> https://kabas.online/ <comment><AUTHOR> What's under the hood?

The source code is parsed to an _AST_ , which is then executed. It began with a _Java-Applet_. When _WebAssembly_ appeared, I ported it to _C++_ with a _JS /HTML_ interface and added graphics. Recently I converted the _C++_ part to _C_ , which halved the size of the _wasm_ file.

<comment><USER2> > Recently I converted the C++ part to C, which halved the size of the wasm > file.

So you are saying that "costless" abstractions of C++ actually had some cost? Who would have thought.

<comment><USER3> I believe the philosophy of C++ is that features _you don't use_ should have zero _runtime_ cost.

<comment><AUTHOR> The C++ runtime library needs space in the wasm file.

</comment></comment></comment></comment><comment><USER4> The original Javascript was intended as an easy language that runs in the browser. And so it was in 1996.

In reality, it seems, a successful language that runs in the browser had no way around becoming a "full-blown" language, as opposed to "easy".

<comment><USER5> JavaScript still _can_ be easy if you ignore the rest of the ecosystem.

Thing is, sooner or later, at the very least you find yourself wanting something like lodash or jQuery[1]. Ironically, the latter also came into existence to make programming across different browsers simpler back in the bad old days (of not that long ago), and did an admirable job of that.

Unfortunately jQuery, and similar libraries of the mid-noughties era, provided the breach in the dam that's seen us swamped in a torrent of new, sometimes baffling, and on occasion arguably unnecessary platforms, libraries, and tools, which have made the process of developing for the client a much more intimidating process for those starting out.

 _[1] Although these days you 'd be surprised by how much of what jQuery offered is now built into the browser APIs and works consistently across them._

<comment><USER6> It's Jevon's Paradox [1] applied to the browser, where increasing the efficiency in which the browser can be scripted paradoxically causes websites to become slower and harder to build. Not because JavaScript itself (or any of the libraries built on top it) got worse, but because by allowing developers the ability to script more and more complex applications, sites became slower, and developer tools became heavier to support these more complex demands.

[1]: [https://en.wikipedia.org/wiki/Jevons_paradox](https://en.wikipedia.org/wiki/Jevons_paradox)

<comment><USER7> That sounds similar to Wirth's Law. In general the more efficient a computer's hardware is, the more bloated its software will be, so the slower the whole system will run.

<comment><USER8> While certainly related, I think GP was suggesting more that the same idea also applies to ease of development rather than raw hardware power.

<comment><USER9> I think that the raw-hardware-power aspect is actually a _special case_ of ease of development - in that case, it's the ease of not having to think about low-level consequences of your abstractions or spend effort on optimizing your code!

<comment><USER8> Fair point!

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> I'm a huge fan of these sorts of simple programming languages. I consider the simplicity of Purebasic and TI-83/89 Basic to be one of the main reasons I was able to successfully get into programming in high school despite not having any programming classes. I was also simultaneously learning x86 ASM and while that was also super fun, my Basic programming was far ahead of my ASM. Being able to successfully build simple games in Basic is extremely motivating when learning to code.

</comment><comment><USER11> Nice! It would be more fitting for the style of this language if 'end' were used instead of '.' imho

<comment><AUTHOR> `end` as block delimiter makes much noise. _Python_ code looks clean because it only uses indentations to delimit blocks. But this makes it at least difficult to refactor the code and to reformat it automatically.

<comment><USER8> I'm not sure why you are downvoted. Your statement is just an opinion, but it's _your_ language and you are perfectly entitled to aesthetic judgements like that.

The . kind of reminds me of Smalltalk, which uses it to end statements (instead of the ; we all know from other languages at the time)

<comment><USER12> Erlang also uses the . to end statements.

</comment></comment></comment></comment><comment><USER13> I think there's more value to be had in building better learning resources around programming languages than continually trying to simplify them at the expense of power.

Besides that, if someone is already looking for a simple scripting language, Lua already exists. I personally prefer its syntax as well.

<comment><USER14> Lua might have little features and have a small footprint, but it's not simple to understand

<comment><USER15> That's only half-true. Understanding all of Lua is hard, specially if you want to get into optimizing your code, but for simple tasks it's an incredibly easy to learn language. That being said, it really depends on the resources you're using to learn; Tutorials meant for programmers that bombard you with slang and exotic concepts will confuse newcomers.

</comment></comment></comment><comment><USER16> Reminds me of the old educational graphics programming language Logo.

[https://en.wikipedia.org/wiki/Logo_(programming_language)](https://en.wikipedia.org/wiki/Logo_\(programming_language\))

<comment><USER17> I grew up with Turing, which was nice later in life because it resembled pseudo code pretty closely.

[https://en.wikipedia.org/wiki/Turing_(programming_language)](https://en.wikipedia.org/wiki/Turing_\(programming_language\))

I think the best teaching language is one that can have a ball bouncing across the screen with as little ceremony as possible.

</comment></comment><comment><USER18> Really cool. Is there a project repository, and are you accepting contributions?

<comment><AUTHOR> Not yet

</comment></comment><comment><USER19> Great! Brings back the simplicity of Basic and QBasic. How easy it was to draw, animate and create music.

</comment><comment><USER20> The source is actually pretty readable, do you have a git repository anywhere so others can contribute?

</comment><comment><USER21> Nice work! I would love it if we could get something like a codepen.io for this language, so we could share and debug code with people learning to program.

<comment><USER8> Look under _" More Features"_ of the main website[0][1].

> _kabas.online can be opened passing a programm within an URL parameter_

If the author of the language reads this: you could try shortening the URL with LZString[2].

[0] [https://kabas.online/main.html](https://kabas.online/main.html)

[1] [https://kabas.online/run.html?title=Run&code=sz%20%3D%20100%...](https://kabas.online/run.html?title=Run&code=sz%20%3D%20100%0Acolor%20900%0Awhile%20x%20%3C%2050%0A%20%20move%20x%20x%0A%20%20line%20x%20%2B%20sz%20x%0A%20%20line%20x%20%2B%20sz%20x%20%2B%20sz%0A%20%20line%20x%20x%20%2B%20sz%0A%20%20line%20x%20x%0A%20%20sz%20-%3D%2010%0A%20%20x%20%2B%3D%205%0A).

[2] [http://pieroxy.net/blog/pages/lz- string/index.html](http://pieroxy.net/blog/pages/lz-string/index.html)

</comment></comment><comment><USER22> Impressive! But why?

<comment><AUTHOR> There are so many programming languages - but imho not a real easy one for beginners.

<comment><USER23> Python?

Python is really easy to learn and has reached VB levels of snowball network effects.

<comment><USER24> But even Python requires installing the dev environment etc. I can see the use for this, for throwing people with no experience into "coding right away".

<comment><USER25> There exists several python web environments.

[https://www.pythonanywhere.com/try- ipython/](https://www.pythonanywhere.com/try-ipython/)

[https://repl.it/](https://repl.it/)

[https://trinket.io/python](https://trinket.io/python)

</comment></comment></comment></comment></comment><comment><USER26> Fantastic work. Hope it finds its niche and grows

</comment><comment><USER27> Very cool, both the graphics capabilities and the fact that it's Wasm!

Is this project open source?

</comment><comment><USER28> Cool! Thanks for sharing.

</comment><comment><USER12> There should be something like this except functional/immutable

<comment><AUTHOR> Many design decisions are based on keeping the language simple in syntax and semantics.

<comment><USER12> I have found that immutability is actually simpler to understand (and, potentially related, results in fewer bugs per LOC written), it's just that everyone's (including mine!) first language is usually procedural/mutable so that seems to be what is "easy"

<comment><USER29> I can't fathom why you've been downvoted. I think you're absolutely correct: there's nothing "obvious" about either flavor.

It's really clear, in hindsight, why we started out with mutability (and self- modifying code and all sorts of "crazy" bit-twiddling goodies from the age of hardware-specific software).

But I don't think it takes too much imagination to envision a world in which immutability becomes the default, even (or especially) for beginners!

<comment><USER12> Now that RAM is relatively cheap and garbage collection algos are pretty decent, immutability should be the default, because it eliminates an entire class of bugs while enabling benefits like "easy" lock-free concurrency, etc.

Here's a project I'm excited about, I think it's the first fully- functional/immutable/declarative cross-platform GUI library in existence [https://github.com/boydm/scenic](https://github.com/boydm/scenic) Still pretty rudimentary but has made great strides in a year and the demo is pretty amazing (especially given how NOTORIOUSLY gui toolkits are typically OOP/procedural/mutable)

[https://www.youtube.com/watch?v=1QNxLNMq3Uw](https://www.youtube.com/watch?v=1QNxLNMq3Uw)

Boyd Multerer got famous via Xbox Live.

</comment></comment></comment></comment></comment><comment><USER30> much logo</comment>


Passports for Bitcoin - <AUTHOR> http://passportsforbitcoin.com/

<comment><USER2> I think that with the required $400k you can buy the whole country...

</comment><comment><USER3> Let's keep fueling the illusion that Bitcoin is still alive. And, yet, solidify the impression that Bitcoin is for criminals.

<comment><USER4> Richard Branson just invested millions in Bitpay, a bitcoin company. You should tell him it's dead!!

<comment><USER3> He has money to lose - like most Bitcoin investors.

<comment><USER4> Yeah he's a bad investor..

don't hate mate.

<comment><USER3> I don't hate, but you guys (which vested interest in Bitcoin, like, holding a couple of bitcoins waiting for them to be worth a million dollars) surely do hate people who don't resonate with you and you feel threatened by those who can get in the way of your pipe dreams.</comment></comment></comment></comment></comment>


Zynga Nightmare Continues - <AUTHOR> http://www.forbes.com/sites/terokuittinen/2012/05/21/zynga-nightmare-continues/

<comment><USER2> Does anyone else feel that this couldn't happen to a better company? :) And by that I mean, they've built their business off of stealing the work of other people. I cheer for them to fail.</comment>


Why is a minute divided into 60 seconds? (2007) - <AUTHOR> http://www.scientificamerican.com/article/experts-time-division-days-hours-minutes/ <comment><USER2> I'm surprised there's no mention of the relationship between time and the origin of the metric system in this article. Interestingly enough, it takes a 1-meter pendulum 1 second to swing from side to side, no matter how high you raise it. Mechanical clocks relied on pendulum motion to keep time, and so the meter was created (at least partly) to standardize the size of the pendulum needed to create an accurate clock! Now that's an interesting story.

<comment><USER3> > it takes a 1-meter pendulum 1 second to swing from side to side, no matter > how high you raise it

If I remember correctly, this is true if sin _x_ = _x_ ; IOW, "how high you raise it" is exactly what takes you away from this approximation being true.

Also, the meter was also originally defined as 1/10 000 000 of the distance from the north pole to the equator. It just so happens that these two come pretty close (it turns out that they had a big error in their measurement; a quarter-circumference of the earth is closer to 12 000 km than 10 000).

</comment></comment><comment><USER4> 10 vs 12 - A coworker nicely put it to me when I asked him why US is still using 12, as 12 inches is one foot, and he told me that during the great US expansion it was much easier to deal in units of 12 - you can divide 12 by 2, 3, 4 and 6, while 10 was divisible only by 2 and 5. Now whether that was really the case I don't know, but sounds like cool theory.

</comment><comment><USER5> > Interestingly, in order to keep atomic time in agreement with astronomical > time, leap seconds occasionally must be added to UTC. Thus, not all minutes > contain 60 seconds. A few rare minutes, occurring at a rate of about eight > per decade, actually contain 61.

So the second should be a tiny bit longer? Like 1.00000015 times longer?

<comment><USER6> Leap seconds can be positive or negative (never happened) and are irregularly announced up to two times a year.

The wikipedia article on leap seconds[1] is worth reading and has this[2] great figure showing the variations of day length over the years due to various astronomical factors. It also shows how these small deviations slowly desync UT1 and UTC (red line), and that this issue cannot be solved by a linear scaling.

[1] [https://en.wikipedia.org/wiki/Leap_second](https://en.wikipedia.org/wiki/Leap_second)

[2] [https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/De...](https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Deviation_of_day_length_from_SI_day.svg/512px- Deviation_of_day_length_from_SI_day.svg.png)

</comment></comment><comment><USER7> 60=2x3x2(4)x5 makes sharing farming products easier among 2,3,4,5,6,10,12 people.

77=7x11 is cool too, but families/partners with 7 or 11 people are less common

<comment><USER8> It feels like this reasoning is related to 13 being an "unlucky" number. You really can't split anything nicely between 13 people, or put them in even groups.

</comment></comment><comment><USER9> The babylonians where into base 60 doesn't everyone know that?

<comment><USER10> The Sumerians, not the Babylonians.

The Babylonians were an Assyrian people who conquered the Sumerians and adopted much of their culture, much the way the Romans took over the Greeks.

The answer to why the Sumerians used base 60 is: nobody knows, although as the article points out, 60 is a very convenient number with lots of useful factors, including 12, which was the number of divisions of the night.

<comment><USER11> 5 fingers, 4x3=12 phalanxes to point with thumb. 5x12=60.

<comment><USER10> 5 fingers, 2x5 = 10

4 fingers, 2x4 = 8. Same for feet => 16

7 major joints on each side of the body, 2x7 = 14

"Explanations" of this kind "explain" what was chosen, but offer no information about why it was chosen over all the other possibilities, and can always be easily adapted to "explain" the alternatives equally well, so in fact they tell us almost nothing.

Whereas we know that heliacal rising stars were used to divide the night into six "double hours" (danna) by Sumerians which gave twelve divisions of the sky. This was based on lunar months, which are a unique--albeit approximate-- source of the number 12, as opposed to anatomical speculations, which clearly are capable of generating just about any number you care to name.

Furthermore, simply because knuckle-counting "just makes sense" to some modern humans does not do more than incrementally raise the plausibility of the claim that it was the source of the base-12 system in the ancient world.

If you accept that the fact that it "just makes sense" to you is somehow compelling evidence for it, you have to accept that fact that it seems completely ridiculous to me counts as compelling evidence against it. The alternative--that I am an ignorant idiot and therefore my intuitions on this matter don't count--is... empirically untenable.

</comment></comment></comment></comment><comment><USER11> > Although it is unknown why 60 was chosen

Oh come on. This is not rocket science.

There is an easy way to count to 12 with just one hand: place your thumb on one of your phalanxes. There are three for each of the remaining fingers: hence, 4*3=12 possibilities. Babylonians counted this way, hence the duodecimal system.

You can use your second hand to count to five, as usual. 5x12 = 60. If you show one of the fingers on your left hand, and one of the phalanxes of your right hand, it really clearly communicates a number from 1 to 60.

This is why the base 60 was chosen.

<comment><USER10> To treat this as "obvious" rather than "interesting" is an epistemic error.

In particular, any ancient counting system could be given a similar anatomical analysis. Base 30? Well, it's only the first two knuckles that are easy to bend. Base 14? Clearly the major joints were being used for counting (because counting, not communication was the issue: people had words for communication and didn't need hand gestures). Base 8? Again, the first two knuckles, multiply both hands to get 64... so maybe that would be base 64?

Simply because you can give an account of what people actually used is no proof of explanation: you also have to given an account of why what was not used was not used. Claiming it was because it was somehow less convenient than the system you suggest is not an explanation, but a classic example of begging the question (assuming the consequent.)

</comment></comment><comment><USER12> This is why the metric system is stupid.

Now, last time I said this I was downvoted, but just because base 10 is convenient academically doesn't mean it is a great idea in day to day life.

The metric system is great for doing science but totally annoying for doing carpentry, for example. Centimeters are generally not a convenient size, and something as simple as dividing a piece of wood in to thirds is unnecessarily imprecise.

<comment><USER13> Well, they did think about making metric base-12, but in the hurry of the revolution, they left that out. It WOULD be more useful with base 12.

However, the nice water/gravity correpondence is really nice in practice (1 kg = ~ the mass of 1 L of water, 1 L = 1 dm cube). For instance you can easily estimate the weigth of something given its metric dimensions and the density of the material relative to water.

<comment><USER14> If you make the units base-12, leaving the numerical system base-10, it will be really inconvenient. How much is 12^4? 10^4 is obvious, even if this 10 is a decimal 12.

Base 2 is successful partly due to the fact that 2^10 is very close to 1000, which matches the kilo, mega scheme.

Edit: asterisks are for formatting, easier to use ^.

<comment><USER15> That's kind of the whole crux of the problem. It's all just culture. Some mathematicians like to act like math is a universal language that transcends culture, but ultimately it's impossible for a human to communicate anything without the basis of culture, so that dirty, ol' not-hard-science has to creep in somewhere. Or else what is the use of it, if you can't talk about it?

If we were more focused as a culture on a base-12 system, we'd be using a base-12 numbering system as well, in which case 12^4 would be written as "10000", leaving us to ask the crazy base-10 proponents just how easy they expect it would be to represent 10^4 ("5954" in base 12).

Base-2's popularity has nothing to do with 2^10 ~= 10^3, except where it serves hard drive manufacturers to sow uncertainty around the difference between Mibi- and Mega- bytes. Otherwise, it's just happy circumstance. Base-2 works because binary computers were ready for work before trinary computers were developed.

The numbers are only useful because we've decided as a culture that they should be so. There is nothing sacrosanct about the number 10 (or 12 for that matter). As all things, mathematics is only useful _insofar as it is useful_. We get to choose the way we abstract reality. It's kind of one of the things that makes us human.</comment></comment></comment></comment>


"Corn-Pone Opinions" by Mark Twain (1901) - <AUTHOR> http://www.paulgraham.com/cornpone.html <comment><USER3> I agree with this sentiment in general, especially that our own feeling of self-worth can be determined to a fair degree by the opinions of people we surround ourselves with, and that this drives conformity.

However, I'd add an important corollary, and one that is perhaps a bit less cynical :). Even if it's mostly true that you can only feel fulfilled when your actions and opinions are approved by your peers, you still get to chose _who_ your peers are, and whose opinion you value. That's true today more than ever, with the world's interconnectedness and the huge diversity of on- and offline communities.

Near the end of the article, Mark Twain says "Men think they think upon great political questions, and they do; but they think with their party, not independently." If you take the time to choose the parties/communities you align yourself with carefully, you can live a very principled and self- determined life even accepting the inevitability of some degree of group conformity.

<comment><USER4> I agree that it's more possible now to switch/choose alignment than it ever was. And being 'more possible' means that it must be happening more.

But "why people choose the communities they align with" doesn't seem much different than "why people have their opinions." Are we carefully choosing who we align ourselves with, any more than we're choosing our opinions?

I don't think there's any test for this, or any cure, other than making time to ruminate, meditate, and ponder. Alone. Free from distractions and other voices.

And if that's the case, isn't it at least _possible_ that today it's both easier to switch communities, and less likely to be for any principled reason?

<comment><USER5> Good point - one might choose one's peers based on the desired feedback - <USER2> end result wouldn't be more healthy than adapting oneself to the feedback of a static peer group.

</comment></comment></comment><comment><USER6> _that it is born of the human being 's natural yearning to stand well with his fellows and have their inspiring approval and praise -- a yearning which is commonly so strong and so insistent that it cannot be effectually resisted, and must have its way._

Actually I think it's the other way round. The force to conform doesn't come from seeking "approval and praise" but from avoiding the negative consequences of possibly being isolated and ostracised.

This is because where there's isolation there is disrespect and even violence from the peer group around the corner, apart from the negative effects it has on one's value on the "partnership market".

</comment><comment> irq-1 Twain was wrong on his main point. He claims self-approval as the motivator for following fashions (of all sorts), but most conformity comes from trying to navigate life: we do what works for our neighbors. IBMs dress code wasn't adopted because coworkers liked the fashion, but because if you wanted to work at IBM you needed to wear the uniform. Time and interest are limited, and copying is easy.

> Do you believe that a tenth part of the people, on either side, had any > rational excuse for having an opinion about the matter at all? I studied > that mighty question to the bottom -- came out empty. ... We all do no end > of feeling, and we mistake it for thinking. And out of it we get an > aggregation which we consider a boon.

This mistakes the value of expressing an opinion or having a vote. First, there's the personal benefit of investigating and thinking something through (even if you "came out empty"). Second, there's a public benefit to encouraging people to investigate and thinking something through on their own. Third, the 'wisdom of the crowd' might be right, show us new ideas, or offer insight and wisdom.

\--

This essay seems related to Paul Grahams "How to Do What You Love". He wrote that when deciding what to do, we should consider what's respected by our peers.

> What you should not do, I think, is worry about the opinion of anyone beyond > your friends. You shouldn't worry about prestige. Prestige is the opinion of > the rest of the world. When you can ask the opinions of people whose > judgement you respect, what does it add to consider the opinions of people > you don't even know? [4]

> This is easy advice to give. It's hard to follow, especially when you're > young. [5] Prestige is like a powerful magnet that warps even your beliefs > about what you enjoy. It causes you to work not on what you like, but what > you'd like to like.

[http://www.paulgraham.com/love.html](http://www.paulgraham.com/love.html)

</comment><comment><USER7> Now we all assume that this perfectly explains the people we disagree with. _They_ are victims of conformity and group think. Thankfully _we_ have reasoned opinions based in fact.

<comment><USER8> It is a cloistered mind that cannot make that leap.

<comment><USER9> Cloistered by the filter bubbles. Though Twain's essay suggests it's not a new phenomenon. People tend to group together with those who believe similar things, each confirming preconceptions of the others. From inside of such a group it always looks like everyone else are sheeple that need to wake up.

<comment><USER8> Your _are_ sheeple. But that doesn't mean that _I_ am not. People are very painfully reminded of their imperfection on a daily basis. They don't always care to keep it in mind, or take it into account, but surely they are aware.

Are you saying that people are not aware that their opinions may not be as ironclad as they sometimes seem?

<comment><USER9> > _Are you saying that people are not aware that their opinions may not be as > ironclad as they sometimes seem?_

That's the impression I'm getting when talking with many of them. Or when browsing the more mainstream parts of the Internet.

<comment><USER8> Here's to us being the lucky few who can see that we're fallible, while the sheeple wallow in their ignorance ;-)

<comment><USER9> :). I'm happy being a meta-sheeple, because thinking too much about it only makes one's head hurt. ;).

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> If, like me, you didn't know what corn-pone is, it's a type of corn bread.

</comment><comment><USER11> I like how Emerson felt about conformity, not to mention - to be exceptional, one must be a non-conformist. Many would rather be accepted than exceptional At some point in your life, you may have to make that choice.

The problem with public opinion in our times is this idea that the crowd is always right. Something is true because the crowd says it is. The crowd saying it is, makes it true. Sane is not the same as true. Sane just means many people believe it. The person who believes something no one else does is considered insane. When the lone "insane" person is vindicated, great wealth and success seem to follow.

<comment><USER12> > Many would rather be accepted than exceptional. At some point in your life, > you may have to make that choice.

I think that there are two problems with this. (EDIT: 'Problems' is a bad word. I don't mean two things that make it wrong, but rather two things that require a careful reading to appreciate fully. I should have said 'subtleties'.)

First, and most importantly, I think that people think that being exceptional _is_ a choice. Thus, someone says "ah, I shall be exceptional!", and sits back and waits to become so. _Being_ exceptional is not a choice; _becoming_ exceptional is, though it is a slow process, subject to setbacks, and not guaranteed of even eventual success (unless—as is perhaps uniquely appropriate here—one is willing to move the goalposts when necessary, and find success where one is).

Second, I take your explicit point to be the decision between being accepted and being exceptional, but I'll respond as if it were between being conformist and being exceptional (which I took to be the subtext). After the problem is surmounted of deciding just to _be_ exceptional, and realising that one must _become_ so, I think that sometimes people confuse blind non-conformity with reasoned non-conformity. That is, a decision not to conform just in order _not_ to be part of the crowd is just as silly as a decision to conform to be part of the crowd. Rather, make your decisions as you will, regardless of the pressure of the crowds, with the idea that you will sometimes agree with and (probably) more often disagree with them, and become exceptional in that way.

<comment><USER11> In deciding to be exceptional, I'm thinking of the courageous act of separating yourself from the crowd when it's safer to conform, when faced with a binary decision between the two. That decision can have lifelong consequences. A friend who could have been an olympic swimmer - but it was easier to not put in the effort, get a job at the local health food store, and get married. Or the brilliant kid with enormous potential, growing up in the bad neighborhood, who purposely fails at school to not get bullied.

I would separate the non-conformist whose conscience puts them at odds with a misguided but widespread popular opinion, from the contrarian. Being contrarian just for the sake of it can be silly sometimes. Although I still like those who consider that so much of their identity that they take contrarian positions even when unnecessary. If anything they are a non- yielding role model and we could use some of that, cause the scales are definately tipped in favor of conformity. It's drilled into us from a very young age.

<comment><USER12> > In deciding to be exceptional, I'm thinking of the courageous act of > separating yourself from the crowd when it's safer to conform, when faced > with a binary decision between the two.

I still think that this decision is one concerning _becoming_ , rather than _being_ , exceptional. That is, I think of being exceptional as the accumulated result of good decisions, rather than as a single (even lifelong) decision in its own right.

> I would separate the non-conformist whose conscience puts them at odds with > a misguided but widespread popular opinion, from the contrarian.

Oh, I see; I was reading 'non-conformist' as essentially synonymous with 'contrarian'. Your clarifying the distinction addresses my second point.

> Although I still like those who consider that so much of their identity that > they take contrarian positions even when unnecessary. If anything they are a > non-yielding role model and we could use some of that, cause the scales are > definately tipped in favor of conformity.

As a reflexive contrarian (I am so inclined to test positions by argument that, if it happens that I convince my interlocutor, then I sometimes unthinkingly argue _against_ his or her new position, which I had been urging, and _in favour of_ the original position), I am not sure that there is as much value to such a position as you suggest; but that is a minor point, and besides I could easily be wrong.

<comment><USER11> The technologist, specifically the software developer (ie.. the typical HN reader) seems to often be a non-conformist, even contrarian. Why do you think that is? Cause we think we're smarter than everyone else, is my theory!

<comment><USER12> > The technologist, specifically the software developer (ie.. the typical HN > reader) seems to often be a non-conformist, even contrarian. Why do you > think that is?

I'm a mathematician, not a software developer, but I think that both professions train their practitioners to believe that the way to get good tools (or theorems) is to try to break them, and to keep only those that survive—a 'confrontational' approach that it is (perhaps too) easy to carry over into the rest of life. I won't deny that there is probably some arrogance in my contrarianism, too.

<comment><USER9> Often not immediately agreeing with a mainstream opinion, or stopping by to evaluate pros and cons, is seen as being contrarian or signaling smarts.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> True as far as it goes, but there are a lot of other factors influencing why some ideas spread widely and others don't. These days we can even track how it happens.

</comment><comment><USER14> Mark Twain had no trouble thinking for himself. Paul Graham does the same kind of thing in his essays. Every great standup comedian in the world does it.

It's a skill that few practice, but thinking for yourself is not actually difficult. Most people have the basic reasoning skills required, they just never care enough to "rock the boat". And that's the real cause of conformity: apathy.

<comment><USER9> I don't think it's apathy. I feel it's culture and economic considerations.

Society generally doesn't reward you for thinking outside the box, or outside the "outside the box" box. It rewards you for following the same rules, norms and patterns as everyone else does. Meanwhile, you have your spouse and kids to feed, house to keep warm, lots of work to do for little pay and an abusive boss that keeps you on your toes. Most people don't have time to think for themselves, even if it was rewarded, and it isn't.

</comment></comment><comment><USER15> [https://www.youtube.com/watch?v=hP8IxKizzWc](https://www.youtube.com/watch?v=hP8IxKizzWc)

</comment><comment><USER16> Monkey see, monkey do.

</comment><comment><USER17> And it readily brings the famous Nazareth's evangelist to mind- Jesus Christ.

Considering the 6 stages of his blasphemy trials between Religious and Roman courtrooms (especially, the burdened denial of right and wrong between Herod and Pilate).

And contrasting that unfortunate crisis of conscience greeted by the reported mob justice with how many billion people today follow a religion borne of Christ's personality, it just reinforces one thing Mark Twain endeavored to deliver as important message in that essay:

"Think for yourself."

<comment><USER8> I like how on an article about crowd conformity and how certain thoughts may come to be frowned upon, your comment gets downvoted because it's Christian.

</comment></comment><comment><USER18> "Corn-Pone Opinions" by Mark Twain, written in 1901, first published in 1923. Twain died in 1910.

<comment><USER19> Thanks, we put that year in the title.

</comment></comment><comment><USER20> Here's what I have to say about that: That's pretty rich, coming from you, Paul Graham.

You have All The Money, and All The Respect amongst your large and powerful peer group. Given your considerable wealth, there's nothing anybody can do to negatively impact your life in any meaningful way. Yet you continue to bang the same drum you've been banging for a decade now, at least.

Time to tell us something we _don 't_ already know.

<comment><USER13> Paul Graham didn't write it.

</comment></comment><comment><USER21> He started his essay w/ the example of a gay person.

If you want to be outside of the herd in thought, it would aid you to be something alien to the majority.

edit: you're right.

<comment><USER22> I'm pretty sure, in 1901, gay meant:

keenly alive and exuberant : having or inducing high spirits <a bird's gay spring song></comment></comment>


David Bowie's Top 100 Books (2013) - <AUTHOR> http://www.openculture.com/2013/10/david-bowies-list-of-top-100-books.html <comment><USER17> The complete list can be found here [http://www.davidbowie.com/news/bowie-s- top-100-books-complet...](http://www.davidbowie.com/news/bowie-s- top-100-books-complete-list-52061) as explained in the comments. It has the 100 books whereas there are only 75 of them here.

</comment><comment><USER18> Interesting to see him studying traditions he stepped into. Like African American origins of rock, sort of starting from Richard Wright to soul and rock histories. I thought J. R. R. Tolkien would have been in there.

Been listening to a lot of his stuff, focussing on the sort of literary thread about his Berlin era that you can read about on his wikipedia page[0].

[0] [https://en.wikipedia.org/wiki/David_Bowie#1976.E2.80.9379:_B...](https://en.wikipedia.org/wiki/David_Bowie#1976.E2.80.9379:_Berlin_era)

</comment><comment><USER19> There are some corkers here. Wonderful to imagine Bowie reading _The Leopard_ (I wonder if he enjoyed the masterful Visconti film of it, too).

</comment><comment><USER20> So he didn't seem to like Sci-Fi too much; only two standards (1984, Clockwork Orange) among the top 100 as far as I can see.

<comment><USER21> It's a great reading list, but I was also surprised not to find Arthur C. Clarke's "Childhood's End" in there. I had always assumed his song "Oh! You Pretty Things" was about that book. While the book and the song share themes, they are apparently independent. I did like this passage I found online about the themes though [1]:

 _The resonance of "Oh! You Pretty Things" comes from how it uses these Nietzschean SF trappings as a metaphor for how a generation regards its successor with longing, fear and resentment (never more so than with the so- called Greatest Generation and their children the Boomers), or, even closer to home, how a parent can regard his or her children. Once you become a parent, you lose precedence in your own life—your own needs and desires are shunted aside, and you spend years as servant and guide to your replacement, who will go on to have richer experiences and greater opportunities than you ever had (that's if you're lucky). More bluntly, once you reproduce, your genetic purpose is fulfilled and all that remains is age, redundancy and death._

[1] [https://bowiesongs.wordpress.com/2010/02/05/oh-you-pretty- th...](https://bowiesongs.wordpress.com/2010/02/05/oh-you-pretty-things/)

</comment></comment><comment><USER22> by author:

Beano (comic, '50s)

Private Eye (satirical magazine, '60s – '80s)

Raw (comic, '80s)

Viz (comic, early '80s)

Paris Review, The - Writers At Work: The Paris Review Interviews edited by Malcolm Cowley

Ackroyd, Peter - <USER2>

Alighieri, Dante - <USER3>

Amis, Martin - <USER4>

Baldwin, James - Fire Next Time, The

Barnes, Julian - Flaubert's Parrot

Bellow, Saul - <USER5>

Braine, John - Room At The Top

Broyard, Anatole - Kafka Was The Rage

Bulgakov, Mikhail - Master And Margarita, The

Bulwer-Lytton, Edward - <USER6>

Burgess, Anthony - A Clockwork Orange

Burgess, Anthony - Earthly Powers

Cage, John - Silence: Lectures And Writing

Camus, Albert - Stranger, The

Capote, Truman - In Cold Blood

Carter, Angela - Nights At The Circus

Chabon, Michael - Wonder Boys

Chatwin, Bruce - Songlines, The

Cohn, Nik - Awopbopaloobop Alopbamboom: The Golden Age of Rock

Cork, Richard - David Bomberg

Crane, Hart - Bridge, The

Danto, Arthur C. - Beyond The Brillo Box

DeLillo, Don - White Noise

Diaz, Junot - Brief Wondrous Life Of Oscar Wao, The

Döblin, Alfred - Berlin Alexanderplatz

Dos Passos, John - 42nd Parallel, The

Edwards, Frank - Strange People

Elliot, T.S. - Waste Land, The

Faulkner, William - As I Lay Dying

Figes, Orlando - A People's Tragedy

Fitzgerald, F. Scott - Great Gatsby, The

Flaubert, Gustave - Madame Bovary

Friedrich, Otto - Before The Deluge

Gillete, Charlie - Sound Of The City: The Rise Of Rock And Roll, The

Ginzburg, Eugenia - Journey Into The Whirlwind

Guralnick, Peter - Sweet Soul Music: Rhythm And Blues And The Southern Dream Of Freedom

Hall, James A. - Halls Dictionary Of Subjects And Symbols In Art

Harding, Douglass - On Having No Head

Hirshey, Gerri - Nowhere To Run The Story Of Soul Music

Hitchens, Christopher - Trial Of Henry Kissinger, The

Homer - <USER7>

Isherwood, Christopher - Mr. Norris Changes Trains

Jacoby, Susan - Age Of American Unreason, The

Jaynes, Julian - Origin Of Consciousness In The Breakdown Of The Bicameral Mind, The

Kerouac, Jack - On The Road

Kidd, David - All The Emperor's Horses

Koestler, Arthur - Darkness At Noon

Laing, R. D. - Divided Self, The

Di Lampedusa, Giusseppe - Leopard, The

Larson, Nella - <USER8>

Lautréamont, Comte de - <USER9>

Lawrence, D.H. - Lady Chatterly's Lover

Lebowitz, Fran - Metropolitan Life

Lévi, Eliphas - Transcendental Magic, Its Doctine and Ritual

Lewis, Wyndham - <USER10>

Marcus, Greil - Mystery Train

McEwan, Ian - In Between The Sheets

Milligan, Spike - <USER11>

Mishima, Yukio - Sailor Who Fell From Grace With The Sea, The

Mitford, Jessica - American Way Of Death, The

Nabokov, Vladimir - <USER12>

Norman, Howard - Bird Artist, The

Norris, Frank - <USER13>

O'Hara, Frank - Selected Poems

Orwell, George - <USER14>

Orwell, George - Inside The Whale And Other Essays

Packard, Vance - Hidden Persuaders, The

Pagels, Elaine - Gnostic Gospels, The

Paglia, Camille - Sexual Personae: Art And Decadence From Nefertiti To Emily Dickinson

Petry, Ann - Street, The

di Pirajno, Alberto Denti - A Grave For A Dolphin

Priestley, J.B. - English Journey

Rechy, John - City Of Night

Sadecky, Peter - Octobriana And The Russian Underground

Saunders, Ed - Tales Of Beatnik Glory

Savage, Jon - <USER15>

Selby, Hubert Jr. - Last Exit To Brooklyn

Spark, Muriel - Prime Of Miss Jean Brodie, The

Steiner, George - In Bluebeard's Castle

Stoppard, Tom - Coast Of Utopia, The

Sylvester, David - Interviews With Francis Bacon

Thomson, Rupert - Insult, The

Thurman, Wallace - Infants Of The Spring

Toole, John Kennedy - A Confederacy Of Dunces

Waterhouse, Keith - Billy Liar

Waters, Sarah - <USER16>

Waugh, Evelyn - Vile Bodies

Weschler, Lawrence - Mr. Wilson's Cabinet of Wonders

West, Nathanael - Day Of The Locust, The

White, Charles - Life And Times Of Little Richard, The

Wilson, Colin - Outsider, The

Wolf, Christa - Quest For Christa T, The

Wright, Richard - Black Boy

Yokoo, Tadanori - Tadanori Yokoo

Zinn, Howard - A People's History Of The United States

</comment><comment><USER23> No Sartre or Camous? No Hundred Years of Solitude? No Pamuk? No Henry Miller? No Nabokov's "Dar", no "Ishmael", no " Stepenwolf", no "Dorian Gray", no "Brave New World"? No "mockingbird", no "Atlas", no motorcycle maintenance? No dr.Zhivago? What kind of a list it is?

<comment><USER24> No Thomas Pynchon? No David Foster Wallace? No Heidegger? No Kundera? No Kerouac?

Oh what a terrible person, to have not read everything in an arbitrary list of authors and books that I feel like name dropping.

<comment><USER25> Is David Foster Wallace really among that calibre of writers? I'm reading Infinite Jest and I think I kind of hate it. There's some amazing chapters, but I can't help but loathe a lot of it. It just feels like pretentious drivel that isn't written by someone who has a clear idea of whats occurring, but by someone who wants to write what sounds "cool". Its almost like emotional pandering. I know he lead a pretty eventful life, but he tries to have so much depth on so many different topics, I get the impression he did the shallowest research time could afford. It would be the equivalent of trying to fit as many highlighted words on a wikipedia page into a paragraph as possible, without understanding what any of them mean. For example, he has some kind of obsession with optics, but every description reduces to convex/concave mirrors and "it just works".

<comment><USER24> I'm currently reading it too, and am about 100 pages from the end. On a chapter-by-chapter level I think the quality is mixed, in the sense that not every chapter has the impact and pace that the first chapter has. In some chapters it feels like he labours a point for far longer than necessary for the reader to get the message.

One criticism I hear quite a lot is that DFW's own writing on the topics of addiction and depression are shallow and not representative, though I can't really say how true this is for lack of experience.

The thing that worked for me in I.J. was that Wallace managed to combine elements of comedy and tragedy, not just in alternating parts but often within the same body of text. Some chapters are so tragic as to be darkly comic, in particular some of the descriptions on the backgrounds of members of AA.

Plus, don't forget that Infinite Jest isn't his only work :). I haven't read The Pale King but have heard that it does away with a lot of the verbosity of I.J. and feels a lot more focused, and there are other works of fiction and also his more journalistic essay-ish type pieces, for example those from A Supposedly Fun Thing I'll Never Do Again (the collection, though the titular piece in particular I found very funny).

I'd also highly recommend Oblivion: Stories, even if Infinite Jest didn't gel with you, as I think it's as emotionally powerful but a lot more focused than Infinite Jest. It doesn't attempt the comedy/tragedy mix that I.J. seems to try for, and as a result the stories are shorter but just as affective.

<comment><USER25> The addiction and depression thing really bothered me too. There was so much detail on those topics I figured he must have gone through a lot of it himself before I read up on his life. Yet, there's so much that's just patently false. He goes on and on about the depravity of marijuana addiction and has a character try to commit suicide from overdosing on 4-5 hits. Ridiculous. I guess this is some parallel world where things aren't exactly as they are in real life, but that just seems like an easy out.

I'm barely half way through the book, but some of the chapters written in ebonics are atrocious. He starts out the chapter totally over the top and it seems like he forgets he was writing in ebonics by the end. There's just so many places it feels like his editor just said "fuck it".

There are definitely a few sections that I get lost in, but they are so few are far between. I could probably count them on one hand. More than half the book I am pretty neutral about and then there is a significant portion I read and just look up and think, this author is full of shit. I guess its just his style, but when he was describing "eschaton", he has no idea how the game would actually be played. I guess the way it ends up kind of describes that the depths of the game were really beyond those playing it, but thats kind of pattern throughout the book. He has this grand idea and exposition, but it just sort of devolves out of laziness, or style, or I don't know what. I'm kind of terrified I'm going to reach the end of the book and he's going to devolve again and introduce some stupid non-sequiter to wrap things up cleanly, without any introduction or background.</comment></comment></comment></comment></comment>


Why Silicon Valley is better than the rest of the places I've visited - <AUTHOR> https://plus.google.com/u/0/111091089527727420853/posts/ewhtNzwTGPr

<comment><USER2> That's a poor comparison, though. Engelbart, in his genius, was pushing the boundries of technology itself. If he was socially inept, it's irrelevant.

Google, in this example, is pushing the boundries of _the application of technology_. (Look! You can Skype as you skydive!)

As technological progress continues, we're going to find wonderful uses for new technology that we've never thought of before -- and we're going to find horrible uses as well. We'll say to ourselves later, "What were we thinking?" Certainly our instinct is to be shy of new uses of technology, but that doesn't mean such an instinct is always wrong.

<comment><USER3> In other words, Engelbart was "inventive", while Google is "innovative".

<comment><USER2> Everyone is innovative. A child using a tablet in a new way is innovative. Engelbart was the kind of genius that creates tsunamis that everyone (you, me, Google) are still surfing off of decades later. Call it what you want.

</comment></comment></comment><comment><USER4> Nice argument for building products for leads users - <http://en.wikipedia.org/wiki/Lead_user>

</comment><comment><USER5> What? It's cool to be in tech now...been that what for years now and we're all social out here in Palo Alto. Of course I came from New York...</comment>


Ask HN: Software Craft in Your Company - <AUTHOR> I'm Tech Lead in a relatively sized Corp (3k employees, 200 developers) and I'm doing full stack from mobile to Backend and APIs, and I was wondering what is the state of Software Craft in your company.
More specifically how did you create a culture of development quality (well-crafted software) ?
What are successful insights to install a culture of learning and make other developers unite and want to improve (create a community of professionals)?
I've been trying for a year (events, katas, coding games, tech watch, ...) and definitely had some results with some of our passionate developers (approx. 20 to 30) but having a hard time to gather more developers, either they don't have a time or they don't want to take it (probably they see no value?).
OTOH we have recurrent defects and code smell, code review is not always enforced properly, unit tests not optimal (or not properly understood).
Do you do weekly katas? Weekly events? Some kind of university inside the Corp? Mob programming? Did you reach to your management to have developers having actually time to devote to improvement? Deliver somewhat less but with more confidence, better quality meaning less people woke up at night.
I know I won't be able to have everyone interested, but I'd be delighted to see what the HN crowd is doing regarding this subject, so that I can also improve :) <comment><USER2> What is your goal with the coding events? These things have more to do with motivation, innovation, and team-bonding but very little bearing on quality of employee code culture. Also, the most productive engineers will not want to waste valuable time on these types of events.

The only genuine way to foster a software engineering culture is to hire proven and experienced software veterans as leaders in your engineering organization. And when I say proven and experienced, I don't mean people that are just good at LeetCode. Remember, culture always comes top-down and the best coding culture is a result of good mentoring and calling out bad coding practices before it spreads like a virus.

<comment><AUTHOR> My goal was to start something and make people know each other around drinks and some interesting stuff like conference videos, latest news (react hooks / suspense at the time, Spring related, Functional Programming, ...).

Really just to start something because teams are isolated and don't speak that much to each others. When something is not working it's always another team fault for example.

I think you're right regarding the hiring process, ours is just "soso", we would definitely need more software veterans. We have some but not so many and even with events/meetings/chapters where we make decisions about code review/best practices and so on, but not enough seniors to actually enforce this in each team and keep an eye on it.</comment></comment>


Idea Sunday - <AUTHOR> I think it is good to post this again and want to hear good Idea's from HN community. <comment><USER2> Obligatory (valid) reason why Idea Sundays were stopped: [https://news.ycombinator.com/item?id=7693640](https://news.ycombinator.com/item?id=7693640)

Previous discussion on the previous attempt to restart it (which includes a comment from the OP of this discussion): [https://news.ycombinator.com/item?id=10326550](https://news.ycombinator.com/item?id=10326550)

<comment><USER3> [I know this thread has already been killed, but for the record:]

A cool feature would be submitting a poll or Ask with a flag called [community]. It would mean: I give up any karma for posting this poll|Ask thread.

It's simple: one bit. Anyone who asks a community question and doesn't mark it [community] is fair game to be flagged into oblivion.

That should remove the incentive for karma-whores, while giving the community at large a chance to self-administer itself.

Those idea threads are clearly popular, losing them is kind of sad.

</comment></comment><comment><USER4> if this is going to happen again it needs to avoid the problems of the last threads.

1) Monthly at most. Certainly not weekly.

2) A separate account needs to create threads. Some blurb needs to be put in place to prevent other accounts racing to create the threads.

3) The threads create work for the mods. That needs to be understood, and stuff put in place to minimise it.

I said all that the last time you posted this thread.

The way you're doing it now is not avoiding those problems, and is doomed to fail. Ignoring minimaxir's link to mod comments seems sub-optimal. Have you tried emailing mods? (Make sure you address the points I mention above if you do).

<comment><AUTHOR> Thanks. I agree with you. I think it is better if HN community moderators post this with separate account. Monthly would be better. No I did not contact anybody. I think they know well and It is their responsibility.

</comment></comment><comment><USER5> Labful ideas: [https://twitter.com/LabfulHQ](https://twitter.com/LabfulHQ)

</comment><comment><USER3> I posted my latest, crazy idea here: [http://foundrs.com/the-most-insane-idea- ever](http://foundrs.com/the-most-insane-idea-ever).

This startup idea will

a. Restore democracy

b. Provide jobs to thousands

It's a startup that's

c. Profitable from day one

d. Extremely viral

e. Easy to launch with very little technical development

f. Media coverage almost guaranteed

<comment><USER6> Kind of like TaskRabbit

<comment><USER3> Exept that TaskRabbit doesn't accomplish a, d, e and f. So no, not really. You could build it on top of TaskRabbit I guess is what you are saying. True.</comment></comment></comment>


Building Your First Profitable Startup - <AUTHOR> http://www.layeredthoughts.com/startups/12-rules-for-building-your-first-profitable-startup <comment><USER2> This article is not using a correct definition of "scalable".

A business that needs more staff to service more customers is still scalable, as long as each staff member generates more revenue than they cost. McDonalds is awesomely scalable.

If you insist on limiting yourself to zero-touch businesses, you'll find yourself competing with all the other geeks who are afraid to go out and sell and build business relationships. On the other hand, if you're willing to build high-touch relationships, you'll discover opportunities that you would have never otherwise found.

<comment><USER3> I'm afraid to report that you have also abused the term "scalable". You have instead described something normally called the Marginal Product of Labour (MPL). I believe the claim in the OP is to the effect that Groupon's MPL is flat or even declining and have to concur with the assessment that a high- touch business model requires huge technological support to remain competitive.

Indeed your own example, McDonalds, is a prime example of this. They are not particularly notable as a human business, rather they represent a case study in very high efficiency supply chain management and process optimisation.

I don't disagree that a strong and skilled front line enables enduring customer relationships and opens many doors. But the recommendation - to do so with a clear eye for efficient processes & systems - is exceedingly sound.

<comment><USER4> I thought McDonalds was a real estate and leasing company? McDonalds only owns about 15% of its restaurants, it doesn't have as many employees as you think.

<comment><USER3> This simply highlights how efficient they are. The franchise operators are subject to the same systems & processes irrespective of ownership.

</comment></comment></comment></comment><comment><USER5> One thing I'd like to add to this from my own experience is: The trick is to Just do it.

Lately, I've seen this trend when guys are waay to concerned over which language, platform, etc to use. Will it be Coffeescript or Javascript, CSS or Less, Apache or nGinx, PHP or Python, Mysql or MemSQL, and on and on.

At least in my own experience it doesn't matter that much, but what matters most is that you pick something (if it's your first startup it has to be something fast + easy) and you stick with it with just one goal: you get the first version out of the door as soon as possible and start collecting leads.

Because funny thing is, your lead does not care how you're inserting him into the database as long as you do it okay. But if you waste a month deliberating the perfect tools the time you've lost has been lost forever. The same time you could have spend converting those leads into paying customers, getting your site in Google, marketing, etc are all a month behind (even worse sometimes due to analysis paralysis you get either too busy learning something or too bored with the idea that you may never finish).

<comment><USER6> And this is 100% correct. Nobody cares about technology while it works from the customer perspective and gives you enough momentum to evolve your product.

A great set of hints was also given by JHH in his widely known presentation: <http://www.youtube.com/watch?v=0CDXJ6bMkMY> (but still so valid!)

<comment><USER7> And as evidence, I present the current contents of the posted link:

Bandwidth Limit Exceeded

The server is temporarily unable to service your request due to the site owner reaching his/her bandwidth limit. Please try again later.

</comment></comment></comment><comment><USER8> One thing I'm having trouble with (that I haven't seen mentioned anywhere) is getting alpha testers. I'm building extensible analytics software (www.instahero.com, not all ready yet), and I need people who could use something like that, but I haven't been able to find any.

You can't try marketing at this stage (people will expect something that works) and all the people I know are at least slightly out of the target market.

Maybe it all comes down to having a good network... If anyone would like to try Instahero (even for very custom stuff, like hardware sensor tracking), get in touch with me (info is in my profile) and I'll set you up.

<comment><USER9> You NEED a free forever plan, especially if you're in alpha. Do something like MailChimp where you get a low number of monthly 'credits', and when websites gain traffic if they're getting value from your product they will upgrade. The other thing this does is gets you a lot of 'leads' -- people who are clearly interested in your product and already have it on their site. You can send a monthly newsletter reminding them of all the great features that come with the paid plan and stay in the back of their mind. (You can always kill the free plan if it's costing you a ton of money)

<comment><USER10> If you set a precedent that your product is free, no one who's signed up will ever pay for it.

Charge right away.

<comment><USER11> This is true, except when it's false. Stormpulse was free for years and is now collecting subscription revenues from customers that used to be free users. And they are not the slightest bit angry about paying.

</comment></comment></comment></comment><comment><AUTHOR> Hey I'm the OP: Ran out of bandwidth in the middle of the night (blog is hosted on a very old hosting account) and am waiting for the host to get back to me so I can fix the situation. Will be moving to a Linode later today to prevent these issues in the future.

Here's the Google Cache for the time being: [http://webcache.googleusercontent.com/search?q=cache:www.lay...](http://webcache.googleusercontent.com/search?q=cache:www.layeredthoughts.com%2Fstartups%2F12-rules- for-building-your-first-profitable-startup)

</comment><comment><USER12> It's no coincidence that several of the companies which recently found themselves hurt by being locked out of APIs did not heed:

"Rule #1: Sell something" - several had no evident source of revenue and as the article says "if your company isn't selling something, you have a project, not a product"

"Rule #7: Avoid products that rely on a community to exist". As these projects found, when the community (or truckload of data) is taken away from you, you no longer exist.

A good article with pragmatic tips.

</comment><comment><USER13> My biggest fear has always been customer service. How does one launch a startup and still provide ample customer service without the humans there to serve? Having lots of docs isn't enough for some people, especially in a B2B setting. Do you have to build a customer base slowly because of this?

<comment><USER11> I've found that there is a lot less customer service work to do in a B2B setting than B2C.

</comment></comment><comment><USER14> What if you dont want to build a freaking SEO, ecommerce, fashion or other site but want to do something that really makes a difference?

Most of your rules only allow you to pick the lowest hanging fruit.

<comment><USER15> The rules are intended for "bootstrapping a reliably profitable company". So yes, they do restrict you to the lowest hanging fruit.

This post isn't directed at people who want to start a company that "really makes a difference", unless those people are trying to start a profitable prior company to fund the really big idea.

<comment><USER14> What if I want to bootstrap a game changing company? :) ... I understand the inherent risks but I'be been willing to try so far.

</comment></comment></comment><comment><USER16> Strongly disagree with Rule #5: Have a "no-touch sales process".

That's actually one of the biggest reasons why companies fail. Founders are afraid to talk to people, they just build something and wait until somebody comes and buys it. What happens next I guess all of us know.

You have to interact with your prospective customers and having a high-touch sales process is one way to do that.

<comment><USER17> I don't think he's saying to be afraid to talk to people. Often when I'm making purchases for my job though I'll run into a product that doesn't show any pricing and says to contact their sales staff if you are interested in making a purchase. I _hate_ that and will pretty much always pass them over for a site that doesn't _force_ me to contact them and doesn't _restrict_ my purchase options.

If I can see pricing and purchase at any time but I still have questions, sure I'll contact them. But don't force me to.

<comment><USER16> In my previous company we had a solution that cost $1000/mo. When we advertised it on the website - we had virtually no signups at all. When we removed it - some people start contacting us and some of them were converted into customers.

When you don't see the price - in most cases that means that this solution is strictly B2B, so you're not their target market.

</comment></comment></comment><comment><USER18> Can anyone post some examples of #3?

The biggest thing I'm struggling with is finding a product idea I can even test against the potential market.

<comment><AUTHOR> OP here: I have an ambitious goal of expanding each rule into full posts with examples so I'll see what I can do for ya

<comment><USER18> Thanks! I'll keep an eye out for it.

</comment></comment></comment><comment><USER19> Any reposts? Looks like HN crowd has exceeded his bandwidth...

<comment><USER20> Try Google Cache: [http://webcache.googleusercontent.com/search?q=cache%3Awww.l...](http://webcache.googleusercontent.com/search?q=cache%3Awww.layeredthoughts.com/startups/12-rules- for-building-your-first-profitable-startup)

<comment><USER21> For ctrl-F:ers: mirror

</comment></comment></comment><comment><USER22> Would be interested in a followup article on how to find niches/markets. I like his About page - keeps it real.

</comment><comment><USER23> Another name for "no touch sales process" is "self service".

</comment><comment><USER24> I know it may be formulaic, but I love these lists of "rules" because I go down them one by one and say "yep, got that" or "nope, don't have that, now lets think about it."

I want to talk about rule #3. And lets use DuckDuckGo as an example. DDG is a competitor to google, and it is safe to say google is a "must have" because you can't find nothing on the web without it.

But is DDG a "must have" because it is providing the same service as google, and that service is a must have?

Or is DDG a "nice to have" because it is a slightly better version of the service Google is providing and thus I don't have to have DDG because if it disappeared I would switch to google and not feel much pain.

No criticism of DDG is meant here, I admire that business, and it was just the best example of the area where I'm not sure. For my personal business, there is an undisputed king of the hill, like google, but I know there is sufficient business for us to service niches. I just wonder if the king of the hill existing makes it impossible for your product to be a "must have", or if this is just a question of the nature of the product/service, and not the market dynamics of the space.

<comment><USER11> Some people have decided that DDG's privacy is a must-have while others have decided that (at least for now) it is a nice-to-have. And DDG is of course betting that more and more people will arrive at the conclusion that DDG's privacy "feature" is indeed a must-have. I know I'm on the verge.

<comment><USER24> I switched to DDG, and occasionally I go back to google when DDG doesn't get me the results I am looking for, or when I want to do an image search.

But you're right, for a segment of the market, the DDG privacy is a feature that would be a must have.

</comment></comment></comment><comment><USER25> Good read. Nothing will make me happier than creating a profitable bootstrapped business that sells a product. Once and _if_ I am through that, may be I will work on the next big viral idea which is not a bad thing to think about but not for my first shot at webpreneurship.</comment>


The Left-Handed Kid: On the Chinese Typewriter - <AUTHOR> https://www.lrb.co.uk/v40/n05/jamie-fisher/the-left-handed-kid <comment><USER2> I've just finished reading that book. A very interesting look at the effort to create Chinese Typewriters. Looking forward to the author's second book, which will cover the Chinese characters in the era of computing.

Here's my own review of the book [1].

[1] [https://www.goodreads.com/review/show/2124197693](https://www.goodreads.com/review/show/2124197693)

<comment><USER3> I loved that book! It's interesting with the perfect balance between being academic and still being accessible.

I was just gutted, that it stops in the middle of the 20th century... Yeah, i know, Mulaney will publish a second part, but it was a bit of a let down.</comment></comment>


The strange lab that lured athletes - <AUTHOR> http://sportsillustrated.cnn.com/nfl/news/20130129/the-strange-lab-that-lured-numerous-athletes/

<comment><AUTHOR> I find this economy interesting. Athletes who need to recover quickly and the pseudo science they turn to. It is quite scary for what may come as now they are selling a product claiming to help in brain issues.</comment>


Announcing PilMCU, the Lisp Machine on a Chip - <AUTHOR> http://email@example.com/msg04823.html <comment><USER2> I'd really like to see the full ISA, etc for the chip. A few years ago, I was doing research on building a byte code based vm after working through Peter Michaux "Scheme From Scratch" [http://peter.michaux.ca/articles/scheme-from- scratch-introdu...](http://peter.michaux.ca/articles/scheme-from-scratch- introduction). (I highly recommend running through his code, but do the GC earlier, it's easier to get it from the start than to try to add it.) I couldn't find anything online listing the kinds of instructions you'd want in a lisp chip.

<comment><USER3> There are a few PDFs with the original Lisp Machine ISA, if I am not mistaken.

I will have a look when back at home.

<comment><USER4> If you mean the LMI Lisp machine: [http://bitsavers.trailing- edge.com/pdf/lmi/](http://bitsavers.trailing-edge.com/pdf/lmi/)

There's significantly more for the TI Explorer Lisp machine: [http://bitsavers.trailing- edge.com/pdf/ti/explorer/](http://bitsavers.trailing- edge.com/pdf/ti/explorer/)

And even more for Symbolics: [http://bitsavers.trailing- edge.com/pdf/symbolics/](http://bitsavers.trailing-edge.com/pdf/symbolics/)

Xerox did a Lisp machine line: [http://bitsavers.trailing- edge.com/pdf/xerox/interlisp/](http://bitsavers.trailing- edge.com/pdf/xerox/interlisp/)

There's a working emulator for the MIT CADR: [http://www.unlambda.com/cadr/index.html](http://www.unlambda.com/cadr/index.html)

And you can of course run Genera in an emulator: [http://www.unlambda.com/download/genera/](http://www.unlambda.com/download/genera/)

There's a semi working TI explorer emu: [http://www.unlambda.com/lispm/](http://www.unlambda.com/lispm/)

<comment><USER5> There is an emulator for the Xerox machines.

There is a working TI explorer emulator: [http://www.unlambda.com/index.php?n=Main.Meroko](http://www.unlambda.com/index.php?n=Main.Meroko)

An updated version of the MIT CADR emulator: [http://trac.common-lisp.net/mit- cadr/browser/](http://trac.common-lisp.net/mit-cadr/browser/)

Source trees for the LMI and TI machines are also on the web.

<comment><USER6> > Source trees for the LMI and TI machines are also on the web.

Without usable licenses.

<comment><USER5> I have been told that TI have given formal permission to redistribute their code, I will chase up the person who got hold of it to get it properly labelled.

The LMI stuff is more interesting to me as it is complete, I'm still a fair bit of work away from being able to build it though. I have spoken with RG about it but we didn't discuss the licence.

<comment><USER7> Texas Instruments sold their computer systems division at the start of the 1990s. I've read that that sale did not include the Explorer copyrights.

At the start of 2011 I got in touch with Robby Holland, then the head of TI's patent licensing division. Holland could not find anything about the computer systems division, or anything about the sale, or whether they owned the rights anymore. They did not have any of the project materials on hand. Holland said he would have someone look into the archives. When I tried to get in touch with him a few months later Holland was no longer with Texas Instruments.

<comment><USER6> Texas Instruments closed their Lisp Machine activities in 1990. Nobody picked that up.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Yes, it can be done, but the half-dozen or so LISP machines of the 1980s were not very successful. Price/performance was worse than compiled LISP on common CPUs.

There's no indication of hardware support for garbage collection. It would probably be more useful to have tag bits to support GC than a LISP-oriented instruction set, especially if it allowed concurrent GC.

</comment><comment><USER9> Talking about Lisp machines. I don't know much about these things but I was thinking about them recently when Hewlett Packard announced its so-called _Machine_ ^1. They want to build a new kind of OS for it, but wouldn't a Lisp machine just do?

[1] [http://www8.hp.com/hpnext/posts/discover-day-two-future- now-...](http://www8.hp.com/hpnext/posts/discover-day-two-future-now-machine- hp)

</comment><comment><USER10> where's the kickstarter page? I want a few of those.

</comment><comment><USER11> I've heard a few people suggest that with single core performance stagnating we may see more ASICs. I admit I'm skeptical, but this line of development does seem worth exploring.

</comment><comment><USER12> So this is something where the hardware is executing something that actually looks like LISP sort of like the Reduceron does with Haskell?

</comment><comment><USER13> No links to the actual thing?

<comment><USER2> I wonder if this is related: [http://picolisp.com/wiki/?home](http://picolisp.com/wiki/?home)

<comment><USER14> That's PicoLisp, which you can apt-get install.

This is PilMCU, a hardware design which apparently runs PicoLisp.

Right now, only the design exists, not physical hardware. But you can run the design in a simulator. ...Something, something, verilog. (This is not my area of expertise.)

[Edit: the devs use the Icarus verilog compiler to provide simulations. It's available in Debian repositories.]

</comment></comment></comment><comment><USER15> Do you have a diagram with the design? Something to amuse people even more? ;)

<comment><USER16> It's a Verilog design for a new chip. But it doesn't seem to be open, at least I can't find the source anywhere.

It seems they're interested in launching this is a commercial (or crowd- funded) hardware product.

</comment></comment><comment><USER17> Where is the verilog source?

</comment><comment><USER18> Is verilog design small enough to put on an FPGA?

<comment><USER19> FPGAs are the only context in which I've ever encountered Verilog, so I'm gonna say yes?

<comment><USER20> Verilog is a hardware description language. You describe the hardware connections, sometimes with high level constructs like "for" loops, sometimes by just describing the gates themselves. In a sense, it is similar to HTML describing a web page, but with higher level constructs available like loops. So maybe HTML with a templating engine is a closer analogy.

For simulation, you compile verilog with a software tool into something executable by a VM or natively. This is heavily event based in execution, with events being edge transitions (a signal going from 0->1 or 1->0) occuring at specific times--for most (but not all) cases.

For producing something usable by an FPGA or a foundry for an ASIC, instead of compiling you synthesize. Different tool. Synthesis is the process of taking higher level hardware descriptions and outputing the lower level descriptions usually called a netlist. It's akin to translating C into assembly for example.

Device specific tools can take that netlist and create a bitstream for configuring an FPGA, or the foundry can take that netlist and go through a process called "physical synthesis" which takes the netlist and chooses from the foundry's library the components that will work best for that netlist to operate at speed, figure out where to place them on the die, and insert buffers as needed.

What the GP was asking: is this design small enough to fit in an FPGA. This question is orthogonal to the language used to describe the hardware.

I'd say they answer is yes, depending on the FPGA you choose. Some FPGAs are pretty high capacity these days, and even fast.

</comment></comment></comment><comment><USER21> Can it run emacs?

<comment><USER14> I think emacs needs a POSIX-like environment to run inside. (I think this is roughly correct even though I don't really understand what it means.)</comment></comment>


Show HN: SearchTrack - The home for your research. Product Demo YC W2017. - <AUTHOR> https://www.youtube.com/watch?v=fJ0_7ndPV7o <comment><AUTHOR> My team and I have built SearchTrack. It's the easiest way to save and organize the best links around any topic, collaborate with peers while doing so and make this research available to others looking for it.

This way, we're building a knowledge-sharing community where people compile their research around any topic and share it with the world.

Would appreciate feedback and suggestions. Thanks.</comment>


NTFS read-write GPL kernel driver - <AUTHOR> https://email@example.com/ <comment><USER2> Discussion on reddit:

[https://www.reddit.com/r/linux/comments/i9mdqg/ntfs_readwrit...](https://www.reddit.com/r/linux/comments/i9mdqg/ntfs_readwrite_driver_gpl_implementation_by/)

When exFAT was announced about 5 months ago Paragon Software posted a rant against that.

Rant is posted here:

[https://news.ycombinator.com/item?id=22688058](https://news.ycombinator.com/item?id=22688058)

<comment><USER3> Paragon, purveyors of fine commercial filesystem drivers that corrupt my hard drives? LOL

<comment><USER4> It's easy to laugh at them but I've had corruptions from Linux's existing NTFS implementation too.

<comment><USER3> I've had issues with their Windows HFS+ driver too, shit wiped out a hard drive that I later learned wasn't fully backed up

<comment><USER5> +1 had the same issue with their HFS+ driver, and it did not take that long to encounter it.

</comment></comment></comment></comment></comment><comment><USER6> Everyone (on Reddit at least) seems to be yakking away at whether Paragon Software is good or bad and what other things they've done. I just wanna know if this driver is actually good or bad. Far as I know, a Linux NTFS driver capable of writing has been a long time coming.

<comment><USER7> I've been writing to NTFS for at least 10 years. It was the FUSE driver but it worked fine and I never had any problem other than it seemed to write at half the speed of ext4

<comment><USER8> NTFS is super slow on windows too. You get better performances with ext4 on WSL2 (Windows Subsystem for Linux).

<comment><USER9> how is this possible? isn't wsl 2 literally a virtual machine image on top of your ntfs drive?

<comment><USER10> I think it bypassed windows filesystem filters (antivirus, etc) and thus faster because there is nothing taxing your read/write operations.

<comment><USER11> It least with WSL1 I know that McShield and cyvera/Traps are fully capable of massively interfering with linux performance under windows 10 just like they do normally. Whether they actually effective is not something I know but it seems like they understand ELF files.

</comment></comment></comment></comment></comment></comment><comment><USER12> Beware, it does not seem functional: [https://firstname.lastname@example.com/](https://firstname.lastname@example.com/)

</comment><comment><USER13> Any guesses why Paragon decided to GPL the driver now?

<comment><USER14> They didn't. Samsung wrote this driver and Microsoft proposed its inclusion into the kernel.

<comment><USER15> Are you thinking about exFAT?

<comment><USER14> Yep. My bad.

</comment></comment></comment></comment><comment><USER16> Does anyone really care anymore since ntfs-3g works well enough?

<comment><USER17> I haven't checked in a really long time, but years ago ntfs-3g had a really suboptimal allocation pattern for spinning drives, and optimizations were only present in the commercial version. I don't know if the block allocation strategy was owned by ntfs-3g or if it was using something in-kernel.

If I'm remembering the details right (and that's a _big_ if), large files written sequentially would have their blocks scattered all over the place, making e.g. large uncompressed videos written from Linux too slow to play due to seek times, when if they'd been written sequentially they would have been fine.

<comment><USER16> ntfs-3g is entirely implemented in userspace, and is far more trivial to hack on and test if anyone wanted to improve its file allocation strategies than an in-kernel driver would be.

Years ago when working on a commercial virtualization product I had to support initializing sparsely populated ntfs filesystems where only the filesystem metadata was written out, with allocated file extent ranges logged externally and not actually populated in the sparse ntfs filesystem in a file. I ended up deriving a small library for doing this from the libntfs-3g code, I don't think it took more than a weekend to do, the code was quite sane.

What I recall from that experience was that the file extent allocations on an empty filesystem was perfectly efficient, since you'd have to go out of your way to make it bad.

The real trouble was when the filesystem was used enough for its free space to become fragmented. At that point the allocation strategy was quite poor and you would end up with extremely fragmented large files. IIRC it was really just a naive sequential find first free blocks kind of thing. This aspect is consistent with your memory.

I think if we wanted to improve that aspect of linux ntfs support we're better off working from ntfs-3g.

Not to mention there are some significant down sides to in-kernel filesystem drivers, they are rather problematic when it comes to containers/namespaces and the security story is a total shit show.

</comment></comment></comment><comment><USER18> I don't know about Paragon's NTFS driver for Linux, but their non-free Ext4 driver for Windows had a nasty bug until recently, where it would change file ownership and permissions in the filesystem mounted in read-only mode, rendering the driver unusable. The bug apparently got fixed after numerous complaints on their support forum, but I still use this driver very cautiously and only when I have to.

So I wouldn't trust Paragon's claim of "decades of expertise in commercial file systems development and huge test coverage."

</comment><comment><USER19> I haven't used a dual boot Linux/Windows machine in ages. How popular is this setting these days?

On Linux, my most common scenario of deploying Windows is in small VMs and, when needed, mounting host folders from them. It's true these VMs will probably not run some PC software well, but I have a Windows PC for that. The other way around we have WSL and WSL2 on Windows, again kind of erasing the need to dual boot from the Windows user side.

The money for them here is in support. Maybe they have licensees that bind them to support contracts and merging software into mainline Linux counts as a "best effort" to make others chip in at keeping it functional as the kernel evolves.

<comment><USER12> I haven't dual-booted in ages, but sometimes use NTFS for USB drives or inspecting drive images from Windows machines / VM guests.

<comment><USER20> I used to do this in my college days. Had some work (Unity projects especially) that would run into FAT32 file size limits and NTFS was the one more advanced filesystem that at least somewhat worked on all three OSes, albeit read only on OS X. exFAT support for Linux didn't exist at the time and FAT32, exFAT and NTFS are still the only three OS portable filesystems. Windows cannot handle HFS+/APFS, ext* is not supported on Mac and poorly supported by third party software on Windows, and ZFS/btrfs/XFS/UFS/etc are out of the question for both Windows and Mac.

<comment><USER12> Amusingly, there was actually a historical Mac ZFS effort that was, I think, sidelined in favor of APFL: [https://arstechnica.com/gadgets/2016/06/zfs-the- other-new-ap...](https://arstechnica.com/gadgets/2016/06/zfs-the-other-new- apple-file-system-that-almost-was-until-it-wasnt/)

They had a GPT GUID for it: 6a898cc3-1dd2-11b2-99a6-080020736631.

<comment><USER19> Time Machine looks like it was designed for ZFS and some poor engineer at Apple had to graft it on top of HFS.

</comment></comment></comment></comment></comment><comment><USER21> Perhaps I'm confused, but we not had a NTFS r/W driver working since many years ago ?

<comment><USER22> There is a userspace driver with rw capability, and a kernel driver that only supports ro.

</comment></comment><comment><USER23> Given the initial testing people on the mailing list have done and the issues that have arisen I hope the kernel maintainers so f no and push back until said issues are fixed and code quality rises.

</comment><comment><USER24> Was there something terrible about NTFS-3G?

<comment><USER13> Uhhhmmm... Speed? Stability? NTFS-3G has been a lifesaver for occasional use, but with anything remotely I/O intensive, it starts to crack.

<comment><USER25> Why would you use NTFS on linux for anything other than occasional use?

<comment><USER26> It's the only non-FAT filesystem with good cross-OS compatibiity.

<comment><USER13> This. Now, you've made me wonder if in the near future you could have both Windows and Linux installed on the same _-gasp-_ partition.

<comment><USER27> I had for years. Usung Wubi.

</comment></comment></comment></comment></comment></comment><comment><USER28> Has this been merged already?

<comment><USER29> It was posted yesterday and the replies indicate issues with the patch. So as these things usually go it'll probably take a long time until it's ready for inclusion.

</comment></comment><comment><USER30> Argh, Russian hackers, again!

</comment><comment><USER31> Bad line-wrapping in the first commitmsg does not leave a first good impression.

</comment><comment><USER32> Good news because NTFS-3G seems dead - last update[1] was in 2017. Moreover, quality of the code is far from perfect.

[1] [https://tuxera.com/opensource/ntfs-3g_ntfsprogs-2017.3.23.tg...](https://tuxera.com/opensource/ntfs-3g_ntfsprogs-2017.3.23.tgz)</comment>