Why you should have a programming blog - <AUTHOR> http://samurails.com/lifestyle/why-you-should-have-a-programming-blog/ <comment><USER2> Have to disagree. If you feel like you have something to say, and you feel like a blog is the right format, then sure, have a blog. But this article feels a lot like one those "self-help" articles that instructs you how to improve your life (Work-out in the morning! Eliminate all sugar from your diet! etc.) In the end, this author has the audacity to say that since something works for him then that necessarily means that everyone else should follow that formula.

TL;DR: do you whatever you want to, man.

Oh and disclaimer: I have a blog but I rarely post technical insights. It's more or less a way for me to post my creations and occasionally a cool picture or something. [http://davidvedvick.info](http://davidvedvick.info)

<comment><AUTHOR> Author here, I'm sorry if the article felt that way, that wasn't my intention at all. It's true that it's something that works for me (and probably for other people) but people should obviously do what they want.

I still think everyone should give it a try. Maybe it won't work for them but it could be great for others. They won't know until they've tried ;)

</comment></comment><comment><USER3> I don't particularly like the blog format. The central organizational feature of a blog is a timeline. The idea that something you said is "older" than some other thing is not a particularly useful relationship. Though you would probably like to hear if a particular idea had been updated.

I think something closer to a personal wiki would be better. All links to entries would be versioned so external links always point to the content they intended, but the entry would contain a link to an updated version if it exists.

I think having a medium where people tend to curate their content, rather than forgetting about it and moving on, would lead to higher quality content over time.

<comment><USER4> From reading the comments here, I keep thinking Stackexchange. A lot of people don't use SE this way, but it's possible to ask a question and right away answer it yourself, before submitting even. I do this sometimes when I learn something. Let the upvotes speak and mark the answer as "wiki" to make sure it's future-proof, so others can update outdated/bad info.

</comment></comment><comment><USER5> I don't blog because I have found the internet is too hateful, too quick to judge, and too eager to bring torches and pitchforks. I have a quiet, calm, low-stress life and I can not justify giving up those for free, just to share a few thoughts on the internet to others. My mental health is worth more than whatever notoriety I might gain from internet blogging.

<comment><USER6> I feel the same way, even if its just one comment out of 20. That one comment can really get to you, and it lowers the mood to a generally negative level. I have a lot of respect for people can blog about "new" ideas on a regular basis. It takes a lot of persistence.

<comment><AUTHOR> Author here. I totally get you guys. I'm just the same and I actually got a few hate comments (not on HN) about this article. Of course, it makes me sad and I'm thinking about just stopping everything. Why bother? But the next day, I realize there are people who liked it and that makes me want to try again and get better :)

<comment><USER7> The beauty about comments is that if you own the blog, you can moderate them. Having a discussion about why something is wrong is a good thing as long as it does not turn into "I hate your f-ing post" type of deals.

</comment></comment></comment></comment><comment><USER8> Nitpick: the Einstein quote is predated by Nicolas Boileau in _l 'Art Poétique_ (1674)

        Avant donc que d'écrire, apprenez à penser.
        Selon que notre idée est plus ou moins obscure,
        L'expression la suit, ou moins nette, ou plus pure.
        Ce que l'on conçoit bien s'énonce clairement,
        Et les mots pour le dire arrivent aisément.

Translated on the spot (as a futile attempt from a non-native speaker to preserve the poetry of it):

        Before even to write, one should learn to think.
        As the idea is more or less obscure
        Expression follows, 'ther less clear, or more pure.
        What's best understood is uttered clearly,
        And those words to tell come that easily.

I'm always surprised how much such a work originally applying to art and poetry is also relevant to code. Another example:

        Hâtez-vous lentement, et sans perdre courage,
        Vingt fois sur le métier remettez votre ouvrage,
        Polissez-le sans cesse, et le repolissez,
        Ajoutez quelquefois, et souvent effacez.

T9n:

        Haste yourself slowly, keep up with bravery,
        Your work on the bench put again a twenty,
        Endlessly you polish, and again polish it,
        Occasionally add, much often delete.

<comment><AUTHOR> Thanks for the reference and nice job on the translation. Pas aussi poétique, mais presque ;)

</comment></comment><comment><USER9> Best part about having a programming blog: writing a quick post about how you fixed some error, then a year or two later, googling for the error and finding your own blog post on it.

<comment><AUTHOR> So true!

</comment></comment><comment><USER10> Disagree. I have nothing to say that I don't already say via twitter, on github (repos, gists), comments on HN/reddit/blogs.

I already have all these tools at my disposal to impart my knowledge. I find blogs too self-indulgent.

<comment><USER11> If someone googles a question about programming, the top results are probably not going to be your github repos, your twitter tweets, or your comments on various sites. The top result will probably be Stack Overflow (for small questions) or a blog post (for the more comprehensive ones).

It seems to me that if "helping others" is the goal, your method of sharing is much more self-indulgent than a blog.

Also who imparts actual programming knowledge with their tweets? You find a way past the character limit? That must be one of the sillier things I've heard on here in a long time.

<comment><USER10> Your imagination is failing you.

Twitter is an excellent medium to share links to tools with others who may find them useful. An example from today: [https://twitter.com/izolate/status/596262146553597952](https://twitter.com/izolate/status/596262146553597952)

<comment><USER12> To play devil's advocate, this is little more than a recipe to be followed by rote. I didn't learn anything in that gist other than a clever way to find the version number. Why does it get installed into my home directory... is that suitable for a production server? And so on...

</comment></comment></comment></comment><comment><USER6> I stopped blogging because my skin isn't thick enough. I wrote a post a few times, and several people complemented it each time. But then a few very angry people shredded my ideas to pieces. Emotionally it really bummed me out, so I deleted it.

</comment><comment><USER13> Please don't maintain a blog just to improve your job prospects.

Not that I think it won't work. But it's an arms race, it only helps you until everyone else starts doing it as well. The cost of playing along is that everybody who wants a job needs to spend more time doing something they don't care about, and "has a blog" becomes less valuable as a signal to employers.

...besides, I already have a blog, and I don't want to lose that advantage.

</comment><comment><USER14> I agree with the idea that one effective way of learning about something you don't understand is to write an article about it. The act of writing it forces you to clarify your understanding to the point that you can explain it to someone else.

Here are some articles I wrote where at least 50% of the article consists of things I _didn 't_ understand prior to writing the article:

[http://blog.reverberate.org/2014/09/what-every-computer- prog...](http://blog.reverberate.org/2014/09/what-every-computer-programmer- should.html)

[http://blog.reverberate.org/2014/02/react- demystified.html](http://blog.reverberate.org/2014/02/react-demystified.html)

[http://blog.reverberate.org/2013/05/deep-wizardry-stack- unwi...](http://blog.reverberate.org/2013/05/deep-wizardry-stack- unwinding.html)

</comment><comment><USER15> If everybody, regardless of their skill level, is writing a blog, we'll end up with a lot of new tutorials how to build yet another insecure fileupload or bypassable .htaccess files.

<comment><USER16> One of my biggest pet peeves is these blogs that do things like "Getting started with Express.js" and they cover HALF of what the Getting Started docs on Express' website offers.

</comment></comment><comment><USER17> Suppose there are 10M programmers worldwide. Should there be 10M programming blogs?

<comment><USER11> Why not? If we were manually filtering all these blogs, then of course not.

But we're not. We have google. It doesn't matter how many blogs there are as long as you are not re-hashing solved problems that have been explained thoroughly.

<comment><USER18> That, however, is the exact problem; most blogs _are_ simple rehashes of the same things that have already been said. One could argue that this very article is a prime example of why someone shouldn't have (or rather, doesn't need) a blog. It isn't an attack against the quality of the writing itself, but the reality is that there's little of value because everything there has already been said countless times before.

<comment><USER19> Depends on the purpose of the blog. I maintain a blog for two reasons. One, writing about problems and their solutions as if I'm explaining them to someone else is a great way to solidify the concepts in one's mind (similar to rubber-duck debugging[1]). Two, having an active blog is a signal to potential employers that you're active in the ecosystem. Even if someone is rehashing the same old topics in their posts, seeing how they think and how well they write can be a useful indicator.

[1][http://en.wikipedia.org/wiki/Rubber_duck_debugging](http://en.wikipedia.org/wiki/Rubber_duck_debugging)

<comment><USER20> Those are both reasons why it is in your interest to write a blog. They are not reasons why someone might want to actually read your blog (short of prospective employers).

It's interesting that this is the focus.

</comment></comment></comment></comment></comment><comment><USER21> Why should I contribute to an increasing entropy?

<comment><USER22> Why you just did?

<comment><USER21> Precisely. I'm contributing anyway (by commenting, breathing, farting), so why should I do it even more?

<comment><USER23> Blogging is far preferable to commenting in a public forum because I don't have to see it unless I visit.

To use your own analogy: commenting on Reddit/Youtube/HN is like farting in public. Blogging is like farting in your own home.

<comment><USER21> There is a significant nuance: blogs are most likely indexed by google. Comments are invisible. So blogs are generating much more noise, obstructing an access to information much more dramatically than even a 5000-strong comments thread could ever do.

<comment><USER23> Comments are most certainly indexed by Google. They are not usually ranked highly. And crappy blogs are also indexed and not ranked very highly.

Both generate comparable "noise", except that there are a lot more comments.

<comment><USER21> Then why, when I search for a solution to a specific problem (e.g., googling for an error message, etc.), I most likely land in some blog full of ramblings with no solution whatsoever? And often the real solution is hidden somewhere in comments on SE, never listed in any sane google query.

</comment></comment></comment></comment></comment></comment></comment><comment><USER24> I'm not sure it makes sense for every person to create their own blog, especially if they're not going to post frequently. Few people will end up reading anything they write. Instead it may make sense to contribute content to another website. This could either be open-source / CC content, or they could be paid for their articles.

(Perhaps they could also embed programming challenges in their articles so people can try out what the article discusses...)

</comment><comment><USER25> You should change the title to: Why you should have a fast programming blog. Making reader waiting for the content fully loaded on your blog is really annoying :)

</comment><comment><USER26> I do have a blog, but I only post when I have something to say to the subject - something that wasn't already said elsewhere. Otherwise I'd consider it just waste of space...

</comment><comment><USER27> I agree, but only if you have something interesting or useful to say. And I agree with the comments saying that the blog isn't the best format to convey information since it's sorted by time. I'm in the process of redesigning my blog to sort by subject matter, and separate small code snippets from more in- depth posts.

On another note, I wish more programmers wrote about topics other than just technical ones.

</comment><comment><USER28> So what happens if you have a blog but can't find the time to write to the level of detail you'd like? I haven't had a chance to write a thing since my aborted series on JPEG encoding, two years ago; there are multiple abandoned series lurking in there, waiting for me...

</comment><comment><USER29> A rails blog doesn't seem like a great soapbox to stand on here.

</comment><comment><USER30> Not persuaded.

<comment><USER31> Write a blog article about why!

</comment></comment><comment><USER32> You could just have your own technical diary so to speak if you don't want to share it with the world.</comment>


What does a hacker do after an accident which puts him to bed rest? - <AUTHOR> http://dy-verse.blogspot.com/2009/02/i-had-accident-yesterday-and-am-totally.html

<comment><USER2> I bought an Eee PC when I had to stay horizontal last month with a broken leg. That's a fun little computer to keep on your lap.

<comment><USER3> Don't get me wrong, I love my little netbook. But to me that's the worst solution for someone on bed rest. The problem is you run into something that requires the full computer (usually due to screen size like a video tutorial that's too tall to watch on the Eee PC). So now you're supposed to stay in bed but your computer's just 4 ft away and you really want to watch this video (or whatever).

Maybe others here just have more self control but for me having the netbook is just a recipe for disaster.

<comment><USER2> Well, if it really, really hurts when you put your foot anywhere below your heart, then its not a problem to stay laying down.

</comment></comment></comment><comment><USER4> Ouch. If he got off with just 'bed rest' he is in very very good shape. Most motorcycle -> bus 'close encounters of the third kind' do not end that well.

<comment><AUTHOR> That is true. He was really lucky.

So many replays yet I cannot believe the way the bus dodges its way in busy traffic. But knowing that he is fine and doing some fun things it made me ROFL.

</comment></comment><comment><USER5> I love the animation. Glad you are ok, if you are still laid up in bed, check out itunes U for some lectures on something interesting an give your eyes/body a rest.

</comment><comment><USER6> That's exactly why I don't ride bikes on public roads or understand those who do. Clowns who can't drive can wreck you without getting more than a minor door ding in their SUV (or in this case, bus). I just don't trust strangers with my life.

</comment><comment><USER7> When I broke my right hand last year I couldn't really use my hands for any sort of typing and spent a lot of time doing code reviews and reading up on libraries/books/languages. Pretty much anything that didn't involve using my hands much. I also took a ton of notes and after the cast was off I then put them all together in a fun blog entry.

<http://benjamin-meyer.blogspot.com/2008/07/cast-off.html>

</comment><comment><USER8> Catch up on his reading list?

<comment><USER3> Personally I like to use the time to read all those theoretical books that I bought and intended to read but just never got around to (you know, the ones filled with pseudo code).

Either that or read up on non-programming related programming texts like Extreme Programming books. Basically the idea is to read something that will keep your interest and keep the "programming juices flowing" but at the same time won't inspire you to get up and try anything on the computer (which would violate your bed rest)

</comment></comment><comment><AUTHOR> Oh, follow the link. He hacked a video depiction of how the accident happened.

Its quite hilarious how the big bus dodges on green and throws him on the pavement. Happens only in Bangalore.

</comment><comment><USER9> Get a "bed" computer and play world of warcraft.

</comment><comment><USER10> While(stdin) continue hacking...?</comment>


MVVM and Databinding for Unity3d - <AUTHOR> https://github.com/vovgou/loxodon-framework <comment><AUTHOR> LoxodonFramework is a lightweight MVVM (Model-View-ViewModel) framework. It is specially designed for Unity3D game development. It refers to the MVVM design of WPF and Android. It provides data binding and localization of views and view models , A simple service container, configuration file component, thread tool component, application context and player context, asynchronous thread and coroutine task components, and other basic components. It also provides a framework for UI views. All code is designed based on the idea of object- oriented and interface-oriented, and almost all functions can be customized. In addition, performance optimization has been performed in the data binding part. On the platform that supports JIT, the delegation method is used for binding. On the platforms that do not support JIT, reflection is used by default, but it can be optimized by injecting the delegate function!</comment>


Unix : 'cal 1752' - look at september - <AUTHOR>

Pretty cool! eh ? <comment><USER2> Background: <http://en.wikipedia.org/wiki/Old_Style_and_New_Style_dates>

</comment><comment><USER3> Why does cal max out at 5875706?</comment>


Command-line tools can be faster than a Hadoop cluster (2014) - <AUTHOR> http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html <comment><USER2> I feel like every time something like this comes up people completely skip over the benefit of having as much of your data processing jobs in one ecosystem as possible.

Many of our jobs operate on low TBs and growing but even if the data for a job is super small I'll write it in Hadoop (Spark these days) so that the build, deployment, and schedluing of the job is handled for free by our curent system.

Sure spending more time listing files on S3 at startup than running the job is a waste but far less than the man hours to build and maintain a custom data transformation.

The main benefit of these tools is not the scale or processing speed though. The main benefits are the fault tolerance, failure recovery, elasticity, and the massive ecosystem of aggregations, data types and external integrations provided by the community.

<comment><USER3> Having worked at a large company and extensively used their Hadoop Cluster, I could not agree more with you.

The author of the blogpost/article, completely misses the point. The goal with Hadoop is not minimizing the lower bound on time taken to finish the job but rather maximizing disk read throughput while supporting fault tolerance, failure recovery, elasticity, and the massive ecosystem of aggregations, data types and external integrations as you noted. Hadoop has enabled Hive, Presto and Spark.

The author completely forgets that the data needs to transferred in from some network storage and the results need to be written back! For any non-trivial organization ( > 5 users), you cannot expect all of them to SSH into a single machine. It would be an instant nightmare. This article is essentially saying "I can directly write to a file in a local file system faster than to a database cluster", hence the entire DB ecosystem is hyped!

Finally Hadoop is not a monolithic piece of software but an ecosystem of tools and storage engine. E.g. consider Presto, software developers at Facebook realized the exact problem outlined in the blogpost but instead of hacking bash scripts and command line tools, they built Presto. Which essentially performs similar functions on top of HDFS. Because of the way it works Presto is actually faster than "command line" tools suggested in this post.

[https://prestodb.io/](https://prestodb.io/)

<comment><USER4> But, as I pointed out in another comment, what about systems like Manta, which make transitioning from this sort of script to a full-on mapreduce cluster trivial?

Mind, I don't know the performance metrics for Manta vs Hadoop, but it's something to consider...

<comment><USER3> From my experience organizations have adopted, Hive/Presto/Spark on top of Hadoop. Which actually solves a whole bunch of problems that "script" approach would not. With several added benefits. Executing scripts (cat, grep, uniq, sort) do not provide similar, benefits, while they might be faster. A dedicated solution such as Presto by Facebook will provide similar if not even faster results.

[https://prestodb.io/](https://prestodb.io/)

<comment><USER4> Ah, so it doesn't solve data storage, and runs SQL queries, which are less capable than UNIX commmands. If your data's stuck inside 15 SQL DBs, than that'd make sense, but a lot of data is just stored in flat files. And you know what's really good at analyzing flat files? Unix commands.

<comment><USER3> Did you even read it? Presto reads directly from HDFS, which is as close to distributed "flat files" as you can get. As far as "SQL being less capable than UNIX commands", you have got to be kidding me. SQL allows type checking, conversion, joins all of which are difficult if not impossible with grep | uniq | sort etc.

<comment><USER4> I read it.

>Presto allows querying data where it lives, including Hive, Cassandra, relational databases or even proprietary data stores. A single Presto query can combine data from multiple sources, allowing for analytics across your entire organization.

That doesn't sound like HDFS to me. I mean, I assume it _can_ read from HDFS, but Presto is backend agnostic. You could probably write code to run it on Manta. That would be neat for people who like Presto, I guess.

Type checking and conversions, no, and table joins only matter when you're handling relational data.

Also, how many formats can Presto handle? Unix utilities can handle just about any tabular data, and you can run them against non-tabular data in a pinch (although nobody reccomends it). I doubt Presto is _that_ versitile.

<comment><USER5> Hive operates on top of HDFS.

Presto absolutley runs directly on HDFS.

<comment><USER4> Huh. Well then, I don't understand HDFS, or Facebook needs to fix Presto's front page. Both are reasonably likely.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> Most "big data" problems are really "small data" by the standards of modern hardware:

* Desktop PCs with up to 6TB of RAM and many dozens of cores have been available for over a year.[1]

* Hard drives with 100TB capacity in a 3.5-inch form factor were recently announced.[2]

 _CORRECTION: THE FIGURE IS 60TB, NOT 100TB. See MagnumOpus 's comment below. In a haste, I searched Google and mistakengly linked to an April Fool's story. Now I feel like a fool, of course._ Still, the point is valid.

* Four Nvidia Titan X GPUs can give you up to 44 Teraflops of 32-bit FP computing power in a single desktop.[3]

Despite this, the number of people who have unnecessarily spent money and/or complicated their lives with tools like Hadoop is pretty large, particularly in "enterprise" environments. A lot of "big data" problems can be handled by a single souped-up machine that fits under your desk.

[1] [http://www.alphr.com/news/enterprise/387196/intel- xeon-e7-v2...](http://www.alphr.com/news/enterprise/387196/intel- xeon-e7-v2-servers-support-6tb-of-ram)

[2] [http://www.storagenewsletter.com/rubriques/hard-disk- drives/...](http://www.storagenewsletter.com/rubriques/hard-disk- drives/incredible-record-of-100tb-into-3-5-inch-hdd-with-hamrhelium/)

[3] [https://news.ycombinator.com/item?id=12141334](https://news.ycombinator.com/item?id=12141334)

<comment><USER7> > Hard drives with 100TB capacity in a 3.5-inch form factor were recently > announced

That is an April Fools story.

(Of course you can still get a Synology DS2411+/DX1211 24-bay NAS combo for a few thousand bucks, but it will take up a lot of space under your desk and keep your legs toasty...)

<comment><USER8> Earlier this year, Seagate were showing off their 60 TB SSD[1] for release next year.

So 100 TB in a single drive isn't too far off.

EDIT: Toshiba is teasing a 100 TB SSD concept, potentially for 2018 [2]

[1] 11th August - [http://arstechnica.co.uk/gadgets/2016/08/seagate- unveils-60t...](http://arstechnica.co.uk/gadgets/2016/08/seagate-unveils-60tb- ssd-the-worlds-largest-hard-drive/)

[2] 10th August - [http://www.theregister.co.uk/2016/08/10/toshiba_100tb_qlc_ss...](http://www.theregister.co.uk/2016/08/10/toshiba_100tb_qlc_ssd/)

<comment><USER9> 60TB at 500Mb/s transfer will take +1 day to read the data. This is the problem of drinking the ocean through a straw. Even with SSD transfer rates, is still a problem at scale. Clusters give you no only capacity, but also multiplication factor for transfer rates.

<comment><USER10> Just use 24 of them interleaved/stripped and it will take just one hour for loading the data.

<comment><USER9> But then you need small disks (eg. 2TB). My point is that huge capacity drives are not appropriate in _compute_ environments, as Hadoop is. They're more for cold storage.

</comment></comment></comment></comment></comment></comment><comment><USER11> This idea was the subject of a paper at a major systems conference. The paper is called "Scalability! But at what cost?" \- It goes well beyond this simple example above to explore how most major systems papers produce results that can be beaten by a single laptop. Here's the paper and the blog post describing it.

[http://www.frankmcsherry.org/assets/COST.pdf](http://www.frankmcsherry.org/assets/COST.pdf)

[http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...](http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html)

<comment><USER3> The paper is frankly stupid and a great example of difference between practice and academia. it looks good because they are using a snapshot of Twitter network from 2010. In reality the work flow is complex, e.g. the follower graph gets updates every hour. 10 different teams have their different requirements as to how to set up the graph and computations. These computations need to be run at different (hourly, weekly, daily) granularity. 100 downstream jobs are also dependent on them and need to start as soon as previous job finishes. The output of the jobs gets imported/indexed in database which is then pushed to production systems and/or used by analysts who might update and retry/rerun computations. Unlike a bunch of out of touch researchers the key concern isn't how "fast" calculations finish, but several others such as ability to reuse, fault tolerance, multi user support etc.

I can outrun a Boeing 777 on my bike in a 3 meter race but no would care. The single laptop example is essentially that.

<comment><USER12> > The paper is frankly stupid and a great example of difference between > practice and academia. it looks good because they are using a snapshot of > Twitter network from 2010.

We used these data and workloads because that was what GraphX used. If you take the graphs any bigger, Spark and GraphX at least couldn't handle it and just failed. They've probably gotten better in the meantime, so take that with a grain of salt.

> Unlike a bunch of out of touch researchers the key concern isn't how "fast" > calculations finish, but several others such as ability to reuse, fault > tolerance, multi user support etc.

The paper says these exact things. You have to keep reading, and it's hard I know, but for example the last paragraph of section 5 says pretty much exactly this.

And, if you read the paper even more carefully, it is pretty clearly not about whether you should use these systems or not, but how you should not evaluate them (i.e. only on tasks at a scale that a laptop could do better).

<comment><USER3> "The paper says these exact things. You have to keep reading, and it's hard I know, but for example the last paragraph of section 5 says pretty much exactly this."

Thanks, that addresses my concern. I take back my comment.

But why stop at Rust implementation, there are vendors optimizing it down to FPGA. This sort of comparison is hardly meaningful.

<comment><USER12> The only point of the paper is that the previous publications sold their systems primarily on performance, but their performance arguments had gaping holes.

The C# and Rust implementations have the property that they are easy and you don't need to have any specific skills to write a for-loop the way we did (the only "tricks" we used were large pages and unbuffered io in C#, and mmap in Rust).

The point is absolutely not that these are the final (or any) word in these sorts of computations; if you really care about performance, use FPGAs, ASICs, whatever. There will always be someone else doing it better than you, but we thought it would be nice if that person wasn't a CS 101 undergraduate typing in what would literally be the very first thing they thought of.

</comment></comment></comment></comment></comment><comment><USER13> > 1.75GB

> 3.46GB

These will fit in memory on modest hardware. No reason to use Hadoop.

The title could be: "Using tools suitable for a problem can be 235x faster than tools unsuitable for the problem"

<comment><USER14> This is exactly the point he was making.

People have a desire to use the 'big' tools instead of trying to solve the real problem.

People both underestimate the power of their desktop machine and the 'old' tools and overestimate the size of their task.

<comment><USER15> Yeah, someone was telling me they need big data for a million rows. I laughed and said SQLite handles that...

<comment><USER16> I would not want to be the one on-call for a million row SQLite database!

<comment><USER17> I would. That's a pager that's never going off.

<comment><USER4> Yeah. No significant software is bug-free, but if there was one, SQLite would be a good candidate.

<comment><USER18> Especially having looked at how thorough their test suite is.

<comment><USER4> That's what I'm talking about. What other database tests against power outages and hardware failure?

<comment><USER15> I'm sure many do...

<comment><USER4> Do they do it for every release? What about OOM error testing? IO error tests? fuzz tests? UB tests? Fuzz tests? Malformed DB tests? Valgrind analysis? Memory leak checking? Regression tests?

SQLite does all of that and more with every release ([https://sqlite.org/testing.html](https://sqlite.org/testing.html)). There's a reason I consider their test coverage impressive...

<comment><USER15> I don't know. I remember reading that other DBs use SQLite's test suite

<comment><USER4> Huh. Well, I learned something new today.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER4> ...And when you _do_ have 140 TB of chess data, you can move to Manta, and you get to keep your data processing pipeline almost exactly the same. Upwards scalability!

I don't know how the performance would stack against Hadoop, but it'd work.

<comment><USER19> Actually, just posted essentially the same thing, before reading your comment. I'm wondering as well how the performance would/will scale. It likely depends on how the data is scattered / replicated, but presumable they've worked out decent schedulers for the system. If not, it is open source! Lovin it.

<comment><USER4> Well, all the code running against the data would already have the paralellization advantages of a shell script, as described in this article. It would additionally probably be running accross multiple nodes, meaning that the IO speeds increase the number of records that can be processe)d simultaneously. The disadvantage is that that data has to be streamed over a network to the reducer node, which could add a good chunk of latency, depending on how fast that is (if you can do some reduction during the map, it would help, but it's possible that Manta spawns one process and virtualized nods per object (and indeed, this is likely), meaning this is impossible), and how many virtual nodes are running on the same physical hardware (but then you're running into the same boundaries you hit on a laptop, just on a much beefier system), as the network latency is near zero if the reducer and the mapper nodes are on the same physical system.

But if you're processing terrabytes, the network latency is probably barely factoring into your considerations, given how much time you're saving by processing data in parallel in the first place.

<comment><USER19> That's pretty similar to my thinking on the performance. Though your point about the combination of shell script streaming and parallelization is a good way to express it.

The real benefit of this system would be compared to "traditional" (modern?) big data tools like spark, then the network latency cost of the reduce phases should be comparable. Though since manta localizes the compute to the data, there should be an overal order of magnitude less network transfer which should significantly reduce the of of manta based solutions compared to spark/s3 solutions.

In theory at least, it'd be great to test this on equivalent hardware, or at least equivalent;y priced hardware. But that would require a nice test data set which I don't have the resources to setup. Any suggestions on data code that could test the above assumptions would be handy ( _ahem_ HN peeps got anything?).

_Edits: grammar_

<comment><USER4> I got nothing on data code. You could try running a comparable S3/EC2 against Manta on Joyent, but that would be relatively expensive, and I have no idea of the differences between Amazon and Joyent's datacenter layout, so such a test would not be optimal, although it would test each in its most common use case.

It's also worth mentioning in performance analysis. that Manta is backed by ZFS and Zones, so it has the performance characteristics of those.

</comment></comment></comment></comment></comment><comment><USER20> Reminds me of one of my all time favorite comments about Bane's Rule:

[https://news.ycombinator.com/item?id=8902739](https://news.ycombinator.com/item?id=8902739)

<comment><USER16> And yet a fourth dimension to the problem: time/difficulty multiplier.

        Going   Multiplier
        -----   </comment></comment><comment>---
        High    1x
        Wide    4x
        Deep    8x

Make the best decision given your current circumstances and engineering resources.

<comment><USER21> Bane's "deep" approach saved tens of millions of dollars in one example, months of time in another. In all cases it was many times easier for the team to keep up, day after day, year after year.

Bane advocates good old-fashioned refactoring. With humble tools like Perl, SQL, and an old desktop, he bested new, fancy, expensive products. Bane deserves the salary of a CEO, or at least a vice president, for the good he has brought to his company.

I think ego leads us to make choices that appeal in the short run but are bad in the long run. Which is more impressive sounding: that you bought a distributed network of a thousand of the newest, shiniest machines, running the latest version of DataLaserSharkAttack; or that you cobbled together some Perl, SQL, and shell one-liners on a four-year-old PC?

Also good-old fashioned hard work is painful. It is a good kind of pain, like working out your body, rather than a bad kind of pain, like accidentally cutting yourself. But it is just good old-fashioned, humble, hard work to sit down, work through the details, and come up with a better plan.

Before that, it is even more humble, hard work, to learn the things that Bane had learned. Not just anybody could have done what he did. First you have to learn the ins and outs of Perl, SQL, and all the little shell commands, and all their little options. He knows about a lot of different programming problems, like what a "semantic graph" is (I can't say I do), what an "adjacency matrix" is (nope), whether something is an O(N^2) problem or an O(k+n^2) problem (I know I've seen that notation before).

</comment></comment><comment><USER22> Arguable if you can keep everything on one box it will almost always be faster (and cheaper!) than any soft of distributed system. That said, scalability is generally more important than speed because once a task can be distributed you can add performance by adding hardware. As well, depending on your use case you can often get fault tolerance "for free".

<comment><USER23> There is nothing preventing distributed systems to be faster than one box for this kind of thing. But they don't always bother to pursue efficiency on that level, because things are very different once you have a lot of boxes and something that used to look important for a couple of boxes doesn't anymore.

<comment><USER24> Yes, there is, you have a lot of overhead in any case for the same tools.

<comment><USER23> You don't have the same tools. You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system.

EDIT: For something that beats a single box easily I envision an interpreter with JIT running on each node in a distributed system and on the same process that stores data, having pretty much no overhead to access and process it.

<comment><USER4> >You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system.

Yeah, but Manta's mapreduce does something close, and it seems to work okay.

</comment></comment></comment></comment></comment><comment><USER25> This seems to be the Manta [0] way. Letting you run your beloved Unix command pipeline on your Object Store files.

[0] [https://www.joyent.com/manta](https://www.joyent.com/manta) But the youtube videos with Bryan Cantrill are even better at explaining.

</comment><comment><USER26> If you like to do data analyses in bash, you might also enjoy bigbash[1]. This tool generates quite performant bash one-liners from SQL Select statements that easily crunch GB of csv data.

Disclaimer: I am the main author.

[1] [http://bigbash.it](http://bigbash.it)

<comment><USER4> That's pretty cool.

Do you think you can get it to support Manta? I think a lot of people in that ecosystem could benefit from it if you could. I'd help, but I don't really know Java all that well :-(.

</comment></comment><comment><USER27> It's all about picking the right tool for the job. I think shell scripting is a great prototyping tool and often a good place to start. As the problem gets more complex and bigger, eventually it will warrant a full scale development.

<comment><USER28> I think people overlook the fact that the author made an even more strong point by using shell scripting, which is relatively inefficient compared to using a compiled language. I guess it would hit the I/O cap without even going parallel.

</comment></comment><comment><USER29> Date on article: Sat 25 January 2014

I am not a Big Data expert, but does that change any of the comments below with reference to large datasets and memory available?

I use J and Jd for fun with great speed on my meager datasets, but others have used it on billion row queries [1]. Along with q/kdb+, it was faster than Spark/Shark last I checked, however, I see Spark has made some advances recently I have not checked into.

J is interpreted and can be run from the console, from a Qt interface/IDE, or in a browser with JHS.

[1] [http://www.jsoftware.com/jdhelp/overview.html](http://www.jsoftware.com/jdhelp/overview.html)

<comment><USER30> There isn't exactly a direct relationship between the size of the data set and the amount of memory required to process it. It depends on the specific reporting you are doing.

In the case of this article, the output is 4 numbers:

      games, white, black, draw

Processing 10 items takes the same amount of memory as processing 10 billion items.

If the data set in this case was 50TB instead of a few GB, it would benefit from running the processing pipeline across many machines to increase the IO performance. You could still process everything on a single machine, it would just take longer.

Some other examples of large data sets+reports that don't require a large amount of memory to process:

      * input: petabytes of web log data. output: count by day by country code
      * input: petabytes of web crawl data. output: html tags and their frequencies
      * input: petabytes of network flow data. output: inbound connection attempts by port

Reports that require no grouping (like this chess example) or group things into buckets with a defined size (ports that are in a range of 1-65535) are easy to process on a single machine with simple data structures.

Now, as soon as you start reporting over more dimensions things become harder to process on a single machine, or at least, harder to process using simple data structures.

      * input: petabytes of web crawl data. output: page rank
      * input: petabytes of network flow data. output: count of connection attempts by source address and destination port

I kinda forget what point I was trying to make.. I guess.. Big data != Big report.

I generated a report the other day from a few TB of log data, but the report was basically

      for day in /data/* ; do #YYYY-MM-DD
        echo $day $(zcat $day/*.gz | fgrep -c interestingValue)
      done

</comment></comment><comment><USER16> There's a lot of operational benefits to running on Hadoop/yarn as well. You get operational benefits from node resiliency (host went down? Run the application over there). You also get the Hadoop filesystem which conveniently stores your data in S3 and distributed HDFS.

These systems were designed by people who probably managed difficult etl pipelines that were nothing but what the author suggests: simplified shell scripts using UNIX pipes.

Besides going up against Hadoop MR is easy. I'd like to see you compete against something like Facebook's presto or spark which are optimized for network and memory.

</comment><comment><USER31> What is the point? Who would want to use Hadoop for something below 10GB? Hadoop is not good at doing what it is not designed for? How useful.

<comment><USER32> Kind of depends on what the 10 GB is. For example, on my project, we started on files that were about 10 GB a day. The old system took 9 hours to enhance the data (add columns from other sources based on simple joins). So we did it with Hadoop on two Solaris boxes (18 virtual cores between them). Same data; 45 minutes. But wait there's more.

We then created a two fraud models that took that 10+ GB file (enhancement added about 10%) and executed within about 30 minutes a piece. But concurrently. All on Hadoop. All on arguably terrible hardware. Folks at Twitter and Facebook had never though about using Solaris.

We've continued this pattern. We've switched tooling from Pig to Cascading because Cascading works in the small (your PC without a cluster) and in the large. It's testable with JUnit in a timely manner (looking at you PigUnit). Now we have some 70 fraud models chewing over anywhere from that 10+ GB daily file set to 3 TB. All this in our little 50 node cluster. All within about 14 hours. Total processed data is about 50 TB a day.

As pointed out earlier, Hadoop provides an efficient, scalable, easy distributed application development platform. Cascading makes life very Unix- like (pipes and filters and aggregators). This coupled with a fully async eventing pipe line for workflows built on RabbitMQ makes for an infinitely expandable fraud detection system.

Since all processors communicate only through events and HDFS, we add new fraud models without necessarily dropping the entire system. New models may arrive daily, conform to a set structure, and are literally self-installed from a zip file within about 1 minute.

We used the same event + Hadoop architecture to add claim line edits. These are different from fraud models in that fraud models calculate multiple complex attributes then apply a heuristic to the claim lines. Edits look at a smaller operation scope. But in cascading this is pipe from HDFS -> filter for interesting claim lines -> buffer for denials -> pipe to HDFS output location.

Simple, scalable, logical, improvable, testable. I've seen all of these. As the community comes out with new tools, we get more options. My team is working on machine learning and graphs. Mahout and Giraph. Hard to do all of this easily with a home grown data processing system.

As always, research your needs. Don't get caught up in the hype of a new trend. Don't be closed minded either.

</comment></comment><comment><USER33> i agree that scalable infrastructure is needed to manage a production pipeline, as others have explained well.

i found this article was a useful reminder, because sometimes a job doesnt require a fully grown infrastructure. i commonly get these requests that dont overlap with existing infrastructure and wont need any followup. in that particular case a hadoop cluster, heck even loading into a pg db would be wasted effort.

but i wouldnt want to manage our clickstream analytics pipeline with shell scripts and cron jobs.

is there any lightweight tooling out there that can schedule/run basic pipeline jobs in a shell environment?

<comment><USER34> Airflow? It might not be what you consider lightweight, though.

</comment></comment><comment><USER17> In my experience you can classify people into two herds: those who, when faced with a problem, solve it directly; and those who, faced with the same problem, try to fit it to the tools they want to use. I like to think this is a maturity question, but I can't think I've actually seen someone make the transition from the latter to the former type.

<comment><USER35> > those who, when faced with a problem, solve it directly

I think you mean "use the tools they already know".

> those who, faced with the same problem, try to fit it to the tools they want > to use

I think you mean "use the correct tools for the job".

> I like to think this is a maturity question

It is, but the direction of maturity is from the first case to the second.

</comment></comment><comment><USER36> This is just plain click bait.

Obviously if a dataset is small enough to possibly fit in memory, it will be much faster to run on a single computer.

<comment><USER37> I guess the author should have called it out more explicitly for some, but I think that's the point.

I've seen the testimony dozens of times on HN, and I've heard it from a friend who manages Hadoop at a bank, and I've seen it with people building scaled ELK stacks for log analysis: People are too eager to scale out when things can be done locally, given moderate datasets.

<comment><USER38> Though sometimes hadoop makes sense even if local computation is faster. For example you might just be using hadoop for data replication.

<comment><USER39> > For example you might just be using hadoop for data replication.

Good point. The reason someone who holds data for 7+ years uses hadoop is not because it is faster.

The processing aspect of the system is only tangential to the failure tolerance when you consider the age of the data set.

HDFS does waste a significant amount of IO merely reading through cold data and validating the checksums, so that it safe against HDD bit-rot (dfs.datanode.scan.period.hours).

The general argument about failure tolerance is off-site backups, but the backups tend to have availability problems (i.e machine failed, the Postgres backup restore takes 7 hours).

The system is built for constant failure of hardware, connectivity and in some parts, the software itself (java over C++) - because those are unavoidable concerns for a distributed multi-machine system.

The requirement that it be fast takes a backseat to the availability, reliability and scalability - an unreliable, but fast system is only useful for a researcher digging through data at random, not a daily data pipeline where failures cascade up.

</comment></comment></comment></comment><comment><USER40> The Hadoop article linked is available at [https://web.archive.org/web/20140119221101/http://tomhayden3...](https://web.archive.org/web/20140119221101/http://tomhayden3.com/2013/12/27/chess- mr-job/)?

</comment><comment><USER41> Ok, now do it for >2tb.

Our prod hadoop dataset is now > 130tb, try that!

<comment><USER42> What kind of data do you have? Is this mostly text, or more like compressed time series?

<comment><USER43> We're analyzing genome sequence data on that scale: [https://github.com/hail- is/hail](https://github.com/hail-is/hail)

</comment></comment></comment><comment><USER44> But what would happen if those exact same command-line tools were used inside a Hadoop node? What would be the optimum number of processors then?

<comment><USER14> you would just be adding management-overhead.

More software != more efficient software.

<comment><USER44> But faster because parallel.

</comment></comment></comment><comment><USER45> Yes, you do not need a 100 node cluster to crunch 1.75GB of data. I can do that on my phone. What's the author's point?!

<comment><USER46> That hadoop's reputation of being worth the hassle and complexity that managing it entails is undeserved.

</comment></comment><comment><USER43> Is there a place to download the database that isn't a self-extracting executable? (Seriously?)

</comment><comment><USER47> Not just faster to run but also much faster to write

</comment><comment><USER48> >cat | grep

why

</comment><comment><USER49> (2014)

Here's the previous comments from the submission a couple years ago: [https://news.ycombinator.com/item?id=8908462](https://news.ycombinator.com/item?id=8908462)

</comment><comment><USER3> This article is a great litmus test for checking if someone has experience working at scale (Multi Terabytes, Multiple analysts, Multiple job types) or not. Anyone who has had that experience will instantly describe why this article is wrong. It's akin to saying a Tesla is faster than Boeing 777 on a 100 meter track.

<comment><USER50> I'd hope people who have worked at scale still are capable of recognizing when the tools they used there are totally overkill. I'd suspect they would, since they'd also be more aware of their limitations (vs somebody without experience, who has to believe the "you need big data and everything is easy" marketing).

That you wouldn't use a Boing 777 _IF_ your problem is just a 100m track is the entire point of the article. It's explicitly not saying that you never should use the big tools.

<comment><USER3> They are not overkill at all, rather they are tuned towards different set of performance characteristics. E.g. in the Boeing 777 example above, transatlantic journey.

In the article above, the data and results stay on the local disk, however in any organization, they need to be stored in a distributed manner, available to multiple users with varying levels of technical expertise. Typically in NFS or HDFS, preferably if they are records stored/indexed via Hive/Presto. At which point the real issue is how do you reduce the delay resulting from transferring data over the network. Which is what the original idea (moving computation closer to data) behind Hadoop/MapReduce.

<comment><USER4> _rolls eyes_

The point is that if you've got such tiny quantities of data, why are you storing it in a distributed manner, and why are you breaking out the 777 for a trip around the racetrack? Grab the 777 when you need it, and take the Tesla when you need the performance characteristics of a Tesla.</comment></comment></comment></comment>


Verifying Software with Timers and Clocks - <AUTHOR> https://insights.sei.cmu.edu/sei_blog/2016/12/verifying-software-with-timers-and-clocks-stacs.html <comment><USER2> I've always wondered how to use formal verification to prove software on 8 bit microcontrollers. There are so many peripherals like PWM/timers/counters, ADC, UART, and complicated configurations like clock prescalers, self-programming, and interrupts.

<comment><USER3> Not to mention what the clock source is. PLLs (internal or external) can become unlocked. RC oscillators have a lot of temperature dependence.

<comment><USER4> My most amusing observation of how much this can matter is the obvious obvious pitch/tone change in rail road alerts (at-grade rail crossing signals) on the Metra rail lines around Chicago in the winter. There is a noticeable difference even between 30F and 10F. IIRC correctly (it's been a year since I've been riding the train in the winter), the "beat" of the signal rapidly increases at lower temperatures and the pitch also seems to increase.

Edited for clarity.</comment></comment></comment>


Ask HN: What steps do you take to initiate contract/freelance work? - <AUTHOR> Once contacted by a potential client, or in other situations, what steps do you take to initiate contract work in general? Specifically asking as a developer. For instance, I need access to their code at some point. What does a successful set of interactions look like in general? <comment><USER2> If the client wants to hire you, then execute the contract by getting a depost, starting the hourly billing, whatever terms you have set together with the contract.

Request access to the github etc. The architecture of code repositories and development process varies between projects/clients, so this is usually on a case by case basis with the simplest being a simple github repo access, ranging to giving the client your public key to get access to their server where the repo exists or code lives, possibly without a vcs at all.

I recommend you commence billing before you look at the code, but I and others have given short free consultations before work begins, so that is an option to you but only if you think it's necessary to win the contract and/or scope the work. If there is definitely work and/or advice to be given, why not start the paid work on a daily or weekly basis.</comment>


Last Minute Advice for YC Applicants - <AUTHOR> http://blog.garrytan.com/last-minute-advice-for-yc-applicants-updated-for-2014

<comment><USER2> Great article. While I agree that writing skill is incredibly important, I disagree with Mr. Tan's advice to just 'write lots and keep reducing your text' as the best way to go about writing clearly.

I certainly didn't learn how to write clearly at University and I had to write loads of reports and essays for 5 years.

I actually owe a great debt of gratitude to my first job out of University where I worked as an analyst for a finance and economics consultancy. The CEO was fanatical about making everything Plain English writing style. It was a trial by fire.

Anyway, my advice is to go on a Plain English writing course. The course instructors usually critique your writing and offer helpful insights on how to improve. If that's not possible, go through the guides on the Plain English website ([http://www.plainenglish.co.uk/free- guides.html](http://www.plainenglish.co.uk/free-guides.html)). I also highly recommend applying the writing principals in Style: Toward Clarity and Grace ([http://www.amazon.co.uk/Style-Clarity-Chicago-Writing- Publis...](http://www.amazon.co.uk/Style-Clarity-Chicago-Writing- Publishing/dp/0226899152)). This book is worth its weight in gold.

</comment><comment><USER3> Thanks for the great advice. We would love to have you look at our application ;)

</comment><comment><USER4> First time applicant here. I was wondering, How should I ask for feedback on my application from YC alums without sounding like spam email?

<comment><USER5> Are there any yc companies that would really benefit from your company? Reach out to them and ask them for feedback and explain that you chose them because you believe they understand the problem because they experience it.

<comment><USER4> We're working on cloud automation using NLP, so anyone with cloud infrastructure would benefit from it. But I see your point. Thanks for the advise.

</comment></comment></comment><comment><USER6> Are there any YC alumni in the NYC area who would kindly oblige to have a look at our application? We'll buy beer :).

<comment><USER5> add your email to your profile if it isn't already and I'm sure someone would at least look it over remotely.

</comment></comment><comment><USER7> Odd title. It's sage advice but I struggle with the inclusion of "last minute". There's nothing "last minute" about this IMO - if you're having to figure any of this out at the last minute, you aren't the YC type, I would guess.

<comment><USER8> There are plenty of super-early or even idea stage companies that apply, right? Seems relevant in that case.

<comment><USER9> Agree. And couple of points are applicable to any stage company, so makes perfect sense.

</comment></comment></comment><comment><USER10> In one of the photos, in the background, there are several crock pots. What's up with that? What are they cooking and why so many?

<comment><USER11> Dinner. With over 50 startups a batch (75 in the current one), we have to feed almost 200 people every week. Hence, lots of crock pots.

<comment><USER12> Will there ever by a YC recipe book? Pretty please? I want to know all of PG's culinary secrets.

<comment><USER13> [http://paulgraham.com/ramenprofitable.html#f1n](http://paulgraham.com/ramenprofitable.html#f1n)

</comment></comment></comment></comment><comment><USER9> Any YC alumni in India? (Preferably Bangalore) Would love to get some feedback on my idea/application. Thanks!

</comment><comment><USER14> Any YC alums care to give my application a once over?

</comment><comment><USER15> Should be first minute advice.

</comment><comment><USER16> Thanks for this!

</comment><comment><USER17> Amazing advice.

</comment><comment><USER18> I'm not a chick and don't have an axe to grind other than my business. Did not apply.</comment>


Why PHP is (so much) more better than Ruby? - <AUTHOR> http://www.slideshare.net/camilleroux/why-php-is-so-much-more-better-than-ruby

<comment><USER2> The getter/setter lines in PHP on slide 9 is the code of a ruby dev that doesn't know PHP. You can easily reduce all those lines to a three line _call() method.

<comment><USER3> I thought PHPers got sens of humor... where is yours ? :D

</comment></comment><comment><USER4> Writing PHP is 25% faster than Ruby (strlen(PHP) = 3, Ruby.length = 4)

<comment><USER3> But we have draw here:

"Ruby.length".length == "strlen(PHP)".length</comment></comment>


An Example of Upgrade Pricing for iOS - <AUTHOR> https://itunes.apple.com/us/app/ftp-on-go-pro-upgrade-for/id705015112?mt=8 <comment><AUTHOR> They use a URL Scheme to check for the existence of previous versions on the device, and depending on the result, offer one of three in-app purchase prices for the same result: activating the new version. See also [http://www.macdrifter.com/2013/10/an-interesting-ios-app- sto...](http://www.macdrifter.com/2013/10/an-interesting-ios-app-store- upgrade-example.html)</comment>


Why C++ is a viable alternative to C in embedded systems design - <AUTHOR> http://www.embedded.com/design/opensource/212100638?cid=RSSfeed_embedded_news <comment><USER2> I developed C++ software for a launch vehicle. It flew fine. The toughest thing was convincing the old timers to change. They were much more comfortable with assembly a lot more time in their schedules.</comment>


Photo WebAPP using Almond,Underscore,Backbone + Marionette, usejsdocs.org - <AUTHOR> http://photomania.net/

<comment><AUTHOR> The code is wrapped with AMD, each class is in its own file (namespaces style) requirejs in dev, almond in production jquery for the DOM underscore... Backbone + Marionette Architecture influenced mainly by [1]. [2] for the documentation.

references: [1][http://www.slideshare.net/nzakas/scalable-javascript- applica...](http://www.slideshare.net/nzakas/scalable-javascript-application- architecture) [2][http://usejsdoc.org/](http://usejsdoc.org/)</comment>


Book club for startup founders, tech pioneers, and indie makers - <AUTHOR> https://thestartupbook.club <comment><USER2> The fact that you are using affiliate links should be disclosed _before_ the "Buy" button is clicked.

<comment><AUTHOR> It's not my club, I'm just sharing the link.

Though from what I saw there is an information that the site uses affiliate links?</comment></comment>


Wattvision (YC W09): The Tale of the Mystery Load - <AUTHOR> http://blog.wattvision.com/the-tale-of-the-mystery-load-what-happens-aft <comment><USER2> I immediately wanted to purchase this for our business. My enthusiasm was greeted by a server error msg on the mobile site, with no way of viewing the full site...

:/

<comment><AUTHOR> Thanks for catching this. We've fixed it -- please contact us at firstname.lastname@example.com with any other questions or issues. We look forward to having you as a customer!

</comment></comment><comment><USER3> Why is the Wattvision system $249? What are the big pieces that make the cost so high? I'm wondering if it can be done cheaper.

<comment><USER4> Who else can you buy this from?

<comment><USER5> Www.Theenergydetective.com is another option. I think I paid $135 and found $50/month savings right away.

<comment><USER6> Curious as to what the 50 per month was from?

<comment><USER5> The biggest chunk of the $50 was a malfunctioning electric water heater. But we went on to save $160/mo. more. The biggest surprises were 3x 20 watt doorbell inverters, the DVD player that never shut off, the desktop computers, the old TV, and 5-6x 10 watt draws that were hard to track down.

Now our electric bill is 15% what it used to be with no real change in our behavior. ($40/mo.) Actually, we've increased the amount of light in several rooms.

</comment></comment></comment></comment></comment><comment><USER7> Will Wattvision someday be able to tap into the (GE) SmartMeter data that already exists? I'm in a multi-unit condo and my wifi closet is a few hundred feet from our meter stack. Is there any way to interrogate the meter via the onboard mesh radio?

<comment><AUTHOR> Yes, definitely something on our radar.

</comment></comment><comment><USER8> Are digital electricity meters (which I gather Wattvision requires) common in the US? Most homes I know in Australia still use an analog/physical device counter.

<comment><USER9> Here in the Netherlands it definately was a hot issue in regards to privacy. But likely the new meters will come: they could bring savings to the power companies by way of better knowledge of usages.

</comment></comment><comment><USER10> Very cool product!</comment>


Tell HN: Linux Mint still serves their hash sums from http not https - angry-hacker You can verify your ISO from downloads page[0] by choosing your version, but when you want to verify the checksum, you are asked to visit a domain http://mint-mirror.gwendallebihan.net to download a file of the hash (and/or gpg key) that is served over plain http[1].<p>[0]: https://linuxmint.com/verify.php<p>[1]: http://mint-mirror.gwendallebihan.net/isos/stable/18.1/sha256sum.txt <comment><AUTHOR> They really can't be taken seriously anymore.

<comment><USER2> Agreed. Its bad that many non tech people still recommend it because they dont are able to understand why its a bad idea...

</comment></comment><comment><USER3> On a slightly related topic, ubuntu apt-gets are still usually served on http. Dodgy.

<comment><USER4> I don't know the details but any packages that you get served should include pgp signatures, that apt should verify before installation.

<comment><USER5> Private keys can be stolen or extracted via bribery, etc.

<comment><USER4> Are you saying if ubuntu served apt-gets over https, stolen private keys wouldn't be a problem?

<comment><USER5> Multiple layers of security is industry practice.

<comment><USER4> Sure. In this particular setting, having both https and pgp is definitely better and a third layer would be even better, though I cannot think of one. Successful transfer from https Ubuntu mirror says my package was not tampered with during transmission, but nothing about the validity and integrity of the package itself. The Ubuntu project most likely does not have absolute control over the various mirrors available to users, so pgp verification provides a further assurance that mere https cannot.

If I had to chose between https and pgp, I would pick pgp. The OpenBSD project has a similar scheme (called signify).

</comment></comment></comment></comment></comment></comment><comment> angry-hacker This is relevant because their ISO's have been already compromised once.[0]

[0]: [https://news.ycombinator.com/item?id=11149839](https://news.ycombinator.com/item?id=11149839)

</comment><comment><USER6> Genuinely curious — how is this a problem since the iso is served over https?</comment>


Cetacean Palaeobiology: Archaeoceti - <AUTHOR> http://palaeo.gly.bris.ac.uk/Palaeofiles/whales/archaeoceti.htm <comment><USER2> Have a listen to this as well!

[http://www.bbc.co.uk/programmes/b00kfqm6](http://www.bbc.co.uk/programmes/b00kfqm6)</comment>


Learn how to become linux kernel developer - <AUTHOR> http://eudyptula-challenge.org/ <comment><USER2> This is really interesting! I've always wanted to get in to stuff like this, but I haven't been able to find any good resources for it, I'll definitely check this out over a weekend.</comment>


Would Your Dog Eat You If You Died? - <AUTHOR> http://news.nationalgeographic.com/2017/06/pets-dogs-cats-eat-dead-owners-forensics-science/ <comment><USER2> If my dog is going to eat me, I at least hope he likes the way I taste and he remembers it fondly.

<comment><USER3> Unfortunately, a dog that is believed to have eaten a human corpse will be euthanized.

So, best not, for all involved.

</comment></comment><comment><USER4> So what? I'm dead. Have at it. Hope she enjoys the meal.

</comment><comment><USER5> Yet another great reason not to have pets!

<comment><USER6> It's not really. Too often we anthropomorphizing out pets and even if they are not that different from us on the social-emotional level you cant blame for doing this - they have no concept of moral wrong of doing it. Historically, consider some human cultures not that long ago where cannibalizing your close ones was a honorable way to show them admiration (and also get the prion disease).

</comment></comment><comment><USER7> I legit worry about this. Not sure why.</comment>


Don't pin the butterfly: not all hobbies need to become hustles - <AUTHOR> https://nesslabs.com/hobbies-versus-hustles <comment><USER2> Here is the point that stands out for me in this article -- _The lower your self-complexity, the more fragile you are. Let's say you're a man who thinks about his life mainly in terms of work and family. If your manager gives you harsh feedback, that's 50% of your sense of self taking a hit. But if your self-complexity is higher—let's say you have a couple of hobbies, personal goals outside of work, and relationships outside of your family members—the impact will be lessened._

People will often define themselves by what they do, this has been true since the stone age when people named themselves after their profession. The more things you do, the less impact there is on your opinion of yourself if someone says that one of the things you do is substandard.

Having hobbies will help avoid the trap that people get into when they take a job "because it pays well" and not because they are interested in it. The trap is sprung because it is harder to work at a job you're not invested in, and you don't generally do as well as people who are invested. So you can end up taking time away from other fulfilling activities to get told your doing badly at the thing you are spending all your time trying to do. That is a horrible place to be mentally.

<comment><USER3> >Having hobbies will help avoid the trap that people get into when they take a job "because it pays well" and not because they are interested in it.

What? People take those jobs because they need money to live and that's their only option (or at least they think it is). Unless your hobby pays, it has absolutely nothing to do with it.

<comment><USER2> FWIW I have known Vice Presidents with this problem. Their entire notion of self worth was wrapped up in their job performance and what people said of them.

People internalize their value in different ways, some are more resilient than others. Having something that you can be passionate about and good at outside your job has been shown to help resiliency. That's a good thing for anyone's mental health I suspect.

</comment></comment></comment><comment><USER4> I started doing astrophotography ~5 years ago because I found the mysterious beauty of the incomprehensibly large cosmos a rejuvenating, wondrous thing. In the course of my fun I built a star tracking mount that people seemed to like, so I refined the design, built 10 and sold them on Etsy to test the waters of entrepreneurship. Today, I sell 20-30/mth via a dedicated store.

How often do I do astrophotography these days? Never. In fact my camera broke and I haven't bothered to replace it. I don't have the time.

Has the entrepreneurship been worth it? Probably. But I do miss looking at the stars.

<comment><USER5> Looks like a great product!

If you don't mind my asking: Do you have another day job? Some external investment? Are you expecting to grow to a much larger scale?

Selling 250–300 devices per year at $100 each doesn't seem like it would pay the bills even for one person, after materials and other business expenses.

<comment><USER4> Thanks. :) This is a hobby for me, an experiment in entrepreneurship. I have a day job that pays well enough that I'm free to quit the side business at any time, which I may do soon.

The alternative I'm mulling over in my mind is to push the next rev that I've been testing which will reduce material costs another 25%, and then actually market the product to increase volume 3-5x (100% of current sales are from organic traffic, mostly google searches). That kind of effort requires a lot of upfront investment in time I currently don't have, though. And even then I wouldn't be comfortable quitting my day job. So we'll see. My heart is telling me to stop, but my brain is chanting, "student loan payoff."

</comment></comment></comment><comment><USER6> Good article, I get it.

I've had 3-4 different hobbies over the past 20 years that have stuck. Bicycling/Motorcycling, Photography, and Music.

They all have semi-obvious side hustles that are pretty poor business opportunities that suck people in and ruin the hobby.

Bicycling - Was on a team, sucked up a lot of time, racing was fun, team members in charge of finance embezzled the team finances. We ran a weekly race series so the amount of money was pretty shockingly large. Ruined it for a lot of us. Racing is massive effort and was never that fulfilling. At no level do the prizes start to make any sense.

Bicycling/Motorcycling - I would say less so for these hobbies but there are some traps around making/producing accessories & trinkets.

Photography - The side hustle you see people get into is small time sales, wedding/event photography, etc.. the problem being most of this stuff is a really expensive way to not make much money and take almost all the artistic fun out of it.

Music - people get into bands and try to get paid to play concerts, etc.. pay is crap, it's hard work, gotta have a lot of equipment, etc..

<comment><USER7> Software development - Was on a team, sucked up a lot of time, programming was fun, team members in charge of finance embezzled the team finances. We ran a weekly sprint so the amount of money was pretty shockingly large. Ruined it for a lot of us. Programming is massive effort and was never that fulfilling. At no level do the prizes start to make any sense. I would say less so for ios development but there are some traps around making/producing apps & widgets. The side hustle you see people get into is small time sales, short projects, freelance, etc.. the problem being most of this stuff is a really expensive way to not make much money and take almost all the artistic fun out of it. People get into bands and try to get paid to support in-house, etc.. pay is crap, it's hard work, gotta have a lot of equipment, etc..

Sorry for abusing your comment :) but it seems that programming is a king of all hobbies.

<comment> em-bee if you do it that way on the side to a fulltime job then i agree with you. both programming and biking as a side project are more fun and relaxing when they are done without pressure.

</comment></comment></comment><comment><USER8> I definitely think this is a useful perspective, but if you're creating something, it can be nice to turn your hobby into a hustle in order to increase the amount that you can create.

If you're getting into glass blowing, you can only have so many vases (or bongs, whatever) in your house, so selling them is a good way to cover the cost of your materials while letting you continue to engage in your hobby. The key is to remember the purpose of selling them - it's not to make a bunch of money, it's just to enable you to blow more glass. Don't give refunds, discounts, or anything else - just price your stuff so it sells and move on after each sale.

<comment><USER9> That's an intersting way of looking at it that I didn't think about, thanks for sharing it.

</comment></comment><comment><USER10> I love building speakers, and I've been doing it off and on for almost 20 years.

Here's the thing: I can't build speakers very often, because it's expensive, they take up space, and I really don't need multiple sets of speakers for every room in my house.

Furthermore, if I built speakers for money, I'd have to charge thousands of dollars for a set just to justify my time; or I'd have to figure out how to grow volume to get anywhere near the low cost and high quality of factory- built speakers. In both cases, that means spending a lot of time "hustling."

So, I just build a new set of speakers whenever I want a new set. I don't try to turn it into a business.

(For the curious: This page is from 2003: [https://andrewrondeau.com/Voight%20Pipes/index.html](https://andrewrondeau.com/Voight%20Pipes/index.html))

<comment><USER11> You don't have to charge enough to justify your time if it's an activity you enjoy doing for free. Could easily sell them on a commission basis, limiting how many you offer to whatever you would enjoy, and then just charge materials plus a small profit that you can put toward funding your own projects. Less charging for your labor as securing funding for your hobby.

</comment></comment><comment><USER12> There's an old saying in the amateur radio hobby. If you want to make a small fortune in ham radio, start out with a large fortune.

</comment><comment><USER13> My highest self-complexity is in my work and main goals. Like composing an orchestra there any many moving parts that need to come together. Hobbies and sports are generally less complex and more simple.

I don't think dropshipping chinese tea counts as a passion or method of self- expression for her and she's articulating that across the whole category. She's clearly more interested in consuming and learning about Chinese tea than expressing an inner desire to be a drop-shipping tycoon, that's cool. Responding to money stress by forcing your hobby into a job is not a great idea, I agree.

<comment><USER14> > My highest self-complexity is in my work and main goals. Like composing an > orchestra there any many moving parts that need to come together. Hobbies > and sports are generally less complex and more simple.

I think that depends a lot on the hobby and the individuals dedication (and ability to devote time) to the hobby. For example, one of my hobbies is I'm working on writing an opera. So that is literally writing for a full orchestra, but with the additional complexity of writing a story and libretto. In truth most of my hobbies can easily get as complex as my job, if I want to take it that far. However, for many people (sometime myself included) the fact that they are hobbies means we are limited in how much depth we have the time to commit.

<comment><USER13> Do you think there's any overlapping benefit from increasing your hobby complexity to the same level as your work complexity?

I wonder if high performance individuals commit to a lifestyle that has high performance across the board and reap a certain kind of structure or skills increase from doing so.

<comment><USER15> That's a really interesting question.

When I look over my hobbies, they are all pretty much as complex and demanding as my job as a software dev. But from a very young age, I always preferred complex activities (I suspect that's a big part of why I was drawn to software engineering).

</comment></comment></comment></comment><comment><USER16> As I once read here on HN:

Don't ruin a perfectly good hobby by making it a job.

<comment><AUTHOR> Great way to put it.

</comment></comment><comment><USER15> I agree with this. The problem with monetizing your hobby is that it turns your hobby into a business. That eliminates a whole lot of the benefit and joy of having the hobby.

</comment><comment><USER17> I wholly agree. I've been programming for about 20 years professionally and I've always been juggling too many hobbies to count until I stumbled on one to two that stayed. About 10 years ago I started to paint and draw and that gave me so much satisfaction and filled a whole in my life that to this day I continue to do this almost daily. I've had epiphanies, realizations or simply enjoyed quietly how drawings unravelled in front of me. I have learned and perfected the craft over the years and even though the 'stream of creativity' continues to this day I am happy this is just a hobby. I would not like to deal with the art world in any way. I think that if I went to art school I'd be quietly pursue computer science or computers or something like that. Hobbies have the potential to be something in which you have complete freedom and don't need to justify to anyone or compete with anyone.

</comment><comment><USER18> There is a corollary to this as well, where people say they are starting a business, where actually they are trying to start a hobby.

A business exists to make money, and usually is not particularly enjoyable (with some exceptions of course).

Whenever someone tells me about their business idea I internally rank it on a hobby<->business scale, although I usually try to hold my judgements to myself.

</comment><comment><USER19> It's great advice, but if I'm not sometimes tempted to skip a day for a hobby (hello running) it doesn't feel as fulfilling to me.

I'm under no illusions that I will ever be a great runner, for example, but pushing my boundaries is really important to me.</comment>


The Cronologics team is joining Google - happy-go-lucky https://www.cronologics.com <comment><AUTHOR> [https://ourincrediblejourney.tumblr.com/](https://ourincrediblejourney.tumblr.com/)</comment>


Unity 4 to offer support for Linux - <AUTHOR> http://arstechnica.com/gaming/2012/06/unity-4-will-let-devs-do-so-much-more-with-so-much-less

<comment><USER2> The title has been changed here on HN and doesn't reflect the article:

      Unity 4 is also the first version of the engine to
      offer support for "the estimated 10 percent of the
      game-hungry PC market" that runs Linux, though the
      feature will only be available in "preview" form
      for the time being. Those that pre-order Unity 4
      starting today will get access to a beta test for
      the new engine, ahead of a full release that is
      still undated.
        -- from the article

</comment><comment><USER3> Couldn't find an official statement, that the Unity3D development software will run under Linux. It sounds more like the new update will only allow to build _games_ that run under Linux [1].

I hope I'm wrong.

<comment><USER4> Still... VERY interesting possibilities open up if a game server can be run on Linux. Cloud based Windows licenses KILL the pocketbook once your users grow.

So even if it is not the Unity IDE... it's still good news.

<comment><USER5> Game servers for Unity can already be run on Linux. You'd write a multiplayer game in such a way that it just uses vanilla C#, and data structures that are easy to replicate (like Vector3, Transform, etc).

Running Unity apps themselves on the server even with this announcement would be a big mistake, IMO. It is not a stable piece of software.

</comment></comment></comment><comment><USER6> My first thought looking at this was the unity desktop in Ubuntu... that was confusing.

<comment><USER7> Yeah, it is unlikely that Ubuntu Unity will ever work properly on Linux.

<comment><USER8> On this off-topic excursion, I must say: I have to give Unity another shot. I tried it and ran back to KDE screaming, but it occurs to me that it wasn't really fair and I didn't spend a couple weeks with it, to get over the "new interface aaaah!" reaction.

</comment></comment></comment><comment><USER9> The momentum of Unity is quite amazing, one out of every 20 installs of new apps on the Google Play market is using Unity 3D currently: [http://www.appbrain.com/stats/libraries/details/unity/unity-...](http://www.appbrain.com/stats/libraries/details/unity/unity-3d)

</comment><comment><USER10> That's a great news! Seems that Linux is getting a lot of gaming love in the last period. We should thanks the guys behind Wasteland 2 for helping or providing the port[1].

[1] [http://www.kickstarter.com/projects/inxile/wasteland-2/posts...](http://www.kickstarter.com/projects/inxile/wasteland-2/posts/228226)

<comment><USER11> In the 2D world, Moai now runs in Linux too. I started working for Zipline recently (working on the docs ATM), and (shameless plug) the more I become familiar with the engine, the more I appreciate its simple yet powerful programming model and its architecture.

Each graphical element (a "prop" in Moai parlance), when moved, rotated or scaled, defines its own coordinate system, and other props can be added as children and share some or all of the new coordinates. It makes it very easy to compose elements. People familiar with 3D graphics would know this as affine transforms and UV coordinates.

For example, you can tie the position and scale of a text label to a sprite, but not its rotation, and tie the scale, rotation and position of the sprite to the background. You can then zoom everyting in one command, rotate the wole game world or and keep the label next to the sprite and rotate the whole scene or the sprite relative to the map while leaving the label straight. You don't have to track the relative angles, the engine does it for you.

Similarly, every object whose properties depends on time ("actions") can be attached or detached not only to the root action (which provide a tick each frame) but to other actions as well. When you detach an item, its descendants are also paused. Actions include animations, coroutines, physiscs worlds, you name it...

Both trees are defined declaratively in Lua, and implemented in C++, giving the best of both worlds: expressivity and speed.

Moai's biggest flaw at the moment is the sparse documentation, and we're tackling it. Right now, the main focus of the dev team is the cloud offering (which is our revenue source, after all), so it does not proceed as fast as we would like, but come back in a few weeks and you should see a vastly improved documentation.

<http://getmoai.com>

<comment><USER12> I did have a look at Moai (I work with Unity normally) but indeed, the documentation just wasn't there so I gave up and went back to Unity. I'll keep an eye on it.

</comment></comment></comment><comment><USER13> I got very excited, because I thought at first from the headline that this meant that Ubuntu's Unity interface now supports Linux. Kind of disappointed to realize that's not the case. I guess we'll have to wait a few more versions..

</comment><comment><USER14> I was really shocked when I found out, that Unity is not running under Linux. They are flooding the Monologue (Mono) news feed - but their product is not running under linux. Couldn't quite understand this.

<comment><USER15> They use Mono to power their scripting engine and MonoDevelop is the Unity code editor.

</comment></comment><comment><USER16> I thought that Linux support was already basically provided via Chrome native client exporting?

The part that was interesting to me. _"the estimated 10 percent of the game- hungry PC market" that runs Linux_

This kind of runs contrary to the popular philosophy that gamers are very unlikely to be running Linux, mainly because the games support has historically always been very bad.

Of course there may be a mass of people who are running dual boots and are just waiting for a load of games to be released in order to delete their Windows partitions (people like me).

On the other hand, I run a dual boot and mainly use Linux as my desktop but since Windows is only a reboot away whether or not a game is available for Linux is not a massive part of my purchasing decision for the game. So it's possible it won't really affect their sales figures.

This surely also brings DirectX into question in terms of it's future as a platform. If I were to develop a game today I would almost certainly use OpenGL for portability reasons even if DirectX was a better library and it doesn't make a load of sense to make seperate DirectX and OpenGL versions.

<comment><USER17> Native Client only really gives support for browser-based games. This export option will let you ship standalone games for Linux, that run natively on the platform.

</comment></comment><comment><USER18> This is really good news. I think the second most important thing holding Linux back in taking over the desktops (after OEMs) is gaming. But if you look at Humble Indie Bundles and kickstarter (so many projects using Unity and/or promising Linux support) - things are changing for the better.

<comment><USER19> I think there's a bit more than just OEMs and gaming preventing Linux from overtaking the desktop. When netbooks first began they were mostly linux for cost cutting. Games also wouldn't have been an issue because netbooks weren't sold as gaming machines. In any case the market wholly rejected Linux on the netbook in favor of windows.

<comment><USER18> Netbooks are not desktops. The market rejected netbooks altogether in favour of notebooks and tablets.

What's the other thing holding Linux back from the desktop in your opinion that ranks higher than either of reasons I mentioned?

<comment><USER19> Only after the iPad recreated the tablet market. For a while netbooks were gaining traction. Just not linux netbooks.

>What's the other thing holding Linux back from the desktop in your opinion that ranks higher than either of reasons I mentioned?

You're shifting the burden of proof here. For what possible reason would anyone believe the market _wants_ Linux on the desktop? I can think of no reason. It doesn't look as nice, it doesn't run the major apps people are used to (e.g. Photoshop, Office, etc.). What "killer feature" does Linux have that would make the masses want to switch to it, and don't say "open source" since I doubt that even a majority of people who know what that means care about that.

</comment></comment></comment></comment><comment><USER20> Good news I'd say, hope it continues to progress and become the number one engine on Linux. Epic from my understanding gave up on Linux but not Id and now there's Unity.

</comment><comment><USER21> It makes sense for Unity to do this in anticipation of Steam for Linux.

</comment><comment><USER22> RMS must be furious.

<comment><USER23> As I understand things, he doesn't care so much about games.

<comment><USER24> Unity is not just about games, but more generally about 3D application development. And with Unity Web Player we even got a new closed source browser plugin that starts competing with Flash. I don't care about closed source games (generally I think it increases my freedom to have those compared to not having them at all - kind of like fun manacles which increase choice of action in some moments despite being used to restrict movement). But I do care about the base of software on which other software depends no longer being free and am a little worried about yet another proprietary browser plugin gaining popularity when flash is finally starting to get replaced by html.

<comment><USER25> To be fair, if that's going to happen, it's going to happen if Unity is on Linux or not, Thus it's better to have it on Linux I figure.</comment></comment></comment></comment>