The key value store we ignored (Postgresql) - <AUTHOR> http://blog.creapptives.com/post/14062057061/the-key-value-store-everyone-ignored-postgresql Postgres had this hstore for structure free storage and nobody ever mentions it. This post will hopefully bring attention to something we ignored long ago. <comment><USER2> I agree with the author that hstore is very interesting, but the data structures are not the key selling point in the NoSQL space in my opinion. The most overlooked advantage to things like Cassandra and Riak are the fact that you have no single point of failure in the system. If an individual node fails, there is no operational impact.

Postgres does have (finally!) a nice replication story, so you have data protection on a second server, but the failover mechanics are much more complicated and you still have a single point of failure at the master/writer node. The story gets even more operationally complex when you talk about sharding and the fact that you now have to have a replica per master -- and it really needs to be the same size as the master if you want to be able to trust that you have enough capacity to fail over to it. Suddenly you need to have half of your database capacity sitting idle.

Now, don't get me wrong, I think Postgres is a wonderful database server. For the vast majority of applications it is the correct default choice. Very few applications ever get to the point where they need to scale further than they can get vertically on a single node and most can tolerate a few minutes of downtime to fail over to a backup server. Hand-wavy "sharding is easy" comments, however, ignore a lot of operational reality and that's dangerous.

Understand your use case. Understand the failure modes. Decide how available your datastore needs to be and how much data you need to support. Know the consequences of the choices you make.

<comment><USER3> Agreed, but of course the massive write parallelism and fault tolerance of DBMS like Cassandra comes at the cost of dropping ACID, which may cause a lot of complexity elsewhere in the system. It also comes at the cost of limiting the types of queries you can perform without resorting to procedural code (at least in the case of column family based architectures). In other words, it comes at the cost of productivity.

So, even if sharding is not easy, you can do quite a lot of it in the time you save by not having to code around the limitations of most noSQL DBMS.

The day will come when RDBMS are secret productivity weapon of smaller companies that don't feel they have to act like they were Google.

That said, there are very good reasons not to use RDBMS in cases where the data model or specific access patterns just don't fit. But in most of those cases, I find that using in memory data structures combined with file system storage or something like BerkeleyDB is a much better fit than any server based DBMS.

<comment><USER4> _> comes at the cost of dropping ACID_

Do remember though that, as discussed here recently, most RDBMSs do not act in a fully ACID compliant way by default. IIRC it is providing a complete isolation guarantee often also provides a hefty performance hit so compromises are made in this area unless you explicitly tell it to be as careful as it can. I imagine this can cause quite a nightmare for master<->master replication.

There are a lot of people using "noSQL" options for the wrong reasons (such as to be buzzword compliant, or because they don't understand SQL), but there are issues that traditional RDBMSs have that stick-in-the-muds like me (who cringe at the phrase "eventual consistency" should be more aware of than we generally are.

Making the right choices about your data storage can be _hard_.

</comment></comment></comment><comment><USER5> The main selling point of the various NoSQL products out there today isn't the schemaless storage, instead it's the ability to grow beyond a single server that's compelling.

228MB of data is nothing, it fits in RAM of any machine. What would the examples in this blog post look like if it was 228GB of data spread across 10 servers instead? How would you grow/shrink such a cluster? How would you perform a query that cuts across servers and aggregates data from some of them? How would you replicate data across multiple servers?

<comment><USER6> _228MB of data is nothing, it fits in RAM of any machine. What would the examples in this blog post look like if it was 228GB_

You bring up an interesting point. At what data size do the companies using the various NoSQL DBs feel they _have_ to move beyond a traditional RDBMS? I work with some traditional RDBMS stores now that are >500GB in size with tables that add 20M-30M rows/month and querying still isn't an issue. Admittedly it takes expertise to optimize the system and make it work efficiently, but that's going to be the case with any datastore.

<comment><USER7> I can sustain this claim. I work with a system implemented almost entirely in Oracle PL/SQL. Some tables in the system are nearing 800-900 columns, their size often exceeds 600 GB per table (not many of such large tables though). Querying isn't a problem at all. Large schema changes are also mostly painless. The only point at which one has to be really careful is when a schema change requires actual calculations based on historic data with an additional write on each record.

<comment><USER8> A lot of criticisms I read about the immutable limits of RDBMSes turn out, upon closer inspection, to be criticisms of MySQL. Oracle sort of sails past these limitations like a superliner: expensively and gracefully.

Not that I particularly like Oracle as a programmer. I have to check my calendar every time I hit the 32-character limit for names or once again have to write

        some_field_expressing_truth  varchar2(1);
        constraint "y_or_n" check some_field_expressing_truth in ('Y','N');

But relational databases already scale to petabytes and millions of transactions per minute. Just not in the opensource world ... not yet, anyhow.

<comment><USER9> Well PostgreSQL does scale to millions of simple read-only transactions per minute, at least in some benchmarks. And in PostgreSQL 9.2 it will scale much better, in the synthetic benchmark it reached about 13 million per minute at high concurrency (compared to about 2.5 million for 9.1).

[http://rhaas.blogspot.com/2011/09/scalability-in- graphical-f...](http://rhaas.blogspot.com/2011/09/scalability-in-graphical- form-analyzed.html)

<comment><USER8> Hence "not yet".

It's been amazing watching the performance surge in postgres these past few years. I wonder if Red Hat or similar will try sponsoring a tilt at the TPC-C crown in future.

<comment><USER10> Back before Sun bought MySQL they started doing a lot of performance work on Postgres. They didn't go after TPC, but they did show it was only(?) 12% slower than Oracle for SpecjAppServer: <http://www.informationweek.com/news/201001901>

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> I've been using a lot of MySQL and some NoSQL solutions over the years. I don't _need_ most of what NoSQL is supposed to offer and uhm, a couple of years ago I tried PGSQL.

I've never found the same level of finish, performance, etc. It's also very customizable and very dependable. You can get extremely good performance out of it.

Finally... the devs are plain AWESOME. Every time I had an issue, question, etc, not even necessarily related to PGSQL but to SQL and performance in general they would reply quickly, and accurately (I owe one of them many, many beers).

To give you an example, a company I worked for was doing a 10M to 10M comparison (for very specific needs). All businesses we have found were using custom NoSQL solutions, or even Oracle, with huge clusters, costing a lot, and taking ~24h for a result.

So we implemented ours. After a year of testing, coding, etc, it was settled on PGSQL (on FBSD), with the help of the said dev, we were doing the same comparison,in 3H, on a single quad core system (with heaps of ram).

Sure, maybe the other businesses never cared _that_ much for performance, but its still astonishing. Most optimizations were made thanks to PGSQL's dev help were usually speeding up the process by thousand of times.

Disclaimer: I can't detail the comparison (its still under NDA), but it included very specific stuff, in case you wonder about the time taken, etc.

<comment><USER12> _Finally... the devs are plain AWESOME._

A few years ago I got a dreaded call from a customer. A point of sale system was taking 45 seconds suddenly to post invoices. This is a big deal for a cash register role, as you can imagine.

So I went on site, ran some diagnostics, isolated the problem query, etc. At this point they were into the slow part of the afternoon so I asked about it on the -perform list along with query plans and everything else I could think of.

Within an hour I had received an email from Tom Lane explaining:

1) What the planner was getting wrong

2) Why the planner was making an error there, and why this was a special planner case.

3) Giving me suggestions as to how to fix it.

First rate support, let me tell you. (Also, planner behavior was changed in a release soon after to accommodate my use case.)

</comment></comment><comment><USER13> In my opinion, Postgresql + hstore + PLv8js (Javascript as a language for stored procedures, see <http://code.google.com/p/plv8js/>) could really rock the world if it became an accepted standard and got some people behind it.

It's not so great if you start using something and one year later see that the original authors lost interest and move on, leaving you with 10k lines of code that depend on a pray-for-no-bugs unsupported piece of software.

<comment><USER14> Totally agree. I've been building out a new system with hstore and it's been great, planning to do some benchmarking with plv8js as well. If you've got a dataset that you can conceivably scale vertically to accomodate, postgres is a solid _and_ feature-packed database.

</comment></comment><comment><USER15> > ...database grew to a size of 239691780 bytes (Just 228MB)

> ...gives me 14689 rows just under 360ms on average

Uh, 360ms seems like an awfully long time to query such a small dataset.

<comment><USER16> I think that includes the time to transfer the data.

<comment><USER15> I sure hope not. That's usually not the way you benchmark database queries.

<comment><USER17> Then its a good thing that line in the article had nothing to do with benchmarking a database query.

</comment></comment></comment></comment><comment><USER18> reddit used (and still for the most part uses) postgres as a schemaless storage (although they don't use hstore).

Also, I'm pretty sure Heroku uses hstore, because they're the ones that taught me about it.

So not _everyone_ forgot about it.

<comment><AUTHOR> Just read the comments around and you will see the dogma people face on daily basis despite the huge databases of reddit!

<comment><USER19> dogma... what do you mean?

</comment></comment></comment><comment><USER20> Documentation about hstore in postgresql <http://www.postgresql.org/docs/9.1/static/hstore.html>

</comment><comment><USER21> Anybody using this with rails?

<https://github.com/softa/activerecord-postgres-hstore>

<comment><USER22> Yes -- but not with that module. We're using it to tag documents (individual rows in Postgres and Solr) with user-defined metadata.

[http://blog.documentcloud.org/blog/2011/05/arbitrary- metadat...](http://blog.documentcloud.org/blog/2011/05/arbitrary-metadata/)

... which can then be used to power custom indexes of particular collections of documents, like this one:

[http://www.nytimes.com/interactive/2011/12/02/us/oil-and- gas...](http://www.nytimes.com/interactive/2011/12/02/us/oil-and-gas- leases.html)

The fun bit being, that in both Postgres and Solr, you can still run a precise search (including joins for access control, sharing, and boolean full text clauses) with a single query because the HStore keys and values are indexed.

<comment><USER23> How did you build it? Are you using hstore SQL directly? Do you just have migrations with raw SQL in them?

<comment><USER22> Let's dig through it a bit. Our hstore column isn't directly on the document row, but rather in a joined "docdata" table. There was a semi-raw migration to add it:

[https://github.com/documentcloud/documentcloud/blob/master/d...](https://github.com/documentcloud/documentcloud/blob/master/db/migrate/20110429150927_add_data_to_documents.rb)

The "Docdata" model handles serializing JSON hashes of keys -> string values into the hstore format, and parsing 'em back out. This is a little loose, but works for most of the simple things you'd like for tagging documents (senator: Kerry, statecode: WV, recordid: 019281)

[https://github.com/documentcloud/documentcloud/blob/master/a...](https://github.com/documentcloud/documentcloud/blob/master/app/models/docdata.rb)

... the resulting effect is that from the Rails side, working with "document.data" directly is like working with a lazy-loaded hash.

The more interesting bit is the search side of things, here's the query parser and query object (which generates SQL and Solr queries):

[https://github.com/documentcloud/documentcloud/blob/master/l...](https://github.com/documentcloud/documentcloud/blob/master/lib/dc/search/parser.rb)

[https://github.com/documentcloud/documentcloud/blob/master/l...](https://github.com/documentcloud/documentcloud/blob/master/lib/dc/search/query.rb)

... for any given query, we detect if Solr is required (if full-text is involved), or if the search can be satisfied purely in Postgres. So there's one way of generating the key:value query for each back end.

Hope that answers the question.

(Edit:) Forgot to mention that we add a little extra on top of vanilla hstore "key:value" queries. You can write "dob: *" to show documents with any senator tagged, and "dob: !" to show documents that haven't been tagged with a date of birth yet.

<comment><USER23> This is awesome, thank you (didn't think to check if you were in Github, but the explanations are great too).

</comment></comment></comment></comment></comment><comment><USER24> Thoughtbot posted a similar Postgres article recently:

[http://robots.thoughtbot.com/post/13829210385/the-durable- do...](http://robots.thoughtbot.com/post/13829210385/the-durable-document- store-you-didnt-know-you-had-but)

</comment><comment><USER25> I tried hstore on a project for many of the reasons mentioned. I just hated having to always represent data as strings.

<comment><USER23> Isn't this effectively what Redis does too?

There are whole Salvatore posts about doing binary data structures in Redis string values.

<comment><USER26> <http://redis.io/topics/data-types>

Strings are the most basic kind of Redis value. Redis Strings are binary safe, this means that a Redis string can contain any kind of data, for instance a JPEG image or a serialized Ruby object.

</comment></comment></comment><comment><USER27> how does this compare to MySQL's HandlerSocket?

<comment><AUTHOR> I haven't done benchmarks yet but if you are looking for HandlerSocket benchmarks <http://blog.creapptives.com/post/3329352663/the-nosql-dogma> is the link

<comment><USER27> Those benchmarks are very flawed as HandlerSocket only works in InnoDB. This is a better post on the subject: [http://yoshinorimatsunobu.blogspot.com/2010/10/using- mysql-a...](http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as- nosql-story-for.html)

</comment></comment></comment><comment><USER28> @chrismealy.. <http://croak.eu/rTBVKX>

<comment><USER29> Quit spamming this bullshit here. HN is a text-based discussion forum.</comment></comment>


Ask HN: How do you securely enter your LastPass Master Password on Android? - <AUTHOR> I have installed LastPass on my Windows PC and secured it with a strong, complex password. Now it is time to enter that password on my android phone. How do you get that password securely transferred from the PC to Android?
I can, of course, type it in (which you have to do frequently) email it to my self, store it on Dropbox in plain text, etc., but I don't want to do that. It is the master passwords to my other passwords I then have to either share with external cloud service or place in plain text on an Android operating system that hasn't received any updates for years.
It surprises me that LastPass has no QR code option or similar allowing me to point my camera towards my pc monitor and securely set up a phone.
My current solution is to use an easy-to-enter-password and then only use LastPass for sites I do not care so much about like blogs and similar websites that have no financial or sensitive info on me.
Entering a complex password, again and again, is something I don't want to do. Maybe I have overlooked something in settings? What do you do? <comment><USER2> I enter it only once and then enable the fingerprint login

<comment><USER3> This is also what I used to do. But also, I think it's good to remember the master password, it's really the safest place where to keep it. I anyway moved away from LastPass to bitwarden, that has the same fingerprint functionality or a pin one.

<comment><USER4> Ok but then you have the same problem</comment></comment></comment>


React Armor: Protect your DOM from third-party tampering - <AUTHOR> https://github.com/elierotenberg/react-armor <comment><USER2> What they thought this will achieve is: people will stop using scripts that changes "ul li .Bar".

What it will actually achieve is: people will spend more scripts to heuristically check which element under "ul li" could be ".Bar" based on the layout, attributes, and contents. Then they'll change "ul li .whatever", sometimes mistaking the class and getting a broken website. Or worse, they'll select on "ul li @background-color='red'" (or whatever the syntax is), which you can't obfuscate any further, but is almost guaranteed to randomly be incorrect.

I get where they come from (user extensions affecting how the website works), but I'd say the alternative it worse :/

<comment><USER3> _user extensions affecting how the website works_

That's really the point of having them, isn't it? To make the site work the way the _user_ wants. In fact I'd say what it will actually achieve is less visitors to your site... they're just going to leave and go somewhere else.

I classify this along with the anti-right-click, disabling select/copy/paste, changing the status bar, resizing windows, and other general "DO NOT WANT" aspects of sites.

<comment><USER4> If only users were consistent and fair about what they wanted.

I wouldn't personally run something like this, but I've definitely experienced the frustration of having a user install an extension that changed my website, and then reported a bug in the extension to me.

Even when the extension is bug-free, it still adds a support burden. For instance, there's an extension that adds icons to usernames on my site. Users have asked me how to change their icon, and then gotten mad at me when I say I don't know.

It also adds a maintenance burden: Whenever I deploy various updates, I get complaints that the updates break one extension or another.

Not to mention, there are extensions not intentionally installed by users. For a while, there was some very common malware that rendered my website unusable. It added the text

        <script src="http://example.com/malware.js"></script>

But it did it badly, so it changed:

        <script src="/legitimatescript.js"></script>

to

        <script src="/legitimates<script src="http://example.com/malware.js"></script>cript.js"></script>

Which, of course, rendered my site unusable.

<comment><USER5> If you want or need pixel perfect and all kinds of fancy javascript to run your site, maybe you should just make a program instead, even if it's a wrapper around some browser engine.

<comment><USER4> I actually do offer that too, for precisely that reason, but of course many people prefer the online version. And browser extensions and things like Stylish let people do some pretty neat things. It's just a support nightmare. :(

</comment></comment></comment></comment></comment><comment><USER6> This should be called React Obfuscate rather that React Armor.

I've done a fair amount of web scraping before and each of their tricks can be broken with enough care. Obfuscating html not only breaks many of the good things about the web but also makes things harder to debug.

<comment><USER7> We handle over 1600 Japanese institutions at the company I am working for in Tokyo (Moneytree.jp), and we haven't arrived at a situation where we were defeated by what React Armor is trying to prevent.

<comment><USER4> I'm sorry, I've reread your sentence several times, and I still don't understand what it's trying to say.

Does your company have the same goals React Armor has (this seems to be what your double-negative is saying), or is it the party React Armor is trying to protect against, or is it relevant to React Armor in some other way?

What kind of Japanese institutions are these and what does handling them entail? Are you scraping or modifying DOM, or are the institutions? Are you obfuscating DOM, or are the institutions?

<comment><USER8> In the context of moneytree.jp the institutes in question are most likely banks that are reluctant to offer APIs, so the only option for Software Integrators is Web-Scraping.

In Germany we have figo.io that allows the local startup scene to access german banks. From what i can gather from their API its also done in part by scraping.

Some Banks want to obstruct the scraping and the author claims that none of the tricks React Armor offers have thwarted his company so far.

</comment></comment></comment></comment><comment><USER9> This is such a horrendous idea. Apart from the fact that this does virtually nothing, this is a prime example of the difference between security and obscurity and how neither is the other.

</comment><comment><USER10> Hooray, get ready for broken websites with modal overlays, CSS popups, ads, and tracking built-in, and no way to disable them!

<comment><USER11> broken user scripts and styles. f-up accessibility and screen readers.

seriously. wtf.

<comment><USER12> But think of the ads!

Can't get in the way of that!

<comment><USER11> not sure if you were joking, but ads messing up with page is a huge problem, and is in no way solved by this.

But the solution already exists for years: [http://www.iab.com/guidelines/safeframe/](http://www.iab.com/guidelines/safeframe/)

any publisher not using this is not worthy their salt!

</comment></comment></comment></comment><comment><USER13> Sounds like a great way to make my browser waster more RAM & CPU on rendering your site (useless span-subtrees?!?!)

WTF, really?

<comment><USER14> It's not really a great way to do that. JavaScript loops are a much better way to do that.

</comment></comment><comment><USER15> Great example of security theater.

[https://en.wikipedia.org/wiki/Security_theater](https://en.wikipedia.org/wiki/Security_theater)

</comment><comment><USER16> Yeah, I genuinely don't understand what they're going for here. Anything that happens client-side can be modified by the client. Because, you know... it's on their side. Front-end validation and stuff is nice for UI/UX, but nothing that comes from the client should ever be trusted just because you put an obfuscated property on an input or something.

At the end of the day, it all has to be valid HTML tags and javascript that runs without crashing. In terms of security, nothing on the front end even registers on the scale. Absolute worst case scenario, it's like solving a Wheel Of Fortune where I have most but not all of the the letters.

I'm pretty uncomfortable with the description of a user on their own machine, running their own browser running plugins they chose and installed as a "third party" that's engaging in "tampering."

</comment><comment><USER17> This is kind of gross.

Pretty sure this is why we need shadow dom.

</comment><comment><USER18> But why stop there? Compile Chrome in Emscripten and render to canvas, with flickering to prevent screenshots! No DOM, no problems, right?

Until someone just types the data into Excel and manipulates it themselves. If you don't trust your user with data, don't show it to them in the first place!

</comment><comment><USER19> The end of transparent web source code. I hate this. :(

<comment><USER20> Don't be so sure. This will push the client authentication problem further up the stack.

</comment></comment><comment><USER21> The extra work needed to do this pretty makes it very unlikely to be implemented much. Additionally the DOM level obfuscating is just bad for performance.

if you're looking to prevent Ad Blockers, this is not your solution.

</comment><comment><USER22> Performance would probably not suffer at all...

EDIT: But if you really want to make this a useful lib, bake in a way to disable the `armor` for debugging/development.

</comment><comment><USER23> The tool is cool but in practice just detecting that DOM is violated and notifying the user in one way or another (like reducing functionality of the site) should work better. Also it is much harder to defeat especially if detection takes into account the layout.

Update: the detection-only also helps with accessibility as obfuscated DOM makes it impossible to use with screen readers etc.

</comment><comment><USER24> "Such third-party scripts include browser extensions (adblockers...)"

ROFL, soon all the annoying garbage ads will be written in React. Yay!

</comment><comment><USER25> I thought this was a joke when I saw it at React Conf. Apparently not.

</comment><comment><USER26> The headline lead me to believe this was some kind of DOM integrity technology, which would be cool. I have a vague notion this is bad for both security and performance.

</comment><comment><USER27> I know this is malicious to those who want the web to be open, but I love it. Not because I particularly agree with the motive, but it's just such a cool demonstration of the power of React. Doing this has never been so easy. Google has been doing this for a while for a few of their products (Google Plus is one of them, I believe), but they've got large engineering teams who can commit time and resources to protecting data.

If you have a justifiable business reason for doing this, then your life just got a little easier. I think this would also help against some forms of XSS too - so there's some silver lining for you.

<comment><USER28> The class-name obfuscation on Google+, GMail, etc. is for latency reasons, not security. When Google wants security they serve content out of a seamless iframe on https, because obfuscation doesn't give it to you.

</comment></comment><comment><USER29> so many haters here...

Make things that solve your problems.

Have fun while doing it.

Continue.

<comment> chao- Widespread deployment of this will make a number of people's lives less fun. In response, pro-fun countermeasures will be created to solve _their_ new problem.

Or, to quote a comment on a technology discussion site about this project:

 _" And so begins an arms race."_</comment></comment>


Opera Mini Up 84 Percent From 2009 - <AUTHOR> http://techcrunch.com/2011/01/20/opera-mini-had-over-85-5-million-users-in-december-up-84-percent-from-2009/ <comment><USER2> I've found opera mini to be awesome when internet connection is slow. Some pages that never load in safari do so quickly with acceptable quality in mini.</comment>


14 Reasons You're Tired All the Time - <AUTHOR> http://time.com/2828800/bad-habits-drain-energy/ <comment><USER2> I hate to nitpick an otherwise well-intentioned and decent article, but on the point with regard to melatonin, the author makes the following suggestion:

"[If you cannot resist checking your device before bed, then] hold it at least 14 inches away from your face to reduce the risk of sleep interference." This is utter hogwash, for two reasons:

1) Good luck reading a smartphone screen from a distance of 14 inches. This is an unrealistic suggestion.

2) The light intensity and your reception thereof is basically unchanged, which is all that is needed to hem melatonin production. Small amounts of light are enough to disrupt this mechanism. I would cite some good papers as evidence, but I currently cannot find them. I suggest to anyone interested to do research into the properties of melatonin --- it's pretty interesting, not to mention useful.

<comment><USER3> A better solution would be to use something like fl.ux or lux to cut out the blue wavelengths of light during night-time hours.

<comment><USER4> Since the backlight is still blue light, this doesn't have as great of an impact as you would think.

The only smartphone that can completely disable blue light are those with AMOLED screens; and only after being rooted:

[https://play.google.com/store/apps/details?id=mobi.pruss.Gal...](https://play.google.com/store/apps/details?id=mobi.pruss.GalacticNight)

</comment></comment></comment><comment><USER5> Even TIME surrendered to listicles, journalism is officially a thing of the past.

<comment><USER6> This is an article from health.com, it's linked at the bottom and at the top, they're a subsidiary of Time Inc.:

[http://www.health.com/health/gallery/0,,20818045,00.html?xid...](http://www.health.com/health/gallery/0,,20818045,00.html?xid=Time- energy-060314)

Link building is - in spite of panda - alive and well, just more subtle.

Maybe google should penalize Time or Health.com?

Oh, never mind, google is now in the spam and link building business for themselves (spam received on my gmail account, which is not exactly advertised as linked to ww.com):

Google+ link request: Link [http://ww.com/](http://ww.com/) to the vanklanhnguyen Google+ page

email@example.com has sent you a request to link [http://ww.com/](http://ww.com/) to the vanklanhnguyen Google+ page.

Linking a Google+ page to a website not only helps build relationships with friends and followers, but also makes a page eligible to show up on the right- hand side of search for relevant queries which may make the page more discoverable. Learn more

Recommended action

Approve or reject this request. Got feedback? Leave it here. Be sure to include this message ID: [WMT-95212] Google Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043 | Unsubscribe.

</comment></comment><comment><USER7> For those interested in improving the whole "screen lights mimic the sun and keep you awake" thing, consider checking out Flux -- [https://justgetflux.com/](https://justgetflux.com/)

It adjusts the colors of your screen late at night.

<comment><USER8> If you're on Linux, Redshift is a good alternative to Flux. It's in Fedora and Ubuntu package managers - 'yum install redshift' or 'apt-get install redshift' respectively (if memory serves).

</comment></comment><comment><USER9> Or you have hypothyroidism, and need to treat it (I am gathering more data on myself, I will write about it when I have enough interesting data to write a full blown article).

</comment><comment><USER10> Yes, TIME sure has a point there, I noticed that taking extra vitamins and exchanging coffee and cola for water and pure juice definitely improved my overall performance

</comment><comment><USER11> Not related to the article, but I really like time.com's new-ish ([http://time.com/13113/time-coms-redesign/](http://time.com/13113/time-coms- redesign/)) design. They have finally understood that people do not want to click through to the next page. I found myself scrolling down to read other articles, which was as painless as reading through a paper magazine.

<comment><USER12> yes. but it looks like I can't middle click links for some reason...</comment></comment>


Portraits in Oil - <AUTHOR> https://nplusonemag.com/online-only/online-only/portraits-in-oil/ <comment><USER2> Louvre Abu Dhabi's "10,000 Years of Luxury" exhibit looks pretty buzz worthy. Luxury was never a thing that interested me historically. But as a driver of human behavior its a locus of conflict. The recent addition in Manhattan of spaces like The Shed and newly opened Edge observation deck are inherently seductive. Situating this history in a ten millenia window may provide some illumination.

The Meaning of Luxury

[https://www.nytimes.com/2019/11/04/fashion/luxury- exhibition...](https://www.nytimes.com/2019/11/04/fashion/luxury-exhibition- abu-dhabi-cartier.html)</comment>


Emacs isn't for everyone (2010) - <AUTHOR> http://briancarper.net/blog/534

<comment><USER2> _Emacs is good at integrating with Git too. So good that there are four or five different Emacs-Git libraries, each with a different interface and feature set. I gave up eventually and went back to using the command line._

This is superficially true but fundamentally incorrect. There is _one_ Git mode that everyone uses: Magit. Magit is incredible; it is one of a couple Emacs integrations that make the tool it integrates with better, in a significant and meaningful way.

I'm not a "use Emacs as my shell", "read my email in Emacs" kind of person. And I am nowhere nearly as effective in Git on the commandline as I am with Magit.

Use Magit.

<comment><USER3> _There is one Git mode that everyone uses_

This is actually a bit of a problem. Even with many things available out there, some things are more useable than others, and it takes quite a bit of digging to figure out which packages you want to install.

I think people who write up their experiences in emacs and give suggestions about which packages are genuinely useful, and which alternatives aren't, are being really helpful.

<comment><USER4> Actually magit is the second most downloaded package from MELPA. It's kinda hard to miss it.

<comment><USER3> _Actually magit is the second most downloaded package from MELPA._

Thanks. I didn't know MELPA had download counts, now I do. But there is nothing in `list-packages' that hints at this, really, it's just a list of packages.

 _It 's kinda hard to miss it._

The person who wrote the article managed this easily enough. Emacs packages also interact with each other (e.g., completion minor modes and language major modes, things like flycheck). I still think there's a fair amount of uncertainty about what packages one should install, and that projects like [https://github.com/bbatsov/prelude](https://github.com/bbatsov/prelude) are really helpful in this regard.

<comment><USER5> > The person who wrote the article managed this easily enough.

Since it's a single data point I think it really says more about the author of the article than about Emacs...

</comment></comment></comment></comment></comment><comment><USER6> The title of this blog-post is _Emacs isn 't for everyone"_. Another posts on the same blog is charmingly titled _" Emacs undying hatred"_ ([http://briancarper.net/blog/290/emacs-undying- hatred](http://briancarper.net/blog/290/emacs-undying-hatred) ). Just putting two and two together here, I'd come to the reluctant conclusion that emacs is not for brian carper (the blogger).

<comment><USER7> I used emacs for about 10 years. I got to be pretty ok with it. On a lark I decided to try vim for a few weeks and see what it was like. About 2 weeks in I was already more productive in vim. I've never even considered switching back. I don't hate emacs, but emacs isn't for everyone. But vim isn't for everyone either. Use what you like.

</comment></comment><comment><USER8> You can always use emacs like Notepad until your curiosity takes hold. There are so many resources to learn emacs that it's easy to browse and learn. Reddit and StackExchange are "newer" communities.

[http://emacs.stackexchange.com](http://emacs.stackexchange.com)

[http://www.reddit.com/r/emacs/](http://www.reddit.com/r/emacs/)

<comment><USER9> More resources for a quick start:

[http://emacs.sexy](http://emacs.sexy)

        brew install emacs --HEAD --use-git-head --cocoa --with-gnutls --with-rsvg --with-imagemagick

</comment></comment><comment><USER10> I suspect that if i had to learn emacs with the "big-bang" theory, I would be less satisfied. Big-bang is where you try to get it all at once.

My path to emacs started with MicroEMACS. Then, I ended up with a DEC workstation with emacs. What worked for me was starting with the minimum viable command set--open a file, find some text, add or change some text, save the file, exit. Then add a new command or two as you go along. In the 22 years that i have been using emacs, there hasn't been a period of concentrated study. Maybe a few minutes at a time picking up something new.

If I had to teach someone learning emacs to "Refactor this java code", I'm not sure how I would go about instructing someone to do that.

But it serves me well for most of my programming and writing.

</comment><comment><USER11> Over four years have gone by since this post was written and there are now a number of other options for working with Clojure. For example, Counter Clockwise adds Clojure support to Eclipse and Light Table is an editor built from the ground up (in ClojureScript) that adheres more closely to modern UI idioms.

You can see which development environments are most popular in the 2014 State of Clojure survey results [0]. It's interesting to note that Emacs is still by far the most popular.

[0] [https://cognitect.wufoo.com/reports/state-of- clojure-2014-re...](https://cognitect.wufoo.com/reports/state-of- clojure-2014-results/)

</comment><comment><USER12> I am less then convinced that relying on Emacs as a primary environment is a problem, but I think that not documenting the relevant features of emacs and the relevant emacs mode(s) for your language or tool the same as you would if it was a custom, language-specific environment is a problem. Too often the relevant emacs mode is documented with the assumption than any new user of the language already has a broad general proficiency with Emacs.

</comment><comment><USER13> it's nice to see an article that says "X isn't for everyone" in the field of text editors.

While the author talked about emacs being "painful" to learn, they later rephrased in terms of a time investment. But I don't think it's just about a time investment. Some people find the process of learning new key bindings, workflows, etc. more painful than others.

I notice that every single person on my ~5 person team uses Eclipse, instead of Emacs or Vim. That's probably partly due to the influence team members have on each other. But I think it's also because we chose (or were assigned) this team based on certain personality traits. The team's product is focussed on making certain hard to use things easier to user. I think that the people who could relate to this product, are the sort of people who experienced the "pain" of learning things like emacs more than others.

I think that we need to acknowledge that people are different. We should value the people who are capable of learning things like emacs with little psychological difficulty. But those of us who like things to be easy and intuitive from the start, should use our visceral experience or hard to use products, to guide us in making easier to user products for others.

</comment><comment><USER14> And the alternatives are... nothing beyond text editor + terminal, it seems. So does anyone have any plans?

<comment><USER15> If you're working on clojure, there are many alternatives to emacs. Counterclockwise for if you want to do editing in eclipse. Enclojure if you want to do work in netbeans. vsClosure if you want to work in visual studio. Emacs is not the alpha and omega; there are other tools out there which quite possibly have better workflows depending on your need.</comment></comment>


ASK HN: Slight Career Change - <AUTHOR>

I am working as a QA Test/Automation Engineer in a Networking Company since ~2yrs, I'am now planning to move to a Web2.0 or Web App Company like Google, Facebook, LinkedIn as an Apps Developer. How/What should I prepare for the interview ? <comment><USER2> Make sure you can reverse a linked list in C++.

<comment><USER3> And know the differences (in structure and operation speeds) between arrays, linked lists, hash tables, etc.

<comment><AUTHOR> Yes, I have started to brush up on those. I came out of school 2 yrs ago.

</comment></comment></comment><comment><USER4> What are your skills? (What programming technologies do you know?)

<comment><AUTHOR> I have programmed in C,C++,Java,Assembly. I also have some knowledge on PHP,Perl,Python. I am actually enjoying Python.

</comment></comment><comment><USER5> The way for a tester to move into development is to a) have a lot of experience writing automated tests and b) to have a reputation for writing bug reports that let developers quickly locate and fix problems.

</comment><comment><USER6> Choose one of the web frameworks RoR, web.py or Cappuccino and write something cool on your own. Something small enough to show your future employers that you can write good software. It would help you to make the transition and also gives companies confidence that you are capable of making this career change.

<comment><AUTHOR> I am actually playing with Django. What project would you call small enough to show to a prospective employee ?

<comment><USER6> Your could write your own blog, since your were in QA something related to that like a web application to view continious integration status or test results, a web app to show what web 2.0 sites are up or down.

Write down a few requirements for what you plan to do if you end up having more than 10 requirements for the project choose another.</comment></comment></comment>


Making the Mind: Why we've misunderstood the nature-nurture debate (2003) - <AUTHOR> https://bostonreview.net/books-ideas/gary-marcus-making-mind <comment><USER2> _According to the gene shortage argument, genes can't be very important to the birth of the mind because the genome contains only about 30,000 genes, simply too few to account even for the brain's complexity—with its billions of cells and tens of billions of connections between neurons—much less the mind's. "Given that ratio," Ehrlich suggested, "it would be quite a trick for genes typically to control more than the most general aspects of human behavior."_

To me, the clearest argument against this position is that horses can walk within a couple hours after being born. That doesn't seem like anywhere near enough trials to learn how to walk from first principles. The method to walk must be encoded in the horse's DNA.

Humans can't walk at birth, obviously. But Occam's Razor suggests that if the ability to walk can be encoded within a horse's DNA, it can also be encoded within a human's DNA.

Anecdotally, when I watch a baby learn to walk, it doesn't really seem like they are trying a zillion things and picking the one that works. It seems much more like their instincts are telling them to move a certain way, and once they grow strong enough, it succeeds and they are walking.

<comment><USER3> I'm going to provide some anecdata. As a new father, you get to talking with a number of new parents too, with kids that are similar in age to your own. One of the potential downsides to this is that when you talk about your kids, you start to compare them to each other and create a competition where none should exist. It's hard, but as a new parent, you should really try to avoid this and instead focus on nurturing your child to be the best he/she can be.

(Aside: what's the currently accepted pronoun there? I was always taught "they" is incorrect grammar, but without devolving this discussion, that seems to be accepted now?)

My son is 13 months old now, and crawls like a maniac, ever since he was roughly 5 months old. Very social, talkative, active, all those good things. Pulls up and stands all the time, can climb in and out of toy vehicles, climbs stairs, all of that. When I hold him while he stands, he is barely putting any weight on me. Even if he's not walking, he should be able to stand on his own. But he doesn't, and he absolutely refuses to do so. He sees cousins and other big kids walking, and isn't afraid to crawl to them and play, but it doesn't inspire him to walk.

We know a lot of other kids that are younger than him, his age, or a few months older, and they are walking. He's stronger and more capable than them, but they're walking and he isn't.

My wife and I comment on this fact all the time, about how it seems that walking has nothing to do with strength, and it's more about readiness. I doubt that my son isn't walking because he's not strong enough, nor is it because he doesn't have a good environment, save for the fact that I'm not pushing him to walk (i.e. I'm enjoying the calm before the storm.)

Readiness for walking is definitely dependent on a number of factors, but I'm willing to bet that genetics is a big part of it too. Apparently I didn't really walk until I was 2 years old.

Babies and walking is a very interesting phenomenon that really lets you see into the working of genetics and environment.

<comment><USER4> Quick reply to your aside: singular they is very grammatical and has been in use for centuries.

From Merriam-Webster: "they has been in consistent use as a singular pronoun since the late 1300s; that the development of singular they mirrors the development of the singular you from the plural you... and that regardless of what detractors say, nearly everyone uses the singular they in casual conversation and often in formal writing."[1]

[1] [https://www.merriam-webster.com/words-at-play/singular- nonbi...](https://www.merriam-webster.com/words-at-play/singular-nonbinary- they)

<comment><USER5> All politics and grammar rules aside, I personally find 'they' to be really confusing when referring to a specific person.

IE: "is Pat coming to lunch with us?"

"Yes, they're coming in a moment."

Leaves me wondering if 'they' is perhaps referring to some implied group of which Pat is a part, like a SO or team at work (meaning we might need to take two vehicles and get another table at the restaurant) or just Pat.

I wish there was an acceptable non-gendered singular pronoun we could all agree on.

<comment><USER6> That's confusing because "are" ("they're" is a contraction of "they are") is used for the plural. "They is" would be singular, just like "he is" or "she is". It would be ungrammatical for plural "they", but AFIACT is correct for singular "they".

So the unambiguous (but weird sounding to those unused to it) version:

"Is Pat coming to lunch with us?"

"Yes, they is coming in a moment."

<comment><USER7> > "Yes, they is coming in a moment."

Perhaps one of:

"Yes, they will be here soon / joining us shortly / along later."

would be better.

<comment><USER5> > "Yes, they will be here soon / joining us shortly / along later."

None of these clarify singular vs plural of 'they.' All of them work whether 'they' refers to one person or multiple.

<comment><USER7> Good point.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> """Their position rests on two arguments, what Stanford anthropologist Paul Ehrlich dubbed a "gene shortage" and widespread, well-documented findings of "brain plasticity." According to the gene shortage argument, genes can't be very important to the birth of the mind because the genome contains only about 30,000 genes, simply too few to account even for the brain's complexity—with its billions of cells and tens of billions of connections between neurons—much less the mind's"""

I'm sorry but if that's the best argument you can come up with if you're a Stanford Prof, you owe it to yourself to refresh yourself on the literature of how genes work and how they lead to phenotypes. It looks like he ascribes to the "exact blueprint" mental model of phenotypes, rather than the competitive/generative model that is more consistent with modern science.

<comment><USER9> The article is a presentation of what the researchers in genetics and development have learned. Perhaps it would be good if you responded to that.

<comment><USER8> The rest of the article is mostly fine (if very limited).

</comment></comment></comment><comment><USER10> Mods: maybe edit title to reflect date (2003)?

</comment><comment> seph-reed In the past, it would seem people had a lesser nurture:nature ratio. In the future it seems we will have greater.

I think the greatest ratio would be an AI that can fully program itself (including its emotions).

Point being, I suspect we're a lot less nurture than we'd like to think.

</comment><comment><USER11> It's very unfortunate that we still do not have any cloned people. Cloning, especially cloning famous scientists, would greatly help in understanding what part of their success is due to genes, and what is nurture/luck.

<comment><USER12> We have identical twins.

<comment><USER13> Most identical twins have very similar upbringings and share a lot of genetic code.

<comment><USER12> By definition, identical twins share all of their genetic code.

<comment><USER14> Not necessarily true:

[https://www.livescience.com/24694-identical-twins-not- identi...](https://www.livescience.com/24694-identical-twins-not- identical.html)

paper (PDF) : [https://www.researchgate.net/profile/Constantin_Polychronako...](https://www.researchgate.net/profile/Constantin_Polychronakos/publication/257756275_Somatic_point_mutations_occurring_early_in_development_A_monozygotic_twin_study/links/56cef7d708ae85c823430b5d.pdf)

</comment></comment></comment></comment></comment><comment><USER9> I found this article quite fascinating. I had no idea how much the researchers have learned about genetics and development, and the general picture that has developed.

</comment><comment><USER15> If we could make bets according to the markets perception of the Nature- Nurture debate, my feeling is that natures role is undervalued.

</comment><comment><USER16> 2003\. Gary Marcus again.

<comment><USER17> Is there anything wrong with Marcus?

<comment><USER16> No, I agree with some of what he says, but he seems to get an article flagged on here every 5 minutes.

Also I'm not sure how representative of the field he is, and (speaking as somebody who shares some of his scepticism) you would think the consensus would be the prevailing tone.

</comment></comment></comment><comment><USER18> The key to thinking about nature versus nurture that I never see mentioned:

If you believe in evolution, you have to believe in the genetic determinism of genes.

Sure, there is leeway for nurture, but a lot of people are holding contradicting opinions (they believe in evolution but not as much in "nature") because we as a society are hiding the truth of human determinism.

<comment><USER2> If you read a decent amount of genetics research, you usually end up discovering some conclusions that are politically unacceptable. Even a comment as bland as yours, you can see some people consider it Hitlerian. But at the same time plenty of people do read genetics research and perform cutting edge experiments that advance the state of the art. It is acceptable to do these things, as long as you use enough academic jargon that laymen and politicians can't understand what you're saying.

<comment><USER3> I'm not exactly sure why this is controversial at all. And I think most of us know and believe this to be fact.

I am 5'10" tall, born and raised in Western society. I am taller than my most of my ancestors, but only marginally taller than my father. He grew up in India, but his family was non-vegetarian in a village of pure vegetarians.

We might have been able to be a little taller if we understood nutrition a bit better, but it's probably within an inch or so. Our height is limited by genetics, not by environment.

It stands to reason that if physical attributes like height are subject to genetic determinism, aspects of our brain & mind should be too. Now, comparing brains to height is very reductive, since the brain is infinitely more complex. But there's a probably a pretty good reason why my father and I have very good memories, and are fairly intelligent people. But we're also not superstar athletes, or great artists. Genetics and environment have optimized our brains in a certain way. And I'm already seeing the same traits in my 1 year old son.

But isn't that the beauty of genetic determinism? Isn't that the beauty of all of us having vastly different life experiences? I don't lament the fact that I don't have the genetics of Usain Bolt, but I can admire and appreciate his prowess. Similarly, I don't lament the fact that I'm not a great creative like Picasso or Mozart. Something about their genetics and environment allowed them to achieve greatness in areas I am not capable of. And that's fantastic, because our society is enriched because of this diversity.

I think a society that celebrates the individual, gives them the resources and environment to achieve whatever they're capable of, and thus, reach the limits of their own genetics is one I'd like to live in. Realistically, most of us are not even close to be limited by our genetics, but man, how awesome would it be if we were?

<comment><USER19> > It stands to reason that if physical attributes like height are subject to > genetic determinism, aspects of our brain & mind should be too.

And if you don't believe in creationism and also don't believe that intelligence is subject to evolution, then you need some very good explanation where intelligence comes from in the first place.</comment></comment></comment></comment>


Microsoft Flow - <AUTHOR> https://flow.microsoft.com/en-us/ <comment><USER2> Well those look exactly like IFTTT recipes. What can this offer that IFTTT doesn't?</comment>


Why Don't Software Developers Use Static Analysis Tools to Find Bugs? - <AUTHOR> http://www.viva64.com/en/b/0335/ <comment><USER3> I don't know how is why! (And if I learned, then my _coworkers_ wouldn't know how, so until I got everyone else on board, I would be making the bed only to have them jump all over it again.)

Don't tell me to hunt down random third-party tools that none of my coworkers has heard of, that I have to convince them to adopt, and which each solve a little (usually overlapping) bit of the problem so I have to run all of them in series. That way leads to madness, and horrible one-off UIs developed by people more concerned with their tool's special-snowflake analysis algorithm than its output.

Instead, ship an extensible, plugin-based linter _with your language 's toolchain_ (e.g. mix for Elixir, lein for Clojure, etc.) Make its output beautiful. Write one or two simple plugins, and make installing lint-plugins as simple as adding something to the project's dependency file. _Then_ I (and everyone else) will use this stuff.

<comment><USER4> Given that this article is written by/for pvs studio, which works mostly inside visual studio (you can get it to work outside but it seems to require some amount of setup), anyone using visual studio's too chain has a static analysis tool at their disposal. Also, clang has built in static analysis, and I can't speak for gcc.

<comment><USER5> It is simply a conspiracy theory :-) PVS-Studio team doesn't belong to the article somehow. However, we knew about it and took into account.

</comment></comment></comment><comment><USER6> Mainly because most of the tool reports freely mix critical issues with stuff that is just opinionated bullshit (e.g. 80 character length lines).

When the output is as long as your arm and you have pick through it with a fine toothcomb to find the things that matter (and even they aren't necessarily causes of bugs), the whole idea becomes substantially less appealing.

<comment><USER7> Guessing you're referring to the infamous Python PEP-8 limitation. Most linters let you trivially change this line length in a way which either affects the linter permanently, or on a per-project basis.

For example, you can get very fine grained control over pylint by asking it to generate a config file, and reading through it (it's remarkably well commented).

<comment><USER8> You still have to go through the config and enable/disable things for your project. That's the biggest barrier; the lack of good default configs to use for flake8 and pylint. If Google or Microsoft or someone else supplied one, I'd just take it and drop it into every project. One less headache to worry about.

<comment><USER7> That's a remarkably low barrier to entry. Not to mention that the coding standard they are enforcing is based on the official language coding standards - that makes them very reasonable defaults.

<comment><USER8> Yep it is and yet the coworkers don't want to do anything about setting up the infrastructure for static analysis or code checking of any kind. It's an attitude I've encountered at many other companies which is unfortunate.

I think it boils down to being timid and not wanting to impose one standard on others which reduces the importance of coding standards to tabs vs spaces -_-'

</comment></comment></comment></comment></comment><comment><USER9> I use IntelliJ for my java development. It has a lot of analysis built in (whilst you're writing code). I'm not sure if there are that many static analysis checks (Findbugs etc) that are missing outside of those real time checks?

I can often determine whether someone had used Eclipse or IntelliJ; there tend to be a lot fewer analysis warnings if someone has used IntelliJ (out of the box) => perhaps we need this analysis in real time?

<comment><USER10> Good static checkers perform pretty deep analyses you can not (yet) do in real time. Real time checking is of course useful but it is no replacement for a deeper static analysis.

<comment><USER11> A tool like IntelliJ can do deep analysis off-line and in the background and only tell you about the results. This is a good tradeoff between fast feedback and deep analysis.

<comment><USER10> I don't exactly know which kind of analyses IntelliJ performs but assuming that they are comparable to what ReSharper does they are quite different from what for example Coverity looks for.

<comment><USER11> Oh yes, they do some static analysis but it's not even close to the level of Coverity, sorry if my initial comment was misleading.

</comment></comment></comment></comment></comment><comment><USER12> Well, I did it the other way around. When my co-workers drove me crazy. I just launched Findbugs on their Repo/Projects and filed some bugs in the Bugtracker. That is just a matter of 5 Minutes to get them down to the ground and show that they are not that kind of Rockstar-Devs they pretend to be.

But never reveal your secret tools!

Well, and if they get curious about your magical Bug-Finding-Skills, they are easy to convince to use these tools. Its a kind of marketing.

<comment><USER8> using clonedigger for python I found lots of copy/paste code and surprise! where did we find bugs? right in every spot clonedigger pointed out.

</comment></comment><comment><USER13> In one of my recent projects, we used SonarQube. I had actually never worked with that tool before. What bothered me most was how it marked things as errors but did not explain _why_ they were wrong. When it comes to code style, there's a lot of room for personal opinion, and this just led to me arguing with the architect about whether something is a false positive or not. That time could have been spent writing actual, working code or more unit tests.

<comment><USER14> I use SonarQube myself and have generally found its rules to make sense after a quick Google, at most.

Aside from its code formatting rules, which are just silly. They're easy enough to modify or turn off, though, so there's that.

</comment></comment><comment><USER15> I used lint for C some 20 or 25 years ago, then moved to using dynamical languages (Perl, Ruby, JS.) Not much use of static analysis tools for those languages I thought (but for example check [https://www.infinum.co/the- capsized-eight/articles/top-8-too...](https://www.infinum.co/the-capsized- eight/articles/top-8-tools-for-ruby-on-rails-code-optimization-and-cleanup)). I used Java sometimes and took advantage of Netbeans/Eclipse's hints about Java code but I use emacs for everything else (some vim too).

I used jslint on a JavaScript project a few months ago. It catched a few things I overlooked, I learned a couple of things (example: it's useless to define a var inside a loop) but it's also very opinionated and insisted that I write code in a way that pleases its author. I had to refactor working code to make most of the warnings go away. I ended up with something that worked as well as the original but was much more complicated to read (at least for me). I stopped using it and regretted making all those changes to the code. There are some configuration switches but nothing that could make it work for me. I should check jshint but unfortunately jslint primed me against that kind of tools.

I prefer to catch errors with tests and test coverage tools. They must be used anyway and they bend to me, not the other way around.

<comment><USER16> Ditto and likewise - same applies to pylint, which must have been the most opinionated piece of software I've ever used. The trouble is that there's some very useful warnings buried in the opinion spew - warnings which would be an error in other languages.

<comment><USER7> pylint is also trivially easy to configure to omit most of these stylistic warnings.

I say this as someone who took 10 minutes to generate and configure the pylintrc for a new project. After taking this time, pylint has been quite useful.

</comment></comment></comment><comment><USER11> ReSharper (on C# in Visual Studio) and Visual Studio itself do perform static analysis and find out bugs like "possible multiple enumerations of IEnumerable" for you pretty readily, developers also use code coverage tools. Still, good article.

<comment><USER17> ReSharper does lightweight static analysis - I would venture to go as far as to place it into the "code smell analysis" category. Using something like Code Contracts, properly, drastically changes your outlook on static analysis.

 _ReSharper:_ you've enumerated this enumerable multiple times. This if statement is redundant. _Basically stuff within the scope of a single method._

 _Code Contracts:_ you've passed an integer to this function. The function asks that you check that the integer falls into the length of the array that you are also passing in and that it's a multiple of two. Also, the function doesn't promise that it won't return a null value and you are using the result without checking it first. _Basically looks at your entire project and can work out code paths across methods._

There's worlds of a difference. ReSharper is definitely useful, but doesn't come close to what proper static analysis can do. If you like what ReSharper is doing for your codebase I thoroughly recommend having a look at Code Contracts (it's a free Microsoft Research project).

<comment><USER18> The problem with contracts and static analysis is that large projects are often so tangled up by the time that someone (me) applies contracts to them that the static analysis takes days or weeks of execution time.

I add the contracts anyway because they really do help find bugs at run time too and also act as executable comments.

</comment></comment></comment><comment><USER19> It's possible to ship working dynamically-typed code that is just absolutely littered with problems that static-analysis tools complain about (even to the point of being "accidentally correct"). So then you can spend a bunch of hours making it compliant, and it's a big refactor that takes a while for other people to review/qa/approve, meanwhile the product folks are grousing about the time it's taking away from the sprint when it's not a roadmap feature, and as a programmer you might completely believe it's worth the effort to clean up the code, but you're still having to deal with a lot of friction.

<comment><USER20> If you had static analysis in from the start, that wouldn't be a problem.

If you add static analysis to a messy project, define success thresholds based on the current violation count, and/or disable violation categories that you can live with. The rule then should be "don't make it worse, try to make it a bit better when you can", rather than "don't commit anything to main until the checker says it's perfect".

<comment><USER21> Our approach was, fix the really horrible ones now, and don't add new ones that we care about. Then, as we have time, fix the next-most-horrible ones.

</comment></comment></comment><comment><USER22> Very interesting article! I am developing static code analysis tools for Python myself ([https://www.quantifiedcode.com](https://www.quantifiedcode.com)) and have thought a lot about this problem as well. Since we offer a SaaS solution instead of an IDE plugin the boundary conditions are a bit different for us (e.g. we have less strict "realtime" requirements but the feedback time is also longer), but the basic question is the same: Why doesn't everyone use static analysis to improve their code?

After having surveyed most of the available static analysis tools I think that part of the answer is that often the tools simply do not provide enough value for developers, especially those analyzing dynamic languages like Python / Ruby / Javascript (for C++/Java the tooling is much better). We are currently trying to change that by developing a new, data-driven approach to code analysis, which (we hope) should improve the quality of the analyses quite a bit and provide better and more actionable feedback.

Part of the problem is also cultural of course: People have different ideas about what "good code" is and they usually do not like having their code critiqued. Using an automated tool rather than manual code review to check some aspects of code quality can be beneficial though, since it is normally easier to accept harsh feedback from a machine than a human.

<comment><USER23> It looks like it takes quite a while to analyze :-( It just says Waiting for analysis to start (queue position: 46)

</comment></comment><comment><USER24> The Dart analyzer ([https://pub.dartlang.org/packages/analyzer](https://pub.dartlang.org/packages/analyzer)) is used by various IDEs and text editors like IntelliJ, Eclipse, Sublime and Atom to provide semantic autocompletion, documentation and inline warnings/errors.

</comment><comment><USER25> Nice paper from 2013, but I was confused at first how it related to the blog. This paragraph just after the abstract explained, but I missed it the first time through:

       The original article in PDF[1] was published at the NCSU COE
       People site. It was translated and published at our blog
       by the authors' permission. At the end of the article[2], we
       added a short section about the PVS-Studio analyzer, where
       we describe which of the recommendations suggested in the
       article are or aren't implemented in our tool and why.

[1] [http://people.engr.ncsu.edu/ermurph3/papers/icse13b.pdf](http://people.engr.ncsu.edu/ermurph3/papers/icse13b.pdf)

[2] [http://www.viva64.com/en/b/0335/#ID0EU4CK](http://www.viva64.com/en/b/0335/#ID0EU4CK)

<comment><USER26> Yes, I also found this quite confusing. The way this paper was attributed seems quite sloppy. It mentions previous publication at the "NCSU COE People site" and gives a link, but omits mention that this paper was originally presented at the International Conference on Software Engineering (ICSE 2013):

[http://2013.icse-conferences.org/index.html](http://2013.icse- conferences.org/index.html)

These notes shouldn't have been tacked onto the abstract. Instead they should have been clearly separated from the paper itself.

In addition, the PVS Studio team "added a short section" which appears as if it were a modification to the paper, and and it also appears in the augmented table of contents. If one is not reading carefully, or if the section heading has scrolled of the top of the screen, one might be misled into thinking the "added section" was also authored by Johnson et. al. instead of by the PVS Studio team. The "added short section" should also have been published entirely separately.

</comment></comment><comment><USER27> They should. There is no excuse: clang has it for free, Visul Studio, too (Express Editions, since 2013). Also, for Open Source projects Coverity is free.

</comment><comment><USER28> Because neither an advanced type system nor static analysis could catch bugs in program logic?)

The benefits of static typing (complie-time checks) are grossly exaggerated. If the claims were true, Java itself and Java projects would be much less buggy.)

<comment><USER29> This reasoning makes no sense to me. It's like:

Alice: "Doing X will prevent bugs like Y!"

Bob: "Oh, but it does nothing for bugs like Z. I just won't bother at all, then".

Why would you _not_ want to try and remove an entire class of bugs if it were within your power to do so? Just look at all the effort companies like Facebook have poured into exactly this kind of problem with things like Hack and Flow (which make use of OCaml).

<comment><USER30> I think you're missing each other in terms of what static typing you mean.

Haskell or ML style static typing is very useful, completely changes the way you do things and gives you many guarantees.

C or Java static typing is almost useless with regards to bugs (see null) and serves mostly to annoy you. It has many of extra costs of stronger static typing, but gives you very little of the benefits.

<comment><USER31> Static analysis is more than just static typing. Notably, it is possible to leverage static analysis for solid gains in reliability against code bases _without_ having to rewrite them to take advantage of a different typing paradigm.

To repeat, you can harden a code base _without rewriting it_ by using static analysis tools. This is not as true with static typing.

</comment></comment></comment></comment><comment><USER32> I do! For cheapness, Flawfinder, CPPCheck, and Product > Analyze in xcode (thanks clang) (the walkthroughs in xcode of how they deduced the problem are really great, blue arrows everywhere, very neat).

EDIT: Additionally, given the changes in C++11 we should be able to push all the work onto the compiler for checking type problems, remove dangerous operations, dangling pointers, naked new/delete and avoid casting where possible. It would make for bug-free software. I know someone who writes their C++ like it is late 80s C and firstly, it's horrible to read. And secondly, it does really dangerous things.

<comment><USER33> I refactored some code this week that used realloc to allocate memory for a 'vector' (which was a typedef for float*), and then a loop with additional realloc's to allocate memory for each element in that vector, to end up with a matrix 8| This is code that was written in the last few years, too. Welcome to academia :(

</comment></comment><comment><USER34> Additional work. Developers don't like to deal with false positives. If static analysis tools were like a laser. It's more like a shotgun that throws a bunch of shrapnel. Sure it hits some really difficult to find bugs. But developers don't like to have to go through a long list of warnings that aren't really guaranteed to find a valid bug, especially in the applications' context. For example null checking is good, but is it needed if I know it is never really going to be null? Isn't that arguably extra effort of typing and compiling and couple of extra lines to compute? I think we will truly move to better adoption if: 1) We had better configuration for customizing it for our application. 2) More education among developers about what is the correct way to modify the tool's behaviour to aid their work. Help the tool, help them in return. 3) More tools that do specific and general work. Example tools extremely focused on mobile development but also tools that not just help you with correcting design and coding problems but also security vulnerabilities.

</comment><comment><USER35> I ended up writing my own for the bespoke language my company uses internally. The lack of tools like java findbugs had been annoying me.

Luckily the server side 'workflow' language, and the client side templates are all written in xml, so its trivial to parse.

The first version was painfully slow. The second version tracked the range of possible states as it did a depth first search of the AST, and then stored the requirements of each node of the tree before it moved on to the nodes next sibling or parent. That way if another part of the software later called into an equivalent subtree, I could compare the current range of possible states to the requirements of the first node in the subtree and record any mismatches, then move on. It went from taking over a minute to evaluate a fairly trivial app, to parsing the biggest apps my company has ever shipped in 1-2 seconds.

I'm sure the technique is quite common. I wish I could remember if I read about it, or applied it from a different context. In any case the exercise was more useful for learning than as a finished product.

</comment><comment><USER36> The problem I have is that there are many cases where the tools are too defensive. If I want to override the tool, I'd typically have to add some annotation in the code to signal to the tool that I am ok with this warning and don't want to see it again. And I, for some reason, just hate adding style checker and static analysis annotations in my code. It just looks ugly to me.

</comment><comment><USER37> Same reason people use email and code html in editors that are barely more capable then notepad.

Sophisticated solutions tend to fail from time to time when applied. And when they fail they are tossed aside for simpler more resilient ones.

</comment><comment><USER38> I think the reason is fairly simple: even the most evident-no-doubt clear SQL Injection vulnerability found by a SCA tool may never be exploited at all under production (for instance because of a WAF). Then the obvious benefits of static analysis are not that obvious for your employer.

Sometimes we forget companies do not want a perfect code or the best possible well designed software but a product that make them earn money.

My experience is that developers only use those kind of tools if they are forced to by their QA managers of bounded by contract. Programmers usually don't want to fix or track bugs.

<comment><USER39> So, you are saying "Developers don't want to deliver quality."?

If that it true, than I don't want to work with them.

<comment><USER38> I am saying "Developers do not want : 1 - Pay A LOT of money for advanced solutions that are more than AST checkers (hello SonarQube) or big piles of false positives. 2 - Add overhead to their workflows (more than an IDE plugin is harmful, and what happens with those devs not using an IDE?). 3 - <USER2> time on figuring out if the static analysis results make sense or not, one by one.

A typical SCA tool can report hundreds or thousands of occurrences for a certain code base. How are developers going to deal with them?

<comment><USER39> I am from engineering background and not soley a software guy, so forgive me my different view on this topic.

I learned, that every error you can fix early on will cost you about 10x to fix in the next stage.

All the new principles like Agile have not changed that.

<comment><USER30> I think the idea is not that it's not worth to fix errors as soon as possible (which it is), but that static analysis tools provide too many false positives and too many non-errors to be useful.

<comment><USER40> I guess you can combine the points: if you use static analysis from the start / have it configured right then the amount of false positives should stay relatively low.

</comment></comment></comment></comment></comment></comment><comment><USER41> It's hard enough sometimes getting people to turn on compiler warnings.

</comment><comment><USER42> I once was told that any compiler warning message act as static analysis.

I think static analysis is a complex subject, because it can touch some sensitive subject like programming style and other more expert subjects like compiler back end and how the language defines such and such code behavior.

Static analysis should be made more mainstream, it would be such a great way to teach everybody how to write better code, including and especially students.

</comment><comment><USER18> For the same reason my colleagues won't use contracts: they get in the way of the job you are trying to do today. I have explained over and over again that they are an investment that pays off by helping to avoid having to fix bugs in the future but most people discount the value of a lack of future bugs so heavily that they feel that the convenience of not being nagged by the machine is worth it.

</comment><comment><USER43> We do. (some of us do, anyways).

> Mike: "Clang is my favorite. Its built into the compiler. You don't have to > invoke anything special."

I agree with Mike. I've also used Coverity Static Analysis and various other tools. Some of them are better than others.

Also, "-Wall -Wextra" or "-Weverything" are fantastic for a simple first-pass static analysis.

</comment><comment><USER44> I don't know tools for Scala/Haskell making me excited, that's why. Would be happy to see something here.

</comment><comment><USER5> Parallel Reddit discussion: [https://www.reddit.com/r/programming/comments/3b67pi/why_don...](https://www.reddit.com/r/programming/comments/3b67pi/why_dont_software_developers_use_static_analysis/)

</comment><comment><USER45> no button in the IDE :)

<comment><USER46> Have you used one of the popular IDEs recently (Visual Studio, Xcode etc)? It's built into most these days.

Everybody I know who actually uses static analysis does so because it's built into their IDE and very easy to use.

<comment><USER47> In Xcode at least there's a 'Build & Analyze" command (cmd-shift-b).

What I like to do is to force a deep analyze on every build. Takes a little longer to build but at least I catch some bugs when I introduce them.

<comment><USER4> My compile is already 10 minutes, I don't want to add a static analysis pass to every build thanks.

<comment><USER32> Why is your compile 10 minutes? Mine is about 20 seconds for ~82000 lines of C++ (not comments), with 43000 header lines and 7669 lines of C (according to cloc), and that's on a 2008 quad-core Xeon Mac Pro.

Do you not parallel build?

<comment><USER4> Parallel build with 40 cores(2x intel xeons), 32GB ram using Visual Studio. According to cloc, we have 11k C++ files. 18k C/c++ header, with 4.5M lines of C++, and 1.5M C/C++ Header, and 700K C. We use a unity build to speed it up (to 10 minutes) without it, it's roughly 40 minutes.

<comment><USER32> Haha my project is tiny, yours is mental haha that's bonkers

</comment></comment></comment></comment></comment></comment></comment><comment><USER48> As a Python programmer, I live and die over Pylint. Yes it's annoying to configure it, but it saves me hours every _day_. The two biggest benefits are:

\- syntax or undefined variables in exception handlers

\- showing which modules are no longer used, so we can have a clearer import block

<comment><USER8> could you share your pylintrc for others to learn from?

</comment></comment><comment><USER49> Static analysis tools are great for simple bugs. However, most compiler tools already fix these simple bugs. If there is a class of bugs not addressed by a compiler, most developers have a script to catch these. It may not find the harder to detect bugs (such as those arising out of nested calls), but it gets the job done. The complex tools pushed out by the static analysis guys usually has a high false positive rate that at first becomes annoying to use and later is completely ignored.

As for more complex bugs, most developers are aware these bugs exist but do not want to fix them immediately. The reason is on many occasions, these bugs represent some bigger problem in the code base that requires significant re- factoring. And applying the quick fix as suggested by the code analysis tool simply buries these problems instead of fixing them the right way.

<comment><USER5> Yes, static analyzers mostly catch simple bugs. But it doesn't really matter if a bug is simple or complicated. You see, programmers believe they never make or make very few simple bugs: [http://www.viva64.com/en/b/0116/](http://www.viva64.com/en/b/0116/)

But they are wrong. They do make quite a lot of them. Here's, for instance, a bug database we have collected and keep updating: [http://www.viva64.com/en/examples/](http://www.viva64.com/en/examples/).

Moreover, some bugs can take quite a while to find, despite being simple. Here's a nice example:

The conclusion is: the bug we had wasted about 50 hours to track was detected at once with the first run of the analyzer and fixed in less than an hour! Source: [http://www.viva64.com/en/b/0221/](http://www.viva64.com/en/b/0221/)

True, false positives aren't good, but they are not that much trouble. Static analysis tools provide numbers of means to suppress them. At least, we in PVS- Studio do have a lot of false positive suppression mechanisms. But it's a long story, so you'd better refer to the documentation.

</comment></comment><comment><USER50> Because we already have a better tool for finding bugs: Production users

<comment><USER5> Therac-25: [https://en.wikipedia.org/wiki/Therac-25](https://en.wikipedia.org/wiki/Therac-25)

</comment></comment><comment><USER51> cl : Command line warning D9040: ignoring option '/analyze'; Code Analysis warnings are not available in this edition of the compiler

Ooops.

</comment><comment><USER52> This is something that Checkmarx.com (checkmarx.com) does. They recently received substantial investments and seem to be popular.

</comment><comment><USER53> Nice advertisement.

It's Alt+F11 in Visual Studio. Between that and Resharper, life is good.

<comment> PVS-Studio Not so simple. Review John Carmack article: [http://gamasutra.com/view/news/128836/InDepth_Static_Code_An...](http://gamasutra.com/view/news/128836/InDepth_Static_Code_Analysis.php)</comment></comment>